Difference between revisions of "Awk/scripts"

Revision as of 05:59, 6 August 2007 (view source) Christoph (Talk | contribs) ← Older edit		Revision as of 06:02, 6 August 2007 (view source) Christoph (Talk | contribs) (→‎Built-in functions) Newer edit →
Line 81:		Line 81:
	===Built-in functions===		===Built-in functions===
		+	see: [[Awk#String functions|String functions]] for full list.
	Arithmetic functions include: <code>sqrt</code>, <code>log</code>, <code>exp</code>, and <code>int</code>		Arithmetic functions include: <code>sqrt</code>, <code>log</code>, <code>exp</code>, and <code>int</code>

---

Revision as of 06:02, 6 August 2007

see: Main Awk article.

Basic examples

Note: The examples in this section have been taken directly from the original Awk paper.^[1]

Printing

• Print all input lines whose length exceeds 72 characters:

length > 72

• Print all lines with an even number of fields:

NF % 2 == 0

• Replace the first field of each line by its logarithm:

{ $1 = log($1); print }

• Print the third and second columns of a table in that order:

{print $3,$2}

• Print all input lines with an A, B, or C in the second field:

$2 ~ /A|B|C/

• Print all lines in which the first field is different from the previous first field:

$1 != prev { print; prev = $1 }

• Print each record preceded by the record number and the number of fields:

{print NR, NR, $0}

• Write the first field, $1, on the foo foo1, and the second field on the file foo2:

{print $1>"foo1"; print $2>"foo2"}

• Append the output to the file foo:

print $1>>"foo"

• Use the contents of field 2 as a file name:

print $1>$2

• Print $1 as a floating point number 8 digits wide, with two after the decimal point, and $2 as a 10-digit long decimal number, followed by a newline:

printf "%8.2f %10ld\n", $1, $2

BEGIN and END

Note: The special pattern BEGIN matches the beginning of the input, before the first record is read. The pattern END matches the end of the input, after the last record has been processed. BEGIN and END thus provide a way to gain control before and after processing, for initialization and wrapup.

• As an example, the field separator can be set to a colon by:

BEGIN { FS = ":" }
... rest of program ...

• Or, the input lines may be counted by:

END { print NR }

If BEGIN is present, it must be the first pattern; END must be the last if used.

Regular expressions

• Print all lines which contain any occurance of the name "smith":

/smith/

• () (parentheses) — for grouping;
• | (pipe) — for alternatives;
• + — for "one or more"; and
• ? — for "zero or more"

• Print all lines which contain any of the names "Aho", "Weinberger", or "Kernighan", whether capitalized or not:

/[Aa]ho|[Ww]einberger|[Kk]ernighan/

• Match any string of characters enclosed in slashes:

/\/.*\//

• Print all lines where the first field matches "john" or "John" (also matches "Johnson", etc.):

$1 ~ /[jJ]ohn/

• Match exactly "john" or "John":

$1 ~ /^[jJ]ohn$/

Relational expressions

The relational operators are: <, <=, ==, !=, >=, and >.

• Select lines where the second field is at least 100 greater than the first field:

$2 > $1 + 100

• Print lines with an even number of fields:

NF % 2 == 0

• Select lines that begin with an "s", "t", "u", etc.:

$1 >= "s"

• Perform a string comparison (note: In the absence of any other information, fields are treated as strings):

$1 > $2

Combination of patterns

A pattern can be any boolean combination of patterns, using the operators || (or), && (and), and ! (not).

• Select lines where the first field begins with "s", but is not "smith":

$1 >= "s" && $1 < "t" && $1 != "smith"

Pattern ranges

• Print all lines between start and stop:

/start/,/stop/

• Perform an action for lines 100 through 200:

NR == 100, NR == 200 { ... }

Built-in functions

see: String functions for full list.

Arithmetic functions include: sqrt, log, exp, and int

• Print each record, preceded by its length:

{print length, $0}

• Print the length of its argument (here the length of the input line):

{print length($0), $0}

• Print lines whose length is less than 10 or greater than 20:

length < 10 || length > 20

The function substr(s,m,n) produces the substring of s that begins at position m (origin 1) and is at most n characters long. If n is omitted, the substring goes to the end of s

The function index(s1,s2) returns the position where the string s2 occurs in s1, or zero if it does not.

The function sprintf(f,e1,e2,...) produces the value of the expressions e1, e2, etc., in the printf format specified by f

• Set x to the string produced by formatting the values of $1 and $2:

x = sprintf("%8.2f %10ld", $1, $2}

Variables, expressions, and assignments

The arithmetic operators (done internally in floating point) are: +, -, *, and % (mod). Increment ++ and decrement -- operators are also available, and so are the assignment operators: +=, -=, *=, /= and %=.

• Assign 1 (a number) to x:

x=1

• Assign "smith" (a string) to x:

x="smith"

• Assign 7 (a number) to x:

x = "3" + "4"

• Print the sums of the first two fields:

    {s1 += $1; s2 += $2}
END {print s1,s2}

Field variables

• Replace the first field with a sequence number:

{$1 = NR; print}

• Accumulate two fields into a third:

{$1 = $2 + $3; print $0}

• Assign a string to a field (here, replace the third field by "too big" when it is):

{if($3 > 1000)
    $3 = "too big"
 print
}

• Field references may be numerical expressions:

{print $i, $(i+1), $(i+n)}

• Field treated as strings:

if($1==$2) ...

• Split the string s into array[1],...,array[n] (note: If the sep argument is provided, it is used as the field separator; otherwise FS is used as the separator):

n = split(s,array,sep)

String concatenation

• Return the length of the first three fields:

length($1 $2 $3)

• Print the two fields separated by " is ":

print $1 " is " $2

Arrays

• Assign the current input record to the NR-th element of the array x:

x[NR] = $0

• Increment counts for the named array elements, and print them at the end of the input:

/apple/  {x["apple"]++}
/orange/ {x["orange"]++}
END      {print x["apple"], x["orange"]}

Flow-of-control statements

The flow-of-control statements are: if-else, while and for. Also, the break statement causes an immediate exit from an enclosing while or for; the continue statement causes the next iteration to begin. The next statement causes Awk to skip immediately to the next record and begin scanning the patterns from the top. The statement exit causes the program to behave as if the end of the input had occurred.

• Print all input fields one per line:

i = 1
while(i <= NF) {
   print $i;
   ++i
}

• Print all input fields one per line (same as above):

for(i=1; i<=NF; i++)
   print $i

• Perform statement with i set in turn to each element of array (an associative array):

for(i in array)
   statement

Other examples

Note: Taken directly from Patrick Hartigan's awk page. The # is the comment character for awk. 'field' means 'column'.

# Print first two fields in opposite order:
  awk '{ print $2, $1 }' file

# Print lines longer than 72 characters:
  awk 'length > 72' file

# Print length of string in 2nd column
  awk '{print length($2)}' file

# Add up first column, print sum and average:
       { s += $1 }
  END  { print "sum is", s, " average is", s/NR }

# Print fields in reverse order:
  awk '{ for (i = NF; i > 0; --i) print $i }' file

# Print the last line
      {line = $0}
  END {print line}

# Print the total number of lines that contain the word Pat
  /Pat/ {nlines = nlines + 1}
  END {print nlines}

# Print all lines between start/stop pairs:
  awk '/start/, /stop/' file

# Print all lines whose first field is different from previous one:
  awk '$1 != prev { print; prev = $1 }' file

# Print column 3 if column 1 > column 2:
  awk '$1 > $2 {print $3}' file

# Print line if column 3 > column 2:
  awk '$3 > $2' file

# Count number of lines where col 3 > col 1
  awk '$3 > $1 {print i + "1"; i++}' file

# Print sequence number and then column 1 of file:
  awk '{print NR, $1}' file

# Print every line after erasing the 2nd field
  awk '{$2 = ""; print}' file

# Print hi 28 times
  yes | head -28 | awk '{ print "hi" }'

# Print hi.0010 to hi.0099 (NOTE IRAF USERS!)
  yes | head -90 | awk '{printf("hi00%2.0f \n", NR+9)}'

# Print out 4 random numbers between 0 and 1
  yes | head -4 | awk '{print rand()}'

# Print out 40 random integers modulo 5
  yes | head -40 | awk '{print int(100*rand()) % 5}'

# Replace every field by its absolute value
  { for (i = 1; i <= NF; i=i+1) if ($i < 0) $i = -$i print}

# If you have another character that delimits fields, use the -F option
# For example, to print out the phone number for Jones in the following file,
# 000902|Beavis|Theodore|333-242-2222|149092
# 000901|Jones|Bill|532-382-0342|234023
# ...
# type
  awk -F"|" '$2=="Jones"{print $4}' filename

# Some looping commands
# Remove a bunch of print jobs from the queue
  BEGIN{
	for (i=875;i>833;i--){
		printf "lprm -Plw %d\n", i
	} exit
       }

# Formatted printouts are of the form printf( "format\n", value1, value2, ... valueN)
#   e.g. printf("howdy %-8s What it is bro. %.2f\n", $1, $2*$3)
#	%s = string
#	%-8s = 8 character string left justified
# 	%.2f = number with 2 places after .
#	%6.2f = field 6 chars with 2 chars after .
#	\n is newline
#	\t is a tab

# Print frequency histogram of column of numbers
$2 <= 0.1 {na=na+1}
($2 > 0.1) && ($2 <= 0.2) {nb = nb+1}
($2 > 0.2) && ($2 <= 0.3) {nc = nc+1}
($2 > 0.3) && ($2 <= 0.4) {nd = nd+1}
($2 > 0.4) && ($2 <= 0.5) {ne = ne+1}
($2 > 0.5) && ($2 <= 0.6) {nf = nf+1}
($2 > 0.6) && ($2 <= 0.7) {ng = ng+1}
($2 > 0.7) && ($2 <= 0.8) {nh = nh+1}
($2 > 0.8) && ($2 <= 0.9) {ni = ni+1}
($2 > 0.9) {nj = nj+1}
END {print na, nb, nc, nd, ne, nf, ng, nh, ni, nj, NR}

# Find maximum and minimum values present in column 1
NR == 1 {m=$1 ; p=$1}
$1 >= m {m = $1}
$1 <= p {p = $1}
END { print "Max = " m, "   Min = " p }

# Example of defining variables, multiple commands on one line
NR == 1 {prev=$4; preva = $1; prevb = $2; n=0; sum=0}
$4 != prev {print preva, prevb, prev, sum/n; n=0; sum=0; prev = $4; preva = $1; prevb = $2}
$4 == prev {n++; sum=sum+$5/$6}
END {print preva, prevb, prev, sum/n}

# Example of defining and using a function, inserting values into an array
# and doing integer arithmetic mod(n). This script finds the number of days
# elapsed since Jan 1, 1901. (from http://www.netlib.org/research/awkbookcode/ch3)
function daynum(y, m, d,    days, i, n)
{   # 1 == Jan 1, 1901
    split("31 28 31 30 31 30 31 31 30 31 30 31", days)
    # 365 days a year, plus one for each leap year
    n = (y-1901) * 365 + int((y-1901)/4)
    if (y % 4 == 0) # leap year from 1901 to 2099
        days[2]++
    for (i = 1; i < m; i++)
        n += days[i]
    return n + d
}
    { print daynum($1, $2, $3) }

# Example of using substrings
# substr($2,9,7) picks out characters 9 thru 15 of column 2
{print "imarith", substr($2,1,7) " - " $3, "out."substr($2,5,3)}
{print "imarith", substr($2,9,7) " - " $3, "out."substr($2,13,3)}
{print "imarith", substr($2,17,7) " - " $3, "out."substr($2,21,3)}
{print "imarith", substr($2,25,7) " - " $3, "out."substr($2,29,3)}

References

1. ↑ Aho AV, Kernighan BW, Weinberger PJ (1978). "Awk - A pattern scanning and processing language". Second Edition, Bell Laboratories, 8 pp.

External links

• AWK Book Code — contains all the programs from The AWK Programming Language, by Aho, Kernighan and Weinberger (Addison-Wesley, 1988). They have been packed by the bundle program found on page 81, and can be unpacked by the unbundle on page 82, also included here. A text editor will also do this pretty easily.
• Patrick Hartigan's awk page