运用之妙，存乎一心

Introduction

Awk is a program that you can use to select particular records in a file and perform operations upon them. It is installed on Linux and Mac by default.

Awk is an interpreting program language with a simple programming paradigm: find a pattern in the input and then perform an action, which often reduce complex or tedious data manipulations to a few lines of code. Since it’s easy and powerful, why not using it to avoid tedious openning/searching/matching/changings using one or few lines of awk codes?

This blog is for my own notes and records, mainly from Dougspeed and The GNU Awk User’s Guide. The awk is far more powerful than the content/examples listed in this blog.

Basic composition of an awk program: rule-action

When you run awk, you specify an awk program that tells awk what to do. The program consists of a series of rules (it may also contain user defined functions). Each rule specifies one pattern to search for and one action to perform upon finding the pattern. By default, the action of awk will be the {print} (or equally {print $0}) if not specified.
Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in braces to separate it from the pattern. Newlines usually separate rules. Therefore, an awk program looks like this:\

1
2
3
4
5
pattern1 { action1 }
pattern2 { action2 }
pattern3 { action3 }
pattern4 { action4 }
…

Running awk

No input files

awk applies the program to the standard input, which usually means whatever you type on the keyboard. This continues until you indicate end-of-file by typing Ctrl-d. \

1
2
#No input files
awk 'program'

Input with files

1
2
3
4
5
6
7
8
9
10
# Input with single file
awk 'program' input-file1
awk <input-file1 'program'
zcat xxx.gz | awk 'program'

# Input with multiple files
# The 'dash' argument indicate awk to read its standard input from stream '|'
awk 'program' input-file1 input-file2 …
zcat xxx.gz | awk 'program' input-file2 -

Read ‘program’ from a script

1
2
#When the program is long,
awk -f program-file input-file1 input-file2 …

System Variables and Functions

System variables in awk

The awk program defines a number of special variables that can be referenced or reset inside a program.\

System functions in awk

The awk program defines a number of special variables that can be referenced or reset inside a program.\

Regular Expressions in awk

AWK is very powerful and efficient in handling regular expressions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
echo -e "cat\nbat\nfun\nfin\nfan" | awk '/f.n/'

echo -e "This\nThat\nThere\nTheir\nthese" | awk '/^The/'

echo -e "knife\nknow\nfun\nfin\nfan\nnine" | awk '/n$/'

echo -e "Call\nTall\nBall" | awk '/[CT]all/'

echo -e "Call\nTall\nBall" | awk '/[^CT]all/'

echo -e "Call\nTall\nBall\nSmall\nShall" | awk '/Call|Ball/'

echo -e "Colour\nColor" | awk '/Colou?r/'

echo -e "ca\ncat\ncatt" | awk '/cat*/'

echo -e "111\n22\n123\n234\n456\n222"  | awk '/2+/'
#Grouping
echo -e "Apple Juice\nApple Pie\nApple Tart\nApple Cake" | awk '/Apple (Juice|Cake)/'

#substitute
echo "apple apple\npineapple apple\n" | awk 'sub(/apple/, "nut")'
echo "apple apple\npineapple apple\n" | awk 'gsub(/apple/, "nut")'

# Regx with variables
## awk can match against a variable if you don't use the // regex markers
## need to build up the required regex as a string
echo "apple apple\npineapple apple\n" | awk 'BEGIN{r = "eapp"} $0~r'
# Using variables from bash terminal
rex=eapp
echo "apple apple\npineapple apple\n" | awk -v r="$rex" '$0~r'

Examples by Practice

Find the intersect (overlap) of two files

1
2
3
awk '{if(NR==FNR){arr[$1];next}}($1 in arr){print $1}' file1.txt file2.txt
# Or 
awk '(NR==FNR){arr[$1];next}($1 in arr){print $1}' file1.txt file2.txt

Get details of the overlaps

1
awk '(NR==FNR){arr[$1]=$2;next}($1 in arr){print "SNP:",$1, "P-Value1:",arr[$1], "P-Value2:", $2}' file1.txt file2.txt

Remove duplicates

Remove duplicates in file1.$1, keep only the first record.
Step by step intepretations:

1. check \$1 in seen, if not: seen[\$1]=0;
2. check !seen[\$1], if TRUE: {print \$0};
3. seen[\$1]++;

1
awk '(!seen[$1]++)' file1.txt >file1_rmDuplicate.txt

An example of awk script

From Steve\

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
BEGIN {
    FS = "[. ]"
    OFS = "."
}

FNR == NR {
    domain[$1] = $0
    next
}

FNR < NR {
    if ($2 in domain) {
        for ( i = 2; i < NF; i++ ) {
            if ($i != "") {
                line = (line ? line OFS : "") $i
            }
        }
        total[line] += $NF
        line = ""
    }
}

END {
    for (i in total) {
        printf "%s\t%s\n", i, total[i]
    }
}

• Previous
  Rmarkdown chunk code options
• Next
  git multiprofile settings