Unix for poets - Notes
@unix @cli
See: http://doc.cat-v.org/unix/for-poets/kwc-unix-for-poets.pdf
Goal: use basic UNIX tools to process text and extract information
What are the tools?
grepfor pattern searchsortuniq -ccount duplicates (uniq: report or omit repeated lines)trtranslate characterswcword countsededit stringawksimple programming languagecutpastecommjoin
1: Counting words
a) tokenize with tr
b) sort with sort
c) count duplicates with uniq -c
Solution: tr -sc 'A-Za-z' '\012' < input | sort | uniq -c
The tr command translates some input characters specified by given pattern into some other specified character.
We read the input from the file input.
The -c flag stands for complement, meaning that now every character is translated that is not part of the set specified by the pattern.
So it says: replace everything that is not an alphabetical character to a newline character, thereby effectively tokenizing.
'\012' equals '\n' (might make a small difference on some very old systems I guess).
The -s flag stands for single occurrence: "replaces repeated characters listed in the set1 with single occurrence". We are dealing with complements of the basic alphabet. Take for example a space. If we match two spaces in a row, we do not want to translate both spaces into a newline. Instead, first we "squash" the two spaces into a single space, and then translate that to a single newline.
< reads from input file, whereas > writes to an output file.
To see the first 5 results, pipe sed 5q
Why sort before running uniq -c? Efficiency?
Sorting on word frequency instead of alphabet
To again sort on frequency, we can pipe the whole command into another sort command sort -n -r
Since the frequencies are listed on the beginning of the line, this will sort of word frequency.
Solution: tr -sc 'A-Za-z' '\012' < input | sort | uniq -c | sort -n -r
Make the count case insensitive
Can be achieved by translation all upper case letters to lowercase, before doing the previous command:
tr 'A-Z' 'a-z' < input.
Solution:
tr 'A-Z' 'a-z' < input | tr -sc 'A-Za-z' '\012' | sort | uniq -c | sort -n -r
Counting lines instead of words
Simply use the l-flag: wc -l foo