Unix for poets - Notes

@unix @cli

See: http://doc.cat-v.org/unix/for-poets/kwc-unix-for-poets.pdf

Goal: use basic UNIX tools to process text and extract information

What are the tools?

  • grep for pattern search
  • sort
  • uniq -c count duplicates (uniq: report or omit repeated lines)
  • tr translate characters
  • wc word count
  • sed edit string
  • awk simple programming language
  • cut
  • paste
  • comm
  • join

1: Counting words

a) tokenize with tr b) sort with sort c) count duplicates with uniq -c

Solution: tr -sc 'A-Za-z' '\012' < input | sort | uniq -c

The tr command translates some input characters specified by given pattern into some other specified character. We read the input from the file input.

The -c flag stands for complement, meaning that now every character is translated that is not part of the set specified by the pattern. So it says: replace everything that is not an alphabetical character to a newline character, thereby effectively tokenizing. '\012' equals '\n' (might make a small difference on some very old systems I guess).

The -s flag stands for single occurrence: "replaces repeated characters listed in the set1 with single occurrence". We are dealing with complements of the basic alphabet. Take for example a space. If we match two spaces in a row, we do not want to translate both spaces into a newline. Instead, first we "squash" the two spaces into a single space, and then translate that to a single newline.

< reads from input file, whereas > writes to an output file.

To see the first 5 results, pipe sed 5q

Why sort before running uniq -c? Efficiency?

Sorting on word frequency instead of alphabet

To again sort on frequency, we can pipe the whole command into another sort command sort -n -r Since the frequencies are listed on the beginning of the line, this will sort of word frequency.

Solution: tr -sc 'A-Za-z' '\012' < input | sort | uniq -c | sort -n -r

Make the count case insensitive

Can be achieved by translation all upper case letters to lowercase, before doing the previous command: tr 'A-Z' 'a-z' < input.

Solution: tr 'A-Z' 'a-z' < input | tr -sc 'A-Za-z' '\012' | sort | uniq -c | sort -n -r

Counting lines instead of words

Simply use the l-flag: wc -l foo

Sorting