Zipf's Law

@IR @text_mining

The frequency of any word is inversely proportional to its rank in the frequency table. In the English language, about 10-15% of all word occurrences are accounted for by the top four most frequent words. The top 50 words account for about 35-40% of all word occurrences.

E.g. top terms are "the", "of", "and", "to", "in" etc.

The second term will be about 1/2 as frequent as the most frequent term. The third term will be about 1/3 as frequent etc.

Zipf originally thought this was a property relating to human cognition. It turns out that if you give a monkey a keyboard with a space bar 20% of the keyboard, this monkey will also be able to reproduce the Zipfian distribution. (Anecdote from Arjen de Vries, see here)