TF-IDF scoring
We should treat matches on non-frequent terms as more valuable than ones on frequent terms, without disregarding the latter altogether. The natural solution is to correlate a term's matching value with its collection frequency. (Karen Spark Jones, 1972)
$$IDF(w) = log( \frac{M+1}{df(w)})$$
Where M is the total number of documents, and df(w) total number of documents containing w.
A common variant leaves the "+1" out.
Q: Explain the rationale behind the TF.IDF weighing scheme.
Q: Why is a logarithm used in the DF part of the TF.IDF formula?
Case distinction: A) When a word occurs in all documents, we have (M+1)/M -> 1, with log(1)=0. So extremely frequent terms (i.e. with high df) are heavily penalized. B) When a word is extremely rare, df=1, then with M=100.000 we have log(100.000) is 5 (assuming base 10 for simplicity here). If we wouldn't have taken the log here, then we would completely dominate the TF in TF.IDF.
"This makes sense when the term occurs so frequently that it’s unlikely to differentiate two documents’ relevance (since the term is so common)" (Zhai, Ch6. p.101).
Using the log also penalizes very frequent terms most, expressing that frequent terms should have a very low contribution in determining relevance. When it concerns a comparison between infrequent and very infrequent terms, the difference in penalization/reward is smaller, because it's OK here if TF can do its work. A simpler way to say this is: the point is not to heavily promote the most rare words.