Learning to Rank | Section 10.4 by Zhai

The Rocchio pseudo-relevance feedback algorithm discussed is one of the first ML approaches used in IR.

BM25 ranks based on some features; PageRank ranks on some others in another context. We can combine many features for ranking with machine learning. This can improve accuracy, but also support other goals, such as improving the robustness of the ranking function so it's not easily manipulated by spammers by manipulating one or two features.

Features can be many things:

  • content-based
  • ranking scores
  • query likelihood
  • link-based scores
  • link-descriptions
  • etc.

To combine the features into a single relevance score we need to learn appropriate weights for each one of them.

Training data input:

  • relevance judgements on query-document pairs; either by real users or approximated by clickthrough data.
  • (query, document, judgement) tuples

Depending on what assumption we make, we can formulate different machine learning models. E.g. if we assume that the relevance of a document with respect to the query is a linear combination of features, we can formulate a logistic regression model, which outputs the probability P(R=1|d,q).

We assume that the log odds can be expressed by some weighted linear combination of features.

$$ log \frac{ P(R=1|q,d)}{1-P(R=1|q,d)} = \beta_0 + \sum_{i=1}^n \beta_i X_i$$

It is common to express the probability of relevance in terms of this logistic function so it is clamped between 0 and 1. This way we can connect arbitrary combinations of feature scores.

Q: Why do we use logistic regression instead of linear regression as a scoring function in learning to rank?

A: because we want a probability of relevance in the range [0,1]. The use of the logarithm maps the linear combination of features onto a probability.

Then we ask: for which parameters $\beta$ (feature weights) do we maximize the P(q,d,R=1) or relevant documents as well as maximize the P(q,d,R=0) of irrelevant documents.

Note! Such a regression optimization objective is not necessarily directly related to a retrieval measure and thus may not result in a good ranking (cf. p. 211).

Development of machine learning in IR

Summary of points from last paragraph on p. 212

  • Contemporary algorithms are "driven by the availability of massive amounts of training data in the form of clickthroughs."
    • Clickthrough data is a relevance signal
    • This data was not massively available before the Web
    • And because it's available from the Web, it's possible to acquire it in massive amounts
  • Web search also drove the need to combine more features because the web simply offers way more features to be used.
  • The Web also poses its own downsides (spam!), which even stronger reinforces the need for more robust rankings (which can be achieved by combining more features)