IR Guest Lecture: Neural Networks for Information Retrieval | by Mostafa Dehghani
Summary: When applying NNs to IR, we have to deal with the problem that in IR the supervised data sets are extremely small compared to what data-hungry NNs normally need. There are many ways to deal with data scarcity, and Mostafa argues that using weak classifiers to produce pseudo-labels is a very good option.
Works at Google Brain.
From lexical to semantic matching using more dense word representations (in the wake of word2vec).
Traditional IR estimates relevance using lexical matches of query terms in the document.
Both lexical and semantic modelling can be modelled with neural networks.
Semantic Matching
Pro
- Synonyms and related words
- Robustness to spelling variations
- Catches cases where lexical matching fails
Con:
- Computationally expensive
Deep Structured Semantic Model (DSSM)
Huang et al, 2013.
- Start: term vector (500k params)
- Word hashing (30k)
- Multi-layer non-linear projection (300)
- Semantic features (128)
The step from 30k to 300 is most interesting: the multi-layer non-linear projection uses a Siamese network with contrastive loss.
Interaction Matrix Based-Approaches
Make query words x document words matrix.
So instead of feeding the network with separate representations for query and documents, you can input the whole "interaction" matrix. Where i-th words in source sentence is compared with j-th word in target sentence.
Comparisons can be both lexical or semantic.
This is a fine-grained approach, but this has obviously too high cost.
Lexical and Semantic Duet
Some terms may be very rare and be rarely seen during training. Having little training samples is a large problem for neural networks.
At the same time, the patterns of rare words in documents is very important for relevance estimation.
Mitra et al. 2017: train a dual model with two "towers", one local network and one distributed network. Input to model is interaction matrix.
You feed a presence-absence interaction matrix to the local network.
The distributed network learns vector representations and compares similarity of those vectors.
Dealing with data scarcity
Neural nets with imperfect supervision
NNs are super data hungry. In IR the training data sets were just too small for usage in neural networks. The TREC datasets have maybe 50 topics with 10.000 relevance judgments. That's nothing in terms of neural networks.
Two ideas:
- Using distant or heuristic supervision
- Using incidental signals that are independent of the task but correlate with the target task.
- E.g. label polar bears on north or south pole based on how light the sky is
- This is very problematics and not robust, but does show a possibility for classifying in absence of labels
Dealing with data scarcity in practice
- Bootstrapping, self-supervised feature learning, data augmentation
- Transfer learning to generalize knowledge across domains/tasks
- Using active learning and response-based supervision (feedback from environment interaction)
- Using structured prior knowledge
- Indirect supervision: learn a companion task
- Zero/one/few-shot learning: learning models that can be extended to new tasks with just a few examples
- exploiting noisy and inaccurate labels
- e.g. crowd-source labeling data; lower-quality but on the larger scale of things that perhaps averages out
- use a biased or weak classifier to produce the labels (pseudo-labels)
- injecting inductive biases into algorithms
- encoding modeling assumptions to generalize better to unobserved data
BERT uses self-supervised representation learning; no idea what that is.
Labeling programmatically
If too little labels are available, we can programmatically with a weak classifier.
Using heuristic based methods as a weak annotator to generate pseudo-labels for a large set of unlabeled instances. (They are not labels because they are not accurate nor guaranteed to be).
Then train a network to be trained on a weak supervision signal.
This seems sketchy theoretically, but works pretty well in practice!
We can also produce relevance judgments this way, e.g. using BM25's top 100 results to label documents as relevant or irrelevant (pseudo-relevance feedback).
From here discussion of one of Mostafa's papers.
If they ask a question about this, look at the slides and the paper.
Things like "pair-wise" and "point-wise" models are introduced in IR-8-Learning_to_rank.md .
Take home messages
- Define an objective which enables your model to go beyond he imperfection of the weakly annotated data
- We do in fact do this by learning to rank instead of computing calibrated scores!
- Let the network decide about the representation. Feeding the network with featurized input kills the model "creativity"!
- Earlier Mostafa made a remark that much preprocesing on text is not needed for NNs when you have enough data; preprocessing can be seen as a way of imposing useful bias, but in some cases you may not want to do this.
- This is why both dense and sparse vector representations do not replicate BM25 performance.
- Feature engineering can also hurt generalization when the network has "limited capacity"
- By just seeing the individual local instances of the data, the network learns a global statistic when using embeddings.
- Slide 34: learned weights have a very high correlation (0.7,0.8) with IDF scores!
- E.g. IDF can be picked up as well using local examples only
- Worth it to learn embeddings instead of using pretrained ones, but this requires much much data; but this is possible using weak supervision!
- I.e. the benefit of the NN (its non-linearity etc.) is that it can also learn useful representations (requires deep networks)
- Weak supervision with much data is preferable over full supervision with little labeled data
- When using weak supervision, it may be useful to do meta-learning on the label's quality.