Conversational Search | IR Guest Lecture by Jeff Dalton

Outline:

  • Motivation and history
  • What is 'conversational search'
  • Understanding and ranking conversational language
  • Future directions

Note that although conversational search is a good match with spoken input/output, the actual searching still works with indices and is not inherently tied to the spoken modality.

Motivation

It took Google a long time to care about question answering. The rise of mobile search changed that, because people often Google a quick question on phone (the majority is now mobile).

Rise of conversational voice assistants; voice search accounts for about 1/3 of all mobile queries. Currently: rise of "hearables" (e.g. from Apple) that include smart speaker devices.

The top Alexa skills are currently " single turn" actions, e.g. commands like " play song X". We want to move to more conversational interactions.

History

CHAT-80 (Pereira 1980)

  • Proof-of-concept natural language interface to a database (NLIDB )
  • Could answer questions about geography
  • Implemented in Prolog
  • hand-built lexicon & grammar

Two axes: coverage and understanding.

Coverage: brittle, narrow coverage <-> robust, broad coverage

Understanding: fuzzy, partial vs precies, complete

The old systems had narrow coverage; Google has broad coverage but often

Classical Voice Assistant Architecture

Speech Recognition -> Natural Language Understanding -> ( Dialogue Manager <-> Task Manager ) -> (Natural Language Generation) <- TSS ->

Q: Why is TTS bi-directional?

Progression over time

Comparison of Siri performance from 2014 until now.

When using conversational search, you could have a request like "Find restaurants near me" which Siri 2014 handled.

But if you would then ask a follow-up question like "Are any of them Italian", it is likely that we will instead be forwarded to pages that offer Italian translations, in this case of "Are any of them" or "any". That is, it falls back to Web Search. "Tell me about the second one" also fails ("Sorry, can't do that"). In itself this is not so strange, but in the context of a conversation it is (as it ignores the context of the preceding question).

Siri 2016 handled follow-ups like "Tell me more about the first one". "Are any of them Italian" also applies a filter.

NLP/IR background

Neural nets are changing the field of IR.

Neural Language Model: Transformer Overview

  • Non-recurrent sequence-to-sequence encoder-decoder model
  • Good for GPU parallelisation
  • Task: machine translation
  • Data: parallel sentences in each language
  • Uses "attention"

BERT is a transformer encoder

  • Masked Language Modeling: Hide 15% of the words of input and try to predict them
  • Inherently bi-directional information flow since attention allows all words to contribute to the prediction
  • Secondary task: next-sentence prediction

GPT-2 tranformer encoder in one direction

  • Uses standard Language Modeling task to predict the next work in a sequence
  • The unidirectional nature suits text generation
  • Improves on training time nad long-term dependency recall

Q: Distinguish standard Language Modeling task and Masked Language Modeling task

Q: distinguish uni- and bi-directional

(Is about information flow)

Seq2Seq architecture model (missed details).

What's really a new possibility is that BERT model fine-tuning through transfer learning: just learn a classifier on the top layer for each task you fine-tune for. This includes ranking!

Dialog systems

Chit-chat chatbots.

  • No specific goal, focus on natural responses
  • "Ungrounded" system (no clear ground truth)

Task-oriented

  • Focus on actions
  • Accomplish a task in the world
    • SIRI, car interfaces, book a restaurant, etc.

Conversational AI Systems balance social conversation and task accomplishment.

  • Virtual AI assistant (low social; high task)
  • Socialbot(bit of both)
  • Chat bot (high social, low task)

Taxonomy of conversational systems: shows how these three branches with the same endpoint. We can reach conversational search system by functionally extending either the tracks from classic IR systems with statefulness and speech and language capabilities, chatbots with IR capabilities and user task modeling, and dialogue systems with data retrieval capabilities and IR capabilities.

conversational search

You could also say that classic IR systems must then become more "social" whereas the more social applications should become more task-oriented.

Conversational Information Systems (CIS)

  • interactive IR system with speech and language processing OR
  • retrieval-based chatbot with user task modeling OR
  • information-seeking dialogue system with information retrieval capabilities

Cf. the previous graph and the different "tracks".

Can be defined as such: an IR system with the user task expressed through a series of natural language interactions.

From web search to conversational search: opportunities and challenges

The following are "ad hoc (web) search" vs. Conversational Search

Keyword-ese Queries vs Natural Queries

  • Necessity: increase in speech/mobile interfaces
  • Opportunity: More natural and explicit expression of information needs
  • Challenge: Query understanding and sparse retrieval

Q: What's sparse retrieval?

Blue-Links vs Natural Responses

  • Necessity: Speech/mobile interfaces
  • Opportunity: More direct and easy access to information
  • Challenge: Document understanding; combine and synthesize information

Good example of this is that nowadays Google displays a natural language default answer on top before displaying links for many questions.

Single-Shot Query vs. Multi-Turn Dialog

A CIS is stateful i.e. it has memory and should leverage this. Think of the restaurant example where you ask a follow-up question.

  • Necessity: N.A.
  • Opportunity: Serving complex information needs and tasks
  • Challenge: Contextual understanding and reasoning

Passive serving vs. active engaging

Whereas in web search we serve up results, we ideally display them in an actively engaging way. For example, in conversational search we can deal with ambiguity in another way, namely by reporting back to the user "Did you mean interpretation X?".

  • Necessity: N.A.
  • Opportunity: Collaborative information seeking and better task assistance
  • Challenge: Dialog management; less lenient user experience (e.g. takes more time)

RRIMS desirable properties of CIS

These are properties an CIS should have.

  1. R. User Revealment
    • Help user express or discover their true information need and long-term preferences
  2. R. System Revealment
    • Reveal to the user what the system can and can't do and set the right expectations.
  3. I. Mixed-Initiative
    • Both user and the system should contribute, i.e. have a interactive strategy that is also flexible in the sense that the human/computer contribution is adjusted for whatever is most appropriate in a given situation.
  4. M. Memory
    • Should be able to reference to previous information
  5. R. Set Retrieval
    • CIS should be able to reason about the utility of complementary items.
    • Helps in deciding where to go next in your "information journey"

Other properties

  • Recovering from communication breakdowns
    • Prompt with clarification queues
  • Representation generation
    • generate useful representations that are shared between user and system (not sure what this means)
  • Multimodality
    • May accept other input modalities
  • Speech:
    • May involve speech-based input and output
    • Q Are conversational agents necessarily speech based?
  • Analyzing conversations
    • Analyze human-human conversations and intervene to provide contextually relevant information
  • Personalized understanding and reasoning about user
  • Collaborative (not entirely what was called collaborative above)
    • involve groups of users instead of a single individual
    • What does this look like?
  • Role
    • Take on different roles: teacher, scribe, mediator

Core CIS tasks

  • Conversational Query Understanding
  • Conversational Document Ranking
  • Mixed-Initiative: Learning to Ask

TREC Conversational Assistance Track (CAsT)

Jeff Dalton organized a TREC track to push the boundaries for conversational search.

Task: candidate passage response ranking in context

  • Input: Pre-defined static dialogues
    • This track provides "canonical response" for each turn
  • Read dialogue context
  • Do retrieval of candidate response information (passages)

Evaluation: is still very open-ended in this research area.

How do you evaluate a conversation? For now, start from well-known measures. This track has multi-level relevance judgments.

  • Topic-level effectiveness
    • N.B. it's ensured that all topics are approximately the same length in turns; otherwise evaluation is biased for some topics
  • Turn-level effectiveness:
    • Primary: NDCG@3
    • Mean reciprocal rank
    • MAP@1000l with binary relevance threshold=2

But these measures do not capture the conversational aspects well. They are really more measuring question answering by passage retrieval. We need a conversation-centric evaluatiom measure that rewards when:

  • Responses are consistently relevant across the conversation
    • Define what "consistent" means
  • Models know how to deal with hang-ups in the conversation
  • Models are able to interact and ask questions back, e.g. for clarification

Typical entry architecture

Preprocess

  1. Build index of MC MARCO and TREC CAR paragraphs

Inference time:

  1. Use a re-written question as a query and search
    • "magic" step: see "Conversational Query Understanding"
  2. Re-rank top 1000 passages using BERT re-ranker

Conversational Query Understanding (CQU)

The thing is that we work with regular indices, so we somehow need to extract queries from a running conversation in order to search our indices. So we need to go from an implicit query in a running conversation to a fully resolved query that gives good results when searching the indices.

This requires us to deal with:

  • Coreference resolution
  • Topic shifts
  • Zero anaphora / implicature
    • "What makes a good location" [for a food truck]
    • The latter part is meant but left out; it's clear from the context of the conversation though
  • Dependance on previous responses

A lot of traditional NLP systems fail here.

We can manually resolve turns in a conversation into queries (e.g. by explicitly stating the mentions) and so make a ground truth we can use for training.

CQU approaches

  • Traditional NLP pipeline
    • Named Entity Recognition (NER)
    • Coreference resolution
    • Named Entity Linking
  • Rule-based
    • Simple rules that leverage patterns in conversational structure
  • Supervised term selection
    • Train a classifier to identity the query terms
  • Neural Sequence-to-Sequence Generation
    • Train a model to generate a query given the previous history

Examples:

Y1 baseline CQU: NLP pipeline. Find entities and mentions, and replace the coreferents with the canonical name.

Team Waterloo@CAsT 2019 used rules:

  • Custom stopword list
  • Add words from first turn
  • Add words from previous n-turns

Supervised classification: Query Resolution by Term Classification (QuReTeC):

  • Use BERT encoder + a supervised classifier on it to find 'good' words from the history
  • Per previous turn, classify which term in that turn (in practice one sentence) is most relevant for the current turn

Sequence-to-sequence models:

  • Learn to translate a context-resolved query given the conversation history
  • So for query i take query 1 until i-1, and use that information to translate/rewrite query $q_i$ into a resolved query.
  • Leverage Natural Language Generation (NLG) pre-trained model, e.g. Apple used GPT-2

A clear potential issue here is the matter of training/fine-tuning the model.

CAsT has 50 topics, 10 turns per topic, and 20 topics with relevance judgements. I didn't really get the gist here, but the suggestion seems to be that in fine-tuning you can do with relatively little training data (few-shot-learning). The results are not perfect though. (Slide showed example where a slight topic switch from Lamborghinis to food trucks is not picked up (model fills in "Lamborghini Food Truck")).

Generative Rewriters

These "generative rewriters" are now the best performing approaches. How we do this rewriting depends on whether we use a uni-directional or bi-directional model.

GPT-2:

Recall that e.g. GPT-2 has a unidirectional information flow. We can sort of take an "incremental-restarting approach" (my words, not sure how to summarize this otherwise).

  1. Q1 [SEP] Q2 -> GPT-2 Rewrite Q2 into Q2r (r for rewritten)
  2. Q1 [SEP] Q2 [SEP] Q2 -> GPT-2 Rewrite Q3 into Q3r
  3. etc.

BART-LT (Last Turn):

  1. Q1 [MASKED QUERY] Q2 -> BART fills in the MASK given the context of both Q1 and Q2, generating Q2r
  2. Q1 Q2r [MASKED QUERY] Q3 -> similar
  3. etc.

BART (Full context):

Now in the output, do not just rewrite the last Q, but also add in another [MASK] in the output, i.e. ask BART to rewrite the history of the previous queries as well. This forces BART to reiterate the previous context and to take it into account when rewriting the current query.

Really nice property is that BART is able to pull in knowledge from the pre-trained model that is not actually in the previous context, but is sometimes correct. A problem here is that if we keep reitering the past history, BART keeps "hallucinating" and may significantly change the history.

Conversational Passage Ranking

BERT and ColBERT Ranking

Pointwise ranking vs. pairwise ranking :

MonoBERT is a pointwise ranker, which means it takes a query and a document and then gives you a relevance score.

DuoBERT is a pairwise model that takes a query and a sequence of documents and then picks which one is more likely.

Pairwise ranking is very expensive to run.

State tracking

Including the whole history is also expansive. What BERT models seem to need is a summary of previous/important information states. E.g. by extracting previous key queries and results, or by producing an (automatic) summary.

These approaches do not outperform history appending based on rules, yet.

You also see that the higher the conversation depth (i.e. the turn in a conversation), the worse the ranking performance gets for automatic methods. For manual query rewriting, the conversation depth is not problematic and may actually increase performance (more contextual information).

Summary

  • Conversational information seeking is important for conversational AI.

    • It serves as a benchmark for machine reading and understanding
    • It is distinct from web search and interactive retrieval
  • CQU is a key building block

    • Sequence-to-sequence models are most effective
    • It is surprisingly easy to fine-tune them on small datasets given pretrained nets
    • Failure in key areas: hallucination problems, long-term relationships (bias to recency, I guess), bad scaling to longer conversations
  • The goal of meaningful interaction for CIS remains challenging and we need new evaluation measures for conversations

  • Adding statefulness and interactivity really re-opens the domain of search from a new perspective

Q: What's the difference between conversational information seeking and interactive retrieval?