Guest lecture on spoken content retrieval | by Martha Larsson
- What is it and what are the commercial applications?
- What are differences with text retrieval?
- When do we need it?
Goal: return speech media results relevant to information need expressed in a user query.
In SRC the made text represenantions does not necessarily have to be human-readable! They can be optimized for retrieval instead. It's not just about machine translation.
SCR is different than "voice search" in which the query is spoken, but where the searched collection may be textual.
Examples of commercial systems:
- Podscope: audio video search engine (out of business)
- Everyzing:
- SpeechFind
- Radio Oranje: search speeches of Queen during WWII.
Does YouTube use automatic speech recognition transcripts for answering queries?
- They obviously do have machine translation.
- But is the transcript used in the index?
- Evidence that it's not done this way: we do not get links to relevant snippets in video in YouTube.
- Someone mentioned
Speech Project of Martha: Recognizing the voices of patients to retrace the opinions about medicines
Overview of Spoken Content Retrieval
The same problems have more or less remained the same the last ten years.
- 1990s: making of broadcast news collections
- 2000: TREC-Spoken Document SDR benchmark
- Different than SCR! Because SDR wants to retrieve a specific document that is known to be relevant.
- Later: focus on SCR instead of SDR
Q: What is the difference between SCR and SDR?
Mentions vs. meaning
- "Spoken Term Detection" (STD) just finds mentions of query words. But this is not yet information retrieval.
- Spoken terms can be used in indexing though
- In some applications, this limited search is sufficient for users
- Core SCR focuses on meaning or "aboutness" to relate multimedia to the user query, e.g. using:
- Vector Space Model
- Language Modeling Framework
Q: What's the difference between Spoken Term Detection (STD) and Spoken Content Retrieval (SCR).
Text IR vs. SCR
Speech transcripts contain a lot of errors due to:
- High speech variability (especially spontaneous speech)
- Pauses
- Sub-optimal recording conditions
- Speech transcripts are inherently unstructured
- Speech is time continuous and is hard to be skimmed
Use all info you have
Additional indexing features are helpful because ASR word errors rates are 30-50% when we apply it on more realistic audio (so more than just clear news broadcasts).
- Metadata is essential for finding spoken content.
- Derive indexing terms from titles or descriptions
- Alignment of audio with human generated transcripts
Q: What are ways of dealing with noise and errors in ASR?
ASR
Cf. slide 25.
New interesting concept: ASR-Phoneme is the smallest unit of sounds convenient for acoustic modeling.
Strategy: learn components of speech and build up from there
Challenges
- Challenging channel conditions
- Spontaneous speech
- Uncontrolled vocabulary
- Proper names
- Grapheme to phoneme conversion
- Pronunciation variation
- Regional differences
- Variation within a speaker related to context
- Multiple languages
- Some are morphological complex so we cannot divide acoustic model from dictionary and language model.
- Low resource languages: less data available for model training.
ASR with Hidden Markov Models
HMM is a statistical model for sequence data.
- Allows combination of multiple knowledge sources
- Outputs most likely sequence of words
- Dominant base model in ASR
We need:
- Incoming acoustic signal
- Feature extraction
- Acoustic model f(X|W): likelihood of acoustics given the linguistic unit
- Language model P(W): probability of a word sequence
- Lexicon: transcription of word as a sequence of linguistic units
- Search: $argmax_W f(X|W)P(W)$
- Essentially argmax over posterior
Q: What are the typical components of a HMM-based ASR system?
Acoustic Models
Acoustic likelihood of observed speech vectors for each sound class (in this case a phoneme): p(X|W)
This is the first thing that got replaced/implemented in neural networks, modelling links to certain phonemes based on constituent sounds.
Language models
Some word sequences are more likely to be spoken than others.
Language model is a prior p(W) encoding the probability of linear word sequences given a particular language.
Pronunciation dictionary
A.k.a. "lexicon": How do acoustic models represent and link to phonemes?
out-of-vocabulary (OOV) problem: a word that's not in the lexicon can never be recognized by the ASR system.
Combining
Score of a word sequence is computed by stringing together acoustic models ... missed this.
Performing recognition
Viterbi algorithm is a dynamic programming algorithm.
Comments on speech recognition
- People do not do perfect speech recognition either
- ASR introduces errors but also gives other information than just the translation i.e. conventional text
- E.g. word repetitions in speech can perhaps compensate for errors when documents are short
- ...
- ...
Features from N-Best
Finding best option from list of N-best hypotheses.
Impose several assumptions:
- Ignore with p()=0 everything outside best N
- N-best are equally probable
- N-best hypotheses are independent
- Terms in N-best hypotheses are independent
Then estimate the counts of word v in document d by summing over the count of all N-best hypotheses and taking the average (same for length; take average length of top n-best):
$$\hat{c_{d,v}} = \frac{1}{N} \sum_{n=1...N}c_{d,v}^{(n)}$$
- Improved retrieval for larger N
Lattice for hypothesis space
This lattice is a time-synchronous directed graph that encodes the pruned hypothesis space. Time-synchronous, i.e. this lattice is managed during recognition and is thus also pruned on the fly.
- Each node has a timestamp
- Each edge is associated with a word
- and edge is related to likelihood of transition (acoustic and language model scores)
The challenge of indexing lattices is to convert the information about the likelihood of transitions, into something that actually reflects the probability of that word occurring.
When you are a calculating a probability of a word in the language modeling approach to IR; calculating the count using vector models or e..g using tf-idf you have the actual words; but we do not have that. We only have probability scores on confidence, which say nothing about the expected frequency of words in a sequence. We cannot really count TF directly in this way. (This mistake was made early on in the field)
If we had a true posterior, we could compute a confidence score by summing over all sequence ... Didn't get this, see 48.
One option: use confusion network.
Confusion network
- Alignment of multiple strings
- Alternative words represented in parallel
- From the lattice, we preserve the order but not the timing
- Each link has a posterior probability; factors need to be scaled.
This is used to approximate P(X) to normalize the argmax above and get a proper posterior.
Exploiting Additional information
- Log of the number of phones in a word
- Fit with a language model
- Speaking rate
- SNR-related features
Combine these predictor features together with ASR-information and provide confidence scores. The choice of features is application dependent of course.
When do we need spoken content retrieval
See 51-57
Audio stream or file is seldom structured
- The basic unit of retrieval must be determined automatically.
- Basic unit is needed for:
- calculation of relevance to user query
- displaying query results to users
- Transcripts generated by automatic speech recognition (ASR) contain errors
- If recognizer vocabulary does not contain a certain word, another word will be substituted for it
- Speech recognition works best on audio recorded by professional speakers speaking in studio conditions
Do we really need spoken content retrieval? Isn't metadata enough?
Until slide 57, summary of a study that suggests that when we use content-based features in podcasts (so speech recognition), rather than only metadata, this improves performance.