Automatic Summarization | Guest lecture by Maya Sappelli
Learning goals:
- Difference extractive and abstractive summarization
- Design both types
- Explain input, purpose and output factors related to summarization
- List the three typical steps in a neural-based summarization pipeline
- Understand limitations of neural-based summarization tools
- Describe appropriate evaluation metrics for automatic summarizers
- Explain the challenges for automatic summarization evaluation
Types and goal
- Single-document summarization: produce a shorter text than the original
- Multi-document summarization: combine information from multiple texts into a single shorter text
The goal in both cases is to produce a concise and fluent summary of the most important information of the input documents.
- Most important information requires understanding semantics of the text
- Writing a concise and fluent summary requires the ability to "reorganize, modify and merge information"
The interpretation of documents is already hard for humans, and is not surprisingly also beyond the state of art for automatic summarization.
Applications
-
Single-document summarization
- News articles
- Scientific articles
- Meeting reports / minutes
- Book plot
-
Multi-document summarization
- Summarize search engine output
- Summarizing news about a topic from different sources
- E-mail messages in a thread
- Summarize reviews about the same product
Aspects of summarization
Input factors
- Single-document or multi-document?
- Input-output relation: Monolingual, multi-lingual, cross-lingual?
- What's the difference between multi-lingual and cross-lingual?
Purpose factors
- Indicative
- indicates the type of information and "aboutness"
- useful for alerts
- more "meta-data"
- Informative:
- Includes quantitative/qualitative information
- Evaluative:
- Evaluates the content of the document (i.e. performs some processing e.g. aggregation on the content)
Example of indicative:
The report from yesterday's meeting is available (3 discussion topics, 1 hour, 5 members)
Example of indicative:
Decisions from yesterday's meeting: John will prepare the presentation for next week. Anne will contact ... etc
Example of evaluative:
31 reviews for this product, of which 23 positive. Most positive aspect: speed. Most negative aspect: battery.
- Generic vs user-oriented
- User-oriented may involve personalization and query-based summarization
- General purpose vs. domain specific
- Time dependency?
- Alert-based systems
- "While you were away" timeline from Twitter (extractive selection)
Output factor
An extract is composed of material from the source.
An abstract contains material that is not verbatim in the original source; e.g. it makes use of paraphrasing.
Possible outputs:
- Single paragraph
- Set of selected messages/responses
- Term cloud
- A single headline
- Overview of opinions or sentiments
- Bullet points
Baseline comparison
The most-often used baseline is surprisingly simple and surprisingly strong!
You simply take the first k sentences or the first paragraph of a document as its summary. This is called LEAD.
Extractive systems:
- Select most important nuggets (sentences) using feature representations
- Classification or ranking task
- Classification: For each sentence, include in summary yes or no?
- Ranking: assign score to sentences and choose top-k
Pros and cons:
- Efficient
- After sentence selection fixes are required to make the selected sentences run fluently
Sentence clustering
- Cluster sentences on similarity (e.g. cosine similarity)
- Select one representative sentence from each cluster
- General idea is that a cluster with sufficient sentences represents an important topic
This abstracts away from the fact that a sentence may express multiple topics. It will only be assigned to one topic.
Traditional approach for multi-document summarization:
- Cluster similar documents
- Per cluster, show one document/paragraph
Graph-based methods for sentence ranking
Build a graph of a document with nodes being sentences and edges being their similarity.
The improvements:
- this method is that graph clustering methods allow assignment to multiple clusters.
- allows usage of graph algorithms like PageRank, or use of Markov chains, of lexical centrality
Sentence selection as supervised learning
Approach 1:
- Engineer features, e.g.
- document position
- similarity to full document or title (representativeness)
- cue phrases ("to sum up" etc.)
- Train classifier on features (e.g. SVM or decision tree)
Approach 2:
- Use a ConvNet or RNN
- Requires no feature engineering because we can use word embeddings as input
- Word embeddings allow for a more semantic interpretation of content; they already take relations between words into account
- Word embeddings also naturally include several features, such as sentence position or term frequency
- Requires lots of computational power and training data
The NN approach needs an additional step to create sentence representations from word embeddings.
The NN approach needs to optimize two things on sentence representations:
- Coverage (is all information included)
- Redundancy (do not include the same information twice)
Issues with extractive methods
- Chosen sentences may include unresolved references to sentences that are not in the summary, leaving a gap.
- "Our investigations show this to be true"; what does "this" mean?
This means that extractive methods require some improvements after sentence selection, e.g.
- Sentence ordering
- Sentence revision
- Sentence fusion
- Sentence compression
You could include this in your optimization in the NN-based approach, but you'd need relevant training data for this specific task to do so.
Abstractive systems:
- Learn a text-to-text-transformation (cf. translation)
- Training data: pairs of longer and shorter texts
- Algorithms: translation models, RNNs; or very old-school using templates
We (and the required reading for this lecture) focus on NN-based abstraction methods.
Pros and cons:
- More difficult to implement, although now feasible with deep learning
- Risk of deviating from original content
An abstractive NN system usually includes two main parts:
- Encoding
- Capture a meaning representation of the whole document
- Attention layer focuses on ensuring coverage
- Decoding: produce shorter version of the meaning representation
- Estimates probability distribution generating word at timestep t
- I.e. this is a generative model that learns how to "write" abstracts
- Attention layer focuses on reducing redundancy
- Pointer/copy mechanism: how to deal with rare words i.e words that are not yet seen when training the model? Typical example: a relevant name.
"Attention layer": probability distribution that encodes focus on the words containing most information.
I did not really understand the details of the network; probably review.
Evaluation
One main challenge is that abstractive systems get the "freedom" to rephrase texts. How do we evaluate the quality of the abstract, including this rephrasing?
Intrinsic
Intrinsic: test the summarizer on its own task.
Methods:
- Compare to reference summary on the same text
- This is especially useful in evaluation of NN-based methods, because then these reference summaries are required anyways as learning targets
- In the case of extractive summaries, we can actually compute precision and recall with the "true" responses in the reference summary, and the selected responses by the system
- In abstractive summarization, we do not select but generate sentences. We can instead measure the overlap with a reference summary, usually using the ROUGE metric or ROUGE-N
- Ask human experts/annotators
- Look into "Pyramid method"
- On Coherence / Informativeness/ Redundancy / Representativeness
- On Fluency: syntactically correct, no weird gaps and messed up grammar etc.
- One approach is compare manual reference and automatic summary side by side an record which one is better by human judgement
ROUGE (abstractive)
ROUGE-N measures the proportion of n-grams from the reference summary that occur in the automatically created summary. This is recall oriented, because it takes the produced summary as a reference in the denominator.
You can generalize this and use multiple reference summaries:
- By summing over references in the denominator, this encodes that there may be multiple good summaries!
- By summing over references in the enumerator, we reward abstracts with words that occur in all reference summaries.
In practice there is often only one reference summary available though! Which is a bummer, because the whole point of abstractive summarization is learning how to generate new words and phrasings; this aspect is not promoted when you only have a single reference summary as being "correct".
Other variants are ROUGE-L / W / S/ SU
Human judgements about relevancy and fluency are strongly correlated to each other, but unfortunately ROUGE often has a weak correlation with human judgements.
Extrinsic
Extrinsic: test effectiveness of summarizer on another task.
This description is a bit vague, but what this means is that you design a task where the summaries need to be used. A good example is assessing the relevance of documents for some topic of interest. It turns out that automatic summarization is very useful for this. You can measure the quality of automatic summarization by measuring to what extent it helps (or detriments) the external task.
- How useful is it for experts?
- Speed gain w.r.t. humans
State of the art and problems
Because almost all NN-based approaches train on the same CNN news articles dataset and also evaluate using the same metrics, there are some issues:
- News articles have a very typical layout so generalization to other types of articles may be bad
- CNN data is apparently quite noisy
- In general the task is underconstrained: there is not just one good way to write a summary, but multiple. We need more reference summaries.
- We also saw that ROUGE correlates terribly with human judgements. So as a measure, this does not even track well what humans consider to be a good summary.
- ROUGE does not account for factual correctness, i.e. that the summary is representative of the data; 8-26% of the summaries are reported to contain factual errors.
Dealing with factual errors is important before these methods can be deployed. The lecturer is not aware of any real-world serious applications of this technique. Open challenge how to deal with factual incorrectness. A human-in-the-loop scenario seems most plausible.