Automatic Summarization | Guest lecture by Maya Sappelli

Learning goals:

  • Difference extractive and abstractive summarization
  • Design both types
  • Explain input, purpose and output factors related to summarization
  • List the three typical steps in a neural-based summarization pipeline
  • Understand limitations of neural-based summarization tools
  • Describe appropriate evaluation metrics for automatic summarizers
  • Explain the challenges for automatic summarization evaluation

Types and goal

  • Single-document summarization: produce a shorter text than the original
  • Multi-document summarization: combine information from multiple texts into a single shorter text

The goal in both cases is to produce a concise and fluent summary of the most important information of the input documents.

  • Most important information requires understanding semantics of the text
  • Writing a concise and fluent summary requires the ability to "reorganize, modify and merge information"

The interpretation of documents is already hard for humans, and is not surprisingly also beyond the state of art for automatic summarization.

Applications

  • Single-document summarization

    • News articles
    • Scientific articles
    • Meeting reports / minutes
    • Book plot
  • Multi-document summarization

    • Summarize search engine output
    • Summarizing news about a topic from different sources
    • E-mail messages in a thread
    • Summarize reviews about the same product

Aspects of summarization

Input factors

  • Single-document or multi-document?
  • Input-output relation: Monolingual, multi-lingual, cross-lingual?
    • What's the difference between multi-lingual and cross-lingual?

Purpose factors

  • Indicative
    • indicates the type of information and "aboutness"
    • useful for alerts
    • more "meta-data"
  • Informative:
    • Includes quantitative/qualitative information
  • Evaluative:
    • Evaluates the content of the document (i.e. performs some processing e.g. aggregation on the content)

Example of indicative:

The report from yesterday's meeting is available (3 discussion topics, 1 hour, 5 members)

Example of indicative:

Decisions from yesterday's meeting: John will prepare the presentation for next week. Anne will contact ... etc

Example of evaluative:

31 reviews for this product, of which 23 positive. Most positive aspect: speed. Most negative aspect: battery.

  • Generic vs user-oriented
    • User-oriented may involve personalization and query-based summarization
  • General purpose vs. domain specific
  • Time dependency?
    • Alert-based systems
    • "While you were away" timeline from Twitter (extractive selection)

Output factor

An extract is composed of material from the source.

An abstract contains material that is not verbatim in the original source; e.g. it makes use of paraphrasing.

Possible outputs:

  • Single paragraph
  • Set of selected messages/responses
  • Term cloud
  • A single headline
  • Overview of opinions or sentiments
  • Bullet points

Baseline comparison

The most-often used baseline is surprisingly simple and surprisingly strong!

You simply take the first k sentences or the first paragraph of a document as its summary. This is called LEAD.

Extractive systems:

  • Select most important nuggets (sentences) using feature representations
  • Classification or ranking task
    • Classification: For each sentence, include in summary yes or no?
    • Ranking: assign score to sentences and choose top-k

Pros and cons:

  • Efficient
  • After sentence selection fixes are required to make the selected sentences run fluently

Sentence clustering

  • Cluster sentences on similarity (e.g. cosine similarity)
  • Select one representative sentence from each cluster
  • General idea is that a cluster with sufficient sentences represents an important topic

This abstracts away from the fact that a sentence may express multiple topics. It will only be assigned to one topic.

Traditional approach for multi-document summarization:

  • Cluster similar documents
  • Per cluster, show one document/paragraph

Graph-based methods for sentence ranking

Build a graph of a document with nodes being sentences and edges being their similarity.

The improvements:

  1. this method is that graph clustering methods allow assignment to multiple clusters.
  2. allows usage of graph algorithms like PageRank, or use of Markov chains, of lexical centrality

Sentence selection as supervised learning

Approach 1:

  • Engineer features, e.g.
    • document position
    • similarity to full document or title (representativeness)
    • cue phrases ("to sum up" etc.)
  • Train classifier on features (e.g. SVM or decision tree)

Approach 2:

  • Use a ConvNet or RNN
    • Requires no feature engineering because we can use word embeddings as input
    • Word embeddings allow for a more semantic interpretation of content; they already take relations between words into account
    • Word embeddings also naturally include several features, such as sentence position or term frequency
    • Requires lots of computational power and training data

The NN approach needs an additional step to create sentence representations from word embeddings.

The NN approach needs to optimize two things on sentence representations:

  • Coverage (is all information included)
  • Redundancy (do not include the same information twice)

Issues with extractive methods

  • Chosen sentences may include unresolved references to sentences that are not in the summary, leaving a gap.
    • "Our investigations show this to be true"; what does "this" mean?

This means that extractive methods require some improvements after sentence selection, e.g.

  • Sentence ordering
  • Sentence revision
  • Sentence fusion
  • Sentence compression

You could include this in your optimization in the NN-based approach, but you'd need relevant training data for this specific task to do so.

Abstractive systems:

  • Learn a text-to-text-transformation (cf. translation)
  • Training data: pairs of longer and shorter texts
  • Algorithms: translation models, RNNs; or very old-school using templates

We (and the required reading for this lecture) focus on NN-based abstraction methods.

Pros and cons:

  • More difficult to implement, although now feasible with deep learning
  • Risk of deviating from original content

An abstractive NN system usually includes two main parts:

  • Encoding
    • Capture a meaning representation of the whole document
    • Attention layer focuses on ensuring coverage
  • Decoding: produce shorter version of the meaning representation
    • Estimates probability distribution generating word at timestep t
    • I.e. this is a generative model that learns how to "write" abstracts
    • Attention layer focuses on reducing redundancy
    • Pointer/copy mechanism: how to deal with rare words i.e words that are not yet seen when training the model? Typical example: a relevant name.

"Attention layer": probability distribution that encodes focus on the words containing most information.

I did not really understand the details of the network; probably review.

Evaluation

One main challenge is that abstractive systems get the "freedom" to rephrase texts. How do we evaluate the quality of the abstract, including this rephrasing?

Intrinsic

Intrinsic: test the summarizer on its own task.

Methods:

  • Compare to reference summary on the same text
    • This is especially useful in evaluation of NN-based methods, because then these reference summaries are required anyways as learning targets
    • In the case of extractive summaries, we can actually compute precision and recall with the "true" responses in the reference summary, and the selected responses by the system
    • In abstractive summarization, we do not select but generate sentences. We can instead measure the overlap with a reference summary, usually using the ROUGE metric or ROUGE-N
  • Ask human experts/annotators
    • Look into "Pyramid method"
    • On Coherence / Informativeness/ Redundancy / Representativeness
    • On Fluency: syntactically correct, no weird gaps and messed up grammar etc.
    • One approach is compare manual reference and automatic summary side by side an record which one is better by human judgement

ROUGE (abstractive)

ROUGE-N measures the proportion of n-grams from the reference summary that occur in the automatically created summary. This is recall oriented, because it takes the produced summary as a reference in the denominator.

You can generalize this and use multiple reference summaries:

  • By summing over references in the denominator, this encodes that there may be multiple good summaries!
  • By summing over references in the enumerator, we reward abstracts with words that occur in all reference summaries.

In practice there is often only one reference summary available though! Which is a bummer, because the whole point of abstractive summarization is learning how to generate new words and phrasings; this aspect is not promoted when you only have a single reference summary as being "correct".

Other variants are ROUGE-L / W / S/ SU

Human judgements about relevancy and fluency are strongly correlated to each other, but unfortunately ROUGE often has a weak correlation with human judgements.

Extrinsic

Extrinsic: test effectiveness of summarizer on another task.

This description is a bit vague, but what this means is that you design a task where the summaries need to be used. A good example is assessing the relevance of documents for some topic of interest. It turns out that automatic summarization is very useful for this. You can measure the quality of automatic summarization by measuring to what extent it helps (or detriments) the external task.

  • How useful is it for experts?
  • Speed gain w.r.t. humans

State of the art and problems

Because almost all NN-based approaches train on the same CNN news articles dataset and also evaluate using the same metrics, there are some issues:

  • News articles have a very typical layout so generalization to other types of articles may be bad
  • CNN data is apparently quite noisy
  • In general the task is underconstrained: there is not just one good way to write a summary, but multiple. We need more reference summaries.
  • We also saw that ROUGE correlates terribly with human judgements. So as a measure, this does not even track well what humans consider to be a good summary.
  • ROUGE does not account for factual correctness, i.e. that the summary is representative of the data; 8-26% of the summaries are reported to contain factual errors.

Dealing with factual errors is important before these methods can be deployed. The lecturer is not aware of any real-world serious applications of this technique. Open challenge how to deal with factual incorrectness. A human-in-the-loop scenario seems most plausible.