Argumentation Mining: State of the Art and Emerging Trends | Lippi, Torroni
Motivation
Argumentation has come to be increasingly central as a core study within artificial intelligence [Bench-Capon and Dunne 2007], due to its ability to conjugate representational needs with user-related cognitive models and computational models for automated reasoning. (1)
There's also some work suggesting that human reasoning itself is argumentative in nature (Mercier and Sperber, 2011).
Main goal of argumentation mining:
The main goal of argumentation mining is to automatically extract arguments from generic textual corpora, in order to provide structured data for computational models of argument and reasoning engines. (2)
Steps:
- Recognize argumentative sentences
- Find argument components from the argumentative sentences
- Predict links between argument components (e.g. which evidence supports which claim)
- Infer connection between arguments to create a complete argument graph that captures the whole argumentation
Application potential:
Currently, the techniques used to extract information from these sources are chiefly based on statistical and network analysis, as in opinion mining [Pang and Lee 2008] and social network analysis [Easley and Kleinberg 2010]. An argumentation mining system instead could enable massive qualitative analysis of comments posted on online social networks and specialized newspaper articles alike, providing unprecedented tools to policy-makers and researchers in social and political sciences, as well as creating new scenarios for marketing and businesses. (3)
Goal of current review paper:
- "first structured survey of models, methods, and applications of this exciting and rapidly developing research area" (3)
- propose a unifying view on argumentation mining
- "The motivation behind it is that many efforts made under the general umbrella of argumentation mining in fact aim to solve a constellation of subtasks, whereas a single unifying view is still missing." (3)
Argumentation models
Toulmin's model (1958) proposes that human argumentation and reasoning have a structure with six categories (p.3-4):
- incontrovertible datum (i.e. it cannot be disputed; is recognized as fact), which is the basis of ...
- a claim
- that follows from the datum by a rule of inference (warrant)
- with some qualifiers that indicate how certain we are of the claim
- or with conditions in which the claim holds (rebuttal)
- or with a justification to the warrant (backing)
This model is relatively complex and thus not always used in actual applications.
In the field of computational argumentation three type of models have been used:
- rhetorical: "emphasis on the audience and on persuasive intention" (4)
- dialogical: "describe the ways arguments are connected in dialogical structures" (4)
- monological: "emphasize the structure of the argument itself, including the relations between the different components of a given argument" (4)
Another distinction is between abstract argumentation and structured argumentation.
- Abstract (based on Dung; I see this more often)
- Each argument is an atomic entity without internal structure
- Dialogical model that provides a framework for modeling and analyzing attack relations between arguments
- which may (or may not) be justified according to some semantics
- Structured:
- Proposes internal structure for arguments, i.e. between forms of support and claims
- When extracting arguments from free text, this requires a structured argumentation model that defines the structure of a (type of) argument
I also noticed when reading other papers that I sometimes used "argument mining" in my notes instead of "argumentation mining". These can be conceptually distinct, but are sometimes used interchangeably anyways:
Argumentation has historically referred to the process of constructing arguments and, since the advent of computational argumentation, to the process of determining the set of justified conclusions of a set of arguments. However, argumentation mining and argument mining are often used interchangeably and in a broad sense, as the field yet retains a strong element of conceptual exploration. (4)
Tasks and subtasks
- Identification/extraction: detecting premises and conclusions of an argument
- Claim detection
- Evidence detection
- Authorship attribution
- Completion: "infer implicit argument components such as enthymemes and tacit assumptions related to commonsense reasoning" (5)
- Relation prediction (both inter- and intra-argument relations)
Other related tasks:
- Rhetorical characterization of sentences
- Opinionated claim analysis
- Premise verification
Methods
Taxonomy of problems
- granularity of input
- paragraph level
- most work nowadays on the level of sentences
- intrasentence argument component boundaries
- genre of input
- e.g. legal, news, essays
- each genre has its own characteristics
- most systems therefore focus on a single genre
- argument model
- majority use simple claim/premise model
- granularity of target
- particular argument component?
- whole argument?
- goal of analysis
- detection
- classification
- relation prediction
- attribution
- completion
Argument component detection
Typically two subproblems:
- Extraction of argumentative sentences
- Detection of component boundaries
Most systems do these steps separately; a few systems assume the boundaries will be determined by other means.
Extraction of argumentative sentences
Three approaches:
- Binary classifier argumentative vs. non-argumentative. Prediction the component type (e.g. premise or claim) requires another classifier.
- This approach is taken by Mochales (2011)
- Multiclass classifier that directly classifies all components of the argument model, e.g. premise, claim, non-argumentative.
- Assumption a sentence can contain at most one argument component!
- One binary classifier per argument component; since they can be applied separately to a single sentence, a sentence can now contain multiple argument components
- Or equivalent: a multi-class components where instances can have multiple classes
ML classifiers used so far (see paper for many references per method):
- SVM
- Logistic Regression
- Naive Bayes
- Maximum Entropy Classifier
- Decision Trees
- Random Forests
Note that they are all supervised.
Feature design: The authors note that little focus is on these classifiers; most energy goes into designing very informative features to be used with these classifiers.
BoW representation with TF-IDF vectors, common text analytics features, and bigram or trigrams, are very common. They have two obvious limitations:
- They do not consider word order (albeit a small amount of context with bi- and trigrams)
- They do not consider semantic similarity between terms
- Advanced features using ontologies, thesauri, lexical databases e.g. WordNet, can help alleviate this problem
- What about word embeddings?
More advanced features with POS-tags:
- bi- and trigrams on particular POS tags
- Information on punctuation
- ... or verb tenses
- ... or discourse markers
Or features "obtained by external predictors" (7)
- Subjectivity scores
- Sentiment detectors
- Named Entity Recognition systems
Because many AM systems are developed for one particular domain and use very context-dependent features. This does of course hurt generalization of these systems. In these cases we call this:
- Context-Dependent Claim Detection (CDCD)
- Context-Dependent Evidence Detection (CDED)
There are therefore also attempts for systems without relying on context-specific feature engineering:
Lippi and Torroni [2015] have proposed an SVM-based method for Context-Independent Claim Detection (CICD), which exploits structured kernels on constituency parse trees (in particular, the Partial Tree Kernel [Moschitti 2006] was employed) to measure similarity between sentences. The constituency parse tree is very often able to capture the rhetorical structure of a sentence, which in many cases is highly indicative of the presence of a claim. (8)
"Also, Rooney et al. [2012] exploit structured kernels, even though they only consider kernels between sequences of words and/or parts-of-speech tags." (8)
TODO: I reckon there are now also approaches that use deep learning and specifically word embeddings as features! Look into that.
Argument Component Boundary Detection
This is a segmentation problem. Note that it's not necessarily the case that a whole sentence corresponds to a single argument component. A simple example sentence that contains both a claim and an evidential statement would be" We claim X, because Y". The goal is to determine the boundaries of X and Y.
There are three scenarios (9)
- portion of sentence coincides with argument component
-
= 2 components in a sentence
- a single argument component spans across sentences
In practice, AM systems make simplifying assumptions. Whether these are appropriate also largely rests on the type of corpora used.
Other principles approaches to segmentation in what the authors call the "collective classification framework" (9)
The collective classification framework has proven to be extremely powerful not only for sequences, but for any kind of relational and structured data, where relationships and dependencies among the examples have to be taken into account [Getoor 2005]. The works by Goudas et al. [2014], Sardianos et al. [2015], and Park et al. [2015] represent a first attempt in this direction, by exploiting Conditional Random Fields to segment the boundaries of each argument component. In particular, Sardianos et al. [2015] also employ Recursive Neural Networks to build representations of words. (9)
Argument Structure Prediction
Possible impact:
Edges in the graph may represent different relations, such as entailment, support, or conflict. With regard to the context of social media and Web documents, such a graph would be an invaluable instrument for a multitude of applications: analyzing the dynamics of social debates on the Web and the spread of influence among social networks, evaluating the acceptance of different arguments and the role of specific users within this kind of process, detecting anomalous behaviors, and even studying the most effective rhetorical strategies. The impact on social sciences, computational linguistics, formal argumentation, and discourse analysis would be dramatic. (10)
The output graph representation is heavily influenced by the chosen argument model.
In case of a claim/premise model, the output will be a bipartite graph.
Summarization of approaches:
Current approaches to the structure prediction task make several simplifying hypotheses. For example, in the corpus by Aharoni et al. [2014], an assumption is made that evidence is always associated with a claim. This in turn enables using information about the claim to predict the evidence. In this case, the support links are thus obtained by definition when predicting the evidence. Mochales Palau and Moens [2011] have addressed the problem by parsing with a manually built context-free grammar to predict relations between argument components. The grammar rules follow the typical rhetorical and structural patterns of sentences in juridical texts. This is a highly genre-specific approach and its direct use in other genres would be unlikely to yield accurate results. Work by Stab and Gurevych [2014b] instead proposes a classic machine learning solution, by employing a binary SVM classifier to predict links in a claim/premise model based on work by Freeman [1991]. Biran and Rambow [2011] apply the same technique mentioned earlier to premise detection also for the prediction of links between premises and claims. Finally, another important research direction adopts Textual Entailment [Cabrio and Villata 2012a], with the goal of inferring whether a support or attack relation between two given arguments holds. (10)
Similarities between AM subtasks and ML-NLP tasks
See p.11.
Argumentative sentence detection is similar to:
- Sentence classification
- Hedge cue detection
- Sentiment analysis
- Question classification
- Subjectivity prediction
Argument component boundary detection:
- Sequence labeling
- Named Entity Recognition
- Text segmentation
Argument structure prediction:
- Link prediction
- Discourse relation classification
- Semantic textual similarity
Corpora and applications
Structure prediction
Because these corpora contain argument maps, you often don't have negative training data on non-argumentative text.
- Corpora from University of Dundee
- Annotated argument maps in various formats
- incl. AraucariaDB and transcripts of the moral Maze radio program
- NoDE benchmark, maps of various sources, can be used for "interargument support and attack relations"
- Boltuzic and Snajder (2014): "they consider a small corpus of documents consisting of user comments on two controversial arguments, and they develop a system for the classification of the relation between each comment and its associated argument into five different classes (strong attack, attack, strong support, support, none)"
- German corpus of Kirschner et al. (2015): "a collection of scientific publications annotated with argumentation structures (supports, attacks, details, sequence)" (13)
Legal domain
None of these are publicly available (as of 2016)
- Mochales, Palau and Moensa (2011) on European Court of Human Rights ( ECHR )
- Mochales, Palau and Moensa AraucariaDB
- Ashley and Walker (2013) Vaccine/Injury Project (V/IP)
Etc.
- Biology and Medicine (p. 13)
- Humanties (p. 13)
- User-generated content
- Quite a few datasets here
- E.g. large set by IBM Research on Wikipedia articles; evidence and claims labeled
Skipped a bunch.
Challenges
SECTION ON DEALING WITH UNSUPERVISED DATA!
Dealing with Unsupervised Data
Deep learning techniques, which have recently obtained breakthrough results in many AI domains, including natural language processing, certainly represent one of the most appealing choices in this direction. The ability to deal with large and unsupervised corpora is indeed one of the most crucial aspects of deep networks, and among the chief reasons of their impressive success. Unsupervised learning is in fact employed in many deep architectures as a pretraining phase whose aim is to extract hierarchical sets of features directly from the data, while the interaction with supervisions (the symbolic layer) only applies at a further stage. In very recent years, a huge amount of deep learning research has focused on language tasks, producing a wide variety of sophisticated systems dedicated to specific applications (e.g., see LeCun et al. [2015], and references therein). Many of these systems are capable of capturing advanced semantic information from unstructured text, and within this context, the Web is an invaluable mine of multifaceted and heterogeneous information. A very successful approach in this direction is given by the so-called word embeddings, sometimes also called simply word vectors, that basically consist in automatically learned feature spaces encoding high-level, rich linguistic similarity between terms [Mikolov et al. 2013]. Recurrent Neural Tensor Networks [Socher et al. 2013] and Tree-Structured Long Short-Term Memory Networks [Tai et al. 2015] and also a language-specific version of Convolutional Neural Networks [Kim 2014], are examples of deep architectures capable of dealing with input sequences of arbitrary length, which have recently achieved state-of-the-art results in language-specific applications, including sentiment analysis. Therefore, they could be naturally adapted to perform some of the steps in the AM pipeline, such as sentence classification. (18)
So I've been playing with the idea to use word embeddings/vectors. Interestingly, the authors refer to a paper here!
Sardianos et al. [2015] presents a first approach to exploiting word vectors (for the Greek language) within an argumentation mining system.
Christos Sardianos, Ioannis Manousos Katakis, Georgios Petasis, and Vangelis Karkaletsis. 2015. Argument extraction from news. In Proceedings of the 2nd Workshop on Argumentation Mining, Lecture Notes in Computer Science, Vol. 8445 (2015), 56–66.