Context-Independent Claim Detection for Argument Mining | Marco Lippi and Paolo Torroni, Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015)
The goal of this paper is straightforward and admirable: a lot of ML-based argument mining applications rely on domain-specific feature engineering. This seriously hurts the generalizability of these systems. Instead, this paper states that argumentative sentences have common rhetorical structures - across domains, is the claim - that can be leveraged to recognize argumentative sentences.
This paper specifically focuses on the detection of claims (which is just another name for the conclusion of an argument that requires support).
Discourse analysts use several techniques for recognizing (cf. p 185):
- argument markers
- cue phrases
- linguistic elements, e.g. "more precisely", "for example"
- Knott and Dale 1994 identify many of these
Existing expert systems for argumentation and reasoning require a lot of knowledge engineering by a domain expert. IBM Debater has the large ambition to argue pro and con any subject without requiring domain experts to import domain knowledge.
To this end, the IBM research team has designed a set of sophisticated classifiers that use contextual information, in order to extract valuable features for evidence, claim, and argument detection tasks in a given context [Levy et al., 2014]. (185)
Hope that Levy et al. go into detail on those contextual factors.
One example given in this paper is "knowledge of the topic", which in the case of the debater is selected by the user. But what about the cases where the topic is not clear; e.g. in social media the topic may be undefined, "posts may contribute to several topics, and the discussion may often shift from topic to topic" (186). The point of this paper is to not assume this contextual information.
Purpose statement:
The purpose of the present work is then to devise a method for the detection of claims in unstructured corpora, without necessarily resorting to contextual information. (186)
Hypothesis:
... argumentative sentences are characterized by common structures that reflect rhetorical processes, and can therefore be extremely informative of the presence of a claim. (186)
Problem definition
Goal: identify sentences that contain conclusions of arguments
The definition of "claim" is not entirely clear:
CD is a quite subtle task, even for humans, since the concept itself of claim could hardly be framed by a compact set of well-defined rules. Indeed, in some cases claims consist of opinions supported by some evidence facts, while in other cases they are represented by statements that describe relevant concepts [Levy et al., 2014], as it happens, e.g., in legal domains. For these reasons, building a large annotated corpus for claim detection is a complex and time-consuming activity. (186)
Methodology
Most ML-based solutions to argument mining so far still rely on 1) large annotated datasets and 2) highly engineered domain-specific features.
But they rarely use the fact that language is structured:
In particular, information coming from NLP could be naturally encoded into structured data which in turn can be extremely informative for many argumentation mining problems, such as claim detection. (2)
Claims have (186):
- Common structure i.e. a particular type of constituency tree that depends on verbs like assert, argue, believe, maintain, sustain, assert
- Claim can be introduced by a colon, like -> "X states: "Claim claim claim"
- Comparison between two concepts or arguments: "Sustained strong growth over longer periods is strongly associated with poverty reduction, while trade and growth are strongly linked." (see p. 186)
- Claims as conclusions in an argument
These examples are used to show that claims tend to occur in particular linguistic structures
Approach:
We therefore built a claim detection system based on a Support Vector Machine (SVM) classifier which aims at capturing similarities between parse trees through Tree Kernels [Moschitti, 2006a]: Figure 1 shows two parse trees whose sentences contain two distinct claims, where boxes highlight the many common parts of their internal structure (p.3)
TODO: See Moschitti 2006b
Authors' explanation of a Tree Kernel:
By considering different definitions of fragments, several TK functions are induced: for example, one could consider only complete subtrees as allowed fragments, as well as define more complex fragment structures. Intuitively, each possible tree fragment is associated to a different feature in a high-dimensional vectorial space, where the j-th feature simply counts the number of occurrences of the j-th tree fragment: the TK can therefore be computed as the dot product between two such representations of different trees. (187)
Implementation wise, I can't quickly find a default implementation of this.
But the scikit-learn SVM implementation supports custom kernel functions.
The authors of this paper use the Partial Tree Kernel (PTK), also by Moschitti, but different paper -> 2006a. Using this kernal function you automaticall generate a feature set.
But the feature vector based on PTK can be combined with other "plain" feature vectors in order to further enrich the description of the instance! (I.e. just get a weighted combination)
Pipeline:
- sentence tokenizer
- parse each sentence -> constituency tree
- discard sentences without verb (VP)
- stem only words (i.e. the leaves of the tree)
- SVM classification
Data sets
Problems:
- Little publicly available
- Expensive to make, esp. with good annotator agreement
- lack of negative examples as many of these datasets only contains arguments
- Discussion of datasets I've already seen before, Dundee sets, NoDE, AraucariaDB
- IBM dataset on Wikipedia articles available on request for research, labelled for claims
IBM dataset was used because it contains annotated claims.
Caveat: a claim in IBM dataset was defined relative to a topic. So for topic T, only claims relevant for topic T were labelled. This is problematic, since the goal of this paper was to classify claims independent of any contextual information.
Second dataset is 90 persuasive essays, from Stab and Gurevych 2014 on various topics, annotated on claims, major claims, premises/evidence and attack/support relations.
Results
See paper.
Summary from the conclusion:
We evaluated our solution against the largest and richest known to date data set containing annotated claims. Our results are comparable with those of state-of-the-art methods, which rely on context. This is a significant achievement, if we consider that the most dominant features in state-of-the-art methods are all context dependent [Levy et al., 2014].