Context Dependent Claim Detection | Ran Levy Yonatan Bilu Daniel Hershcovich Ehud Aharoni Noam Slonim
- define task of automatic claim detection in a given context
- discuss associated unique difficulties
- preliminary supervised learning solution with cascade of classifiers to handle skewed data
Introduction
The one thing that's common to argument models is the presence of a claim or conclusion.
In this project, the goal is not to detect claims in general, but in a specific usage context. So the work definitions are:
- Topic: "short phrase that frames the discussion"
- Context Dependent Claim (CDC): general, concise statement that directly supports or contests the given Topic
These CDCs were labeled for given Wikipedia topics.
Challenges:
- CDC is similar to a statement that defines a concept related to the topic
- and to claims that are not relevant to the topic
- and to statements that just repeat the topic in different words
- some claims may be more a matter of opinion than fact
- facts may be simple true or false, while opinions may not have an as strict truth value
- very skewed/unbalanced data; only 2% of Wikipedia articles are highly relevant to a topic
Task definition and related work
- practical importance in decision support and persuasion enhancement.
- subtask of argumentation mining
- task more specific than claim detection, because it's not just claims in general, but detection of claims relevant to a particular topic.
- at the same time more general in the sense that the topics are not limited to a domain, but can be general Wikipedia topics on anything
- CDCD as formulated in this paper is not just sentence classification but requires boundary detection.
- "in our framework it is quite natural that the Topic, or the associated CDCs, will correspond to imperative sentences, or even to exclamatory sentences."
Data
Labelling criteria for CDCs:
- Strength, directly support of contesting of the topic
- Generality, deals with relatively broad idea
- Phrasing, grammatically correct and semantically coherent
- Keeping text spirit
- Topic unity, discusses at least one, at most two related topics
Technical approach
Set up as a "funnel" that increasingly focuses on smaller text segments and divides the CDCD problem into smaller more tangible problems.
- Detect sentences
- Detect CDCD boundaries
- Rank candidates
The funnel approach is designed 1) for tackling severely imbalanced data and 2) for filtering out as much irrelevant text as early as possible to allow for more demanding techniques 3) and allows more focused feature design for a specific subtask :
Thus, in principle, for each Topic we consider around 300,000 candidate CDCs, of which typically only 30 represent positive examples. By breaking the problem into independent sub-problems, at each stage the skew between positive examples and negative examples is less daunting, thus easier to handle by classical machine learning techniques. In addition, since much surplus text is filtered along the cascade, “downstream” components typically examine much smaller amounts of text, and thus can plausibly make use of more demanding algorithms. Finally, this conceptual separation naturally allows to develop features tailored individually to each task; for example, the grammatical correctness of a text fragment is clearly relevant for boundaries detection, while being irrelevant when classifying whole sentences. (1492)
In the development of one of the funnel components it is implicitly assumed that the preceding components work perfectly.
Sentence component
- Binary classification problem (it's a quite rare exception that a single sentence contains multiple CDCs)
- Logistic Regression classifier
- Effort focused on finding highly discriminative features
Two main type of features:
- Context features: capture relation with a topic
- Context-free: based on content of the candidate sentence and probability of a "claim like" statement
Most dominant features (inspiration!):
- MatchAtSubject: cosine similiarty of sentence with topic
- ExpandedCosineSimilarity: idem but with semantic WordNet expansions of candidate sentence (specifically expansions to synonyms, hypernyms, hyponyms)
- binary ESG features (ESG parser)
- does sentence contain "that" token assigned as "conjugate", as in "critics such as Dave Grossman and Jack Thompson argue that violence in games hardens children to unethical acts" (table 2 contains this example). This type of "that" indicates the introduction of a claim.
- Is the verb in present tense
- Is it an infinitive verb
- presence of year
- presence of named location
- Subjectivity score (classifier-based)
- Ratio of sentiment words
- SequentialPatternMatch from Sequential Pattern Matching (SPM) algorithm
- Not familiar with this algorithm! Seems interesting!
Boundaries component
Coarse filter:
- Generate sub-sentences
- Filter out sub-sentences without verb and a noun
- Do MLE probabilistic classification, in this case using Naive Bayes
Fine-grained filter to select the top 10 candidates from the coarse filter:
Specifically, the candidate boundaries induce a division of the containing sentence into three parts: prefix, candidate body, and suffix, where the prefix and/or suffix might be empty. The features are then calculated for each of these three parts independently. Thus, for example, the presence of the word “that” in the prefix as opposed to its presence in the candidate body, will increase or decrease the confidence of the examined boundaries, respectively. In addition, the LR classifier considered features derived from the probabilistic model defined by the Boundaries Coarse Filter, that also aim to assess the probability that the examined boundaries yield a CDC (1494)
Further discussion of features.
- So the tokens around the candidate substring are also taken into account in these features.
- Probability that substring is grammatically correct given token positions
- "ModifierSeparation: The ESG parser (McCord et al., 2012) describes the modifiers of its parsed tokens, such as the object of a verb. Typically, a token and its modifier should either be jointly included in the CDC, or not included in it. This notion gave rise to several corresponding features." (1495)
- Parse Sentence Match
- "that-conj" matches CDC: A binary feature indicating whether in the ESG parsing we have a subordinator ”that” token, whose corresponding covered text matches the examined boundaries.
- Digit count
- Unbalanced quotes or parenthesis; guess this is an indication of whether the substring is complete/correct
Ranking
Input are all previous scores.
Additionally:
CandidateComplexity, a score based on counting punctuation marks, conjunction adverbs (e.g., ”likewise”, ”therefore”), sentiment shifters (e.g., ”can not”, ”undermine”) and references, included in the candidate CDC. (1495)
Sentiment, ExpandedCosineSimilarity and MatchAtSubject, as in the Sentence Component above, estimated specifically for the CDC candidate (1495)
Experiments
Leave-One-Out evaluation on topic-level, so training on 31 topics, then evaluate on at least one CDC from the left out topic.
Why this way? Do they care a lot about
Error analysis
Error analysis by five annotators using these labels:
- Accept - The candidate should be accepted as is.
- Accept with correction - The candidate should be accepted with minor corrections.
- Generality failed - The candidate is too specific.
- Multiple Candidates - The candidate contains more than one claim.
- Repeats Topic - The candidate simply reiterates the topic (or its negation) rather than claim something about it.
- Strength failed - The candidate does not directly and explicitly supports or contests the topic.
- ** Text Spirit failed** - The candidate does not keep the spirit of text in which it appeared.
- Incoherent - The candidate is not a coherent claim.
- Not a sentence - The candidate is not grammatical.