Argumentation mining: How can a machine acquire common sense and world knowledge? | Moens, Argument & Computation 9 (2018), p.1-14
Introduction
Argumentation: "discourse phrases or sentences or larger discourse units that have an argumentative function in this discourse" (p.1-2)
Argument: "the set of one or more phrases or sentences that function as "premises", along with another set of phrases or sentences that function as “conclusions” or "claims" and that together form the argument." (p.1-2)
Argumentation mining 1) detects argumentation structure and 2) classifies its argumentation components and 3) their argumentative relationships.
As encountered in many argumentation mining papers, the argumentation model of Toulmin serves as a good basis for argumentation mining work, but in practice simplifications are used.
Difficulty; linguistic markers are often not clear enough; :
Arguments are sometimes revealed by the presence of discourse markers (e.g., in English adverbs such as “because”), but such markers have often ambiguous meanings or are missing. Moreover, the premises in favor of the same claim might be far apart in a discourse, making it difficult to automatically link them together. Also argumentation structures might take the form of graphs, where nested tree structures (e.g., a full argumentation composed of a claim and its supporting premises or counter elements form the premise for a more general claim [46]) (2)
But the main problem is lack of common sense and world (& domain) knowledge. Example from p.2:
“Technology negatively influences how people communicate. Some people use their cellphone constantly and do not even notice their environment.”
The second sentence supports the claim in the first, but finding this relation requires "the knowledge that using a cellphone is a way of communication" (2).
Ok, so how to get this world and domain knowledge?
- Explicit knowledge engineering
- drafting a lot of knowledge patterns
- manually annotating training data
- downside is clearly that the world keeps changing and these approaches would then require more manual effort to keep up
- Representations learned from given data
- "This knowledge would take the form of reusable representations learned from data" (p.2)
- Can be updated by providing new data
Outline:
- Current methods for argumentation mining and language understanding and their difficulties
- Solutions of representation learning and deep learning (unsupervised setting!)
- But what is the bottleneck for learning common sense and world knowledge?
- Promising paths for automated acquisition of common sense and world knowledge.
Language understanding
- semantic role labeling: 'recognizing "who" does "what", "where" and "how"' (p.3)
- coreference resolution: "identifying coreferring expressions in a discourse (e.g., that refer to the same entity)" (p.3)
- "recognition of temporal and spatial relations between actions and entities, and identification of other relationships" (3)
- "detecting modality, that is, the factual status of a statement and can involve negation, possibility and obligational entailments of statements" (3)
- the output is typically for use by a machine
Short history:
- 1960-70s: symbolic knowledge representation; domain specific; much insight in inference but little to no adaptability
- 1990s: focus on machine learning, knowledge representation now mostly in design of reusable standard labels, e.g. PropBank style semantic role labeling
- 2000s: joint learning: "The last decade witnessed an interest in joint learning and inference of semantic labels with structured learning models, having the advantage that certain correlations between the occurrences of output labels can be captured during training or prediction. For instance, in [34] a spatial relation, the objects involved in this relation, and the relation’s attributes are jointly recognized (e.g., “the man rides a horse” where “man” is spatially and externally connected to the “horse”). Structured learning algorithms are popular for semantic recognitions in language processing, especially maximum margin (max-margin) graphical models [63] and max-margin structured support vector machines [66]." (3)
- I want to have a look at structured SVMs. I think Lippi (2015) also uses this for unsupervised(?) recognition of argument structures?
- Structured SVMs output structured labels, such as a parse tree.
Why is language understanding such a difficult task for a machine?
- sentence and discourse grammars are ambiguous, but to get the meaning of words right you need to know the context and "grammatical configuration" (4)
- linguistic patterns do not sufficiently capture the meaning of free speech and writing
- so this is a limitation both of manually designing language patterns as well as labelling (which also assumes a fixed context? Not sure what the point is here)
- interpretation requires inference using contextual background information
Extra challenge of argument mining
We could make models using common sense knowledge in their reasoning; but the real question is how to acquire common sense reasoning.
Arguments add another layer of pragmatic understanding, i.e. in arguments words are used to achieve some result. The problem is that "discourse markers that signal arguments to a proposition are often ambiguous or missing" (4), meaning that if we want to teach a machine this layer of pragmatic understanding, we need to add signals (e.g. labels) to the text itself.
The hardest problem of all is that language use in general is incomplete in the sense that a lot of information is not expressed explicitly. Argumentation can require "reading between the lines".
(As an aside, in legal text, it is probably the case that you can way better use argumentative markers).
Current proposed solutions: Representation learning and deep learning
- Latent Semantic Indexing (LSI)
- Probabilistic Latent Semantic Indexing (pLSA) ... not pLSI?
- Latent Dirichlet Allocation (LDA)
- RNN based approaches
- CNNs approaches
- recursive tree RNNs -> "which infer latent concepts that are not apparent in the raw data" (5)
These techniques embed the context of the word in their representations. Most well known are the models used e.g. in word2vec:
Word embeddings commonly take the form of word vectors, that is, each word is associated with a real valued vector in an N-dimensional space (usually N = 50-1000). The most famous architectures are the CBOW and Skip-gram NN-LM models [43], the latter referring to the skip-gram neural network language model, and the GloVe model [52]. These models are characterized by a simple, single-layered architecture, where their main differences regard the inputs (CBOW: words within short window; Skip-gram NNLM: the current word) and output (CBOW: the current word is predicted; Skip-gram NNLM: the context of the word is predicted). The hidden layer in these architectures implements a linear function, which guarantees an efficient computation. The trained weights of the nodes (dimensions) of the hidden (or exceptionally the output) layers of the network form the word embedding vectors (5)
Some lines of research.
- Language models that predict content with a relational structure in an unsupervised manner
Recently, we witness a growing interest in language models that predict content using neural networks that train on large text data sets. For instance, [53] predict the likely order of events or actions in a discourse possibly enriched with coreference chains [51]. Although results are preliminary, the above authors foresee a large applicability especially in automated understanding of narrative discourse. The grounding of language relations by visual data only starts to emerge. For instance, [26] ground causality of action verbs by using video data. We know very little on how we could use visual and multimodal representations to improve the accuracy and efficiency of language understanding by machines.
- (Automatic) composition of statistical representations into more abstract concepts
Another interesting development is the composition of the statistical representations into more abstract concepts. The principle of compositionality in language states that the meaning of a complex phrase is a function of the meanings of its parts and their mode of combination. For instance, in case of vector representations of individual words, recent work shows that the meaning of phrases or sentences could be inferred by mathematical operations on the word embedding vectors [28]. Such models often keep the syntax-driven compositionality of formal semantics while retaining the empirical nature of distributional models [45]. (6)
TODO check sources on current approaches:
Current composition models rely on vector, matrix or tensor additions and multiplications, Frobenius algebras, packed dependency trees or fully supervised deep neural networks (for overviews see: [4,28,49]), and even provide logical entailment operations over vectors [30]. (6)
Three lacks of embeddings:
- embeddings do not exploit the structure of language (but only take local word context into consideration):
The embeddings usually do not exploit the structure of language, and certainly not the structure of the perceptual reality, resulting in representations that are not specific enough for effective language understanding, and most importantly they lack real anticipatory power to predict semantics (6)
TODO: which approaches with word embeddings then DO take language structure into account?
- Word embeddings also do not add much generalization to other subject domains (p. 6).
- Embeddings do not clearly address the problem of lack of common sense and world knowledge
Question for the next paragraph: "Could more sophisticated deep learning techniques help in language understanding and argumentation mining and offer a solution for these problems?" (7)
Reasoning about entailment
Example of a double LSTM architecture, one for premise one for claim, with attention mechanism between them. Difficult to understand for me.
PASCAL challenge recognizes need for world knowledge and common sense.
Human language understanding is very efficient; partly because humans anticipate a lot of the relevant circumstances and outcomes of actions.
Learning common sense and world knowledge
The generative lexicon
In other work the idea of a generative lexicon is proposed, which is a type of knowledge representation that combines lexical items with world knowledge. This contains:
- constitutive role of object: physical proporties and knowledge of the object; explicit semantic typing
- formal role: configuration w.r.t. other objects, likely location and spatial relation
- telic role: function, use and purpose of object
- agentive role: how the object was created or produced; its origin
- domain knowledge: predicate-argument structures, event structures, causal chains, role subtyping
But of course the question again is... do we need to make all of this by hand? How can we acquire this automatically?
Hypothesis of this work (not proven but tentative):
As explained in the next sections, we hypothesize that telic and agentive roles might be learned from large text collections, that constitutive and formal roles might be learned from large imagery collections and that event structures and causal chains might be learned from large text and imagery collections. (9)
Learning common sense and world knowledge from text
Author believes that "event semantics can be learned from unannotated text data" (9).
- cf. unsupervised dependency parsing
- cf. probabilistic graphical models (e.g. Bayesian models with latent variables) to estimate language models
- deep belief networks
- idea is to predict event structures
- if this is successful, you can do implicit semantic role labeling
- future work how to relate this to argumentation mining
Learning common sense and world knowledge from imagery
A lot of human world knowledge comes from years of visual perception. There is some work in this direction to learn object properties from observation object manipulations.
Learning common sense and world knowledge from imagery and language data
Weak linguistic supervision:
When humans learn about the world their perception is often guided by the weak supervision of language data. We tell a child what an object is, what its properties are, often when watching or manipulating the object. This brings us to the question, how can a machine learn common sense and world knowledge from text and visual data? Can the machine ground language in visual perception and as such acquire the necessary common sense and world knowledge? (10)
How to integrate with argument mining?
Open question.