Toward Constructing Evidence-Based Legal Arguments Using Legal Decision Documents and Machine Learning | Kevin D. Ashley, Vern R. Walker
Part of abstract:
For decision texts, we use the Vaccine/Injury Project (V/IP) Corpus, which contains default-logic annotations of argument structure. We supplement this with presuppositional annotations about entities, events, and relations that play important roles in argumentation, and about the level of confidence that arguments would be successful. We then propose how to integrate these semantic-pragmatic annotations with syntactic and domain-general semantic annotations, such as those generated in the DeepQA architecture, and outline how to apply machine learning and scoring techniques similar to those used in the IBM Watson system for playing the Jeopardy! question- answer game. We replace this game-playing goal, however, with the goal of learning to construct legal arguments. (p. 176)
Introduction
Why are decision texts relevant for argumentation in new cases?
- "Authorative applications of the rule conditions and concepts to identified situations"
- "a ground truth for testing predictions about outcomes in new cases with new evidence"
- "patterns for successful and unsuccessful argumentation"
- "guidance in retrieving, extracting, and organizing evidence for new arguments and new situations"
Practically speaking, "decision texts can provide syntactic and semantic variability for machine learning" (176)
We propose using the DeepQA architecture to generate arguments about findings of fact based on factual evidence (evidence-based legal arguments), as contrasted with arguments about rules based on principles or policies. (176)
- Default-logic models can model a) requirements as trees of rule conditions; b) inference rules/ chains of reasoning
- Additional labels that encode "background information": a) information w.r.t. argumentation function (involved entities, events, relations); b)a confidence level for the succes of a particular argumentation pattern
The idea is to use the (dual) model to provide a scoring function for possible arguments, similar to how IBM Watson scores possible answers using the DeepQA framework.
Decision-Document Corpus
Vaccine/Injure Project corpus. Often mentioned in other papers, but not public. Concerns decision texts (35, 15-40 pages each).
A rule tree in the DLF framework is used, which encoded presuppositions where edges are implications of other presuppositions/propositions with possible values "true", "undecided" or "false".
Each case’s logic model (or “case model”) is divided into two major parts [6]: the governing system of legal rules and the factfinder’s assessment of the evidence in the case. The governing system of legal rules is formally represented as an inverted “rule tree” (a directed acyclic graph with the root node at the top). (177)
This is pretty mind-boggling and requires much work:
The researchers assign plausibility-values to the terminal or leaf assertions in the inference tree, so that the plausibility-values and truth-values throughout the model reflect the factfinder’s evaluation. The evidence assessment inference trees for the 35 vaccine decisions in the V/IP Corpus contain over 1,000 assertion nodes, with the vast majority of those being attached to the three branches of the Althen causation conditions. (177)
Again, very complicated (I introduced the enumeration for readability; it is NOT in the original quote)...
"The process of extracting the DLF structure of a decision document can be decomposed into various tasks, including:
- (A) the identification/annotation of text sentences that state findings of fact on critical rule conditions;
- (B) the identification/annotation of text sentences with respect to their “horizontal relevance” (i.e., the particular branch of the rule tree in which the sentences play an important inferential role);
- (C) within each branch of the rule tree, the identification/annotation of the horizontally relevant sentences with respect to their “vertical relevance” (i.e., their location in the inferential levels of support within the branch);
- and (D) the extraction from each of the sentences in steps (A)-(C) of the precise content or information (the evidentiary assertion) that plays an inferential role in the reasoning" (177-178)
I glossed over the rest of the paper. I have the impression this paper describes the data mostly and proposes a framework, but not yet the development of an actual system. I find it very hard to see how an actual system would work in this framework and how the annotated data will be used in training and allow application of insights to unseen data.