Recognizing cited facts and principles in legal judgements | Olga Shulayeva, Advaith Siddharthan, Adam Wyner
- Reasonable agreement between human annotators on annotating which sentences contain cited facts and principles
- Why "cited" facts?
- It is feasible to automatically annotate sentences containing legal facts and principles based on linguistic features
- precision and recall of 0.79 and 0.89 and kappa of 0.72 with gold standard! That's actually pretty amazing.
"Citations from existing case law are used to illustrate legal principles and facts that define the conditions for application of legal principles in the current case." (108)
What's quite nice about the introduction is that it clearly distinguishes "material" and formal citations.
Citation analysis can help legal practitioners to identify which principles have applied in a certain case and which facts have been selected as the 'material' facts of the case, i.e. the facts that influenced the decision and which are crucial in establishing the similarity between two cases. (108)
Ideally finding similar facts and principles, which is the basis for arguing why a current case should be a precedent. The following section also nice highlights an idea I was playing before, namely that this type of analysis allows us to differentiate the juridical relevance of a case for different stakeholders.
The information that is embedded within the cited cases includes the legal principles and facts that are used to reason to a decision. Optimally, a legal professional finds a cited case with the same facts and legal principles, and so can argue that the decision for the current case should be that of the precedent; similarly, the opposing party may identify precedents with opposing principles to argue the decision should be otherwise. (108)
INTERMEZZO:
Research idea: distill "fact" statements from a text and compute similarity over those w.r.t. other cases; semantic similarity; similar entities (NER)? The idea is: with similar facts and principles you can argue that the decision in the current case should be similar to the decision in the other case. (Something I seem to assume here: for a similar type of accusation!) There can also be scenarios where you have similar facts and principles, but the decision is different. These cases may be more relevant to other stakeholders, such as the opposing party (case law for the counter decision) as well as for legal scholars (what distinguishes these cases that have similar facts and principles but opposing conclusions?) Ideally this requires a label w.r.t. the conclusion.
Evaluation: comparison with FATE system. Current evaluation is very topic-based: about half of the top k relevant documents is on the same topic. Can we improve on this by factoring in legal facts and principles?
Explainability: how can we define counterfactual explanations with this setup? I see two options:
1) if we label with case outcome, it would be interesting for several stakeholders to automatically see which "facts" or "principles" would minimally have to be different in order to change the case outcome 2) more like the current FATE setup: which facts or principle would assign the current case to another "topic" label
Labeling with case outcome:
- Mining "beslissing" section. Feasible?
- Separate classifier that uses sentiment analysis?
- Conclusions are only comparable when the type of claim is similar
- "For example, citations can be classified according to sentiment polarities: confirmative or negative (Moravcsik and Murugesan 1975); positive, neutral or weak (Teufel et al. 2006)." (113)
For mining facts I can probably exploit the structure of case texts
- Extract facts based on structure and directly use these facts
- Use step 1 to create a labeled dataset for training a separate classifier that finds evidentiary statements
END INTERMEZZO
In this paper, the chosen approach is more that argument (or evidence-) mining here can easily provide more information about cited texts.
Further relevance of identifying relevant facts and principles:
To support citation analysis (discussed further in Sect. 3.1), existing electronic tools, such as electronic databases,2 provide one word summaries for relationships between cases (e.g.‘applied’). However, it is uncommon for them to extract information about the facts and the legal principles of the cited cases. This means that on many occasions readers are required to make themselves familiar with the full text of multiple law reports in order to identify the applicable law and the correct way to apply it. Thus, citation analysis tools save some labour by providing a preliminary filter on relevant cases, yet, identification of particular cases and the essential details require further manual effort. (108)
- "this work is focused on the computational analysis of the language of the law rather than on legal theory" (109)
- "cases are considered to contain ratio decidendi, which can be translated as a reason for a decision, an important piece of reasoning that is incorporated into the argumentation structure of future decisions." (109)
- See p. 109 for some definitions, which are roughly used for annotation as well
Data: "A gold standard corpus is created, with sentences containing cited legal principles and facts manually annotated." (109)
Method: "A Bayesian Multinomial Classifier is then applied to the corpus using a set of linguistic features to automatically identify these sentences." (109)
Future work:
"The reported studies lay the basis for further applications, including creation of metadata for search and retrieval purposes, compilation of automated case treatment tables containing summaries about legal principles and material facts of cases, and automated analysis of reasoning patterns and consistency applied in legal argumentation." (109)
Motivation in a legal context
- "improve the compilation and use of tertiary sources" (109)
Geist (2009) state that three type of tools help with finding factually similar cases:
- Primary: case law, legislation records
- Secondary: commentary about case law and legislation
- Tertiary: "legal digests and citators that provide collections of systemised case law information" (110)
- Wasn't familiar with this
- "Legal citators, which constitute tertiary sources, are systemised collections of law reports that facilitate a faster search of cases and provide brief case treatment summaries, indicating which cases were cited during the discussion." (110)
Benefits of automated analysis of citations:
- Dealing with information overload of increasing case base as well as the size of unstructured citation info (111)
- E.g. a case may have 7000 incoming citations, 1000 outgoing; this is a bizarre amount of work to estimate the relevant "material" facts from all these cases
- "the complexity of interpretation and variance in conceptualisation can also make processing of citation information more challenging." (111)
- Too much irrelevant facts and citations in cases
- Sometimes insufficient agreement between legal professionals concerning case law
- "searching for legal principles supported by a case may be complicated as there has never been a definitive guide on how to correctly interpret a case." (112)
- "Instead, the law that is supported by a certain case will be defined as legal principles. Existing interpretations of case law provided by courts as citations are binding and can be used to illustrate the legal principles that are supported by the cited case and material facts that are necessary for the principles to be invoked." (112)
I'm not sure I get the latter point.
Related work
Citation analysis
Focuses on citation studies. Two strands of approaches:
- network analysis on citations (e.g. see ) Van Opijnen (2014)
- Classification systems that estimate the "treatment" status of the cited case
I find this one particularly interesting: reasons for citing
The researchers note that different parts of a case can be cited. Studying the reasons for citation can provide valuable information for a legal researcher. Their approach relied on RFC (reason for citing), a patented technology that allows extracting reasons of why the case has been cited. RFC performance was summarised in the patent (Humphrey et al. 2005), which explored a methodology of ‘‘identifying sentences near a document citation (such as a court case citation) that suggest the reason(s) for citing (RFC)’’. (113)
Apparently this approach doesn't use machine learning and mostly uses word frequences.
This study does not classify citations.
Many features for annotating statements are domain-specific; this work is more domain neutral:
our work is domain neutral. The features (n-gram and entity types) used in Grabmair et al. (2015) are not as clearly tied to statement annotations as in our approach. Our approach does not consider ranking or high level properties of decisions, but tying specific statements of legal principles and evidence to citations within decisions; our results are promising.
Argument extraction
- "identification of statements of legal principle and evidence tied to a citation" (115)
- argumentation zoning
- is focuses on the argumentation of the current case, not the cited case
- citations and summarizations of other papers are usually an "OTHER" or "PROCEEDINGS" category
This paper does not work on overall argumentation structure, but mostly places this work as one step that can be used in other systems and work.
Discusses several approaches that rely on manual work, e.g. knowledge base of cue phrases that are then matched through regular expressions.
Refs to ...
- Wyner 2010 - Approaches to text mining arguments from legal cases with context-free grammar
- Ashley 2013 -Toward constructing evidence-based legal arguments using legal decision documents and machine learning
There's a distinction between ratio (binding statements based on the facts on which the decision is based) and obiter (more persuasive statements); this distinction does not play a further role here.
Manual annotation study
- 50 common law reports on limited amount of topics
- identify sentences containing legal principles or facts of the cited cases; other sentences are "neutral"
Guidelines, some key points:
- annotation areas were paragraphs with at least one citation
- 14% of the control corpus sentences were in an annotation area (control: check annotator agreement)
- citation instances were pre-annotated
- Principle work definition: "A legal principle is defined as any statement which is used, along with facts, to
reach a conclusion." (117) Linguistic markers:
- deontic modality: must, must not, may
- Fact work definition: "statements bearing on what uncontroversially exists, occurred, or is a piece of information" (118)
- "For our purposes, only sentences that refer to events which occur outside the court hearing are annotated; this excludes procedural facts." (118) -> This is because they are specifically investigating cited facts; this qualification may not be appropriate for other applications
- non-modal expressions and denoting expression: specific, actual, definite.
Results: very good kappa; particular annotator 1 had intra-annotator kappa of k=0.95 so this annotator (who had legal training) created the golden standard of 50 reports for the automated annotation.
Automated annotation study
Naive Bayesian Multinomial Classifier
Features:
- Part of speech tags.
- Facts are expected to be in past tense
- Legal principle sentences often use epistemic and deontic modal qualifiers
- Unigrams.
- Dependency pairs.
- Goal is to help overcome the limitation that BoW unigrams does not factor in word order
- These pairs are grammatically linked, not just collocated like with n-grams
- Length of the sentence (in words)
- Position in the text (scale 0-1)
- Cit — a feature which indicates whether there is a citation instance in the sentence.
ML method:
Due to limited training data, approach was to design features thought to be useful, then prune the irrelevant ones automatically.
- TF-IDF scaling of feature counts
- Attribute selection (InfoGain)
- Naive Bayes Multinomial
- 10-fold cross validation
Data was unbalanced: 60% neutral, 30% principles and 10% facts.
Results
- Feature selection reduced features from 51576 to 887
- Good precision and recall across categories
- Facts and principles are easy to distinguish (see confusion matrix)
- Neither category results in most mistakes
- Classifier does slightly better than annotator 2
- Suggests that automated annotation is a feasible option!
Table 4 lists top 100 features by info gain. Mainly POS tags, Unigrams, and Dependency pairs.
I wasn't familiar with dependency pairs, examples are : case-that, is-there, concern-was, concern--case (and so on).
More insight in the model, for principles:
In the learnt Bayesian model, part of speech features such as ‘MD’ (modals), ‘VBZ’ and ‘VB’ (present tense), ‘WRB’ (WH-adverbs), as well as unigrams such as ‘a’ and ‘an’ (indefinites), ‘is’, ‘be’ and ‘has’ (present tense), ‘may’,‘must’, ‘will’ and ‘can’ (modals), ‘if’, ‘whether’, ‘or’, ‘unless’ and ‘where’ (conjunctions) have higher probability for the the principles class, as do general purpose nouns such as ‘person’ and ‘party’. This is intuitive as principles are often stated in present tense, scoped with a modal verb, presented in general terms (thus using indefinites or general nouns such as ‘person’ or ‘party’) and also typically relate more than one clause using conjunctions. (122)
For facts:
On the other hand, the main features that predict facts are the use of past tense (‘VBD’, ‘VBN’,‘was’, ‘were’, ‘had’, ‘concerned’, etc.) and proper names (‘NNP’, ‘Mr’, ‘Mrs’, etc). The principle features that select for the neither class are the use of the first person (‘I’, ‘my’ and ‘judgment-my’), other references that indicate the sentence is about the current case rather than a cited one (‘this’, ‘case-this’, etc), words indicating that the judge is summarising or quoting (‘says’, ‘says-he’, ‘summarised’) and the use of adjectives and non-WH adverbs (‘JJ’ and ‘RB’), which might indicate an opinion being expressed. (122)
Evaluation per feature group:
Finally, we evaluated the performance of the classifier on each type of feature (part of speech, unigram and dependency) separately, as reported in Table 5 along with the majority class baseline that labels all sentences as neutral. While using only part of speech tags achieves 63% accuracy, dependency features by themselves achieve 81% accuracy. The combination of all three feature types results in the best results (85% accuracy). All feature sets outperform the baseline. (122)
Error analysis
First of all, I really like the structure of this paper.
W.r.t. mislabeling as "neutral" the authors think this is because a lot of facts are not presented in separate sentences, but as part of legal reasoning.
This type of error is relatively obvious:
The main cause of error for the automatic annotation of principles was that the gold standard only annotated principles from cited cases, but often these were linguistically indistinguishable (in our machine learning approach) from discussions of principles by the current judge; i.e principles expressed by the current judge should have been annotated as neither, but were frequently annotated as principles.
Conclusion & Future work
Automated analysis of legal principles and facts within cited cases allows identifying the key information about the cited case which can be used for many purposes, including creation of detailed case treatment summaries, improvement of search and retrieval methodology for the case law and many others. (124)
This work demonstrates the feasibility of automatic identification of legal principles and facts that are associated with a case citation. This functionality could, for example, allow a legal practitioner to not only search, say in Google, for citations mentioned in a case, but also the associated legal principles and facts, providing deep access to and insight into the development of the law. (124)
Although useful for IR tasks, the authors have not currently used the minig information for ranking.
Finally, we have only addressed accessing cited legal principles and facts, which is distinct from ranking and relating precedents, i.e. Shepardisation. In future work, the source material annotated here could be used to investigate the automation of Shepardisation as well. (124)