Utilizing Vector Space Models for Identifying Legal Factors from Text | Mohammad H. Falakmasir a and Kevin D. Ashley
Goal is to identify case factors:
An important target of argument mining efforts in the legal field has been to extract factors from case texts. See [1, Chapter 10]. Legal factors are stereotypical patterns of fact that tend to strengthen or weaken a side’s argument in a legal claim. [2, p.27].
Classification criterium factor:
For purposes of modeling, a conclusion that a factor applies in a case is based on classifying at least one sentence in the text as an instance of the factor.
Notice that this classification is very explainable! Why does this case contain this factor? Because of this sentence.
Approach includes multiple steps:
Our main research goal is to improve the performance of automatically classifying the texts of trade secrets misappropriation cases by their applicable factors. As an initial step we asked how well vector space models (VSMs) can identify factors in the case texts (see Section 2). Using the domain model of Figure 1, once factors are identified, one could also identify the legal issues litigated in the case. (p. 3)
- Identify/classify factors
- Experiments with vector space models here
- Four using TF-IDF representation
- Four using dimensionality reduction techniques
- Given factors (and a domain model on those factors), classify which legal issues are discussed in the case
Vector Space Models
Statistical studies of semantics represent meaning as a probability distribution over a set of latent dimensions using the bag-of-words hypothesis or the distributional hypothesis [18]. The key idea is that if units of text have similar vectors in a term frequency matrix, they tend to have similar meaning. (p.~185)
- BoW hypothesis: word frequencies are indicative of relevance
- For document relevancy; too crude for topic similarity
- Distributional hypothesis: "words that appear together in the same context tend to have similar meaning" (186)
- "The context could be a sentence, or perhaps even a fixed window of words. In general, shorter windows tend to capture syntactic features while longer windows tend to capture more semantic relations." (186)
Vector space models of semantics represent meaning as a coordinate in a high- dimensional “semantic space” (186)
Vector Space Models use the distributional hypothesis by trying to represent texts in "semantic space", i.e. in terms of vectors in a latent space. Semantic similarity between texts can then be computed through distance metrics, such as cosine similarity.
Another approach is dimensionality reduction techniques for topic modelling, such as LSA (SVD on term-document matrix) or LDA. I wasn't familiar yet with non-negative matrix factorization (NMF).
Data & Methods
- Scrapted texts with references to two sources of legal rules
- Gold-standard corpus (172 cases; VJAP corpus) with each sentence labelled according to 26 trade secret factors.
- Couldn't find anything about this VJAP corpus
- VJAP corpus sources include sources from HYPO, CATO, SMILE, VJAP programs.
- This CATO?: https://www.cato.org/criminal-justice
This work is without feature engineering;
Our goal in this project is to see how far one can go without feature-engineering. At the same time, we remain open to applying techniques to efficiently incorporate domain knowledge where feasible, a task for future work.
Pipeline & Experiments
- Tokenization, PoS tagging, extracting main verbs
- n-grams with n=1,2,3
- Vectorization. Term-document matrix using BoW hypothesis
- 1) BoW representation
- 2) BoV (bag-of-verbs) representation because many factors relate to agent actions
- 3) TF-IDF on terms
- 4) TF-IDF on verbs
- IDF weights computed on larger corpus of 1600 scraped cases, rather than gold standard of 172 case
- Transformation / dimensionality reduction
- 5) LSA ( K = 20, 50, 100, 200
- 6) NMF ( K = 50, 100 )
- 7) LDA ( K = 20, 50, 100, 200
- 8) Hierarchical Dirichlet Process (HDP) (the only non-parametric one, i.e. doesn't require specification of K dimensions!)
- Prediction and Evaluation
- Using the 8 representations, train a supervised classifier to predict factors
- SVM classifier with linear kernel
- "investigate the association of each VSM with the targeted labels, the legal factors" (188) -> How exactly? Coefficients of linear kernel?
- F1 metric
Extra processing:
- Filter out extremely frequent words (>90% of docs)
- Filter out extremely rare words (< 5 counts)
With respect to the BoV model, note that the chosen verb representation is atypical and does not use verb stems:
We use a modified form of the verb by concatenating the immediate conjunct of the verb according to the dependency parse results. For example in this form, we have separate tokens for the verb “disclose” including, disclosed, not disclosed, not to disclose, have disclosed, etc. This way of representing verbs is different from the forms used in the topic modeling literature that often uses the stemmed version of the verbs (“disclos” for all of the above forms) and removes the conjuncts as stop words. The main reason is that in the legal context in general and in considering the factors in particular, the verb tense and the modals play a pivotal role in the fact finding process of the decision maker and should not be mapped into the same dimension (considered as the same “token”) in the feature space. (187)
Results & Discussion
TF-IDF (Verbs), the TF-IDF model that used n-grams of the verbs, and LSA (400) did best. On the positive side, this performance (and that of all the VSMs tested) is better than that of previously reported efforts (see Section 6). (189)
The five most predictive features for some factors seem to make sense, which suggests that the VSM have picked up some of the factor semantics, despite minimal domain modelling:
These results shows some potential for applying VSMs in the legal domain with minimal domain modeling.
Why minimal? No explicit connection to a taxonomy except through the labelling based on the domain model.
Problem 1: sentence-level vs. document-level labeling
One way to explain these results is that our gold-standard corpus provides labels at the document level while trade secret factors are usually discussed on a sentence level, a problem also pointed out in [15].
Problem 2: False negatives
Moreover, the labels are annotated mainly to study the interaction of factors in the trade secret domain, and there are some false negatives due to cases where the factor was mentioned but not applied in the decision (e.g., the factor may have been discussed in a description of a case cited for other reasons.)
Problem 3: possible improvement in preprocessing to deal with noise
In an ideal scenario, the document representation should be able to filter-out noise and irrelevant case-specific information from the raw text files and aggregate information that discusses the factors actually applied and the issues actually decided. This may require the identification of sentence role types such as court’s findings of fact.
Problem 4: do all factors have enough training samples?
Related work
Starting from description of factors + WordNet expansion
Wyner and Peters [15] tried to solve this problem by starting from the description of the factors and using WordNet expansions and expert knowledge to generate factoroids, plausibly semantic terms that are related to each factor. They used factoroids to generate rules as a part of GATE system to annotate cases with respect to factors and pointed out the utility of creating a gold-standard corpus for machine learning. (191)
GATE: https://gate.ac.uk/
TODO Wyner and Peters: Wyner AZ, Peters W. Lexical Semantics and Expert Legal Knowledge towards the Identification of Legal Case Factors. JURIX 2010. p 127-136.
Unsupervised clustering approaches:
In e-discovery, unsupervised learning enables exploratory clustering of documents and selecting seed sets for supervised learning. For example, the Categorix system clusters documents for review using PLSA, a probabilistic alternative to LSA as we used [6]. In earlier work, Uyttendaele, et al. applied an unsupervised, non-hierarchical clustering method and a TF-IDF vector space model like our TF-IDF (Terms) VSM to group paragraphs in court opinions thematically for purposes of summarization [7]. Schweighofer and Merkl applied self-organizing maps, a kind of unsupervised neural network, to explore and cluster documents in a corpus of European legal texts concerning public enterprises [8]. (191)
[8]: Schweighofer E, Merkl D. A learning technique for legal document analysis. ICAIL 1999. ACM Press. p 156-163
Not clustering cases, but instead clustering case citations:
Winkels, et al. applied unsupervised learning to identify natural clusters of case citations of statutes, (as opposed to clusters of cases themselves as we do) for eventual use in a legal recommender context [10]. (191)
Call for unsupervised approach to learning legal semantics to generate structured case notes: [13], McCarty
Word embeddings
Highlighting this because this is of interest to me for possible thesis work.
NMF + topic coherence via word2vec to study topic drift over time:
More recently, Panagis, et al. applied non-negative matrix factorization to a large set of judgments from the EU Court of Justice and European Court of Human Rights and selected clusters using topic coherence via word2vec to study topic drift over time [11].
[11]: Panagis Y, Christensen ML, Sadl U. On Top of Topics: Leveraging Topic Modeling to Study the Dynamic Case-Law of International Courts. JURIX 2016. p 161-166.
word2vec word embeddings for extracting similar obligations and provisions:
Landthaler, et al. employed word embeddings (word2vec) in extracting similar obligations from the text of an EU Data Protection Directive 94/46/EC (EU-DPD) and similar provisions from a collection of German rental contracts [12].
[12]: Landthaler J, Waltl B, Holl P, Matthes F. Extending Full Text Search for Legal Document Collections Using Word Embeddings. JURIX 2016. p 73-82