Measuring Similarity among Legal Court Case Documents | Mandal, Chaki, Saha, Ghosh, Pal, Ghosh
Abstract
Prior works have proposed network-based and text-based methods for measuring similarity between legal documents. However, there are certain limitations in the prior methods. Network-based measures are not always meaningfully applicable since legal citation networks are usually very sparse. On the other hand, only primitive text-based similarity measures, such as TF-IDF based approaches, have been tried till date. In this work, we focus on improving text-based methodologies for computing the similarity between two legal documents. In addition to TF-IDF based measures, we use advanced similarity measures (such as topic modeling) and neural network models (such as word embeddings and document embeddings). (1)
The results from this article indicate that neural network models that use word or document embeddings outperform traditional text-based similarity methods, network methods (e.g. using citation graphs), as well as hybrid text- and network approaches.
Introduction
Good intro on the importance of finding relevant cases in common law.
Network-based approaches have the limitation that citation networks are typically very sparse, so often they cannot be meaningfully used.
Text-based methods are often used and work, but are what the authors call "primitive" (i.e. TF-IDF approaches have been around for ages; although I do have to say primitive is in my opinion too derogatory since they do work well especially with SVMs).
STEP 1: Experiment with different document representations; specifically try to isolate and use the meaningful and important units of the document.
Among the different possible representations of a legal document, we consider (i) the whole document, (ii) the set of paragraphs of the document, (iii) summaries of the document, and (iv) the set of RFCs (the text surrounding a citation) in the document. These representations are attempts to separate out the meaningful / important units of the document. (2)
STEP 2: Compute similarity.
- Prior work uses TF-IDF vectors + cosine similarity.
- This paper: vectors based on LDA, word embeddings, document embeddings
- Authors claim to be the first to do this: "To the best of our knowledge, this is the first attempt at using neural network based models to measure the similarity among legal case documents" (2)
Related work
Network-based
Kumar et al proposed network similarity measures:
- Bibliographic coupling: "Bibliographic coupling between two documents is the number of common ’out-citations’ of these two documents." (2)
- Co-citation: "co-citation is the number of common ’in-citations’ between the two documents." (2)
Minocha et al (2015) proposed:
- Dispersion: "For a given pair of nodes, the Dispersion measure attempts to measure to what extent are the common neighbors of the nodes not themselves well connected." (2)
- Embeddedness: "Embeddedness, in simple terms means, how close the common neighbors of a node are to form a clique." (2)
All these methods depend on how well-connected the citation graph is. In practice, this is disappointing. The paper's dataset had 80% isolated nodes, which makes network measures unusable to find these documents. This can be explained by the fact that lawyers do not note than the whole case based, but at best only the most relevant prior cases.
Text-based and hybrid
Several typical TF-IDF + cosine approaches
Kumar et al. (2013) introduce hybrid approach called paragraph-links (PL)
- Break docs into constituent paragraphs and treat them as separate entities
- Compute cosine similarity between pairs of paragraphs of docs A and B
- Paragraphs have TF-IDF represntation (but for IDF terms from first paragraphs only seem to be used, which is surprised to me)
- If similarity is higher > threshold, add a paragraph link between them
See Sushanta Kumar, P. Krishna Reddy, V. Balakista Reddy, and Malti Suri. 2013. Finding Similar Legal Judgements under Common Law System. 103–116.
Measuring document similarity
- Find document representation
- Measure similarity between documents
Document representation
This paper experiments either with ways of partitioning the document, and with ways of selecting relevant sections from it.
- Whole document
- Problem: one legal document may discuss various legal issues
- e.g. "validity of an evidence, the severity of a crime committed and the j.interpretation of a law" (3)
- So a prior case may not be that similar except on one discussed legal issue; this whole document approach does not really capture this partial similarity
- Document summary
- Summaries highlight the most important concepts from a document
- This paper creates a selective summary, i.e. just a concatenation of the important sentences
- Paragraphs
- Remove paragraphs < 20 chars
- N.B.Separators between paragraphs is corpus-specific
- Compute pairwise similarity between all paragraphs of d1 and d2
- Document similarity must then be some weighted average mean
- RFCs of the document (reason for citation)
- "Prior works [10] hypothesized that such RFCs include the important legal issues discussed in a document, which also provide the reasons why a future case should cite the current case.4 Thus, focusing on the RFCs seems to be a promising way of identifying the important legal issues described in a document." (4)
Measuring similarity
- TF-IDF vectorizer
- LDA Topic model
- Also cosine similarity on LDA vectors used
- Word2vec word embeddings
- As in many papers, vector size 200 works best
- Paragraph or RFC embeddings by taking weighting average of word embeddings (TF-IDF weighted)
- Doc2vec document embeddings
- Vector size 100 used
Experiment
Dataset of crawled case documents, as well as gold-standard with similarity judgements by Kumar et al.
Correspondence of scores with similarity judgements by experts are computed with the Pearson correlation coefficient.
Results
- Whole document: Doc2Vec does best in combination with whole documents.
- Overall, this combination had the highest correlation with expert opinion
- Summary: LDA does well on summaries (but depends on summarization algorithm)
- Paragraph: both word2vec and doc2vec perform equally well
- RFC -> LDA