Extending Full Text Search for Legal Document Collections Using Word Embeddings | Landthaler, Watl, Holl, Matthes in JURIX 2016

Abstract

In this paper we explore a novel method that provides the ability to find not only exact matches, but also semantically similar parts for arbitrary length search queries. We achieve this without the application of ontologies, but base our approach on Word Embeddings.

Recently, Word Embeddings have been applied successfully for many natural language processing tasks. We argue that our method is well suited for legal document collections and examine its applicability for two different use cases: We conduct a case study on a stand-alone law, in particular the EU Data Protection Directive 94/46/EC (EU-DPD) in order to extract obligations. Secondly, from a collection of publicly available templates for German rental contracts we retrieve similar provisions. (73)

Introduction

Normally, there's a need for ontologies:

Depending on the use case it might be necessary to consider synonyms and related words for each word in the search query. A common approach to address this problem is the application of domain ontologies documenting relationships such as synonyms or antonyms. However, the creation and maintenance of such ontologies is often difficult and expensive. (73)

Usefulness of approach:

Our approach can be used to find related rights and obligations or arguments in legal documents. The proposed method is not restricted to process single text documents. It can be used to find related text passages in a collection of laws, judgments or contracts, too. Due to the fact that representing the context of legal terms is very important, the word embeddings approach is well suited for legal documents. (74)

The following sentence suggests that legal jargon suffers less from ambiguity than everyday natural language:

In addition, terms within a legal domain are often used in a consistent way of understanding and interpreting. This is a beneficial characteristic for the application of our method. (74)

Using word embeddings can be seen as an implicit form of query expansion (because it takes context into account) when used for text search, cf. p 75.

Related work

Many, but just cherry picking stuff I may read.

Unclear which system, but it's a system for capturing concepts:

Grabmair et al. have adapted and implemented an analysis pipeline to find relevant semantic patterns in vaccine injury decisions [9]. They trained a system to find relevant linguistic and semantic patterns that capture legally relevant concepts and their context. (74)

Word embeddings for argument mining:

Word Embeddings have been applied on legal texts for argumentation mining: Rinott et al. [10] detect evidences for claims in debates. In particular the authors calculate the cosine similarity of all pairs of Word Vectors in a claim and candidate text fragments that can contain evidence for the claim and rank the candidates. (74-75)

Naderi and Hirst [11] detect text fragments that support claims in political debates by computing similarity scores between sentences followed by a Support Vector Machine classification. (75)

Rule-based techniques are dominant:

most approaches to extract rights and obligations from privacy laws rely on rule-based techniques. (75)

Word embeddings

Summary:

Traditional sparse vector representations (bag-of-words) consume either a lot of memory or require specialized vector operations for sparse vectors and typically drop the word order. While Word Vectors are still considered a bag-of-words approach (addition is commutative), Word Vectors do encode the general context of words in dense vectors with manually chosen size. Mathematical operations (vector addition, multiplication) can be carried out on the vectors while preserving their inherent semantic characteristics. (75)

  1. Use word2vec to compute word embeddings for the German Civil Code
  2. Accumulate vectors of words per norm
  3. Compute t-SNE dimensionality reduction per norm vector
  4. Result (figure 2) shows a good separation of the norms, suggesting that the underlying word embeddings do capture semantic differences between norms

Method description and MVP

The point of the chosen method is to not only find literal matches (which apparently is the norm in legal search?) but also sentences that are semantically very close. As word embeddings capture some semantics and context, using them is a strategy that makes sense:

The goal of our method is to provide a way to find not only exact matches of arbitrary length search phrases, but also to find parts in the text that might be relevant or semantically close, but differ in one or multiple words. We use Word Embeddings and the ability to sum or average vectors over multiple words. (77)

  1. Accumulate word vectors of search phrase to produce a single search phrase vector
    • N is the number of words in the search phrase
  2. Define a window of size N and move it over the corpus with step size n/2
  3. Compute cosine similarity of search phrase with accumulated vectors
    • Unclear: accumulated per ... norm?
  4. Rank vectors on cosine similarity and select top X
  5. Refinement: per top match from step 4, do a more detailed shift with step size 1 but now with window size 3*N, and again repeat step 3 and 4

Parameters word2vec: some standard choices, but minimal word count is set to 1 so that all words have a word vector.

Dealing with words in the search phrase that do not exist:

Missing word vectors for words of the search phrase can be a problem and so far we ignore words where no corresponding Word Vector exists or restrict the user to search by interactively selecting text in documents of the corpus.

Not optimized prototype:"If a document is appended to or edited in a collection, we recalculate all Word Vectors within that collection." (78)

Note on implementation: " Mikolovs word2vec implementation also outputs a vocabulary and we use this vocabulary to calculate an integer representation of all texts in a collection, because integer comparisons are much faster than string comparisons (used for the lookup table that maps words of a text to their Word Vectors)." (78)

Case study

I would like to find such hybrid approaches with word embeddings and parse trees:

We assume that our approach could be improved by using Word Embeddings trained on larger datasets and also from hybrid approaches with parse trees and rule-based systems.

Conclusion

Chosen method is computationally too intensive and is not scalable to larger corpora.

For the future, it would be interesting to experiment with other natural language pre-processing steps in combination with Word Embeddings, for example stemming, stop-word removal, part-of-speech tagging or parsing. The development of a standardized quality measure for Word Vectors could help to improve the understanding and reliability of this technology. (81)

The authors say it's hard to evaluate their method, because relevance is subjective. That is true, but that is trivially true of all information retrieval problems which also do have to offer some indication of score. The authors also do in fact give their manual subjective interpretation of which results are relevant by doing a manual search. This could have been treated as a golden standard to compute e.g. a MAP score.