On Top of Topics: Leveraging Topic Modeling to Study the Dynamic Case-Law of International Courts | Panagis, Christensen & Sadl

Code: https://github.com/derekgreene/dynamic-nmf

In this paper we use dynamic topic modeling, a family of unsupervised machine learning techniques, to gauge the shifts in the content of the case-law of international courts over longer time spans. Our results indicate that dynamic topic modeling is a powerful and reliable tool to systematically and accurately track legal change over time and enhance our understanding of courts and their influence on the law.

Goals (all on p. 161):

1) Provide a proof of concept that topic modeling can be used to identify the topics in a dynamic case-law collection, 2) identify possible shortcomings of these methods, and finally, 3) make machine learning useful to researchers who study case-law.

Previous work

  • "Livermore et al. [5] used LDA to study the agenda of the U.S. Supreme Court."
  • "Panagis and Sadl [6] used LDA together with network analysis to detect the topics discussed in the most cited paragraphs of the case-law of CJEU, relevant to European citizenship, however, without examining how the topics develop through time."
  • "A different use of topic modeling was demonstrated by Winkels [9], where it was applied to build case-law recommender systems."

Method

Dynamic modelling on each time window with NMF. I.e. for each time window a separate topic model is trained. Then for each of these time windows, the "topic coherence" score is computed using Word2Vec:

In the present paper we use a topic selection process called Topic Coherence via Word2Vec (TC-W2V) [3]. With this method the topic coherence is computed as the sum of pairwise cosine similarities of the topic terms in the word embedding space. Thereafter, model coherence coh(T), is simply the average coherence over all topics.

See [3]: Greene, D. and J.P. Cross, Exploring the Political Agenda of the European Parliament Using a Dynamic Topic Modeling Approach. Political Analysis, (forthcoming).

Topic models

Optimal value K is determined per time windows. The K (topics) that optimizes the coherence score is picked.

Model interpretation

  • The topic model gives a certain amount of topics per time period
  • These topics are labeled by hand
  • The frequency of these topics is tracked throughout the time windows
  • This shows general shifts in focus by the investigated legal bodies

A topic is just a collection of words. So what I don't understand is how the authors identify the "same" topic throughout time windows, because I imagine none of these topics is exactly the same. Also, per time window you have different amounts of topics.

Another thing, e.g. with LDA you can get a different word distribution per topic on each rerun because this uses a stochastic process. Perhaps the authors of this paper do not have this issue, because they don't use LDA but the deterministic NMF. Even then, I'm curious how they can label the "same" topics throughout the time windows (which are 5-10 years long, depending on the amount of judgements in that decade).

Conclusion

First conclusion is that the analysis shows that some topics remain relevant throughout the lifespan of courts.

Second, dynamic topics that consist of a mixture of topics indicate either the presence of a generic term, e.g. directive, that is attached to a number of subjects (the dynamic topic T5 of CJEU), or that there has indeed been a topic drift in the Court’s thematic.

It's unclear to me how in the main text this concept of "dynamic topic" as a "mixture of topics" is used.