Using Factors to Predict and Analyze Landlord-Tenant Decisions to Increase Access to Justice | Hannes Westermann

Introduction

Paper on results from JusticeBot project.

we identified 44 factors that occur in disputes where the tenant seeks a remedy due to problems with the rented apartment, such as the existence of bedbugs, high noise levels or problems with insulation. (1)

Identifying and analysing patterns can give insight in characteristics of winning cases:

For the second dataset (39 cases with bedbugs, drawn from the first dataset), we developed in-depth factors and used them to tag the cases. We found a number of plausible correlations, such as the average damage award being higher in cases with infestations of high intensity. (1)

Data: 1 million case decision texts

Goal of JusticeBot is better decision making for the tenants:

The goal of the project is to provide a chatbot interface to tenants wanting to know which remedies are available to them. The chatbot will ask a number of questions to determine the tenant’s factual situation, and then provide the chatbot user with statistical information about their likelihood of success and averages of awarded damages, as well as displaying previous similar cases. This information could help the tenant get an overview of their legal situation and help inform their decision making. (1)

One good takeaway: good idea to have such a specific niche of law to start developing this type of application with.

From 149 cases a taxonomy was created with which these cases were tagged. (I assume taxonomy is the same as "ontology" here).

Prior work

  • representing legal rules and cases with rule trees and factors
  • text analytics to identify factors
  • predicting case outcomes
  • explaining predictions in terms of legal concepts and case analogies

  • Rule tree: "tree of authoritative rule conditions", adapted from Default Logic Framework.

    • Combined with "domain model comprising legal concepts" (2)

ML-approach that predicts factors using word embeddings:

Falakmasir and Ashley [8] assembled a corpus of trade secret cases, employed word embeddings, trained an ML algorithm for each factor using a subset of manually-classified cases, and predicted the factors that apply in cases, achieving F1 values of .65, an improvement over the results in [4].

Approach that starts with factor descriptions, expands on them, and converts them into rules to identify factors:

Starting from factor descriptions, Wyner and Peters [15, 16] applied WordNet expansions and expert knowledge to generate factoroids, finer-grained semantic terms related to each factor from which they generated factor-annotating rules in GATE.

How to explain predictions?

Integrating machine learning (ML) and expert knowledge for purposes of explanation is an important topic of current interest [6]. For example, IBP based its predictions on factors identified automatically in the case texts and explained them in terms of intermediate legal concepts in its domain model and factor-based analogies to similar cases [4]. Similarly, our project will base predictions on automatically identified factors and explain them in terms of legal concepts from the rule trees, as far as we know, a first for a civil law domain. Other factor-based prediction approaches take values into account [7, 9], which we may also be able to incorporate. (2)

Methodology

Rule tree and Default Logic Framework:

We used the Default Logic Framework [11] to represent a legal rule as a set of propositions, one of which is the conclusion and the remaining propositions being the rule conditions (i.e., the conditions under which the conclusion is true). An integrated system of such rules that governs a governmental action can be represented as a “rule tree” [14] (2)

Rule trees allow us:

  1. classify cases by legal issue that is decided
  2. classify portions of text within those decisions by the particular legal issue they address
  3. match new cases with old cases based on the rule conditions

Source:

Use of the rule tree conditions as a method for classifying cases decided by the RDL allows us to organize cases by the legal issues that are decided, and to classify the portions of text within those decisions by legal issue. This enables us to match past cases to future cases, with the unit of analysis (or topic) being the legal rule condition (3)

Steps of Grounded Theory Method; incremental manual determining of factors by creating memo's and working them out; factors are worked out and placed in a detailed taxonomy (in this case about details related to bed bugs); cases are annotated with the taxonomy (individual sentences not annotated)

Output is a "factor vector":

"The result of this annotation process is to create a “factor vector” for each case - a profile that contains the specified factors of each case as well as the outcome." (4)

Results

Dataset 1

149 cases with 86 cases of rent reduction.

44 factors.

Some notable observations:

  • Very low inter-annotator agreement on the annotations
  • Some factors are easy (bedbugs, yes/no), but more refined factors had low agreement as well
  • There is a correlation between the amount of confirmed factors and winning the case (i.e. I guess this shows the factors are reasonable)
  • Also some correlation with the amount of rent reduction
  • Some factors more strongly correlated with outcome and thus better indicators
  • Factors treated as binary: present/not present.
  • Prediction of judge decision using Random Forest Classifier on factor vectors
  • Prediction of months of rent reduction using regression
    • But the regression performed worse than a baseline that just takes the average amount of rent reduction

Dataset 2

Subset of 39 cases with bedbugs that are tagged with bedbug taxonomy.

34 factors, including 12 from bedbug taxonomy.

Also didn't outperform baseline.