Distilling Jurisprudence through Argument Mining for Case Assessment Rachel Rietveld Julien Rossi Evangelos Kanoulas r.d.rietveld@uva.nl j.rossi@uva.nl e.kanoulas@uva.nl Law School Amsterdam Business School Amsterdam Business School University of Amsterdam Institute of Informatics Institute of Informatics Amsterdam, Netherlands University of Amsterdam University of Amsterdam Amsterdam, Netherlands Amsterdam, Netherlands ABSTRACT 2 OPEN NORMS IN LAW This paper describes an AI-based legal assistant that would support Open norms are general, flexible terms, used to provide justice in case assessment based on the extraction of arguments from court individual cases and are therefore not restricted to specific situa- decisions. Open norms are often used in law as a way to set a leg- tions that need to be written in advance. In this way, yet unknown islative frame that allows for individual justice under the specific cases can always fit in the norm.[4] Most common examples are circumstances. By extracting the underlying arguments from exist- terms like such as reasonable, fairness or equity. Some open norms ing corpus of annotated court decisions, a reliable legal framework are referred to as semi-open since they are stricter and clearer with can be formed in order to give more insight and clarity to both less room for interpretation. The more open the norm, the more judges and parties in the case. This contributes to legal certainty room there is for subjective interpretation and discussion. and equality, without losing justice in individual cases. The system By applying open norms, judges can deliver tailor-made solutions provides legal specialists a practical tool to help their clients in a that suit the material status quo of the specific case. This is a rather legal procedure, without the necessity of going through all relevant difficult task. A judge is supposed to carefully collect the evidence, case law themselves. The general public will also benefit from the retrieve all arguments that can be of any value, without benefiting available knowledge. one of the parties over the other. At the same time, he needs to take into account the inequality of parties and therefore protect KEYWORDS the weaker one. As a result, the judge may ask questions to get a Open norms, Legal, Arguments Retrieval, Legal certainty, AI, justify better idea of the underlying case, but too much translating facts to legal grounds is not allowed. The broader scope of an open norm gives judges a desired flexibility. In the ideal situation, the judge 1 INTRODUCTION will base his conclusion on all the relevant factors and still in the It seems Artificial Intelligence and Intelligent Assistants already framework of the law, of the purpose of the legislator and reflect have their place in law firms, although much of this place is devoted the social view of that time. to Document Management Systems and various Information Re- Although justice might be reached, one of the downsides of open trieval systems. We suggest that under a stricter scrutiny, it would norms is the high level of legal uncertainty, which leads to unpre- be clear most of these systems were pushed by the business need to dictability and a lack of transparency.[5] Parties for example, do not reduce the cost of support systems, a vital need under the paradigm know what the outcome of legal reasoning will be. A settlement is of the billable hour [10]. difficult to reach if a clear framework is missing. This may also lead We observe that most of the Information Retrieval or Natural to more legal procedures and therefore higher costs, since a judicial Language Processing problems are by large unsolved in the legal verdict is the only way to determine who is correct. In addition, the domain. The specifics of the language, or the manipulation of con- judge, who of course is in favour of maintaining his independence cepts and abstractions contribute to create this distance between and stature, needs some sort of framework to come to a justified legal texts and general literature, as observed in [1]. ruling. If such a scope is lacking, it is too difficult to come to an We make the hypothesis that lawyers internalise a summarised independent justified outcome, because then a judge’s subjective knowledge of case law, that allow them to assess individual situa- opinion can be the only guide. tions with regards to norms, and that this process can be formulated In the common law system very few laws are written. The legal as a knowledge extraction and summarising task[7] [6]. An Intel- scope is set by a constitution, filled with open norms and some ligent Assistant can leverage this acquired knowledge to come prohibiting laws. What is not written is allowed, unless there is case forward with the right pointers for the assessment of the individual law on the certain topic or situation. The value of jurisprudence is situation, and put it in perspective of the legal landscape crafted by significant, since it sets the scope of rules and (new) legal outcomes. court decisions. Although the civil law system is more about written laws and gives the assumption to be covering all possible situations beforehand, In: Proceedings of the First International Workshop on AI and Intelligent Assistance jurisprudence is still of great value. It may be impractical or even for Legal Professionals in the Digital Workplace (LegalAIIA 2019), held in conjunction with ICAIL 2019. June 17, 2019. Montréal, QC, Canada. impossible to foresee, and at the same time describe, every set of Copyright ©2019 for this paper by its authors. Use permitted under Creative Commons characteristics of a case in all possible combinations. This is why License Attribution 4.0 International (CC BY 4.0). also in the civil law systems open norms are used. The result in Published at http://ceur-ws.org. Rietveld, Rossi, Kanoulas Figure 1: From unstructured documents to data Figure 3: Modelling Legal Qualification Figure 2: Training and Evaluation Figure 4: System Usage and Workflow both systems is a lack of certainty how judges will rule. It is also To this end, we design an annotation process, pictured in Fig. 1, quite demanding for judges themselves to deal with a broad and driven by data needs for the downstream tasks, as assessed by unclear range of possibilities. Would it not be better to provide all a collaborative team of AI and Law practitioners, providing the (legal) actors with a clear scope of arguments that influence the necessary expertise to shape both content and form, and align final outcome? If not defined in advance, it seems better to establish the outputs of the system with the needs of the practitioners, in a the ruling aspects, apparently found to be of value by other judges perspective that makes sense for the business at hand. This business and parties. If possible, giving such insight will lead to more legal perspective will as well guide the choice of relevant metrics to equality and certainty to all involved. optimise, as there is no one-size-fits-all answer, but only tailored answers to specific needs. 3 SYSTEM DESCRIPTION Given the designed task and tools, annotators proceed with iden- tifying which fragments of the text qualify as Arguments, either as 3.1 Argument Extraction facts or as statute law references, and which pairs are tied together. We formulate the Argument Extraction problem as an Information The pairing associates one fact to one statute law, while each fact Extraction task that has to be repeated once per jurisdiction and/or can be paired to multiple statute law references, and each law article language. We consider that an information system capable of link- can be paired to multiple fact. ing an argument of a case to an article of the common law, can only The training and evaluation process, pictured in Fig. 2, follows be valid within the boundaries of the jurisdiction where the article the classical work flow, under the supervision of an AI expert that applies. Further usage of the system is bounded by the accessible provides the modelling setup and the evaluation tools. The selected underlying data. metrics will provide the drive for system improvement. Distilling Jurisprudence through Argument Mining for Case Assessment We formulate the Precision-Recall balancing problem with re- judge is most likely to rule in a case. By developing these kinds of gards to the usage of the system and the type of downstream errors AI aids, legal research can become less time-consuming and more we aim at minimising. It seems natural to prioritise Precision over effective. They can even contribute to more equal outcomes, thus Recall, for a knowledge that is correct but maybe not exhaustive. legal certainty and equality. This however, is disputable, since the We observe recent similar researches in the field of Legal Infor- predictions are still mainly black boxes. Therefore, parties cannot mation Retrieval, using either traditional techniques: Ontologies define how their case might differ from the data the prediction is and Combinatory Categorial Grammar in [3], [2] introduces Active based on. Judges furthermore, miss out on the substantiation and Learning with non-neural Classifiers; or deep learning techniques: therefore, will not be able to have the complete legal or practical Hierarchical RNNs in [11], Dense Word Embeddings and Topic Mod- framework in order for them to implement the prediction. eling in [9]. We also refer to recent implementation architectures, In our practical research, we aim for the public community to such as [8]. benefit from the result and the usage of legal professionals to differ from existing use seen in the reduction of time and costs, as well 3.2 Modelling Legal Qualification as in the creation of clarity in the legal practice, specifically, in the We consider legal qualification as the association of 3 information: use of open norms. Parties can either solve their own conflicts once a fact or a collection of facts, a statute law article, and a judgement they know what is relevant to a case and what ought to be required whether the facts constitute a breach of that article or not. Having behaviour, and thus prevent a conflict about the expected behaviour. annotated those links in the existing corpus of decisions, we for- Consequently and in the most optimal result, this would lead to mulate a system that models the legal qualification of facts. It is fewer justice seekers, which can be a relaxation of the judicial pictured in Fig. 3 system. If parties still want a judge deciding on their dispute, our As an AI system is also an attempt to reverse-engineer a decision research will be an aid to come up with the valuable circumstances, scheme, it enables both prediction based on known inputs, and since some details brought up by parties can be irrelevant and may description of which inputs are the most likely to make a given lead to piles of legal documents for a judge to read through. On the decision. other hand, parties may be incomplete and consequentially harm The review and the evaluation of that model will be driven by their case. In the Netherlands for instance, a judge can only base his legal expertise. The descriptive power of the model will be evaluated ruling on facts presented by parties. He is allowed to ask questions with regards to its capacity to summarise which clusters of facts and to order a hearing, but should always keep equality of parties explain a legal qualification, to show which factors can explain in mind and is, thus, limited. Once less necessary effort is needed, variation in the certainty of a qualification. this will as well lead to a relief of burden of the judicial system. 4 LEGAL SCOPE In this paper we want to focus on open norms in labour law, more 6 RESEARCH GOALS AND PRACTICAL specifically, on dismissal law. One of the open norms in several RESULT legal systems is about culpable acts. If for instance, an employee The goals of our research and development are multiple and depend violates the law or breaks the rules within the company, it is up to on the user type. We aim to reach the following: the judge to decide whether, in his perspective, the act is a reason for dismissal or if there are reasons to rule in favour of the employee. • Lawyers and legal support: Through an easily accessible Acts under the scope of culpable behaviour are diverse. It covers for desktop tool they can assess a case that is based on available example theft, being late for work, sexual harassment and breaching facts extracted from previous case law. The goal is not to a non-compete clause. This wide range of specific situations, makes simply provide them the chance of a possible breach of the the outcome even more uncertain and unclear for both parties open norm, but to give them an overview of relevant prac- and even for the judge. First of all, it is not decided beforehand if a tical arguments. These arguments can be used to build up situation will fall under the scope of possible culpable behaviour and their case and that will colour in the open norm with actual secondly, if it does, it is not definite the situation will be qualified and practical circumstances. These users would only need as a culpable act. The outcome relies on a variety of details, from to define and ultimately substantiate and/or prove which the view of both the employer and employee. What leads to fair aspects are applicable. It saves them time and they do not dismissal within one company may not be severe enough within have to compile extensive pleadings and other procedural another, whilst the act can be exactly the same. documents that might contain irrelevant arguments that If machine learning makes it possible to come up with an insight only distract from the valuable ones. into arguments that are of value for legal decision making, it will be • Judiciary: For judges the system will also be available through a great gain. This way of extracting relevant factors from previous a desktop. Similar to lawyers, they have the ability to assess judge-made law, can contribute to these main principles in law, jurisprudence. They can quickly see what aspect(s) is or was achieving greater legal equality, certainty and even justice. relevant in other comparable cases where there was a breach of the invoked open norm. Higher courts will prevail over 5 PRACTICAL IMPACT lower judiciaries. As a result, judges have a practical tool AI is influencing the work of legal practitioners in several ways. In that gives them an idea of what the common relevant and addition, AI is used nowadays to give insights on how a specific justified aspects are that colour in the open norm, since the Rietveld, Rossi, Kanoulas outcome is based on a large number of previous cases, with- (legal experts, legal operations experts) and the form (AI experts) in out losing their own power to decide on the case. The higher a holistic manner. Cross-education of team members on key topics goal is to create more legal equality and certainty. is essential to lower the walls and break free from a top-down • Legal sciences: Most research studies demand case studies. waterfall approach. Questions that arise, such as what the trend in jurisprudence is or whether there has been a change in ruling opinions, 8.3 Research Questions are mostly about open norms and can only be answered RQ1. How to arrange annotations of Court Decisions? How to use by going through a reasonable number of cases. It takes unstructured legacy annotations? time and requires skills as constructing databases to get a RQ2. How to extract arguments from court decisions? How to sufficient insight. With the targeted practical tool research optimise on a business relevant metric? How to assemble a can be done in less time and might be of higher quality since knowledge base? How to evaluate accuracy and complete- the amount of investigated cases should increase. ness? • Public community/Parties: Law should be accessible to RQ3. How to model a decision boundary based on facts associa- everyone. This includes not only access to court, but also tions? How to model facts importance? How to model facts having knowledge about what rules are applicable and what composition? the practical results of these rights or prohibitions are. By setting a clear scope of practical behaviour that colours in an 9 CONCLUSION open norm, we can increase the desired accessibility. Parties In this paper, we introduced a novelty Intelligent Assistant based on would have access to an online tool and could perform a Artificial Intelligence techniques, that would leverage the knowl- self-assessment. This could form the basis of negotiations edge contained in court decisions in order to support stakeholders between parties to either prevent a conflict or to solve one. of the judicial system at large in assessing the effectiveness of poli- Summarised, the research goal and practical result are aimed cies, the actual usage of legislation and norms, the merits of new to allow a new value proposition for different users. Instead of cases with regards to jurisprudence. predicting outcomes, we are focusing on how to get to a justified We associate the tool to multiple stakeholders of the judicial outcome that is in line with common ideas. Furthermore, we will try system with different use cases where the additional information to relief the judicial system by preventing conflicts, since potential benefits all parties. parties should get a clearer insight of how they should behave We recognise that the challenge lies not only on the technology towards each other, in order for them not to breach the applicable itself, but on the capacity to produce a fruitful collaboration between open norm. the world of legal work and the world of Artificial Intelligence, and the co-engineering of a solution that is driven by business or societal 7 RESEARCH PLAN needs. We will be applying this research on an existing open norm, culpable act, in order for us to have enough relevant case law. The legal REFERENCES [1] Piyush Arora, Murhaf Hossari, Alfredo Maldonado, Clare Conran, Gareth JF research on this topic has been mostly completed humanly, so Jones, Alexander Paulus, Johannes Klostermann, and Christian Dirschl. 2018. we tend to know what relevant practical arguments judges use to Challenges in the development of effective systems for Professional Legal Search. support their ruling in case of the underlying question whether (2018). [2] Cristian Cardellino, Serena Villata, Laura Alonso Alemany, and Elena Cabrio. certain behaviour is culpable. In this way we can train the system, 2015. Information Extraction with Active Learning: A Case Study in Legal Text. and see whether the generated outcomes are correct. In Computational Linguistics and Intelligent Text Processing, Alexander Gelbukh (Ed.). Springer International Publishing, Cham, 483–494. [3] Mauro Dragoni, Serena Villata, Williams Rizzi, and Guido Governatori. 2016. 8 METHODOLOGY Combining NLP Approaches for Rule Extraction from Legal Documents. In 1st Workshop on MIning and REasoning with Legal texts (MIREL 2016). Sophia 8.1 Data Sources Antipolis, France. https://hal.archives-ouvertes.fr/hal-01572443 [4] H.L.A. Hart. 1994. The Concept of Law. Oxford: Oxford University Press. We consider using Court Records as a primary source of data. A [5] Atilla Kun. 2018. How to Operationalize Open Norms in Hard and Soft Laws: properly formed court ruling will motivate the final decision by Reflections Based on Two Distinct Regulatory Examples. International Journal of elaborating on how the given facts were legally qualified. The prolif- Comparative Labour Law and Industrial Relations 34, no. 1 (2018). [6] Gloria T. Lau, Kincho H. Law, and Gio Wiederhold. 2005. Legal Information eration of "Open Data" policies increase the amount of information Retrieval and Application to e-Rulemaking. In Proceedings of the 10th International made public each year. Most of this data comes unstructured, and Conference on Artificial Intelligence and Law (ICAIL ’05). ACM, New York, NY, USA, 146–154. https://doi.org/10.1145/1165485.1165508 most of the structured data provide mainly a structure for docu- [7] K Tamsin Maxwell and Burkhard Schafer. 2008. Concept and Context in Legal ment meta-data, such as bibliographical details, time stamps and Information Retrieval.. In JURIX. 63–72. document origin. We have to consider that our main source of data [8] Marc Opijnen, Nico Verwer, and Jan Meijer. 2015. Beyond the experiment: the eXtendable legal link eXtractor. In Workshop on Automated Detection, Extraction will consist of raw text files. and Analysis of Semantic Information in Legal Texts, held in conjunction with the 2015 International Conference on Artificial Intelligence and Law (ICAIL). [9] Srishty Saha and Karuna P Joshi. 2019. Cognitively Rich Framework to Automate 8.2 Multidisciplinary Teamwork Extraction and Representation of Legal Knowledge. UMBC Faculty Collection The annotation project receives a lot of attention with regards to (2019). [10] JD Supra. 2018. When Will the Billable Hour Die? Conversation with Law Firm aligning its design with the needs of the downstream tasks. We Change Agent John Chisholm (Part Two). https://www.jdsupra.com/legalnews/ form an open collaboration team able to tackle both the content when-will-the-billable-hour-die-95132/ Distilling Jurisprudence through Argument Mining for Case Assessment [11] Rao Xi and Ke Zhenxing. 2018. Hierarchical RNN for Information Extraction Engineers and Computer Scientists, Vol. 1. from Lawsuit Documents. In Proceedings of the International MultiConference of