=Paper=
{{Paper
|id=Vol-2632/MIREL-19_paper_2
|storemode=property
|title=Reasoning in Court with the Outcome of a Machine Learning Algorithm with Case Law as an Input
|pdfUrl=https://ceur-ws.org/Vol-2632/MIREL-19_paper_2.pdf
|volume=Vol-2632
|dblpUrl=https://dblp.org/rec/conf/jurix/Vanderstichele19
}}
==Reasoning in Court with the Outcome of a Machine Learning Algorithm with Case Law as an Input==
https://ceur-ws.org/Vol-2632/MIREL-19_paper_2.pdf
Reasoning in court with the outcome of a machine
learning algorithm with case law as an input*
Geneviève Vanderstichele1
1
Judge in the court of appeal of Ghent (Be)
Deputy judge in the Benelux Court
Affiliate researcher in the KULeuven Centre for IT & IP Law (Be)
LL.M (U.Va - USA) - MPhil in Law (U. of Oxford - UK)
DPhil in Law candidate (U. of Oxford - UK)
1 Introduction
Imagine a lawyer invoking in a bankruptcy case as an argument in favour of her
client the outcome of a natural language processing (NLP) and machine learning (ML)
algorithm that analyses past judicial decisions in bankruptcy cases, and ‘predicts’ out-
comes, taking into account many different dimensions of bankruptcy cases – such as
assets, liabilities, debtors, creditors, economic sector, the judge and the lawyers in-
volved etc. These technologies give insights in patterns and probabilities, whereas ad-
judication is about arguments, reasoning and giving reasons for a decision.
This paper addresses the question how courts, if at all, can integrate such outcome
in their practice. How should a court approach a case where a plaintiff or defendant
uses the outcome of NLP and ML technologies as an argument? Is the outcome of such
an NLP and ML algorithm admissible in a court procedure? If so, what is its value?
How can a court engage with it? This essay argues that such outcome is not to be con-
sidered as a fact in a court case, nor as law, nor as a secondary source of law such as
comments or treatise. Furthermore, it claims that it is a sui generis concept, a concept
of its own kind, with its own set of questions to be addressed before it can be integrated
in a reasoned court decision, deciding on its (in)admissibility and its value.
The paper aims to contribute to the integration of the quantitative methods of com-
puter science, data science and statistics with the qualitative method of the law, taking
the rule of law into account.
*
I thank Prof. Dr. Henry Prakken and Prof. Dr. Kevin Ashley, warmly for their valuable com-
ments on the earlier draft of this paper. I express an equally deep gratitude to my colleague
Dr. David Harvey for his detailed and valuable comments on my M.Phil thesis (see n4 below).
Copyright © 2020 for this paper by its author. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
2
2 Preliminary remark with regard to the terminology
Supervised machine learning (SML) and NLP are the technologies on which I
mainly focus in this article. Hereafter I replace the term ‘supervised machine learning
with case law as an input’ by ‘Legal Analytics’ or ‘LA’. The term ‘outcome of Legal
Analytics’ and its abbreviation ‘ocLA’ substitute ‘outcome of supervised machine
learning with case law as an input’. This allows for a more fluent reading. In addition,
SML with the aim to ‘predict’ court decisions is a part of the broader subject of Legal
Analytics, to which intelligent search, legal information extraction and automated case-
based and rul-based reasoning also belong.
3 Qualification of the ocLA for use in court
In current practice, courts analyse the facts and the law. In coming to a decision,
there is a subsumption of facts and law in a continuous process; facts are selected and
analysed from the perspective of the relevant law, and at the same time the relevant law
is selected on the basis of the facts of the case. This process can be iterated if necessary,
depending on the arguments submitted by the parties. Furthermore, courts in common
law countries are bound by stare decisis, the principle ‘that decisions of earlier cases
sufficiently like a new case should be repeated in the new case’.1 On the other hand, to
create a supervised machine learning algorithm, an input, system and output are neces-
sary. With supervised machine learning, defining the problem implies defining the out-
come variable.2 In addition, facts, law, metadata of cases and societal values of a num-
ber of cases are selected for the input.3 Information is combined in a different way
compared to traditional dispute resolution. In addition, machine learning algorithms
sort, search, count and classify according to their requirements and specifications, ren-
dering patterns and probabilities. The results are interpreted, and the outcome consti-
tutes new information that is abstracted from the original cases that were used as an
input. Each step in the algorithm-making process uses a methodology that is at least
partially different from the methodology used in adjudication, distinguishing existing
legal dispute resolution practices from supervised machine learning.4 I claim that it is
1
Ronald Dworkin, Law’s Empire (Hart Publishing, first published 1986, 1998) 24.
2
Solon Barocas and Andrew D. Selbst, ‘Big Data’s Disparate Impact’ (2016) Cal. L. Rev. 677-679.
3
For a critical discussion from the perspective of a judge of 9 machine learning experiments
with case law as an input, see G. Vanderstichele, ‘The normative value of Legal Analytics. Is
there a case for statistical precedent?’ (August 30, 2019), Chapter 2. Available at
SSRN: https://ssrn.com/abstract=.
4
AI and Law also share in part a method, semi-formal modelling, and a common subject matter,
the coordination of human behavior. See Bart Verheij, ‘Towards ICAIL 2013 in Rome: the
start of the next 25 years of the research program AI and Law’, in: Trevor Bench-Capon et
al., ‘A history of AI and Law in 50 papers: 25 years of the international conference on AI and
Law’, AI and Law, DOI 10.1007/s10506-012-9131-x.
3
precisely because of this partial difference in methodology that the ocLA cannot qualify
as any of the concepts used in current adjudication.
1. I first argue that the ocLA cannot be considered and treated as a fact.
An issue is one of fact where it concerns the reliability or credibility of direct evi-
dence, or inferences from circumstantial evidence.5 Facts relate to the case that is being
heard. They are often concrete and must be proven by evidence specific to the case and
given by the parties. In principle, the facts of a case cannot be proven by reference to
facts in other cases.6 By contrast, the ocLA gives a correlation between parts of a num-
ber of court decisions. As the input for the algorithm consists of an interpretation of
both facts and law and metadata of an amount of earlier cases, and as the input was
turned into a vector and a statistical analysis was performed, the ocLA is about more
than a concrete fact or set of facts in a concrete case. For instance, the concrete trade-
mark and its use by the trademark holder are specific to a case against an alleged in-
fringer. The ocLA of a number of trademark cases rendering a prediction on infringe-
ments in similar cases constitutes information that, for instance, allows to compare the
given case to a larger number of cases, and thus might provide a new context for a
concrete case, but does not meet the legal standard of proof in a given dispute. Also,
the input for and the methodology of the algorithm discern it from a mere statistical
analysis of a phenomenon. The latter can be analysed as a fact, whereas the former has,
as discussed, aspects to it which do not allow to reduce it to a fact.
To what extent could the ocLA be equated to expert evidence, if at all? Evidence
involving a technological or scientific aspect and the ocLA both have their technicality
in common. The skilled witness must demonstrate to the court that they have relevant
knowledge and experience to give either factual evidence, which is not based exclu-
sively on personal observation or sensation, or opinion evidence.7 Where the skilled
witness establishes such knowledge and experience, they can draw on the general body
of knowledge and understanding of the relevant expertise.8 Hence, the question must
be examined whether the ocLA could receive the status of a ‘non-human skilled witness
about the general body of knowledge and understanding of the relevant expertise’. I
argue that this is not the case. First, new technologies represent information in a novel
way, involving interpreted facts and the applied law from earlier cases, which is epis-
temologically as well as methodologically different from a general body of knowledge
and understanding of the relevant expertise.9 Therefore, the ocLA cannot receive the
same treatment as expert evidence currently receives in court procedures. For the same
reason, the ocLA is different from expert opinions on foreign law. By analogy, the use
of technology assisted review (TAR) or e-discovery is considered by the U.S. District
5
S.H. Bailey and M.J. Gunn, Smith & Bailey on the modern English legal system (Sweet &
Maxwell, 3d ed., 1996) 415.
6
There are exceptions in case of established facts of criminal cases, a discussion of which is
beyond the scope of the paper.
7
Myers v The Queen [2015] UKPC 40 [2015] 3 WLR 1145, para 63.
8
Ibid.
9
See G. Vanderstichele, (n3), Chapter 1.
4
Court for the Southern district of New York as a tool to assist the parties in the discov-
ery process,10 and not as a fact in itself.11 The ML algorithm is a tool with the possibility
to assist in dispute resolution. Second, in the UK experts can and often do give evidence
of fact as well as opinion.12 However, in the UK, as well as in civil law countries, it is
unlawful to shift the responsibility for deciding the issues brought before the court to
an expert witness,13 be it a scientist of any kind, or a computer engineer. By law, such
decisions belong to the competence of the court.14 The input for the ML algorithm is in
part the law, and a ML algorithm requires selective decisions about the facts and the
law, by selecting input data, requirements, specifications, a specific ML algorithm, etc.
Therefore, equating the ocLA to expert testimony would equate to the court shifting
their decision-making responsibility, which goes against the fundamental principle
‘iura novit curia’ or ‘da mihi factum, dabo tibi ius’,15 applicable in both civil and com-
mon law systems. Finally, it can be questioned whether evaluating new technologies
exclusively from the perspective of scientific peer review sufficiently respects the prin-
ciple of due process. Parties ought to have the possibility to adequately contest the al-
gorithms,16 by questioning them without intermediation.
Moreover, for the first and second reasons it would not be legal to have a human
expert witness explain the methodology and the significance of the ocLA to the court,
as this would amount to explaining the court what the law and its application is, which
is the task of the courts. At least, lawyers and judges ought to have a sufficient under-
standing of the involved quantitative methodologies to be able to question thoroughly
an expert witness with regard to the invoked ocLA in a court procedure. This poses a
question with regard to legal education: what exactly do lawyers need to study, in ad-
dition to the substance of the law? It also raises the question about organising the time
of sitting judges in order to allow them to acquire new knowledge and skills.
The technical nature of algorithms and their outcome, might incite to consider the
outcome as a fact with special authority, as has been done to establish social science in
law.17 In that concept, findings of social science research that are relevant to create a
rule of law are treated as a source of authority, rather than a source of facts, and treated
in the same way as legal precedent.18 The outcome of social science research would
have similarities with facts, as it is empirical research, and with the law, as both research
findings and the law are general; findings of social science research extend beyond the
10
Rio Tinto PLC v. Vale S.A. [2015] (S.D.N.Y.) WL 8722294.
11
See infra for further discussion.
12
Kennedy v Cordia (Services) LLP [2016] UKSC 6, [2016] 1 W.L.R. 597, para 40.
13
See e.g. Kennedy v Cordia (Services) LLP [2016] UKSC 6, [2016] 1 W.L.R. 597, para 49;
Advisory Opinion on the Western Sahara, [1975] I.C.J. Rep. 12, 138; Article 112.1 Dutch
constitution; Article 40 and 144.1 Belgian constitution.
14
Ibid.
15
Resp. ‘the court knows the law’ and ‘give me the facts and I shall give you the law’.
16
Rebecca Wexler, Life, Liberty, and Trade Secrets (2018) 70 Stan. L. Rev. 1343, 1375.
17
John Monahan and Laurens Walker, Social Science in Law. Cases and Materials (Foundation
Press, 1st ed. 1985, 7th ed. 2010) 387.
18
Ibid. 388.
5
concrete facts involved in the study.19 However, this special status for social science
findings is not necessary to allow them in court. They are related to empirical facts and
can be analysed as such. Even when scientists speak about ‘scientific laws’, they remain
empirical findings. The arguments discussed above with regard to expert evidence in
general are valid here as well.
2. Second, the ocLA is not law as we know it today, for the following reasons.
First, the ocLA is conceptually too different from current precedent to be equated.
The ocLA renders a probabilistic outcome after a statistical treatment of data, which is
very often not transparent. In general, statistical outcomes of machine learning algo-
rithms rendering an estimation of x%, compared to the chosen baseline, of what a court
might decide, call for the conclusion that predicting how a court might decide in a future
case may be sufficient for a lawyer to tentatively advice and orient their client who
would be willing to estimate what their chances are of winning or losing a case, but is
not a precedent in common law, nor a statute or an authoritative court decision in civil
law countries. Importantly, LA do not give reasons for their outcome, while reasons are
precisely what doctrine of precedent and jurisprudence require to qualify a court deci-
sion as law. A proposition of law in precedent is only binding when it forms part of the
ratio decidendi, i.e. the reason(s) for the decision of the court: ‘As with judicial or other
opinions, what carries weight is the reasoning, not the conclusion’.20 Furthermore, Ar-
ticle 6 §1 of the European Convention for the Protection of Human Rights and Funda-
mental Freedoms (ECHR) prescribes that in the determination of their civil rights and
obligations or of any criminal charge against them, everyone is entitled to a fair and
public hearing within a reasonable time by an independent and impartial tribunal estab-
lished by law. The guarantees enshrined in Article 6 §1 include the obligation for courts
to give sufficient reasons for their decisions. A reasoned decision shows the parties that
their case has truly been heard.21 Additionally, the Convention and the case law holding
the proposition that reasons are essential in adjudication are supported by the jurispru-
dence of Joseph Raz. According to his theory, all normative phenomena are normative
in as much as, and because, they provide reasons or are partly constituted by reasons.22
For Raz, courts ought to act on exclusionary reasons, which are legal standards23 and
court decisions are issued for reasons which are deemed to justify them.24
Such reasons are absent in Legal Analytics. At best, they can be inferred from the
computation, which requires an additional reasoning. Without the justifying reasons,
19
Ibid.
20
Dingley v Chief Constable, Strathclyde Police 1998 SC 548, 604. See also the discussion in
G. Vanderstichele (n3), Chapter 1, section 3.
21
H. v. Belgium, 30 November 1987, § 53. See also: Suominen v. Finland, 24 July 2003, § 36;
Carmel Saliba v. Malta, 29 November 2016, § 73; Ruiz Torija v. Spain, 9 December 1994, §
29; Hiro Balani v. Spain, 9 December 1994, § 27. All cases accessible via ; see also ; both accessed last on 20 July 2019.
26
David Lehr and Paul Ohm, ‘Playing with data: what legal scholars should learn about Machine
Learning’ (2017) 51 U.C. Davis L. Rev. 671, 707-708.
27
Ibid. 707.
28
Ibid. 715.
29
Ibid. 709.
30
They have been implemented for random forests, gradient boosting algorithms, but not for
complex methods of deep learning. See ibid. 710.
31
Ibid.
32
See G. Vanderstichele (n4) chapter 1, section 3.
7
however resorting to merely referencing it, as this could amount to basing the decision
solely on automated processing. Courts rubberstamping a decision in an individual
case, decided solely on the basis of an ocLA violate the provision of Article 22.1 GDPR.
There are conditional exceptions in the following sections of Article 22 GDPR.
First, explicit consent is such an exception. This legal ground is for instance used in
current Dutch court cases about health care insurers who obtained explicit consent from
their contracting patients to handle disputes by private, partly automated courts. Upon
discussions whether such consent can be qualified as really voluntary, and whether the
due process requirement of article 6.1 ECHR was respected, the health insurers almost
entirely discontinued their project on private dispute resolution.33 In addition, auto-
mated decision-making could be authorised by Union or Member State law, on the con-
dition that suitable measures are taken to safeguard the data subject’s rights and free-
doms and legitimate interests. Thus, a legal intervention is arguably required before the
ocLA would be admissible, let alone before it could obtain normative value. Also, it
will have to be defined what such suitable measures would be.34
Although it is not a law, I point to principle 5 ‘Under user control’ of the European
ethical Charter on the Use of Artificial Intelligence in judicial systems and their envi-
ronments of the European Commission for the Efficiency of Justice (Cepej), as it illus-
trates a line of thinking about the research question and is intended for AI developers
as well as ‘public decision-makers’.35 The 5th principle states among others36 that ‘pro-
fessionals in the justice system should, at any moment, be able to review judicial deci-
sions and the data used to produce a result and continue not to necessarily be bound by
it in the light of the specific features of that particular case’.37
For the above mentioned three groups of reasons, the ocLA cannot qualify as prec-
edent by itself, but needs further discussion in and consideration by the court to be
useful in dispute resolution.
3. Third, the outcome of machine learning algorithms is not to be equated with a
secondary source of law for the following reasons.
The ocLA and secondary sources of law, such as comments on the law, have in
common that they both analyse the facts and law. The ocLA might in addition have
metadata as an input. Some information on lawyers or judges, not related to the law or
the facts of the cases that are commented, can be incorporated in a classic comment as
well, be it not as a precise parameter. However, the difference in methodology comes
again to the forefront. A comment discusses the case in its context. Part of the discus-
sion of a comment may provide the reader with a different view on the context. By
33
See Marc Van Opijnen, ‘Robotrechters’ (8 February 2018) IBestuurOnline
(in Dutch); see also http://www.e-court.nl/wp-
content/uploads/2018/11/20181123-VVG-e-Court-De-Staat-NL-site.pdf (in Dutch) both ac-
cessed 13 June 2019.
34
This is beyond the scope of the paper.
35
Adopted on 3-4 December 2018, accessed 20 July 2019.
36
I refer to the text of the charter for the three other elements of this principle; ibid..
37
Ibid.. See also the lemmata 7 and 35 of Appendix I of the Charter.
8
contrast, supervised ML algorithms initially remove context. The ocLA possibly con-
stitutes a whole new result that can be used as a new element of context, but this comes
after the numerical treatment and is not part of it. The ocLA is a different set of observ-
ables, compared to the observables of law, facts and metadata from the input. The al-
gorithm creates a proxy for reality.38 By contrast, even when the classical comment
sheds a new light on the content, the set of observables remains unaltered. In addition,
a comment offers an interpretation after the analysis, whereas the Legal Analytics ren-
der patterns, with at least two instances of interpretation. There is a preliminary inter-
pretation that was built into the system through the selection of the facts, or the require-
ments of the system by humans. Afterwards, the outcome is open for a second analysis
and interpretation, again by humans. Finally, a comment or treatise is most often fo-
cused on a qualitative analysis, whereas a supervised machine learning algorithm ren-
ders a quantitative analysis. Hence, I conclude that the ocLA cannot qualify as a sec-
ondary source of law.
4. Fourth, if the ocLA cannot be qualified as any known category or concept in
current legal practice, be it a fact, or law, or a comment, I claim that by elimination the
only remaining logical conclusion is that the ocLA ought to be considered as a sui gen-
eris concept in dispute resolution, a concept of its own kind.
With regard to the terminology, I propose to keep using the term ‘outcome of Legal
Analytics’ or ‘ocLA’, at least for the time being. It is preferable to the expression ‘syn-
thetic facts’. Although, the word ‘synthetic’ might be useful to refer to the aggregation
of metadata and data, encompassing facts and law, limiting the term to ‘facts’ renders
it inaccurate, since Legal Analytics take as an input data that not only come from the
facts of court cases.39 The same argument is valid for the term ‘derived facts’. ‘Syn-
thetic precedent’ and ‘synthetic case law’ imply normative value, which it hasn’t as
such. Moreover, ‘synthetic legal information’ is unnecessarily vague. Finally, ‘quanti-
tative legal argument’ probably comes closest to what the outcome could be in dispute
resolution. However, it is too early to fix the concept on a narrower term than ocLA.
4 Implications of the qualification of the ocLA as a concept of
its own kind for the use in court
Introducing the ocLA as a sui generis concept in court proceedings has several ad-
vantages. The first important advantage that it allows for conceptual clarity; the parties
and the court can consider the ocLA at its own level of abstraction, distinct from the
other parts of a dispute and its resolution. It furthermore allows to reason and to argue
with this new kind of information. The sui generis concept encourages the parties and
38
Patrick Allo ‘Mathematical values and the epistemology of data practices’, Emre Bayamlioglu
et al. (eds.) Being profiled: cogitas ergo sum (AUP, 2018) 20, 23.
39
See the discussion in G. Vanderstichele (n3) chapter 1, section 2.
9
the court to explicitly ask specific questions concerning the different aspects of the in-
put, the system with its different aspects of the software cycle, and the output. The
parties can develop separate arguments e.g. with regard to the number of cases that
were used, the kind of data and metadata, the cleaning of the data, the requirements
submitted to the software developers, the chosen model with its design decisions40, the
proxies that were used, the outcome, the inferences made, etc. A good documentation
of all elements of the NLP and ML algorithm and its outcome will facilitate and en-
hance the discussion in court. Parties and judges probably will also discuss related is-
sues such as problems of privacy, reasonable inferences, datafication, opacity, intellec-
tual property rights v. transparency, ‘gaming’ the system, etc.41 Third, treating the ocLA
as a concept of its own kind in the court procedure helps preserving the rights of due
process and fair trial in an information society.42 One of the dangers of the argument
that ‘courts decide x in 79% of this kind of case/legal issue’ is indeed that parties and
judges fall in the trap of automation bias – the propensity to uncritically favour auto-
mated decision-making systems, only because it is technology – as well as the
McNamara fallacy – decision-making based solely on quantitative observations, while
ignoring important observations of another kind43. By considering the ocLA at its own
level of abstraction, these traps, which would violate the rights to due process and fair
trial, can be averted. The entire algorithm can receive the required adversarial discus-
sion in a court case, before the judge rules either to admit the ocLA in the proceedings
or to declare it inadmissible, and in the former case to decide what weight to accord it
in the proceedings. In addition, it permits to give specific reasons for a judicial decision,
which is essential in common law,44 as well as civil law45 systems. Moreover, the sui
generis concept offers a method to safeguard human oriented values in adjudication,
and in society in general, as opposed to a reductionist approach of conflict, as it is the
human judge who has the proverbial last word. The judge adds context by studying the
case and analysing it for facts or legal questions a programmer could not have foreseen,
compares their analysis with the ocLA, and corrects where necessary. In general, it is
an element in preserving the rule of law when Legal Analytics are used in adjudica-
tion.46
40
See e.g. Remi Wieten, Floris Bex, Henry Prakken, Silja Renooij, ‘Supporting discussions
about forensic Bayesian networks using argumentation’,
accessed 3 November 2019.
41
It is beyond the scope of this contribution to discuss these issues; see G. Vanderstichele (n3),
section 3.5.
42
It is beyond the scope of the current contribution to elaborate on the implications for these
rights.
43
See e.g. Michael H Basler, ‘The utility of the McNamara fallacy’, (2009)
BMJ 339 accessed 3 June 2019.
44
See supra n20, n22, n23.
45
See supra n21.
46
It is beyond the scope of this contribution to further discuss this.
10
From the above, it becomes clear that the ocLA is an additional factor for judges to
consider in the decision-making process.47 Courts presented with an ocLA will interpret
facts and law as they do today. They will furthermore hear the parties’ thorough analy-
sis of the algorithm they invoke. Once the question about a sufficient correctness of the
algorithm48 is established, the court will confront the outcome of the concrete inter-
preted facts and law with the ocLA and it might reinterpret its preliminary finding on
the basis of the new information. A subsumption of principles of computer program-
ming, statistics, law and facts will take place. At the same time, law is put into the
ocLA.
Common law courts might establish a precedent that involves an algorithm. In such
a way the algorithm does not have normative value in itself but acquires at least indi-
rectly normative value as it is interpreted in a given set of facts and law. The ocLA will
have to be used in a specific context in order to become a precedent, giving the ocLA
limited precedential value as, and only as, tied to the facts of the case. This would add
context to the data. It would interpret the data and reconnect them to society. Addition-
ally, this modus operandi would preserve the fairness, consistency and predictability
precedent provides.49
In civil law systems, as the law is created by Parliament, a statute is required for an
algorithm to acquire the force of law. Alternatively, the legislator could establish a reg-
ulatory body with the power to grant licenses for tested and approved algorithms.50
However, I claim that courts should not necessarily declare the ocLA inadmissible as
long as there is no statutory intervention of a kind. The applicable code of civil proce-
dure must be applied. Where there is sufficient possibility for interpretation, the ocLA
as an argument of its own kind could be admitted in the procedure and subjected to
discussion by the parties.
It ought to be clear that I have proposed here a method for judges to engage critically
with the ocLA when submitted in court, as a portal to a further discussion about the
content and the methodology of the algorithm. However, a comprehensive discussion
of the relevant questions and criteria for assessment of the ocLA in court is outside the
scope of this contribution and will be a research project in itself.
Therefore, I offer in this article only the following general remarks with regard to
the content of a further argumentation and reasoning in court. It is to be foreseen that
using only ocLA with case law as an input will have a rather limited use. First, the
quantity and the quality of the data used as an input for the natural processing and ma-
chine learning algorithm is crucial. This poses a problem to the extent that access to
47
Matt O’Brien and Dake Kang, ‘AI in the court: when algorithms rule on jail time’ (31 January
2018) accessed 27 May 2019.
48
The definition of correctness and the criteria for it are outside the scope of this paper.
49
Frederick Schauer, ‘Precedent’ (1987) 39 Stanford Law Review 571, 595–600.
50
Richard Susskind, The future of law. Facing the challenges of information technology (OUP,
1998) 94xlii. See also Sandra Wachter and Brent Mittelstadt, ‘The right to reasonable infer-
ences: re-thinking data protection law in the era of big data and AI’ 2019 (2) Columbia Law
Review 494.
11
court data is limited in many countries, and access to the submitted evidence is even
more restricted. More specifically, whenever the algorithm is made with court cases as
an input, and not with the entire case file, its input is the facts as interpreted by the
courts and not the full facts as presented to the courts. It is to be taken into consideration
that there might be a difference between the ‘raw’ facts of a case and the withheld facts
in the court’s decision.51 Moreover, if ‘wrong’ court decisions, i.e decisions that do not
uphold a rule of law are used as an input, the outcome will be legally wrong, even when
the prediction is correct. Hence, it is valid to point to the limited value of the accuracy
measure of an algorithm. In a legal context this measure can be misleading. On its own,
it does not allow to conclude that the outcome will predict in a legally correct way the
outcome of a new individual case. Additionally, it has to be repeated that NLP and ML
algorithms render patterns and probabilities. They do not give reasons, whereas reasons
are essential in adjudication in the common law52 as well as in the civil law53 traditions.
Also, NLP and ML algorithms with case law as an input do not seem to be the appro-
priate tool to address legal cases with complex and multiple legal issues. This approach
to Legal Analytics could be used for a system to assist judges in processing cases where
the legal question is relatively straight forward, and the variables are restricted to a
limited number of rules and facts involved, and where the intervention of a judge is
required as a safeguard for fundamental rights such as due process. For instance, it
could be used to select ‘outliers’ for (faster) treatment on appeal.54 Finally, the quality
of the program, the training and the testing of the data also determine the quality of the
NLP and ML algorithms.
5 Conclusion – future work
I discussed above a method for parties and judges to examine in a court case the out-
come of an NLP and ML learning algorithm with case law as an input. I proposed to
discuss the outcome of a machine learning algorithm with case law as an input at its
own level of abstraction in court cases. In adjudication, it is a sui generis concept, pre-
liminary called ocLA, to be distinguished from facts, law and secondary sources of law.
By considering it as a concept of its own kind, parties and judges can reason with this
new kind of information.
51
See G. Vanderstichele (n3), section 2.1.
52
See e.g. Joseph Raz, From normativity to responsibility (OUP 2011) 85; Joseph Raz, Practical
reason and norms (2nd ed., OUP 1999) 142-145, 155 and 170-171.
53
See e.g. Article 6.1 European Convention for Human Rights; European Court of Human
Rights (ECtHR), Guide on Article 6 of the European Convention of Human Rights. Civil rights
(limb) 71-72 last accessed 11
February 2020.
54
See Ryan Copus, Ryan W. Copus, Machine Learning and the Reliability of Adjudication
(2017) 1, 33
ac-
cessed 3 March 2019.
12
This method is taking into account that courts operate today in a mixture of the so-
called printed press paradigm, where the affordances of the printed press determine the
rule of law and its application in adjudication, and the digital paradigm, whilst legal
analytics are evolving towards a fully digital paradigm. For instance, several scholars
are researching systems that imitate the human back and forth argumentation in court.55
Also, researchers create algorithms that account for legal values in dispute resolution.56
The latter allows for a so-called empirical law making, as an algorithm can be ablated
to simulate the effect of values on the outcome. As courts will further develop their role
into that of an interface between legal analytics and humans in a fully digital paradigm,
I claim that the proposed method probably will be less used to the extent legal analytics
are accepted in practice but will remain valid for judicial review of algorithms. Future
research will examine what the benchmarks are for assessing the outcome of Legal
Analytics, in light of the rule of law.
55
See e.g. Katie Atkinson, Pietro Baroni, Massimiliano Giacomin, Anthony Hunter, Henry
Prakken, Chris Reed, Guillermo Simari, Matthias Thimm, and Serena Villata: ‘Towards arti-
ficial argumentation’, (2017) AI Magazine. Vol. 38(3), pp. 25-36; Henry Prakken, ‘Logical
models of legal argumentation, In M. Knauff & W. Spohn (eds.): The Handbook of Rational-
ity. Cambridge, MA: MIT Press, 2020,
.
56
Kevin D. Ashley, ‘Accounting for Legal Values’, forthcoming; see also Katie Atkin-son and
Trevor Bench-Capon, ‘Taking account of the actions of others in value-based reasoning’,
(January 2018) Vol. 254 Artificial Intelligence January 2018, 1-20, https://www.sciencedi-
rect.com/science/article/abs/pii/S0004370217301078; Mat-thias Grabmair, ‘Predicting trade
secret case outcomes using argument schemes and learned quantitative value effect tradeoffs’
DOI: 10.1145/3086512.3086521.