This paper touches upon the intertwining of AI technology and criminal justice systems and assesses especially the issue of using AI as an evidence-generating mechanism in criminal trials. The paper revolves, in particular, around three focal points. Firstly, it sets the context for the following analysis and gives a short definition of AI. Secondly, it examines some thorny parameters of the evidentiary proceedings and focuses on the most important AI weaknesses that could jeopardise the smooth incorporation of AI in the criminal justice systems. Thirdly, it presents the ways in which AI could affect basic procedural rights of the defendant and concludes with some safety requirements and suggestions that could facilitate the transition to an AI-criminal-justice-era.
It is trite to say that Artificial Intelligence (AI) will reshape and is indeed already reshaping many aspects of our reality, and yet it is true. The digital transformation of the global society due to AI does not leave the justice systems around the world unaffected, bringing at this very moment the first challenges for crime control and criminal justice to the surface.
As crime becomes more and more complex, sophisticated and opaque, it is extremely difficult for the law enforcement agencies to detect certain criminal behaviours and find their operational patterns. This fact has a negative impact on the social credibility of traditional justice systems.1 In this context, the use of AI in the criminal justice system may prove to be of strategic importance and a game changer for prevention, investigation, fact finding and procedural economy. This in turn calls for a deeper understanding of AI’s functions and operation processes within the scope of criminal justice. Given that software programmes of predictive policing,2 predictive analytics and face recognition are already being used in police departments in the U.S.,3 Europe4 and China,5 and criminal justice systems have begun to use machine learning to assist in investigations for fraud and other white-collar crimes, it is imperative that legal scholars begin to dig in this uncharted and unlegislated territory. This paper focuses on the use of AI as evidence in the context of the traditional criminal trial. After briefly dealing with the definition of AI, the paper outlines the use of AI as an evidence-generating mechanism, examines the procedural rights of the defendant in view of the problems inherent to the AI technology and concludes the proposed solutions and guidelines. 2.
In the field of criminal law, there is a long-standing, close bond between criminal justice and technology. Over the past 150 years, criminal courts have deployed the so called ‘machine evidence’, in order to form and support their verdict, and the ‘silent testimony of instruments’ has supplemented the testimony of humans.6 One could just think of toxicology, ballistics, anthropometry, fingerprints, uhlenhuth test, maturation, forensic graphology and DNA-test.
The striking difference, though, between the AI—in the form of face, voice or video recognition, machine learning for the detection of fraud or other crimes, etc.—and the previous forensic methods of past decades is that machines, then, whenever put into use, operated according to rules that humans painstakingly programmed by hand.7 By contrast, an offshoot of AI, that is, machine learning, refers to a programme’s ability to extract patterns from raw data.8 The machine has now the ability to keep improving its performance without humans having to explain exactly how to accomplish a task.9 Indeed, many times the programmer herself cannot account for how the machine came to a particular result, even if the result is accurate.10 For the purposes of this paper, we adopt the High-Level Expert Group on Artificial Intelligence (AI HLEG) general definition of AI, specifically that of its AI Ethics Guidelines: ‘Artificial intelligence (AI) systems are software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge, or processing the information, derived from this data and deciding the best action(s) to take to achieve the given goal. AI systems can either use symbolic rules or learn a numeric model, and they can also adapt their behaviour by analysing how the environment is affected by their previous actions. As a scientific discipline, AI includes several approaches and techniques, such as machine learning (of which deep learning and reinforcement learning are specific examples), machine reasoning (which includes planning, scheduling, knowledge representation and reasoning, search, and optimization), and robotics (which includes control, perception, sensors and actuators, as well as the integration of all other techniques into cyber-physical systems).’11 Core to the concept of AI, as stated above, is the notion of an agent capable of taking relatively autonomous decisions, depending on its perception and cognition of its environment. The emphasis on agency implies that we are not dealing with a rigid execution of rules but with systems capable of learning how to improve their performance on the basis of feedback.12 When it comes down to the use of such programmes as evidence in a criminal trial, and 7 V Fomin, ‘The Shift from Traditional Computing Systems to Artificial intelligence and the Implications for Bias’ in JS Gordon (ed), Smart Technologies and Fundamental Rights (Brill | Rodopi, 2020, to be published). 8 I Goodfellow, Y Bengio and A Courville, Deep Learning, 9th edn (Cambridge, MA, The MIT Press, 2016) 2–3; H Surden, ‘Machine Learning and Law’ (2014) 89 Washington Law Review 87‒115, 88: ‘Machine learning systems are computer algorithms that have the ability to learn or improve in performance over time on some task’. 9 On AI, in general, see the reference book of S Russell and P Norvig, Artificial Intelligence: A Modern Approach, 4th edn (New Jersey, Pearson, 2020). 10 A Holzinger, C Biemann, CS Pattichis and DB Kell, ‘What Do We Need to Build Explainable AI Systems for the Medical Domain?’: ‘Often the best-performing methods are the least transparent’, 2, available at: https://arxiv.org/pdf/1712.09923.pdf. 11 Available at: https://ec.europa.eu/futurium/en/ai-alliance-consultation, 36. potentially as basis for the subsequent acquittal or conviction of the defendant, the focus must be, thus, on the compatibility of this particular characteristic of AI with the traditional purposes and guarantees of the criminal trial and the evidentiary process. 3.
Adjudicative fact-finding as such is bound to be conducted in conditions of uncertainty. The evidentiary process of the trial rests upon probabilities, not certainties, and hence involves a risk of error. That is why it is impossible to eliminate erroneous convictions and acquittals.13 The courts may aspire to ascertain the truth, but at the end of the day they must come to a decision.14 For this reason, the criminal courts often turn to science, in order to help them reach a verdict that is, as much as possible, objective and facts-based. As already mentioned, the use of machines, scientific evidence and expert witnesses in the evidentiary proceedings is not a novelty for criminal justice. Equally old is also the fact that scientific evidence might get it wrong sometimes. By their conduct, courts have expressed, over the years, a tolerance for some level of both ignorance and risk in machine evidence: ignorance in how these processes work, and risk that they might not ‘get it right’ every time.15 The purpose of the finality of trial trumps the purpose of finding of truth, if the latter ever was possible. The beyond-all-reasonable-doubt standard itself recognizes this inevitability of sporadically convicting innocent people.16 Further, evidence is not necessarily produced during trial. It is instead the outcome of the process of appraising what is produced at trial by the fact finder, who in doing so invokes a large storehouse of ‘evidence’ that is summed up in her beliefs.17 This makes it almost impossible for the fact finder to avoid heuristics and cognitive bias. Increasing reliance on machines in litigation could consequently, for some scholars, help minimize the ‘whim and caprice’ of the bench or jury. All in all, no one could refuse that the generation of unpredictable, idiosyncratic decisions is the antithesis to the rule of law.18 Therefore, the AI could serve as an auxiliary mechanism, assisting the court in the fact-finding process, by reducing judicial arbitrariness, systematizing the proof process and improving trial efficiency. Some proponents of the deployment of AI in the judicial system even invoke phenomena of judicial corruption, to advocate in favour of the use of the AI in the judicial field.19 This could be the case, though, if 12 M Hildebrandt, ‘Criminal Law and Technology in a Data-driven Society’ in MD Dubber and T Hörnle, The Oxford Handbook of Criminal Law (Oxford, Oxford University Press, 2014) 175‒96, 188. 13 A Stein, Foundations of Evidence (Oxford, Oxford University Press, 2005) 2. 14 A Keane, P McKeown, The Modern Law of Evidence, 12th edn (Oxford, Oxford University Press, 2018) 2‒3. On the search for the truth as purpose of the criminal trial, see E Billis, Die Rolle des Richters im Adversatorischen und im Inquisitorischen Beweisverfahren (Berlin, Ducker & Humblot, 2015) 93‒120. 15 PW Nutter,’ Machine Learning Evidence: Admissibility and Weight’ (2019) 21(3) Journal of Constitutional Law 919‒58, 925. 16 ibid 173. 17 RJ Allen, ‘Artificial Intelligence and the Evidentiary Process: The Challenges of Formalism and Computation’ (2001) 9 Artificial Intelligence and Law 99‒114, 103. 18 ibid 101. 19 Y Cui, Artificial Intelligence and Judicial Modernization (Singapore, Springer, 2020) 22. the AI deployed in the criminal justice field, could promise a high percentage of objectivity and accuracy. Research so far has shown, however, that AI is susceptible to biases and, as a result, its outcome accuracy cannot be fully trusted.20 Nevertheless, since AI applications are already being used in many criminal jurisdictions around the world, it is imperative to properly examine their weak spots.
As artificial intelligence does not equal artificial perfection, the weaknesses of AI as an evidence-generating mechanism must be put under a magnifying glass, in order to figure out satisfactory safety requirements conditioning its use in the realm of the criminal justice. The most problematic characteristics of AI could be summed up around the following terms: 1. inexplicability, 2. discrimination and bias and 3. lack of accountability.
AI is revolutionary in its applications and capabilities, though, with respect to its potential uses in criminal justice, it is functionally similar to traditional software: data go in and conclusions come out. In between, there is a ‘black box’ of calculations that not only is occasionally inaccessible to the experts themselves but also few in the courtroom would understand.21 Here lies the danger of the AI being improperly afforded a presumption of reliability, objectivity and certainty, due to its mechanical appearance and apparently simple output.22 In order for the bench or jury to make an informed decision on the guilt of the defendant, light must somehow be shed on this black box. Moreover, given that the AI output is often inexplicable, the question of how the defendant will be able to defend herself and contest the evidence produced by it inevitably arises.
At the same time, decisions taken by algorithms could result from data that is incomplete and therefore not reliable: data may be tampered with by cyber-attackers, biased or simply mistaken. Applying the technology as it develops without due consideration would, therefore, lead to problematic outcomes as well as reluctance by citizens to accept its use by the courts, since the risk of malfunction always remains a distinct possibility. Thus, one of the toughest challenges for a successful incorporation of AI in criminal justice is the elimination of all kinds of biases that AI is susceptible to. Indeed, such biases may subsequently lead to poor and unjust judicial decision making, when factored in by the bench or jury. In truth, all these processes have hidden subjectivities and errors that often go unrecognized and unchecked, thus potentially ‘facilitat[ing] the masking of illegitimate or illegal discrimination behind layers upon layers of mirrors and proxies.’23
Furthermore, when data are first gathered or generated, basic human error in collection or interpretation is common.24 Human errors could occur in the training phase of the data or even later in the further development of the programme. Nevertheless, in order to establish accountability, one needs to demonstrate the person behind the programme, who did something wrong. In machine learning systems, where computer scientists are often unable to determine how or why a machine learning system has made a particular decision, this is very difficult to achieve.25 Furthermore, one of the typical supportive arguments from the side of AI experts and AI companies is that AI systems and especially the machine learning ones evolve in unforeseen ways, due to their autonomous and self-learning nature. As a result, no programmer could be held liable for their evolution. 5.
In view of the characteristics of AI outlined above, legal scholars need to come up with new, effective safeguards in criminal procedure, or reinterpret those already existing. Since the use of AI will be a state privilege, at least in this phase of the digital judicial transformation, the defendants need to be equipped with the procedural rights that will preserve the equality of arms between them and the state, and the fairness of the trial. The defendant must be able, in this new criminal procedural framework, to defend herself against the all-mighty AI and contest the evidence produced by the latter.
Legal scholars must also consider changes in the law of evidence. Rules regarding the admissibility of AI generated evidence and methods to determine the reliability of its outcomes, exclusionary rules, and rules on risk-allocation are some of the problems which lie at the heart of this issue. The principle of judicial discretion must, likewise, find its place in this new environment, since the danger of over-evaluating the importance of AI generated evidence could lead to total reliance upon science and to ‘abdication of responsibility by law’.26 Human-computer interaction research on the biases involved in all algorithmic decision-making systems has shown that it is extremely difficult for a human decision-maker to refute a ‘recommendation’ made by a high-tech tool.27 Furthermore, since we already witness a ‘dissolution of the procedural infrastructure within the criminal justice system’28 because of the profiling, risk assessment and predictive analytics techniques, a new conceptualization of the fundamental procedural rights of the defendant is more than necessary. Procedural fairness is the ultimate prerequisite, if we want this new architectural scheme to work and gain social acceptance.
The presumption of innocence was traditionally connected with a temporal distance between the criminal charge and the conviction or the acquittal.29 The new AI environment challenges ‘the linear sense of time’.30 In other words, it challenges the delay inherent in procedural safeguards embodying protection against hasty judgments, as we are confronted with a series of real-time decisions taken by automated decision systems based on machine learning techniques.31 Data-driven surveillance challenges the very foundations of the presumption of innocence by suggesting precognition of criminal intent32 and thus ‘creating a de facto presumption of guilt’.33 Hence, some scholars go as far as advocating the construction of a ‘presumption of innocence by design’34 and the interjection of ‘explanation systems’ into AI solutions, since the inculpatory evidence must have some kind of discernible logic, explanation, ability to be examined or challenged. In the context of law enforcement and intelligence, default settings of the computational technologies should prevent the reversal of the presumption of innocence by the automation of suspicion,35 especially where data-mining is used to flag behaviours.
One of the oldest rights that belongs to the core of the defendant’s procedural arsenal in the Western legal systems is the right to confrontation.36 The accused enjoys the right to be confronted with the witnesses against him, cross examine them and contest the incriminating evidence. Normally, the defendant would be given full access to the evidence against him, in order for him to exercise this very right. Since the inner workings of these tools are trade secrets of the companies that developed them, one wonders how the defendant would be able to effectively defend herself in lack of access to the very algorithmic assessment tool, that brought her to the stand.
With the advance of big data, privacy is the right that has suffered the most. Individuals are investigated, judged and sometimes punished ‘en masse’ and ‘at a distance’,37 blurring the clear distinctions between citizen, suspect, defendant, convict and 29 Hildebrandt (n 12) 181. 30 Hildebrandt (n 12) 182. 31 Hildebrandt (n 12) 182. 32 Hildebrandt (n 12) 194−95. 33 Hildebrandt (n 12) 184. 34 Hildebrandt (n 12) 174, 195. 35 Hildebrandt (n 12) 183. 36 For the U.S., see P Marcus, DK Duncan, T Miller and J Moreno, The Rights of the Accused under the Sixth Amendment, 2nd edn (Chicago, American Bar Association, 2016). For Europe, see S Maffei, The Right to Confrontation in Europe (Groningen, Europa Law Publishing, 2012). 37 Marks, Bowling and Keenan (n 28) 708. 38 Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal acquitted. Even at the level of the European Union where the General Data Protection Regulation38 and the accompanying Law Enforcement Directive39 establish strict data protection standards in the area of criminal offences and penalties, fully automated decision-making remains possible, albeit rarely. The Member States still have the possibility of providing for a decision based solely on automated processing, which produces an adverse legal effect concerning the data subject. Sole prerequisite is the authorisation by Union or Member State law, as long as it provides appropriate safeguards for the rights and freedoms of the data subject.
The above-mentioned impact of AI on the procedural rights disturbs the fair balance between the parties. Procedural equality of arms is designed to ‘treat the accused as a thinking and feeling human being worthy of respect, who is entitled to be given the opportunity to play an active part in procedures with a direct and possibly catastrophic impact on their life, rather than as object of state control to be manipulated for the greater good’.40 Therefore, it is imperative for the defendant to be afforded a reasonable opportunity to present his case and take actively part in the criminal trial including the evidentiary proceedings.
AI has entered the premises of criminal justice systems in the aspiration to improve the procedural justice and economy, their effectiveness and efficiency. In order to fulfil those aspirations, we need greater social acceptance of AI. Trustworthy AI has three components according to the High-Level Expert Group on Artificial Intelligence (AI HLEG): (1) it should be lawful, ensuring compliance with all applicable laws and regulations, (2) it should be ethical, ensuring adherence to ethical principles and values and (3) it should be robust, both from a technical and social perspective since to ensure that, even with good intentions, AI systems do not cause any unintentional harm. The current state of the art does not provide for systems to self-report their decisions, but there is a widely held view in the relevant scientific community that regulators will force developers to interject “explanation systems” into their AI solutions when they are deployed in environments data and on the free movement of such data, and repealing Directive 95/46/EC, available at: https://gdpr.eu/tag/gdpr/. 39 Directive (EU) 2016/680 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data by competent authorities for the purposes of the prevention, investigation, detection or prosecution of criminal offences or the execution of criminal penalties, and on the free movement of such data, and repealing Council Framework Decision 2008/977/JHA, Article 11: 1. Member States shall provide for a decision based solely on automated processing, including profiling, which produces an adverse legal effect concerning the data subject or significantly affects him or her, to be prohibited unless authorised by Union or Member State law to which the controller is subject and which provides appropriate safeguards for the rights and freedoms of the data subject, at least the right to obtain human intervention on the part of the controller. Available at: https://eur-lex.europa.eu/legalcontent/EN/TXT/?uri=celex:32016L0680. 40 P Roberts and A Zuckerman, Criminal Evidence, 2nd edn. (Oxford, Oxford University Press, 2010) 21. where their outputs (or decisions) are likely to have a significant regulatory or human impact.41 According to the Ethics Guidelines for Trustworthy Artificial Intelligence prepared by the same expert group (AI HLEG), the key requirements that any AI system must fulfil in order to be accepted are: a. the human agency and oversight, b. technical robustness and safety, c. privacy and data governance, d. transparency, e. diversity, non-discrimination and fairness, f. societal and environmental wellbeing and g. accountability.42 Overall, the trust in AI control mechanisms poses many challenging regulatory questions, given the fact that trust must not be an elusive and muddled idea, but a reflection and a result of crystal-clear regulation. In order for these requirements to gain true meaning instead of remaining empty vessels, legal scholars must ally with AI experts, in order to come up with solutions that comply with the actual practice of the law as fairly and as efficiently as possible. 7. CONCLUSION This paper served the purpose of highlighting the interplay between the criminal justice systems and the AI technology in connection with AI being employed as evidence tool, the sticking points of this risky venture, and some brief deliberations about possible solutions. The inducement that AI offers to criminal justice systems is big. The challenge will be, as Hildebrandt puts it, for the law to engage with AI ‘without either sacrificing or petrifying its identity’.43 It is imperative, therefore, for legal scholars to cross disciplinary boundaries and work together with AI experts, in order for them to demystify and understand in depth the workings of AI. Only then they can produce relevant and applicable laws that will effectively incorporate AI in our legal reality and justice systems and regulate its possible dangers.