This research explores evidence capture for accountable AI systems. First, di erent scopes of AI accountability are set out by extending existing classi cation. Based on these scopes, two important and fundamental questions in evidence capture are answered: what types of evidence need to be captured and how we can capture them to facilitate better AI accountability. We hope that this research can provide guidance on building better accountable AI systems with e ective evidence capture and initiate further research along this line.
Accountability of AI systems has been increasingly studied in recent years, and
it has attracted much attention from not only academia [
Realising accountable AI Systems entails knowing who the people were
behind the key decisions made throughout the AI system's life cycle, e.g., how the
system was designed and built, how it is being used and maintained, and how
the laws, regulations, and standards were followed [
A crucial step to achieve this is to capture evidence e ectively. To start with, two questions need to be answered: what types of evidence need to be captured and how they can be captured. Answering these two fundamental questions will help implement functional evidence capture components for AI systems, ? Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
This research is supported by the RAInS project (https://rainsproject.org/) funded by EPSRC (EP/R033846/1). We thank all other members of the project for their inspiration and suggestions for this research. thus making AI systems accountable. It will also provide guidance on how we can perform accountability-related investigations (e.g., incident investigation for automated vehicles and bias investigation for AI-assisted recruitment) through e ective evidence gathering.
In this research, we will extensively discuss the above two questions. We do not intend to provide speci c solutions or frameworks for evidence capture; instead, we aim to provide guidelines and suggestions, and we hope this could inspire further research on this topic.
The rest of the paper is organised as follows: rst, di erent scopes of AI accountability are set out in Section 2. Then based on these scopes, in Section 3 a series of \what" questions are answered. This is followed by Section 4, in which the \how" question is discussed. Finally, Section 5 concludes the paper. 2
AI accountability may have di erent scopes and meanings in various scenarios.
Following the brief discussion in [
In this scope, accountability is considered as a feature or component of an AI
system per se. An AI system can o er related functions to make itself accountable.
These functions include explainability, attributability, auditability, and
provenance. Similar to accountability, each of these functions may have di erent scopes
and meanings in various scenarios. Explainability entails enabling the system to
justify its outputs (e.g., decisions and predictions). This can be automated by
XAI (eXplainable AI) tools, whether model agnostic [
Within this scope, accountability aims to hold the persons or organisations
accountable. This is because the AI systems are made by and for humans (we argue
that for the AI systems automatically produced by AutoAI/AutoML [
In the broadest scope, an AI system is viewed as a complex system, for example,
a socio-technical system [
It is noted that, further to the classi cation in [
To be accountable for everything means to be accountable for nothing. Correspondingly, capturing everything is neither feasible nor necessary. To decide what we will actually capture (the action), we need to answer the following three questions: what is the scope of capture, what is the capability of capture, and what is the obligation to capture.
First, the scope of capture is determined by the scope of accountability in consideration (as set out in Section 2); we will discuss this in Section 3.1. Second, the capability of capture is subject to both AI system limitations and external constraints; we will address this in Section 3.2. Third, the obligation to capture is often determined by the requirements of speci c domains, regulations, laws, and standards; we will cover this in Section 3.3. Lastly, what we will actually capture is the ultimate question, which is a ected by the answers of the rst three questions. We will discuss this nal question in Section 3.4.
In what follows (Sections 3.1 3.4), we will not produce an exhaustive list of the types of potential evidence in each subsection (as such an exhaustive list is impossible to generate), but rather, we provide the most essential and representative types of evidence, some of which are accompanied by concrete examples. 3.1
Considering the three distinct scopes of AI accountability set out in Section 2, di erent sets of evidence for capture can be accordingly considered for each scope. We will now discuss them in detail.
Technical Aspect The rst scope of AI accountability is concerned with the technical aspect. First, it is essential to record information about the training and evaluation data (e.g., sources, pre-processing processes, and quality analysis) and about the models, which includes the training paradigm and evaluation procedures. Furthermore, explanations of AI predictions and inference processes, fairness, uncertainty, robustness analysis (and even formal veri cation) for the AI system often need to be recorded for auditing and potential investigations. In many cases, the above information has not been generated or it is not feasible to generate such information beforehand; therefore, whenever possible, the approaches to generating such information should be investigated, initially congured, and documented for post-hoc accountability analysis. For instance, appropriate XAI and fairness analysis tools for the AI system may be prepared and the instructions for using these tools are recorded.
Social and Human Aspect The second scope of AI accountability focuses on human and social activities. Human activities related to the AI system need to be captured in order to hold them accountable. This includes human decisionmaking processes and human-human interactions, either directly or through AI system components, during the life cycle of an AI system. For example, the following information may be captured as evidence: the stakeholders' meetings and discussions on the AI system to be developed, AI designers' decision making processes on using particular AI models, the interactions between AI designers and developers during the implementation stage of the AI system, and how users operate an AI system deployed in the wild.
Complex System Aspect As for the third and broadest scope of AI accountability, we need to capture not only the information regarding the rst two scopes, but also the interactions and information ows of di erent elements of the complex AI system, including the interactions between people, the AI components, the lower-level software and hardware supporting infrastructure, and the environment which the AI system is operated in and interacts with. 3.2
As mentioned at the beginning of this section, what can be captured is subject to the AI system limitations and other external constraints. As in Section 3.1, we will again consider the di erent scopes of accountability set out in Section 2 to discuss this in detail.
Technical Aspect The capture ability is determined by the functionalities and
limitations of the AI system. The limitations of the AI models being used may
a ect such ability. For instance, explaining the inference and reasoning processes
of black-box models is generally more challenging compared to white-box models.
The robustness of some AI models, e.g. some sophisticated deep neural networks,
may be hard to analyse against adversarial attacks. For many cutting-edge AI
models, their formal veri cation may be very challenging or even not possible [
For a speci c AI application, we must consider related laws, regulations, or
standards to capture the required types of information or capture them as
much as possible. One example is one of the UK national standards for
automated vehicles, the BSI standard PAS 1882 [
Having covered the scope, capability, and obligation of evidence capture, we can now discuss what we will actually capture.
It is obvious that deciding what evidence we actually capture should consider the above three factors simultaneously. For a particular application, from a pragmatic perspective, we may start from the obligations, and then examine/improve the capture capability within the scope of capture. By doing this we will get a narrower set of evidence to be captured.
For the above re ned set, we propose the following three principles to further re ne it: rst, evidence capture should not signi cantly a ect the system performance (e.g., accuracy, e ciency, and reliability) or take too much resource (e.g., computational time, storage, and human labour), and we call this the performance principle. Second, evidence capture should be less invasive to the AI system and its environment (e.g., requiring no signi cant change to the AI system or environment), and we call this the friendly principle. Third, considering the above two principles, capturing more is better than capturing less, and we call this the redundancy principle.
Finally, evidence capturing needs to consider the nature of the application domain and the requirements of the particular accountability investigation. Capturing potential evidence for an automated vehicle is more likely to include hardware and environmental data, such as the vehicle's engine information, road and weather conditions; but capturing potential evidence for an AI recruitment system may focus more on the technical aspect, such as bias analysis and decision/prediction explanation. 4
In this section, we discuss the methods of evidence capture. First, the same three principles in Section 3.4 need to be followed when designing capture methods. Second, evidence capture should be carried out throughout the AI life cycle, including requirement analysis, design, implementation, deployment, operation, and maintenance. Related components and work ows which enable evidence capture should be carefully designed and implemented for each stage of the AI system life cycle. Third, based on the degree of automation, capturing methods can be classi ed into three categories: automatic, semi-automatic, and manual. We discuss them in detail below.
Automatic capture does not involve human intervention. Google's TFX
framework [
Semi-automatic capture requires some degree of human input; for instance,
the Model Card Toolkit (MCT) [
Finally, manual capture is the last resort if the rst two approaches are not feasible; for example, gathering stakeholders' meeting minutes and extracting related information from these documents are likely to be done manually. 5
We have extensively discussed two fundamental questions on evidence capture for accountable AI systems: what to capture and how to capture. We hope the discussion can guide more e ective evidence capture and thus contribute to the development of better accountable AI systems.