Artificial Intelligence (AI) systems exert a growing influence on our society. As they become more ubiquitous, their potential negative impacts also become evident through various real-world incidents. Following such early incidents, academic and public discussion on AI ethics has highlighted the need for implementing ethics in AI system development. However, little currently exists in the way of frameworks for understanding the practical implementation of AI ethics. In this paper, we discuss a research framework for implementing AI ethics in industrial settings. The framework presents a starting point for empirical studies into AI ethics but is still being developed further based on its practical utilization.
Artificial Intelligence (AI) and Autonomous Systems (AS) have become increasingly prevalent in software development endeavors, changing the role of ethics in software development. One key difference between conventional software systems and AI systems is that the idea of active users in the context of AI systems can be questioned. More often than not, individuals are simply objects for AI systems that they either perform actions upon or use for data collection purposes. On the other hand, users of AI systems are usually organizations as opposed to individuals. This is problematic in terms of consent, not least because one may not even be aware of being used for data collection purposes by an AI.
To this end, existing studies have argued that developing AI/AS is a
multi-disciplinary endeavor rather than a simple software engineering one
However, despite the increasing activity in the area of AI ethics, there is currently a
gap between research and practice. Few empirical studies on the topic exist, and the
state of practice remains largely unknown. The IEEE Ethically Aligned Design
guidelines have suggested that they have not been widely adopted by practitioners.
Additionally, in a past study, we have presented preliminary results supporting the notion of a
gap in the area
To provide a starting point for bridging the gap between research and practice in terms of empirical research, we present a framework for AI ethics in practice. The framework is built around extant conceptual research in the area of AI ethics, intended to serve a framework for empirical studies into AI ethics. The framework has been utilized in practice to collect empirical data and based on this utilization we discuss the framework in this paper.
The rest of this paper is organized as follows. In section 2 we discuss the theoretical background of the study by going over existing research in the area. Then, in section 3, we present the research framework discussed in this paper. In section 4 we go over the results of an empirical study in which the framework was utilized. In section 5, we discuss the framework and its implications. Section 6 concludes the paper. 2
The academic discussion on AI ethics has thus far largely focused on defining the
area through central constructs and principles. Thus far, the focus has been on four main
principles for AI ethics: transparency
Various real-world incidents out on the field (e.g.,
Attempts to bring this on-going academic discussion out on the field have been
primarily made in the form of guidelines and principles lacking practices to implement
them
These guidelines, however, are unlikely to see large-scale industry adoption based
on what we already know about ethical guidelines in IT. In their study on the effects of
the ACM ethical guidelines
Academic literature has discussed various principles as a way to address ethics as a
part of the development of AI and AI-based systems. Currently, four constructs are
considered central ones in AI ethics: Transparency
To categorize the field of AI ethics three categories have been presented: (1) Ethics
by Design (integrating ethics into system behavior); (2) Ethics in Design (software
development methods etc. supporting implementation of ethics); and (3) Ethics for Design
(standards etc. that ensure the integrity of developers and users)
Out of the aforementioned four principles that have been proposed to form the basis of ethical development of AI systems, we consider accountability, responsibility, and transparency (the so-called ART principles (Dignum (2017)) a starting point for understanding the involvement of ethics in ICT projects. These three constructs form the basis of ethical AI and attempts to identify their possible relations, as well as relations of other constructs that may be involved in the process.
To make these principles tangible, a subset of constructs in the form of actions (Fig.
1 (1.1-3.5)), discussed in detail in subsection 3.1, was formed under each key concept.
These actions were outlined based on the IEEE guidelines for EAD
Transparency is a key ethical construct that is related to understanding AI systems.
Dignum (2017) discusses transparency as transparency of AI systems and specifically
algorithms and data used. Arguably, transparency is a pro-ethical circumstance that
makes it possible to implement AI ethics in the first place
In the research framework presented in this paper, we consider transparency not only
in relation to AI systems but also in relation to AI systems development. I.e., we also
consider it important that we understand what decisions were made, by whom, and why
during development. Different practices support this type of transparency (e.g. audits
and code documentation
For the system to be considered transparent (line 1.a), feature traceability (1.1) (EAD Principle 5) should be present, and the system should be predictable in its behavior (1.2) (EAD Principles 5 and 6). For development to be considered transparent (line 1.c), the decision-making strategies of the endeavor should be clear (1.4) (EAD Principles 5 and 6), and decisions should be traceable back to individual developers (1.3) (EAD Principles 1, 5, and 6). As a pro-ethical circumstance, transparency also produces the possibility to assess accountability and responsibility (line 1.b) in relation to both development and the system.
Accountability refers to determining who is accountable or liable for the decisions made by the AI. Dignum (2017) defines accountability to be the explanation and justification of one’s decisions and one’s actions to the relevant stakeholders. In the context of this research framework, accountability is used not only in the context of systems, but also in a more general sense. We consider, for example, how various accountability issues (legal, social) were taken into consideration during the development.
As mentioned earlier, transparency is the pro-ethical condition here that makes
accountability possible (denoted by line 1.b). We must understand how the system works
in order to establish accountability. Similarly, we should be able to determine why it
works that way by understanding what decisions made during development led to the
system working that way. We consider accountability in a broad sense, thus including
also legal and social concerns related to the system. Much like transparency,
accountability is also considered a key construct in AI ethics
In our research model, accountability is perceived through the concrete actions of the developers concerning the systems itself, 2.1 Preparing for anything unexpected: (actions that are taken to prevent or control unexpected situation) (EAD Principle 8), 2.2 Preparing for misuse/error scenarios (actions that are taken to prevent or control misuse/error scenarios) (EAD Principles 7 and 8), 2.3 Error handling (practices to deal with errors in software) (EAD Principles 4 and 7) and 2.4 data security (actions taken to ensure cyber security of system and secure handling of data) (EAD Principle 3).
Finally, Dignum (2017) considers responsibility a chain of responsibility that links
the actions of the system to all the decisions made by its stakeholders. We do not
consider this definition to be actionable and instead draw from the EAD guidelines
While accountability relates to the connection between one’s decisions and the stakeholders of the system, responsibility is more focused on the internal processes of the developers not necessarily directly related to any one action. In order to act responsibly, one needs to understand the meanings of their actions. Therefore, in the research framework responsibility is perceived through the actions of the developers concerning, 3.1 perception of responsibility (developers have a sense of responsibility and perception what is responsibility in software development) (EAD Principles 2, 4 and 6); 3.2 distribution of responsibility (who is seen responsible e.g. for any harm caused by the system) (EAD Principle 6); 3.3 encountered problems (how errors and error scenarios are tackled and who is responsible for tackling them) (EAD Principles 7 and 8); 3.4 feelings of concern (developers are concerned about issues related to their software); and 3.5 data sensitivity (developers attitude toward data privacy and data security) (EAD Principles 2 and 3). 3.2
The commitment net model of Abrahamsson (2002) was utilized to analyze the data
gathered using this research framework
From this commitment net model, we focused on concerns which were analyzed to
understand what ethical issues were of interest to the developers. Actions were then
studied to understand how these concerns were actually tackled, or whether they were
tackled at all. In commitment net model, actions are connected to concerns because
when actions are taken, they are always driven from concerns
The dynamic between actions and concerns was considered a tangible way to approach the topic of practices for implementing AI ethics. Actions were directly likened to (software development) practices in this context. On the other hand, concerns were considered to be of interest in understanding e.g. whether the developers perhaps wanted to implement ethics but were unable to do so.
In this fashion, existing theories can be used in conjunction with the framework to either make it more actionable for implementing ethics, or for helping analyze or gather data using the framework. 4
The framework was utilized successfully in a recent study
The research framework was utilized to carry out a multiple case study of three case companies. Each company was a software company developing AI solutions for the healthcare industry. More specifically, the case studies focused on one specific project inside each of the case companies. Case
Case Description A B C
Statistical tool for detecting marginali
zation Voice and NLP based tool for diagnos
tics NLP based tool for indoor navigation
Respondent[Reference]
Data analyst [R1] Consultant [R2]
Project [R3]
coordinator Developer [R4] Developer [R5] Project manager [R6] Developer [R7] Developer [R8]
Data from the cases were gathered using semi-structured interviews, for which the
strategy was prepared according to the guidelines of Galletta (2013). The research
framework, described in the preceding section, was utilized to construct the research
instrument with which the data was collected. The questions prepared for the
semistructured interviews focused on the components of the framework. The interviews
were recorded and the transcripts were analyzed for the empirical study. The transcripts
were analyzed using a grounded theory
The findings of the empirical study conducted using this framework were summarized into four Primary Empirical Conclusions (PECs). The PECs were communicated as follows: ● ● ● ●
PEC1 Responsibility of developers and development is under-discussed PEC2 Developers recognize transparency as a goal, but it is not formally pursued PEC3 Developers feel accountable for error handling on programming level and have the means to deal with it PEC4 While the developers speculate potential socioethical impacts of the resulting system, they do not have means to address them.
These results served to further underline the gap between research and practice in
the area. Whereas developers were to some extent aware of some of the goals of the AI
ethics principles, these were seldom formally pursued in any fashion.
Rather than discussing the implications of the empirical findings of the study
utilizing this framework (as was already done in
However, as this framework was constructed in late 2017, it is now two years old. Since its inception, much has happened in the field of AI ethics and AI in general. The discussion has progressed, and whereas in 2017 the ART model was the current topic of discussion and Fairness an emerging construct, now Fairness has also become a central construct in AI ethics discussion (e.g ACM Conference on Fairness, Accountability, and Transparency, https://fatconference.org/) , especially in the discourse of the United States and the Anglosphere.
Moreover, a recent EU report on Trustworthy AI systems
The discussion on principles in the field continues to be active. Morley et al. (2019) recently proposed a new set of constructs intended to summarize the discussion thus far. Only time will tell whether this novel set of constructs becomes as widely used as the existing constructs such as transparency.
Yet, we maintain that it is pivotal that attempts such as this are made to bring
empiricism into this otherwise highly theoretical discussion. Although the field is still
evolving, the industry is not waiting for the discussion to finish. AI systems are developed
with or without the involvement of AI ethics. To this end, even if the academia does
not act, governments and other national and supranational organizations are drafting
their own guidelines
The framework nonetheless does require further development. Aside from including
constructs such as fairness, we argue that it should be essentialized. Essentializing
refers to a process discussed by Jacobson, Lawson, Ng, McMahon & Goedicke (2017) in
the context of the Essence Theory of Software Engineering
In the context of this framework, we see essentialization as one way to make it more understandable for industry experts. Essentializing a practice, or method, or a framework has three steps, according to Jacobson et al. (2017): 1. Identifying the elements - this is primarily identifying a list of elements that make up a practice. The output is essentially a diagram. 2. Drafting the relationships between the elements and the outline of each element - at this point, the cards are created. 3. Providing further details - Usually, the cards will be supplemented with additional gui-delines, hints and tips, examples, and references to other resources, such as articles and books
In this fashion, the framework could be essentialized by e.g. making Essence alphas
out of the principles such as transparency. Alphas, in the context of Essence, are things
to work with which are measured in order to see progress on the endeavor
As it stands, the framework can be utilized for empirical studies in the area of AI ethics. It presents a practice-focused view of AI ethics. However, it does not cover all the aspects of the AI ethics discussion in 2019 (and beyond). Depending on the context, one may wish to extend it to include fairness as the fourth key principle for AI ethics, and/or trustworthiness. 6
In this paper, we have presented a framework for approaching AI ethics through
practice. Having conducted an empirical study using the framework
We seek to develop the framework further ourselves. We have utilized a similar
framework in another study