Time for AI (Ethics) Maturity Model Is Now Ville Vakkuri1 , Marianna Jantunen1 , Erika Halme1 , Kai-Kristian Kemell1 , Anh Nguyen-Duc2 , Tommi Mikkonen3 , Pekka Abrahamsson1 1 University of Jyväskylä, Faculty of Information Technology ville.vakkuri — marianna.s.p.jantunen — erika.a.halme— kai-kristian.o.kemell — pekka.abrahamsson@jyu.fi 2 University of South Eastern Norway, School of Business angu@usn.no 3 University of Helsinki, Department of Computer Science tommi.mikkonen@helsinki.fi Abstract details of what “ethical AI” constitutes and ”which ethi- cal requirements, technical standards and best practices are There appears to be a common agreement that ethical con- needed for its realization”, is up for debate. The ethics of cerns are of high importance when it comes to systems AI systems appear open for initiatives, or as Greene, Hoff- equipped with some sort of Artificial Intelligence (AI). De- mands for ethical AI are declared from all directions. As a mann, and Stark (2019) put it, ‘up for grabs’. These initia- response, in recent years, public bodies, governments, and tives offer goals and definitions for what is expected of eth- universities have rushed in to provide a set of principles to ical AI systems. As stated in Ethically Aligned Design: A be considered when AI based systems are designed and used. Vision for Prioritizing Human Well-being with Autonomous We have learned, however, that high-level principles do not and Intelligent Systems, First Edition, regardless of the eth- turn easily into actionable advice for practitioners. Hence, ical framework we follow, the systems should be expected also companies are publishing their own ethical guidelines to to honor holistic definitions of societal prosperity, not pur- guide their AI development. This paper argues that AI soft- suing one-dimensional goals such as increased productiv- ware is still software and needs to be approached from the ity or gross domestic product. Awad et al. (2018) proposed software development perspective. The software engineering that we were entering an era where intelligent systems can paradigm has introduced maturity model thinking, which pro- vides a roadmap for companies to improve their performance be tasked to “not only to promote well-being and mini- from the selected viewpoints known as the key capabilities. mize harm, but also to distribute the well-being they cre- We want to voice out a call for action for the development of ate, and the harm they cannot eliminate”. Societal and policy a maturity model for AI software. We wish to discuss whether guidelines should be established to ensure that they remain the focus should be on AI ethics or, more broadly, the quality human-centric, serving humanity’s values and ethical prin- of an AI system, called a maturity model for the development ciples (Ethically Aligned Design: A Vision for Prioritizing of AI systems. Human Well-being with Autonomous and Intelligent Sys- tems, First Edition). Introduction Coming up with policies and enforcing them within an The ethics of Artificial Intelligence (AI) have been an organization might seem challenging and unrewarding. De- emerging topic in the field of AI development (Jobin, Ienca, mands for ethical AI are declared from all directions, but and Vayena 2019), and the ethical consequences of AI sys- the rewards and consequences of making or not making eth- tems have been researched in significant amounts in the re- ical initiatives and commitments seem unclear. When com- cent years (Ryan and Stahl 2020). Now that AI has become panies and research institutions make “ethically motivated prevalent in many decision-making processes that have the ‘self-commitments’” in the AI industry, efforts to formu- chance to directly or indirectly impact or alter lives, in fields late a binding legal framework are discouraged, and any de- such as healthcare (Panesar 2019) and transportation (Sadek mands of AI ethics laws remain relatively vague and super- 2007), concerns regarding the currently existing and hypo- ficial (Hagendorff 2020). As Greene, Hoffmann, and Stark thetical ethical impacts of AI systems have been voiced by (2019) suggest, many high-profile companies, organizations, many. With AI systems becoming pervasive, there emerges and communities have signaled their commitment to ethics, an increasing need for guidance in creating AI systems that but the resulting articulated value statements ”prompt more align with our perception of ethical behavior. questions than answers”. A problem may also emerge in the As Jobin, Ienca, and Vayena (2019) suggest, there is situation where – as presented by Hagendorff (2020) – AI an apparent agreement that AI should be ethical. Still, the ethics, like ethics in general, “lacks mechanisms to reinforce its own normative claims”. It might be that the consequences Copyright © 2021 for this paper by its authors. Use permitted under of not enforcing and applying ethical principles in AI de- Creative Commons License Attribution 4.0 International (CC BY velopment are not severe enough to motivate companies to 4.0). follow through. Despite these challenges, many organizations have re- a connection to technology policy has authored or endorsed acted to ethical concerns on AI, for example, by form- a set of principles for AI”. As an example of the aforemen- ing ad-hoc expert committees to draft policy documents tioned policy forming committees (Jobin, Ienca, and Vayena (Jobin, Ienca, and Vayena 2019) and producing statements 2019), some major publications from influential institutions, that describe ethical principles, values and other abstract re- such as The IEEE Ethically Aligned Design: A Vision for quirements for AI development and deployment (Mittelstadt Prioritizing Human Well-being with Autonomous and In- 2019). At least 84 public-private AI ethics principles and telligent Systems, First Edition; and Ethics Guidelines for values initiatives were identified by Mittelstadt (2019), and Trustworthy AI by the High-Level Expert Group appointed the topic evolves dynamically through new initiatives and by the European Commission, have introduced practical de- their iterations. Such initiatives can ”help focus public de- sign approaches and suggested standards and principles for bate on a common set of issues and principles, and raise ethical AI development and implementation. Research insti- awareness among the public, developers and institutions of tutions are only the tip of the iceberg, however; a variety the ethical challenges that accompany AI” (Mittelstadt 2019; of other institutions, such as governments and corporations, Whittaker et al. 2018). have stepped in to publish their own AI ethics guidelines, as So far, these principles and values used to form various discovered by, for example, Jobin, Ienca, and Vayena (2019). guidelines for implementing AI ethics have been the primary Even the Vatican has published their initiative, teaming up tools intended to help companies develop ethical AI systems with IBM and Microsoft to draft a call for AI ethics (Stotler (as we discuss in detail in the next section). However, as al- 2020). ready noted in existing literature (Mittelstadt 2019), these While not legally binding, the effort invested in such guidelines alone cannot guarantee ethical AI systems, and guidelines by multiple stakeholders in the field are notewor- seem to suffer from a lack of industry adoption (Vakkuri thy and influential (Jobin, Ienca, and Vayena 2019), con- et al. 2020). What, then, should be done instead? In this pa- tributing to the discussion of AI ethics. Guidelines can be per, we look at the issue from the point of view of Software seen as “part of a broader debate over how, where, and why Engineering (SE). these technologies are integrated into political, economic, One approach to tackling this issue, from the point of view and social structures” (Greene, Hoffmann, and Stark 2019) of SE, would be to focus on methods, practices, and tools for (p. 2122). We can witness how guidelines have contributed AI ethics, in order to make these principles and values more positively to the development of AI ethics discussion by ob- tangible to the developers working on these AI systems. serving the number of organizations that published their sets Some already exist, as discussed by Morley et al. (2019), of guidelines. Based on the number of organizations that although they are mostly technical ones focused specifically use the common vocabulary of ”keyworded” guidelines, dis- on, e.g., managing some aspects of machine learning. An- cussing transparency, fairness, and other such principles, it other approach, on which we focus here, is the development seems as though guidelines may have developed into a type of a maturity model. Maturity models, which we discuss fur- of ”common language” for AI ethics discussion; a familiar ther in the third section, are used in SE to evaluate the ma- format that is easy to adopt and quick to communicate. turity level of organizational processes related to software Researchers have conducted reviews on AI ethics guide- development. Could a maturity model for AI ethics help or- lines, considering their implications (e.g. Ryan and Stahl ganizations develop ethical AI? (2020)) and looking for unanimity among them (e.g. Jobin, Ienca, and Vayena (2019); Hagendorff (2020)). In the light AI Ethics Guidelines of these reviews, certain prevalent guidelines have emerged. To respond to the concerns and discussions around the ethi- For example, Jobin, Ienca, and Vayena (2019) identified cal and societal impacts of intelligent technology, guidelines a “global convergence emerging around five ethical prin- for ethical AI development have been published in the re- ciples”, namely transparency, justice and fairness, non- cent years by a variety of organizations ranging from cor- maleficence, responsibility and privacy. porations to governmental and research institutions. Still, However, guidelines alone do not cater to the whole spec- there appears to be no acknowledged single standard in trum of AI ethics challenges. Firstly, although some similar- the field, but the guidelines often appear to be either one ities emerge between sources and studies, there is no guaran- “keyword” principles such as accountability or transparency tee of unanimity of their application; even if each organiza- (Ethically Aligned Design: A Vision for Prioritizing Human tion were to adhere to the exact same set of guidelines, their Well-being with Autonomous and Intelligent Systems, First practical application is not guaranteed to be synchronized. Edition) or descriptive sentences that present the organiza- There may be questions related to, for example, interpreta- tion’s approach, such as “We want to develop safe, robust, tion, emphasis and level of commitment, that organizations and explainable AI products” (Bolle 2020). The guidelines need to make for themselves. may serve different purposes for each organization – a cor- In particular, when considering organizations employing poration’s motivation to publishing a set of ethical guide- guidelines in their AI product development, the guidelines lines to follow can be expected to be different from that of a often provide us with the answer to the question “what” is research institution. done, but not “how”. This concept seems supported by Mor- Tending to the need of standards, several organizations ley et al. (2020) when they discuss the same effect on the have stepped in to publish their own guidelines. As phrased mainstream ethical debate on AI. Another problem follow- by Fjeld et al. (2020), ”seemingly every organization with ing from reliance in guidelines is, that their impact on human decision-making is not guaranteed, and they may remain in- the potential drawbacks. For example, a past version of the effective (Hagendorff 2020). CMMI has been criticized for creating processes too heavy As reported by Vakkuri et al. (2019), there appears to for the organizations to handle (Sony 2019; Meyer 2013), be a gap between research and practice in the field of AI and in general being resource-intensive to adopt for smaller ethics when it comes to the procedures of companies, as the organizations (O’Connor and Coleman 2009). SAFe, on the academic discussions have not carried over to industry; de- other hand, has been criticized for adding bureaucracy to velopers consider ethics important in principle, but perceive Agile (Ebert and Paasivaara 2017), leaning towards the wa- them to be distant from the issues they face in their work. terfall approach. In a survey of industry practices, including 211 companies, Nonetheless, these models are widely used in the in- 106 which develop AI products, it was found that compa- dustry, either independently, or in conjunction with other nies have mixed levels of maturity in implementing AI ethics frameworks, tools, or methods. SAFe, for example, has been (Vakkuri et al. 2020). In terms of guidelines, the survey dis- adopted by 70 of the Forbes 100 companies. CMMI has even covered that the various AI ethics guidelines had not, in fact, been adopted in fields other than software development. had a notable effect on industry practices, confirming the Academic studies aside, companies seem to have taken a lik- suspicions of Mittelstadt (2019). ing to maturity models in the context of software. The high variety in both industry practices and AI ethics Indeed, this apparent popularity of these models out on guidelines may make it difficult to assess AI systems devel- the field has, in part, motivated us to write this early pro- opment, especially aspects such as trustworthiness or other posal maturity models in the context of AI ethics as well. ethics-related topics. To answer a need of standardized eval- In an area where we struggle with a gap between research uation practices, we propose a look into maturity models, and practice, we argue that looking at frameworks, mod- and their utility in evaluating software development prac- els, and other tools that are actively used out on the field tices. Maturity models or maturity practices for AI with dif- is a good starting point for further steps. Thus far, guidelines ferent emphases have already been introduced, such as the have been used to make AI ethics principles more tangible, AI-RFX Procurement Framework by The Institute for Eth- but further steps are still needed, and a maturity model could ical AI and Machine Learning (The Institute for Ethical AI be one such step. and Machine Learning 2020) and The AI Maturity Frame- work (Ramakrishnan et al. 2020). Next, we discuss maturity What about an AI Ethics Maturity model or models in general, before discussing them further in the spe- cific context of AI and AI ethics in the fourth section. an AI Maturity Model? Despite the criticism towards maturity models discussed What are Maturity Models? above, maturity models are widely used in the industry. Con- Maturity models are intended to help companies appraise versely, the AI ethics guidelines that have been somewhat their process maturity and develop it. They serve as points well-received in the academia seem to not have seen much of reference for different stages of maturity in an area. In the interest out on the field. We thus propose that an AI devel- context of SE, they are intended to help organizations move opment maturity model might take us closer to standardize- from ad hoc processes to mature and disciplined software able and ethically sound AI development practices. processes (Herbsleb et al. 1997). Since the Software Engi- AI systems are particularly software-intensive systems. neering Institute launched the Capability Maturity Model Only a small fraction of a typical industrial AI system is (CMM) almost twenty years ago (Paulk et al. 1993), hun- composed of Machine Learning (ML) or AI code. The rest dreds of maturity models have been proposed by researchers consists of computing infrastructure, data, process manage- and practitioners across multiple domains, providing frame- ment tools, etc. However, considering the overall analytic work to assess current effectiveness of an organization and capability of AI systems, we need to have code for the supports figuring out what capabilities they need to acquire ML model itself, visualization of the ML model outcome, next in order to improve their performance. data management, and integration of ML into other software Though maturity models are numerous in SE, the Scaled modules. This code is hardly trivial and requires proper engi- Agile Framework (SAFe) and Capability Maturity Model neering principles and practices (Carleton et al. 2020). This Integration (CMMI) are some typical high-profile exam- lends support to the idea of an AI maturity model. ples of maturity models in the field of SE. SAFe is a mix- Seeing as there are already numerous software maturity ture of different software development practices and fo- models, a question worth asking is whether they would al- cuses mainly on scaling agile development in larger orga- ready solve this issue. I.e., do we really need an AI ethics nizations. CMMI, on the other hand, focuses on improve- maturity model? In comparison to traditional non-AI soft- ments related to software development processes. In gen- ware code, AI systems are sensitive to some special quality eral, Software Process Improvement tools are rooted in attributes, such as technical debt, due to various AI-specific Shewhart-Deming’s plan-do-check-act (PDCA) paradigm, issues. While traditional software are deterministic with a where CMMI, for example, represents a prescriptive frame- pre-defined test oracle, AI/ ML models are probabilistic. work in which the improvements are based on best practices ML models learn from data and the model quality attributes, (Pernstål et al. 2019). such as accuracy change throughout the process of experi- Maturity models have been studied in academic research menting. Moreover, ethical requirements, or attributes such as well. Studies have focused on both their benefits and as fairness, trustworthiness, transparency, and explainability (Jobin, Ienca, and Vayena 2019), have unique meanings in tion whether the existing AI maturity related models work the context of AI, and they are not sufficiently addressed in in more detail, and if not, why not? If they do not, what existing software models. Moreover, data is the central com- approach should the new model take to tackle the existing ponent of the engineering process with a lot of new prob- issues? Moreover, how would this model relate to existing lems, such as dealing with missing values, data granularity, software maturity models, and why would those not be ap- design and management of the database, data lake, and the plicable in the AI context? Additionally, how would the de- quality of the training data in comparison to real-world data. velopment effort be communicated to those already involved These differences complicate applying traditional software with existing software maturity models? Would the model be models to AI. competing with existing ones or be a complementary one to Several AI-specific models have been published, for ex- be used in conjunction? ample, a Microsoft nine-step pipeline (Amershi et al. 2019), Whichever approach is chosen, this would be a large en- a five-step “stairway to heaven” AI model (Lwakatare et al. deavor, as we discuss further in the next section. The discus- 2019), and a maturity framework for AI process (Akki- sion on AI ethics has gone a long way in the past decade. raju et al. 2020). However, they are not particularly fo- Though this discussion is still on-going in terms of princi- cused on the quality or ethical aspects of developing AI sys- ples, the time to act is now when it comes to bringing this tems. Besides, while these models reflect processes in par- discussion into practice. Whether or not AI ethics is a part of ticular organizational contexts, there is currently no general AI development, AI systems will become increasingly com- model that could be adopted in SMEs and startup compa- mon, and thus it is important to already make further efforts nies (Nguyen-Duc et al. 2020). Hence, a generic AI (ethics) at bridging the gap. Choices have to be made on which AI maturity model is still needed to benchmark and promote ethics principles and issues to focus on in such models. the proper engineering practices and processes to plan, to implement, and to integrate ethical requirements. Moreover, Call for Action this model should facilitate standardizing and disseminating best practices to developers, scientists and organizations. In a nutshell, we propose the development of an AI (ethics) In devising a maturity model for this area, one important maturity model to cover the entire sphere of technical and question is whether such a model should be an AI Ethics ethical quality requirements. Such maturity model would Maturity Model or simply an AI Maturity Model. Both ap- help the field move from ad hoc implementation of ethics proaches, we argue, would have their own potential benefits (or total negligence), to a more mature process level, and and potential drawbacks. ultimately, if possible, automation (Figure 1). Furthermore, First, an AI Ethics Maturity Model. Being a field-specific we argue that this model should not be an effort for a single model, an AI ethics maturity model would address the nu- researcher or research group, but a multidisciplinary project merous AI ethics needs discussed in academic literature that builds on a combination of theoretical models and em- and public discussion alike. Such a maturity model could pirical results. be devised so that it would directly complement the on- The first step in creating an AI (ethics) maturity model going principle and guideline discussion, and help bring it would be the formulation of requirements for different as- into practice. Moreover, focusing on ethics over SE would pects of AI (ethics) maturity. We may require different types make it potentially suitable for any organization regardless of commonly acknowledged agreements on issues that AI of their chosen development approach, although one should maturity entails. We also need to refine a topic still shrouded still keep in mind its suitability for iterative development ap- in vagueness to some extent, AI ethics, into solid, univer- proaches. sally applicable requirements. On the other hand, were the model too focused on AI In this paper, we have introduced lots of challenges related ethics issues or design-level issues, the practical SE side to the variety of practices and motivations that stakeholders could be lacking. This could result in a situation where the involved in AI systems development face, and this noncon- maturity model would still face the issue of being impracti- formity can pose challenges in making an AI maturity model cal, much like the existing guidelines. In general, the model applicable universally, as much as that can be realistically might risk being detached from industry practice. Compa- striven for. In order to improve the universal applicability of nies should be closely involved when devising such a model a maturity model, we should look into ways to form agree- in order to mitigate these potential drawbacks. ments, preferably ones that are agreed on as universally as Secondly, an AI Maturity Model, an approach where the possible, to avoid unnecessarily limiting the model’s use. focus is not on AI ethics as such. An AI Maturity Model In addition to the numerous AI ethics guidelines and would arguably be more technical; speaking the language of the principles presented in them (Jobin, Ienca, and Vayena the developers, so to say. This would likely make the matu- 2019), we should also consider looking into standards as a rity model more attractive from the point of view of industry. starting point for agreements in this context. As suggested AI Ethics could (or would) still be present, but be embedded by Cihon (2019), AI presents ”novel policy challenges” that into the more practice-focused model as simply one aspect require a coordinated response globally - and standards de- of the model. Moreover, such a model could advance the AI veloped by international standards bodies can support the maturity discussion as a whole and not only from the point governance of AI development. Such widely acknowledged of view of ethics. agreements could be harnessed to build unity and alignment On the other hand, this approach would force us to ques- in defining maturity in AI systems development. AI-related standards might answer the problem of vagueness and dis- agreement, when setting up requirements for what ethical AI maturity should look like. Several organizations have already published, discussed, or suggested standards, so the work is already underway and there are already standards to utilize. Some standards to con- sider regarding ethical AI might be, for example: • ISO/IEC JTC 1/SC 421 , standard for Artificial intelli- gence, created in 2017, an undergoing work that includes some published and several under development ISO stan- dards, and • Standards under IEEE P70002 - Standard for Model Pro- cess for Addressing Ethical Concerns During System De- Figure 1: The maturity model shaped by given requirements sign, that includes several standards that are relevant to AI systems. For example IEEE P7001 - Standards for Trans- parency of Autonomous Systems and IEEE P7006 - Stan- International Conference on Business Process Management, dard for Personal Data Artificial Intelligence (AI) Agent 17–31. Springer. Requirements set for AI systems by internationally ac- Amershi, S.; Begel, A.; Bird, C.; DeLine, R.; Gall, H.; Ka- cepted standards, together with guidelines that have reached mar, E.; Nagappan, N.; Nushi, B.; and Zimmermann, T. a consensus across different domains of business and re- 2019. Software engineering for machine learning: A case search, can perhaps be used as building blocks in forming study. In 2019 IEEE/ACM 41st International Conference an ethically aligned AI maturity model. The numerous AI on Software Engineering: Software Engineering in Practice ethics guidelines should also help in this regard. While the (ICSE-SEIP), 291–300. IEEE. existing AI ethics guidelines, as guidelines, have faced the issue of not being widely adopted out on the field (Vakkuri Awad, E.; Dsouza, S.; Kim, R.; Schulz, J.; Henrich, J.; Shar- et al. 2020), the principles in them are still relevant. Incor- iff, A.; Bonnefon, J.-F.; and Rahwan, I. 2018. The moral porating those principles into a more practical form – a ma- machine experiment. Nature 563(7729): 59–64. turity model is one – is what this call for action is ultimately Bolle, M. 2020. Code of ethics for AI. Technical report, about when it comes to AI ethics. Robert Bosch GmbH. URL https://www.bosch.com/stories/ In distilling the discussion on AI ethics principles, IEEE’s ethical-guidelines-for-artificial-intelligence/. Ethically Aligned Design (Ethically Aligned Design: A Vi- sion for Prioritizing Human Well-being with Autonomous Carleton, A. D.; Harper, E.; Menzies, T.; Xie, T.; Eldh, S.; and Intelligent Systems, First Edition) presents an exten- and Lyu, M. R. 2020. The AI Effect: Working at the Inter- sive set of guidelines. The EU has also produced a re- section of AI and SE. IEEE Software 37(4): 26–35. port that has tried to make these principles more actionable Cihon, P. 2019. Standards for AI governance: international through checklists of questions to be asked during develop- standards to enable global coordination in AI research & de- ment (Ethics Guidelines for Trustworthy AI). The ECCOLA velopment. Future of Humanity Institute. University of Ox- method has also been an attempt at making these principles ford . more actionable (Vakkuri, Kemell, and Abrahamsson 2020). While the discussion on these AI ethics principles is on- Ebert, C.; and Paasivaara, M. 2017. Scaling agile. Ieee Soft- going, decisions are needed to fight vagueness and to incite ware 34(6): 98–103. action. In this regard, we could bring up the Agile Mani- Ethically Aligned Design: A Vision for Prioritizing Human festo3 . A product of its time, it was a declaration of what Ag- Well-being with Autonomous and Intelligent Systems, First ile software development should be like, written by a small Edition. 2019. URL https://standards.ieee.org/content/ieee- group of people. However, it has helped define what Agile standards/en/industry-connections/ec/autonomous- software development is and has encouraged organizations systems.html. to discuss maturity in the context of Agile. Ethics Guidelines for Trustworthy AI. 2019. URL https://ec.europa.eu/digital-single-market/en/news/ethics- References guidelines-trustworthy-ai. Akkiraju, R.; Sinha, V.; Xu, A.; Mahmud, J.; Gundecha, P.; Fjeld, J.; Achten, N.; Hilligoss, H.; Nagy, A.; and Srikumar, Liu, Z.; Liu, X.; and Schumacher, J. 2020. Characterizing M. 2020. Principled artificial intelligence: Mapping consen- Machine Learning Processes: A Maturity Framework. In sus in ethical and rights-based approaches to principles for 1 AI. Berkman Klein Center Research Publication 2020-1. https://www.iso.org/committee/6794475.html 2 Greene, D.; Hoffmann, A. L.; and Stark, L. 2019. Better, https://standards.ieee.org/initiatives/artificial-intelligence- systems/standards.html nicer, clearer, fairer: A critical assessment of the movement 3 http://agilemanifesto.org/ for ethical artificial intelligence and machine learning. In Proceedings of the 52nd Hawaii International Conference Marble, A.; Ok, F.; Ryan, M.; Sobolewski, C.; Suen, P.; on System Sciences. Truong, S.; and Zurof, R. 2020. The AI Maturity Frame- Hagendorff, T. 2020. The ethics of AI ethics: An evaluation work. Technical report, Element AI. of guidelines. Minds and Machines 1–22. Ryan, M.; and Stahl, B. C. 2020. Artificial intelligence Herbsleb, J.; Zubrow, D.; Goldenson, D.; Hayes, W.; and ethics guidelines for developers and users: clarifying their Paulk, M. 1997. Software quality and the capability maturity content and normative implications. Journal of Information, model. Communications of the ACM 40(6): 30–40. Communication and Ethics in Society . Jobin, A.; Ienca, M.; and Vayena, E. 2019. The global land- Sadek, A. W. 2007. Artificial intelligence applications in scape of AI ethics guidelines. Nature Machine Intelligence transportation. Transportation Research Circular 1–7. 1(9): 389–399. Sony, M. 2019. Implementing sustainable operational excel- Lwakatare, L. E.; Raj, A.; Bosch, J.; Olsson, H. H.; and lence in organizations: an integrative viewpoint. Production Crnkovic, I. 2019. A Taxonomy of Software Engineering & Manufacturing Research 7(1): 67–87. Challenges for Machine Learning Systems: An Empirical Stotler, L. 2020. The Vatican Teams with Mi- Investigation. In Kruchten, P.; Fraser, S.; and Coallier, F., crosoft and IBM to Call for AI Ethics. URL eds., Agile Processes in Software Engineering and Extreme https://www.futureofworknews.com/topics/futureofwork/ Programming, 227–243. Springer International Publishing. articles/444670-vatican-teams-with-microsoft-ibm-call-ai- Meyer, B. 2013. What is wrong with CMMI. URL ethics.htm. https://bertrandmeyer.com/2013/05/12/what-is-wrong- The Institute for Ethical AI and Machine Learning. 2020. with-cmmi/. AI-RFX Procurement Framework v1.0. URL https://ethical. Mittelstadt, B. 2019. Principles alone cannot guarantee eth- institute/mlmm.html. ical AI. Nature Machine Intelligence 1–7. Vakkuri, V.; Kemell, K.; Kultanen, J.; and Abrahamsson, P. Morley, J.; Floridi, L.; Kinsey, L.; and Elhalal, A. 2019. 2020. The Current State of Industrial Practice in Artificial From What to How. An Overview of AI Ethics Tools, Meth- Intelligence Ethics. IEEE Software 37(4): 50–57. ods and Research to Translate Principles into Practices. Vakkuri, V.; Kemell, K.; Kultanen, J.; Siponen, M. T.; and arXiv preprint arXiv:1905.06876 . Abrahamsson, P. 2019. Ethically Aligned Design of Au- Morley, J.; Floridi, L.; Kinsey, L.; and Elhalal, A. 2020. tonomous Systems: Industry viewpoint and an empirical From what to how: an initial review of publicly available study. arXiv preprint arXiv:1906.07946 . AI ethics tools, methods and research to translate principles Vakkuri, V.; Kemell, K.-K.; and Abrahamsson, P. 2020. EC- into practices. Science and engineering ethics 26(4): 2141– COLA : a Method for Implementing Ethically Aligned AI 2168. Systems. In Proceedings of the 46th Euromicro Confer- Nguyen-Duc, A.; Sundbø, I.; Nascimento, E.; Conte, T.; ence on Software Engineering and Advanced Applications, Ahmed, I.; and Abrahamsson, P. 2020. A Multiple Case Euromicro Conference on Software Engineering and Ad- Study of Artificial Intelligent System Development in In- vanced Applications (SEAA2020), 195–204. IEEE. doi: dustry. In Proceedings of the Evaluation and Assessment 10.1109/seaa51224.2020.00043. in Software Engineering, EASE ’20, 1–10. Association for Whittaker, M.; Crawford, K.; Dobbe, R.; Fried, G.; Kaziu- Computing Machinery. doi:10.1145/3383219.3383220. nas, E.; Mathur, V.; West, S. M.; Richardson, R.; Schultz, J.; and Schwartz, O. 2018. AI now report 2018. AI Now O’Connor, R.; and Coleman, G. 2009. Ignoring ”best prac- Institute at New York University New York. tice”: Why Irish software SMES are rejecting CMMI and ISO 9000. Australasian J. of Inf. Systems 16. doi:10.3127/ ajis.v16i1.557. Panesar, A. 2019. Machine Learning and AI for Healthcare. Springer. Paulk, M.; Curtis, W.; Chrissis, M. B.; and Weber, C. 1993. Capability Maturity Model for Software (Version 1.1). Technical Report CMU/SEI-93-TR-024, Software En- gineering Institute, Carnegie Mellon University, Pittsburgh, PA. URL http://resources.sei.cmu.edu/library/asset-view. cfm?AssetID=11955. Pernstål, J.; Feldt, R.; Gorschek, T.; and Florén, D. 2019. FLEX-RCA: a lean-based method for root cause analysis in software process improvement. Software Quality Journal 27(1): 389–428. Ramakrishnan, K.; Salveson, C.; Abuhamad, G.; Chantry, C.; Diamond, S.-P.; Donelson, P.; Ebert, L.; Koleilat, W.;