In this paper we summarize the research project currently conducted by the Software Engineering Research LABoratory (SERLAB) at Dipartimento di Informatica, Università degli Studi di Bari ”A. Moro” on the topic of Responsible Artificial Intelligence (RAI). should lead its development and application [4][5]. cial Intelligence (RAI).
reshaping science and society as a whole [1]. While
autonomous decision systems are being used more processed and analyzed [2], autonomous and semi- values”. frequently in several industries, such as healthcare, stead of Responsible.
automotive, banking, and manufacturing, just to cite a few [3]. Given AI revolutionary potential and wide-ranging social influence, there has been a lot of discussion regarding the values and principles that
been focused on concerns that AI may endanger the jobs of human workers [6], be abused by malicious actors [7], avoid responsibility, or accidentally spread bias and as so, erode fairness [8].
sible Artificial Intelligence (RAI) started emerging.
”intelligent algorithms that prioritize the needs of all stakeholders as the highest priority, especially the minoritized and disadvantaged users, in order to make trustworthy decisions. These obligations and mitigating negative impacts, and maximizing the long-term beneficial impact. (Socially) Responsible AI Algorithms constantly receive feedback from users to continually accomplish the expected social
sources published by OECD AI Policy Observatory (OECD.AI1). Since these terms can be treated as synonyms, for the sake of simplicity, in this document we will proceed using only Responsible Artifi
sponded to these societal fears by developing diferent kinds of resources: ethical requirements, principles, guidelines, best practices, tools, and frameworks.
activities intended to support, by shedding lights on problems and providing possible solutions, the realisation of a more responsible AI world.
Section 2 defines some background definitions useful to set the stage; Section 3 highlights the research problem we are studying; Section 4 describes our vision and the research questions we are trying to answer; Section 5 summarizes our most important ifndings to date and, finally, our planned future works are drawn in Section 6. 2.
In this section we provide some preliminary definitions to better understand the concepts that guide 2.1. Responsible AI Principles 2https://digital-strategy.ec.europa.eu/en/policies/ expert-group-ai 3https://www.unesco.org/en/artificial-intelligence/ recommendation-ethics 4https://www.cms-holbornasia.law/en/sgh/publication/ singapore-to-form-advisory-council-for-ethical-use-of-ai 5https://ai.jpl.nasa.gov/ 6https://www.gov.uk/government/groups/ai-council 7https://www.sony.com/en/SonyInfo/sony_ai/ responsible_ai.html 8https://ai.facebook.com/blog/ facebooks-five-pillars-of-responsible-ai/ 9http://www.thefutureworldofwork.org/media/35420/uni_ ethical_ai.pdf 10https://www.internetsociety.org/resources/doc/2017/ artificial-intelligence-and-machine-learning-policy-paper/ around five ethical principles: transparency, justice and fairness, non-maleficence , responsibility, and privacy. Jobin et al. [12] stated that no one of these ethical principles is present in all the documents they reviewed; however, these five principles are mentioned in more than half of all the sources reviewed. Moreover, further in-depth thematic analysis revealed notable semantic and conceptual divergences in interpreting these principles and in the particular recommendations or areas of concern drawn from each of them.
ated ad-hoc expert groups on AI to address the risks connected with the development of AI, frequently with the task of generating policy documents. These organizations include, among others, the High-Level Expert Group on Artificial Intelligence established by the European Commission2, the UNESCO Ad Hoc Expert Group (AHEG) for the Recommendation on the Ethics of Artificial Intelligence 3, the Advisory Council on the Ethical Use of Artificial Intelligence and Data in Singapore4, the NASA Artificial Intelligence Group 5 and the UK AI Council6, 2.2. Chosen AI principles definitions just to cite a few. As highlighted in Section 2.1, there are a lot of
These committees have been appointed to pro- uncertainties and nuances around the definition of duce reports and guidelines about Responsible AI. the principles that mainly characterize RAI, as well Similar initiatives are being made in the commercial as, about the definition of RAI itself. sector, particularly by businesses that depend on In our research, to address the problem of prinAI. Businesses like Sony7 and Meta8 made their ciple proliferation, we have decided to focus on a AI policies and principles available to the public. specific subset of principles, in particular, the four At the same time, professional organizations and principles identified by Jobin et al. [ 12] with the no-profit groups like UNI Global Union 9 and the exclusion of responsibility as this concept is rarely Internet Society10 have all released statements and defined in a clear manner. recommendations. Moreover, to give an authoritative and clear def
The significant eforts of such an ample group of inition for each principle, we decided to consider stakeholders to develop RAI principles and policies the ones provided by the High-Level Expert Group not only show the need for ethical guidance but also on Artificial Intelligence established by the Europoint out their keen interest in reshaping AI ethics pean Commission11 in their Ethics guidelines for to suit their individual priorities [10]. Notably, the trustworthy AI [14]. private sector’s participation in the field of AI ethics The resulting chosen principles are transparency has been questioned since it may be using high-level (often known as explainability), justice and fairsoft policy as a portmanteau to either make a social ness, non-maleficence (often known as security) and issue technical [10] or avoid regulation altogether privacy. [11, 12].
However, many research works highlighted how these proposals often diverged, giving diferent defi- 2.3. Frameworks nitions, resulting in the problem known as principle proliferation [13]. Consequently, several in-depth investigations have been conducted, such as the one by Jobin et al. [12], who found a global convergence
The concept of framework is far well-known in the Software Engineering (SE) field. Already in 1997, Johnson et al. [15] referred to frameworks as ”an object-oriented reuse technique” or ”the skeleton of an application that can be customized by an application developer ”. These are not conflicting definitions; the first describes the structure of a framework while the second describes its purpose.
Shifting the focus from SE to a more general context, frameworks are a form of design reuse.
Frameworks can be considered a collection of suggestions, guidelines and tools to be followed in order to create a product compliant with a defined standard. 11https://digital-strategy.ec.europa.eu/en/policies/ expert-group-ai
Of fundamental importance in our research are tens of millions of people which in 2019 was found frameworks that implement the above-mentioned to be biased against dark-skinned patients in New RAI ethical principles. Jersey, USA; the efects were it was assigning darkskinned patients lower scores than white patients with the same medical conditions15. This could have caused wrong medical prescriptions.
The concept of Software Development Life Cycle 3.2. Non-maleficence (or Security) (SDLC) represents a fundamental piece of knowledge in the field of Software Engineering (SE). Boehm et Since AI models learn from data, there exists the al. [16] in 1976 already were talking about common possibility a malicious actor can provide manipuactivities involved in the production of a software lated training samples which force the model to system. Subsequently, these activities have been learn from a distorted reality. standardised in 2017 into the ISO/IEC 1220712. This threat is known as adversarial machine learnThese standardized activities which compose the ing, which poses great challenges to deep learning SDLC are: models. This threat is particularly harmful in safetycritical scenarios, such as self-driving, where the • Requirement Gathering and Analysis. vision system must be robust to ad-hoc crafted ex• Design. ternal perturbations. An adversarial example is • Implementation or Coding. a malicious input typically created by applying a • Testing. small but intentional perturbation, such that the • Deployment. attacked AI model misclassifies it with high confidence.
According to the development methodology used Diferent specific instances of this threat exist: (e.g. Agile13 or Waterfall14), these activities may Adversarial Examples, Data Poisoning, Model Evabe done sequentially or iteratively. sion, Trojan (or Backdoor) and Model stealing (or Model Extraction).
The victim of such an attack was the chatbot 3. Problem statement ”Tay”, developed by Microsoft in 2016, which was shut down and closed a few hours after its release time because was attacked and forced to post ofensive tweets against users16.
AI models can amplify existing bias coded in data or applied in real contexts, assisting humans in the introduce new forms of bias [17], resulting in unfair most disparate domains. However, the increase decisions in legal or ethical sense. In particular, of their performance has been accompanied by an AI-based systems may produce decisions or have increase of complexity at a level that no one, neither impacts that are discriminatory or unfair, and this their designers, can interpret the inner workings is especially true when AI is deployed in complex leading to decisions. socio-technical systems. Many researchers started focusing on the urgent
Note that cases like these do not have to be inten- open challenge of how to construct meaningful explational on the part of the people who designed/devel- nations for opaque (i.e. systems whose functioning oped these systems. Instead, issues like these may logic is not comprehensible by a human) AI systems arise, for example, due to the datasets or algorithms in the context of AI-based decision-making, aiming used to develop the systems. at empowering individuals against undesired efects
An example of ”justice and fairness issue” in an AI of automated decision-making, implementing the system is a famous smart algorithm guiding care for “right of explanation”, helping people make better decisions preserving (and expand) human autonomy. 12https://www.iso.org/standard/63712.html 13https://www.sciencedirect.com/topics/computer-science/
agile-methodology 15https://www.nature.com/articles/d41586-019-03228-6 14https://www.sciencedirect.com/topics/computer-science/ 16https://blogs.microsoft.com/blog/2016/03/25/ waterfall-methodology learning-tays-introduction/
(XAI).
This is the only principle which does not pose direct harm to the final human users of the model. Anyway, an understandable AI algorithm would instil confidence in its users and help its owners in understanding and debugging unexpected decisions.
non-technical stakeholders, with guidelines, best practices, and tools to support and guide the development of Responsible AI applications in all the Software Development Lifecycle (SDLC).
Privacy is currently one of the first human rights roadmap made of sequential and correlated steps: that has been considered in legal frameworks and realize a framework prototype that helps diferent regulations, e.g. the European GDPR [18]. As a kinds of stakeholders to address Responsible AI consequence, even if a lot of work has been done issues. in scientific literature, there is still the need to in- This roadmap starts with the study of the current vestigate new methodologies and approaches to (a) literature in the field of Responsible AI, with the formally define and automatically detect privacy goal of understanding what has been done so far risks raised by AI systems handling diferent kinds and collecting the discovered gaps and needs. of personal data; (b) design data anonymization al- Since the literature may occasionally miss pargorithms that are suficiently robust to sophisticated ticular problems encountered by the AI practitionattacks but computationally feasible; (c) design AI ers, the roadmap includes a study on the field dialgorithms or plugins able to enforce privacy by rectly with the practitioners, to understand their design constraints; (d) investigate existing measures real needs and validate the results obtained from (or create new ones) to evaluate the privacy risk of the literature study. novel or unusual kinds of data. Finally, the roadmap envisages the development
The privacy attacks the literature already con- of a framework prototype to guide diferent stakeducted against AI models are Membership Inference holders — both with a technical or non-technical and Model Inversion, aimed to reconstruct the data background — in the development of AI applicaon which the model was trained or the model itself. tions.
Another aspect to consider is the possible data One peculiarity of this prototype is that at its leakage which can be caused by the interoperability best it should cover all diferent phases and activof AI models with classical software. Just a few days ities of the SDLC: we consider this a mandatory ago, celebrity ChatGPT has an issue with the titles requirement because, while realizing a product or a given to its users’ chats17: users’ conversations were service, several stakeholders collaborate to achieve randomly exposed to other users without consent, their common final goal — i.e. the product or serwhich may be a violation of GDPR regulation by vice — but each one has diferent skills and provides OpenAI (the company behind ChatGPT). The root a diferent point of view for the product (or service). cause was a ”major issue” with a third-party open- We expanded this overall goal in a few macro source library, according to the company, which has research questions: subsequently been resolved. This incident, added to other missing protections, cost ChatGPT a ban in Italy from the Italian data-protection authority18.
17https://techmonitor.ai/technology/ai-and-automation/
chatgpt-bug-openai-gdpr 18https://www.bbc.com/news/technology-65139406 • RQ1: What is the state of the practice and the correlated literature to approach the Responsible AI development? • RQ2: What do the practitioners think about
Responsible AI? What are their perceived gaps? • RQ3: Is it possible to realize a framework able to support diferent kinds of stakeholders in implementing Responsible AI?
In order to address each of these questions, in the following there are the Actions (A) we intend to perform: • A1: Perform a rapid review to collect all published resources — such as frameworks and tools — related to Responsible AI. • A2: Spread a survey and conduct focus All these findings are also supported by the comgroups with AI practitioners, both from mon worries publicly raised by some experts on AI, academia and industry, to understand if they who stated ”AI systems with human-competitive agree with the problems that emerged from intelligence can pose profound risks to society and the literature and discover new specific gaps humanity, as shown by extensive research”19. To and necessities. summarize, right now does not exist any comprehen• A3: Define a framework prototype to support sive framework whose knowledge can be navigated AI practitioners in the development of AI and exploited by diferent kinds of stakeholders applications. This framework should include (technical and non-technical ones), which can simboth guidelines and tools related to RAI. plify and speed up the adoption of RAI practices.
To answer RQ1, in [19] we investigated the state of As already mentioned in Sec. 4, what we discovered the literature and of the practice by doing a rapid while analyzing the literature may be diferent from review of most of the frameworks proposed by both what practitioners actually need. This leads now to public and private entities to address Responsible continue our research by validating in the practical AI issues. An overview of the findings is presented ifeld our previous findings. below; the interested reader can find all the details Our next step consists in spreading a surin [19]. vey among AI experts (both from industry and
In November 2022 we consulted various search academia) to collect as much structured data as engines, to collect both white- and grey-literature possible, in order to derive an initial preview of the resources. The research lasted one month and ended actual practical gaps in the state of the practice. up with 148 unique resources (without duplicates). By analyzing this data, we aim to extract the key
All the retrieved sources were classified w.r.t. the points requiring a deeper investigation. Then we type of proposing institution (COMPANIES, UNI- intend to eviscerate these key points by conducting VERSITIES, and NO-PROFIT ORG / COMMUNI- focus groups in which we ask the practitioners if TIES / PUBLIC ENTITIES (NPG/COMM/PE)) they agree regarding the gaps that emerged from and according to their type (Principle (P), Guideline literature on Responsible AI. We will also ask for (G), Tool (T) or Other (O)). their points of view, possibly by showing real-world
First of all, our analysis highlighted that most cases in which they would have had suggestions of the filtered frameworks are proposed by NPG/- regarding Responsible AI. This formalized data will COMM/PE (50.7%). Regarding the type, we can enable us to answer RQ2. say that there is a worrying lack of tools: most of Then, once the practitioners’ needs will be clearly the frameworks are just Principles or Guidelines. elicited, we will be one step away from our main
A positive trend is that the majority of the frame- goal to develop a comprehensive framework able to works address all four principles presented in Sec. provide support to multiple diferent stakeholders 2.2, even if sometimes in a ”partial” way: this re- while addressing Responsibility issues in AI across veals an even greater lack of consensus and stan- the entire SDLC. We plan to include information at dardization about which are the best practices to diferent abstraction levels and coming from diferfollow to be compliant with the RAI values. ent knowledge domains (e.g., legal laws as well as
Nevertheless, some frameworks neglect one, two, best practices for software development). A possible or even three of these principles. solution may be the formalization of a knowledge
Anyway, a negative trend is that very few frame- base, but we must be careful to keep knowledge in a works encompass all the SDLC phases, thus not structured and organized form, in order to facilitate providing practical support to practitioners who its query. want to develop, test, and deploy RAI applications; Before obtaining a useful and user-friendly framemost frameworks focus only on the initial phases work, the underlying knowledge base must be valof the SDLC, and in particular on Requirements idated on real-world AI systems, both open- and elicitation. private- source, to identify possible gaps and apply
Finally, another negative trend is that in most the required refinements. cases there is not a practical tool complementing the theoretical frameworks; this is true regardless of the type of entity releasing the tool. 19https://futureoflife.org/open-letter/ pause-giant-ai-experiments/
Finally, we plan to realize an automated and user- and machine learning, in: Hawaii International friendly interface which should support the various Conference on System Sciences, 2019. stakeholders and possibly automate repetitive tasks. [11] Wagner, B. in Being Profiled: Cogitas Ergo Sum. 10 Years of ‘Profiling the European Citizen‘ (eds Bayamlioglu, E., Baraliuc, I., References Janssens, L. A. W. & Hildebrandt, M.), Amsterdam University Press, 2018. URL: https:// [1] Y. N. Harari, Reboot for the ai revolution, mediarep.org/handle/doc/14277. doi:10.1515/ Nature 550 (2017) 324–327. URL: https://doi. 9789048550180.
org/10.1038/550324a. doi:10.1038/550324a. [12] A. Jobin, M. Ienca, E. Vayena, The global [2] M. I. Jordan, T. M. Mitchell, Machine landscape of ai ethics guidelines, Nature Malearning: Trends, perspectives, and prospects, chine Intelligence 1 (2019) 389–399. doi:10. Science 349 (2015) 255–260. doi:10.1126/ 1038/s42256-019-0088-2.
science.aaa8415. [13] L. Floridi, J. Cowls, A unified framework of [3] G. Cornacchia, F. Narducci, A. Ragone, ifve principles for ai in society, Issue 1 (2019).
Improving the user experience and the [14] High-Level Expert Group on AI (AIHtrustworthiness of financial services, in: LEG), Ethics guidelines for trustworthy AI Human-Computer Interaction - INTER- | Shaping Europe’s digital future, 2018. ACT 2021 - 18th IFIP TC 13 Interna- URL: https://digital-strategy.ec.europa.eu/ tional Conference, volume 12936 of Lec- en/library/ethics-guidelines-trustworthy-ai. ture Notes in Computer Science, Springer, [15] R. E. Johnson, Frameworks = (components 2021, pp. 264–269. URL: https://doi.org/ + patterns), Communications of the ACM 40 10.1007/978-3-030-85607-6_19. doi:10.1007/ (1997) 39 – 42. doi:10.1145/262793.262799. 978-3-030-85607-6\_19. [16] B. W. Boehm, Software engineering, IEEE [4] E. Vayena, A. Blasimme, I. Cohen, Machine Transactions on Computers C-25 (1976) 1226 learning in medicine: Addressing ethical chal- – 1241. lenges, PLOS Medicine 15 (2018) e1002689. [17] E. Ntoutsi, P. Fafalios, U. Gadiraju, V. Iosifidis, doi:10.1371/journal.pmed.1002689. W. Nejdl, M.-E. Vidal, S. Ruggieri, F. Turini, [5] E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Hen- S. Papadopoulos, E. Krasanakis, I. Komrich, A. Sharif, J.-F. Bonnefon, I. Rahwan, patsiaris, K. Kinder-Kurlanda, C. Wagner, The moral machine experiment, Nature 563 F. Karimi, M. Fernandez, H. Alani, B. Berendt, (2018). doi:10.1038/s41586-018-0637-6. T. Kruegel, C. Heinze, K. Broelemann, G. Kas[6] n.d., Science must examine the future of neci, T. Tiropanis, S. Staab, Bias in datawork, Nature News 550 (2017) 301–302. driven ai systems – an introductory survey, URL: https://doi.org/10.1038/550301b. 2020. arXiv:2001.09762.
doi:10.1038/550301b. [18] E. Parliament, of the Council of Euro[7] M. Brundage, S. Avin, J. Clark, H. Toner, pean Union, Regulation (eu) 2016/679 of the P. Eckersley, B. Garfinkel, A. Dafoe, european parliament and of the council of 27 P. Scharre, T. Zeitzof, B. Filar, H. Ander- april 2016 on the protection of natural persons son, H. Rof, G. Allen, J. Steinhardt, C. Flynn, with regard to the processing of personal data S. hÉigeartaigh, S. Beard, H. Belfield, S. Far- and on the free movement of such data, and quhar, D. Amodei, The malicious use of artifi- repealing directive 95/46/ec (general data procial intelligence: Forecasting, prevention, and tection regulation) (text with eea relevance), mitigation (2018). 2016. [8] J. Zou, L. Schiebinger, Ai can be sexist and [19] V. S. Barletta, D. Caivano, D. Gigante, racist — it’s time to make it fair, Nature 559 A. Ragone, A rapid review of responsible ai (2018) 324–326. frameworks: How to guide the development of [9] L. Cheng, K. R. Varshney, H. Liu, Socially re- ethical ai, The International Conference on sponsible ai algorithms: Issues, purposes, and Evaluation and Assessment in Software Engichallenges, Journal of Artificial Intelligence neering (EASE) 2023 (To appear).
Research 71 (2021) 1137–1181. [10] D. Greene, A. L. Hofmann, L. Stark, Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence