This document summarizes the activities regarding the development of Responsible AI (Responsible Artificial Intelligence) conducted by the Knowledge Discovery and Data mining group (KDD-Lab), a joint research group of the Institute of Information Science and Technologies “Alessandro Faedo” (ISTI) of the National Research Council of Italy (CNR), the Department of Computer Science of the University of Pisa, and the Scuola Normale Superiore of Pisa.
Big Data typically describes diferent dimensions of the daily social life and are the heart of a knowledge society, where the understanding of social phenomena is sustained by the knowledge extracted from the miners of big data across the various social dimensions by using data mining, machine learning and AI technologies. The worrying side of the story is that this data also describes in detail personal or even sensitive aspects of our lives, thus, privacy leaks, security threats or discriminatory events can occur when Big Data are processed. We are particularly aware of ethical issues related to the processing of personal data, and we were precursors in the ethical management of personal data, especially in the ifelds of privacy, fairness, and explainabilty.
• TANGO - It takes two to tango: a synergistic ap- that we do not fully understand, and, even worse, deciproach to human-machine decision making (HE). sions that are likely to violate ethical principles. In 2018, the European Parliament introduced in the GDPR12 a 1.3. Organization of the paper set of clauses for automated decision-making in terms of a right of explanation for all individuals to obtain In the next section, we will describe, for the scientific “meaningful explanations of the logic involved” when themes related to Trustworthy AI, the work that has been automated decision-making takes place. Also, in 2019, carried out from all the people of the KDD Lab9, with the High-Level Expert Group on AI presented the ethics particular focus on the EU projects listed in Section 1.2. guidelines for trustworthy AI [1]. Despite divergent opinions among legals regarding 2. Research Activities these clauses, everybody agrees that the need for the implementation of such a principle is urgent and that Big Data analytics and AI are not necessarily enemies of it is a huge open scientific challenge. As a reaction to Ethics. Sometimes many practical and impactful services these practical and theoretical ethical issues, in the last based on Big Data analytics and machine learning (ML) years, we have witnessed the rise of a plethora of explacan be designed in such a way that the quality of results nation methods for black-box models [2, 3] both from can coexist with fairness, privacy protection, and trans- academia and from industries. eXplainable Artificial Inparency. The overall scientific objective regarding the telligence (XAI) emerged as a research field to investigate Responsible AI is to develop the scientific foundations for methods to create or complement AI models whose inTrustworthy AI. Trustworthy AI is high on both the polit- ternal logic is accessible and interpretable, thus making ical and scientific agenda for AI [ 1], and it is considered the decision-making process human-understandable, and one of the hallmarks of AI made in Europe. helping people make better decisions preserving (and ex
We especially focus on four ethical dimensions (Ex- panding) human autonomy. For this aim, we defined plainability, Fairness, Privacy, and Trustworthiness), and diferent research lines and we have been achieving the on the intertwining aspects that bound these dimensions. following results: In the following, we will briefly outline, for each dimension, our main goals, the ongoing activities, and the open challenges that we are going to face in the projects listed in Section 1.2. However, for all the activities regarding these topics, we also: (1) engage the scientific community, e.g., organizing inclusive events in diferent conference such as international workshops 10 or special issues on leading journals11, to foster collaborations and the crosscontamination; (2) develop focused solutions to target research problems with links to models and formalisms studied by the foundational themes; (3) generalize these solutions into reusable AI techniques and guidelines for standardized processes.
The impressive performance of AI systems in prediction, recommendation, and decision-making support is generally reached by adopting complex ML models that often “hide” the logic of their internal processes. As a consequence, such models are often referred to as “black-box” models. AI-based systems are likely to lead to decisions • We designed algorithms for the inference of local explanations for revealing the decision rationale for a specific case, developing novel algorithms such as: (a) an algorithm for local explanation that learns factual and counterfactual logic rules [4, 5]; (b) a set of methods that moves the generation of explanations into a latent space to produce exemplars and counter-exemplars for images [6, 7], for time-series [8, 9], and for text [10]; (c) approaches tailored to explaining decision of text classifiers. In particular, we designed an explainer that produces an explanation of a document classifier by generating new documents in its vicinity through word replacement, either by replacing words in the document with their synonyms, antonyms, hyponyms, hypernyms, and definitions. The approach preserves the structure of the original text as it generates synthetic text samples through the use of an ontology [11].
Moreover, we also developed an explainability algorithm for Transformer-based models finetuned on Natural Language Inference, Semantic Text Similarity, or Text Classification tasks [ 12].
The explanation is obtained by extracting a set of facts from the input data, subsuming it by abstraction, and generating a set of weighted triples as explanation; (d) we studied and advanced the state-of-the-art for counterfactual explanations 12https://ec.europa.eu/justice/smedataprotect/ both by design [13, 14] and post-hoc [15, 16]. • We explored languages for expressing explanations, in terms of both expressive logic rules (with statistical and causal interpretation) and models that capture the detailed data generation behind specific deep learning models. We designed a framework that composes rules representing local explanations into a global explanation by merging theories, a form of logical meta-reasoning [17]. • We explored the opportunity of creating a co-design methodology to develop a humancentered, explainable AI system for decision support [18]. Specifically, we designed a prototypetest-redesign loop involving healthcare providers as end-users, which we then used to refine an explanation algorithm and its user interface. We ifrst presented the XAI technique’s conception based on the patients data and healthcare application requirements. Then, we developed the initial prototype of the explanation user interface, and perform a user study to test its perceived trustworthiness and collect healthcare providers’ feedback. We finally exploit the users’ feedback to codesign a more human-centered XAI user interface taking into account cognitive design principles such as progressive disclosure of information. • We investigated unexplored aspects such as the explanation of complex models like Siamese Networks used in few-shot and zero -shot learning [19], and the exploitation of causal discovery to improve the explainers efectiveness [20]. • We endorsed the creation of a common ground for researchers working on explanation from diferent domains, we developed a platform consisting of two parts: (i) a software library that integrates a wide set of explanation methods; (ii) a dedicated visual interface to let the user to interact with the explanation. We survey Explanation methods focusing on benchmarking [21]. The main goal of the scientific research on privacy is to design privacy-preserving solutions that guarantee to achieve both privacy protection (i.e., not revealing any personal or sensitive information about individuals or companies whose data are referring to) and utility of the data-driven services. We have been achieving interesting results in the following research lines: • Privacy-by-design paradigm. We explored the potentiality of privacy-by-design paradigm in designing and developing technological frameworks to counter the threats of undesirable efects of privacy violation, without obstructing the knowledge discovery opportunities of big data analytical technologies, by inscribing privacy protection into the data processing by design. We applied this principle in diferent applications, such as mobility data analytics [22] or call activities [23]. • Privacy Risk Assessment. We developed methodologies for systematically evaluating the risk of re-identification of all the individuals in a certain dataset [24]. This framework can be applied when it is not totally clear what kind of information is owned by a malicious third party, and it has been tested in diferent settings with diferent kinds of data, such as mobility data [24], retail data [25] and psychometric profiles [ 26]. We also proposed an adversarial model based on this framework and developed an optimization algorithm tailored for human mobility data to determine the most damaging adversarial behavior w.r.t. the privacy of the individuals in the data [27]. We also studied the privacy risk of federated leaning systems and we defined a new approach aiming to reduce by generalization the assessed risks [28]. • Privacy Risk Prediction. We extended the previous framework, allowing to obtain a prediction of the privacy risk for previously unseen individuals, i.e., not belonging to the starting dataset [29].
In this field, we aim to push forward the research in Even though privacy is one of the first human rights both directions: (a) developing post-hoc explanations that has been considered in legal frameworks, and a lot that given a black-box model aims to reconstruct its logic, of work has been done in the scientific literature, there is and (b) drive towards explainable-by-design models. This still the need to investigate new methodologies and apmust be done having the performance and interpretability proaches for: (a) defining formally and detecting automattrade-of in mind, trying to achieve both by following a ically privacy risks raised by AI systems handling diferhuman-centered methodology to produce explanations ent kinds of personal data; (b) designing data anonymizasuitable to the cognitive skills of their users. tion algorithms that are robust to sophisticated attacks; Finally, we need to explore what is still missing: for (c) designing AI algorithms that respect by-design privacy example, a formalism for explanations, and standards constraints also in distributed scenarios, where individuand metrics to quantify the grade of comprehensibility of als can cooperate for a common learning goal but with an explanation for humans. These standards need to take diferent privacy requirements; (d) investigating existing into account the research results from the HCI, DataVis, measures (or creating new ones) to evaluate the privacy and Cognitive Sciences communities. risk of novel or unusual kinds of data, especially with respect to the interplay with other ethical dimensions. 2.3. Fairness, equity and justice by-design bounds between explainability and fairness [50] and between privacy and explainability [51, 52]. Concerning AI models’ outputs can be biased against specific individ- the latter one, we studied both: i) how XAI techniques uals or groups [30]. The most relevant efect of such a help individuals to acquire awareness on their potential bias is unfairness or even illegal discrimination against privacy risks providing them with insights on which beprotected-by-law social groups [31]. Equity requires that havior contributes most [51, 53]; ii) how transparency people are treated according to their needs, which does may jeopardize the privacy risks of individuals reprenot mean all people are treated equally [32]. Justice is the sented in data used for training ML models [52]. “fair and equitable treatment of all individuals under the We also developed an ethico-legal framework for relaw" [33]. Fair AI models are designed to prevent biased sponsible data science [54] and we promoted general decisions in algorithmic decision making. Quantitative aspects such as digital ecosystem of trust [55]. definitions of fairness have been introduced in philoso- In addition, a goal that we would like to pursue is phy, economics, and machine learning in the last 50 years also to build Reference Datasets, which may boost both [34], with more than 20 diferent definitions of fairness research along the dimensions above and their combinaappeared thus far in the computer science literature [35]. tions, and may allow to compare the various proposed We contributed to the definition of: solutions among them acting as benchmarks, together with evaluation criteria, to assess whether proposed AI systems are Trustworthy or not.
2.5. The social dimension of AI • Group fairness metrics, measuring the statistical diference in distributions of decisions across social groups, with the pioneering works [36]. • Individual fairness metrics, binding the distance among the decision space and the feature space describing people’s characteristics [37]. • Causal fairness metrics, which exploit knowledge beyond observational data to infer causal relations between features and decisions, and to estimate interventional consequences [38].
mans interact with various forms of AI systems, including assistants and recommenders (AIs), amplifies the possibility for the emergence of large-scale social behaviour, possibly with unintended negative consequences. While AIs may generate individually “good” suggestions, the Based on these metrics, methods and tools have been sum of many suggestions can have unintended outcomes proposed for: because users’ choices, influenced by these suggestions, interfere with each other on top of shared resources. For • Bias detection (discrimination discovery or fair- example, GPS navigation systems suggest directions that ness testing), including our approaches [36, 39, 40] make sense from an individual perspective but may create and case studies [41, 42]. chaos if too many drivers are directed on the same route; • Dataset de-biasing and data processing (pre- and personalised recommendations on social media often processing approaches), including our work con- make sense to the user but may artificially amplify echo necting fairness and privacy [43]. chambers, filter bubbles, and radicalisation. This hap• Training or correcting AI models and representa- pens because AIs are based on ML models, generating a tions through fair algorithms (in-processing and feedback loop: users’ preferences determine the training post-processing approaches), including [44]. datasets on which AIs are trained; the trained AIs then • Monitoring models’ decisions [45, 46]. exert a new influence on users’ subsequent preferences, influencing the next round of training, and so on.
As with other quality objectives, the choice of a fairness As an example, we conducted a study in Milan to invesmetric is crucial for optimizing and for auditing AI mod- tigate the impact of navigation systems on the urban enels [47]. We have recently supported the critiques to vironment in terms of CO2 emissions [56]. We simulated the hegemonic theory of fairness [48], which reduces the behavior of vehicles by assuming that they would the problem to a numeric optimization of some metrics either follow a commercial navigation system or deviate [49]. Pathways for research include, in our view, multi- randomly from the fastest route. The results showed that stakeholders participatory design, integration with other blindly following the recommendations of a navigation trustworthy tools for AI, notably explanation methods, app can lead to trafic congestion in some areas of the city, and the option to reject unfair AI outcomes. resulting in increased travel time and global emissions. We also observed that adding controlled randomness to 2.4. Trustworthy AI as a whole suggested routes results in a better distribution of trafic on the road network, leading to decreased travel time We studied potential tensions but also synergies among and emissions, without penalizing individual travellers. these ethical dimensions. We have been exploring the The study suggests the need for developing routing algorithms that prioritize route diversification, as conforming to a limited set of routes can diminish diversity in drivers’ behavior and lead to ineficient road network use.
Understanding the impact of AIs on STS is emerging as another challenging dimension of trustworthy AI, which may enable the unprecedented opportunity to intervene on such STS to proactively help achieve important agreed goals with a better balance of individual and collective interests. However, achieving such a broader understanding requires a change of perspective that embraces complexity science and the trans-disciplinary integration of network science, AI, and computational social science.
gram under the scheme “INFRAIA-01-2018-2019 – Integrating Activities for Advanced Communities”, G.A. n.871042 “SoBigData++: European Integrated Infrastructure for Social Mining and Big Data Analytics”, by the PNRR - M4C2 - Investimento 1.3, Partenariato Esteso PE00000013 - "FAIR - Future Artificial Intelligence Research" - Spoke 1 "Human-centered AI", funded by the European Commission under the NextGeneration EU programme and by the EU – NextGenerationEU – National Recovery and Resilience Plan – Project: “SoBigData.it – Strengthening the Italian RI for Social Mining and Big Data Analytics” – Prot. IR0000013 – Avviso n. 3264 del 28/12/2021. D. Pedreschi, Privacy-by-design in big data analyt- aware patterns, Data Min. Knowl. Discov. 29 (2015). ics and social mining, EPJ Data Science 10 (2014). [41] M. M. Manerba, R. Guidotti, L. C. Passaro, S. Rug[23] F. Pratesi, L. Gabrielli, P. Cintia, A. Monreale, F. Gi- gieri, Bias discovery within human raters: A annotti, Primule: Privacy risk mitigation for user case study of the jigsaw dataset, in: NLPerspecprofiles, Data & Knowledge Engineering 125 (2020). tives@LREC, 2022. [24] F. Pratesi, A. Monreale, R. Trasarti, F. Giannotti, [42] A. Romei, S. Ruggieri, F. Turini, Discrimination D. Pedreschi, T. Yanagihara, PRUDEnce: a system discovery in scientific project evaluation: A case for assessing privacy risk vs utility in data sharing study, Expert Syst. Appl. 40 (2013). ecosystems, Transactions on Data Privacy 11 (2018). [43] S. Ruggieri, Using t-closeness anonymity to control [25] R. Pellungrini, A. Monreale, R. Guidotti, Privacy for non-discrimination, Trans. Data Priv. 7 (2014). risk for individual basket patterns, in: ECML PKDD [44] D. Pedreschi, S. Ruggieri, F. Turini, Measuring disWorkshops, 2018. crimination in socially-sensitive decision records, [26] G. Mariani, A. Monreale, F. Naretto, Privacy risk in: SDM, SIAM, 2009.
assessment of individual psychometric profiles, in: [45] K. Kenthapadi, H. Lakkaraju, P. Natarajan, DS, 2021. M. Sameki, Model monitoring in practice: Lessons [27] R. Pellungrini, L. Pappalardo, F. Simini, A. Mon- learned and open challenges, in: KDD, ACM, 2022. reale, Modeling adversarial behavior against mo- [46] M. Fontana, F. Naretto, A. Monreale, F. Giannotti, bility data privacy, IEEE TITS 23 (2022). Monitoring fairness in HOLDA, in: HHAI, volume [28] M. Fontana, F. Naretto, A. Monreale, A new ap- 354 of Frontiers in Artificial Intelligence and Appliproach for cross-silo federated learning and its pri- cations, IOS Press, 2022, pp. 246–248. vacy risks, in: PST, IEEE, 2021. [47] D. Pedreschi, S. Ruggieri, F. Turini, A study of top-k [29] R. Pellungrini, L. Pappalardo, F. Pratesi, A. Mon- measures for discrimination discovery, in: SAC, reale, A data mining approach to assess privacy ACM, 2012.
risk in human mobility data, ACM TIST 9 (2017). [48] L. Weinberg, Rethinking fairness: An interdisci[30] E. Ntoutsi, P. Fafalios, U. Gadiraju, V. Iosifidis, plinary survey of critiques of hegemonic ML fairW. Nejdl, M.-E. Vidal, S. Ruggieri, F. Turini, et al., ness approaches, J. Artif. Intell. Res. 74 (2022). Bias in data-driven artificial intelligence systems — [49] S. Ruggieri, J. M. Alvarez, A. Pugnana, L. State, an introductory survey, WIREs Data Mining and F. Turini., Can we trust fair-AI?, in: AAAI ConferKnowledge Discovery 10 (2020). ence on Artificial Intelligence, 2023. [31] A. Altman, Discrimination, in: E. N. Zalta (Ed.), [50] C. Panigutti, A. Perotti, A. Panisson, P. Bajardi,
The Stanford Encyclopedia of Philosophy, 2020. D. Pedreschi, Fairlens: Auditing black-box clinical [32] M. Minow, Equality vs. equity, American Journal decision support systems, Information Processing of Law and Equality 1 (2021). & Management 58 (2021). [33] J. Lehman, S. Phelps, et al., West’s encyclopedia of [51] F. Naretto, R. Pellungrini, A. Monreale, F. M. Nar
American law, Thomson/Gale, 2004. dini, M. Musolesi, Predicting and explaining privacy [34] B. Hutchinson, M. Mitchell, 50 years of test risk exposure in mobility data, in: DS, 2020. (un)fairness: Lessons for machine learning, in: FAT, [52] F. Naretto, A. Monreale, F. Giannotti, Privacy risk of ACM, 2019. global explainers, in: HHAI, volume 354 of Frontiers [35] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, in Artificial Intelligence and Applications , 2022.
A. Galstyan, A survey on bias and fairness in ma- [53] F. Naretto, R. Pellungrini, F. M. Nardini, F. Giannotti, chine learning, ACM Comput. Surv. 54 (2021). Prediction and explanation of privacy risk on mo[36] S. Ruggieri, D. Pedreschi, F. Turini, Data mining bility data with neural networks, in: PKDD/ECML for discrimination discovery, ACM Trans. Knowl. Workshops, volume 1323, Springer, 2020.
Discov. Data 4 (2010). [54] N. Forgó, S. Hänold, J. van den Hoven, T. Krügel, [37] B. T. Luong, S. Ruggieri, F. Turini, k-nn as an im- I. Lishchuk, R. Mahieu, A. Monreale, D. Pedreschi, plementation of situation testing for discrimination F. Pratesi, D. van Putten, An ethico-legal framework discovery and prevention, in: KDD, ACM, 2011. for social data science, JDSA (2020). [38] B. Qureshi, F. Kamiran, A. Karim, S. Ruggieri, D. Pe- [55] J. van den Hoven, G. Comandé, S. Ruggieri, dreschi, Causal inference for social discrimination J. Domingo-Ferrer, F. Musiani, F. Giannotti, reasoning, J. Intell. Inf. Syst. 54 (2020). F. Pratesi, M. Stauch, I. Lishchuk, Towards a dig[39] S. Ruggieri, S. Hajian, F. Kamiran, X. Zhang, Anti- ital ecosystem of trust: Ethical, legal and societal discrimination analysis using privacy attack strate- implications, Opinio Juris in Comparatione (2021). gies, in: ECML/PKDD, 2014. [56] G. Cornacchia, M. Böhm, G. Mauro, M. Nanni, D. Pe[40] S. Hajian, J. Domingo-Ferrer, A. Monreale, D. Pe- dreschi, L. Pappalardo, How routing strategies imdreschi, F. Giannotti, Discrimination- and privacy- pact urban emissions, 2022.