The growing integration of artificial intelligence (AI) in diverse sectors underscores the need for comprehensive and standardized approaches to ensure AI responsibility. However, the absence of a holistic framework to evaluate the fairness, privacy-preserving, secure, explainable, and human-centered facets of AI systems poses a challenge. Addressing this gap, this research paper presents a novel approach to assessing Responsible AI by combining insights from a systematic literature review with a practical evaluation framework. The paper provides a concise overview of the key aspects of Responsible AI and highlights the findings from the literature review. Furthermore, the paper introduces a set of evaluation metrics specifically designed for the current state of the art, using diferent model types and data from the healthcare domain. The framework supports the evaluation of Natural Language Processing (NLP), Computer Vision (CV), and tabular data models for classification tasks. Additionally, the paper briefly demonstrates VERIFAI, an example implementation of the framework, which serves as a comprehensive tool for assessing the responsibility of AI systems. The overall objective of this research is to make a meaningful contribution to the Responsible AI discourse, providing researchers and practitioners with a valuable resource to enhance the overall responsibility of their AI systems.
CEUR ceur-ws.org Framework
The rapid progress and extensive integration of artificial intelligence (AI) systems across diverse sectors have heightened concerns regarding their security, explainability, privacy, and fairness. Moreover, AI is becoming increasingly ingrained in daily life, leading to discussions about the roles of technologies like ChatGPT as artificial generators of text, code, and more. Ensuring Responsible AI (RAI for short) practices is crucial to maintaining trust in these systems and mitigating potential negative consequences. The relevance of RAI is not limited to technological domains. In the humanities, scholars in history, literature, and philosophy increasingly rely on AI tools for research. Beyond academia, industries like healthcare, finance, and automotive also emphasize RAI to ensure the privacy, safety, fairness, and explainability of AI-driven diagnostics, ifnancial predictions, and autonomous driving systems, respectively.
In an efort to systematize the approach to RAI and address a noticeable gap in the literature, this
CEUR
Workshop
Proceedings
paper introduces a comprehensive framework for the evaluation of AI systems. This framework
incorporates insights from our prior research in [
Our initial framework, VERIFAI [
The goal is to bridge the gap between theoretical understanding and practical implementation of RAI principles by providing a solid foundation for further development and refinement. The paper is organized as follows: Section 2 summarizes the literature review findings on RAI aspects, ofering a solid foundation for developing the evaluation framework. Section 3 introduces the metrics and evaluation process for specific model types and tasks. Section 4 briefly delves into the case studies to demonstrate the practical application of the framework on Computer Vision and Natural Language Processing (NLP) models as well as tabular models based on three exemplary data sets. Finally, Section 5 concludes the paper and outlines future research directions.
The insights presented in this section are drawn from our foundational work on Responsible
AI (RAI) as detailed in [
Trustworthiness Trustworthiness in AI is predominantly based on user perceptions of system reliability. Key factors include prioritizing data protection, providing accurate predictions under varying conditions, and ensuring transparent explanations. Furthermore, designing systems to be user-centric and adhering to intended application behavior is essential to foster a sense of utility and fairness.
Ethics Fairness stands out as a crucial ethical requirement for AI. It is of high importance for AI systems to operate without bias and discrimination. Other significant aspects include transparency in decision justifications, alignment with Sustainable Development Goals, and strict compliance with legal standards.
Privacy Ensuring privacy, especially when handling sensitive data, is of paramount importance. Compliance with standards such as the General Data Protection Regulation (GDPR) is vital. Methods like Federated Learning, which aims to decentralize data processing, ofer potential solutions, but they need to align with organizational strategies for robust data protection. Security threats in machine learning include stealing the model or sensitive user information, reconstruction attacks, and membership inference attacks, with the latter being a rapidly evolving research branch.
Security Several security threats exist within the branch of AI that warrant attention. These threats encompass poisoning attacks, where the training data is manipulated to undermine the model’s performance, and adversarial attacks, in which adversaries craft malicious input samples to deceive the model and induce incorrect predictions. Addressing these concerns is an active area of research, and countermeasures are continually being developed. Explainability The inherent opaqueness of some AI models has necessitated the development of Explainable AI (XAI). A human-centered approach to XAI emphasizes tailoring explanations to cater to diverse kinds of users. This not only ensures transparency but also empowers users in their decision-making. An intuitive user interface and visually understandable language enhance comprehension and engagement. Explainability serves as both a functional and non-functional requirement, emphasizing the AI system’s inner workings and efective communication. Human-centeredness Prioritizing human-centeredness is fundamental in RAI. The integration of human feedback and inputs, as observed in the Human-in-the-loop (HITL) approach, ensures that AI technologies align with human values while remaining eficient and user-friendly. This approach ensures that AI technologies are not solely reliant on algorithms but also benefit from human knowledge, experience, and intuition, allowing AI systems to better align with human values, expectations, and ethical considerations.
Interdependence of RAI Aspects The diferent aspects of RAI are closely interrelated. While ethical considerations are pivotal, they must coexist with technical requirements like security and explainability. The holistic approach encompasses both system-side and developer-side perspectives, ensuring a comprehensive RAI framework. Explainability methods must also respect privacy and security, as they are interconnected. Human-centered AI and the HITL approach play crucial roles by including human expertise and perspective. As a dynamic and interdisciplinary process, RAI requires attention and care throughout the entire system lifecycle. In the subsequent sections, we will delve deeper into the metrics of our RAI framework, building on these foundational insights.
In the following, we focus on specific sub-aspects (see second row of Fig. 1), which were selected, because they were identified as most important in the findings of our systematic literature review: The quantitative evaluation of Explainability, Fairness, Adversarial Robustness, and Privacy Leakage. In future work, we will expand this set of metrics.
In the first iteration of our framework, our attention is centered on models for classification problems across diferent domain types, such as language, image, or tabular data. As illustrated in Figure 1, these are grouped into four main aspects. The sub-aspects correspond to those identified as most important in the literature analysis, and below them are the metrics already implemented in our framework.
The metrics of Monotonicity[
As visual explanations for Neural Networks alone are often insuficient and to provide a
comprehensive evaluation of explainability, four metrics were chosen, which measure explanations
from diferent perspectives: The Robustness of the explanation was measured using Max
Sensitivity[
The Fairness metrics encompass various aspects, such as Group Fairness (e.g. Statistical Parity
Diference[
The Robustness metrics of the framework were carefully chosen to provide a comprehensive
security assessment. For tabular models, the Zeroth Order Optimization Attack (ZOA)[
For evaluating Privacy Leakage we tested them using several Membership Inference Attack (MIA) Approaches for each model. The Black-Box-MIA[21] metric was selected for the Tabular model, while for the neural networks (NLP and CV) we employed the metrics of MIA via Shadow Models[22] and MIA via Population Data[22]. These metrics enable the assessment of potential privacy risks in the models by quantifying if some data points of the training data can be reconstructed.
In the current development stage, we compute the Responsibility Score by averaging the fairness, privacy-leakage, robustness, and explainability scores, which are calculated using the averages or worst-case outcomes of their respective metrics. This method assumes an equal contribution from each category to the model’s overall score. The resulting Responsibility Score is then expressed as a percentage, reflecting the model’s responsibility level according to the assessed metrics.
This section presents a part of the current implementation of the framework. Our use cases are designed to assess Responsible AI principles and specifically focus on concerns within the healthcare domain. We selected three use cases to demonstrate the framework’s applicability across diferent tasks and data types: Detecting Skin Cancer, Recommending Medicines based on Sentiment Analysis, and Detecting Heart Diseases.
To train the models for these use cases, we used datasets from tabular data (Heart Disease Dataset[23]), image data (HAM10000 dataset[24]), and text data (Drug Review Dataset[25]). We chose representative models for each task: Random Forest[26] for tabular data, Xception[27] for image classification, and DistilBERT[ 28] for text classification. These models were selected based on their compatibility with evaluation metrics used in the framework, ensuring a comprehensive evaluation of the system’s efectiveness.
To give an insight into the application by showing the evaluation of adversarial robustness of the three diferent models mentioned in section 4 we want to provide a few screenshots of the application in the following: (a) Evaluation of the CV- model. The metric result (b) Evaluation of the NLP- model (without page and explanation are shown in the page header. header) (c) Adversarial examples on tabular data with diference vectors (without
page header)
The evaluation of our CV-model for detecting skin cancer is depicted in Figure 2a. The image illustrates our process of testing the model’s robustness by applying various adversarial attacks, including four previously mentioned algorithms, with the aim of deceiving the model and inducing erroneous predictions. This process involves introducing perturbations to the input images and measuring the model’s accuracy after each round of increasing disturbance. We find that our model has a robustness score of 0.53, indicating its vulnerability to adversarial attacks. This susceptibility means that the model can be deceived with minimal pixel changes, resulting in an ’Accuracy under Attack’ of 52% using the FGSM Algorithm, thus deriving a privacy score of 5 out of 10.
Our Natural Language Processing model, as referenced in Figure 2b, is not immune to such attacks either. The adverse consequences of deceiving this model can be severe, such as the potential for incorrect medication recommendations. We utilize four distinct algorithms, as mentioned in 3, that modify the input text to induce inaccuracies.
The table indicates the following insights: The attack was performed on 200 examples, of which the model initially misclassified 8. This resulted in an original accuracy of 92.0%. We subsequently ran the adversarial attack process on the remaining examples to derive a valid adversarial perturbation for each. 16 of these attacks failed, yielding a success rate of 82.61%. This translates to the model correctly predicting and resisting attacks on 16 out of the total 200 samples, thereby achieving an accuracy under attack of 16.0%. For the successful attacks, on average, 10.69% of words were altered to achieve a prediction change.
The model’s robustness score is computed by considering the after-attack accuracy, the inverse of the adversarial attack success rate, and the degree of input modification required for a successful attack. A high robustness score corresponds to a model that resists adversarial attacks efectively, maintains high accuracy, and demands significant alterations by the adversary to be deceived. Unfortunately, in our case, the model had a robustness score of just 2 out of 10 using the most successful attacks, suggesting a need for mitigation strategies.
Next, we attempt to assess the tabular model. Figure 2c provides a visualization of the adversarial examples, enabling a deeper understanding of how the attacks influence the model’s decisionmaking. The image illustrates features such as age versus maximum heart rate, using black lines to denote the diference vectors between the original and manipulated data points. These visual cues help to interpret the extent of data manipulation by adversarial attacks. Despite these vulnerabilities, the model shows considerable overall robustness, scoring an encouraging 8 out of 10 (or 80%). More comprehensive results, including those on privacy leakage, fairness, and explainability, have been omitted due to paper length constraints but are available in our demo version on our project website1.
In conclusion, determining the level of responsibility in AI systems is a complex task that goes beyond the realm of computational metrics alone. It involves subjective considerations and requires the incorporation of human feedback and human-centered approaches. By taking into account the input and perspectives of the target audience, we can gain a more comprehensive understanding of what responsible AI entails. While our presented framework addresses various aspects of Responsible AI based on a systematic literature review, we acknowledge that it is a work in progress and has certain limitations. We recognize the need for further refinements and expansions to enhance its efectiveness. The current iteration of the framework serves as a solid foundation for evaluating Responsible AI. However, we are aware that there are specific model types, such as generative models like the GPT architecture, that may require additional attention and consideration. Additionally, as AI is applied across diferent domains, the framework will need to adapt and incorporate domain-specific considerations to ensure its relevance and applicability. As we continue to refine and expand our framework, we aim to address these challenges and further improve its capabilities to create a more comprehensive and robust approach to assessing the responsibility of AI systems. arXiv preprint arXiv:1412.6572 (2014). [15] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, A. Vladu, Towards deep learning models resistant to adversarial attacks, arXiv preprint arXiv:1706.06083 (2017). [16] S.-M. Moosavi-Dezfooli, A. Fawzi, P. Frossard, Deepfool: a simple and accurate method to fool deep neural networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2574–2582. [17] J. Li, S. Ji, T. Du, B. Li, T. Wang, Textbugger: Generating adversarial text against real-world applications, arXiv preprint arXiv:1812.05271 (2018). [18] J. Gao, J. Lanchantin, M. L. Sofa, Y. Qi, Black-box generation of adversarial text sequences to evade deep learning classifiers, in: 2018 IEEE Security and Privacy Workshops (SPW), IEEE, 2018, pp. 50–56. [19] D. Jin, Z. Jin, J. T. Zhou, P. Szolovits, Is bert really robust? a strong baseline for natural language attack on text classification and entailment, in: Proceedings of the AAAI conference on artificial intelligence, volume 34, 2020, pp. 8018–8025. [20] S. Ren, Y. Deng, K. He, W. Che, Generating natural language adversarial examples through probability weighted word saliency, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, 2019, pp. 1085–1097. URL: https://aclanthology.org/P19-1103. doi:10.18653/v1/P19- 1103. [21] R. Shokri, M. Stronati, C. Song, V. Shmatikov, Membership Inference Attacks Against Machine Learning Models, in: 2017 IEEE Symposium on Security and Privacy (SP), IEEE, San Jose, CA, USA, 2017, pp. 3–18. URL: http://ieeexplore.ieee.org/document/7958568/. doi:10.1109/SP.2017.41. [22] J. Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, R. Shokri, Enhanced Membership Inference Attacks against Machine Learning Models, 2022. URL: http://arxiv.org/abs/2111. 09679, arXiv:2111.09679 [cs, stat]. [23] A. Janosi, W. Steinbrunn, M. Pfisterer, R. Detrano, Uci machine learning repository-heart disease data set, School Inf. Comput. Sci., Univ. California, Irvine, CA, USA (1988). [24] P. Tschandl, C. Rosendahl, H. Kittler, The ham10000 dataset, a large collection of multisource dermatoscopic images of common pigmented skin lesions, Scientific data 5 (2018) 1–9. [25] F. Gräßer, S. Kallumadi, H. Malberg, S. Zaunseder, Aspect-based sentiment analysis of drug reviews applying cross-domain and cross-data learning, in: Proceedings of the 2018 International Conference on Digital Health, DH ’18, Association for Computing Machinery, New York, NY, USA, 2018, p. 121–125. URL: https://doi.org/10.1145/3194658. 3194677. doi:10.1145/3194658.3194677. [26] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., Scikit-learn: Machine learning in python, Journal of machine learning research 12 (2011) 2825–2830. [27] F. Chollet, Xception: Deep learning with depthwise separable convolutions, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. [28] V. Sanh, L. Debut, J. Chaumond, T. Wolf, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, arXiv preprint arXiv:1910.01108 (2019).