A Framework Quantifying Trustworthiness of Supervised Machine and Deep Learning Models

Abstract

Trusting Artificial Intelligence (AI) is controversial since models and predictions might not be fair, understandable by humans, robust against adversaries, or trained appropriately. Existing toolkits help data scientists to create fair, explainable, robust, and transparent Machine and Deep Learning (ML/DL) models. However, tools to quantify AI trustworthiness according to pillars and metrics relevant for heterogeneous scenarios are still missing. This work proposes a novel algorithm that quantifies the trustworthiness level of supervised ML/DL models according to their fairness, explainability, robustness, and accountability. The algorithm is deployed on a Web application to allow the general public to calculate the trustworthiness of their models. Finally, a validation scenario with models classifying cyberattacks demonstrates the applicability of the Web application and algorithm.

Keywords

• eol>Trust
• Supervised Machine Learning
• Trust Framework
• Deep Learning Trust

1. Introduction

accountability as pillars for trusted AI [ 2 ]. Concerning fairness, bias is one of the main issues against trusting AI Artificial Intelligence (AI) made great strides over the last systems [ 3 ]. Bias can be introduced by human prejudice decade [ 1 ]. Some relevant examples are the victory of in the training dataset or assumptions made during the IBM Watson in the open-domain Q&A game of Jeopardy process of training Machine and Deep Learning (ML/DL) (2011), the first version of Tesla Autopilot, with lane con- models. Furthermore, too small, incomplete, or not ditrol, breaking, and speed limit adjustment (2014), or when verse enough training datasets might introduce bias as AlphaGo defeated world champion Lee Sedol in the Go well. The literature has proposed diferent solutions to degame (2016). In parallel to these achievements, AI has also tect and avoid bias. For instance, IBM AI Fairness 360 [ 4 ] gained relevance as support to human decision-making, contains a set of metrics to detect bias in ML/DL models spanning from trivial to highly complex applications. and datasets. Algorithms to mitigate bias during the preThe diagnosis and treatment of diseases, assessment of processing, in-processing, and post-processing stages are legal issues, or admission to credits are good examples of also available in the IBM toolkit. Another de-biasing apcurrent tasks supported by AI systems. In these scenar- proach is called Fairness GAN [ 5 ], which creates new and ios and many others, delegating partially or entirely the unbiased datasets from original biased ones. However, decision-making process to automated and intelligent aspects dealing with the explainability, robustness, or systems generates a trust dependency. Thus, it is critical accountability of models are not taken into account by to have mechanisms to quantify the trustworthiness level these tools. of AI systems and their predictions. Explainability is another important pillar that consists

Recently, the research community has agreed on the of understanding how ML/DL models come to their conimportance of fairness, explainability, robustness, and clusions. When AI is used in high-risk fields, all stakeholders should understand the main decision drivers. For SafeAI2023: The AAAI’s Workshop on Artificial Intelligence Safety, human decision-making, trust can be gained by explain*FeCborruraersypo1n3–d1in4g,2a0u2t3h,oWr.ashington, D.C. ing the underlying rationale. However, for AI, aspects $ [e-mail] (A. Huertas Celdran); [e-mail] such as the algorithm class, features importance, or model (J. Kreischer); [e-mail] (M. Demirci); complexity are some of the key aspects to explain [e-mail] (J. Leupp); [e-mail] tions [ 6 ]. In this context, the IBM AI Explainability 360 (P. M. Sanchez Sanchez); [e-mail] (M. F. Franco); toolkit [ 7 ] contains a pool of algorithms and methods to (gGer.oMmaer.tbinovezetP@earerzm);asstuiillsesre@.c hifi.u(Gzh. .cBhov(Bet.);Sgtirlleegro)[e-mail] explain model details and predictions. This toolkit also 0000-0001-7125-1710 (A. Huertas Celdran); 0000-0002-6444-2102 includes two metrics to evaluate the goodness of expla(P. M. Sanchez Sanchez); 0000-0002-0208-0521 (M. F. Franco); nations. In addition, the literature has created various 0000-0002-4534-3483 (G. Bovet); 0000-0001-5532-6604 (G. Martinez libraries implementing local explanations methods like Perez); 0000-0002-7461-7463 (B. Stiller)

© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License LIME or SHAPE [ 8 ]. Despite the contributions of these CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) solutions they only focus on explainability and do not consider other important aspects of trusted AI like the • A Web application [ 15 ] that allows stakeholders robustness of ML models. to use the proposed algorithm to compute and

In this sense, robustness is the third pillar of trusted compare the trustworthiness level of their superAI and refers to the model ability to deal with adver- vised ML/DL models. sarial samples. Even in black-box approaches, where • A case study focused on models classifying malmodel details are unknown to adversaries, it is possi- ware afecting Internet of Things (IoT) devices to ble to cause unexpected predictions using adversarial at- demonstrate the suitability of the developed Web tacks. Therefore, to ensure trusted ML/DL models, their application and proposed algorithm. predictions must be stable and robust, even when adversaries are present [ 9 ]. In this context, [ 10 ] proposes The remainder of this paper is structured as follows. a comprehensive taxonomy for adversarial robustness Section 2 analyzes previous works in the area of trusted and discusses potential consequences of attacks consid- AI. Section 3 presents the pillars and metrics relevant for ering data integrity, confidentiality, and privacy. From trusted AI. Section 4 introduces the design and implea diferent perspective, the IBM Adversarial Robustness mentation of the trusted AI algorithm. While Section 5 Toolbox (ART) [ 11 ] ofers a collection of adversarial at- presents the functionality of the Web application, Sectack/defense implementations, run-time attack detection tion 6 validates the algorithm in a cybersecurity scenario. methods, poisoning detection, and robustness metrics. Finally, Section 7 draws some conclusions and future These metrics are very relevant to calculate the trustwor- work. thiness level of AI models, but others dealing with the methodology followed to train models are missing.

To measure the quality of the ML pipeline or method- 2. Related Work ology used to train ML/DL models, the fourth main pillar First, it is important to mention that trust in AI is an of trusted AI proposes to use accountability and trans- incipient field, and, to the best of our knowledge, there is parency aspects. More in detail, train/test splitting strat- no solution automatically assessing the trustworthiness egy, data pre-processing, normalization, or feature extrac- level of ML/DL models in multiple dimensions. Howtion and selection are some aspects providing valuable ever, related work has proposed metrics and tools for insights to trust AI systems. In this sense, [ 12 ] proposes individual dimensions or pillars impacting model trustthe creation of FactSheets as a form of AI documentation. worthiness. Therefore, this section reviews work dealing Additionally, [ 13 ] studies the documentation needed by with AI trustworthiness in terms of fairness, explainabilstakeholders to trust AI. Finally, the IBM AI FactSheets ity, robustness, and accountability. 360 [ 4 ] provides a guide and tool for the manual prepara- Starting from fairness, the authors of [ 16 ] introduced tions of factSheets. the notions of group fairness and individual fairness and

Despite the valuable contributions of previous works, discussed how model developers could address the topic. trusted AI is an emerging research field that needs more In the same direction, [ 17 ] proposed an algorithm for efort. In particular, the state-of-the-art lacks a compre- fair classification that complied with the two previous hensive and unified collection of relevant metrics per notions of fairness. To mitigate unfairness in AI, [ 5 ] pillar to identify and quantify trusted ML/DL models. proposed a novel de-biasing approach, called Fairness Furthermore, existing solutions focus on detecting and GAN, capable of creating a new dataset that approximitigating diferent issues per pillar. However, there is mates a given original biased one. Reviewing existing no solution combining the pillars and computing a global tools and frameworks, IBM AI Fairness 360 toolkit [ 4 ] trustworthiness level of ML/DL models. Last but not contains a comprehensive set of fairness metrics such least, IBM toolkits are helping data scientists to create as statistical parity, equal opportunity, or average odds fair, explainable, robust, and transparent ML/DL mod- that can be used to detect bias in ML models and datasets. els. Nevertheless, tools to quantify AI trustworthiness in Furthermore, the toolkit ofers algorithms capable of mitiheterogeneous and real-world scenarios are still missing. gating bias during the pre-processing, in-processing, and

To address the previous challenges, this article presents post-processing stages. Facebook internal Fairness Flow the following contributions: toolkit [ 18 ] is another solution composed of a Python • A taxonomy with the four main pillars of trusted library that provides a simple API that requires a data AI (fairness, explainability, robustness, and ac- set of predictions, labels, and group membership. As countability) and their most relevant metrics to an output, the API ofers informative metrics, statistical quantify the trustworthiness level of supervised confidence, and how to interpret the results. Microsoft ML/DL models. Fairlearn [ 19 ] is another open-source project that com• An extensible, adaptive and parameterized algo- bines visualization capabilities with unfairness detection rithm (available in [ 14 ]) to quantify the trustwor- and mitigation algorithms. thiness level of supervised ML/DL models with Explainability is another important dimension for tabular data according to the pillars and metrics trusted AI. In this sense, IBM AI Explainability 360 of the proposed taxonomy. toolkit [ 7 ] contains several algorithms and methods to create explanations for ML models or make explainable models like decision trees more accurate. It also includes two proxy metrics to evaluate the goodness of explanations. Compared to the work at hand, this toolkit does not evaluate the level of model explainability but provides explanation methods for diferent types of models.

Additionally, various python libraries implemented local explanations methods like LIME or SHAPE [ 8 ]. All those solutions aim to explain the decisions of models but not to classify models on their explainability level, as the paper at hand does.

Robustness against adversarial attacks is something well-studied in the literature. The authors of [ 10 ] presented a comprehensive taxonomy with explanations for existing attacks and defenses. They also discussed the potential consequences of adversarial attacks in terms of data integrity, confidentiality, and privacy. However, the authors did not analyze metrics measuring the models robustness. In terms of metrics, the literature focuses mainly on intrinsic and post-hoc explainability methods. In terms of existing tools, IBM ART [ 11 ] is an opensource Python library that collects adversarial robustness techniques on ML models. It consists of adversarial attack/defense implementations, run-time attack detection methods, poisoning detection, and robustness metrics.

This toolbox implements multiple attack detection and defense techniques. Similar to the work at hand, ART also gathers possible robustness metrics.

Dealing with accountability, [ 12 ] proposed a methodology for creating factsheets. Factsheets provide information regarding ML models, such as training data, model type, or training methodology. All stakeholders involved in the AI lifecycle contribute to the Factsheet creation, and it covers the expertise gap between AI producers and consumers. The authors of [ 13 ] studied the necessities of developers and other stakeholders to understand what content to include in factsheets. They highlighted the importance of documenting how a model was structured, what training data was used, and how features were engineered. In terms of existing tools, IBM AI FactSheets 360 [ 4 ] provides a guide for preparing factsheets. Transparent and well-documented communication between model creator and model operator increases trust and enables more eficient integration of pre-trained models.

One limitation of the previous works is that the creation of factsheets requires a lot of manual efort. The paper at hand automates this process as much as possible.

In conclusion, despite the contributions made by the previous works, there is no automatic tool or solution combining diferent pillars and metrics to compute the trustworthiness level of supervised ML models. 3. Main Pillars & Metrics of

Trustworthy AI This section describes the pillars and metrics identified by this work as the most relevant to quantify the trustworthiness of supervised ML/DL models (see Figure 1). ML/DL models can be classified as unfair due to diferent reasons such as i) biased training data, ii) unbalanced or lack of training data, or iii) discrimination of protected groups, among others. The concept of protected group refers to a set of samples sharing a common property or feature value. For instance, in a job application, women could be a protected group to avoid gender discrimination. Considering these reasons, this work proposes the following metrics to quantify the fairness level of supervised ML/DL models.

• Underfitting : detects if the model is unable to learn the relationship between inputs and outputs accurately. In this work, Eq (1) calculates underfitting as the diference between the model accuracy with train data and a baseline threshold (established according to the expected accuracy of the selected application scenario).

(ˆ = 1| = 0)| = − (1) • Overfitting : measures the model generalization capabilities. In this work, Eq (2) calculates overfitting as the diference between the model accuracy with train and test datasets.

= −

(2) • Class Balance: measures the ratio of samples belonging to diferent classes in the training dataset.

Class imbalance is a problem for predicting tasks of infrequent classes with few samples. In this work, the class balance is computed by using Eq. (3) and chi-square (2) distribution. It performs a statistical test that checks the deviation between a perfect samples distribution () for all classes () and the actual distribution ().

2 = Σ ( − )2

• Statistical Parity Diference : computes the spread between the percentage of samples belonging to the majority group receiving a favorable outcome compared to a protected group [ 20 ]. This metric and the following three detect discrimination in predictions using samples of diferent groups.

Eq. (4) measures the Statistical Parity Diference (SPD) between both groups. ˆ = 1 represents a favorable prediction and (ˆ = 1| = ) is the probability of receiving a favorable prediction if the sample belongs to the protected minority ( = 1) or the unprotected majority ( = 0). If SPD is close to zero, the classifier has statistical parity.

(ˆ , , ) = | (ˆ = 1| = 1)− (3) (4)

Fairness

Explainability

Robustness

Accountability

Trusted AI Taxonomy Confidence Score

DT, RF, GBDT Clique Method Class Balance Statistical

Parity

Difference Legend Overfitting Underfitting Equal

Opportunity Average Odds

Difference Difference

Disparate Impact Model Size Correlated Features Algorithm Class Feature Relevance FactSheet

Normalization Missing Data Completness

Train/Test Regularization

Split NN, LG, SVM ER Fast Gradient

NN Pillars

Metrics

Algorithms

ER Carlini Wagner

ER DeepFool

Loss Sensitivity

CLEVER Score • Equal Opportunity Diference : measures the 3.2. Explainability Pillar spread between true positive rate (TPR) and false positive rate (FPR) of protected and unprotected Explainable artificial intelligence focuses on enabling a groups [ 21 ]. Eq. (5) calculates the Equal Opportu- deeper understanding of the inner workings of ML/DL models. To quantify the explainability of models and nity Diference (EOD). {ˆ = 1| = 0, = their predictions, the following metrics (which assume 0u}nparnodtecte{dˆan=d1p|rot=ect1e,dgr=ou0p}s,arreestpheecFtPivReloyf. tabular data format) are used.

Additionally, {ˆ = 1| = 0, = 1} and {ˆ = 1| = 1, = 1} are the TPR of unprotected and protected groups. (ˆ , ) = {ˆ = 1| = 1, = }− {ˆ = 1| = 0, = }, ∈ 0, 1

(5) • Average Odds Diference : calculates the mean absolute diference in TPR and FPR between protected and unprotected groups [ 22 ]. Eq. (6) measures the Average Odds Diference (AOD) as 1/2[( − ) + ( − )] (ˆ , , ) = Σ ∈{0,1}| {ˆ = 1| = 1, = } − {ˆ = 1| = 0, = }|

2 • Disparate Impact: measures the ratio of a protected group receiving a favorable prediction divided by the ratio of an unprotected group receiving a favorable outcome. Eq (7) calculates the Disparate Impact (DI) and uses the previously defined variables.

(ˆ , , ) = {ˆ = 1| = 1} {ˆ = 1| = 0} (6) (7) • Algorithm Class: indicates the model explainability degree according to the algorithm type and its complexity. This work defines a predefined and configurable score for each algorithm, which is inspired by the literature. • Correlated Features: measures the percentage of highly correlated features. High correlation among features might lead to biases in most explanation techniques. This work calculates the number of highly correlated features (>= 95%) and predefines configurable thresholds to compute the metric score. • Feature Relevance: calculates the percentage of irrelevant features for a set of predictions. The lower irrelevant features, the better, as they would only make explanations more complex without being relevant [ 23 ]. As in the previous metric, this work calculates the number of irrelevant features (relevance <= 3% ) for the model and predefines configurable thresholds to compute the metric score. • Model Size: indicates the number of parameters used by models. In this work, this metric considers predefined thresholds that are used to evaluate the model comprehensibility, which inversely correlates with the model degrees of freedom determined by the number of parameters. 3.3. Robustness Pillar To trust ML/DL models, their predictions should be stable and robust. If small changes in the input data cause significant deviations in the output, adversarial perturbations can be used to generate undesired outcomes. This work proposes using the following metrics to measure the robustness of supervised ML/DL models. (, , ) = 1 ∑︁ || () − || || ∈ ||||

(10) 3.4. Accountability Pillar Accountability fosters trust in AI by documenting, validating, and notifying the creation, evaluation, and main• Confidence Score : measures the probability of cor- tenance of ML/DL models. This work identifies the folrectly predicting samples. It calculates the stabil- lowing metrics as relevant to evaluating the accountability of predictions, the more stable the predictions, ity of supervised ML/DL models. The following metthe more robust the model [24]. Eq (8) calculates ric scores are calculated using predefined configurable the confidence score as the mean over all the pre- thresholds that can be found in [ 14 ]. cision scores for all thresholds. Where are the labels, and ℎ are the probabilities that one vector is classified correctly. • Train/Test Split: measures the ratio between the number of samples used for training and testing.

If the training and testing dataset do not contain enough representative data, the model will not perform well, it will not be able to generalize, and the predictions will not be reliable. • Missing Data: evaluates how missing values of features of the training dataset are handled. If the model is trained on a dataset containing many missing values, the model will not be reliable. • Normalization: evaluates if some models have been trained with normalized or non-normalized data. It helps mitigating the efects of outliers and ensures that features values are in the same range [29]. • Regularization: measures if the ML/DL model used generalization techniques during training. It avoids model parameters taking extreme values, and it is necessary to avoid memorization during training NN with millions of parameters. • FactSheet Completeness: measures if the FactSheet includes all necessary information that stakeholders need in order to trust the model and its predictions. A FactSheet summarizes important metadata regarding the steps followed to train a model, purpose, algorithm, and data, among others.

1 = | ℎ| Σ + +

(8) • Loss Sensitivity: calculates the largest variation of the output of a Neural Network (NN) under a small change in its input. Overall, it quantifies the smoothness of a model [24]. The smaller the variation in the output, the smoother the model.

Eq (9) calculates Loss sensitivity (g), where ℒ is the loss function.

= ⃦⃦⃦⃦ ℒ ⃦⃦⃦⃦ 1

(9) • Cross Lipschitz Extreme Value for Network Robustness (CLEVER) Score: measures the minimal perturbation that is needed to change the classification outcome [25] using the local Lipschitz constant [ 11 ]. It is applicable to NN. • Clique Method: finds the exact minimal adversarial perturbation or a guaranteed lower bound of it [26]. It is applicable to Decision Trees (DT), Random Forests (RF), and Gradient Boosted Decision

Trees (BGDT). • Empirical Robustness: measures the average minimal perturbation that needs to be introduced 4. Algorithm Quantifying to change the model prediction [ 11 ]. Empirical Trustworthy AI robustness (ER) is applicable to NN, Logistic Regression (LG), and Support Vector Machine (SVM) This section presents a novel, extensible, and algorithms. Eq. (10) calculates ER, where is a parametrized algorithm able to quantify the trusttrained classifier, is an untargeted attack, worthiness level of supervised ML/DL models. Figure 2 is the test data. First adversarial inputs () shows the algorithm schema and lifecycle. are crafted and the classifier is tested against First, for each pillar, the algorithm computes the metthem. In the equation only the adversarial in- rics explained in the previous section. For that, each puts which successfully fooled the model are be- metric receives as an input the i) training and testing ing considered. So only the indices ∈ 1, 2, datasets, ii) trained ML/DL model, and iii) FactSheet with where () ̸= ( ()) must be taken. Select- the metadata of the training methodology. Then, the aling attacks is a challenging task, and this work gorithm evaluates if the inputs fulfill the conditions of considers the success ratio and calculation speed each metric. If so, each metric is independently calcuof each attack to select Fast Gradient [ 9 ], Carlini lated according to its formula and input data. Table 1 & Wagner [27], and DeepFool [28] attacks. shows for each metric, its inputs, conditions, and output.

Input Input Inputs Inputs Inputs

Metric 1 Metric 2 MMeettrriicc 3n MMeettrriicc 4n MMeettrriicc nn

Output Output Outputs

Score 1 Score 2

Scores Outputs

Scores Outputs

Scores

Explainability

Score

Robustness

Score

Accountability

Score

Mapping FuPnilcl atiroAnggregation FuncGtiolonbsal Aggregation FunTcrutisotnScore

The metrics outputs cannot be interpreted as trust scores because they have diferent data types, scales, and meanings. Therefore, each metric output must be interpreted and translated into a standard trust score using a mapping function. The proposed trust score for all metrics ranges from one to five, where one corresponds to the worst score, and five represents the best score. The mappings from metrics outputs to trust scores are predefined according to good practices indicated in the literature.

However, this process could involve some arbitrary decisions adding biases. To avoid it, the mapping function is parameterized and can be fine-tuned by stakeholders according to the data domain, metric, or scenario.

The next step consists of aggregating all the metrics scores of each pillar and calculating a score per pillar. The algorithm proposes a weighted approach where each metric has particular importance in the pillar score. It is up to discuss whether all metrics are equally important and how weighted they should be. Because of that, default weights for every metric are defined, but stakeholders can modify them according to the scenario characteristics.

Finally, the four pillar scores are aggregated into a global trust score, which is the return value of the algorithm. Computing the global trust score is done analog to calculating the pillars scores. Independent weights are assigned to each pillar, and the global trust score is the weighted average of each pillar. Since the importance of each pillar depends on the scenario, the predefined configuration of the algorithm (equal importance per pillar) can be modified by stakeholders. Algorithm 1 shows the pseudocode implementing the previous steps of the proposed Trusted AI algorithm.

Algorithm 1 Trusted AI Algorithm 1: function Trusted_AI(, , , ℎ, , ℎ) 2: _ ← 0 3: ← , , , 4: _ ← 5: for do 6: ← 0 7: _ ← 8: ← _() 9: for do 10: ← [] 11: _[] ← _() 12: 13: 14: 15: 16: 17: 18: 19: ← ℎ[] ← + * _[] ←

for ( ∈ , ∈ ) _ do for ( ∈ , ∈ ) _ do ← ℎ[] _ ← _ + * return _ 5. Web Application This section presents a Web-based application hosting the proposed algorithm and allowing stakeholders to calculate the trustworthiness level of supervised ML/DL models [ 15 ] in an intuitive way. To understand the application functionality, it is important to introduce the concepts of scenario and solution. Firstly, a scenario is the application context where supervised ML/DL models are created to solve a particular task. Classifying malware in IoT devices, or fraud in credit cards are two examples of scenarios. Secondly, a solution is a combination of i) the training & testing data, ii) a trained supervised ML/DL model, and iii) a FactSheet solving a given scenario task.

Thus, the Web application allows stakeholders to: • Create scenarios and upload solutions for existing

scenarios. • Calculate and graphically see the trustworthiness

level of a solution. • Re-calculate the trustworthiness level of a given

solution according to customized parameters. • Compare the trustworthiness levels of two solutions, explaining the meaning and results of each metric and pillar.

The Python framework dash 2.1.0 was used to implement the Web application backend and frontend components. The backend implements the algorithm and a database to store the input data needed by the algorithm. The frontend provides stakeholders with a graphical interface composed of the following four pages. The Appendix contains one screenshot of each page. 5.1. Web Application Functionality The main functionality of the proposed Web application is organized in the following main pages: scenario, upload, analyze, and compare. Below, more details about them are given.

Scenario Page. This page allows stakeholders to create new scenarios with their descriptions.

Upload Page. Stakeholders use it to upload their solutions for a previously created scenario. In particular, the following aspects are needed: • Scenario & Solution. • Description: Brief description of the solution (op

tional). • Training & Testing data: Two datasets (format:

csv or pickle). • Protected Feature: Protected features of the Train

ing data (optional). • Protected values: Protected values for the pro

tected features (optional). • Target Column: Column of the training dataset to

be predicted. • ML/DL model: Supervised ML/DL model of the

solution (format: pickle). • FactSheet: Methodological steps (format: json) with the following fields.

– Model Name (optional); – Purpose: Supervised ML/DL model goal

(optional). – Domain: Where the model is used (op

tional). – Data: Description of the data and the pre

processing techniques (optional). – Model information: Information about the model (optional). stage, it is worth mentioning that the mapping function and weights per pillar and metric of the algorithm are not shown for the sake of simplicity and room. However, they can be found on the compare page of the Web application [ 15 ] once both solutions are selected.

7. Conclusions

This article introduced a novel, adaptive, and parameterized algorithm able to quantify the trustworthiness level of supervised ML/DL models with tabular data. The algorithm considers twenty-three metrics grouped into four pillars of trusted AI (fairness, explainability, robustness, and accountability). It combines the metrics outputs to compute a global trustworthiness score of supervised ML/DL models according to their training and testing data, model, and FactSheet. Also, the algorithm was deployed on a Web application where a cybersecurity scenario demonstrates the practical applicability of the algorithm. In conclusion, this work outlines the importance of considering not only traditional performance metrics but also the trustworthiness level of ML/DL models.

Future work plans to improve the limitations of the current solution with the support of unsupervised models and the inclusion of suggestions to improve the trustworthiness level of ML/DL models. Finally, the Web application will be redesigned to reduce computation time.

Acknowledgments This work has been partially supported by (a) the Swiss Federal Ofice for Defense Procurement (armasuisse) with the CyberTracer and RESERVE (CYD-C-2020003) projects and (b) the University of Zürich UZH. tion models, SN Applied Sciences 3 (2021) 1–12. [24] F. Y. et al, Interpreting and evaluating neural network robustness, in: 2019 International Joint Conferences on Artificial Intelligence, 2019, pp. 4199– 4205. URL: https://www.ijcai.org/proceedings/2019/ 0583.pdf . [25] T.-W. Weng, H. Zhang, P.-Y. Chen, J. Yi, D. Su,

Y. Gao, C.-J. Hsieh, L. Daniel, Evaluating the robustness of neural networks: An extreme value theory approach, arXiv preprint arXiv:1801.10578 (2018). [26] H. Chen, H. Zhang, S. Si, Y. Li, D. Boning, C.-J.

Hsieh, Robustness verification of tree-based models, Advances in Neural Information Processing

Systems 32 (2019). [27] N. Carlini, D. Wagner, Towards Evaluating the

Robustness of Neural Networks, in: 2017 IEEE symposium on security and privacy, IEEE, 2017, pp.

39–57. [28] S. Moosavi-Dezfooli, A. Fawzi, P. Frossard, Deepfool: a simple and accurate method to fool deep neural networks, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2574–2582. [29] T. Jayalakshmi, A. Santhakumaran, Statistical normalization and back propagation for classification, International Journal of Computer Theory and Engineering 3 (2011) 1793–8201.

A. Images from web application Choose File no file selected Drag and Drop or Select File Choose File no file selected Drag and Drop or Select a File

P R OT E CT E D F EAT U R E P L EA S E S E L E C T T H E P R OT E C T E D F EAT U R E

Select Protected Feature

P R OT E CT E D VA LU ES P L EA S E S E L E C T T H E P R OT E C T E D VA LU E S FO R T H E

P R OT E C T E D F EAT U R E Select Values of the Protected Feature belonging to the Protected Group

Select Target Column

FAVO RA B L E O U TCO M ES P L EA S E S E L E C T T H E FAVO R A B L E O U TCO M E S FO R T H E

TA R G E T CO LU M N

Select Favorable Outcomes ChooseDFrilaeg anondfilDersoelpecoterdSelect File

CREATE FACTSHEET Choose File no file selected Drag and Drop or Select File

ANALYZE ! ! ! ! ! ! ! ! ! ! ! NormaliTzeasttioSnplit 8N0o.n0e0/20.00 Train Technique PTNruRamibOnerP/OETfResTt I DEaSta 468000 samples / 12000 samples SFieazetures RMeogduelarTiypzaetion nRoatndospmeFcoirfeisetdClassifier Technique NormaliTzeasttioSnplit 80.00/20.00 Train

None TTercahinniq/ueTest Data 48000 samples / 12000 samples NSuimzbeer Of FReeagtuulraersization Technique Normalization Technique 6 6 None not specified Model Type RandomForestClassifier NaAmcecuracy 0.94 This TCsrhceiedsniatsrcieCoanraidrsioAcpopinrscoevcraonlnecderwnietdh wdietchiddiencgidiifngthief atphpeliacpaptliiocnatfioorn afocrreadictrecdairtd ciasrd is Model Information suppossuepdpotsoedbetogrbaentgerdanotreddeonriedTder.naiTenhde.TedTsahteaSsdpealttiatcsoent8t0ac.io0nn0st/a2ti0hn.es00ntuhmebenrumobferchoifldcrheinl,drtehne, the DescrDiepGstlciorobinapltiRoencall 0.94 anualTahnsiuasallasrcsyea,nlatrhiyeo, hitoshuesceohnoacunesdrenceaadnrdwoicwtanhrerdosewhcniiepdrissnhtgiaptiufssttaohtfeustahpoepflaitpchpaeltiiacopanpnltif.ocraBnaats.ecdrBeaodsnietdthcoiansrdthis Authors inforsimunapftpoiorosmneadttihoteon abtpehpelgiracapanpttlieiodcnatofrioorndTertnfahoieerndc.t/rheeTdheiecstrtedcadDatiarattdsaecita4sr8dc0eo0ini0tsthaseiearinmstpahlpetephrsreoa/vpneup1dmr2bo0oev0rre0drosefoajrmecpchrlteieeljsdde.rcetne,d.the DeCslcarsisptWieoinghted anual salary, the house andPSiRczaeOr PowEneRrTshIiEp Sstatus of the applicant. Based on this Link LCioPRnektEcaaRcltFl IOnfRoMrmaAtiNonC0.E94M E T R I ChSttpshi:nt/ft/opwrswm:wa/.t/kiwaowgnwg.lkteha.egcgolampe/p.alclioecmxa/itasilboecnxoioskb/csoyonkt/hseytnitchd-eictriecd-airctdr-ecdiasirtd-e-icataphrpedrr-oavpaaplpr/romovevatelad/dmaeotrtaadreajteacted. Refgourlatrhiezactrieon

RnaontdosmpFeocriefsiteCdlassifier LiAGnclkcoubraalcyPrecision 00..9944 https://www.kaggle.com/alexTMieosdcbehclnoioqTkyu/peseynthetic-credit-card-approval/metadata ModelPMoRTdyeOplePTyEpRe T I ERaSndoRmaFnodroemsFtoCrleasstsCilfaisesrifier AccurPAaccEcyuRrFacOy R M0A.9N4 C0.E94M E T R I C S O V E R A L L S C O R E

• T R U S T W O R T H I N E S S TrainMTorTdaeeislnt TTSyeppsletitSpl8i0t.00R8/a02n.0d0.o00m/0F2o0r.e0s0tClassifier GlobaAGlclcoRuberacalclyRlecall0.94 00..9944 """!! TrainTr/aiTnes/TteTsDteasttSaplD4ia8t0a008408s.0a00m00p/l2se0as.m0p/0le1s20/0012s0a0m0plseasmples

Size Size 3.R1egu/lTR5areagiiuznlaat/riioTzneasttioDnaotta s4np8oe0tc0i0sfpieseacdmipflieesd / 12000 samples

r O V E R A LTeLchnSTiSieqczuehCeniOqueR E

Number Of O V E R A L L FSeatCurOes R E """!!

P R OPPREORPTEI ERST I E S 3.1/5

NormaRNleoigrumlaalrioiznzaattiioonnNone nNootnespecified

zati ""Te"ch!nTie!qcuhenique

Technique 3.FN1euamtb/ueTFNN5rreeouecarmsOhtmbfnuaeirlrqeiusOzefation 6 6None

NFuemabteurreOsf 6 M O DMECOlLaDsIsENWLFeiOIgNhRtFeMOdART00M..I99OA44TNI O N

GPlroebcailsioRencall ModelMMoNCGOdalleEmoaDlebsRsaNFlaWmOeFei1RghSMtceoAdreN C00..E9944M E T R I C S

E L I N F O R M A T I O N P

Recall PurpoMPsoueCdrlpeDaloessseNcarmiepti

WDeeisgchrotinepdtion Accuracy

GFl1obSaclorePrecision 00..9944 DomaiPDnuormDpaeoissncerDieDpestscicrorinippttiioonn

CGlaosbsalWeRiegchatleld 00..9944

Precision TrainDTiornCmaglaiansisngWeDsDiacegtrshaictpreDtideipsotcniroinption iDnatDae

0.94

GRleocbaall F1 Score 0.94 Model•TMroIadniefTnloirnIRmngaftUoDiarotmnaaStDiTeosncWriptOionR T H I N E S S

CGlaosbsalWePirgehctiesdion 00..9944 AuthoMArousFdt1ehloSrcsIonrfeormation

CPlraescsisWieoinghted 0.94 ContaACcuotntthIaoncrftsorImnaftoiromnation

Global F1 Score 0.94 Contact Information •ClTassRWeUighStedT W O R T H I N E S S

F1 Score 0.94 P E RPFEORRFMOARNMCAEN MCEE TMREI CT SR I C S ClassCGlWoaebsiasglhWteReeidcgahltle0d.94 00..9944 RecalRlecall

Class Weighted GlobaGRlelcoPabrlaelciPs5rieocnis0i.o9n4 00..9944 ClassCGlWoaebsiasglhWtePeridegchitse0id.o9n4 00..9944 PreciPsrieocnision

Class We4ighted GlobaGPlrleocbiaslSicoFon1reSco0r.e94 00..9944

F1 ClassGClWoaebsiasglhWteFe1idghSctoe0rd.e94 00..9944

5 F1 ScFo1reScore3

CFl1asSscoWreeighted 0.94 • T•RTURSU5TSWT OWROTRHTI HN EINSESS S

• T R U31 S T W O R T H I N E S S 2 0 3 1 2 0 1 EXPLAINABILITY

EXPLAINABILITY

ROBUSTNESS

ROBUSTNESS

ACCOUNTABILITY

ACCOUNTABILITY EXPLAINABILITY

ROBUSTNESS

ACCOUNTABILITY EXPLAINABILITY

EXPLAINABILITY EXPLAINABILITY

ROBUSTNESS

ROBUSTNESS ROBUSTNESS

ACCOUNTABILITY

ACCOUNTABILITY ACCOUNTABILITY

G E N2 E R A L I N F O R M AT I O N 0 O la

U ve

C nd B a lan

• C O M PA R E 1 U

O Purpose Description ndAttack Classifivceation

r er E X P L AfittI N

IT Siencgurity Domain Description

A B I L IfTittinYg S C O R Ealan

B Description TrItaiSneicnIgncDiadteant Classification

!!!!!

Alberto's classification data MoIMtIdteeSSlleiekcIecInsnIfncSocirvdimdceaentEntitoCXCnllaPasssLifiCfilcAicaasttIisoiiNonfn2iAcat.Bi0onI L/scIeT5naYrio S C O R E C O M P ADescRripEtion M O D E L I N F O R M A T I O N

0 U

O er r SCENARfIiOttMSineglike' sfittSVC2

UPinLgOAD

S tis E X P L A I N

A B I L I T Y IT Security

D

S C iOffe R E !!!!! r ity t ra icalAPNaALYZEte ren ren p act Melikes Svc2

U SCENndARIOS er l Pa t ra MicAaT I OteN C laCCOOMMPPAARREE ss B a lan × × × ×× × × DReesccarlilption

Split •TPruaripGbnoaislneEgDeDN2as0ct.arE8i7pRtioAn LAttaIcNkCFlasOsifRicaMtioAn

T I O N Glo Alberto's

classifTircaaitnio/n Tdeasta 3510 samples / 391 !!!!!

F O R M AT I O N MoMdeellikIensfSorvmca2tion Authors

E X P LClAasIsiNf2iAcat.Bi0onI L/scIeT5naYrio S C O R E FA

EX P RO

BU

S IL IL

UN

B COMaPlaAnRE SVC none M 7o4de S ize ADuetshcorrisption

0 MClEasTs R I C S Model Information

SIkTlSeaercnuriSVtMy Classifier

Regularization MPordeeclisIinofno0rmation [e-mail] Contact IDnefsocrrmiapttiiooAnnlg Attack ClassifiocrarNetuimobner Of Purpose Class

Melike Demirci Normalization Melike's SVC2ClassTiefcihenriqfuoer IT SNeocnureriittyy IInncciiddeenntt Sklearn SVM

Data Size

samples for IT Security Incident Alberto's classiPfiRcaOtiPonE RdaTtasIE S

Fea

Felatteudres PGCllEoabsRasFl O R0M.8A7 N C E

PTrRatiOendPTFCeeEasRt T I E S

ize APAFuc1tEchuSRorcrFaoscrOye R0M.8A7 NRC OE BMelUikeCSlaDTsesmNircEi SSpSlitS tOu R9E0.00/10.00 MGlEobTalR I C S

re 0.87 CRoenctaalclt Information melike.de!mi!rc"i@!uzTMhro!.daciehnl /TyTpest

Data Size CAlcacsusraTcyR0.U87 S T W O R T H I TRSNrepagliuiEnltaSrization 9n0o.n00e00//1100..0000

TesSt

5 GGlloobbaall SCRceloacrsaesll

F1400..8877 CPClrlaeascssission FGR1elcobaSclaolre 0.87

F1 CPlraescsision4 2 4 Precision31 Global F1 WFe1igShctoTerdeR0.U88 S T Score Class

AAlltteerrnnaattiivveeSSttyyllee W

O R T H I FNeatEurSes S

O V E R A L L S C O R E T R2 U S T W !!"!!

O R T H I N E S S 0

C fid O V E ReAn L L

E r Fa S Cst O R E

G E r D A

D W agn W agn eep foo tta Altercnkative Style eep ACCO foo TAB tta ITY

FA IRN

3 Melikes Svc

1 RO

BU S C O TRNEESS

TAB

ITY

IMMTeelliSiekkceeu''rssiStSyVVCC NTNurumambienr/OfTest 3550 samples / 888

74 Regularization Technique Technique Number Of FWF1e1iSgSchcotToreredeR0.U85 S T W

O R T H I FNeatEurSes S Score TTRR2UUSSTTWWOORRTTHHIINNE S S

!!!!! 0 Class

C r

E r

E 2 on Fa Weighted 0.85 SSCsCOORRE

OOVVEEfRiRdeAnALLLL t G F1 Score 1 tta O VFAEIRRN

ck T R0 U SCoTn W O33REr..TFa11sHI//NEE5rCSarS 5 fiden t G lini 0 ce

A LS LEXSP C OradRienEROBU co LA E r D A

D W agn W agn eep foo eep ACCO ck foo l A IL A tta tta ITY MGoldoeblalName

20.85 MCPMoorodnedetcelailcsItiInonfIDnfone0orfsrmocmarratmitiapiottoniinooAnnlg [e-mail] Of Purpose Class o r MMietelhliikkee DDeemmiirrccii

m

IT SecCurity

M none 7o4de S ize

PTPrRRatiOendPTFCeeEasRt T I E S

ize APAFuPc1tEcEhuSRoRrcrFaosFcrOyeORR0M.M8A5ANNRCCOEE BMelUikeCSlaDTsesmNircEi SSpSlitS tOu R8E0.00/20.00 O 3.5/TTrSr5aaEinS/CTeOst R3E550

Technique VREORBAULSL TSNCEOSPNDRDaoaRtrtmaOalSPizEaeRtiTonI EsTSraamipnliensg

Technique 5

F1400..8855 Class CPClrlaeascsision4

2 WWeeiigghhtteedd400..8855 Precision31 Global F1 2 4

ITY !!!!! !!"!! r !!"!f!itting 2.8/5 !!!!!

ing fiittttiinngg

ITY !!"!! !!"!! IRN IRN

ESS ESS

EX EX P LA P LA IN IN AB AB IL IL ITY ITY RO RO

BU BU S TN S TN ESS ESS

ACCO

UN

TAB

IL

ITY

ACCCOOUUNNTTAABBIILLIITTYY F A I R N E S S

S C O R E FA EXPR !!"!!

TRO tsh ACCO

R S M O issve alfiztatAitniogInR N a

D EintagS S F p lit ra tsh r ity

C p act UN C TAB IL ten ITY e ce ten !!"!! U fitfittt ing inigng

O O r fitt S r fitt ing tis ing l Pa FA IRN

ESS FFAAIIRRNNEESSSS

EX EEXXP P LA IN AB

IL PLLAAIINNAABBIILLI

RO RO

BU

BU ITY ITY

S TN S TN ESS

ESS

EXPR !!"!!

TRO t ion M ra BU c

Fa aFFEASASIisIRsRingNNEELAeSSIgNuSAlSaBrIiLzSaSC OTeTRRN E

CinOS E tsh liza 2t.a8 !!!""/!!!I!5TtYion D a ! stESSS

ACCO N o rm

R c tsh ACCO ACCO

UN UN

TAB

TAB UN C TAB IL IL ITY ITY IL ten ITY C B a lan p le ten U nd U

S C O R E

ss r er I N A B I L IfTiTttinYYg SSCCOORREEBa E X P LAfAittI N A B I L I E X P L !!!!!

O 2.0/5 !!!!! !!!!! o r ithm

22..00//55 E X PAlgL A I N A B ICLo I T Y

rre 2.0/5

Fea S C OMoRd E ss C Cllaassss B allaannccee C la t ica t ica

C la p act te D iffe ing

t !in!g!!!ica A B ISLta I T Y DSispCa O RCcElea

tis

la tu S C OMoRd E

References

[1] ThinkML, The Growing Timeline of AI Milestones , https://achievements.ai, 2021 .

[2] J. Wing , Trustworthy ai, Commun. ACM 64 ( 2021 ) 64 - 71 .

[3] G. Saposnik , D. Redelmeier , C. Ruf , P. Tobler , Cognitive Biases Associated with Medical Decisions: A Systematic Review, BMC medical informatics and decision making 16 ( 2016 ) 1 - 14 .

[4] R. Bellamy , et al., AI Fairness 360 : An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias , arXiv preprint arXiv: 1810 . 01943 ( 2018 ).

[5] P. Sattigeri , S. Hofman , V. Chenthamarakshan , K. Varshney , Fairness GAN : Generating Datasets with Fairness Properties Using a Generative Adversarial Network , IBM Journal of Research and Development 63 ( 2019 ) 3: 1 - 3 : 9 .

[6] S. Dash , O. Günlük , D. Wei , Boolean Decision Rules via Column Generation , in: 2018 International Conference on Neural Information Processing Systems , 2018 , p. 4660 - 4670 .

[7] A. Vijay , et al., The Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques , arXiv preprint arXiv: 1909 . 03012 ( 2019 ).

[8] S. Lundberg , S. Lee , A Unified Approach to Interpreting Model Predictions , in: Advances in Neural Information Processing Systems 30 , Curran Associates , Inc., 2017 , pp. 4765 - 4774 .

[9] I. Goodfellow , J. Shlens , C. Szegedy , Explaining and Harnessing Adversarial Examples, arXiv preprint arXiv:1412.6572 ( 2014 ).

[10] E. Tabassi , K. Burns , M. Hadjimichael , A. MolinaMarkham, J. Sexton , A Taxonomy and Terminology of Adversarial Machine Learning , NIST IR ( 2019 ) 1 - 29 .

[11] M. Nicolae , et al., Adversarial Robustness Toolbox v1.0 .0, arXiv preprint arXiv: 1807 . 01069 ( 2019 ).

[12] J. Richards , D. Piorkowski , M. Hind , S. Houde , A. Mojsilovic , A Methodology for Creating AI FactSheets , arXiv preprint arXiv: 2006 . 13796 ( 2020 ).

[13] M. Hind , et al., Experiences with Improving the Transparency of AI Models and Services , in: 2020 CHI Conference on Human Factors in Computing Systems , 2020 , pp. 1 - 8 .

[14] J. Leupp , M. D. Demirci , J. Bauer, TrustedAI Git Repository, https://github.com/JoelLeupp/ Trusted-AI, 2022 .

[15] J. Bauer , M. Demirci , J. Leupp , A. Huertas , Web Application to Quantify Trusted AI , https: //www.trusted-ai.net, 2022 . User: trusted-ai; pass: mp@csg2021.

[16] S. Feuerriegel , M. Dolata , G. Schwabe, Fair AI , Bus. Inf. Syst. Eng . 62 ( 2020 ) 379 - 384 .

[17] R. Zemel , Y. Wu , K. Swersky , T. Pitassi , C. Dwork , Learning fair representations , 30th International Conference on Machine Learning , ICML 2013 ( 2013 ) 1362 - 1370 .

[18] I. Kloumann , J. Tannen , How we're using fairness flow to help build ai that works better for everyone , 2021 . URL: https://ai.facebook.com/blog/ how -were-using-fairness-flow-to-help-build-ai-\ that-works-better-for-everyone.

[19] Fairlearn , Improve fairness of ai systems , 2021 . URL: https://github.com/fairlearn/fairlearn.

[20] R. Zemel , Y. Wu , K. Swersky , T. Pitassi , C. Dwork , Learning fair representations , in: International conference on machine learning, PMLR , 2013 , pp. 325 - 333 .

[21] M. Hardt , E. Price , N. Srebro , Equality of opportunity in supervised learning , Advances in neural information processing systems 29 ( 2016 ).

[22] D. Solans , B. Biggio , C. Castillo , Poisoning attacks on algorithmic fairness , in: Joint European Conference on Machine Learning and Knowledge Discovery in Databases , Springer, 2020 , pp. 162 - 177 .

[23] M. Saarela , S. Jauhiainen , Comparison of feature importance measures as explanations for classificaShow Configuration Attack Classification 2.7/5 90.00/10.00 AWuetihgohrtsed 0.88 Domain Des1cription Precision Contact Information TGrlaoibnailngF1Data 0 .87 DPSecsEocrRreiFpOtioRnM A N C E Weighted 0 .88 DColmaasisn Description Model Info31rI0mN.a8tF7iOonR M AClTaIsOsiNfication scenario MTWreOaiigDnhitEnegdL Data Alberto's classifTieccahtniioqnuedata Recall Weighted 0.87 Global Recall 0.87 Recall Precision MWeEigThtRedIC 0S.87 GAWlceocibugarhlatceyd 00..8878 Precision 0.87 Weighted 0.88 WWeeiigghhtteedd 53500..8887 0.87 Weighted 0.88 Class 0 .87 • MSeuGlpikpeEosrtSNVvceEctoRrCAlasLsifieIr N01 Contact In5formation melike . [e-mail] WGAGelcliocogbubhartlalecdy 00..8855 Precision 00..8855 PPrreecciissiioonn WWeeiigghhtteedd 00..8855 WWeeiigghhtteedd 5500..8855 FGRG1ellocobabalall FF131 Score 00..8855 0.85 3.1/5 4 Figure 7: Compare55Page withinPgerformance Metrics acne d Tessrustworthin55ess Scores of Two Solutions.rence 2 .5/5 p le p le E X P L A I N A B I L I T Y