Performance evaluation of Machine Learning systems have been usually limited to performance measures on curated and clean datasets that may not properly reflect how robustly these systems can operate in real-world situations. One key element in this understanding of robustness is instance dificulty . The efect of instance dificulty on robustness could be understood as how unexpected would be that a customary system fails on a particular instance of certain dificulty. In order to provide further understanding on this issue, we estimate IRT-based instance dificulty for an illustrative set of supervised tasks and we implement and test perturbation methods that simulate noise and variability depending on the type of input data. With this, we evaluate the robustness of diferent families of machine learning models, which we select and characterise according to their behaviour. The preliminary results of this work in progress allow us to define a novel taxonomy based on the robustness of the diferent models and the dificulty of the instances addressed. This study is a significant step towards exposing vulnerabilities of particular families of machine learning models.
The success of AI and specially Machine Learning
(ML) technologies caused these type of systems to
spread across many applications from diferent domains,
e.g., medical, financial, social or autonomous transport,
among others [
Robustness is defined by the IEEE standard glossary of software engineering terminology [ 4] as: The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental conditions. In the context of ML, robustness measures the resilience of a system towards perturbations in any of its components (the data, the learning program, or the framework) [5]. Earlier works [6] assessed model robustness by perturbing instances in the training set, test set, or both. A general way to perturb instances is adding noise, a method that has been extensively applied in the adversarial ML field for the generation of adversarial examples ([7]). This is why most of the research in
(IRT) [13], where dificulty is inferred from a matrix of items (instances) and respondents (systems), giving more relevance to conformant systems. In addition, IRT gives a scaled metric of dificulty that follows a normal distribution and can be compared directly against the ability of a system.
In this work in progress paper, we present, as a proof of concept, an evaluation setting to analyse the robustness output of the model changes. Attribute noise has been of diferent ML models empirically, considering the dif- also used in diferent approaches to improve the robustifculty of the instances. We also perform a hierarchical ness of models to adversarial examples. For instance, [20] clustering to derive taxonomies of ML models according shows that the injection of noise in the training dataset to their robustness. The setting is general as we em- results in models more robust to attacks since they are ployed datasets from diferent domains, a wide set of rep- able to detect the perturbed instances beforehand. Simiresentative ML techniques, and an instance perturbation larly, adding adversarial instances to the training set can function that introduce random noise with no specific improve the robustness of neural nets [8], and make a goal. Therefore, it can be adapted to more specific prob- Speech Emotion Recognition system more robust [21]. lems by changing the datasets, models and perturbation On the other hand, artificial label noise is useful to simufunction to the domain of interest. In this general evalu- late wrongly annotated instances or other sources of data ation framework, we measure the robustness of a model corruption. Label noise has been used, for instance, to as the agreement modulo instance dificulty between the evaluate robustness in computer vision applications [22]. output of the model for the original and the perturbed Additionally, label noise in the training set can be emtest sets. ployed to enhance the robustness of models, for instance,
The paper is structured as follows. In Section 2 we by reducing errors derived from overfitting [ 23]. review part of the literature related to the assessment There has been proposed diferent ways of assessing of robustness in presence of noise, the estimation of in- the robustness of a model in noise environments. The stances dificulty and a taxonomy of machine learning most general method consists in measuring the correcttechniques derived from a notion of behavioral similarity. ness loss of models with noise in the data, with respect to Section 3 describes the method we developed to assess the case without noise, regardless of where the noise is model robustness. We apply that method and describe located (training set or test set). For noise in the training the experiments in Section 4. Finally, 5 concludes the set, the metrics used to quantify the loss are the standard paper. classification metrics such as accuracy and F-measure [14] and the Equalised Loss of Accuracy (a metric for measuring a classifier’s noise robustness [ 24]). For the 2. Background sub-field of adversarial robustness (where noise is used in the test set to generate the adversarial examples) there In this section we revisit some key concepts related to has been used specific metrics such as adversarial accumodel robustness, instance IRT-based dificulty and the racy [8]. There are other general techniques for evaludefinition of behavioural taxonomies of machine learning ating robustness, including mixed integer programming techniques. [25], abstract interpretation [26], and symbolic execution [27, 28, 29]. 2.1. Robustness in a noisy framework In this work, we are interested in studying how the robustness of a model is afected by the distortion produced by injecting diferent levels of noise into test instances considering the dificulty of the perturbed instances. As far as we know, instance dificulty has not yet been taking into account to evaluate the robustness of models.
characterise their behavior [5], being particularly suitable for checking whether the system behaves as expected to changes in the data. A common way to simulate those changes is injecting noise into the data, given that real world data often contain some degree of noise [ 14].
In the literature, there is a large number of approaches 2.2. Instance dificulty
for adding artificial noise to datasets [
IRT dificulty is characterised by being systemindependent an domain-generic unlike the other metrics developed subdiscipline in psychometrics [36], only redescribed above [11]. It also has some advantages over cently brought to AI and machine learning [37, 12, 38, 39, using average performance as a metric of dificulty, in 40]. In IRT, the probability of a correct response for an terms of distribution, stability and predictability, as has item is a function of the respondent’s ability and some been studied in the literature of IRT. item’s parameters. The respondent solves the problem and the item is the problem instance itself. We focus on the dichotomous models where the response can be either correct or incorrect.
with = 1 for a correct response and = 0 otherwise. For the basic one-parameter logistic (1PL) IRT model, the probability of a correct response given the examinee’s ability is modelled as a logistic function: ( = 1| ) =
1 1 + (− ( − )) (1) by dificulty logistic function.
is the dificulty of . If ability equals item dificulty then there are even odds of a correct answer (cutting the , curve as exactly 0.5, as the light blue dashed shows in
( ), which is the location parameter of the characteristic curves (SCC), also follow a logistic func- (measured as output agreement), especially considering 2.3. Behaviour-based Machine Learning
families
machine learning techniques is by defining ‘families’, according to their formulation and learning strategy (e.g., neural networks, Bayesian methods, etc.) [43, 44, 45].
However, this taxonomy of learning techniques does not take into account the intrinsic behaviour of the models predictions in sparse zones where insuficient training data was available. If we want to characterise the robustt1.00 c e answer the most dificult items will be assigned to high ness of ML models, we need to analyse a diverse set of the preliminary objective of testing the efectiveness of models, as many as possible, under diferent parameters our setting), but is also limited by those benchmarks as well. In this regard, [42] derived a taxonomy of ML where there is a suficiently large number | | of examples techniques for classification, where families are clustered (articles) and | | of models (respondents). according to their degree of (dis)agreement in behaviour, i.e., the diferences between models on how they dis- Dataset # Instances # Features # Classes tribute the output class labels along the feature space. letter 20000 16 26 We considered both dense and sparse zones (where train- wall-roobpottd-ingaitvsigation 55642506 644 140 ing data is scarce or inexistent), using Cohen’s kappa statistic [46]. While in dense areas diferences between Table 1 models may be dificult to find, in sparse areas the algo- List of datasets for the experiments. rithms diverge significantly, and unveil the characteristic behaviour of the trained models using those techniques. Regarding ML models, we employed a set of 18 ML
The methodology was based on comparing the be- models from diferent ML families (see Table 2), derived haviour of 65 diferent learning models (including hyper- in [42]. These 18 model families were obtained from a parameter variations), performing a pairwise comparison pool of 65 models learned and evaluated on a wide range (based on Kappa) and averaging the results obtained for of datasets for categorisation into diferent families, as 75 datasets. For grouping in families, authors applied a described in the section 2.3. For each family we selected hierarchical clustering so that the models that presented a single model, its centroid (i.e., representing the center similar behaviour fell in the same cluster, which is consid- of each family cluster), assuming it to be representative ered a model family (see the 18 diferent families obtained of its family. Thus, we can assume that the 18 selected in Figure 2). This method is useful to objectively quantify models are diverse enough to provide a wide view of how how diferent two models (or model families) are. diferent model families behave in terms of robustness.
In this section we describe the experimental methodology performed to obtain a taxonomy of classification algorithms according to their robustness. We start introducing the set of representative datasets and learning models we have employed. Then, we describe how we estimate instance dificulty, the approach followed to introduce noise in the data and, finally, how we deifne the taxonomy of ML families. All the data, code, complete experiments, plots and results can be found in https://github.com/rfabra/family-robustness. 3.1. Data and Classifiers
marks that had instance-wise results of a good number of models. It is recommended to have at least 10-20 responses per item [47]. More importantly, we need the instance-wise results, i.e., a | | × | | matrix with the performance of each system ∈ for each instance ∈ .
Finding experiments not reported in an aggregated way was not an easy task. As an exception to the instancewise result problem, we find platforms such as OpenML [48], a repository in which AI researchers and practitioners can share data sets and results in as much detail as possible. The platform also provides several curated datasets such as OpenML-CC18, from which we address a set of 3 benchmarks for supervised learning (see Table 1).
The selection is guided by the illustrative character (with 3.2. Estimation of Dificulty As mentioned in Section 3.1, in order to estimate the dificulty of the instances, we first check that for each benchmark selected from OpenML there is at least 10-20 reponses (model evaluations) per item/feature (e.g., we would need between 640 and 1280 responses for optdigits) and that they are suficiently diverse (diferent architectures or technologies). Next, we obtain their responses for unseen instances (e.g., we will be using the test folds, so it is actually test performance, even if we cover the whole dataset). This will be our | | × | | matrix with all binary responses .
We follow the recommendations from [12] for the application of IRT. In practice, for generating the IRT models, we used the MIRT R package [49], using Birnbaum’s method, as explained above. The package MIRT (as many other IRT libraries) output indicators about the goodness of fit which can be used to quantify the discrepancy between the values observed in the data (items) and the values expected under the statistical IRT model. Item-fit statistics may be used to test the hypothesis of whether the fitted model could truly be the data-generating model or, conversely, we expect the item parameter estimates to be biased. In practice, an IRT model may be rejected on the basis of bad item-fit statistics, as we would not be reasonably confident about the validity of the inferences drawn from it [50]. In the present case, none of the estimated models were discarded because of bad item-fit statistics or inconsistency in their results. 3.3. Introduction of Noise We need a method to generate noise, representative and general enough, so that the experimental results can be adapted to other noise settings, e.g., to include adversarial attacks. Hence, we will work directly with noise levels, generated randomly by using some well-known probability distributions, following a similar procedure as in [16]. Instances are perturbed by changing their attribute values into a range of possible values. The process to select among the possible values depends on whether the attribute is nominal or numerical: • Numerical attributes: Let be the level of noise a value in is modified as ′ ∼ (, ⋅ ) to be injected into a numerical attribute , and the standard deviation of all values of . Then,
, i.e., we follow a normal distribution using as mean and multiplied by the noise level as standard deviation. • Nominal attributes: Let { 1,..., } be the set
of the possible values of a nominal attribute List of the 18 models employed for the experiments, along with the parameters used. 3.2.1. System Characteristic Curves One of the most powerful visualisation tools that de- Kappa metric, as we describe below. assuming that they are mapped from contexts. Noise is 1 is considered more robust than model 2 if , and the vector that represents the empirical distribution of , that is, = ( 1, … , ), where, is the frequency of value . Consider we have an instance of value = in , we represent as the vector = ( 1, … ,
) with = 0 ∀ ∈ {1..}, ≠
, and = 1. To insert a noise level , we calculate = 1 − (−) , and then compute a new vector of probabilities ′ = ⋅ + (1 − ) ⋅
. Finally, we use ′ in order to sample the new value ′ of the attribute.
For the experiments, we will generate noisy datasets (test set) using a noise level = 0.2 . We vary the proportion of perturbed instances in each bin, from = 0 (keeping unperturbed the original test set) to = 1 (perturbing the whole test set). This is performed under a
3.4. Model robustness to noise and dificulty We compare the behaviour of ML models from diferent families by classifying the same test set from a particular benchmark, to which we introduce diferent levels of noise. The more the behaviour of a model changes under noise, the less robust it is. This diference in behaviour can be measured with Cohen’s Kappa metric [46]. More concretely, given the domain of all data sets we can create from all possible inputs, a test set ∈ , a perturbation function ∶ →
to introduce noise into a data set, the perturbed test set ′ = ( ) , the predictions of a model for the original test of a model for the perturbed test ′ = ( ) , the predictions = ( ′) and two models 1 and 2 learned on the same data, the model ( 1 , ′ 1) > ( 2 , ′ 2)
between the predictions of a model on the original and the perturbed test sets. It is important to notice that we are not accounting for the real class label, since adding noise to the input attributes of an instance implies that the actual class is probably not the same as it was originally.
Instead, we compare the model predicted labels for the original test set (without noise) with the ones predicted for the noisy test sets. Our goal is not to determine the well-performance of a model to solve a task, but to assess how the behaviour of the model changes under diferent levels of noise applied to instances of diferent levels of dificulty. As we want to analyse whether the model robustness may vary depending on the dificulty of the 3.5. Experimental questions instances addressed, we estimated the dificulty of each typical in educational measurement. In health measureinstance in the dataset following the procedure described ment, however, these values are usually much higher and above. Later, we grouped instances into dificulty bins to around 4. In our case, when addressing ML benchmarks, analyse the robustness (to produce SCCs), as explained dificulty values around -3 and 3 are the norm (see [ 12]). above. For this reason, we decided to remove those instances
Analysing the data from the SCCs for diferent mod- whose dificulty is out of the range [−6, 6], which are els we also derive a ML robustness model taxonomy at- considered outliers. This happened in all benchmarks tending at the diferent shapes of the SCCs and models’ for very easy instances for which all techniques are corbehaviour. In this regard, for each dataset, we built a ma- rect, never afecting more than 0.1% of the instances. trix where each row represents a model and each column Figure 3 shows the IRT-dificulty distribution per benchrepresent a combination of dificulty bins and proportion mark, with a standard deviation around 1 (as expected). of noisy instances per bin. Each element represents the In terms of location (Q1), the letter benchmark contains similarity (i.e., the kappa metric) between the predictions more dificult instances (mean dificulty of −1.50 ± 0.92 ) of the model for the original test set and the predictions than the others (−1.92 ± 0.67 for optdigits and −2.36 ± 0.9 for each noisy test and bin. By averaging these across all for wall-robot navigation). Although the distribution is the datasets, we may perform a hierarchical clustering generally normally distributed, the wall-robot-navigation with the aim of obtaining diferent grouping of models dataset presents a higher number of dificult instances, by robustness, showing the similarity between diferent skewing the distribution to the right. This may be due to ML families, in a data-driven fashion. the diversity of the population of systems used for the dificulty estimation (similar cases can be observed in [11]).
tewall−robot−navigation tasa optdigits d letter −4 0 4
Difficulty Figure 3: IRT-dificulty distribution per dataset. Benchmarks sorted by average dificulty.
want to investigate the relationship between the robustness of the models and the dificulty of the instances, the latter having been altered with diferent levels of noise.
For this, we set 3 experimental questions. Q1: How do dificulties distribute per benchmark for the IRT-dificulty metric estimated? Q2. Can we see diferences in robustness for diferent models based on the dificulty of the instances? Q3. Can we group models by robustness?
Regarding Q2, for each technique in Table 2, we compare its predictions on the original test set (for each dataset in Table 1) with the predictions of each of the 4. Experiments noisy test sets, by means of the Kappa metric. The SCCs produced (using Kappa values on the y-axis) are shown in 4.1. Setup Figure 4. Obviously, Kappa takes values equal to 1 when the test set is not perturbed ( = 0 ), since we are comWe employed R language with caret package [51] to paring the output labels of the diferent trained model carry out our experiments, i.e., training and evaluation with themselves. As we increase the amount of perturbed of the models. All the models were learnt from scratch, instances (the same proportion for each bin of dificulty), so we did not used any pre-trained model. We used the we can appreciate diferences in the behaviour for the MIRT R package [49] for estimating IRT 1PL models. To techniques analysed. feed the IRT method, we obtained the predictions from a As expected, the most dificult instances are those that wide variety of models by using OpenML API [52]. In total, are more sensitive to noise, and this can be seen in terms we employed the predictions of (up to) 2000 evaluations of the level of performance of the diferent techniques for per dataset. the most dificult instance bins. This behaviour may indicate that these instances are located close to the decision 4.2. Results boundary or regions with class overlap, so the behaviour for most techniques is more unpredictable in those reIRT dificulties are built to approximately follow a nor- gions that in easier ones. In general, we may find some mal distribution with standard deviation 1 but diferent patterns of behaviour for diferent sets of techniques. locations depending on the dataset. When it comes to First, we identify cases where robustness decreases nonthe item dificulty parameters, what you find acceptable linearly with increasing levels of dificulty. This is the depends very much on the purpose of your test and the most common case, but with diferences in robustness population of interest. For instance, values around 1 are variations for diferent models and datasets (see, e.g., CI_T, FDA, 3NN, MLP , MLR or SVM). Second, we also see cases letter
FDA
GBM
JRip 3NN 1.00 0.75 0.50 0.25 a0.00 appK1.00 NB PART RBF RDA RF RFRules RPART PLS SVM 0.75 0.50 0.25 0.00−2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5 −2.5 −2.0 −1.5 −1.0 −0.5
Difficulty CI_T
FDA
GBM
JRip 3NN
LVQ
MLP
MLR where robustness is mostly afected by the noise level the last bin (the most dificult) for this same model and and less by the dificulty of the instances (see, e.g., NB, dataset, we can see that it presents similar behaviour RBF or RDA). Finally, there are cases in which robustness compared with the first bin. However, this phenomenon is barely altered by either dificulty or noise level (see, happens for a diferent reason. In this case, the model e.g., PLS, PART or LVQ). predicts 160 instances of class “1” for = 0 , whereas
On the other hand, if we analyse the results at the for = 1 , the number of instances predicted of this dataset level, we see that the behaviour of some tech- class increased up to 244, i.e., this bin tends to absorb the niques changes significantly. For instance, techniques predictions of class ”1” the more noise is introduced. Both such as C5.0, CI_T and JRip for the dataset optdigits exhibit cases may constitute a robustness flaw for a particular an interesting behaviour. These techniques seem more model. prone to change theirs predictions in easy instances than Finally, for Q3, we derive a taxonomy to group similar medium (even hard) instances. Analysing the results in techniques in terms of robustness behaviour considermore detail, we have seen that this is due to the class dis- ing dificulty. To measure the dissimilarity between sets tributions in those more easy bins: these bins are usually of observations, we employed the Kappa metric comcomposed of many instances of a single class (usually the puted for each model, aggregated accross all datasets, majority class), but these instances may be misclassified dificulty bins, and proportions of perturbed instances . as we increase the amount of noise, thus reflecting a drop We performed an agglomerative hierarchichal clustering, in the Kappa value. employing the euclidean distance and the complete link
This is the case, for instance, with the JRip model learnt age method as a linkage criteria. The result of applying on the optdigits dataset. The first bin is composed of 479 the hierarchical clutering is shown in Figure 5. We found instances of the class “6”, without introducing any noise three main clusters. The first cluster show that CI_T, JRip ( = 0 ). After perturbing all the instances ( = 1 ), only and C5.0 have very similar behaviour, joining with NB. 256 instances in this bin are predicted of class “6”, which The second cluster is composed of the models GBM, RF, explains the observed descend in Kappa. If we focus on MLR , MLP y FDA, joining with RPART and RFRules at
analysing robustness may be very useful and insightful. We would need to inspect the test SCCs as an exercise before selecting and deploying models in real-world situations. SCCs can thus be used to select the (set of) best classifier(s) according to the their robustness for diferent dificulty ranges. Since we may not know the dificulty values of these unseen examples in a test/validation set, we may estimate them in diferent (an straightforward) ways such as by averaging the dificulty values of the most similar examples in the original set [12] or training a dificulty estimator [ 11]. We could even do this with small sets or even for single instances, always running the dificulty estimator to determine which model to use for it. If we can predict the dificulty of instances, we could set a threshold to use the system only for the easy instances for which it is robust.
In this work we propose an evaluation setting to analyse the robustness of diferent ML models, from diferent ML families, when addressing noisy instances attending to their dificulty. Furthermore, we established a ML model taxonomy based on the robustness. Our results shown that there are models afected by noise, instance dificulty, or both. Some models are more prone to change their prediction when adding noise to the most dificult instances, while other models also performs similarly with easy instances. This might be caused by the concentration of
gian Research Council grant 329745 Machine Teaching for Explainable AI, also by the EU (FEDER) and Spanish MINECO grant RTI2018-094403-B-C32 funded by MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe”, Generalitat Valenciana under grant PROMETEO/2019/098, EU’s Horizon 2020 research and innovation programme under grant agreement No. 952215 (TAILOR), and INNEST/2021/317 (Project cofunded by the European Union with the “Programa Operativo del Fondo Europeo de Desarrollo Regional (FEDER) de la Comunitat Valenciana 2014-2020”) and the UPV (Vicerrectorado de Investigación) grant PAI-10-21. 2021, pp. 25–28. [43] J. Hernández Orallo, C. Ferri Ramírez, [29] G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochen- M. Ramírez Quintana, Introducción a la Minderfer, Reluplex: An eficient smt solver for veri- ería de Datos, Pearson Prentice Hall, 2004. fying deep neural networks, in: International con- [44] P. Flach, Machine learning: the art and science of ference on computer aided verification, Springer, algorithms that make sense of data, Cambridge Uni2017, pp. 97–117. versity Press, 2012. [30] M. R. Smith, T. Martinez, C. Giraud-Carrier, [45] M. Fernández-Delgado, E. Cernadas, S. Barro, An instance level analysis of data complexity, D. Amorim, Do we need hundreds of classifiers Mach. Learn. 95 (2014) 225–256. URL: https:// to solve real world classification problems?, The doi.org/10.1007/s10994-013-5422-z. doi:10.1007/ journal of machine learning research 15 (2014) s10994-013-5422-z. 3133–3181. [31] O. Russakovsky, J. Deng, H. Su, J. Krause, [46] R. Landis, G. Koch, An application of hierarchical S. Satheesh, S. Ma, Z. Huang, A. Karpathy, kappa-type statistics in the assessment of majority A. Khosla, M. Bernstein, et al., Imagenet large scale agreement among multiple observers, Biometrics visual recognition challenge, International journal (1977) 363–374.
of computer vision 115 (2015) 211–252. [47] B. D. Wright, M. H. Stone, Best test design, Mesa [32] D. Liu, Y. Xiong, K. Pulli, L. Shapiro, Estimating press, 1979.
image segmentation dificulty, in: International [48] J. Vanschoren, J. N. Van Rijn, B. Bischl, L. Torgo, Workshop on Machine Learning and Data Mining OpenML: networked science in machine learning, in Pattern Recognition, Springer, 2011, pp. 484–495. ACM SIGKDD Explorations Newsletter 15 (2014) [33] S. Vijayanarasimhan, K. Grauman, What’s it going 49–60.
to cost you?: Predicting efort vs. informativeness [49] R. P. Chalmers, mirt: A multidimensional item for multi-label image annotations, in: 2009 IEEE response theory package for the r environment, conference on computer vision and pattern recog- Journal of statistical Software 48 (2012) 1–29. nition, IEEE, 2009, pp. 2262–2269. [50] A. Maydeu-Olivares, Goodness-of-fit assessment [34] B. Richards, Type/token ratios: What do they really of item response theory models, Measurement: Intell us?, Journal of child language 14 (1987) 201–209. terdisciplinary Research and Perspectives 11 (2013) [35] D. L. Hoover, Another perspective on vocabulary 71–101.
richness, Computers and the Humanities 37 (2003) [51] M. Kuhn, Building predictive models in R using 151–178. the caret package, Journal of Statistical Software, [36] S. E. Embretson, S. P. Reise, Item response theory Articles 28 (2008) 1–26. URL: https://www.jstatsoft.
for psychologists, L. Erlbaum, 2000. org/v028/i05. doi:10.18637/jss.v028.i05. [37] F. Martínez-Plumed, R. B. C. Prudêncio, A. Martínez- [52] J. N. van Rijn, B. Bischl, L. Torgo, B. Gao, Usó, J. Hernández-Orallo, Making sense of item V. Umaashankar, S. Fischer, P. Winter, B. Wiswedel, response theory in machine learning, in: ECAI M. R. Berthold, J. Vanschoren, OpenML: a collabo2016 - 22nd European Conference on Artificial rative science platform, in: Machine Learning and Intelligence, 2016, pp. 1140–1148. doi:10.3233/ Knowledge Discovery in Databases, Springer, 2013, 978-1-61499-672-9-1140. pp. 645–649. [38] F. Martínez-Plumed, J. Hernández-Orallo, Dual [53] B. J. Petit, B. Stottelaar, M. Feiri, F. Kargl, Remote atindicators to analyse AI benchmarks: Dificulty, tacks on automated vehicles sensors: Experiments discrimination, ability and generality, IEEE Trans- on camera and lidar black hat europe, 2015. actions on Games 12 (2020) 121–131. [39] J. P. Lalor, Learning Latent Characteristics of Data and Models using Item Response Theory, Ph.D. thesis, Doctoral Dissertations, 1842, 2020. [40] Z. Chen, H. Ahn, Item response theory based ensemble in machine learning, International Journal of Automation and Computing 17 (2020) 621. [41] A. Birnbaum, Statistical Theories of Mental Test
Scores, Addison-Wesley, Reading, MA., 1968. [42] R. Fabra-Boluda, C. Ferri, F. Martínez-Plumed,
J. Hernández-Orallo, M. J. Ramírez-Quintana, Family and prejudice: A behavioural taxonomy of machine learning techniques, in: ECAI 2020, IOS Press, 2020, pp. 1135–1142.