=Paper=
{{Paper
|id=Vol-3068/short4
|storemode=property
|title=Collaborative Human-ML Decision Making Using Experts’ Privileged Information Under Uncertainty
|pdfUrl=https://ceur-ws.org/Vol-3068/short4.pdf
|volume=Vol-3068
|authors=Mansoureh Maadi,Hadi Akbarzadeh Khorshidi,Uwe Aickelin
|dblpUrl=https://dblp.org/rec/conf/aaaifs/MaadiKA21
}}
==Collaborative Human-ML Decision Making Using Experts’ Privileged Information Under Uncertainty==
https://ceur-ws.org/Vol-3068/short4.pdf
Collaborative Human-ML Decision Making Using Experts’ Privileged
Information Under Uncertainty
Mansoureh Maadi1, Hadi Akbarzadeh Khorshidi2, Uwe Aickelin3
The University of Melbourne1,2,3
mmaadi@student.unimelb.edu.au1, hadi.khorshidi@unimelb.edu.au2, uwe.aickelin@unimelb.edu.au3
Abstract tion, etc. In some of these areas like clinical decision mak-
Machine Learning (ML) models have been widely applied for ing, ML models have been applied largely. However, the re-
clinical decision making. However, in this critical decision liability of these models is always under question. It is re-
making field, human decision making is still prevalent, be- ported that ML models are not enough in critical clinical de-
cause clinical experts are more skilled to work with unstruc-
cision making (Itani, Lecron, and Fortemps 2019), (Zerilli
tured data specially to deal with uncommon situations. In this
paper, we use clinical experts’ privileged information as an et al. 2019).
information source for clinical decision making besides in- Human-in-the-loop ML models pave the way to imple-
formation provided by ML models and introduce a collabo- ment human expertise in ML models. In these models, hu-
rative human-ML decision making model. In the proposed man experts can interact and collaborate in different stages
model, two groups of decision makers including ML models
of ML process to improve the performance of ML models.
and clinical experts collaborate to make a consensus decision.
As decision making always comes with uncertainty, we pre- Clinical experts can collaborate in three stages of data pro-
sent an interval modelling to capture uncertainty in the pro- ducing and data processing (Huang et al. 2020), (Wrede,
posed collaborative model. For this purpose, clinical experts Hellander, and Wren 2019), ML modelling (Cai et al. 2019),
are asked to give their opinion as intervals, and we generate and ML evaluation and refinement (Alahmari et al. 2019) to
prediction intervals as the outputs of ML models. Using In-
improve the performance of ML methods for clinical deci-
terval Agreement Approach (IAA), as an aggregation func-
tion in our proposed collaborative model, pave the way to sion making (Maadi, Khorshidi, and Aickelin 2021). How-
minimize loss of information through aggregating intervals ever, in human-in-the-loop ML approaches, ML models are
to a fuzzy set. The proposed model not only can improve the principal decision makers and clinical experts can guide the
accuracy and reliability of decision making, but also can be models according to their expertise and experience.
more interpretable especially when it comes to critical deci-
ML models are generated using training data and used to
sions. Experimental results on synthetic data shows the
power of the proposed collaborative decision making model predict the test samples. So, these models decide based on
in some scenarios. training data information. However, there are some infor-
mation that are available at the training stage but not availa-
ble for test data. This information called privileged (hidden)
Introduction information (Vapnik, Vashist and Pavlovitch 2009), (Vap-
Machine Learning (ML) has experienced a surge in recent nik and Vashist 2009). In this study, we introduce another
years. They have been used to develop models for facilitat- type of privileged information that is available at the testing
ing decision making processes in different areas. The devel- stage but not available (recorded) for training. For example,
opment of these models is based on the idea that computers at the time of diagnosing a patient’s disease, clinical experts
can process big data and make predictions whilst it is often have an estimation about the diagnosis (with different level
hard for human experts. However, humans are more skilled of confidence) based on their experience, patient’s appear-
to work with unstructured information and deal with uncom- ance and reviewing documents and test results. These esti-
mon situations. That is the reason that human decision mak- mations are not normally recorded so that they cannot be
ing is still prevalent in many areas like clinical decision used in training models. However, they can be captured for
making, defence commanding, criminal punishment predic- each patient at the diagnosis (testing) stage. In this paper, we
Copyright © 2021 for this paper by its authors. Use permitted under Crea-
tive Commons License Attribution 4.0 International (CC BY 4.0).
propose a framework to capture experts’ privileged infor- Interval Agreement Approach (IAA) is an aggregation
mation and integrate with trained ML models in a collabo- method that generates fuzzy sets from interval-valued data
rative decision making. to minimize the loss of information in aggregation process.
As ML models and clinical experts use different sources (Havens, Wagner, and Anderson 2017; Khorshidi and Aick-
of information to make decision, a consensus decision mak- elin 2020). This approach is introduced by (Wagner et al.
ing approach can improve decision making. In this paper, 2015). In the proposed collaborative model, we use IAA as
unlike human-in-the-loop ML models, we introduce a col- the aggregation function to improve decision making
laborative human-ML decision making model where both through capturing more uncertainty and minimizing the loss
clinical experts and ML models are decision makers. Clini- of information.
cal decisions should not be made individually. We believe Thus, in this paper, we make two important contributions:
clinical experts and ML models can help each other through (1) we present a collaborative human-ML decision making
a group decision making process. Collaborative decision model to use two important sources of information in clini-
making approach can provide a trust between ML models cal decision making from two groups of decision makers,
and clinical experts to have trustable and explainable mod- ML models and clinical experts, and (2) we measure uncer-
els. In addition, this approach provides an opportunity to tainty of both decision maker groups through intervals and
have a precise investigation on the ML models’ and clinical capture decision uncertainty in the collaborative model us-
experts’ performance in clinical decision making process in- ing interval modelling and IAA.
dividually and jointly. Besides, this approach improves de- The structure of the paper is as follows. In the next sec-
cision making especially when a decision maker is not avail- tion, we describe the technique to generate intervals as the
able due to disruption in connectivity or information is pro- outputs of ML models to capture ML models’ uncertainty.
vided intermittently for decision making. Also, IAA is described in this section. Then, the proposed
Uncertainty is inevitable in decision making. Decision collaborative decision making model is explained. After
makers may have different levels of uncertainty based on that, the performance of the proposed model is investigated
their knowledge and level of access to information. When it using a synthetic dataset. Finally, conclusions of the paper
comes to human decision makers, decision making accom- are presented.
pany with two kinds of uncertainty named inter-expert un-
certainty and intra-expert uncertainty. Inter-expert uncer-
tainty shows the variation among different decision makers Preliminary
and intra-expert uncertainty is related to the change of the
mind of each decision maker during the time on the same Generating Uncertainty Intervals for ML Models
situation (Havens, Wagner, and Anderson 2017). To capture To capture uncertainty of clinical experts’ opinion, we use
decision maker’s opinion with its uncertainties, linguistic interval data. When decision makers are ML models, we can
variables and computing with words paradigms have at- capture the uncertainty of ML models using prediction in-
tracted the researchers recently (Khorshidi and Aickelin tervals. Here, we describe how we can generate intervals as
2020). In these techniques, opinions as words encode to the prediction of ML models based on the approach we in-
fuzzy sets (Borovička 2019), cloud models (Khorshidi and troduced in (Maadi, Aickelin, and Khorshidi 2020). Let C
Aickelin 2020), intervals (Wu, Mendel, and Coupland be an ML classifier such as decision tree or logistic regres-
2012), to name a few, and computational analysis on them sion. From a training dataset, we can generate different
provides decisions. Expecting exact values from clinical de- training datasets using bagging method. Bagging is a sam-
cision makers is unrealistic. They should be given an oppor- pling strategy proposed by Breiman (Breiman 1996). In this
tunity to express their opinions with a level of uncertainty. method, some samples are elicited randomly from the train-
So, in the proposed collaborative decision making approach, ing dataset with replacement and generate a new training da-
we capture the uncertainty using intervals. We ask each clin- taset named as bag. By training the ML model on different
ical expert to give their opinion as an interval and show the bags, we have different classifiers. Applying them on the
level of the uncertainty using the width of the interval. test dataset generates multiple probabilities related to class
In ML models, data uncertainty and model uncertainty are prediction. Using these probabilities, we can generate an un-
two important sources of uncertainty. To capture uncertainty certainty interval (UI) for the prediction of the ML model.
of ML models by intervals, we recently have introduced an UI captures the uncertainty of the ML model (Maadi, Aick-
interval modelling technique to capture uncertainty in en- elin, and Khorshidi 2020).
semble learning (Maadi, Aickelin, and Khorshidi 2020). In Suppose 𝑝1 , 𝑝2 ,…, 𝑝𝑏 as probabilities determined by b
this technique, for each ML model in an ensemble, an inter- classifiers (generated using bags), an UI for ML model is
val is generated as a prediction. In the proposed collabora- generated by calculating the first quartile (𝑄1 ) and the third
tive decision making model, we use this technique to capture quartile (𝑄3) of the probabilities as (1).
uncertainty of each ML model by intervals.
Figure 1: Generating an Uncertainty Interval for an ML Model on a Dataset to Capture Uncertainty
UI = [𝑄1 , 𝑄3 ] (1)
According to (1), UI is defined as the inter quartile Collaborative Human-ML Decision Making
range of 𝑝1 , 𝑝2 ,…, 𝑝𝑏 . In Figure 1, the framework of gen- Model
erating uncertainty intervals for ML models is shown.
We consider clinical decision making problem as a classifi-
cation problem. Considering three classifiers (ML models)
Interval Agreement Approach (IAA) and three clinical experts as decision makers, the process of
IAA is an aggregation function to aggregate decision mak- collaborative decision making is depicted in Figure 2. In this
ers’ opinions while the opinions are presented as intervals. process, decision makers determine the probability that a
IAA aggregates intervals to a fuzzy set. Let 𝐴 = test sample belongs to the main class and present it as an
{𝐴1 , … , 𝐴𝑚 } be a set of intervals given by m decision makers interval. For example, in a cancer diagnosis problem, deci-
as their opinions where 𝐴𝑖 = [𝑙𝑖 , 𝑟𝑖 ] (i = 1, 2, …, m). Ag- sion makers determine the probability that the test sample is
gregating intervals of set A generates a Type 1 Fuzzy Set malignant.
(T1 FS) in IAA with the membership function of 𝜇𝐴 which In the proposed model, both classifiers and clinical ex-
is defined as (2) (Wagner et al. 2015). perts have access to the electronic health records. The rec-
yi ords are used to train classifiers. Using generating uncer-
μA= ∑m
i=1 ⁄ m-i+1 m-i+2 (2)
(⋃j =1 ⋃j =j +1 … ⋃mj=j +1 ( Aj1 ∩… ∩ Aji )) tainty intervals explained before, we have an interval as the
1 2 1 i i-1
In (2), yi = i⁄m is the degree of membership and ‘/’ refers output of each classifier. This interval shows the probability
to assignment of degree of membership. The degree of of belonging a test sample to the main class. Also, clinical
membership in IAA is related to the number of intervals experts are asked to give their opinions about the test sample
overlapped in a point. So, the value of one for the degree of as intervals. IAA aggregates all intervals to a T1 FS. This
membership in a point shows all intervals overlap at that fuzzy set shows how much all decision makers are in agree-
point. To simplify (2), 𝜇𝐴 can be written as (3) for a point ment. A T1 FS can be shown as a list of tuples which each
like x. tuple indicates different region of change over the member-
ship function as (4).
∑m
i=1 μA
̅ (x)
i
μA (x)= (3)
m 𝑇𝐹𝑆 = [𝑅1 , 𝑅2 , … , 𝑅𝑣 ], 𝑅𝑖 = ([𝑅𝑖𝑙 , 𝑅𝑖𝑟 ], 𝑅𝑖ℎ ) (4)
1 𝐿𝐴̅ 𝑖 ≤ 𝑥 ≤ 𝑟𝐴̅ 𝑖 Where 𝑙 is the left point, 𝑟 is the right point and ℎ is the
Where 𝜇𝐴̅ 𝑖 (𝑥) = {
0 𝑒𝑙𝑠𝑒 height or the membership function value of the tuple 𝑅𝑖 cal-
culated using (3).
Figure 2: Collaborative Human-ML Decision Making Framework
To make the collaborative decision about a test sample, we Experimental Analysis Using Synthetic Data
calculate the centroid of this fuzzy set. The centroid of the
fuzzy set is calculated using (5). To show how the proposed collaborative approach works,
we use Liver Disorders dataset from UCI as electronic
∑𝑣𝑖=0(𝑅𝑖ℎ ×𝑅𝑖𝑙 )+(𝑅𝑖ℎ ×𝑅𝑖𝑟 ) health records (Dua 2019). This dataset has 345 samples and
𝑐𝑒𝑛𝑡𝑟𝑜𝑖𝑑 (𝑇𝐹𝑆) = ∑𝑣𝑖=𝑜 2(𝑅𝑖ℎ )
(5)
5 features that are all blood tests which are related to liver
disorders arise from alcohol consumption. In this dataset,
If the value of the centroid is more than 0.5, it shows the target value is alcohol consumption and features’ values are
test sample belongs to the main class. For example, in the integer numbers. To have a bi-class classification dataset,
cancer diagnosis problem, if the centroid is more than 0.5, we follow the strategy described in (Turney 1994). We as-
it shows the test sample is malignant. sign class 0 to the number of drinks less than 3, and class 1
In the proposed approach, accompany with making a col- to the number of drinks equal or more than 3. This dataset is
laborative decision, we can evaluate the performance of a balanced dataset.
both groups of decision makers separately. Specially in high In this experiment, three classifiers of Decision Tree
risk conditions, this provides us important information to (DT), Logistic Regression (LR) and Gaussian Naïve Bayes
make decision in a timely manner. (GNB) are considered as ML models. We split the dataset to
In the proposed collaborative decision making approach, training dataset and test dataset with the ratios of 75% and
we can calculate the width of the intervals presented by clin- 25% respectively. Regarding the proposed approach, we
ical experts and ML models as a measure for uncertainty. calculate uncertainty intervals for classifiers. For clinical ex-
The width of intervals provides us information about how perts’ opinion, we generate synthetic data as their decision
much certain clinical experts and ML models are in their de- (the probability of assigning one sample to class 1, here as
cision making separately and collectively. So, we can rec- the main class) to examine different scenarios in collabora-
ognize decision makers that are highly deviated from the tive decision making approach.
consensus to extract informative insights for updating the In the first scenario, we assume that both groups of deci-
decision-making process. sion makers (ML models and clinical experts) make similar
In Algorithm 1, the steps of the proposed collaborative decisions and have close uncertainties. So, we construct
decision making approach for m ML models (classifiers) three synthetic intervals as clinical experts’ opinions using
and n clinical experts is described. three uncertainty intervals generated by classifiers. To gen-
Algorithm 1: Collaborative expert-ML decision making process erate one opinion interval from one uncertainty interval, we
Input: randomly select two numbers from the range of numbers be-
1. Electronic health records
2. Test sample u tween endpoints of the uncertainty interval added by an
3. Collection C1, C2, …, Cm of classifiers and these numbers are the endpoints of the opinion interval.
4. Clinical experts’ opinions about the test sample u (n clinical experts)
Results of this scenario in three different experiments are
5. The number of randomly picked samples in bagging (V)
6. The number of bags generated using bagging (H) shown in Table 1.
Output: The decision about getting infected to the disease for test sample Performance Collaborative ML Clinical
Experiments
u measure model models experts
1. For i from 1 to m do Accuracy 0.790 0.795 0.784
2. For j from 1 to H do F-score 0.870 0.873 0.866
3. Select a bag of training samples (V samples with replacement) 1 G-mean 0.563 0.564 0.555
using bagging Width of the
0.156 0.151 0.161
intervals
4. Train classifier Ci on the selected bag
5. Compute the probability for the given test sample u using the Accuracy 0.775 0.770 0.770
F-score 0.859 0.856 0.855
trained classifier and assign it to Pj
2 G-mean 0.560 0.556 0.556
6. end Width of the
7. Compute the first quartile of {P1, P2, …, PH} and assigns it to Q1i intervals
0.150 0.144 0.155
8. Compute the third quartile of {P1, P2, …, PH} and assigns it to Q3 i Accuracy 0.773 0.771 0.773
9. Determine the uncertainty interval [Q1i, Q3i] for classifier Ci re- F-score 0.858 0.856 0.857
garding test sample u 3 G-mean 0.539 0.537 0.553
10. end Width of the
11. Ask the probability of belonging the test sample u to the main class as an 0.154 0.149 0.158
intervals
interval from clinical expert j and named it [DLj, DRj] (j = 1, 2, …, n) Table 1: Collaborative and Individual Results for Two Groups of
12. Aggregate [Q11, Q31], [Q12, Q32], …, [Q1m, Q3m], [DL1, DR1], [DL2,
DR2], …, [DLn, DRn] using IAA and generate a T1 FS
Decision Makers (ML Models and Clinical Experts) in Scenario 1
13. Calculate the centroid of the T1 FS
14. If the centroid >= 0.5 In Table 1, three experiments show possible situations in
Return ‘Test sample u belongs to the main class’ scenario 1 where each group of decision makers (collabora-
Else tive model and individual models) has relatively better per-
Return ‘Test sample u belongs to the subordinate class’ formance than others. However, the difference among the
performance of these three groups of decision makers is not
significant. According to this table, the decision-making re- data as experts’ opinions, we use the strategy mentioned in
sults in terms of accuracy, F-score and G-mean are similar previous scenarios to generate opinions intervals. Then, we
to each other for three decision maker groups in all experi- consider the middle points of the intervals as clinical ex-
ments. Also, in this scenario the widths of intervals are close perts’ point predictions to determine the class label of test
together for all groups. We know that always the width of samples. Finally, using majority voting on point predictions
the intervals for collaborative model is between the width of generated by ML models and clinical experts, we determine
the intervals for clinical experts and ML models. Totally, we consensus point prediction for test samples. The results on
can say that the performance of the collaborative model in comparing the proposed collaborative model to its equiva-
the case that the performance of two individual groups is lent point prediction model are shown in Table 3. In the first
like each other is better than at least one of them. experiment, the opinion intervals are generated according to
In the second scenario, we examine the effect of the bad
scenario 1 and in the second experiment opinion intervals
performance of one of decision makers’ groups on collabo-
are generated according to scenario 2.
rative decision making. We generate opinions’ intervals so
that the predictions for clinical experts are far from the real Performance Proposed Point prediction
Experiments
class of the test samples to some extent. In experiment 1 of measure model model
Accuracy 0.780 0.774
this scenario, we generate intervals with wider width and in 1 F-score 0.862 0.858
experiment 2, we generate intervals with narrower width. G-mean 0.555 0.555
The results related to this scenario are shown in Table 2. Accuracy 0.766 0.715
2 F-score 0.849 0.813
G-mean 0.602 0.602
Performance Collaborative ML Clinical
Experiments
measure model models experts
Table 3: Proposed Collaborative Model Compared to Its Equiva-
Accuracy 0.766 0.774 0.333 lent Point Prediction Model in Scenario 3
F-score 0.849 0.859 0.238
1 G-mean 0.602 0.534 0.353
Width of the The results in Table 3 show better performance of the pro-
0.327 0.154 0.500
intervals posed model than its equivalent point prediction model in
Accuracy 0.773 0.777 0.336
F-score 0.855 0.861 0.251
terms of accuracy and F-score. It means that interval mod-
2 G-mean 0.639 0.524 0.182 elling in the proposed collaborative model improves deci-
Width of the
intervals
0.121 0.153 0.09 sion making through capturing uncertainty.
Table 2: Collaborative and Individual Results for Two Groups of Regarding all described scenarios and experiments, we
Decision Makers in Scenario 2 conclude that the proposed collaborative model can be an
effective model to capture uncertainty, use clinical experts’
The results in Table 2 shows the power of interval mod- privileged information and implement the power of ML
elling as well as IAA as interval aggregation function in de- models in clinical decision making. However, real datasets
cision making. In the cases that one group of decision mak- and scenarios can present more accurate analysis on the per-
ers makes bad decisions, results show it does not affect the formance of the proposed collaborative human-ML decision
collaborative decision significantly. So, the proposed model making model.
is robust and is suitable for critical collaborative clinical de-
cision making. Considering Table 2, we conclude that when
it comes to the width of the intervals as a measure for uncer- Conclusion
tainty, we can investigate the decision made by the group In this paper, we use expert’s privileged information in ad-
with high value of width of intervals. Decision making by dition to ML models in clinical decision making and present
this group of decision makers can affect decision quality. So, a collaborative human-ML decision making model. In the
we can ignore them in decision making process. proposed model, both clinical experts and ML models are
In the third scenario, we investigate the effect of interval considered as decision makers. To handle the uncertainty of
modelling in the proposed model on decision making per- decision making in collaborative model, we use intervals.
formance. We believe that the proposed interval modelling Clinical experts’ opinions are asked as intervals, and we de-
improves decision making through capturing uncertainty. velop an approach to have intervals as the outputs of ML
To test it, we compare the performance of the proposed col- models. IAA as a powerful interval-based aggregation func-
laborative model with its equivalent point prediction model. tion is used to aggregate decision makers’ interval predic-
In this scenario, we use majority voting as the most common tions to a T1 FS. The generated FS is used to determine the
used aggregation function in point prediction approaches. final consensus decision. In the proposed collaborative
To create point prediction model, for each ML model like model, the width of the intervals is a measure for decision
DT, we create different classifiers using bagging, then we makers’ uncertainty. So, it provides information about the
uncertainty of each group of decision makers to improve de-
use majority voting to determine the class label of each test
cision making. To show how the proposed model works, we
sample using classifiers. To create synthetic point prediction
consider different scenarios and experiments using synthetic Canberra, Australia.
data and test the performance of the proposed model. Re- Maadi, M.; Khorshidi, H.A.; and Aickelin, U. 2021. A Review on
sults show the power of intervals and interval modelling in Human–AI Interaction in Machine Learning and Insights for
the proposed collaborative model to capture uncertainty and Medical Applications. International Journal of Environmental
making more effective and robust decision in clinical deci-
Research and Public Health 18 (4): 1–27.
sion making. Also, as a significant result, we observe that
weak performance of a group of decision makers does not Turney, P.; 1994. Cost-Sensitive Classification: Empirical
affect the collaborative decision in the proposed model sig- Evaluation of a Hybrid Genetic Decision Tree Induction
nificantly. For the future work, we are collecting real data to Algorithm. Journal of Artificial Intelligence Research 2: 369-409.
examine our proposed model in different scenarios. Also, Vapnik, V.; and Vashist, A.; 2009, A New Learning Paradigm,
we will develop the proposed method for multi-class classi- Learning using privileged information, Neural Networks, 22 (5-6):
fication decision making problems. 544-557.
Vapnik, V.; Vashist, A.; and Pavlovitch, N. 2009. Learning using
References Hidden Information (Learning with Teacher). In the international
Joint conference on Neural Networks, 3188-3195
Alahmari, S.; Goldgof, D.; Hall, L.; Dave, P.; Phoulady, A.H.; and
Wagner, C.; Miller, S.; Garibaldi, J.; Anderson, D.; and Havens, C.
Mouton, P. 2019. Iterative Deep Learning Based Unbiased
2015. From Interval-Valued Data to General Type-2 Fuzzy Sets.
Stereology with Human-in-the-Loop. In Proceedings - 17th IEEE
IEEE Transactions on Fuzzy Systems 23 (2): 248–69.
International Conference on Machine Learning and Applications,
ICMLA, 665–670. Orlando, FL, USA. Wrede, F.; Hellander, A.; and Wren, J. 2019. Smart Computational
Exploration of Stochastic Gene Regulatory Network Models Using
Borovička, A. 2019. New Approach for Estimation of Criteria
Human-in-the-Loop Semi-Supervised Learning. Bioinformatics 35
Weights Based on a Linguistic Evaluation. Expert Systems with
(24): 5199–5206.
Applications 125 (July): 100–111.
Wu, D.; Mendel, J.; and Coupland, S. 2012. Enhanced Interval
Breiman, L. 1996. Bagging Predictors. Machine Learning 24 (2):
Approach for Encoding Words into Interval Type-2 Fuzzy Sets and
123–40.
Its Convergence Analysis. IEEE Transactions on Fuzzy Systems 20
Cai, C.; Reif , E.; Hegde, N.; Hipp, J.; Kim, B.; Smilkov, D.; (3): 499–513.
Wattenberg, M.; Viegas, F.; Corrado, G.; Stumpe, M.; and Terry,
Zerilli, J.; Knott, A.; Maclaurin, J.; and Gavaghan, C. 2019.
M. 2019. Human-Centered Tools for Coping with Imperfect
Algorithmic Decision-Making and the Control Problem. Minds
Algorithms during Medical Decision-Making. In Conference on
and Machines 29 (4): 555–78.
Human Factors in Computing Systems, CHI, 1–14. Glasgow,
Scotland, UK.
Dua, D. and Graff, C. 2019. UCI Machine Learning Repository:
Citation Policy. Irvine, CA: University of California, School of
Information and Computer Science.
Havens, T.; Wagner, C.; and Anderson, D. 2017. Efficient
Modeling and Representation of Agreement in Interval-Valued
Data. In IEEE International Conference on Fuzzy Systems,1-6.
https://doi.org/10.1109/FUZZ-IEEE.2017.8015466.
Huang, Q.; Chen Y.; Liu, L.; Tao, D.; and Li, X. 2020. On
Combining Biclustering Mining and AdaBoost for Breast Tumor
Classification. IEEE Transactions on Knowledge and Data
Engineering 32 (4): 728–38.
Itani, S.; Lecron, F.; and Fortemps, P. 2019. Specifics of Medical
Data Mining for Diagnosis Aid: A Survey. Expert Systems with
Applications118(March):300–314.
Khorshidi, H.A.; and Aickelin, U. 2020. Multicriteria Group
Decision-Making under Uncertainty Using Interval Data and
Cloud Models. Journal of the Operational Research Society 1-15.
Maadi, M.; Aickelin, U.; and Khorshidi, H.A. 2020. An Interval-
Based Aggregation Approach Based on Bagging and Interval
Agreement Approach in Ensemble Learning. In IEEE Symposium
Series on Computational Intelligence, SSCI 2020, 692–99.