The paper is devoted to the development of an approach to machine learning for Big Data under epistemic and aleatoric uncertainties which are taken into account with the help of corresponding minimax criteria. The keystone of the method is processing subsets of training data in parallel, using the partitioning based on complexity is analyzed and compared with sequential data processing. An example from a medical application is considered, where the method is investigated for different learners and resampling strategies.
performance algorithm for solving the Machine Learning (ML) problems. To cope with Big Data parallel computing is one of the most effective solution in the case of ML. It leads to the necessity of the partitioning on Big Data sets. Voronoi diagrams are traditionally used for such type of problems.
problems [
The upper limit of the probability of incorrect classification can be used as an explicit indicator to
assess the reliability of classification models. A version of MPM with parametric reduction was
proposed in [
2022 Copyright for this paper by its authors.
The problem of supervised ML, which means the prediction of with the help of , loss function , and the set of probability distributions on ( , ) can be formulated as minimax problem with the respect to , provided that the maximization is due to all possible distributions and minimization is with the respect to decision rules ∈ .
∈ ∈ [ ( , ( ))] where [•] is expectancy.
maximum principle. Mathematical description of the problem of supervised ML in systemic medical research was presented in [21], [22]. Here we formulate it in the case of minimax criterion. Mathematically ML problem for the systemic medical research is based on the following data. We have dataset , which includes tuples
D X i | i 1, N
In order to model aleatory uncertainties, consider supervised ML regarding the distribution of learning tuples. For the class of all subsets of we introduce , including the distributions of classes of training and testing datasets (Dtrain, j , Dtest, j) D D | Dtrain, j Dtest, j , Dtrain, j Dtest, j D, j 1,2N ,, where Dtrain, j and Dtest, j are all possible datasets for training and testing correspondingly. In practice, resampling strategies are distributions of the classes of tuples which are characterizing aleatoric uncertainties the best. We introduce the resampling strategies ⊂ (Dtrain,k , Dtest,k) D D | Dtrain, j Dtest, j ,k Dtrain,k k Dtest,k D, (2.4) (2.1) (2.2) (2.3) (2.5) 2.1.
Each th tuple = ( 1, 2, . . . , , ) consists of input data ( 1, 2, . . . , ) (called by also attributes) and output data .
Let raw = ( 1, 2, . . . , ) present the value of th attribute of all tuples. Output attribute = ( 1, 2, . . . , ) includes all output data. The attributes 1, . . . , and (depending on the tasks of classification or regression) can accept both numerical and categorial values.
In the simplest case, the supervised ML problem is to predict, using the certain predictor, the value of output attribute +1 based on the values of attributes 1 +1, . . . , +1. The predictor should maximize the accuracy of prediction of output attribute, namely the probability { 1 +1, . . . , +1} for arbitrary ∈ {1, . . . , }1. Further, applying minimax approach, we introduce ℎ ∈ for the considered class of ML models ℎ( , ), which can be trained and tuned for the data ⊂ and assessed taking into account certain resampling strategies . Comparing different ML models, the goal is to minimize expected losses. But you also need to consider resampling strategies, which should also assess the loss function. This formulation of the ML problem considers two types of uncertainty. Namely, uncertainty in oversampling is aleatory because it is related to data. At the same time, the uncertainty in the choice of models is epistemic. Mathematically, the minimum problem of MN is described as a search for a model ℎ due to ℎ∈ ∈ [ ( , ℎ( , ))] strategy the ones with the greatest differences. The principal components analysis method (PCA) is one of the widely used methods of dimensional reduction. Although it is used for unsupervised ML problems, it helps us refine the results when used for supervised ML, such as a classification or regression problem. The task of reducing the number of attributes is extremely important for medical use in interpreting the results. Below we propose a method of its application in conditions of aleatory uncertainty.
When regarding the Voronoi tessellation, the dimension reduction algorithm has to be applied for each Voronoi cell. Moreover, since in the minimax ML (machine learning) problem the loss function is calculated for all resampling strategies, the dimension reduction algorithm must be applied separately for each strategy . We can present arbitrary resampling strategy as = ( ), where ( ), ∈ 1, is lth sample of indices from 1 to N, which corresponds to training tuples of the Let ( ) be input data coming from , if sample of indices ( ) were applied. Namely, ( ) = {(
1, 2, . . . , , ) } the lth training sample.
∈{1... }∩ ( )
, where Nl < N is the number of training tuples in
= {( 1, 2, . . . , , ) } =1 , Output: principle components together with the attributes. 1 transform data D into the matrix A including all numerical entries; 2 apply resampling strategy for А: = {( 1, , 2, , . . . , , , , ) } =1 ∈ 1+1× , = 1, ; 3 4 5 6 7 8 9 , := 1
calculate : = 1 1 for each ( ), = 1, do ∑ =1 , , = 1, 1; ( ): = ∑ 1 =1
( , ); ( , ), = 1, 1; ′ (
′ ) ∈ 1., ′ : = { , − , } =1, 1, =1,
∈ 1× .; calculate eigenvalues: ,1 ≤ ,2 ≤. . . ≤ , 1 10 calculate
eigenvectors 11 , 1respectively Var(PC1 ), ExplainedVar(PC2 ): = Var(PC2 ); Var( ) ExplainedVar(PC12): =
∑ ExplainedVar(PC12) 1 =1 17 return names of attributes з ( , 1 ), = 1, і ( , 1−1), = 1,
values of raws (Step 4), variance
( , ), = 1, 1 (Step 5), general variance (sum of sample variances)
( ) (Step 6), deviation matrix ′ ∈ 1× (Step 7), covariance matrix ∈ 1 (Step 8), eigenvalues of matrix due to increasing order (Step 9), eigenvectors (Step 10). Here we consider eigenvectors , 1
and , 1−1 ∈ 1, which correspond to , 1and , 1−1 respectively. At the step 11 we get two principle components (PC2l) respectively (Step 13). Next, we organize the values of the eigenvectors , 1і , 1−1 in descending order of their absolute values. For this purpose, we use permutations ( , 1) and ( , 1−1). We use the denotion ( ) for a permutation that organizes the vector x in descending order of the absolute values of its elements.
Next we return the names of the first ExplainedVar (PC1l) 100% attributes in permutation ( , 1 ) and the first ExplainedVar (PC2l) 100% attributes in permutation ( , 1−1) (Step 14).
resampling strategy \ gamma (Step 16). Finally, we return the names of the first ExplainedVar (PC1l) 100% attributes, which are most common in permutations ( , 1 ), = 1, and the first ExplainedVar (PC2l) 100% attributes that are most common in permutations ( , 1−1), = 1, (Step 17).
As a result of reduction of dimension we receive some numerical matrix = ∈ 2+1× , 2 ≤ 1. These data can then be used as training to solve ML {( 1; 2, . . . , 2, ) }
=1 problems based on the minimax approach.
convert all categorical attributes that are widely used in systemic medical research into boolean data. Second, when considering the two main components traditionally used for planar presentation of training kits, we propose an approach to selecting some reduced number of attributes for further research (e.g., developing a ML model). This number is related to the number of variations explained. The latter assumption allows us to truly reduce the size of ML problems in systemic medical research under uncertainty.
is low. In such cases, we need to take into account the components PC3, PC4 and so on to obtain the appropriate dispersion. Steps 10-14 and other algorithms should be changed accordingly.
Note 3. It should be noted that the PCA should be calculated depending on the resampling strategy, as the PCA is applied to training tuples. ( ) (not for the whole data set D). Therefore, in Step 14, different features may be selected depending on the sample of indices the selection of attributes in the last step of the algorithm for the entire resampling strategy. ( ). In turn, this affects 2.2.
minimax approach with the possibility of accurate, acceptable and stable results. The MN model, formulated under conditions of certainty, is presented in [22]. Here, we summarize a flowchart for solving the problem under uncertainty in both the model and the resampling strategy.
collected in EMR systems. Methods of importing data sets from EMR systems are presented. Note that the choice of open source EMR systems over commercial ones is extremely important because it allows open access to clinical data that can be processed and selected for subsequent stages of ML [22].
Then we should define the task from the point of view of MN. This can be regression, classification, grouping, and so on. Resampling strategies are also defined. For example, resampling strategies supported by the mlr package include: cross-validation (cv), cross-validation (LOO), re-crossvalidation (RepCV), color subsampling, also called Monte Carlo cross-validation (Subsample), Holdout method (training / testing) (Holdout) [23]. In a real application, we are dealing with a large number of attributes. Only some of them can be important for the tasks of MN. Therefore, it is natural to try to reduce the dimension by discarding the attributes with the largest deviations.
the choice of parameters for the methods, which affects the accuracy of the model. In the next cycle, configure the parameters for each model with Ѱ based on all resampling strategies , which are used.
Prepar ation of data Determ ining task Regression, classification, grouping, ...
Voronoi tesselat ion Minimax ML model Result voting for Voronoi cells Resam pling strateg
ies ∈ Γ Choos
ing model ℎ ∈ Ψ
Minimization of ℎ ∈ Ψ Dimension reduction Tuning model ℎ( , γ) Assessment of loss function ( , ℎ( , γ)) Maximization of ∈ Γ
a set Ѱ, which includes the method of a 4-layer neural network with the number of neurons , , , on layers based on inverse error propagation and method C5.0 induction of decision tree height ℎ.
≔ ( + ⌈2⌉)
Error! Reference source not found.:
Computational complexity of resampling based on -fold cross validation is Error! Reference source not found.: : = ( #( )( +
+ )) 5.0 ≔ (ℎ#( )
#( )) = ( #( )) (2.6) (2.7) (2.8)
= ( + 5.0) (2.9)
evidence with clinical experience and patient expectations [30]. They are aimed at improving health care in the future. Systematic medical research helps doctors and researchers gain knowledge about human health and disease. They also allow you to find more effective ways to prevent and treat disease. Assessment of health is based on a comprehensive and systematic examination of the patient, which includes history, objective examination of the body, analysis of laboratory blood tests and various secretions, instrumental and interventional studies, including X-ray, CT, MRI, endoscopy, biopsy and others methods.
Nowadays, cardiovascular diseases attract attention because they are "the number one cause of death in the world" [31]. In the study of cardiac diseases, there are quite a number of nuances and indicators that experts pay attention to during diagnosis. Diagnostic criteria include both physical tests and history, as well as laboratory, instrumental research methods. During the survey, the doctor may ask questions about the patient's family members (genetic predisposition), lifestyle and habits. Physical inactivity (sedentary lifestyle), unhealthy diet, alcohol consumption and smoking significantly increase the risk of cardiovascular disease. During laboratory studies, much attention is paid to the assessment of the level of lipids and their fractions (lipid profile). It includes indicators of total cholesterol, triglycerides, high, low, very high and very low lipoprotein density, as well as the level of atherogenicity. Lipid imbalance increases the risk of atherosclerosis. Among other things, the patient's overweight is one of the dangerous risk factors for heart disease. Blood glucose and glycated hemoglobin are among the most important indicators of carbohydrate metabolism in the body and markers of diabetes. Diabetes is a separate disease, but its presence significantly increases the risk of cardiovascular disease. In addition to the risk assessment, the necessary extended hematological, biochemical and instrumental studies are performed. In addition to the general blood test, the patient's blood pressure is measured, the following instrumental methods are used: electrocardiogram (ECG), Holter monitoring, echocardiography, coronary angiography, MR angiography.
This experimental study includes data from 1651 patients diagnosed with myocardial infarction. The target attribute of forecasting is life expectancy. Each patient's data includes 97 attributes that contain both numerical and categorical values. Such information includes data on the type of heart attack (focal or transmural), the location of the heart attack (anterior or posterior). Mortality information (hospital, short-term and long-term) is also used. The presence of concomitant pathologies is described. And here we use a detailed analysis, because such pathologies can be combined. Risk factors typical of cardiovascular diseases are investigated, namely, clinical evaluation includes data on such risk factors as gastritis, gallstone disease, lung disease, nephrological disorders, rheumatic thyroid disease, angiopathology, gastrointestinal diseases, oncology, chronic obstructive pulmonary disease, hypertension, diabetes, smoking.
The considered detailed clinical course includes indicators of vital functions, namely heart rate, systolic blood pressure and diastolic blood pressure, analyzes of heart attack complications in the form of arrhythmias, in particular, detailed heart attack complications developed in the hospital. The data lists all indicators of the general analysis of blood. Special attention is paid to leukocytes (WBC), biochemical analysis of blood is presented, information on medicines which the patient received in hospital is included. After the dimension reduction algorithm, the following features remained: sex, age, re-myocardial infarction (RMI), life expectancy after MI (death_days), body mass index (BMI), leukocyte density (White_blood_cells_count), left ejection fraction ventricle (LVEF).
base kernel (regr.ksvm), and random forest (regr.ranger).
Resampling strategies include cross-validation of cv3, cv5, cv7, cv9, cv10. The loss function L was calculated as the rmsse rmse and the training time. In the case of RMSE as an indicator of efficiency (Table 2.1), the regr.ksvm model is a solution of the ML problem based on the minimax. Namely, we first compare the error values for all the models considered. In the second step, we see that the RMSE value for the ksvm model will be minimal among the maximum. In Figure 2.2 we can see the analysis of the effectiveness of ML models with different resampling strategies for standard deviation as an indicator of efficiency.
compared to the model of SVM. In this situation, the choice of a random forest model would lead to unexpected losses arising from aleatory uncertainty.
Therefore, the minimax approach proposes to establish a resampling strategy with the maximum ("worst") value of the loss function, on which the desired model should behave best (get the minimum value of the loss function). 5. References [14] Z. Deng , L. Cao , Y. Jiang , S. Wang , Minimax probability tsk fuzzy system classifier: a more transparent and highly interpretable classification model, IEEE Trans. Fuzzy Syst. 23 (4) (2015) 813–826 . [15] K. Huang. , H. Yang , I. King , et al. , The minimum error minimax probability machine, J. Mach.
Learn. Res, 5 (4) (2004) 1253–1286 . [16] K. Huang , H. Yang , I. King , M.R. Lyu , Imbalanced learning with a biased minimax probability machine, IEEE Trans. Syst. Man Cybern. Part B 36 (4) (2006) 913–923 . [17] T. Strohmann , G.Z. Grudic , A formulation for minimax probability machine regression, in: 2002
[18] T. Strohmann , G.Z. Grudic , A formulation for minimax probability machine regression, in: 2002
[19] E. Lughofer , Single-pass active learning with conflict and ignorance, Evolving Syst 3 (4) (2012) 251–271 . [20] E. Lughofer , O. Buchtala , Reliable all-pairs evolving fuzzy classifiers, IEEE Trans. Fuzzy Syst 21 (4) (2013) 625–641. [21] V. Martsenyuk, L. Babinets, Y. Dronyak, O. Paslay, O. Veselska, K. Warwas, I. Andrushchak, and
diagnostics of the comorbid states," in 2019 10th IEEE International Conference on Intelligent
IEEE, Sep. 2019,pp313-318 doi:10.1109/idaacs.2019.8924345. [22] V. Martsenyuk, V. Povoroznyuk, A. Semenets, and L. Martynyuk, On an approach of the solution of machine learning problems integrated with data from the open-source system of electronic medical records: Application for fractures prediction, Artificial Intelligence and Soft Computing, Springer International Publishing, 2019, pp. 228-239.doi:10.1007/978-3-030-20915-5_21. [23] Resampling, https://mlr.mlr-org.com/articles/tutorial/resample.html, (Accessed on 10/08/2020) [24] Jun Ma, Liming Yang, Yakun Wen, and Qun Sun, Twin minimax probability extreme learning machine for pattern recognition, Knowledge-Based Systems 187, (2020) 104806. doi:10.1016/j.knosys.2019.06.014. [25] Zhaohong Deng, Junyong Chen, Te Zhang, Longbing Cao, Shitong Wang, Generalized Hidden
intelligent models, Information Sciences 436–437 (2018) 302-319. doi:10.1016/j.ins.2018.01.034. [26] Jun Ma ,Jumei Shen, A novel twin minimax probability machine for classification and regression,
Knowledge-Based Systems 196(2020) 105703. doi.org/10.1016/j.knosys.2020.105703. [27] Khadiev K. The Quantum Version Of Classification Decision Tree Constructing Algorithm C5.0 / K. Khadiev, I. Mannapov, L. Safina. – 2019. [28] Z. Pawlak. Rough sets. International Journal of Information and Computer Sciences 11 (1982) 341–356. [29] W. Xizhao . Learning with Uncertainty / W. Xizhao, Z. Junhai., 2016. – (Taylor & Francis Groups). [30] D.L. Sackett , W.M. Rosenberg, J.A Gray , R.B. Haynes RB, W.S Richardson , Evidence based medicine: what it is and what it isn’t, BMJ, 312(7023),(1996) 71–72. [31] Cardiovascular diseases, https://www.who.int/health-topics/cardiovascular-diseases/#tab=tab_1, 366 (Accessed on 11/24/2020).