The key direction in modern medicine is the development of software systems that allow us to analyze a large amount of data in an adaptive way and interpret the results obtained, ensuring high accuracy of results. Predicting mortality from myocardial infarction (MI) and identifying significant factors influencing this mortality is an urgent task, since the share of cardiovascular diseases annually accounts for more deaths than any other cause. The purpose of this study is to develop a machine learning model based on the gradient boosting technique for predicting mortality from MI, identifying the most significant signs, assessing the influence of meteorological factors, and improving the accuracy of predicting using data balancing methods. The paper used a depersonalized sample of all residents of the Voronezh region who had a STROKE in 2015-2017. The archive of weather data for 2015-2017 was taken from the site rp5.ru. Data analysis and model development were performed in the Python programming language version 3.6. Five undersampling strategies were reviewed and the method that achieves the highest prediction accuracy was chosen - the cluster centroid method.
Myocardial infarction (MI) is the most common and most fatal problem [
Machine learning algorithms are often referred to as black box models because there
is no clear explanation of how exactly the model comes to predictions. Medicine is an
industry where the interpretation of results is an important step. When determining
signs that affect the risk of disease, you can eliminate these factors in a timely manner
with the help of targeted medical intervention. Many works are devoted to identifying
the dependence of the morbidity and mortality of the population on climatic conditions
[
Thus, the purpose of this study was to construct models for predicting one-year mortality among residents of the Voronezh region who underwent MI, based on meteorological, socio-demographic, and clinical factors. The initial data were provided by the Voronezh Regional Cardiology Dispensary.
To build the model, one of the popular methods was chosen – gradient boosting,
which is highly accurate [
In recent years, many works have been published devoted to predicting mortality
from cardiovascular diseases, including those using gradient boosting methods [
During processing real data, there are often situations when the share of examples of one class in the training dataset is too small (minority class), and another is strongly represented (a majority class). Depending on the task, the problem of class imbalance can lead to a serious bias towards the majority class, a decrease in predicting efficiency and an increase in the number of false predictions. In tasks of a medical nature, this is especially important. When analyzing diagnostic results, there is often a problem of uneven distribution of classes in the training sample, since the number of patients suffering from a particular disease in the initial data is significantly lower (or, conversely, higher) than the number of healthy patients. Misclassification of the patient may lead to a lack of timely action. One approach to solve this problem is to use various sampling strategies. There are two ways to restore class balance. In the first case, delete a certain number of examples of the majority class (undersampling), in the second – increase the number of examples of the minority class (oversampling). This article discusses various undersampling methods and compares their accuracy. 2 2.1
For the analysis, we used depersonalized data on all patients who were admitted to Voronezh region’s hospitals in 2015-2017 with a diagnosis of MI, and dependencies in the sample with fatal cases of myocardial infarction (MI) for the same years. Total recorded cases of MI in 2015 – 3810, 2016 – 3837, 2017 – 3679. In 2015, 2016 and 2017, there were 684, 657 and 606 fatal cases of MI, respectively.
The source file contained information on 15 attributes presented in Table 1. The analyzed dataset was supplemented with six meteorological indicators. Weather data were downloaded from the rp5.ru website archive. Data preprocessing was performed using the Oracle SQL developer integrated development environment in the SQL language.
Attribute’s type Categorical variable Continuous variable
Attribute Gender, whether the myocardial infarction is repeated (MI), localization, KILLIP class, whether the patient underwent thrombolytic therapy (TLT), percutaneous coronary intervention (PCI), whether the patient has a history of diabetes mellitus (DM), atrial fibrillation (AF), acute cerebral circulation disorder (CCD), chronic obstructive pulmonary disease (COPD), chronic cardiovascular failure (CHF), arterial hypertension (AH) Age, Maximum of the temperature (Max_T), humidity, atmospheric pressure, wind speed, cloudiness When working with a real sample of data, there is often a situation often when the volume of one class are much larger than the volume of another. This data is called unbalanced. Building a model in such a situation may turn out to be ineffective and the error of the predicted data will be large. For medical issues, the data balancing stage is important. The predominance of the number of cases of one class leads to a shift in the model towards the majority class. Mortality prediction is a type of "early warning" tasks of an event. It is more important to predict the onset of death than to assume that the patient will survive and get the opposite result. Since the number of patients who died in the initial sample is much less than the number of survivors, it is necessary to balance the training data to correctly build a predictive model and reduce false negative results.
The class with the largest number of cases is called majority, with the smallest
minority. The class of the deceased in the problem under consideration is the minority
class. Various sampling strategies are used to correct the class imbalance [
Killip class (%)
I II III
VI Age PCI (%)
Yes
No CHF (%)
H I H IIА H IIБ H III
No CCD (%)
Ischemic stroke TIA Hemorrhagic stroke
No COPD (%)
Yes
No Cloudiness Max_T Undersampling is considered the simplest and at the same time the most correct in the tasks of medical research. Therefore, it was this method that was chosen to solve the problem. The undersampling technique can be implemented in several ways. To achieve the highest prediction accuracy, five algorithms were considered, and the accuracy of their work was compared.
instances of the majority class are randomly deleted. The number of deleted records is determined empirically. The advantages of this method include simplicity of implementation and high performance. The main drawback is the high probability of losing significant data.
The method of cluster centroids. This strategy removes examples of the majority class using cluster analysis methods. The majority class is divided into the number of classes according to the number of instances of the minority class by the k-average method. Then the centroids of each class are selected, which eventually form a new class. The main advantage of this approach is that it preserves important topological properties of the sample.
Strategy for searching for Tomek links. All entries of the majority class that are included in the Tomek link are considered "noisy" and deleted. A pair (Ei, Ej) is called a Tomek link if there is no example of El such that the set of inequalities is valid: (, ) < (, ) {(, ) < (, ) (1) d – is the distance between the samples Ei, Ej.
the nearest neighbor. It then removes any instance whose class label differs from the
class in at least two of its three nearest neighbors. The idea of this method is to remove
instances from a majority class that is near or around the boundary of another class in
order to improve the accuracy of classifying minority instances rather than majority
instances [
The general parameter of XGBoost. This is a built-in data balancing method in the XGBoost – scale_pos_weight gradient boosting method, which allows you to increase the weight of the minority class examples. 2.3
The gradient boosting machine learning model was used to predict the mortality of patients from MI. Five-fold cross-validation was used to train the model to correct hyperparameters, the final testing was carried out on a deferred validation set, the volume of which is 20% of the initial data.
Gradient boosting is a machine learning technique, the main idea of which is the
iterative process of sequentially building partial models. Each new model is trained
using information about the errors made at the previous stages, and the resulting
function is a linear combination of the entire ensemble of models, considering the
minimization of some penalty function [
One of the advantages of this method is its high performance, which is important when working with large data sets. The gradient boosting method is resistant to outliers, which are normal when working with real data.
In machine learning tasks, various metrics are used to assess the quality of models. When calculating these metrics, a classification error matrix is used. Based on the matrix elements for each class, the following indicators are calculated: (TP) – True Positive (the number of correctly predicted samples of class 1), (FN) – False Negative (the number of false negative samples, incorrectly predicted class 0), (TN) – True Negative (correctly predicted class 0), (FP) – False Positive (number of false positives, incorrectly predicted class 1). The main metric of classification problems is the proportion of correct answers, which is calculated using the formula: accuracy = (TP+TN)/(TP+TN+FP+FN) (2)
However, this indicator is ineffective for unbalanced samples. More often, the indicators are considered precision (precision) – the proportion of true positive examples from the total number of positively predicted examples, and recall – the proportion of true positive examples from the total number of positive examples.
precision = TP/(TP+FP) recall = TP/(TP+FN) (3) (4)
Recall is preferable when evaluating models in medical problems. This metric helps
to reduce the number of false negative predictions. Medical diagnostics has its own
metrics that determine the accuracy of the method [
Specificity= TN/(TN+FP) (5)
To create an optimal diagnostic system, it is necessary to find a compromise between
the obtained indicators of sensitivity and specificity of the models. A common way to
visualize the relationship between these metrics is to use the ROC-curve – a graphical
characteristic of the quality of a binary classifier, the dependence of the sensitivity on
the indicator (1-specificity) when varying the threshold of the decision rule of the
model. The optimal position for the ROC curve is as close as possible to the upper left
corner where specificity and sensitivity are at their maximum. The value of AUC ROC
– the area under the ROC-curve is a compromise metric widely used in medical
research [
Building a machine learning model and data analysis was performed using the libraries of the Python programming language. The gradient boosting model was built using the tools of XGBClassifier library. XGBoost is an optimized distributed library that is highly efficient, flexible and portable: you can train many models with different loss functions. The developed code is applicable in the main common environments (Kubernetes, Hadoop, SGE, MPI, Task).
Table 3 presents the metrics of the quality of the model's assessment, which were obtained based on true and false answers on the test sample: sensitivity, specificity, percentage of correct answers, and shows the percentage of correct answers in the final cross-validation. Table 4 shows the most significant features. The significance of each factor is calculated as the average normalized result of the decrease in the branching criterion caused by this factor. The branching criterion calculates the measure of uncertainty at the nodes of the trees. The Gini index was used as such a criterion. The most significant feature is index of heart failure severity on the Killip scale. This is the expected result, since this indicator is determined by specialists when the patient is admitted to the hospital and characterizes the severity of the patient's condition during the initial visual examination. Age is expected to be a significant factor; survival is worse in patients belonging to the older age group. Also, predictors influencing the results include indicators: whether percutaneous coronary interventions (PCI) were performed, whether the patient has chronic heart failure (CHF), has a history of stroke (CCD) and chronic obstructive pulmonary disease (COPD). It can be noted that in comparison with clinical indicators, meteorological factors are less significant, however, there is an influence of cloudiness indicators (Cloudiness) and maximum daily temperature (Max_T).
As noted earlier, the initial sample has an imbalance in the data for the predicted variable – there are much more patients who survived after MI than who died. Therefore, despite the rather high indicators of Accuracy and AUC ROC, in Table 3 it can be seen that the constructed model has an extremely low sensitivity index (Sensitivity), which is unacceptable for the task of predicting mortality, since the model is poorly able to identify patients with a high risk of mortality.
An undersampling method is used to balance the classes, which can lead to an increase in the sensitivity of the machine learning model. The following undersampling strategies were considered: the random deletion of examples, the cluster centroid method, the Tomek link search strategy, and the concentrated nearest neighbor rule. Also considered is the built-in method of balancing data in the XGBoost gradient boosting method – scale_pos_weight.
Table 5 shows the main metrics for assessing the quality of the model: Accuracy (train) – the accuracy of the resulting model on the training set, sensitivity, specificity, Accuracy (test) – the proportion of correct answers on the test set, the area of the ROCcurve. Based on the results of the model quality metrics, we can conclude that the best indicators of accuracy when using strategies of random deletion, cluster centroids and Tomek links. In this case, the method of cluster centroids provides the highest sensitivity.
Method Random undersampling Cluster centriods undersampling TomekLinks undersampling NearMiss undersampling Scale pos weight
To assess the impact of weather conditions, the same models were built for a dataset without weather attributes. Table 6 presents the results. Comparing the indicators in Table 5 and Table 6, we can conclude that considering weather conditions helps to improve the accuracy of the model and has an impact on predicting mortality from MI. All (Training) TomekLinks undersampling Random undersampling Cluster centroids undersampling
NearMiss undersampling 4
This study was conducted to build a gradient boosting model for predicting mortality after myocardial infarction and to determine the most significant factors for mortality in MI. The accuracy of the model was improved by balancing the original sample using undersampling. The accuracy of five sampling strategies has been demonstrated. The most effective methods are the method of random removal of samples, the method of cluster centroids and the method of Tomek links. The best accuracy (Accuracy) was obtained using the Tomek links method (on test data, it is equal to 0.85). The best sensitivity is provided by the cluster centroid method.
With the help of the constructed model, significant factors were found for predicting mortality after the onset of myocardial infarction. The most significant are: Killip class, age, percutaneous coronary interventions, whether the patient has chronic heart failure, whether there is a history of stroke and chronic obstructive pulmonary disease, as well as weather factors – cloudiness and maximum daily temperature.
Acknowledgments. The research was supported by the Russian Foundation for Basic Research (RFBR), project № 20-37-90029.