=Paper=
{{Paper
|id=Vol-3609/paper18
|storemode=property
|title=Classification of Patients with the Development of Alzheimer's Disease using an Ensemble of Machne Learning Models
|pdfUrl=https://ceur-ws.org/Vol-3609/short4.pdf
|volume=Vol-3609
|authors=Mariia Nykoniuk,Nataliia Melnykova,Yurii Patereha,Dariusz Sala,Dariusz Cichoń
|dblpUrl=https://dblp.org/rec/conf/iddm/NykoniukMPSC23
}}
==Classification of Patients with the Development of Alzheimer's Disease using an Ensemble of Machne Learning Models==
Classification of Patients with the Development of Alzheimer's Disease using an Ensemble of Machine Learning Models Mariia Nykoniuka, Nataliia Melnykovab, Yurii Paterehac, Dariusz Salad, Dariusz Cichońe a,b,c Lviv Polytechnic National University, Stepan Bandera 12, Lviv, 79013, Ukraine d,e AGH University of Krakow, al. Adama Mickiewicza 30, 30-059 Kraków, Poland Abstract Every year, the number of diagnosed cases of Alzheimer's disease (AD) continues to grow. Dementia affects memory, orientation, language, learning ability, and the ability to perform daily activities. It is very important to correctly diagnose the stage of Alzheimer's disease, as each stage requires different treatment and support strategies for the person and their caregivers. Machine learning (ML) methods have been shown to be effective in the classification of AD patients based on medical images, such as magnetic resonance imaging (MRI). However, individual ML models often have limited performance due to overfitting or the inability to capture all of the complex patterns in the data. In this study, an ensemble of ML models is proposed to improve the classification of patients with the development of AD. The ensemble model combines the predictions of multiple individual ML models, such as Random Forest, Multi-Layer Perceptron and SVM, to produce a more accurate and robust prediction. The ensemble model achieved an accuracy of 96% in classifying patients into five stages of AD: cognitively normal, early mild cognitive impairment, late mild cognitive impairment, mild cognitive impairment, and Alzheimer's dementia. Keywords 1 Classification, Alzheimer's disease, magnetic resonance imaging, MRI, machine learning 1. Introduction Alzheimer's disease is a progressive neurodegenerative disorder that affects the brain, especially in areas related to memory, thinking, and behavior. According to the World Health Organization, by 2030, the number of dementia patients will increase by 40% to 78 million [1]. Currently, there is no cure for Alzheimer's disease, but modern rehabilitation and therapy methods can slow down the rate of disease development. There are often three stages of Alzheimer's disease, but experts also use a more detailed scheme that breaks down the stages into five categories: preclinical, mild cognitive impairment, mild dementia, moderate dementia, and severe dementia. Traditional methods for diagnosing AD, such as cognitive tests and neuropsychological assessments, are subjective and can be time-consuming. Medical imaging techniques, such as magnetic resonance imaging (MRI), can provide objective and quantitative data on the structural and functional changes in the brain that occur in AD. Alzheimer's disease is characterized by the loss of brain tissue and the formation of abnormal protein deposits that can be visualized using MRI. The classification of Alzheimer's disease stages using MRI is relevant for early detection, which allows for improvement of patients' condition through early intervention. In addition, the correct determination of the stage allows for the planning of high-quality treatment, as it directly depends on the stage of the disease. The contribution of this work is highlighted below: DDM’2023: 6th International Conference on Informatics & Data-Driven Medicine, November 17 - 19, 2023, Bratislava, Slovakia EMAIL: mariia.nykoniuk.mknssh.2023@lpnu.ua (A.1); nataliia.i.melnykova@lpnu.ua (A.2); yurii.i.patereha@lpnu.ua (A.3); dsala@zarz.agh.edu.pl (A.4); dcichon@agh.edu.pl (A.5) ORCID: 0000-0003-3521-1740 (A.1); 0000-0002-2114-3436 (A.2); 0009-0002-5110-008X (A.3); 0000-0003-1246-2045 (A.4); 0000-0003- 4198-1530 (A.5) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Wor Pr ks hop oceedi ngs ht I tp: // ceur - SSN1613- ws .or 0073 g CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings • The development of an ensemble of Random Forest, Multi-Layer Perceptron and SVM for AD classification; • ML model evaluation using multiple performance measures such as Accuracy, Recall, Precision and F1-score; • Multiclass classification by predicting 5 classes, namely CN, EMCI, LMCI, MCI and AD. 2. Methods and tools 2.1. Dataset The dataset used in this paper is from Kaggle [2]. This dataset contains MRI images of patients' brains divided into five stages of Alzheimer's disease. This set consists of 2953 JPEG files with a size of 208x176. The images are already divided into training and test images, but for convenient analysis and processing, it is worth combining them into one set. Examples of images of each stage are shown in Figure 1. Figure 1. Example of images of each stage of the disease in the set As you can see, the images have already removed non-brain tissue to focus only on brain structures. 2.1.1. Dataset preprocessing To process the dataset, the steps shown in Figure 2 are performed. This consists of the steps of checking for duplicates, removing duplicates, balancing the dataset, and normalizing the images. Figure 2. Stages of preprocessing an MRI image dataset First of all, we have checked for duplicates in this set. For this purpose, we used the SHA-256 algorithm. This algorithm is one of the most popular hashing algorithms used to find duplicate images. One of the main advantages of SHA-256 is that it is more secure and reliable because it generates a longer hash code (256 bits) than MD5 (128 bits). This means that the probability of a collision when generating a hash code for different data is very low. Thus, the chances that two different photos have the same SHA-256 hash code are very small. So, all detected duplicates have been removed and are not used for research. In addition, the study of the dataset revealed data imbalances. The problem with an unbalanced image set is that the number of images in different classes is unequal. This can lead to the learning model paying more attention to the categories with more images, thereby reducing the classification accuracy for the less represented categories. To balance the dataset, artificial images were added to the less represented classes. This is done by augmenting the data by displaying the images relative to the vertical axis and rotating them by a certain angle. The next step in data processing is image normalization. MRI brain images of the five stages of Alzheimer's disease are normalized to ensure standardization of the brightness scale. This is done to better control the effects of different light sources and other differences between images. Normalization involves converting pixel values to a range from 0 to 1 using a standard formula: 𝑋 − min(𝑋) (1) 𝑛𝑜𝑟𝑚𝑎𝑙𝑖𝑧𝑒𝑑_𝑋 = , max(𝑋) − min(𝑋) where X is the original image, min(𝑋) is the minimum pixel value of the image, max(𝑋) is the maximum pixel value of the image. For further work, the data was divided into a training, validation, and test sample, keeping the balance of classes in each set at 60:20:20. 2.2. Models used for classification Image classification is the process of assigning each image to a specific class based on its properties and characteristics. This task is usually solved using machine learning methods that provide class identification using patterns and algorithms. The task of medical image classification is to automatically identify the characteristics and features of medical images and classify them based on these features. For the study, four classification methods were selected, including Random Forest, Decision Tree, SVM, Multi-Layer Perceptron. 2.2.1. SVM One of the most popular machine learning methods for image classification is the support vector machine (SVM). The basic idea of the method is that the image data is first converted into vector forms and then used to train the SVM. Each image vector becomes a separate training example, and the classification is based on the position of the vector relative to the hyperplane that separates the different classes. When training the model, a kernel is first selected that determines how the classes are separated in space. Next, the SVM searches for the optimal hyperplane that maximizes the separation of points from different classes and minimizes classification errors. This process is called hyperplane optimization [3]. Once trained, the SVM can classify new images by converting them to vector form and determining their position relative to the hyperplane built during training. The distance from the hyperplane to each image vector allows the SVM to determine the probability that the vector belongs to a class. SVM usually performs well with small data and provides good resolution, so it may be suitable in this study. The disadvantage of SVM is that it can become computationally demanding when the amount of data increases [4]. Also, when creating a classifier, there are difficulties with choosing the optimal values of hyperparameters, which affect its performance [5]. In addition, this method cannot work well with data containing noise or a large number of outliers. Therefore, SVM can be an effective classification method for MRI brain images for Alzheimer's disease diagnosis, but care must be taken when setting up the model parameters and processing the data accordingly. 2.2.2. Decision Tree Decision Tree is a supervised learning method used for classification and regression. The goal is to create a model that predicts the value of the target variable by learning simple decision rules derived from the characteristics of the data [6]. The most important feature of a decision tree is its ability to transform complex decision-making problems into simple processes, thus finding a solution that is clear and easier to interpret [7]. Thus, it is necessary to create a decision tree model based on the features obtained from the input images. The Decision Tree method is popular in image classification because it is easy to interpret and visualize the decisions made by the algorithm. It also works well for datasets with a small number of classes and a small number of features and does not require complex input processing, but the dataset must be balanced [6]. However, there are disadvantages, as this method may not be the best choice for datasets with many classes or high-dimensional feature spaces, where other algorithms may be more efficient. Thus, the Decision Tree method may be suitable for classifying images of Alzheimer's disease stages, since the dataset is not large and does not contain large image dimensions. However, one should be careful when tuning the model parameters to avoid overfitting. 2.2.3. Random Forest Random Forest is an ensemble learning method that uses a set of decision trees to make predictions. In image classification, it works by using multiple decision trees to assign an image to one of several classes. The algorithm starts by creating a large number of decision trees, each based on a random subset of the input data. Each decision tree in the random forest makes a prediction for the class of the input image based on its own set of rules, and these predictions are combined to make the final prediction for the image. During training, each decision tree is built based on a random subset of the input data, and at each split point in the tree, a random subset of features is considered [8]. This helps to reduce overfitting and increase the model's ability to generalize. At the prediction stage, each decision tree in the forest independently predicts the class of the input image, and the final prediction is the majority vote of all decision trees. The Random Forest method is known for its high accuracy and ability to handle large datasets with high dimensionality compared to a single decision tree model. It also has the advantage of being able to handle data with noise and outliers [9]. However, training a random forest model can be computationally expensive and time-consuming, especially when dealing with large datasets. In addition, the model may not be as interpretable as a single decision tree, as it involves combining the predictions of several trees [8]. Thus, Random Forest can be an effective tool for the task of classifying images of disease stages, provided that the dataset is properly prepared and the model parameters are tuned. 2.2.4. Multi-Layer Perceptron Multi-Layer Perceptron Classifier is a type of neural network that can be used for image classification. A supervised learning algorithm learns from a defined dataset using backpropagation. A multilayer perceptron works by taking an input image and reducing it to a one-dimensional array of pixel values. This array is then fed to the input layer of the neural network. The input layer is connected to one or more hidden layers, each of which consists of several neurons. These hidden layers are responsible for learning the features in the input image. The output layer then produces a probability vector, with each element of the vector representing the probability of the input image belonging to a particular class. During the training phase, the weights of the connections between neurons are adjusted iteratively using backpropagation, which is a gradient descent optimization algorithm. This adjusts the weights in such a way as to minimize the difference between the predicted output of the network and the actual output based on the labelled data set [10]. Once a neural network is trained, it can be used to classify images by inputting new images and obtaining predicted probabilities for each class. The class with the highest probability is then assigned as the predicted class for that image. Thus, a multilayer perceptron can be used for the task of classifying disease stage images because it performs well on small datasets, but it can also be sensitive to the choice of hyperparameters such as the number of hidden layers, the number of neurons in each layer, and the activation function. 2.3. Ensemble Voting Classifier Ensemble learning is a machine learning method that uses multiple models to get better prediction results. In an ensemble method, multiple models are trained on the same training set, and their prediction results are combined to produce the final result. This approach can provide better prediction accuracy than a single model if the models are properly tuned. There are different types of ensemble learning methods, which differ in the type of models used, the data sample, and the decision-making function. One of the most widespread ensemble learning techniques, Voting, was chosen for the study. Voting Classifier is a voting algorithm used to combine the results of different classifiers into one ensemble to achieve better results. In this study, we will use the Voting method because it does not require complex computations. Also, Voting can be a fast method because it does not require additional training of the ensemble model compared to other ensemble learning techniques. Voting is also a flexible method because it allows you to use different combinations of base models with different algorithms, hyperparameters and settings. This allows you to experiment with different models and find the optimal combination for a particular task. The ensemble developed in this study consists of three classification models, which are selected based on the results of image classification separately by each method discussed in Section 2.2. The diagram of the ensemble formation from the selected models using Voting Classifier is shown in Figure 3. Figure 3. Diagram of creating an ensemble of classification models The ensemble is trained on the training dataset, and the final testing is performed on the test dataset. To choose the best type of voting, we conducted a study with two types: hard voting and soft voting. Hard voting is majority voting, where each classifier gives a vote for its predicted class, and then the class with the highest number of votes is selected [11]. This can be useful for high quality data when all classifiers produce accurate results. Soft voting is a probability-based voting where each classifier provides probabilities for each class, and then the class with the highest average probability is selected [11]. This makes it possible to take into account the importance of some classifiers that may produce better results for certain classes. Soft voting usually works better when classifiers have different accuracies. The ensemble is chosen according to the results of the study with the considered voting types. 3. Proposed solution This section presents the results of the study obtained for AD prediction using the selected dataset. To summarize, first of all, the performance of each model for MRI image classification is studied separately. After that, the three models with the best results are selected and combined into an ensemble. To ensure the best result, two types of voting ensemble are investigated and the best one is selected. The development of the software module is carried out on the free interactive platform Google Colab. To create a program module, the following libraries are used: Numpy, Opencv, Matplotlib, Seaborn, Scikit-learn, Hashlib, Scikit-image, Joblib. 3.1. Evaluation of the performance of individual classification models Each model was trained on a training dataset and forecasting is performed on validation data. The Grid Search method was used to select model hyperparameters. Grid Search takes as input a list of parameters that we want to test and their possible values. For each combination of parameters, Grid Search performs cross-validation and calculates the average accuracy on the training set. The best parameter set was defined as the one with the highest average accuracy on the training set. The best parameters were used to build the final classifier on the entire dataset. For the Decision Tree method, the hyperparameters that were selected are criterion and max_depth. Using Grid Search, we found that the best combination of parameters was criterion='entropy' and max_depth=6, which gave the best results on the sample. These parameters were used for further classification using Decision Tree. For the Random Forest method, the hyperparameters n_estimators and max_depth were selected and the best result was obtained with the parameters max_depth=None, n_estimators=300. For the Multi-layer Perceptron method, Grid Search was used to find the best result with the parameters activation='tanh', hidden_layer_sizes=(50, 50, 50), solver='sgd'. For the SVM method, the hyperparameters C=1, gamma=0.001, kernel='rbf' were chosen. These classifications are conducted using different performance metrics in terms of Accuracy, Precision, Recall and F1 score. The results of all four models are summarized in Table 1. Table 1 Comparison of the classification of AD using different models Disease stages Model Metrics AD CN EMCI LMCI MCI Precision 0.87 0.85 0.90 0.79 0.95 Decision Recall 0.91 0.84 0.87 0.84 0.89 Tree F1-score 0.89 0.84 0.88 0.81 0.92 Accuracy 0.87 Precision 0.96 0.93 0.94 0.88 0.98 Random Recall 0.94 0.90 0.97 0.93 0.95 Forest F1-score 0.95 0.92 0.96 0.91 0.96 Accuracy 0.94 SVM Precision 0.95 0.92 0.93 0.92 0.96 Recall 0.93 0.94 0.97 0.90 0.95 F1-score 0.94 0.93 0.95 0.91 0.95 Accuracy 0.94 Multi-Layer Precision 0.95 0.94 0.95 0.91 0.97 Perceptron Recall 0.96 0.93 0.97 0.92 0.95 F1-score 0.95 0.94 0.96 0.92 0.96 Accuracy 0.95 In summary, the Random Forest, SVM, and Multi-Layer Perceptron models outperform the Decision Tree model in terms of overall accuracy. Among these three, the Multi-Layer Perceptron has the highest accuracy. All models have relatively high precision, recall, and F1-scores for most disease stages, indicating their effectiveness in classifying these stages. The best three models were selected for further integration into an ensemble: Random Forest, SVM, and Multi-Layer Perceptron. 3.2. Evaluation of the performance of different ensemble classification methods Table 2 displays the results of classifying MRI images into five stages of Alzheimer's disease using ensembles of two types of voting. In terms of precision, both Hard Voting and Soft Voting are strong, with high precision for the AD and CN stages, indicating low false-positive rates. Table 2 Comparison of AD classification with different ensemble classification methods Disease stages Ensemble Metrics AD CN EMCI LMCI MCI Precision 0.98 0.95 0.93 0.91 0.97 Recall 0.96 0.95 0.99 0.92 0.93 Hard Voting F1-score 0.97 0.95 0.96 0.92 0.95 Accuracy 0.95 Precision 0.98 0.95 0.94 0.94 0.97 Recall 0.97 0.97 0.99 0.91 0.94 Soft Voting F1-score 0.97 0.96 0.96 0.92 0.95 Accuracy 0.96 Soft Voting generally outperforms Hard Voting in terms of recall, achieving higher recall values for CN, EMCI, and MCI. This suggests that Soft Voting is better at identifying true positive cases in these stages. F1-scores are relatively similar between the two methods, with both ensembles maintaining a good balance between precision and recall for most disease stages. Soft Voting has a slightly higher overall accuracy (0.96) compared to Hard Voting (0.95). In summary, Soft Voting Classifier appears to be a slightly better ensemble method in this context, as it achieves higher recall and accuracy while maintaining strong precision and F1-scores for most stages of the disease. The Figure 4 shows the error matrices for both ensembles. From the obtained confusion matrix, we can see that in general, the results are better compared to the confusion matrix of the ensemble with hard voting type. The number of correctly classified images of the AD class is 2 samples higher than when using the ensemble with hard voting type. For the CN class, this number is 4 samples more, and for the MCI class, it is 1 sample more. For the EMCI class, the number of correctly classified images has not changed. However, for the LMCI class, the number of correctly classified data decreased by 2 samples. The problem of classifying the LMCI class may be related to the specificity of the data of this class in the dataset used in the study. (a) (b) Feature 4. The confusion matrix of the models. (a) Hard Voting Classifier; (b) Soft Voting Classifier The Soft Voting Classifier with the highest efficiency and accuracy of 0.96 was selected. The results show good performance of the classification model with high precision and recall for most classes, which indicates its ability to recognize different stages of Alzheimer's disease in the studied dataset. 4. Discussion To evaluate the effectiveness of the developed model, we will compare the results of the soft voting ensemble with the results of individual models used in this ensemble. Comparative results are shown in the Table 3. Table 3 Combined results of the ensemble and individual models tested Disease stages Model Metrics AD CN EMCI LMCI MCI Precision 0.98 0.95 0.94 0.94 0.97 Ансамбль Recall 0.97 0.97 0.99 0.91 0.94 soft voting F1-score 0.97 0.96 0.96 0.92 0.95 Accuracy 0.96 Precision 0.96 0.93 0.94 0.88 0.98 Random Recall 0.94 0.90 0.97 0.93 0.95 Forest F1-score 0.95 0.92 0.96 0.91 0.96 Accuracy 0.94 Precision 0.95 0.92 0.93 0.92 0.96 Recall 0.93 0.94 0.97 0.90 0.95 SVM F1-score 0.94 0.93 0.95 0.91 0.95 Accuracy 0.94 Precision 0.95 0.94 0.95 0.91 0.97 Multi-Layer Recall 0.96 0.93 0.97 0.92 0.95 Perceptron F1-score 0.95 0.94 0.96 0.92 0.96 Accuracy 0.95 Analyzing this table with metrics for each model, the following conclusions can be drawn for each stage of the disease. AD: • The Soft Voting Ensemble has the highest Precision, Recall, and F1-score. • Random Forest and SVM also perform well, but the Soft Voting Ensemble is slightly better. CN: • The Soft Voting Ensemble and SVM have the highest Precision, Recall, and F1-score. • Random Forest and MLP perform slightly worse. EMCI: • The Soft Voting Ensemble has the highest Precision and F1-score. • Random Forest and SVM perform well, with slightly lower Precision. • MLP also performs well but has slightly lower Recall and F1-score. LMCI: • The Soft Voting Ensemble has the highest Recall, while Random Forest has the highest Precision. • The Soft Voting Ensemble has a better F1-score. MCI: • The Soft Voting Ensemble has the highest Precision, Recall, and F1-score. A common metric for evaluating models is accuracy, which reflects the overall ability of the model to correctly classify instances across all classes. In this case, the soft voting ensemble has the highest accuracy (0.96), which means that it correctly classifies 96% of instances from all classes at different stages of the disease. Thus, the developed soft voting ensemble proved to be the best option among the considered models, with high precision, recall, and F1-score for most classes and the highest overall accuracy for the studied dataset. 5. Conclusion The ensemble of Random Forest, Multi-Layer Perceptron, and Support Vector Machine models proposed in this study has achieved the accuracy of 96% in classifying patients into five stages of AD. The multiclass classification of BA into five stages is of great importance in the field of diagnosis of AD. The developed soft voting ensemble proved to be the best option among the considered models, with high precision, recall and F1-score for most classes and the highest overall accuracy for the studied dataset. This is a significant improvement over the performance of individual ML models, which typically achieve accuracy in the range of 90%. The proposed ensemble model is therefore a valuable tool for the early and accurate diagnosis of AD, which can lead to better patient outcomes. Future work could focus on investigating the performance of the ensemble model on other AD datasets, developing and evaluating new ensemble learning algorithms for AD classification, integrating the ensemble model with other clinical data, such as imaging and neuropsychological data, to further improve its performance. 6. References [1] S. Nebehay. Number of people with dementia set to jump 40% to 78 mln by 2030, 2021. URL: https://www.reuters.com/business/healthcare-pharmaceuticals/number-people-with-dementia-set- jump-40-78-mln-by-2030-who-2021-09-02/. [2] Alzheimer datasets 5 classes | kaggle. URL: https://www.kaggle.com/datasets/phamnguyenduytien/alzheimer-datasets-5-classes. [3] H. Bhavsar and M. H. Panchal, “A Review on Support Vector Machine for Data Classification”, International Journal of Advanced Research in Computer Engineering & Technology, Volume 1, No. 10, pp. 185-189, 2012. [4] J. Cervantes, F. Garcia-Lamont, L. Rodríguez-Mazahua, A. Lopez, A comprehensive survey onsupport vector machine classification: Applica-tions, challenges and trends, Neurocomputing 408(2020) 189–215. doi: https://doi.org/10.1016/j.neucom.2019.10.118. [5] D. J. Kalita, V. P. Singh and V. Kumar, “A survey on svm hyper-parameters optimization techniques,” inSocial Networking and Computational Intelligence, Berlin, Germany, Springer, 2020, pp. 243–256. doi: https://doi.org/10.1007/978-981-15-2071-6_20. [6] 1.10. decision trees — scikit-learn 1.2.2 documentation URL: https://scikit- learn.org/stable/modules/tree.html. [7] Priyanka, D. Kumar, “Decision tree classifier: a detailed survey”, International Journal of Information and Decision Sciences, vol. 12, no. 3, pp. 246269, 2020. [8] Ibrahim, I. Abdulazeez, A. The role of machine learning algorithms for diagnosing diseases. J. Appl. Sci. Technol. Trends 2 (2021). doi: http://dx.doi.org/10.38094/jastt20179. [9] Speiser JL, Miller ME, Tooze J, Ip E. A Comparison of Random Forest Variable Selection Methods for Classification Prediction Modeling. Expert Syst Appl. 2019 Nov 15;134:93-101. doi: https://doi.org/10.1016/j.eswa.2019.05.028. [10] Multilayer perceptron / 2023. URL: https://en.wikipedia.org/w/index.php?title=Multilayer_perceptron&oldid=1147594876. [11] Sklearn.ensemble.votingclassifier — scikit-learn 1.2.2 documentation. URL: https://scikit- learn.org/stable/modules/generated/sklearn.ensemble.VotingClassifier.html.