The incapacity of the body to effectively make or use insulin results in diabetes, a chronic illness. Over time, this illness may cause harm to the kidneys, blood vessels, heart, eyes, nerves, and kidneys. Timely treatment is essential to stop the progression of diabetes and requires early detection. We provide a hybrid machine learning strategy in this work that predicts diabetes by combining two strong algorithms. XGBoost (eXtreme Gradient Boosting)-based voting classifier and bagging classifier are the two main components of our system. We evaluated our model using three distinct datasets: the Pima Indian diabetes dataset (PIDD), its extended version, and the Frankfurt Hospital Germany Diabetes Dataset (FHGDD). In comparison to individual algorithms (XGBoost, Bagging with Decision Tree) and other ensemble methods (Voting Classifier, HM-Bag Moov Voting Classifier, XGBoost+ Data feature stitching, and Soft), our experimental results show that our proposed approach achieved higher accuracy of 92.7%, precision of 97.1%, recall of 81.7%, and an F1 score of 88.7%%. Therefore, our results imply that the hybrid machine learning approach that has been suggested can be a dependable tool for the early diagnosis of diabetes, resulting in better patient outcomes and more prompt and efficient therapies.
Diabetes is becoming more commonplace worldwide, which presents a serious public health risk. The early identification and treatment of diabetes can improve the quality of life for persons with the disease by preventing or delaying the onset of complications. Conventional techniques for identifying diabetes, such blood glucose monitoring, can be costly, intrusive, and time-consuming. Because machine learning techniques automate the process and enable more precise and effective disease identification, they hold the potential to completely transform the diagnosis and management of diabetes. Large-scale datasets and sophisticated algorithms are used to find patterns and risk variables that could be hard for human experts to find.
There are two main forms of diabetes: Type 1: Usually identified in young people, it is caused by the immune system targeting cells that produce insulin. It requires daily insulin pumps or injections to control. If left untreated, it can result in heart disease, retinopathy, neuropathy, and nephropathy. Type 2: More frequent, usually affecting older persons, it is characterized by either insufficient or inefficient insulin production or utilization by the body. controlled via dietary adjustments, prescription drugs, and occasionally insulin. If left unchecked, the hazards are comparable to those of Type 1. Because gestational diabetes during pregnancy results in decreased insulin sensitivity and elevated blood sugar levels, it raises the risk of Type 2 diabetes later in life for both the mother and the child.
Numerous researches have examined the prediction of diabetes by taking into account a range of characteristics, including lifestyle, Electronic Health Records (EHRs), environment, and molecular attributes. Based on prior experience and medical records that contain patient conditions and vital signs. The most widely used dataset in these studies is the Pima Indians Diabetes Database [8; 1;6;4], which has 768 samples, 268 of which are patients with diabetes, and 8 independent factors that are used to identify whether a patient has diabetes.
Using the PIMA Indian Dataset, authors in Ref. [
Despite the extensive utilization of the Pima dataset and the notable success in prediction
outcomes derived from it, a significant challenge arises due to class imbalance within the
dataset. Specifically, there exists a prevalence of healthier patients compared to those afflicted.
This inherent imbalance poses a substantial hurdle for classification algorithms, as the minority
classes are overshadowed. Consequently, even if misclassifying every minority instance, the
algorithm could still exhibit low error rates [
One potential remedy for this problem is data augmentation [
The primary aim of this project is to address the challenge of imbalanced datasets in diabetes
diagnosis by leveraging data augmentation techniques and ensemble learning models. The
ultimate goal is to enhance patient outcomes and alleviate the strain on healthcare systems
caused by diabetes. To achieve this objective, the project proposes an effective and efficient
system capable of analyzing clinical data to accurately identify diabetes or determine if an
individual is in the pre-diabetic stage. The project involves the utilization of both bagging and
XGBoost (eXtreme Gradient Boosting) algorithms for diabetes prediction, using an expanded
version of the Pima dataset previously generated via a GAN (Generative Adversarial Network)
algorithm [
Bagging also known as bootstrapping is a prominent ensemble learning technique that
combines predictions from multiple decision trees trained on different subsets of the same
dataset [
The proposed combined approach is designed to improve the accuracy and robustness of the predictive model. Furthermore, the project seeks to advance the field by comparing its proposed methodology with various state-of-the-art research studies, thereby providing insights into the efficacy and superiority of the suggested approach.
The remaining sections of the paper are arranged as follows: In Section 2, the study technique is explained; in Section 3, the evaluation's specifics are outlined and the results are discussed; and in Section 4, the article is concluded and potential future directions are discussed.
2.1. Dataset
Our research utilized the PIMA Indian Diabetes dataset [
Pregnancies: The total number of pregnancies.
Blood glucose level measured in an oral glucose tolerance test after two hours.
DiabetesPedigreeFunction: The function of the diabetes pedigree.
"Outcome," the only dependent variable in the dataset, had binary values of either 0 or 1. The dataset was split into two sets, a training set and a testing set, using a 70:30 ratio in order to assess the performance of the model. Using four folds of cross-validation on the testing set, the model was assessed after being trained on the training set.
The used extension of Pima was generated recently in a previous work [
The Frankfurt Hospital Germany Diabetes Dataset (FHGDD) serves as another resource in diabetes prediction and classification research. Comprising the same attributes as the PIDD but with an expanded size of 2000 instances, it provides a rich data source for analyzing diabetesrelated factors. Table 1 illustrates the distribution of instances across each class, delineating the counts for Class 1 (diabetic) and Class 2 (non-diabetic).
Our methodology combines XGBoost (eXtreme Gradient Boosting) and bagging techniques.
XGBoost, a gradient boosting decision tree (GBDT) algorithm, efficiently iterates weak models
to create a strong one, proven effective in prediction tasks [
Decision trees, particularly CART (Classification and Regression Tree), are pivotal in
XGBoost, where their shallow structure mitigates overfitting risks. XGBoost defines an
objective function to optimize during training, comprising a regularization term controlling
model complexity and a loss function quantifying prediction accuracy against actual values. For
binary classification, the logistic loss function, also called log loss or cross-entropy loss, is
employed as the objective function in XGBoost, denoted by equation (1) [
L = I n (yi − yi^ )2 + ( f ) (1)
n : the number of samples. yi : the true label. yˆi : the predicted label. (f ) : the regularization term, which is a function of the model parameters f.
XGBoost is a powerful machine learning algorithm that trains an ensemble of decision trees iteratively. It utilizes gradients to understand instance deviations and constructs trees to identify patterns efficiently. Weighted updates adjust instance weights based on prediction errors, while ensemble building combines individual predictions using their importance. Regularization techniques control model complexity, preventing overfitting, and control parameters fine-tune its behavior. By aggregating ensemble predictions weighted by importance, XGBoost produces accurate predictions, making it effective for various machine learning tasks.
In our study, we aimed to boost the accuracy and robustness of our model by integrating Bagging Classifier with XGBoost through a Voting classifier. The Voting Classifier, a form of ensemble classifier, combines predictions from multiple base classifiers via a majority or weighted vote. We adopted hard voting, where the final prediction for an input is determined by the majority vote of individual model predictions. Mathematically, for N individual models (f1, f2, ..., fN), the final prediction ypred for input x is obtained using equation (2), where argmax selects the class label with the highest number of votes. Figure 1 illustrates the flowchart of the proposed model.
ypred = argmax(sum(fi(x)))f or i = 1 to N (2)
The experimental hardware setup comprises an Intel Core i3-3110M CPU clocked at 2.40GHz, paired with 8 GB of RAM and a capacious 1 TB HDD for storage needs. Python 3.7 was utilized to develop the machine learning model, with the following libraries employed: NumPy as a fundamental tool for mathematical operations, pandas for efficient data loading, and scikit-learn providing a suite of base classifiers.
Precision=TP/(TP+FP) Rappel =TP/(TP+FN) F1-score=TP/(TP+1/2(FP+FN)) Accuracy=(TP+TN)/(TP+TN+FP+FN) (3) (4) (5) (6)
In the realm of binary classification evaluation for this task, four pivotal terms emerge—True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN)—are fundamental for evaluating classifier model performance. TP denotes correctly classified positive instances, TN signifies correctly classified negative instances, FP represents incorrectly classified positive instances, and FN indicates incorrectly classified negative instances. These terms are pivotal in calculating various evaluation metrics such as precision, recall, F1 score, and accuracy, essential for assessing classifier model performance. Precision measures the proportion of correctly identified positives, recall gauges the proportion of actual positives correctly identified, and F1 score offers a balanced assessment of both precision and recall. Accuracy quantifies the overall correctness of the classifier's predictions. Understanding these terms and associated evaluation metrics is crucial for evaluating and enhancing the performance of binary classification models. The formulas for these metrics are referenced in equations (3-6) as follows:
As demonstrated in Table 2, our approach exhibited strong predictive capability across multiple datasets. Specifically, high accuracy rates of 91% and 92.7% were achieved using the Pima and Pima extended datasets, respectively, indicating the robustness of our model. The precision values for the Pima and Pima extended datasets were notably high at 89.55% and 97.1%, respectively, indicating a high proportion of true positive predictions. Similarly, solid recall values of 81.08% and 81.7% were obtained for the Pima and Pima extended datasets, respectively, demonstrating the model's ability to accurately identify actual positives. Maintaining a balance between recall and precision, crucial in medical diagnostic models, our algorithm delivered F1-scores of 85% and 88.7% for the Pima and Pima extended datasets, respectively, indicating a suitable balance between the two metrics. Furthermore, when applied to the FHGDD dataset, our approach achieved an accuracy rate of 85.6% and a precision of 87.8%, albeit with a slightly lower recall of 68.7%. Nevertheless, the F1-score of 77.1% demonstrates a reasonable balance between precision and recall. These results underscore the effectiveness of our approach in accurately predicting the presence of diabetes across different datasets. For further context and comparison, detailed performance metrics relative to other state-of-the-art methods are provided in Table 3, and Figure 2 respectively.
Let's explore two contrasting scenarios: one featuring a diabetic patient and the other a nondiabetic individual. In the first case, the model identifies the patient, with attributes such as one pregnancy, blood glucose level of 119, blood pressure of 78, skin thickness of 29, insulin level of 180, BMI of 38.19, diabetes pedigree function of 0.53, and age of 25, as non-diabetic. Conversely, the second scenario portrays a patient with four pregnancies, blood glucose level of 129, blood pressure of 70, skin thickness of 18, insulin level of 122, BMI of 29.43, diabetes pedigree function of 1.17, and age of 41, classified as diabetic. This classification suggests that the combined characteristics in the latter set imply a higher likelihood of diabetes, according to the model's interpretation. 0,00 0,20 0,40 0,60 0,80 1,00
Combining XGBoost with decision trees for binary classification tasks leverages the strengths of both algorithms. Decision trees offer intuitive interpretations, robustness to noisy data, and the ability to handle both numerical and categorical features effectively. XGBoost enhances performance through boosting, regularization techniques, and scalability, making it suitable for large datasets.
The results obtained from our model using different datasets showcase its effectiveness in predicting diabetes compared to various existing techniques. Our model achieved high accuracy rates when applied to the Pima and extended Pima datasets, respectively. Additionally, when our model was applied to the FHGDD dataset, it achieved a respectable accuracy rate of 85.6%, indicating its applicability across different datasets.
Comparing our results to those of other techniques, we observe that our model outperforms several state-of-the-art methods in terms of accuracy. For instance, the soft voting classifier, XGBoost, Bagging, Random Forest, and XGBoost+ Data feature stitching techniques achieved accuracy rates of 79.08%, 75.75%, 74.89%, 77.48%, and 80.2%, respectively. Our model's accuracy surpasses these benchmarks, highlighting its superior predictive performance. Moreover, our approach also compares favorably to other ensemble methods, such as HM-Bag Moov Voting Classifier and Voting Classifier, which achieved accuracy rates of 77.21% and 86%, respectively.
In this study, we compare the accuracy obtained by our proposed model with other state-ofthe-art models. While ensemble learning, which combines multiple algorithms, often demands significant computational resources, our approach mitigates this challenge by ensembling just two robust algorithms, Decision Trees (DT) and XGBoost. Additionally, some methods in related works utilize more intricate techniques such as deep learning, which may enhance performance but at the expense of increased computational demands and interpretability challenges. Moreover, disparities in data preprocessing techniques and evaluation metrics further complicate direct comparisons between different models.
This study aimed to predict diabetes using machine learning techniques on the Pima Indian Diabetes dataset. XGBoost and Bagging with Decision Trees were among the techniques used, along with data pretreatment techniques like median imputation. For diabetes prediction, our method produced remarkable accuracy rates of 92.7% with the extended Pima dataset and 91% with the Pima dataset. With a precision of 89.55% for Pima and 97.1% for extended Pima, our model demonstrated a high percentage of accurate positive predictions. The model's recall values of 81.08% for Pima and 81.7% for extended Pima showed that it could recognize real positive cases.
These findings highlight the efficacy of our approach in predicting diabetes, although with acknowledgment of potential biases and incomplete data. Future research should prioritize addressing these limitations by diversifying datasets and incorporating more comprehensive medical information to reinforce the model's accuracy and robustness. Moreover, ensuring the model's applicability across diverse demographic groups is imperative for its generalizability.
In addition, while this study showcased promising outcomes with XGBoost and Bagging with Decision Trees, it's essential to explore additional algorithms and ensemble methods, such as stacking, which may offer complementary benefits.