A Stacked Ensemble Model based on RUSBoost and a Cost-Sensitive Convolutional Neural Network for Class Imbalance in Big Data Analytics

Abstract

- Machine learning algorithms have been designed such that they focus more on attaining high accuracy levels and as such they tend to misclassify instances that belong to the minority class which are often the instances of interest. In big data analytics, the issue of handling class imbalance has been understudied and this is mainly attributed to the problems of small disjuncts of an imbalanced dataset. This study proposes a stacked ensemble model based on RUSBoost and cost sensitive convolution neural network for tackling class imbalance issues in big data analytics, The RUSBoost algorithm handles the class imbalance at the data level by intelligently and randomly removing samples from the majority class while the costsensitive convolutional neural network handles the imbalance at the algorithm level by enabling the convolutional algorithm automatically learn the cost during the training period. The ImageNet and WHOI-Plankton datasets were used to evaluate the proposed model as they met the imbalance ration standard set of 100:1 in addition to containing more than 100,000 records. The results posted showed the proposed stacked ensemble model outperformed existing ensemble techniques such as the SMOTEBoost and AdaBoost which were seen to immensely improve on the classification performance given an imbalanced dataset. The recall, precision and gmean value posted by the proposed algorithm were 80.2%, 95% and 87.3%respectively. Therefore the proposed model is seen as a better, faster and less complex alternative for handling class imbalance in big dataset.

Keywords

• class imbalance
• big data analytics
• random under sampling
• cost
• Convolutional neural network

I. INTRODUCTION

Big data, a term with no formal definition is commonly characterized with 5 V’s. that is, velocity, volume, veracity, variety and value whereby the volume and variety characteristics are coined from the aspect that big data involves massive amount of various structured, semistructure and unstructured data while the velocity characteristic implies that the data is collected with a very high rate. The veracity characteristic is used to indicate the quality of the data collected while the value characteristic explains whether the data collected is of importance, depending on the problem [ 1 ]. With the availability of these massive data, most organizations aim at analyzing the data so as to identify hidden patterns and insights that may not be easily identified by a human, for making informed decisions and support their strategies.

In data analytics, deep learning methods have gained greater attention as compared to other machine learning (ML) classification algorithms they have been associated with improved performance levels especially on domains that involve complex datasets [ 2 ]. This has mainly been attributed to factors such as availability of hardware and software components, availability of data, improvements of the algorithms that help to speed up the time taken to train a model and generalize new data. Despite these advancements, [ 3 ]states that these deep learning algorithms have been seen to perform poorly given an imbalanced dataset whereby the samples from the majority class are seen to dominate the gradient value which updates the overall model’s weight and as a result, the errors from the majority class are reduced while error obtained from the minority class are increased leading to a slow convergence of the network.

Class imbalance, a problem common to classification algorithms, results from having fewer samples from the minority group as compared to the sample amount retrieved from the majority group in the same dataset [ 4 ]. The underlying assumption of both DL and ML classification algorithms is that the classes, present in a dataset, have been represented in almost equal proportions [ 5 ].

Nonetheless, this is contrary to what happens in real world scenarios whereby the aspect being measured is often less represented as compared to its counterpart. Depending on factors such as the imbalance ratio, complexity of the concept represented, classifier involved and overall size of training set, the degree of class imbalance is seen to vary from minor to severe class imbalance and class rarity[ 6 ].

Currently, techniques identified for dealing will class imbalance issues can be broadly classified into data-level, algorithm level and hybrid techniques [ 7 ]. Data level techniques for handling class imbalance have been designed such that they eliminate the class imbalance in the training dataset either through random under-sampling(RUS) that involves removing samples randomly from the majority class or through random over-sampling(ROS) that involves a process of randomly duplicating values from the minority class. The algorithm level techniques aim at modifying the algorithm such that it will not be biased towards the majority class instances while the hybrid techniques ,which are also referred to as ensemble techniques, combine both data level and algorithm modification techniques [ 3 ].

[ 4 ] did a survey paper that aimed at summarizing the research works conducted from 2010 to 2018 to address the issue of class imbalance in big data. To ensure they only considered works that involved big data only, they selected works that utilized datasets consisting of a minimum of 100,000 records.

From their work it was noted that there exists very little work on class imbalance when dealing with big data. Additionally, tackling the issues of class imbalance on the data-level, the ROS gives better performance but this is exclusively on smaller datasets as using this technique on a large dataset may result to generating a very huge amount of dataset that may lead to an increase in the training time thus lowering efficiency. For big data analytics, they suggested use of RUS which is assumed will perform better by removing noise and redundant samples from the dataset. Additionally, the algorithm level techniques that were identified included the mean false error(MFE) loss and mean squared false error(MSFE) proposed by [ 8 ] , focal loss[ 9 ] that aims at reducing the impact samples that are easily classified have on the loss of the classifier and use of a cost-sensitive convolutional neural network (CNN) [ 10 ] that is capable of learning the weights and cost during the training process. Of the identified algorithm level techniques, the cost-sensitive CNN and focal loss technique have been seen to generalize and adapt to various domains that involve complex datasets as they are not dependent on the knowledge of a specific domain. Ensemble algorithms such as AdaBoost[ 11 ], UnderBagging[ 12 ], RUSBoost[ 13 ], SmoteBoost[ 14 ], and overbagging[ 15 ] were identified. Of these algorithms, the SmoteBoost, RUSBoost and Underbagging were seen to post high performance levels. Additionally the RUSBoost was seen to have more superiority in terms of performance and being the least complex algorithm. Despite the great performance levels being posted by the ensemble, it was noted that there still exists a problem of cost definition.

To handle the class imbalance problem in big data analytics, this study proposes an ensemble technique based on RUSBoost and cost-sensitive CNN. Using this approach, it is expected that the deep learning algorithm will be able to perform better even with an imbalanced dataset and the cost associated with the training process will be automatically defined.

II.

RELATED WORKS [ 16 ] did an analysis on data level techniques for handling class imbalance in big data analytics, that is, the ROS, RUS and SMOTE in the MapReduce framework and the impact these techniques have when evaluated using decision trees(DT) and random forest(RF) in Spark and Hadoop. To train the model, the evolutionary computation for big data and big learning competition (ECBDL14) dataset was used. To further evaluate the impact on these techniques given different number of maps in a MapReduce environment, 1,8,16, 32 and 64 maps over both Spark and Hadoop were evaluated. The results obtained showed that SMOTE and RF performed better than SMOTE and DT. Also it was observed that the RUS and ROS performed better than SMOTE in the big data scenario and that the performance level obtained greatly depended on the behavior of the classifier used. To sum it up, it was concluded that the ROS performed independent of the number of maps being used while the data distribution was greatly affected when the RUS was applied on different partitions. As a result, increasing the number of partitions was seen to have severe effect for the RUS and this was mainly attributed to the lack of data.

[ 17 ] Described a Random over Sampling and Evolutionary Feature Weighting for Random Forest (ROSEFW-RF) which won the ECBDL14 big data challenge held for the big data challenge in bioinformatics domain. The ROSEFW-RF technique, based on MapReduce environment was used to tackle the class imbalance problem using random oversampling then the evolutionary feature weighting was used to identify relevant features which were then selected using a threshold. A RF model was then trained using the preprocessed dataset. To evaluate the performance measure of the ROSEFW-RF, 64,192 and 256 mappers were used with 100 trees defined for the RF model. They further did an experiment of using RF classifier with a dataset that used 100% oversampling ratio. The results obtained during this first experiment showed that a very low true positive rate (TPR) was obtained and compared to the true negative rate (TNR) and that this difference tends to increase with a decrease in the number of mappers used. Experiments done using the ROSEFW-RF showed that this technique outperformed other strategies in the competition and was capable of balancing the TNR and TPR which had been a considered to be a difficult task during the competition. For future research, they proposed analysis of the effects of the number of maps and other classifiers in addition to utilizing a strategy that would combine over sampling and under sampling or even use of instance reduction techniques for handling the class imbalance challenge on big data analytics.

In [ 18 ] the authors proposed a supervised technique that would handle the class imbalance between nonortholog and ortholog classes observed on ortholog detection in different yeast species. The proposed methodology was structured such that it involved three steps that involved calculation of the different gene pair features that were supposed to be combined, building the ML classifier and classification of the obtained gene pair features. The proposed technique was then compared using various proposed models such as the RF for Big Data with Cost-Sensitive (RF-BDCS) described in [ 19 ],the Random Oversampling with RF for Big Data(ROS+RF-BD) described in [ 20 ] and the Support Vector Machines for Big Data(SVM-BD) in the Apache Spark environment. During the experiment, the authors selected datasets with various genome yeast pairs, that is: Saccharomyces-Klutveromyces lactis, Saccharomyces cerevisiae-Candida glabrata and Saccharomyces cerevisiaeSchizosaccharomyces pombe. To evaluate the performance measure, the true positive rate (TPR), true negative rate (TNR), the area under curve (AUC) and G-Mean performance metrics were used. The results obtained indicated that the proposed supervised technique for gene pairwise feature combination gave the best for pairwise ortholog detection in big data scenarios. Additionally, using ROS with the SVM-BD classifier gave better results as compared to the other tested techniques. For future works, they recommended use of new gene pair features for the supervised algorithm for pairwise ortholog detection.

III. PROPOSED METHOD

A. Dataset Description

[ 21 ] Defined class imbalance in data as data having an imbalance ratio between the ranges of 100:1 to 10000:1.Consequetly,to train and evaluate the proposed model, datasets with a class imbalance ratio of 100:1 and above were considered in this study. Additionally, to incorporate the big data aspect, the WHOI-Plankton[ 22 ] and ImageNet[ 23 ] datasets with a minimum of 100,000 records, as described in in table 1, were obtained.

The obtained dataset was partitioned to training and test dataset using k-cross fold validation whereby k=10. Nine (9) folds were used to train the model while the other one fold was used to test the performance of the model.

To harness the computational power and great performance of RUSBoost as reported by [ 24 ] and address the issue of cost definition in convolutional neural networks while handling datasets with class imbalance, this study proposes a stacked ensemble based on the RUSBoost and a cost-sensitive CNN. The RUSBoost was used to sample the dataset at the data level by under sampling instances from the majority class and the obtained result fed to the costsensitive CNN for classification as illustrated in figure 1.

The residual network (ResNet) architecture for training deep neural networks was used to develop the proposed model. The decision to adopt the ResNet architecture was motivated by the high level of performance reported when using the ResNet architecture for deep neural networks as a result of its skip connections [ 25 ].

Fig 1: The RUSBoost, Cost Sensitive CNN Stacked Ensemble Model

To enable the CNN automatically determine the cost associated with each misclassification, an additional cost layer was added to manipulate the output of the convolutional layer before it is fed to the softmax layer for classification. The cost layer was designed such that it was capable of automatically updating the cost matrix in equation (1) using an empirical risk value illustrated in equation (2).

Where: R1 (o) is the empirical risk.

Y is the class labels while n is the total number of instances in the dataset.

C is the cost matrix whereby the cost value is set as an imbalanced ratio (IR) when the class predicted q matches the actual class p

The o(i) is the predicted output while the desired output is represented by d(i) .

The RUSBoost was implemented such that the instances from the majority class are under sampled using an intelligent random under sampling technique as illustrated in the table 2 which gives the RUSBoost algorithm. C. Performance Evaluation

To evaluate the performance measure of the developed ensemble model, a confusion matrix shown in table 3, was used to show the true positive, true negative, false positive and false negative values. These values from the confusion matrix were then used to calculate the precision, recall and geometric mean of the true negative rate (TNR) and the true positive rate (TPR) metrics as shown in equation(3),(4) and (5) respectively for evaluation of the proposed model. According [ 24 ] to the accuracy level is not a favorable performance evaluation metric in an imbalanced data as it is not sensitive to class imbalance hence might give misleading results. Consequently it was not considered in this study. (1) (2) (3) (4) (5) 1. Initialize all wi=eights to 1/m for all instances

; D1(i) = 1/m for all i Repeat for t = 1,2,3,… T [ 0,1 ] Dt) Create temporary training set S`t with a D`t distribution using random undersampling D`t weights Call WL with S`t dataset with their Get back a hypothesis ht : X*Y -> Calculate the pseudo-loss ( for S and ( )(1 − ℎ ( ,

)ℎ ( , ))

Calculate

parameter : Update Dt: the weight update = є 1 − є 1(1+ℎ ( , )−ℎ ( , : ≠ )) 2 Normalize Dt+1 : Let = ∑ +1( )

+1( ) = +1( ) 3. Output the final hypothesis ⋲ ( ) = argmax ∑ ℎ ( , ) log 1 =1

Actual Positive True

False

the goal of enabling the classifier to automatically learn the cost during the training process and as a result improve on the performance level of the model. classifier was capable of minimizing the errors as a result of misclassifications. positives and false negatives. According to the literature reviewed, there exists limited studies that focus on class imbalance in big data analytics. Consequently, it has been suggested various techniques for handling class imbalance in traditional machine learning algorithms can be extend to big data analytics using deep learning techniques. In this work an ensemble model based on RUSBoost and costsensitive CNN for handling class imbalance issue in datasets is proposed. The performance measure of the proposed model was compared with the performance posted when using RUS with CNN and RUSBoost with cost sensitive CNN.

According to the obtained results, the ensemble model based on RUSBoost and a cost sensitive CNN model has proved superiority in terms of performance measure as compared to the using the RUSBoost . As a result, this technique can be adopted to a variety of big-data analytics applications for handling imbalanced datasets with a CNN classifier.

For future works, we recommend evaluation of the proposed model on the different big data analytics platforms such as Hadoop and Apache Spark so as to evaluate if it is independent of the big data platform used.

ACKNOWLEDGEMENT

We would like to thank the Africa-ai Japan Project 2020 for the financial support offered for this project

References

[1] I. Kalbandi and J. Anuradha , “ A Brief Introduction on Big Data 5Vs Characteristics and Hadoop Technology,” Procedia Comput . Sci., vol. 48 , pp. 319 - 324 , Dec. 2015 , doi: 10.1016/j.procs. 2015 . 04 .188.

[2] A. Muniasamy and A. Alasiry , “ Deep Learning: The Impact on Future eLearning,” Int. J. Emerg . Technol. Learn., vol. 15 , p. 188 , Jan . 2020 , doi: 10.3991/ijet.v15i01. 11435 .

[3] J. M. Johnson and T. M. Khoshgoftaar , “ Survey on deep learning with class imbalance , ” J. Big Data , vol. 6 , no. 1 , p. 27 , 2019 , doi: 10.1186/s40537-019-0192-5.

[4] J. L. Leevy , T. M. Khoshgoftaar , R. A. Bauder , and N. Seliya , “ A survey on addressing high-class imbalance in big data , ” J. Big Data , vol. 5 , no. 1 , p. 42 , 2018 , doi: 10.1186/s40537-018-0151-6.

[5] S. Iram , F.-B. Vialatte , and M. I. Qamar , “Chapter 1 - Early Diagnosis of Neurodegenerative Diseases from Gait Discrimination to Neural Synchronization,” in Emerging Topics in Computer Science and Applied Computing, D. Al-Jumeily , A. Hussain , C. Mallucci , and C. B. T .-A. C. in M. and H. Oliver , Eds. Boston: Morgan Kaufmann, 2016 , pp. 1 - 26 .

[6] R. A. Bauder , T. M. Khoshgoftaar , and T. Hasanin, “ An Empirical Study on Class Rarity in Big Data,” Proc. - 17th IEEE Int. Conf. Mach. Learn. Appl. ICMLA 2018 , pp. 785 - 790 , 2019 , doi: 10.1109/ICMLA. 2018 . 00125 .

[7] K. Yang et al., “Hybrid Classifier Ensemble for Imbalanced Data,” IEEE Trans. Neural Networks Learn. Syst. , vol. 31 , no. 4 , pp. 1387 - 1400 , 2020 , doi: 10.1109/TNNLS. 2019 . 2920246 .

[8] S. Wang , W. Liu, J. Wu , L. Cao , Q. Meng , and P. J. Kennedy , “ Training deep neural networks on imbalanced data sets ,” in 2016 International Joint Conference on Neural Networks (IJCNN) , 2016 , pp. 4368 - 4374 , doi: 10.1109/IJCNN. 2016 . 7727770 .

[9] T.-Y. Lin , P. Goyal , R. B. Girshick , K. He , and P. Dollár , “ Focal Loss for Dense Object Detection ,” 2017 IEEE Int. Conf. Comput. Vis. , pp. 2999 - 3007 , 2017 .

[10] S. H. Khan , M. Hayat , M. Bennamoun , F. A. Sohel , and R. Togneri , “ Cost-Sensitive Learning of Deep Feature Representations From Imbalanced Data ., ” IEEE Trans. neural networks Learn. Syst. , vol. 29 , no. 8 , pp. 3573 - 3587 , Aug. 2018 , doi: 10.1109/TNNLS. 2017 . 2732482 .

[11] K. Li , G. Zhou , J. Zhai , F. Li , and M. Shao , “ Improved PSO _ AdaBoost ensemble algorithm for imbalanced data,” Sensors (Switzerland) , vol. 19 , no. 6 , 2019 , doi: 10.3390/s19061476.

[12] B. Raghuwanshi and S. Shukla , “ Class imbalance learning using UnderBagging based Kernelized Extreme Learning Machine,” Neurocomputing , vol. 329 , pp. 172 - 187 , 2018 , doi: 10.1016/j.neucom. 2018 . 10 .056.

[13] S. R. Mounce , K. Ellis , J. M. Edwards , V. L. Speight , N. Jakomis , and J. B. Boxall , “ Ensemble Decision Tree Models Using RUSBoost for Estimating Risk of Iron Failure in Drinking Water Distribution Systems ,” Water Resour. Manag., vol. 31 , no. 5 , pp. 1575 - 1589 , 2017 , doi: 10.1007/s11269-017-1595-8.

[14] W. Feng , W. Huang , and J. Ren , “ Class imbalance ensemble learning based on the margin theory , ” Appl. Sci. , vol. 8 , no. 5 , 2018 , doi: 10.3390/app8050815.

[15] L. Nanni , C. Fantozzi , and N. Lazzarini , “ Coupling different methods for overcoming the class imbalance problem , ” Neurocomputing , vol. 158 , 2015 , doi: 10.1016/j.neucom. 2015 . 01 .068.

[16] A. Fernández , S. del Río , N. V. Chawla , and F. Herrera , “ An insight into imbalanced Big Data classification: outcomes and challenges,” Complex Intell . Syst., vol. 3 , no. 2 , pp. 105 - 120 , 2017 , doi: 10.1007/s40747-017-0037-9.

[17] I. Triguero , S. del Río , V. López , J. Bacardit , J. M. Benítez , and F. Herrera , “ ROSEFW-RF: The winner algorithm for the ECBDL'14 big data competition: An extremely imbalanced big data bioinformatics problem,” Knowledge-Based Syst ., vol. 87 , pp. 69 - 79 , 2015 , doi: https://doi.org/10.1016/j.knosys. 2015 . 05 .027.

[18] D. Galpert , S. del Río , F. Herrera , E. Ancede-Gallardo , A. Antunes , and G. Agüero-Chapin , “An Effective Big Data Supervised Imbalanced Classification Approach for Ortholog Detection in Related Yeast Species ,” Biomed Res. Int. , vol. 2015 , p. 748681 , 2015 , doi: 10.1155/ 2015 /748681.

[19] A. Haldankar and K. Bhowmick , “ A cost sensitive classifier for Big Data ,” in IEEE International Conference on Advances in Electronics, Communication and Computer Technology, Dec. 2016 , pp. 122 - 127 , doi: 10.1109/ICAECCT. 2016 . 7942567 .

[20] S. Del Rio , J. Benitez , and F. Herrera , “ Analysis of Data Preprocessing Increasing the Oversampling Ratio for Extremely Imbalanced Big Data Classification ,” in 2015 IEEE Trustcom/BigDataSE/ISPA, Aug. 2015 , pp. 180 - 185 , doi: 10.1109/Trustcom. 2015 . 579 .

[21] H. He and E. A. Garcia , “ Learning from Imbalanced Data,” IEEE Trans. Knowl. Data Eng. , vol. 21 , no. 9 , pp. 1263 - 1284 , 2009 , doi: 10.1109/TKDE. 2008 . 239 .

[22] B. E. F. Sosik Heidi M. , Peacock Emily E., “ WHOI-PLANKTON,” WHOAS: Woods Hole Open Acess Server , 2014 . .

[23] J. Deng , W. Dong , R. Socher , L. Li , K. Li , and L. Fei-fei, “Imagenet: A large-scale hierarchical image database ,” in In CVPR, 2009 .

[24] C. Seiffert , T. M. Khoshgoftaar , J. Van Hulse , and A. Napolitano , “ RUSBoost: Improving classification performance when training data is skewed , ” Proc. - Int. Conf. Pattern Recognit ., pp. 8 - 11 , 2008 , doi: 10.1109/icpr. 2008 . 4761297 .

[25] A. Khan , A. Sohail , U. Zahoora , and A. S. Qureshi , “ A survey of the recent architectures of deep convolutional neural networks , ” Artif. Intell. Rev. , pp. 1 - 70 , 2020 , doi: 10.1007/s10462-020-09825-6.

[26] C. Seiffert , T. M. Khoshgoftaar , J. Van Hulse , and A. Napolitano , “ RUSBoost: A hybrid approach to alleviating class imbalance , ” IEEE Trans. Syst. Man, Cybern. Part ASystems Humans , vol. 40 , no. 1 , pp. 185 - 197 , 2010 , doi: 10.1109/TSMCA. 2009 . 2029559 .