The article discusses the application of methods for reducing the size of majority classes (undersampling) in unbalanced samples to the problem of processing and classifying user requests sent to the technical support of the portal of the Center for Industry Information and Analytical Systems. Sampling methods are investigated, indicators of the classification quality are given - F1-measure for macro and micro averaging. The paper considers a sample consisting of user requests to the technical support service, which has a highly unbalanced form - volume of some classes in the sample exceeds the volume of others by several times. On this sample, various undersampling algorithms are used, a conclusion is made about the advisability of using certain algorithms. A study of the influence of undersampling and oversampling methods on the accuracy of classification of user requests to the portal's technical support service when using gradient boosting on decision trees was carried out. The results obtained showed that an increase in F1-measure compared to the base model was achieved only with the use of the Tomek Links undersampling method; other methods for reducing the size of majority classes did not give an increase inaccuracy, and in most cases even worsened the result.
Nowadays, due to the rapid development of computer technology, a section of data science known as Machine Learning has gained popularity. One of the problems of * machine learning is the classification problem, which often has to be solved in conditions of imbalanced classes. This situation arises in cases when the proportion of objects of one or several classes in the training sample is significantly greater than 1/n, where n is the number of classes. The consequence of such an imbalance in the data is both the worse quality of the model's operation on objects of minority classes (in comparison with the majority) and the bias of classification metrics, for example, accuracy (the proportion of correct answers).
To improve the quality of classification, there are sampling methods, which can be
divided into two main groups: oversampling (generation of additional objects of a
smaller (minority) class) and undersampling (removal of objects of a larger (majority)
class). In this paper, we consider a sample consisting of user requests to the technical
support service [
The purpose of the article is to analyze the well-known sampling algorithms to increase the accuracy of the classification of user requests to the technical support service. 2
Description of samples There are many requests from X users to the portal technical support. Each request, = 1̅̅,̅2̅̅2̅̅0̅̅0̅0̅, belongs to one of 13 classes and is presented in text format with the following structure: a greeting phrase - a description of the problem or the reason for contacting a signature.
The distribution of queries by class is shown in Table 1. It can be seen that the training set has a highly uneven distribution of objects
Class State Tasks CRRR Miscellaneous Performance reports Standard costs “Do the right thin” Portal FM rating Subsidies calculations Advanced trainings and seminars SOI AC of FMCS Salary monitoring Budgeting and FEAP 5.427 46.130 0.616 1.012 0.510 The solution to the problem is divided into 3 stages.
Data preprocessing Building a basic model (on the original sample) Building and analyzing models on modified samples Data processing is the extraction of Russianlanguage terms from the query text, their normalization (reduction to the initial form), and removal of stop words. In this task, the stop word dictionary included auxiliary parts of speech, names of people, and terms most frequently used in greeting phrases.
Figure 1 shows the visualization of the sample using the T-SNE method [
As a basic model for classifying queries, we will consider a classifier built based on
gradient boosting on decision trees (XGBClassifier) without using sampling methods.
The choice of the XGBClassifier is based on the best classification results in
comparison with other classifiers [
The f1-measure with micro and macro-class averaging will be used as an indicator of quality.
recall = =
+
+
(2)
where TP (True Positive), FP (False Positive), FN (False Negative) - error matrix
indicators (confusion matrix) [
Micro-averaging does not take into account the belonging of an object to a class, i.e. each object contributes equally to the calculation of the measure. The total values TP, FP, FN, TN are calculated as the sum of the components for each class, and then the F1 measure itself is calculated from the total values
With macro averaging, the value of the F1-measure is calculated as the average F1 for all classes, each class equally participates in the calculation of the measure.
The choice of the preferred averaging is based on the task at the given moment. The calculation of the F1-measure for micro-averaging is carried out using absolute scales, while for macro-averaging - using relative scales. Having received the general indicators of the error matrix in the first case, a large contribution will be made by the objects of the majority classes (due to their numerical superiority). In the second case, we equalize the contribution of each of the classes, ignoring their numbers. Thus, if the value of identifying minority objects in the problem turns out to be higher than the value of majority objects, then it is worth focusing on macro-averaging (for example, identifying “unreliable” bank customers in the problem of credit scoring).
In our task, the value of each class turned out to be equal, therefore, more attention was paid to micro-homogenization. Next, we investigated several sampling methods based on reducing the size of majority classes. As is known, with an uneven distribution of objects across the sample classes, the classifier model is trained unevenly on different classes, and therefore there is a shift in training towards majority classes, and in the extreme case, it is more profitable for the classifier to assign all objects to the majority class than to try to allocate objects of the minority class.
In studies, we will reduce those classes whose share exceeds 1/n, where n is the
number of classes in the sample. In the literature, there are quite a few sampling
methods associated with reducing the size of classes [
From Table 2 it can be seen that sampling methods associated with reducing the size of
majority classes in most cases do not increase the classification accuracy. A slight
increase in accuracy was achieved only with the use of Tomek Links, the success of which
can be justified by its algorithm, which removes objects of the majority class located
near objects of the minority class. Other methods are likely to reduce the majority class
too much, which leads to the underfitting of the model [
r_i = 〖majority〗 _i / k, where majority i is the number of majority objects and k neighbors of the ith minority object. ─ Normalize
(r_i) ̂ = r_i / (∑r_i) ─ For each neighborhood, calculate the number of generated objects
G_i=G*(r_i ) ̂ ─ New Gi objects are generated according to the formula:
s_i = x_i + (x_zi-x_i) α, x_i is an object relative to which new examples are generated. x_zi is a randomly selected example of a minority class in the x_i neighborhood. α is a random number in the range from 0 to 1.
If there are no examples of a minority class out of k neighbors, then either no new objects are created for xi, or xi is duplicated Gi times.
RandomOverSampler – random selection of objects with the return.
SMOTE can link internal and external values, while ADASYN can focus exclusively on outliers, which in both cases can lead to suboptimal decision functions. In this regard, SMOTE offers two additional sample generation options (SVMSMOTE, BorderlineSMOTE). These methods focus on samples near the border of the optimal decision function. A visual illustration is shown in Table 3 and Fig. 2. The deterioration in the quality of the model can be explained by the low linear separation of the sample, since all methods, except for RandomOverSampler, use the algorithm of nearest neighbors, which can lead to the generation of new objects belonging to the majority class.
A study was carried out of the influence of undersampling and oversampling methods on the accuracy of the classification of user requests to the portal technical support service when using gradient boosting on decision trees. The results obtained showed that an increase in accuracy compared to the baseline model was achieved only with the use of the Tomek Links undersampling method, other methods for reducing the size of majority classes did not give an increase inaccuracy, and in most cases even worsened the result. The use of oversampling methods did not give results on this sample, which is most likely due to the poor linear separability of the sample. A further direction of research is to study the possibility of improving the accuracy of classification by simultaneously increasing the minority classes and reducing the majoritarian ones.
This method is aimed at reducing the error and shows good results when applied in various applied calculations.