This paper considers the building of a classifier in the case of unbalanced training sets, which consist of very short text messages. We use machine learning to automatic classifier construction. High quality of classification is the main requirement for a classifier (we use precision and recall as the criterion of quality). Imbalance in sets means, that the number of texts belonging to one class is significantly lower than those belonging to the other classes. The development of classifiers on the base of unbalanced samples is a well-known complex problem in Text Mining. A lot of theoretical and experimental researches are devoted to searching for a solution. This is especially the case for relatively short text documents, for which the use of known approaches is not always effective. We give a brief review of methods and argue using Gradient Boosting Machine (GBM) for classification short text messages in the case of unbalanced training sets. We diagnose situations in which the classifier fails, analyze them, form a hypothesis on how we can improve text processing, and configure the settings of the classifier. Also, we investigate and compare two schemes of preprocessing Bag of Words and Word2Vec. A final section is devoted to experimental results. Experiments were carried out on datasets collected in the process of the daily operation of one of the large bank's branches. These real data have a rather significant imbalance. In the case of the unbalanced training set our voting approach (GBM) improves performance considerably.
With the rapid growth of unstructured information in Internet, classification has become * one of the key techniques for handling and organizing text data. Methods of classification are used to analysis of documents (scientific articles, news, e-mails, WWW-pages, product reviews, etc.) and dividing them into classes (or categories). This problem arises in various fields of human activity. For example in large corporations (banks, governmental organizations, offices of e-commerce) are frequently required to automatically identify the type of messages sent by customers (or users) and correctly recognize their categories and direct them to the proper department for processing.
In this paper, we are dealing with the problem of automatically dividing service requests that come through the corporate network of the Bank. The purpose is to distribute requests based on the content and send them for execution to the relevant department. These requests may have the following form: "Please replace the cartridge in office #12, printer – HP Color LaserJet CM 1312 MFP, suitable time – from 10 to 12 a.m.”. To be able to solve such a real-world problem it is necessary to develop a classifier, which will efficiently distribute the requests according to the classes (departments) automatically. By doing so we want to exclude an operator from the process of distribution of requests, but maintain the high quality of classification (precision and recall). Thus, the processing of requests should take place without any human participation (more correctly with no or minimum human effort).
It is important to emphasize that the precision and recall of the classifier should not be lower than the precision and recall of the operator having instructions and rules for the direction of the request to the departments. Moreover, it is even possible to increase the accuracy shown by the operator due to eliminating a human impact (errors occurring because of fatigue, negligence, loss of concentration, etc.).
The main approach of creating such classifiers is machine learning – the construction of a classification function (rules) on the examples. To develop such classifiers, we need labeled data (documents and their corresponding categories - labels). The labels are assigned by the experts (supervised learning). A set of labeled data is used to predict the labels of unseen data. In this research, the number of classes is six and all requests can belong to only one category
A variety of methods have been proposed to categorize unstructured information.
The most widely used classifiers are Naive Bayes, Logistic Regression, Nearest
Neighbor, Decision Trees, Neural Networks, Support Vector Machines, and others [
As described above standard classifiers have a bias towards classes that have more
texts and tend to predict the majority class. As a result, we gain misclassification of the
minority class (in comparison with the majority class). That is way conventional
approaches to classifying texts demonstrate rather bad performance when faced with
imbalanced datasets. There are two effective ways how to operate with unbalanced sets
[
First, we can change samples (data preprocessing). The goal is either to increase the number of documents in the minority class or to reduce the number of documents in the majority class. This is done to obtain approximately the same number of instances for both classes. The most popular techniques are random undersampling (randomly eliminating majority class text examples) and oversampling (increasing the number of text instances in the minority class first through simulation). Our experimental studies show that the disadvantages of these approaches outwit the advantages, and the quality of the classification changes only slightly.
Second, we can improve classifiers to make them appropriate for imbalanced data sets. These studies are conducted as part of the construction of ensemble classifiers to improve the performance of single classifiers by voting (and aggregating their predictions).
Many researchers have investigated the technique of combining the predictions of
multiple classifiers to produce a single solution [
In this article, we apply and explore Gradient Boosting Machine for the
categorization of requests. The arguments in favor of usage GBM can be formulated in the
following way:
In comparison with other ensemble classifiers, GBM generally performs better than
Random Forests and AdaBoost (although there is a price for that: GBM has a few
hyper-parameters to tune, while Random Forest and AdaBoost are almost
tuningfree). The possibility of tuning gives researchers more options to achieve better
results on a particular dataset.
The Kaggle competitions (the most popular testing ground of machine learning
techniques) demonstrate high universality GBM and efficiency for a wide range of real
applications (https://www.kaggle.com/).
The availability of software implementations modifications of the method for
solving various practical problems (for example, algorithm XGBoost - eXtreme Gradient
Boost – the variant of classical GBM with some additional heuristics [
This paper is organized as follows. In the first section, we consider pre-processing tasks (extracting text, tokenization, stop-word removal, normalization, named entity recognition) and give descriptions of two models of text representation (Bag of Words and Word2Vec). In the second section, we briefly explain Gradient Boosting Machine, setting its parameters on the simples and compare the quality of classification under using Bag of Words and Word2Vec. 2
For the learner to compute a classification function, it needs to understand the
document. For the learner, the document is merely a string of text. Hence, there is a need to
represent the document text in a structured manner. The most common technique to
represent text is the Bag-Of-Words (BOW) model [
The other model of text representation is Word2Vec [
Word2vec takes as its input a large corpus of text and produces a vector space,
typically of several hundred dimensions, with each unique word in the corpus being
assigned a corresponding vector in the space. Word vectors are positioned in the vector
space such that words that share common contexts in the corpus are located close to
one another in the space [
Word2vec has two main learning algorithms: CBOW (Continuous Bag of Words) and Skip-gram. CBOW is "a continuous bag of words" a model architecture that predicts the current word based on its surrounding context. The Skip-gram architecture works the other way: it uses the current word to predict the surrounding words. The user of Word2vec can switch and choose between the algorithms. The order of context words does not affect the result in any of these algorithms. Obtained at the output of the coordinate representation of the vectors of the words allow us to calculate the "semantic distance" between words. Word2vec technology makes its predictions based on the contextual proximity of these words. Since Word2vec tool is based on neural network training, it is necessary to use large datasets to train it to achieve its most efficient operation. This allows improving the quality of predictions.
To apply models, the original samples first must be processed specially, taking into account the peculiarities of the language to identify the most informative features (Fig.1).
This processing included: Removing punctuation marks and service characters using the Re library and regular expressions. Reducing words to lowercase. Splitting the text into separate tokens using"whitespace". Reduction of words to normal form. Individual tokens were normalized using the pymorphy2 library. The addition of the negative particle to the next word. The meaning of some requests depends on the negative particle and affects the meaning of the entire request. Therefore, it is important to save the negative essence. Removing stop words and template phrases (clichés). It was performed using two dictionaries. Experts compiled the first one manually. It contained a list of words directly related to the automated system. The second dictionary was the standard dictionary from the NLTK library (Natural Language Toolkit). Named entities recognition for further linking. The task of named entity recognition generally involves finding named entities in the text and determining their types, as well as linking a named entity to an object. The latter is important since identical words/phrases can refer to completely different objects. For example, the word “Lena” in Russian can be: The name of the person The name of the river The name of the town The name of the highway The types of named entities in the work included: Dates - 07.02.2019, 28 November 2018, 1995-2003; Numbers – 1, 3.5, 547899.00; Addresses – Novosibirskaya street, 23; Time – 9:30, from 13:00 to 14:00; Currency units and money amounts - 5000 rubles, $10; Names of automated systems - SAP, CRM; Positions - Manager, engineer, senior analyst; To recognize the named entities in the text, both the specificity of their entry and dictionary resources are used: dictionaries of names, geographical names, monetary units, occupations, etc. In this work, regular expressions are used to recognize named entities, in addition to special dictionaries supplemented with the subject of the problem.
It is well known that the quality of classification with GBM is significantly influenced by the dimension of vectors and terms used to describe documents. Therefore, it is important to select the best representation of the documents. In this paper, we conduct a comparative analysis of two approaches to feature extraction BoW and W2V. Based on our experiments, we identify an approach that provides a higher quality of classification with Gradient Boosting Machine. We make our conclusion based on analysis of values of precision and recall (as well as their combination – F1-measure). 3
[
As noted earlier for the building of the classifier we use boosting technique which
creates a highly accurate prediction rule by combining many weak and inaccurate
rules. Each classifier is serially trained with the goal of correctly classifying texts in
every round that were incorrectly classified in the previous round. After each round, it
gives more focus to texts that are harder to classify. The quantity of focus is measured
by a weight, which initially is equal for all instances. The most common representatives
of the boosting family are Adaptive Boosting (AdaBoost), Gradient Boosting (GBM),
and XGBoost (eXtreme Gradient Boost). In Gradient Boosting Machine the principle
idea behind the algorithm is to construct the new base-learners to be maximally
correlated with the negative gradient of the loss function, associated with the whole ensemble
Gradient Boosting is an approach to approximate a function ∗: → by a
func
( ) = ∑ =1 ℎ ( )
(1)
where ∈ R are real-valued weights, ℎ : → are weak learners and x ∈ is
the input vector [
Gradient Boosting can be seen as a generalization of AdaBoost. The latter is
designed to minimize the exponential loss with stump weak learners ℎ : → {+1, −1},
while Gradient Boosting can make use of real-valued weak learners and can minimize
other loss functions [
Let us discuss in more detail the description of the short text documents (requests for service) and training samples. The request that must be classified consists of two parts: “Information” – a brief description of the problem, written by the user and “Options” that may contain additional information about the request, for example, user’s department, system information, etc. As a result of combining these two parts, we obtain a document for classification.
To perform the experiments we downloaded and processed 9000 user requests. After that corpus was divided into two parts – training set and testing set – in the ratio of 67 and 33 percent respectively. Each class in the dataset has a different volume, which is shown in table 1.
To perform the best classification result we need to adjust the parameters of the Gradient Boosting Machine (http://xgboost.readthedocs.io). For some parameters were used the recommendations of the developers. But for the main parameters that have the greatest impact on the accuracy indicators, a special adjusting was carried out with the GridSearchCV module which performs an exhaustive search over specified parameter values for an estimator using k-fold cross-validation (k = 5). The next table shows parameters with the description, range, and selected value.
vents overfitting. After each boosting step, we can directly get the weights of new features, and eta shrinks the feature weights to make the boosting process more conservative.
Maximum depth of a tree, increase this value will make the model more complex/likely to be overfitting. 0 indicates no limit, limit is required for depth-wise grow policy.
Specify the learning task and the corresponding learning objective
To perform feature extraction first we break sentences into words. On this step, we had a deal with 469880 words and after preprocessing our training corpus contained 230905 informative words which we gave to the input of BoW and W2V.
To perform word extraction with Word2vec using CBoW algorithm we applied the following parameters: Learning algorithm - Hierarchical softmax. The minimum number of repetitions of the word in all sentences -min_word_count = 40. The dimension of the resulting space - num_features = 300 (default value). How many words "around" a particular word should be considered Context = 10 (Normally used value). Parameter «sampling» makes the model choose very-frequent words less often in the training process - sampling = 1e-3 (default value).
Evaluation of a classification algorithm performance is measured by the Confusion Matrix which contains information about the actual and the predicted class.
False Negative (FN)
True Negative (TN) =
+ =
+ 1 − = 2( ( ∗ + ) ) (2) (3) (4) The results of classification with GBM and two methods of feature extraction are presented in tables 4 and 52. The results are obtained on the testing set.
C1 0,92 0,89 0,90 C2 0,91 0,92 0,89 C3 0,89 0,90 0,90 С4 0,95 0,97 0,93 С5 0,88 0,86 0,85
As we can see prediction with GBM using Bag of Words provides a slightly better
result than Word2vec but it is still not good enough to replace a human operator.
According to the experience of the Bank human-operators perform such task
approximately with f-score = 0.92 - so our goal is to perform an average f-score of at least 0.93
[
Although that BoW model provides better results on average it also performs the worst result in C5 category. Results of W2V model are smoother than the results of BoW model so we could note that W2V model is more resistant to the size of the dictionary. 4
Currently, the models and methods considered in the article are tested in one of the
branches of a large bank in Russia. During testing, we focused on the analysis of the
causes of errors. The main conclusion is that most of the errors are due to incorrect
filling of requests, in particular, the use of slang, abbreviations that correspond to the
vocabulary of modern messengers and social networks. To improve the prediction
quality, we plan to achieve through the following:
─ as the number of requests increases to expand the training set to achieve the more
precise tuning of the parameters for Word2vec, and also make our sample balanced
by extending small classes and discarding some objects of large classes;
─ try W2V skip-gram algorithm. It is more computationally expensive than CBoW but
performs better results in models using large corpora and a high number of
dimensions [
At the same time, the conducted testing and discussion of its results with the Bank's specialists allows identifying several similar processes that can be automated based on Text Mining techniques.