The paper is focused on comparing the performance of different techniques of text tonality analysis, a widely used approach in business to conduct social listening research. However, there are still debates on what type of models perform better for NLP classi cation tasks. On a corpus of Russian language tweets three models were tested to solve binary classi cation problem: Logistic regression (LR), XGBoost classier and Convolutional Neural Network (CNN). The paper descriptively overviews main techniques useful for data cleaning and preprocessing for these methods and covers its possible tfalls. Based on the study CNN showed best results among chosen models, which goes in line with several articles in the eld for other than Russian languages. Together with high predictive results, neural networks exhibit a computational drawback their performance is poor in terms of timings. Besides this, all methods showed high sensitivity to the way data is preprocessed, which is a product of Russian language variability. This leads to a conclusion that there is still a room for improvement - in future research more emphasis will be put on hyper parameter tuning for "boosting type" models and extending list of applied methods.
Nowadays, more and more communication, services and goods transfer to the Internet where the information is basically provided in the form of text. In this regard, the task of determining the emotional state of a person without personal communication is becoming increasingly crucial. The perspectives of this eld are that, based on the textual information, it is possible to determine a mood of a person, or to estimate a success of political or economic reforms, or to check person's reaction to particular event or decision.
Copyright c 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Due to in practice it is not possible to identify the emotional coloring of large number of texts available in the Internet manually and, in addition, it is too labor-intensive, one of the possible ways to solve this problem is to use machine learning algorithms.
At the moment, there are several key methodologies for determining
emotional coloring of the text:
{ Analysis using pre-complied dictionary[
This paper compares several machine learning techniques to solve binary classi cation problem of text tonality analysis. The dataset used for this task is a corpus of Russian language tweets. After several data cleaning stages logistic regression (LR), XGBoost classi er and convolutional neural network (CNN) are applied. The study compares those models in terms of accuracy and F1 score and discusses possible pros and cons for these methods. 2
Proposed methods are based on the use of machine learning methods. The
developed prototype allows to assess texts as `positively' or `negatively' coloured.
To train these models a set of short texts were collected. Using RuTweetCorp[
Before applying discussed above methods, data cleaning stage should be
completed. To prepare the dataset for the models I used in the paper several classical
for the industry approaches are applied:
{ Capitalisation. The purpose of this stage is to make a proper text cleaning
and frequency calculation. Actually it makes no di erence what type of
register level to use for the modelling since, mathematically, models' algorithms
transform words into digits, thus any case type can be chosen. In the current
paper I have used lowercase register;
{ Punctuation cleaning. As a part of data preprocessing researchers usually
drop all punctuation marks, digits, links, etc. from texts since in most of the
times it does not give any impact on emotional meaning of tweets. Another
e ect of these symbols on the analysis is irrelevant results in terms of word
frequencies and a nities. Punctuation marks are comparatively often appear
in sentences which leads to extremely high metrics for them while in reality
these symbols almost always make no impact on text tonality;
{ Lemmatisation. Other well-known in the industry approaches to reduce
the number of words carrying similar emotional meanings are lemmatisation
and stemming. Both of these methods transform words from its full form
into their short parent version. The di erence between methods is that
lemmatisation derives an in nitive form of the words while stemming simply
cuts beginnings and endings of words to obtain a root. In the current paper
I use lemmatisation as I believe it to be more e cient[
Besides well-known in literature methods of texts cleaning in current paper I
have followed additional step improving algorithm performance - concatenation
of word \not" with the following one. It is intuitively understandable that the
negation word \not" can signi cantly change the emotional meaning of a tweet.
However, it was practically revealed that analysed models captured high
correlation between \not"[
Having done all that, tweets have been transformed to vectors consisting of cleaned in nitives of words. This still resulted in huge amount of unique words, which led to high memory and time consumption, while big chunk of words did not give any impact on model score. To optimise the modelling process words with frequency less than 3 were dropped. After that, a dictionary was created, which included remaining words to which a unique ID was assigned. Using such dictionary we have prepared two input datasets: { For LR and XGBoost: a TF-IDF matrix (term frequency-inverse document frequency), which is a method of "Bag of words"(BoW) class. While term frequency denotes how many times a word is in a document (in one piece of text input), inverse document frequency calculates number of documents where a text has appeared and number of total documents in data. In this method all the words in the data are transformed into a list. Afterwards all the words in this list are assigned to each document as a vector. For example if there is a dataset of 5000 words, each document has a vector of 1 row and 5000 columns. Each column in this vector de nes a word. If the word appears in the document, number of times the word is present in the text is assigned to the column corresponding to that word. When this process is done for all documents, illustration of data as per text frequency is obtained. Inverse Document Frequency (IDF) is calculated by using number of documents where a term has appeared and total number of documents in the data. { For CNN input: a Document-Term Matrix of special form - rows of this matrix were corresponding to tweet ID, columns corresponded to the position of the word in a tweet, while in each cell of the matrix were written an ID of the word from dictionary. Such matrix may have large number of columns in case of presence of very long texts. This leads to high memory and time consumption during computation. To reduce the time, the matrix were cut to 23 columns, which signi cantly improved model calculation time on one side and left all relevant information on another (23 columns cover 99.65% of all tweet lengths) 2.2
For the purpose of the analysis we have tested several machine learning methods and compared its e ciency (more on this in section 3). We have compared three typical models in the text mining eld to conduct semantic analysis of texts: logistic regression, XGboost and CNN. Short overview of these methods is provided below.
Logistic regression This is a well known approach used to solve binary
classi cation problems.[
1
1 + e x
(1)
where z is a set of model input factors (in our case these are vectors of TF-IDF
matrix). More on this type of regression is given by Cramer[
XGBoost Another model we have decided to test on the dataset is
XGBoost, which is highly credited by machine learning competitors[
Convolutional Neural Network (CNN) The last but not least model
type we have tested on the dataset is a CNN. Neural networks recently got high
attention among data scientist due to ability to solve almost any type of problem
once it is stated correctly. Text mining is one of the elds, where CNN showed
high performance[
To test the results of models performance we estimate shares of correct and
incorrect predictions on train and test set. It is a practical and reliable way
to deal with over- and under- tting. To measure the learning performance of
methods we use accuracy and F1-score metrics. These two metrices were chosen
based on research in the literature for binary classi cation[
Below we demonstrate obtained results for all three models. Table 1 shows Logistic Regression results:
Based on the results one can make a conclusion, that model is over- tted. This is for sure not a good sign, however, it can be ne-tuned by changing hyperparameters of the model and using cross-validation techniques. Overall, the score on test is considerably high. Another think to note - training of logistic regression take very small amount of time. This makes the method a good starting model to test how data preparation stages in uence model performance.
XGBoost training took much more time. Results you can see in the table below.
This model shows worse results than logistic regression. Probably, it happened because it needs a better setup of hyper parameters. Since in literature XGBoost proved to be one of the best models for these type of tasks, we are motivated to pay more attention to work with this model in the future. The paper provides with results of applying di erent machine learning techniques to solve the text tonality binary classi cation task. Three model types were estimated: logistic regression, XGBoost classi er and convolutional neural network. Each of them are well-known models among data scientists to solve tasks of that type. However, not that much studies exist where those models are applied and compared on social networks data and Russian language. Results demonstrate that based on F1 measure CNN performs better. However, training such a model needs much more time than LR. Hence, depending on available time and computing power for modelling LR may be preferred. Also, it was a surprise that XGBoost classi er showed signi cantly lower result than other models, while the framework demonstrated high performance on many data science competitions. We suspect that more time should be spent to nd optimal hyper parameters of the model.
For future research we are planning to extend the number of models tested at least, LightGBM and Word2Vec models are interesting frameworks showing good results for this type of problems. Besides that, more focus will be allocated on optimal hyper parameters search, cross-validation techniques and stacking of models. These we believe will lead to even greater model performance.