This paper focuses on the eld of sentiment analysis of natural language texts. To be precise, short texts extracted from social networks are analyzed. At present two groups of applied methods can be distinguished in this eld: machine learning methods and methods based on sentiment lexicons. The paper reviews the principal methods in this eld and proposes an approach for short text sentiment analysis, combining sentiment lexicons and blending of machine learning algorithms for the problem of three-class text classi cation. Various formulas for determining weights of the words in vector representation of texts are considered as well. This approach is applied to a dataset that consists of 10,000 manually marked posts extracted from VKontakte social network. Class distribution of dataset objects for the studied area often tends to be unbalanced. Standard models show moderate quality but only due to the fact that the majority of data points are classi ed as belonging to the dominant class. In this work an approach that shows good metric scores for all the three classes is described.
The amount of data collected from users around the world has grown greatly over the past decade as websites such as Twitter, Amazon and Facebook have facilitated publication and aggregation of micro opinion pieces that allow individuals to record their sentiments towards things, people and events.
Sentiment analysis is the process of determining whether a text relates to
one of given classes. It is also known as opinion mining and aims to determine
the attitude of a writer with respect to some topic or the overall contextual
polarity or emotional reaction to a document, or event, or some other object. The
information received after the analysis is clearly of value to researchers,
organizations and companies trying to understand sentiment both for individuals and
on average, as well as to identify trends [
At present two groups of applied methods can be distinguished in the eld: machine learning methods and methods based on sentiment lexicons. The aim of this work is to develop a method combining the two mentioned approaches as well as a meta ensemble of machine learning models that would cope with the mentioned problem of strong class imbalance.
Blending of Logistic Regression, Random Forest Classi er, Gradient Boosting
Classi er, and set of features is used to combine the information received from
all the parts of the ensemble. Among added features there are metrics received
from using emojis extracted from texts, sentiment lexicons, application of
Relevant Frequency method [
The developed method exceeds quality of standard classi ers on given dataset and shows high value of F 1 metric even with a strong class imbalance.
The paper is organized as follows. In section 2 related work is overviewed, then the proposed approach is described. In section 4 conclusion is provided. 2
Most commonly applied methods in text classi cation and sentiment analysis
tasks in particular can be divided into two groups: methods that use sentiment
lexicons [
Lexicon based techniques rely on an assumption that the collective polarity
of a document or sentence is the sum of polarities of the individual words or
phrases. In [
For machine learning methods texts are usually preprocessed by removing
stop words and irrelevant information[
Neural networks are often applied to text classi cation tasks and for
sentiment analysis in particular. CNN or RNN are used very often. In [
Sentiment analysis is a growing area of research. There have been a lot of related papers at conferences of di erent levels for some years and new works still appear very often. 3 3.1
The dataset for three-class sentiment classi cation was provided by Mail.Ru Group. It was retrieved from VKontakte social network in July, 2017 and consists of about 10,000 manually marked posts. Four people took part in the marking process and each post was assessed by one person. The quality of the classi cation was evaluated by leading assessor.
Posts from social networks have several speci c features. They are usually short, contain mistakes and slang expressions, as well as emojis, useful for polarity identi cation. Additionally, their content is dynamic and often dependent on current events. So, the vocabulary is very rich and training a classi er on embedded texts is rather di cult because of certain inconsistency due to the mixture of domains in the dataset. Most posts in the dataset are written in Russian, although there is a large amount of posts in English and some posts in other languages which were kept as they didn't signi cantly in uence the quality of the model. The average length of texts is 29 words. The distribution of classes in the dataset is very unbalanced. There are 6380, 2700 and 600 posts of neutral, positive and negative classes respectively. 3.2
Raw data usually needs to be preprocessed before being passed to any machine learning algorithm as there are typically a lot of symbols which are meaningless for a given task and keeping them can reduce the quality of the model. The preprocessing of text means cleaning of noise such as cleaning of stop words, punctuation, terms which doesnt carry much weightage in context to the text.
When dealing with vector space models stemming or lemmatization is usually applied to the text to reduce the size of the dictionary and to give a classi er an ability to learn information from the words not running after words forms.
In this paper the following preprocessing is conducted. Stop words are removed using Python nltk library. The standard set of Russian stop words is extended. HTML tags are removed using regular expressions.
Yandex stemmer M ystem1, Snowball Stemmer2 and Morph Analyzer3 are applied for stemming. After stemming TfIdf Vectorizer from sklearn Python library with default parameters is used for vector representation of the texts. The LogisticRegression from sklearn with default parameters is then applied for classi cation. The results are shown in table 1. F 1 metric score for each of the three classes is provided as well as an aggregated F 1 score. F 1 metric is calculated as weighted average of the F 1 scores for all classes. In this section and in all further sections General score represents F 1 metric calculated for all the three classes and Pos-Neg score represents F 1 metric calculated as weighted average for positive and negative classes.
For our task MyStem showed better results in comparison and it was chosen
for future investigations. In [
Among models that are commonly used for text classi cation are Logistic Regression, SV M , Multinomial Naive Bayes, Random Forest Classi er, neural networks.
The dataset is divided into train and test parts and cross validation method is used. The following classi ers over TF-IDF feature vectors with preprocessing described in the previous section are applied: Logistic Regression, Random Forest Classi er, SVM, Gradient Boosting Classi er, KNeighbors Classi er, Multinomial Naive Bayes. The results are shown in table 2.
SVM, Logistic Regression, Random Forest Classi er and Gradient Boosting Classi er demonstrated better results and are chosen for future consideration. The use of neural networks is not applicable in straight-forward way because it requires a large amount of labeled data for training. The SVM classi er over standard TF-IDF model form sklearn from this section is used as a baseline model, and the Logistic Regression classi er over standard TF-IDF model is shown as simple Logistic Regression for comparison with other Logistic Regression models in the next sections. 1 https://tech.yandex.ru/mystem/ 2 http://www.nltk.org/api/nltk.stem.html 3 http://pymorphy2.readthedocs.io/ It is rather di cult to achieve good quality just using standard classi ers. So the task was to add features that would help the model to learn information about the objects of all the classes. As there are a few examples of positive and negative classes the speci ed problem is hard to be solved using only available dataset. That is why it was decided to get unlabelled objects from VKontakte social network and to build a classi er that would extract more examples for negative and positive classes.
The following approach is used for this task. The LogisticRegression classi er is tted on labelled data to predict probability for each of the three classes. The data is preprocessed as it is described in the previous section. The idea is that if the trained classi er is very sure that the object should be related to a certain class the probability of being mistaken when adding this object to the dataset with corresponding label is low. The thresholds for probabilities returned by trained model for positive an negative classes were chosen manually to supplement the positive and negative classes. About 3000 of new examples were extracted. These objects were used to calculate some statistics. The more detailed description is provided in the next sections. There are many emoticons in the social network posts and they are very useful when identifying the sentiment polarity of the text. Smiley symbols received from the text are used in some ways. First they are added as features to TF-IDF model together with lemmatized words.
The probability distribution for each smile symbol to belong to each of three classes is calculated using both data from the original dataset and objects from extracted dataset which is described in the previous section.
Pi;k = NNik;k ; k 2 f 1; 0; 1g
Where Ni;k is the number of posts in which smiley symbol i is found and its label is equal to k in the united dataset, and Nk is the number of posts in which smiley symbol i is found in the united dataset.
The following features are added to the classi er. The average probability distribution for the three classes for all the smiley symbols in each post is calculated and these three numbers are used as features for the classi er. The number of smiley symbols in each post is also added as a feature to the model. The smiley label is calculated for each post as the class with the highest probability considering all the smiley symbols: Cj = argmaxk Pj;k, where Pj;k is the probability of class k for document j considering all the smiley labels in the document j.
Smiles were also added as features to TF-IDF model and this raised the value
of F1 metric. The results can be viewed in table below. The LogisticRegression
classi er over TF-IDF that is described in section 2 is chosen for consideration.
The marked sentiment lexicons from [
Sentj = PnN=j1 sentn Where Nj is the number of words in the text j found in sentiment lexicon, and sentn is the sentiment mark for word n from sentiment lexicon. 3.7
Relevant Frequency [
The value is calculated as RFic = log2( max(1;b) ). Where a is the number documents containing word i that belong to class c in the training dataset and b is the number documents containing word i that not belong to class c in the training dataset.
For each word from train dataset the value of Relevant Frequency is calculated using the above formula using train dataset and additional dataset which is described in section 3.4.
The metric is calculated for each post as the sum of di erences of the relevant frequency for the positive class and the relevant frequency for the negative class for each word in the training set, and this gure is added as a feature for the classi er.
RFsentence = Pwords(RFw+ord
RFword) Where RFw+ord is the Relevant Frequency for a given word for positive class and RFword is the Relevant Frequency for a given word for negative class. 3.8
The vector space model is used very often in machine learning tasks related to
texts. In this approach individual documents are represented as vectors in term
space. Bag of words, TF-IDF, word2vec are three most commonly used vector
space models. Bag of words model was tried but it showed lower metric score
in comparison. FastText [
In TF-IDF vector space model the term weight is given by product of Li;j ,
Gi, Nj where Li;j is a local weight for term i in the document j, Gi is a global
weight for term iand Nj is the normalization factor for document j [
Local weights are functions of how many times each term appears in the document, global weights are functions of how many times each term appears in the entire collection, and the normalization factor compensates for discrepancies in the length of the documents.
The following weighting formulas are applied. For local weight, binary (BN RY ), within-document frequency (F REQ), log (LOGA), normalized log (LOGN ), augmented normalized term frequency (AT F 1) and the one that is used in sklearn. For global weights: inverse document frequency (IDF B), probabilistic inverse (IDF P ), entropy(EN P Y ), global frequency idf (IGF F ), Idf from sklearn, no global weight (N one). For normalization factor: cosine normalization (COSN ), pivoted unique normalization (P U QN ), no normalization (N one).
For this task of classi cation the following TF-IDF models were chosen: h ATF1, None, COSN i and h LOGN, ENPY, PUQN i.
The local weights from chosen models are calculated as follows:
AT F 1 = <80:5 + 0:5 fi;j ; if fi;j > 0
xj :
0; if fi;j = 0 8 1 + log fi;j ; if fi;j > 0 LOGN = < 1 + log aj 0; if fi;j = 0
Where fi;j is the frequency of term i in document j and xj is the maximum frequency of any term in document j.
Where aj is the average frequency of the terms that appear in document j.
The global weight EN P Y is calculated as follows:
EN P Y = 1 + PN j=1 fi;j log fFi;ij Fi log N
Where Fi is the frequency of term i throughout the entire collection and N is the number of documents in the collection.
The normalization factors from chosen models are calculated as follows: 1
COSN = pPim=1(Gi Li;j)2
Where Li;j is local weight of term i in the document j, Gi is global weight of term i, m is the number of terms in document j.
P U QN = (1
1 slope) pivot + slope lj
Where lj is the number of distinct terms in the document j, slope is set to 0.2, pivot is set to the average number of distinct terms per document in entire collection.
The application of these formulas to the model is described in the next
sections. The other formulas and description for them can be found in [
Ensemble methods, such as blending and stacking, are designed to boost
predictive accuracy by combining the predictions of multiple machine learning models.
Recent work has shown that the use of meta-features, additional inputs
describing each example in a dataset, can boost the performance of ensemble methods
[
The two chosen TF-IDF models which are described in the previous section could perform better if to combine their predictions. The model with h ATF1, None, COSN i weights performs better general metric score in comparison to other models while h LOGN, ENPY, PUQN i model show higher metric quality for negative class.
The following approach is used. Two Logistic Regression classi ers are applied over vector representations received from each of the two chosen TF-IDF models. If the classi er over h LOGN, ENPY, PUQN i is very sure that the object should be related to the negative class and the probability for this object to belong to negative class predicted by the classi er over h ATF1, None, COSN i model is not low then the prediction of the rst classi er is chosen. Otherwise the result of the second classi er is taken. These two thresholds were chosen using cross-validation method. The comparison of performance of these two models and their combination is shown in table 4. The pseudocode for described rule is as follows.
if clf 1:result > thr:clf 1 and clf 2:result > thr:clf 2 then
obj.class = clf1.result else
obj.class = clf2.result end if The following designators are used in pseudocode: { clf 1:result is the probability for negative class predicted by classi er over h LOGN, ENPY, PUQN i TF-IDF model; { clf 2:result is the probability for negative class predicted by classi er over h ATF1, None, COSN i TF-IDF model; { obj:class is the resulting label for the object; { thr:clf 1 and thr:clf 2 are the lowest probability of negative class for the rst and the second classi ers respectively for the object to be related to negative class;
The applied blending strategy is as follows.The training subset of the dataset is divided into two parts. The Logistic Regression, SVM, Random Forest Classi er, Gradient Boosting Classi er tted on the TF-IDF vector representations of the texts from the rst part are applied to get the predictions on the second part of the train dataset and on test dataset. For Logistic Regression the output from one of the two classi ers is chosen as it is described in the previous paragraph. For the other two models h ATF1, None, COSN i variant of TF-IDF is used. For Logistic Regression the probability distribution of classes is taken and for the other classi ers the predicted class label is used. The predictions of the classi ers are used as features for another classi er together with the other calculated features described in the previous sections. The feature that corresponds to SVM is excluded from the dataset because without this feature the metric score is higher. At this nal step of blending the Logistic Regression is tted on these features on the whole training data and predictions for the test data are received. Table 4 shows that the applied ensemble method increases the value of F 1 metric. However there are still mistakes in the predictions while the value of some features corresponded to true label. For example, the classi er relates the text to neutral class but the values of sentiment score and relevant f requency score are very low, and the true label is negative. The thresholds were chosen using cross validation according to which the decision is made despite the resulting label of the classi er. The pseudocode of the rules over the classi er for a single object is as follows.
if clf:result ! = 0 then
obj.class = clf.result else if obj:rf < rf:thr and obj:sent < sent:thr and obj:neg > neg:thr and obj:pred > pred :thr then obj.class = -1 else if obj:rf > rf:thr and obj:sent > sent:thr and obj:pos > pos:thr and obj:pred+ > pred+:thr then obj.class = 1 else
obj.class = clf.result end if The following designators are used in pseudocode: { clf:result is the label predicted for the object by the classi er; { obj:class is the resulting label for the object; { obj:rf is the value of Relevant Frequency metric described in the 2 section; { obj:sent is the value of Sentiment metric described in the 2 section; { rf:thr , sent:thr, neg:thr, pos:thr , pred :thr, pred+:thr are chosen thresholds for Relevant Frequency, Sentiment Value, number of negative and positive words from the sentiment lexicon, the probability returned by the classi er for negative and positive class respectively;
The described method helps to improve the quality of the model. The results of this step in comparison to previous step are shown in table 5. 3.11
In this section an overview of experiments and evaluated components is provided.
Three variants of stemmers were applied for preprocessing: My Stem, SnowballStemmer and MorphAnalyzer. My Stem demonstrated better results in comparison both for the rst steps of the applied model and for the whole pipeline. Stop words were removed using N LT K Python library. All meaningless symbols were also removed with help of regular expressions.
Di erent classi ers were applied over vector representations of posts from dataset. Logistic Regression, Random Forest Classi er, SVM, Gradient Boosting Classi er, KNeighbors Classi er, Multinomial Naive Bayes were tested over different TF-IDF formulas for local, global weights and normalization.
Di erent TF-IDF formulas were considered. About 500 models were tested altogether and the best variants were chosen. It was noticed that many models with N one for global weights and normalization performed better. This can point to the fact that local global weights and normalization dont play signi cant role in the task of sentiment analysis classi cation. It was also noticed that using P U QN normalization leads to better results in F 1 score for negative class.
Logistic Regression, RandomForestClassi er, GradientBoostingClassi er as well as h ATF1, None, COSN i and h LOGN, ENPY, PUQN i variants for TF-IDF were chosen according to their better performance for being parts of blending model together with the generated features received from using Relevant Frequency method, sentiment lexicons and emoticons extracted from posts. Rules were applied over the predictions of the classi er to correct its mistakes.
Final model shows much better quality then standard classi ers. The main achievement of the proposed model is the increase of F 1 score for positive and negative classes. 4
In this paper an approach for the problem of sentiment analysis that combines machine learning methods and sentiment lexicons is demonstrated. Di erent classi ers and di erent variants of formulas for computing TF-IDF values for vector representation of texts are evaluated and the best of them for a given task are chosen.
Final model shows better quality according to F 1 score in comparison to standard classi ers. The main achievement of the proposed approach is that it helps to increase F 1 score for positive and negative classes as standard classi ers show moderate quality on test dataset but only due to the fact that the majority of data points are classi ed as belonging to the dominant class because of strong class imbalance.