The article discusses two approaches to solving the problem of sentiment analysis: lexicon-based approach and deep learning. Within the first approach two well-known lexicon-based methods has been adapted for the Russian language - SO-CAL and SentiStrength. For these methods a unified sentiment lexicon has been prepared using a voting procedure based on the existing lexicons. The second approach has used the RuBERT deep learning model. The SentiRuEval-2015 corpora, which provides reviews and tweets, has been used as training and test data. Analysis of the results showed that deep learning model demonstrates higher accuracy compared to lexicon-based methods.
Sentiment analysis is a field of computational linguistics aimed at automated research
of people’s opinions and assessments in relation to various objects mentioned in the
text, for example, products, services, organizations, persons, events [
There are many studies in the field of sentiment analysis, mainly for the English
language [
There are three main approaches to the sentiment analysis in texts – lexicon-based,
machine learning, and hybrid, in which the two indicated approaches are combined
[
Examples of existing lexicon-based sentiment analysis methods are SO-CAL [
The second approach to sentiment analysis is machine learning, within which there
are two areas: traditional machine learning (for example, such methods as SVM,
Naïve Bayes, and Gradient Boosting) and deep learning (for example, models based on
the Transformer architecture such as BERT) [
The purpose of this work is to compare the performance of the lexicon-based methods SO-CAL and SentiStrength adapted for the Russian language and the BERT deep learning model for sentiment analysis, as well as to explore the possibility of adding information from sentiment lexicons to deep learning models. 2 2.1
SO-CAL† (Semantic Orientation CALculator) is a method developed by Maite
Taboada that determines the sentiment of texts [
The first change affected preprocessing – for Russian texts it is required to determine not only the part of speech of tokens, as in the original version, but also their initial form. The rnnmorph‡ module was used for this.
Secondly, we used a lexicon created by combining existing sentiment lexicons (described below). With the help of rnnmorph the combined lexicon was split into four separate lexicons for different parts of speech: nouns, adjectives, verbs, adverbs. If an element was attributed to several of the specified parts of speech, it was used in several lexicons.
Thirdly, Russian-language lists of special words and expressions, used in the algorithm, were formed, in particular, a lexicon of modifiers (words that affect the sentiment of the words to which they belong, for example, less, absolutely) and lists of negations (for example, nothing, without).
As in the original version of the method, the Russian version ignores the sentiment words in the sentence if there is a condition word from the special list in the sentence. Such list includes conditional markers (for example, if), some verbs (like expect, doubt), questions and words enclosed in quotation marks (which may be factual, but † https://github.com/sfu-discourse-lab/SO-CAL. ‡ https://github.com/IlyaGusev/rnnmorph. do not necessarily reflect the opinion of the author). If repeating the same sentiment word, there is a decrease in weight for each subsequent repetition.
The version of SO-CAL adapted for the Russian language, just like the original one, for each text produces a numerical value corresponding to the degree of sentiment of the text: a value greater than zero indicates a positive sentiment, a value less than zero indicates a negative sentiment.
On the training part of the corpora a threshold is determined that separates the texts
into positive and negative ones (for a three-class classification two thresholds are
selected to divide into positive, neutral and negative ones). Next, the sentiment of the
texts of the test part is determined, taking into account the found thresholds.
SentiStrength is a lexicon-based method developed by Mike Thelwall et al. [
The method was originally developed for the English language, we adapted it for the Russian language by changing the linguistic resources required for the algorithm. These are a list of sentiment words, a list of modifiers – words that raise or lower the sentiment score of the following words (for example, bad, a little, very, extremely), a list of negations (for example, not, never), a list of words that indicate the presence of a question in a sentence (for example, how, when, why), etc. The replacement of such language-independent resources as the list of emoticons was not carried out.
Experiments with SentiStrength were performed with two versions of the datasets. The first version is raw, unprocessed data. The second version is preprocessed (lemmatized) data.
RuBERT. In addition to the SO-CAL and SentiStrength methods, we used a model
based on the Russian-language version of BERT – RuBERT [
In experiments with the RuBERT model two variants of text corpora were used: a corpus without preprocessing and a corpus in which positive words present in the combined lexicon were replaced by good, and negative words by bad. This preprocessing procedure made it possible to test the hypothesis of a potential improvement in the performance of sentiment analysis based on RuBERT when adding information from the sentiment lexicon. 2.2
The main linguistic resources for solving the problem of sentiment analyses are sentiment lexicons and text corpora labelled by sentiment.
Sentiment lexicons. The combined sentiment lexicon was formed using nine publicly
available lexicons for the Russian language [6; 12].
1. EmoLex [
The lexicon contains the collocations that received the majority of votes.
6. Word Map [
Each lexicon has been separately processed as follows: ─ neutral words have been removed; ─ all words have been converted to lower case; ─ words that are both positive and negative in the lexicon have been removed (including the analyzed words with the spelling "е" and "ё"); ─ words containing non-Cyrillic letters in the spelling have been removed; ─ one occurrence of each element has been left.
The size of lexicon after preprocessing is given in Table 1. At the next stage a combined sentiment lexicon was obtained from the preprocessed lexicons. This lexicon includes words that are simultaneously found in at least four source lexicons: 1,444 negative words (67%) and 712 (33%) positive words, a total of 2,156 words. At the same time only those words were left in which the sentiment was clearly defined, i.e. there were no controversial cases. A controversial case was considered when the number of lexicons in which a word was classified as negative coincided with the number of lexicons in which it was positive. For the rest of the cases the words were assigned to the prevailing class, i.e. the voting was used. Other values were investigated for the minimum number of lexicons, but for four the best performance scores were obtained on the training data.
We created two versions of the combined lexicon – in the first one positive elements were assigned a sentiment score of 3, negative ones – a score of –3. This version of the lexicon is hereinafter referred to as CLex. In the second version of the lexicon the sentiment score was determined by the number of source lexicons that contained a given word. The second version of the lexicon is weighted and is further called WCLex.
Text Corpora. For experimental research the corpora of the SentiRuEval-2015§ competition were taken. The data consists of labelled reviews of restaurants and cars, as well as tweets about banks and telecommunications companies. Corpora sizes are very different: there are fewer reviews than tweets, but the average review length is almost 10 times longer than the length of a tweet (830 characters versus 85 for training data and 830 versus 82 for test data). This is due to the maximum possible tweet length. The organizers of the competition provided training and test data. The training sample consists of 403 reviews and 9,722 tweets. The size of the test sample is 403 reviews and 8,308 tweets (Table 2). § http://www.dialog-21.ru/evaluation/2015/sentiment/.
Category Training
Test 3
Sentiment Negative Neutral Positive Total Negative Neutral Positive Total Three models were tested – the lexicon-based methods SentiStrength (SS) and SOCAL adapted for the Russian language, as well as the RuBERT deep learning model.
For lexicon-based methods, two variants of the combined lexicon were used: CLex and WCLex. For SO-CAL each corpus preprocessed with rnnmorph was evaluated on both lexicons. For SentiStrength estimates were obtained on the original raw data and on the preprocessed lemmatized data. For the RuBERT model two variants of text corpora were used: corpora without preprocessing and corpora with replaces to good and bad of words from sentiment lexicon.
Two series of experiments were carried out. In the first series only texts with a positive and negative sentiment were used, thus a binary classification was carried out. The second series of experiments was carried out for a three-class classification – texts with a neutral sentiment were also used.
Table 3 shows the values of the macro F1-score metric on test data for binary classification.
Cars 0.8247 Among the lexicon-based methods, on average, for all experiments, SO-CAL showed better results by 10 percentage points (pp), while in the classification of reviews, the results were better for SentiStrength, tweets – for SO-CAL. The use of a weighted lexicon for both methods sometimes even worsens, but on average not significantly improves the performance (by about 9 pp on average, and by 28 pp maximum for tweets about banks when using SentiStrength). In general, among the lexicon-based methods the best result was shown by SO-CAL when using WCLex.
The RuBERT model is superior to lexicon-based methods. On average for all experiments its result exceeds the results of lexicon-based methods by 22 pp (in comparison with the best lexicon-based method – by 15 pp). The experiment carried out to replace words with subsequent fine-tuning of the RuBERT model showed a good result only in one case – for reviews on cars, the quality increased by 2 pp, in other cases such preprocessing deteriorated the performance.
The performance of the classification of reviews is on average 32 pp higher than the performance of tweet classification, the largest difference (by 59 pp) for SentiStrength on lemmatized data with CLex.
Table 4 shows the values of the macro F1-score metric on test data for a three-class classification. With a three-class classification among the lexicon-based methods on average for all experiments again by 10 pp SO-CAL performed better, with better results for both reviews and tweets. The use of a weighted lexicon for both methods has very little effect on the performance. In general, among the lexicon-based methods the best result was shown by SO-CAL when using WCLex.
The best results for the three-class, as well as for the binary classification, are shown by the RuBERT model. On average for all experiments its results exceeds the results of lexicon-based methods by 16 pp (in comparison with the best lexicon-based method – by 10 pp). Replacing words in the original data did not improve the result. The performance of the classification of reviews and tweets on average differs slightly (by 3.5 pp).
The experiments showed that the RuBERT deep learning model in all cases gets better results than lexicon-based methods (compared to the best SO-CAL lexicon-based model with a weighted lexicon – by 15 pp for a binary and 10 pp for a three-class classification). However, the results of the neural network model are difficult to interpret, and the lexicon-based methods provide detailed information about the sentiment words and expressions found in the text. This allows, in particular, analyzing the errors that these methods made.
The analysis showed that for reviews the most common causes of errors are, for example, the following: incorrect search for negation, search for not all sentiment words, prevalence of vocabulary of the opposite sentiment, a problem of the third class, when sentiment 0 is used not only for neutral texts, but for contradictory ones also. There are also two other common types of errors for tweets: lack of knowledge of the context and incomplete phrases, which are related to the specifics and limitation of the number of characters in one text message. 5
The use of the RuBERT deep learning model has a higher performance (by 22 pp for two-class classification and by 16 pp for three-class classification) compared to the adapted versions of the SO-CAL and SentiStrength lexicon-based methods, but its results are difficult to interpret. The ability to analyze errors allows you to identify ways of improving lexicon-based methods.
Adding information from the lexicon during preprocessing data for RuBERT led to an improvement in the result only in one case out of eight.