In this paper, we introduce a specialized Russian dataset and study approaches for aspect-based sentiment analysis of Russian users' comments about the COVID-19. We solve two tasks, namely Relevance Determination (RD), which aims to predict whether a sentence is relevant to an aspect of the pandemic, and Sentiment Classification (SC), which classifies the sentiment expressed towards an aspect in a sentence. We applied and tested various methods of machine learning, including finetuning of the pre-trained RuBERT model. The best results in both tasks were obtained by RuBERT model in the Natural Language Inference (NLI) formulation.
COVID-19 is a dangerous infectious disease caused by the SARS-CoV-2 virus.
Nowadays this infection is declared a pandemic and is one of the main threats
to humanity endangering both physical and mental health of people.
COVIDrelated issues are widely discussed in social media. Such discussions give great
opportunities for psychologists, social scientists to study information
dissemination in social networks, influence of various sources on forming users’ opinions
[
Extracting opinions related to coronavirus can be considered as the
aspectbased sentiment analysis task (ABSA)[
In this paper, we introduce a Russian dataset and an approach to aspectbased sentiment analysis of Russian users’ comments about the COVID-19. The dataset is large enough (about 10 thousand messages) to train modern machine learning methods in order to classify the flow of user opinions on the above and similar issues. A similar dataset could not be found in the current world publications. For the Russian language, there is no other manually annotated dataset of user messages related to the issues of coronavirus infection.
We solve two tasks, namely Relevance Determination (RD), which aims to
predict whether a sentence is relevant to an aspect of the pandemic, and
Sentiment Classification (SC), which classifies the sentiment expressed towards an
aspect in a sentence. We applied and tested various methods of machine
learning, including fine-tuning of the pre-trained Russian BERT, RuBERT [
During last year a lot of work were devoted to users’ posts concerning
COVID19. In [
Among used methods for sentiment analysis of COVID-related texts, general
sentiment analysis prevails based on existing general sentiment classifiers [
There are very few new specialized datasets that are manually annotated
with respect to coronavirus or related aspects. The authors of [
For the Russian language, about 3903 tweets were extracted in the work [
ABSA determines the sentiment expressed to some aspect of an entity in a text.
Typically, one aspect can be represented by several terms or can be not expressed
in a text at all. Early approaches to the ABSA task utilize extensive
featureengineering. So, in [
Neural networks allowed researchers to avoid manual feature-engineering.
First models were based on the LSTM architecture and attention mechanism.
For instance, TD-LSTM [
Introduction of transformers [
The latest innovation in NLP tasks is the utilization of pre-trained generative
language models, such as ELMo [
One of the most important problems that face researchers is the lack of
labeled data. There are diferent approaches to that problem. For instance,
BERTADA [
For the dataset, users’ comments on Covid-2019 related news articles were collected from the VKontakte social network. We selected masks, quarantine (lockdown) or vaccines as aspects for sentiment annotation and extracted relevant comments using corresponding keywords. Also sentiment attitudes towards government measures were annotated for all selected comments. This government aspect is especially dificult for automatic analysis because mentioning of government can be implicit as in the following sentence: – In Germany, a permanent mask, etc. regime, shots from Russia are very surprising, when nothing is observed at all.
The total number of sentences is 10968.
Each sentence was labeled by several experts (three on average). An annotator should indicate sentiment it expresses towards each of the above-mentioned four aspects (or indicate that the sentence is not relevant to the aspect). The annotators’ group included professional linguists and psychologists. We consider six types of sentiment labels, namely: – irrelevant; – positive; – negative; – neutral. This label is used for factual sentences without any visible sentiments; – both positive and negative. For such a label, evident positive and negative attitudes should be seen in a message; – relevant, but impossible to determine. In this case, the presence of a sentiment attitude is seen, but the context of sentence does not give possibility to determine it.
A sentence is considered to be relevant to an aspect, if at least two annotators considered it relevant. Sentences collected using keywords also can be irrelevant, for example a sentence mentioning Elon Musk ( ”Mask” in Russian spelling) is not relevant to the mask aspect. Multiple annotations for a relevant sentence are translated to three sentiment classes: positive, negative, and other (comprising neutral, contradictory, and unclear cases) using the following rules: – a sentence has the positive score, if the number of positive annotations is more than the number of all other annotations for this sentence; – a sentence has the negative score, if the number of negative annotations is more than the number of all other annotations for this sentence; – otherwise the sentence is assigned to another category.
For example, the following sentence “the mask allows you not to maintain health, but to save your family budget” had three diferent labels from annotators: positive, negative, and impossible to determine. This sentence in fact need more context to precisely determine its sentiment towards masks, the attitude depends on interpretation. According to the above described rules, its resulting sentiment category is other.
Table 1 provides sizes of resulting categories for each aspect. It can be seen that the attitudes to masks and quarantine are mainly positive, the attitudes to government actions are mainly negative.
Relevant 5097 2604 3515 1585
Negative 861 (17%) 601 (23%) 244 (7%) 1027 (65%)
Positive 1011 (20%) 538 (21%) 868 (25%) 54 (3%)
Other 3225 (63%) 1465 (56%) 2403 (68%) 504 (32%) The most significant disagreement between annotators concerns assigning positive and negative scores to the same user’s post. We found the following main cases for positive-negative disagreement between annotators.
First case. An author of a comment describes an opinion of another person, disagreement of the author with this opinion can be seen. In such cases some annotators can assess the sentiment of the author; other annotators can give label ”positive and negative” (because two opinions are seen) or ”impossible to determine”; the third annotator can select the described position because it takes most part of the sentence. For example (all examples are translated from Russian): – My father is so ... He endlessly repeats that masks, like vaccination, are a way of enslavement and he has an eternal ”they are watching us” in his mind, I endlessly tease him, they say, be careful. – But, just they think, since they are already sick, they no longer need a mask, there is nothing to defend against and they sneeze at everyone.
Second case. The author tries to ofend another participant of the dialogue using the aspect words: – well, nothing, nothing, someday for people like you they will definitely come up with vaccinations - from stupidity. Here one annotator consider this comment as irrelevant to vaccines, other two annotators provide contradictory opinions (positive-negative)
Third case A comment describes some violations of mask or quarantine regimes. Some annotators consider such sentences as factual, neutral, other annotators try to infer some positive or negative positions. for example: – Because few tourists comply with the quarantine measures.
Also typos may occur, which are dificult to explain. Because of all abovementioned problems, we try to have at least three annotations for each comment.
In the scope of this work, we use the RuBERT-conversational language model as a powerful feature extractor for classification. RuBERT-conversational is the BERT language model pre-trained on a large number of Russian tweets by the DeepPavlov project2. It greatly fits our needs because it was tuned on spoken and informal language data.
As the original BERT model, the input sequence of this model is either one or two sentences framed with special tokens:
[CLS], A1, . . . , Am, [SEP ], B1, . . . , Bn[SEP ] where A1, . . . , Am are tokens of the first sentence, B1, . . . , Bn are tokens of the second sentence, [SEP ] is a special separating token, and [CLS] is a special token, which represents the whole input sequence for classification tasks.
BERT returns hidden representations of every token of the input as the output. Furthermore, the representation of the [CLS] token is processed by a fully connected layer, which was pre-trained for the Next Sentence Prediction objective, and the tanh activation function.
For the relevance determination and sentiment classification tasks, we added two fully-connected layers, containing 256 and K (a number of classes) outputs respectively, on top of the final representation of [ CLS]. These layers are preceded by dropout layers with the rate of 0.5 and followed by the ReLU activation function:
H1d = dropout(0.5)(H[CLS]);
H2 = ReLU (W1H1d + b1);
H2d = dropout(0.5)(H2);
output = W2H2d + b2;
where H[CLS] ∈ [
We formulate and solve the original classification tasks in diferent ways. First of all, we trained separate classifiers for each aspect. In that case, a document is an input to a classifier, and the output is either its relevance (0 or 1) or its sentiment (positive, negative, and other) to a considered aspect. In the second case, a document must be relevant to an aspect.
Secondly, the relevance determination problem was also postulated as a Natural Language Inference (NLI) problem. In that case, a classifier operates with all the given aspects and is able to learn relevance relations for new aspects if new data comes. The input of the classifier is a pair (s, h ) of a sentence and an afirmative hypothesis about its relevance to an aspect, and the output is 2 https://huggingface.co/DeepPavlov/rubert-base-cased-conversational whether h is true (0 or 1). For example, h can state “Is relevant to masks” or “Is relevant to vaccines”.
Thirdly, we formulate the sentiment classification problem as an NLI problem as well. In that case, for each triple (s, h) of a sentence and an afirmative hypothesis about its sentiment towards a relevant aspect, the classifier is trained to predict whether h holds truth (0 or 1). In that case, h may be “Is positive to masks” or “Is negative to quarantine”.
Finally, the sentiment classification problem was stated as a Question Answering (QA) problem. In this formulation, we train a classifier to predict the sentiment polarity given a pair (s, a) of an expression and an aspect. In that case, a is simply an aspect, such as “Masks” or “Quarantine”, and the output is a sentiment category. We decided not to use QA formulation for the relevance determination task because in that case it is equivalent to the NLI formulation.
Task
Epochs LR Sentiment Classification (NLI) Sentiment Classification (QA) Relevance Determination (NLI) RD and SC (aspect-specific) During the experiments, we compare several variants of RuBERT-based models with classical machine learning methods, namely, Support Vector Machine (SVM), Multinomial Nıav¨ e Bayes (MNB), Bernoulli Nıav¨ e Bayes (BNB), Gradient Boosting (GB), and Random Forest (RF). Implementations of the classical algorithms were taken from the scikit-learn library3. These models receive tf-idf vectors as the inputs. To obtain the vectors, we tokenized and lemmatized texts, dropped stopwords, punctuation marks, and words that are seen less than in ifve documents. We tuned their hyperparameters with a Bayesian Optimization algorithm realized in the tune-sklearn library.
We utilized an implementation of the RuBERT-conversational model from the Transformers library. Other steps were performed with the PyTorch library 4. The models were trained with the standard back-propagation algorithm. The size of a batch was set to 64. We utilized cross-entropy loss as a loss function, AdamW as an optimizer. OneCyclicLR with maximum learning rate of 3e-5 was utilised for learning aspect-specific SC and RD tasks (in the standard formulation). Other 3 https://scikit-learn.ru/ 4 https://pytorch.org/
Model Accuracy Precision Recall F1 SVM MNB BNB RF
GB Vaccines
SVM
MNB Quarantine BNB
RF
GB Masks
SVM
MNB Government BNB
RF
GB
SVM MNB BNB RF GB SVM MNB BNB RF GB 98.47 96.93 95.17 98.98 98.98 98.82 96.64 96.76 97.61 98.36 parameters were specific to the tasks and described in Table 2. In addition, we kept track of a current best (according to F1-score) model after each epoch.
All the models were tested with a random stratified train-test split, with a test size of 0.3. More precisely, the original texts were split into those collections, whereas the task-specific datasets were formed based on the same stratified traintest split.
Table 3 shows the performance of classical machine learning methods in the relevance determination task. The low results of classification for the “government actions” aspect can be explained with the diversity of lexical expressions of this aspect in comments. Some sentences do not contain direct mentions of this aspect, but nevertheless, express some opinion. Table 4 provides macro-averaged scores of classical machine-learning methods for the sentiment classification task.
Model Accuracy macroPrec macroRecall macroF1 Vaccines
SVM MNB BNB RF GB SVM
MNB Quarantine BNB
RF
GB Masks
SVM
MNB Government BNB
RF
GB Average
SVM MNB BNB RF GB SVM MNB BNB RF GB
Table 5 and Table 6 compare results of the best (according to F1-score) classical methods to RuBERT-based models in relevance determination and sentiment classification tasks correspondingly.
Aspect
Model
Accuracy Precision Recall F1 Vaccines
As we can see from the tables, both classical methods and neural networks determine the relevance of messages with the high quality when the messages include direct mentions of an aspect. In more complex scenarios, neural networks show better results. In the sentiment classification task, neural networks also achieve higher scores, because they consider context and the order of words.
Finally, the use of the NLI and the QA formulations increased the scores of the sentiment classification task, whereas the QA formulation performs slightly better. It may be explained by the introduction of additional aspect-related features to the input of the models and by the usage of the whole collection of sentences for training. The lowest results of macro measures are obtained for the government aspect, which can be explained with small number of examples in the positive class.
As for the RD task, the introduction of the new formulations did not increase overall quality. This behavior may be caused by the fact that the task is too ’simple’ for the model to improve further.
Accuracy MacroPrec MacroRecall MacroF1 Vaccines
NLI QA RuBERT SVM NLI
QA Quarantine RuBERT
SVM Masks
NLI
QA Government RuBERT
SVM NLI QA RuBERT MNB NLI QA RuBERT
SVM Average
In this paper, we introduce a specialized Russian dataset of Russian users’ comments about COVID-19 aspects. The dataset contains sentences with sentiment scores towards four topics widely discussed such as masks, vaccines, quarantine, and government measures. Each comment is scored by three annotators on average.
We studied approaches to aspect-based sentiment analysis of the created dataset. We solved two tasks, namely Relevance Determination (RD), which aims to predict whether a sentence is relevant to an aspect of the pandemic, and Sentiment Classification (SC), which classifies the sentiment expressed towards an aspect in a sentence.
We applied and tested various methods of machine learning, including finetuning of the pre-trained RuBERT model. The best results were obtained by RuBERT model in special settings called Natural Language Inference (NLI) and Question Answering (QA), in which an additional sentence is added to a classified sentence, indicating a target aspect.
The created collection is publicly available5.
Acknowledgements. The reported study was funded by RFBR according to the research project № 20-04-60296. 5 https://github.com/LAIR-RCC/RussianCovidDataset