This study presents the approach to aspect-based sentiment analysis where a named entity of a certain category is considered as an aspect. Such task formulation is a novelty and opens up the opportunity to determine writers' attitudes to organizations and people considered in texts. This task required a dataset of Russian-language sentences where sentiment with respect to certain named entities would be labeled, which we collected using a crowdsourcing platform. Sentiment determination is based on a deep neural network with attention mechanism and ELMo language model for word vector representation. The proposed model is validated on available data on a similar task. The resulting performance (by the f1micro metric) on the collected dataset is 0.72, which is the new state of the art for the Russian language.
A relevant part of social monitoring is determining the sentiment of a text so as to identify its attitude to significant social events (aspects). Frequently, even one sentence contains several sentiment evaluations concerning various aspects of the text. For example, in the sentence: “Alex is an excellent worker, but the company Foo LLC, in which he works, poorly manages its staf”, there are two named entities mentioned in diferent sentiment. The entity “Alex”, of the category “Person”, is used in positive sentiment, and the other entity “Foo LLC”, of the category “Organization”, in a negative one. This research proposes an approach to aspect-based sentiment determination in text with named entity pre-assigned as an aspect. Such task formulation is novel and opens up the opportunity to determine authors’ attitudes to organizations and people considered in texts, which could be useful for social and political analysis.
There are several datasets in diferent languages available for aspect-based sentiment
analysis task, including SemEval 2015 competition dataset [
This research is based on a specially created dataset containing sentences with aspect-based
sentiment labels. Crowdsourcing was used to extend the number of annotators, their markup
was validated with a special procedure to provide annotation quality (see sec. 2). Analysis
of the related works of the last few years [
There are currently no datasets in Russian for setting up tools for solving problems of aspect
analysis when the aspect is a named entity, so one has been gathered and annotated. For
the formation of the corpus, we collected sentences in Russian from several sources: posts
of the LiveJournal social network [
Generally, sentences do not always contain sentiment estimations and named entities. Thus,
a preliminary selection of sentences for the subsequent annotation process is carried out. The
selection criterion is the presence in the sentence of a named entity and one word from the
sentiment words list. Named entities were extracted using a neural network model from the
DeepPavlov library3, which is a State-of-the-art solution for the Russian language with an
accuracy of 98.1% (f1-score metric * 100% ) obtained on the Collection3 dataset [
The annotators were asked to determine in what sentiment the author uses the named entity in the selected sentences (the classes of sentiment were “Positive”, “Neutral” and “Negative”). The sentence could not be marked with multiple tags. If the annotator was unable to unambiguously determine the sentiment class of the selected aspect, then that example was marked with the label “I find it dificult to determine” and, in the absence of other annotations, was not included in the resulting dataset. If a selected aspect was erroneously defined by a named entity, then such an example was marked as a “Wrong aspect” and was also not included in the ifnal dataset. The final label for the sentence was selected on the basis of the aggregation of annotators labels by majority voting.
The aspect-based sentiment dataset contains 5552 unique sentences (1992 from Twitter, 2050 from the news site “Lenta.ru”, 1500 from the blog platform “Livejournal”). The resulting number of sentences for every presented sentiment label, as well as the number of unique named entities, are presented in Table 1. ber of answers for the selected sentiment label to the number of all answers for all entities. The agreement value was 0.84. The most similar datasets to the one collected from the point of aspect-based sentiment annotation for the Russian language are the datasets from SentiRuEval 2015-2016 competitions. However, the 2015 dataset contains 822 reviews (17000 particular entities) on two pre-defined topics (restaurants and cars) labeled with four sentiment classes (positive, negative, neutral, mixed). The 2016 dataset is more representative (approx. 23600 labeled entities), it contains labeling for pre-defined aspect list, which are possible not to be presented in sentence text. Therefore, the collected dataset is a significant extension of data available for sentiment analysis of Russian-language texts.
The proposed method is based on deep neural network with attention (IAN) [
= [ 1, 2, ..., ],
= [ , ..., ],
where is the number of words in the sentence, and are the indices of the start and the
end of the named entity, respectively. At the first step, the sentence words are vectorized
using the bi-directional language model ELMo4 [
Then, the vectors corresponding to the words of the aspect [ , ..., ] and context
[ 1 , 2 , ..., , ..., , ..., ] are selected. The resulting word embeddings of the
aspect and context we feed into a recurrent neural network based on LSTM layers [
We evaluate the proposed model in terms of F1-macro and F1-micro scores (see section 4.1). (1) (2) 4http://docs.deeppavlov.ai/en/master/features/pretrained_vectors.html#elmo
To evaluate the performance of our models, we use the F1-measure metric as the evaluation score, as in the SentiRuEval 2015-2016 competitions. (3) (4) (5) = /(
+ ) = /(
+ ) 1 − = 2 ∗ ( ∗ )/( + ) where TP is the number of true positives, FP is the number of false positives, FN is the number of false negatives.
In this case, the score is calculated in two variations (F1-macro and F1-micro). For macroaveraging the F1-measure calculation is averaged for each class separately, but for microaveraging it is held for all examples together.
In the experiments to determine the sentiment for a specified named entity, the deep neural network method proposed in this paper is validated on the original SentiRuEval-2015 competition dataset, and then trained and tested on the dataset collected in the current work.
The SentiRuEval-2015 competition dataset was originally split by the competition organizers into train and test set containing 5974 and 6615 samples respectively. Model training was based on examples of three classes: positive, negative, and mixed.
The best performance was achieved using the following hyperparameters for our model:
Batch size – 4, Dropout – 0.3 (we add a dropout layer right before the recurrent LSTM layer),
the number of neurons in the LSTM layers is 150, the learning rate is 0.01, the l2-regularization
value is 0.001, the loss function is cross entropy [
Table 2 presents the model performance in terms of F1-micro and F1-macro scores. As a
result, the method proposed in this work shows better accuracy compared to other solutions
from the SentiRuEval-2015 competition [
The dataset collected in the current study was split into training and testing sets as 80% and 20% samples, respectively. The training set included 1081 named entities with negative sentiment, 1829 entities with a positive sentiment, and 2454 with neutral. The testing set included 460 positive entities, 267 negative entities, and 614 neutral entities. To evaluate the obtained model results, experiments were conducted with baseline methods that solve the usual problem of classifying sentences without paying any attention to the extracted named entity. These methods are built both on the rules using the vocabulary of emotive vocabulary, and with a classifier analyzing the entire sentence. To train such a classifier and select its hyperparameters, the AutoML method based on the TPOT library is used [16]. Thus, for comparison are used: 1. : Random definition of a label for each aspect; 2. : This classifier is based on the positive and negative sentiment word lists that were in Section 2 used for pre-selecting sentences. The sentence is given a sentiment label that belongs to the dictionary the largest number of words from which are present in the sentence. If the number of words included in dictionaries of diferent sentiments is the same, then the label of the most representative sentiment of the corpus is put, that is, “positive” in this case. If the sentence doesn’t contain any words from the sentiment lists, then the sentiment of the sentence is considered neutral. 3. ( ): This classifier is based on the TPOT software library. The average vectors of aspect words of the analyzed sentence obtained from the ELMo model are used as input features. The type of classifier and its parameters were selected automatically by the TPOT library.
The performance of the model on the proposed dataset, in comparison with the baseline methods, is presented in Table 3.
Besides, we have analyzed the ability of the model to classify aspects that are not present in the training set. The average performance across all sources has not changed. This confirms the efectiveness of the proposed approach for working with other named entities.
In this section, we present an example of a visualization of the results of aspect-based sentiment analysis. Experiments were conducted using Russian text corpus of the LiveJournal posts and the SCTM-ru dataset [17], compiled from articles from the Russian Wikinews website. For analysis, 47 news and 40 blog texts on the topic “cinema, oscar” was selected. These texts contain the following keywords: “film”, “role”, “cinema”, “oscar” etc.
Visualization of the results is carried out in the form of Sankey diagrams, which shows the frequency of named entities contained in diferent sentiment contexts (see Fig. 2). The figure 2 shows that authors of news articles have a more positive point of view in their publications, while LiveJournal posts authors often create negative context.
The main issues requiring further research are: 1. Reproducibility of results for other languages. This task is complicated by the fact that there are no labeled data sets where named entities are considered as aspects; 2. Verification and use of modern language models (for example BERT [18]), as well as other implementations of attention mechanisms; 3. Besides, it is planned to further develop the collected dataset, both from the side of increasing the number of examples and from the side of expanding sources and domain areas, which will make it possible to better assess the universality of the developed method; 4. Establishing identity between diferent spellings of the same entities to more accurately determine the integral assessment of their sentiment.
The paper presents a deep-neural-network-based method for aspect-based sentiment analysis, where aspect is expressed by the named entity (organization or person), for textual data in Russian.
To solve the problem, a dataset of annotated sentences for several sources (blogs, microblogs, and news) were collected. The collected dataset is available for researchers upon request with https://sagteam.ru/en website. The developed method for building a dataset based on crowdsourcing resources can be used to extend the dataset size and improve the performance of the proposed classifier. Also, the proposed method can be used in other domain areas to create labeled examples.
Evaluation of the model both on the open dataset from SentiRuEval-2015 competition and on the collected annotated corpus shows the eficiency of the developed solution. The resulting performance is a baseline for this type of task in Russian and allows one to provide aspectbased analysis with clear visualization of the results, an example of which is presented in the paper.
The reported study was funded by an internal grant of the NRC "Kurchatov Institute" (Order No. 1359) and has been carried out using computing resources of the federal collective usage center Complex for Simulation and Data Processing for Mega-science Facilities at NRC “Kurchatov russian tweets, in: Computational Linguistics and Intellectual Technologies: Proceedings of the International Conference Dialogue, 2016, pp. 50–59. [16] T. T. Le, W. Fu, J. H. Moore, Scaling tree-based automated machine learning to biomedical big data with a feature set selector, Bioinformatics 36 (2020) 250–256. [17] S. Karpovich, The russian language text corpus for testing algorithms of topic model,
Intellektual’nyye tekhnologii na transporte (2018). [18] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).