Sentiment Analysis for A Lexicon-Based Approach Russian Academic Texts:

Abstract

In this paper, we explore to what extent sentiment markers can differentiate the polarity of Russian political texts and academic texts for different ages and grade levels. The Corpus compiled for the study contains UN official records and textbooks of different subjects (Social Studies, History, Biology, Ecology, Technology and Science) and grades (1-11). We provide a brief overview of previous research on sentiment analysis of Russian texts and conduct threestage lexicon-based sentiment analysis and evaluate sentiment bias of 28 Russian texts. Based on the data registered in RuSentiLex, we propose an innovative quantitative method of assessing sentiment in academic and political domains. As the results obtained compare favorably with the previously published results on the established sentiment characteristics for English and German texts, the study encourages enlargement of the Corpus with the aim to compute sentiment analysis of texts of other genres and time periods. The research findings provide a broad context for understanding the sentiment bias of texts which may be useful for text writers and test developers.

Keywords

• 1 Russian
• political texts
• academic texts
• lexicon-based sentiment analysis
• RuSentiLex

1. Introduction

Sentiment analysis, also referred to as emotion AI (artificial intelligence) and opinion mining, is a computational text analysis for opinions, emotions, assessments, attitudes. For almost 20 years, it has been one of the most actively developing branches of computational linguistics and a popular research area [ 1 ]. Sentiment analysis proves to be a valuable technique in almost all spheres of human activity as assessments and opinions play an important role in evaluation and management of society and its social values. The main areas of application of sentiment analysis are customers’ reviews of goods and services [ 2 ], public opinion in social networks [ 3 ], news [ 4 ] etc. Sentiment analysis is also important in marketing, finance, political science, communication and health services and science [ 5 ].

However, until recently, it has been sporadically implemented in education and publications in the area are few. Archana R.P. and K. Bagloti pursued a comparative analysis of the role of sentiment analysis in students’ perspectives as well as instructional effectiveness and concluded that incorporated in education sentiment analysis isinvaluable in assessing teaching methodologies and course curricula [ 6 ]. H. Hamdanetet al. [ 7 ] conducted a research aimed at Opinion Target Extraction in book reviews and concluded that sentiment analysis has a strong potential to improve teaching materials (see also [ 8 ]).

As for textbook sentiment assessment, studies in the area are quite rare, though one which is noteworthy is the study conducted by J. Sell and I. G. Farreras [ 9 ] who elaborated a new approach to sentiment vocabulary on the corpus of 66 Introductions to Psychology college textbooks published over the last century. The research demonstrated “a less emotional manner” and “a more guarded tone” of modern textbook authors. These findings are especially meaningful as they allow to contrast sentiment in reviews and academic writing which differ in genre, length and function. Reviews are typically rather brief texts generated predominantly with the purpose to assess a referent. Even if a review consists of a number of paragraphs there is always one aimed at evaluating goods or services. Academic texts are not only much longer, they are typically informative or instructional and as such they require a different approach.

In the educational context, sentiment analysis is widely implemented to process students' feedback and is aimed at monitoring effectiveness of instructions and thus contributing to enhancement of learning effectiveness. Sentiment analysis “for big educational data streams”, including teaching materials, is challenging [ 10 ] and is still viewed as a new area of research where studies are rare and validated methodsare few.

In the paper we aim at the following research questions (RQ): RQ.1: What is the polarity of Russian academic texts used in elementary, middle and high school? RQ.2: What is the polarity of UN Russian texts elicited from the United Nations Parallel Corpus? RQ.3: How different is the polarity of Russian academic and UN Russian texts?

The two hypotheses tested in the research are that (1) academic texts used in high school tend to have a negativity bias and (2) the negativity bias of high school textbooks is similar to that of political texts.

2. Related Work

A review of early research on sentiment analysis is offered in [ 11 ], and a comprehensive latest review in performed by S. Tedmori and A. Awajan in [ 12 ]. The method of sentiment analysis has been developed within a number of approaches. One of the latest approaches is neural networks designed with the advent of deep learning and theera of artificial intelligence [ 5, 13 ].

Another approach utilized in a number of research is dictionary-based, the principles and strategies of which are presented in [ 11, 14 ]. Lexicon-based approach requires sentiment lexicon, i.e., explanatory dictionaries of words provided with connotative (positive, negative etc.) tags. In studies of Russian discourse researchers utilizea manually created dictionary RuSentilex [ 15 ]. An example of dictionarybased approach implementation is described in [ 16 ] where Q. Guang et al. study contextual advertising.

Educational texts imply many more difficulties for sentiment classification than services or product reviews as their authors use more elaborated language of sentiment including various stylistic devices. Educational domain was studied by Z. Kechaou et al. [ 17 ] who utilized sentiment analysis to examine the emotional nature of e-learning blogs [ 10 ]. U. O. Osmanoglu applied a machine learning approach to assess sentiments in distance education course materials [ 18 ].

To the best of our knowledge the only research on application of sentiment analysis of Russian educational texts is performed in [ 14 ] where the authors used subcorpus of Russian Academic Corpus compiled of Social Studies textbooks. The findings confirm the hypothesis of predominantly negative discourse in the textbooks studied and the conclusion received is revealing since the language comprises more positive than negative words [cf. 8 and 18 for Pollyanna effect]. In this regard, another essential contribution is a diachronic study of Iliev R. et al. [ 19 ], who provided clear evidence that the frequency of affective, both positive and negative words in modern discourse has decreased over two centuries.

3. Methods and data

The study is aimed at comparative analysis of sentiments in texts of different types, i.e., political texts and educational texts of different subjects and for various age groups.

For this purpose, we compiled four homogeneous Russian subcorpora, three sets of school textbooks and official records from the United Nations Parallel Corpus: (1) 8 Elementary school textbooks, Grades 1 – 4 for schoolchildren aged 7 – 11; (2) 11 Biology textbooks, Grades 5 – 9 for schoolchildren aged 12 – 16; (3) 9 History textbooks, Grades 10 – 11 for schoolchildren aged 17 – 18; (4) 10 UN Russian texts. The UN Russian texts are elicited from the United Nations Parallel Corpus “composed of official records and other parliamentary documents of the United Nations that are in the public domain” (https://conferences.unite.un.org/uncorpus, [ 20 ]). The grade number of school textbooks labels textbook complexity (readability) and is used as an index to benchmarka sentiment bias of the book. The sizes of all four subcorpora are presented in Table 1 below. To ensure reproducibility of results, we uploaded the Corpus used in the study on the website thus providing its availability online (see Corpus of Russian Academic Texts (CORAT) at https://clck.ru/U7sCt).

In comparison with the previous study where we analyzed textbooks on social sciences only [ 21 ], we significantly expanded the range of text types analyzed.

At present the Corpus of Academic Texts (CORAT, Corpus of Russian Academic Texts [ 22 ]) comprises 11 biology textbooks, 9 history textbooks, and 8 elementary school textbooks (n=28). For contrastive purposes we also computed 10 texts in the Russian language from the official United Nations Parallel Corpus(https://conferences.unite.un.org/uncorpus) to indentify differences the polarity of these texts and the text of school textbooks.

In this study we implemented a lexicon-based approach and estimated textbook sentiments computing frequency of words with positive and negative sentiment orientation. The sentiment with a positive or negative value is traditionally referred to aspolarity. For this purpose, we used RuSentiLex containing over 12,000 Russian words and phrases labeled as positive, negative, neutral or positive/negative (indefinite). Thecategory positive/negative is traditionally applied to those words the polarity of whichdepends on the context. RuSentiLex contains three types of sentiment-related words: (1) opinionated words from Russian sentiment vocabularies; (2) non-opinionated words with connotations conveying information about social phenomena; (3) slang and curse words from Twitter [ 23 ]. Sentiment of “non-opinionated words” is identified based on the context they are used in, i.e., social phenomena they nominate [ 24 ]. The phenomenon is viewed as positive if it is supported, secured, defended and guarded. If it is negative, the phenomenon is disputed, struggled, conflicted with or fought against, etc. All in all, RuSentiLex contains 35 negative and 20 positive vocabulary patterns enabling researchers to elicit connotations of words under study.

Negative patterns include e.g., Rus. borotysya s (struggle against), Rus. obvinit’ v (charge in), etc. Positive patterns can be exemplified with Rus. borotysya za (struggle for), Rus. zashchishchat’ (protect). The type (positive or negative) of non-opinionated words is allocated based on the frequency of its 2 All the books are provided with meta-description containing the number of the grade and the author. E.g. code 01-4ch stands for “The World Around Us”, Grades 1-4, Reference materials, Chudinova E.V., DemidovaM.Yu., 2011, see Corpus of Russian Academic Texts (CORAT).zip.at https://clck.ru/U7sCt collocations: to be computed as negative a word is to be registered in negative patterns 10 times more often than in patterns of positive type. Otherwise, it is viewed as neutral [ 1 ].

The multi-domain origin of the lexicon provides solid foundation for better performance of RuSentiLex in any domain. RuSentiLex is the only Russian sentiment lexicon and as such it is widely used in modern research of Russian discourse [ 25 ]. The Lexicon statistics is presented in Table 2 below.

Neutral and positive / negative words registered in the lexicon were excluded from the study as they amount to less than 0.05% in our Corpus.

4. Evaluation of Sentiment Bias

Text processing was carried out in three stages. First, with the help of the morphological analyzer UDPipe 2 [ 26 ] we performed lemmatization, i.e., ’reduced’ the inflected forms of a word to their initial form grouping them together, so they can be analyzed as a single item. The lemmatization accuracy of UDPipe 2 is considered high with F1 estimated at 96.68%.

On the second stage, we annotated texts under study with the help of RuSentiLex labeling the words as positive or negative. Finally, the total number of positive and negative words in the text was computed as a percentage of the total number ofwords.

As all senses of polysemous words demonstrate the same sentiment [ 1 ] we did notface the problem of semantic disambiguation. Another problem which we avoided in the current study is performing complete syntactic analysis which is viewed as compulsory as a researcher has to detect negation reversing the polarity of words, phrases,and sentences. The research showed less than 1% cases of the kind and as such theydo not affect the experimental data presented.

5. Results and Discussion

The previous research showed that sentiment vocabulary in children’s books is associated with developing higher levels of empathy and even better perspective-defining skills [ 27 ]. Thus, sentiment analysis can be an important feature used not only to classify textbooks vocabulary but also to assess their quality and appropriateness for children.

The complete research results presented in Tables 5, 6 confirm the hypothesis that the majority of the Russian textbooks contain some kind of an emotional bias as all the texts analyzed contain words bearing either negative or positive sentiment.

The bar charts in Figure 1 below show the distribution of positive and negative sentiment in the books under study.

The diagram indicates that positive and negative words frequency is unevenly distributed in elementary school textbooks, History textbooks and UN texts. In the first two cases, these differences are statistically significant (see Table 4 below). 2,5

2 1,5 0,5 1 0 2,8919

In elementary school textbooks, the number of positive words (1.5027) is almost 2 times higher than the number of negative words (0.6903). In History textbooks, on thecontrary, the number of negative words (2.8919) is higher than positive words (2.1598). The unevenness defined is possibly caused by the very nature of the texts: History textbooks narrate of wars, struggle for power, revolutions, etc. which are typically negatively connotated. Elementary school textbooks, on the other hand, are oriented for the target audience, school students aged 7 – 11, who are expected to comprehend mostly positive information. The latter is caused by two factors. Firstly, reading texts in elementary schools are considered not only educational but pedagogical, i.e., disciplinary, and as such are aimed at forming positive personality and a positive picture of the world. Secondly, the texts are supposed to reinforce a positive attitude towards learning. As for textbooks in secondary and high schools, they are expected to develop critical thinking thus exposing students to both positive and negative timelines (https://www.jstor.org/stable/40014056?seq=1). As “products of the author’s professional and personal preferences” (https://www.euroclio.eu/ resource/the-textbook-is-man-made/) modern textbooks reflect “the prevalence of a social representation of history as a process of collective violence” [ 28 ] and “two thirds of nominated historical events were negative” [ 29 ]. Thus, of two possible timelines, i.e., positive and negative, in the majority of cases textbooks authors prefer the latter. Our findings here also coincide with the findings of V. Bagdasaryan et al. [ 30 ] whose research reports on numerous negative images and characteristics in secondary and high school textbooks. 3 p < 0.05 — statistically significant differences

The frequency of positive and negative words in Biology textbooks is almost the same, which indicates an emotionally balanced presentation of information.

As can be seen from the diagram, on average, the frequency of sentiment words in History textbooks is two times higher than that in Biology textbooks (Fig. 1). The research indicates that History textbooks are most emotionally charged when contrasted with Biology and elementary school texts. The differences are statistically significant for both negative and positive sentiment words (Table 5).

The data in Table 5 indicate that there are significant differences in the frequencyof sentiment words in texts. UNPC Russian and History texts demonstrate similaritiesin the frequency of sentiment words, apparently due to the nature of the texts referents. UN PC Russian texts and history textbooks do not only narrate social events, present social phenomena and describe social objects, they provide explicit emotional assessment of the notions and the facts presented. The frequency of positive words in Biology textbooks are similar to those in elementary school textbooks, while the frequencies of negative words are statistically significantly different. As it was already mentioned above, the number of negative words in elementary school textbooksis much lower than in any other type of texts studied (Fig. 1).

History Textbooks (n=9) & Elementaryschool textbooks (n=8) History Textbooks (n=9) & Biology Textbooks (n=11) History Textbooks (n=9) & UN Texts(n=10) Elementary school textbooks (n=8) & Biology Textbooks (n=11) Elementary school textbooks (n=8) & UN Texts (n=10)

Biology Textbooks (n=11) & UN Texts (n=10)

We also implemented a correlation analysis (Spearman Rank Order Correlations) to analyze the relationship between the frequency of positive and negative words inall academic texts studied (n = 28). We excluded UN texts from this analysis as functionally different types of texts. The data obtained indicate (Fig. 2.) a strong statistically significant correlation between the frequency of negative and positive words in all 28 academic texts (0.70 at p <0.05). We observed a rise in frequency of negative words accompanied with a rise of positive words.

6. Conclusion

In this study, we conducted a contrastive sentiment analysis of educational texts for schoolchildren and UN texts. The Corpus of academic texts comprises three sets of 28 textbooks: elementary school textbooks, secondary Biology textbooks, and high school History textbooks. This choice makes it possible to compare texts of social andnatural sciences, as well as texts for younger and older students. This significantly expands the results of [ 21 ], in which we analyzed texts on Social Sciences only.

The shift towards negative vocabulary revealed in [ 21 ] comprises all Social Science textbooks for grades 5 – 11, and as such proved to be significant in textbooks of all age groups, from the 5th through the 11th grade. In this study, we confirmed the earlier findings in the subcorpus of History textbooks for grades 9– 11. A similar shift towards negative vocabulary was observed in UN texts. At the same time, in Biology textbooks (for grades 5 – 7), the number of positive and negative words is approximately the same. The comparative analysis proved the results to be statistically significant.

The sentiment difference in presenting educational material in Russian textbooks on social and natural sciences was revealed for the first time. We also confirmed the hypothesis that positive vocabulary prevails in Russian textbooks for elementary school children: it is true for the three subjects books analyzed, i.e. Ecology, Technology and Science. These results are similar to those received in a recent study by [ 31 ] who showed that English and German fiction discourse for children and adolescents demonstrates a distinct positive bias.

We believe that research on the use of positive and negative vocabulary can have a significant impact on textbook writers and testing material developers. Textbook authors are recommended to pay more attention to the so called positivity superiority effect [ 27 ], as positive words are comprehended faster than neutral and negative words.

7. Acknowledgment

8. References

This paper has been supported by the Strategic Academic Leadership Program “Priority 2030” of Kazan Federal University.

References

1. B. Liu , Sentiment analysis: Mining opinions, sentiments, and emotions , The Cambridge University Press, 2015 .

2. V. Solovyev , V. Ivanov , Dictionary-based problem phrase extraction from user reviews , International Conference on Text, Speech, and Dialogue , 2014 , pp. 225 - 232 .

3. P. Burnap , O. Rana , M. Williams , W. Housley , A. Edwards , J. Morgan , L. Sloan , J. Conejero, COSMOS: towards an integrated and scalable service for analysing social media on demand , International Journal of Parallel Emergent and Distributed Systems , 30 ( 2 ), 2015 , pp. 80 - 100 .

4. A. Moreo , M. Romero , J. L. Castro , J. M. Zurita , Lexicon-based comments-oriented news sentiment analyzer system , Expert Systems with Applications , 39 , 2012 , pp. 9166 - 9180 .

5. Z. Lei , Sh. Wang , B. Liu , Deep learning for sentiment analysis: A survey , WIREs Data Mining and Knowledge Discovery , 8 , e1253 , 2018 , https://doi.org/10.1002/widm.1253, last accessed 2021 /04/05.

6. R. P. N. Archana , K. Baglodi , Role of sentiment analysis in education sector in the era of big data: a survey , International Journal of Latest Trends in Engineering and Technology , Special Issue, 2017 , pp. 022 - 024 .

7. H. Hamdan , P. Bellot , F. Bechet , Sentiment analysis in scholarly book reviews , arXiv preprint arXiv:1603.01595 , 2016 .

8. N. Altrabsheh , M. M. Gaber , M. Cocea , SA -E: Sentiment Analysis for Education , Frontiers in Artificial Intelligence and Applications , Volume 255 , 2013 , pp. 353 - 362 .

9. J. Sell , I. Farreras , LIWC-ing at a Century of Introductory College Textbooks: Have the Sentiments Changed? , Procedia Computer Science , 118 , 2017 , pp. 108 - 112 .

10. Zh . Han, J . Wu, Ch. Huang, Q. , Huang , M. Zhao , A review on sentiment discovery and analysis of educational big‐data, WIREs data mining and Knowledge discovery , 10 ( 1 ), е1328 , 2019 .

11. M. Hu , B. Liu , Mining and summarizing customer reviews , Proceedings of ACM SIGKDD international conference on Knowledge Discovery and Data Mining , 2004 , pp. 168 - 177 .

12. S. Tedmori , A. Awajan , Sentiment Analysis Main Tasks and Applications: A Survey , Journal of the Association for Information Systems , 15 ( 3 ), 2019 , pp. 500 - 519 , https://doi.org/10.3745/ JIPS.04.0120.

13. Y. Goldberg , A Primer on neural network models for natural language processing , Journal of Artificial Intelligence Research , 57 , 2016 , pp. 345 - 420 .

14. S. Kim , E. Hovy, Determining the sentiment of opinions . Proceedings of international conference on Computational Linguistics , 2004 , pp. 1367 - 1373 ,

15. N. Loukachevitch , A. Levchik , Creating a General Russian Sentiment Lexicon , Proceedings of Language Resources and Evaluation Conference LREC-2016 , 2016 .

16. Q. Guang , H. Xiaofei , Zh. Feng, Sh. Yuan , B. Jiajun , Ch. Chun, DASA: dissatisfaction-oriented advertising based on sentiment analysis , Expert Systems with Applications , 37 , 2010 , pp. 6182 - 6191 .

17. Z. Kechaou , M. B. Ammar , A. M. Alimi , Improving e-learning with sentiment analysis of users' opinions, 2011 IEEE global engineering education conference (EDUCON) Apr 4 , 2011 , pp. 1032 - 1038 .

18. U. O. Osmanoglu , O. N. Atak , K. Caglar , H. Kayhan , T. C. Can , Sentiment Analysis for Distance Education Course Materials: A Machine Learning Approach . Journal of Educational Technology & Online Learning , 3 ( 1 ), 2020 , pp. 31 - 48 .

19. R. Iliev , J. Hoover , M. Dehghani , R. Axelrod , Linguistic positivity in historical texts reflects dynamic environmental and psychological factors , Proc Natl Acad Sci USA , 113 ( 49 ), E7871 - E7879 , 2016 .

20. M. Ziemski , M. , Junczys-Dowmunt , B. Pouliquen , The United Nations Parallel Cor pus, Language Resources and Evaluation (LREC'16) , Portorož, Slovenia, 2016 .

21. V. Solovyev , M. Solnyshkina , E. Gafiyatova , D. McNamara , V. Ivanov , Sentiment in academic texts, Conference of Open Innovation Association, FRUCT, 2019 , pp. 408 - 414 .

22. V. Solovyev , M. Solnyshkina , V. Ivanov , I. Batyrshin , Prediction of reading difficulty in Russian academic texts, Journal of Intelligent & Fuzzy Systems , 36 ( 5 ), 2019 , pp. 4553 - 4563 .

23. N. Loukachevitch , A. Levchik , Creating a General Russian Sentiment Lexicon, Open Semantic Technologies for Intelligent System , 6 , 2016 , pp. 377 - 382 .

24. S. Feng , J. S. Kang , P. Kuznetsova , and Y Choi , Connotation lexicon: a dash of sentiment beneath the surface meaning , Proceedings of ACL2013 , 2013 , pp. 1774 - 1784 .

25. Rogers , А., Romanov , А., Rumshisky , S. Volkova , M. Gronas , A . Gribov, RuSentiment: An Enriched Sentiment Analysis Dataset for Social Media in Russian , Proceedings of the 27th International Conference on Computational Linguistics , 2018 , pp. 755 - 763 .

26. UDPipe Versions, https://ufal.mff.cuni.cz/udpipe, 2021 , last accessed 2021 /04/05.

27. J. Lüdtke , A. M. Jacobs , The emotion potential of simple sentences: additive or interactive effects of nouns and adjectives?, Front Psychol ., 6 , 1137 , 2015 .

28. J. H. Liu , D. Páez , P. Slawuta , R. Cabecinhas , E. Techio , D. Kokdemir , et al., Representing world history in the 21st Century: The impact of 9-11, the Iraq War, and the Nation-State on dynamics of collective remembering , Journal of Cross-Cultural Psychology , 40 ( 4 ), 2009 , pp. 667 - 692 .

29. S. Moscovici , L' age des foules . Un traité historique de psychologie des masses, Brussels: Éditions Complexe, 1983 .

30. V. Bagdasaryan , et al.: School's textbook of history and public policy , Moscow: Nauchnyi ekspert, 2009 . In Russian, https://rusrand.ru/files/13/07/23/130723083031_BLOK.pdf

31. A. M. Jacobs , B. Herrmann , G. Lauer, J. Lüdtke , S. Schroeder , Sentiment Analysis of Children and Youth Literature: Is There a Pollyanna Effect? , Frontiers in psychology, 11 , 2020 , pp. 574 - 746 , https://doi.org/10.3389/fpsyg. 2020 . 574746 .