The paper aims to describe the issue considering the most appropriate way of retrieving neutral subset of Internet texts in Russian. The problem described appeared within the project on the design of the classifier of Russian Internet text according to their emotion for which, apart from the eight emotional following the Lövheim's cube model, the neutral training set is needed. Two experimental variants of neutral subsets are described and compared in the paper - one obtained as a result of emotional texts assessment by informants and another one drawn from Wikipedia. Due to the fact that any of the two subcorpora meets the criteria required for our research aim - personal story genre and zero emotion content independently of other classes the neutral subset has been artificially created. Two-classes (emotional / neutral) classifier has been trained on Wikipedia subset in order to be able to retrieve the neutral dataset from social network public groups. The training set included 14000 text fragments for each of two classes from the VK public group “Overheard” and Wikipedia respectively. As a result of the classification 500 from 70000 of posts from social website have been claimed neutral. The study faced a problem within Emotional Text Analysis - the lack of reliable data sources for neutral texts - which has been successfully resolved combining mixed techniques.
The paper is devoted to the problem of building the neutral corpus of texts for training set used in the frame of emotional text analysis.
As our project aims to create a multiclass computer classifier for the Internet texts in Russian, our training set for further machine learning procedure should obviously include a range of labeled emotional text subsets and a neutral one.
The concept of the project previews eight emotional classes according to the classification of basic
emotions proposed by H. Lövheim [
2020 Copyright for this paper by its authors.
It means the texts firstly appeared in paper format and then published on the Internet do not fit for our purposes.
In this way, we will mainly focus in the paper on the criteria that we have elaborated for neutral subset retrieving and on the technological solution suitable for them.
In this logic, we will start by a brief description of the project and the emotional classes used in it (Section 2). In Section 3 we will pay a particular attention to the specificities of two first neutral subsets we were using in our first attempts – a little subcorpus obtained after emotional texts assessment by informants and the subcorpus drawn from Wikipedia. We also discuss the criteria for neutral text subset we have formulated in result of the analysis of two experimental subsets. In the Section 4, we describe the principles and main steps for using deep-learning techniques to train two-class (emotional / neutral) classifier in order to use it further for retrieving texts from our target internet segments in Russian. We conclude the paper by a discussion of the drawbacks and advantages of used methodology. The conclusion itself resumes the key ideas of the paper and offers an opening for the further investigation.
The development of technologies prompting human-computer interaction has rather recently
revealed the necessity of tools permitting Textual Attitude Analysis. Focusing on the appraisal been
expressed in texts the model gave rise to a bunch of directions in NLP domain. Two of them ‒ Opinion
Mining and Emotion Detection ‒ could be considered as the most productive for promoting the ideas
of Affective Computing. This is a “multidisciplinary field that explores how technology can inform an
understanding of human affect, how interactions between humans and technologies can be impacted by
affect, how systems can be designed to utilize affect to enhance capabilities, and how sensing and
affective strategies can transform human and computer interaction” [
Even if the analysis of sentiment has already obtained considerable results in text classification, it
remains especially centered on commercial purpose – to judge about customers’ opinion they express
in texts. However, emotions, affects play crucial role in human communication: they serve to catch the
interlocutor’s attention, to solicit the empathy in relations [
Within the framework, the researches pursuit the goal of designing a computer classifier able to predict the emotion of text. There are two trends in detecting emotion: performing it from the author / writer perspective or from the reader perspective [13]. For our project we have chosen the former. While asking our assessors to label the text, we provided them with the following instruction: Which emotion does the author of the text feel?
Another challenging question that research teams in Emotional Analysis of Texts ask themselves is: how many classes of texts to determine? The problem is not trivial, because the number of emotions mentioned in works of psychologists oscillates between 4 and 40. The projects in Emotion Analysis (see the overview of models in [14]) use the well-known lists of basic emotions proposed by either C. Izard [15] or P. Ekman [16]. The former focuses on 11 emotions, the latter – on six. For our classifier, we took the biochemical model of emotions known as “Lövheim’s Cube”. Having combined his experience in neural sciences and Thomkins’ typology of emotions based on the assumption that emotional affects are biologically determined, H. Lövheim created the model which includes eight emotions under, mostly, doubles names (the first for the softer expression of it, the second – for the stronger). These are Interest / Excitement, Enjoyment / Joy, Surprise, Distress / Anguish, Anger / Rage, Fear / Terror, Contempt / Disgust, Shame / Humiliation.
In Emotional Analysis paradigm several methods are usually applied: rule-based, lexicon-based methodologies or machine learning algorithms. We decided to rely on supervised machine learning method which presupposes as its obvious step the labeling by assessors of training set of texts. To collect such set, we needed to decide about the source of Internet publications which we could submit for analysis.
In research practice different data corpora have been used: in the works of C. O. Alm et al. and E. Volkova et al. [17; 18] they propose to annotators a corpus of children stories, in C. Strapparava’s and R. Mihalcea’s work [19] the data is extracted from news headlines and in the research of A. Neviarouskaya et al. [20], for example, from the social networking website “Experience Project” where people publish personal stories to share the experience they survived. As our project’s aim is to build the classifier targeted on Internet texts, we considered that the personal stories about lived experience could meet our goals. Our attempt to explore the most popular in Russia social network VK showed that three public groups fit perfectly our needs. These are such groups as “Overhead” with 2.5 million of subscribers, “Room № 6” with 6 million of followers and its feminine twin “Caramel”. All three projects publish anonymous personal stories saturated by emotions which are expressed in a very frankly fashion. At the preselection stage, relying on thematic hashtags, we retrieved 15000 texts fragments from mentioned above publics.
Then 2000 Russian informants on the one of crowd sourcing platform assessed 3900 randomly taken texts from the collection. In this way, we obtained a rather representative labelled training set. In a given moment, the weighted average f1-score value is about 0.6.
To increase the accuracy of classification we need a ninth class – the neutral one. In the next section, we will explain main difficulties we have met searching for the “neutral” data.
The first idea we have got about the Internet segment which we could use as source for neutral texts was the Wikipedia. However, we also have a little subset of text fragments annotated as neutral during the procedure of emotional assessment.
In this section, we will characterize these two subsets and put them in comparison. 3.1.
Its major advantages in comparison, for example, with professional blogs (blog of specialists in translation) or thematic groups in social network (for example, treating the theme “architecture and design”) are the following: 1. Wiki texts do not seem to contain any attitude manifestation because with time the platform is coming more and more closer to the model of free and deliberative dialogue based on the ideals of criticism, rationality and pluralism [21]. 2. Wiki is the resource which, despite a pure informative function of its texts, corresponds to the majority of the characteristics of the Internet-communication: 1) anonymity; 2) accessibility; 3) collective share and dialogism; 4) multimodality; 5) high speed of informational uploading. 3. Wiki is technically accessible to the automatic downloading of textual content. Collected in this way our Wiki subset of neutral texts contains 136 184 tokens.
The design of the assessment procedure was elaborated in the way that the assessors had no constraint to attribute zero emotion to a text. After the assessment procedure we obtained, among others, a little subcorpus of texts where the annotators did not see any emotion. Although initially such texts had been submitted to the assessment as emotional ones, but the informants assessed them as neutral. The neutral subset of this kind contains 14000 tokens. Even if it seems to be minuscule, it can serve as an interesting point of comparison for the Wiki subset. 3.3.
The research questions we formulated in order to work with these subsets are: 1. Extracted from different genres of the Internet texts, are the texts neutral in the same degree? Do the texts express neutrality differently? 2. Could we rely on them both, either choose one of them or none of them for building our neutral training subset? In the Table 1,
Table 1 we summarize the results of comparison executed on different linguistic levels of textual content. To feature the subcorpora we use 6 types of markers whose statistical validity for distinguishing emotional classes of texts was proved previously (see more details in [22]). We use them as tertium comparationis to see the degree of neutrality of the subsets. To complete the feature list, we added the frequency of three classes of words – verbs, lexemes-somatisms and deictic words – which were proper for the Internet texts of all emotional classes of texts extracted from the public groups (“Overheard”, “Caramel” and “Room №6”). The aim was to see whether such properties are due to the type of text (Internet texts) or to its genre (personal stories for selected public groups or informative texts for Wikipedia).
As linguistic features, we took the following ones: 1. Normalized frequency of sentences ended by exclamation, because it was strong marker of anger and distress texts. 2. Normalized frequency of construction “intensifier так +Adverb” which was very predictive for ashamed texts. 3. Normalized frequency of words denoting parts of human body (somatisms) – голова (head), глаз (eye), рука (hand), нога (leg) – whose weight in all emotional classes, exempt Fear and Disgust was very important according to the results of TF-IDF method application. 4. Normalized frequency of words giving a reference to things said by others (e.g., мол) – we found such markers very robust to distinguish texts of Interest, Startle and Fear texts from others classes. 5. Normalized frequency of absolutist words [23] – никто (nobody), нигде (nowhere), никогда (never), не (not), нет (no), всегда (always), все (all), везде (“everywhere”) – which are rather good predictors of Fear texts [24]. 6. Normalized frequency of obscenisms – swear words used mostly in Anger and Disgust texts. 7. Normalized frequency of verbs and deictic words (здесь, тогда, вчера, тут, там) whose main function is to transmit the localization in space and time of the author of stories published in three mentioned above publics. Both classes of words are generally frequent in all classes of emotional dataset.
The evidence given by the comparative analysis of two subsets is that they have in common only one trait – the absence of markers of the speech of third person. This characteristic is significant for evaluating neutrality of subcorpora. However, the rest of the features demonstrates considerable discrepancies among corpora. Some of them are due to the differences in genres, for example, it is not authorized to use swearing words in Wiki texts, but, in contrast, in public groups such lexemes are not censored at all.
Moreover, the narrative nature of personal stories determines the abundance of verbs and deictics because they both signify the presence of a person who observes the situation, being integrated in it. The Wiki texts, on their part, do not contain mentioned integration in situation – its author only refers to the information, facts and descriptions provided by other people: scientists, researchers, and experts.
Another evidence is that texts, the informants were given as emotional, but assessed as neutral, contain many verbal markers specific for texts of different emotions. In attempt to find an explanation of it, we hypothesize that these texts were annotated as neutral ones because they had been assessed in comparison to much more emotionally saturated texts. For us, it means that the subset assessed as neutral in such way is biased.
If we turn back to one of our research questions formulated at the beginning of the section “Could we rely on them both, either choose one of them or none of them for building our neutral training subset?”, our resulting answer is that we cannot use them at all.
A new problem we face then is how to collect the subset responding to two main criteria: 1) its texts belong to the genre of personal stories published in social networks: otherwise, the texts of 8 emotional classes and neutral texts will be disparate and heterogeneous and, as result, it will bring some noise in classification; 2) its texts contain zero emotions independently, not in comparison with something else.
A research solution we have found is to elaborate two-classes (emotional / neutral) classifier, trained on Wiki texts so that to retrieve the dataset from social network public groups.
As a data source we used posts from social groups “Overheard”, ‘Caramel”, “Overheard: Man and Woman” which main genre is anonymous confessions. These public groups are very similar in terms of genre, style and audience so that it allows us to treat them as examples of the same discourse domain. Also, they are hosted at VK social network which allows users to download their content using a REST API. 4.1.
Using a specially built client, we have obtained most of these groups’ posts resulting in 70000 samples. As manual annotation of each item in this dataset is very labor consuming, a decision of automated filtering has been made. However, the automated approach has its drawbacks – for successful filtering of neutral samples we have to formalize the notion of “neutrality”, describe this entity and its features and then implement the process of filtering using a programming language. As explicit and formal definition of “neutrality – non-neutrality” is very hard to construct using conventional programming paradigms, it has been decided to opt for machine learning and train a classifier to tell neutral items from non-neutral ones that would allow it to make decisions on the base of training data.
The chosen approach was based on deep-learning techniques, which automate identification of relevant high-order features for given classes and then utilize these features for classifying previously unseen samples. The drawback of this way is non-interpretability of those features for a researcher, but we believe that such a situation can be tolerated because the main objective was downloading and filtering of data samples and the exploratory analysis has been made by other convenient tools such as linguistic corpus tools.
For training purposes, we used 14000 of samples for each class (neutral and non-neutral) and also we used 4000 of sample for validation needs. Both training and validation datasets were balanced. For non-neutral texts, we took a dump of Russian texts from the “Overheard” group that were previously labeled as “emotional”. As neutral dataset, random texts from Russian Wikipedia were obtained using Wikimedia API. The main hypothesis of the approach was that features regarding “neutrality” that are present in Wikipedia by definition could be obtained and discriminated by a neural network and later used for classification and prediction of neutral texts in social networks posts. Architecture of the classifier is presented on the Figure 1.
As it is shown on the Figure 1, the architecture for a classifier is quite straightforward regarding using recurrent network approach. This approach was selected due to available tools and capability to capture highly nonlinear features. We built the classifier on top of AllenNLP framework that is a Pytorch-based research platform for natural language processing domain.
The first layer of the neural network is an embedding layer which responsibility is to map sparse one-hot-encoded vectors to dense vectors with dimension of 512, so the tensor representation of tokens was trained on the classification task objective. We didn’t use any pretrained word embeddings like Glove or Word2Vec as they are very large in size and require much computational resources to be handled but from the point of view of accuracy their performance is comparable to using ad-hoc embedding layers. Hence, we plan to leverage Russian embeddings and language models in further research.
After the embedding layers there is an LSTM-encoder, which task is to read a sequence of embedded tokens and at each step pass the transformed values from embedded inputs and hidden state to the next LSTM-cell. The encoder dimension in our experiment was 256. At the end of the sequence, we can take the hidden state from the last cell and treat this vector as a numerical representation for the word sequence.
The final layer is a simple dense layer which task is to take a sequence vector from the encoder and transform it into logits of dimension (1, 2) which corresponds to two classes prediction. After dense layer, we use softmax function to transform logits into probabilities and derive the most probable class for a text. As a training objective, we used binary cross-entropy loss.
All the training was done using Yandex Datasphere platform with Nividia V100 GPU using 28000 data items. The batch size was 32 and we reached convergence at epoch 30 with maximum accuracy of 96% on validation dataset. Fig 2. presents accuracy performance for 12 epochs with validation patience of 4 epochs.
When the classifier was ready, we could use pretrained checkpoint to predict neutrality-vsemotionality class of the text for any arbitrary text. Since out interest was to filter neutral text from social publics with emotional content, we used this classifier to be such a filter.
The overall process is illustrated on the Figure 3. After having trained the binary classifier we fetched a dump of posts from public groups “Caramel”, “Room № 6”, “Overheard: Man and Woman”. Using automated means we leveraged VK developer API to obtain texts from this publics, we retrieved about 70 000 of posts. Then, we performed preprocessing and data cleansing and piped the obtained texts to the pretrained classifier and filtered neutral samples. As a result, we have got about 500 of posts that had been classified as neutral.
Additionally, we carried out a random assessment of 50 items, and all of them could be referred as “neutral” in terms of selection of words and expressiveness. As for text genres, the resulting subset includes together with personal stories a limited pull of advertisements, asking questions, notifications, call-to-action or invitations, but they don’t dominate the corpus.
To be sure about the neutrality of retrieved data, we compared it with two previously analyzed corpora, using the same “diagnostic” features (see Table 1).
It is to notice, that concerning features predetermined by genre of texts, we can observe the following. If in Wiki texts we found zero obscenisms, in neutral VKontakte texts after assessment their frequency was rather representative, in neutral data obtained after training we found them, but in minimum quantity – 14.08 ipm. In Wiki texts the normalized frequency of verbs and deictic words was fewer than in neutral texts from VKontakte, in data after training its value is fewer than in Vkontakte, but more important than in Wiki – 94873 ipm for verbs and 1126 for deictic words. For us, it means that the obtained data, as word statistic shows, occupy a middle position between information texts and texts of personal stories. The observation could be interpreted as the evidence for concluding that obtained subset conserves, apparently, social networks texts features: deliberativeness (no ban on swearing words) and focus on lived experience (actions described by verbs, time and space perception expressed via deictics).
As for the verbal markers of different emotions, their frequency values are closer to Wiki texts, than to neutral texts from VKontakte.
The current research and its results stress a very intriguing problem we faced in context of Emotional Text Analysis – the lack of reliable data sources for neutral texts.
The problem has several facets. First, the task to find “manually” the source domain in the Internet with neutral texts does not sound realistic. In Internet communication, the frontiers between speech registers and styles are very blurred. The emotions emerge everywhere in texts published on the Internet. Second, despite the weakness of stylistic norms, Internet has already built its own inventory of the written texts’ genres: personal stories, opinion texts, memes, informative texts, and others. Each of them has a range of particular features observable on different levels of textual material: morphology, syntax, vocabulary, even punctuation. When we are working in the machine learning paradigm, all mentioned text genre particularities in using words affect the accuracy of classification.
Thus, the challenge consists in a triple task: to find neutral texts, to perform it in an automatic way and, finally, to realize it in respect of the text genre characteristics.
The solution we found is rather vulnerable to criticism. However, we managed to combine all three tasks by using mixed techniques and methods. Its relevance and effectiveness will be approved or not on the step of running our multiclass classifier. Nevertheless, we believe it is an interesting experience within the Emotional Text Analysis paradigm based on the Internet texts in Russian.
We would like to thank the Russian Foundation for Basic Research for supporting our research (project № 19-012-00205 “Design of Sentiment Classifier for Internet Texts in Russian backed by Lövheim’s Cube Emotion Model”).
[10] Yu. M. Kuznetsova, Osnovaniya razrabotki kognitivnogo assistenta – protektora zdorovogo obraza zhizni [Basis for the development of a cognitive assistant – a protector of a healthy lifestyle], in: I. L. Surat (Ed.), Tendentsii i perspektivy razvitiya sotsiotekhnicheskoj sredy: Materialy IV mezhdunarodnoj nauchno-prakticheskoj konferentsii [Trends and prospects for the Development of the Sociotechnical Environment: Proceedings of the IV International Scientific and Practical Conference], Publisher of the Modern University for the Humanities, Moscow, 2018, pp. 441–452. (in Russian) [11] O. Koltsova, S. Nikolenko, S. Alexeeva, O. Nagornyy, S. Koltcov, Detecting interethnic relations with the data from social media, in: D. A. Alexandrov, A. V. Boukhanovsky, A. V. Chugunov, Y. Kabanov, O. Koltsova (Eds.), Digital Transformation and Global Society, Springer, Cham, 2017, pp. 16–30. doi:10.1007/978-3-319-69784-0. [12] Y.-P. Huang, T. Goh, C. L. Liew, Hunting suicide notes in web 2.0 – Preliminary findings, in: ISMW '07: Ninth Ieee International Symposium On Multimedia – Workshops, Proceedings, IEEE Computer Socierty, NW Washington, 2007, pp. 517–521. [13] P. K. Bhowmick, A. Basu, P. Mitra, Reader perspective emotion analysis, in text through ensemble based multi-label classification framework, Computer and Information Science 2 (4) (2009) 64–74. [14] O. Bruna, H. Avetisyan, J. Holub, Emotion models for textual emotion classification, in: Journal of Physics: Conference Series 772, Symposium: Metrology Across the Sciences: Wishful Thinking?, IOP Publishing, Berkley, 2016. doi:10.1088/1742-6596/772/1/012063. [15] C. E. Izard, Basic emotions, relations among the emotions and emotion-cognition relation,
Psychological Review 99(3) (1992) 561–565. doi:10.1037/0033-295x.99.3.561. [16] P. Ekman, Are there basic emotions, Psychological Review 99(3) (1992) 550–553.
doi:10.1037/0033-295X.99.3.550. [17] C. O. Alm, D. Rot, R. Sproat, Emotions from text: machine learning for text-based emotion prediction, in: R. J. Mooney (Ed.), HLT '05: Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Stroudsburg, 2005, pp. 579–586. doi:10.3115/1220575.1220648. [18] E. Volkova, B. Mehler, W. D. Meurers, D. Gerdemann, H. Bülthoff, Emotional perception of fairy tales: achieving agreement in emotion annotation of text, in: D. Inkpen, C. Strapparava (Eds.), Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Association for Computational Linguistics, Los Angeles, 2010, pp. 98–106. [19] C. Strapparava, R. Mihalcea, SemEval-2007 Task 14: Affective Text, in: E. Agirre, Ll. Marquez, R. Wicentowski (Eds.), SemEval '07: Proceedings of the 4th International Workshop on Semantic Evaluations, Association for Computational Linguistics, Stroudsburg, 2007, pp. 70–74. doi:10.3115/1621474.1621487. [20] A. Neviarouskaya, H. Prendinger, M. Ishizuka, @AM: Textual Attitude Analysis Model, in: D.
Inkpen, C. Strapparava (Eds.), Proceedings of the NAACL HTL 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, Association for Computational Linguistics, Los Angeles, 2010, pp. 80–88. [21] N. Cimini, J. Burr, An aesthetic for deliberating online: Thinking through «universal pragmatics» and «dialogism» with reference to Wikipedia, The Information Society 28(3) (2012), 151–160. doi:10.1080/01972243.2012.669448. [22] A. V. Kolmogorova, A. A. Kalinin, A. V. Malikova, Tipologiya i kombinatorika verbal'nykh markerov razlichnykh emotsional'nykh tonal'nostei v internet-tekstakh na russkom yazyke [The types and combinatorics of verbal markers of different emotional tonalities in Russian-language internet texts], Vestnik Tomskogo gosudarstvennogo universiteta [Tomsk State University Journal] 448 (2019) 48–58. doi:10.17223/15617793/448/6. (in Russian) [23] M. Al-Mosaiwi, T. Johnstone, Linguistic markers of moderate and absolute natural language,
Personality and Individual Differences 134 (2018) 119–124. doi:10.1016/j.paid.2018.06.004. [24] A. V. Kolmogorova, L. A. Vdovina, Leksiko-grammaticheskie markery emotsiy kak parametry dlya sentiment-analiza russkoyazychnykh internet-tekstov [Lexical and grammatical markers of emotions as parameters for sentiment analysis of internet texts in Russian], Vestnik Permskogo universiteta. Rossiyskaya i zarubezhnaya filologiya [Perm University Herald. Russian and Foreign Philology] 11 (3) (2019) 38–46. doi:10.17072/2073-6681-2019-3-38-46. (in Russian)