In this study, we examined how precisely a sentiment analysis and a word list-based lexical analysis predict the emotional valence (as positive or negative emotional states) of 63 emotional short stories. Both the sentiment analysis and the word listbased analysis predicted subjective valence, which however was predicted even more precisely when both analysis methods were combined. These results can, for example, contribute to the development of new technology-based teaching designs, in that positive or negative emotions in the texts or online-contributions of students can be assessed in automated form and transferred into instructional measures. Such instructional actions can, for example, be hints, learning support or feedback adapted to the students' emotional state.
There has been great progress in technology-based
learning in recent decades. Methods and
procedures of learning analytics have recently played an
important role here. In principle, learning
analytics is about collecting data from students during
learning and using it to improve teaching. Despite
progress in Natural Language Processing (NLP),
texts or contributions from students have rarely
been used as a source of information for
learning analytics or for technology-based learning
In the late nineties, Barrett and Russell (1999) developed the circumplex model, a model of emotions with two dimensions; emotional valence and emotional arousal. Emotional valence is the experience of one’s own actual positive or negative feeling. Emotional arousal is the subjective amount of internal activation or energy. Together, these two dimensions form the core affect, “the most elementary consciously accessible affective feelings that need not be directed at anything” (S. 806). The circumplex model provided the theoretical basis for the present work.
Emotional valence, based on the circumplex model,
was measured on a bipolar scale, ranging form very
negative to very positive. This method was
originally conceived by Wundt (1896) and is the most
commonly used method to date. However, like the
sentiment analysis used in this study, some
theories view valence as a bivariate construct (e.g.
Norris et al., 2010; Briesemeister et al., 2012; Shuman
et al., 2013; Kron et
Today, research assumes that individual measurements cannot capture the phenomenon of emotions entirely. This leads to the practice of using multiple measuring methods in scientific investigations, often in conjunction. Self-reports such as questionnaires or single item questions are a popular way of measuring emotions, and for good reasons: they have good validity (as long as response biases are taken into account) and enable quick and simple data collection. Non-verbal alternatives to measure emotions can also be used, such as the SelfAssessment Manikins (SAM scale) from Bradley and Lang (1994), measuring feelings, i.e. the subjective experience of emotions. This instrument contains visual rather than verbal stimuli (i.e. pictures rather than questions), which consist of abstract representations of a human being displaying different emotions. The scale varies in three dimensions; valence, arousal, and dominance. The valence dimension shows pictures ranging from a smiling face to a frowning face, with more neutral expressions in-between; and in the arousal dimension, pictures range from a sleepy and calm figure to a wide-eyed, excited expression. We did not use the dominance dimension that represents the controlling and dominant nature of emotion shown by a tiny figure in the middle of a square for low dominance towards a oversize figure going beyond the borders of the square for high dominance. Raters were instructed to choose the image that best represents their own current emotional state. 2.2
Despite their popularity, self-reports are far from
the only instrument being used in affective
science. Lexical analysis (i.e. analysis based on
single words) is a different, more objective
instrument which historically has been used significantly
less often. In this reg
Before neurological research pointed out these
connections, there was a clear language-emotion gap,
i.e. most emotion theories ignored language
functions, while linguistic theories ignored affective
processes. In order to bridge that gap, the Berlin
Affective Word List (BAWL-R) was developed
The BAWL-R specifically has been utilized for this
purpose
The automatic sentiment analysis system used for
this paper is composed of two parts, namely the
model and the data. The multi-layered
convolutional network model is the same as in Deriu et al.
(2017). The authors trained this network as shown
in Figure 1 with a large number of tweets in
different languages that were weakly supervised, and
demonstrated the importance of using pre-training
of such networks. The specific pre-training
procedure, named distant-supervised learning, is trained
on larger weakly or non-labelled samples2.
Afterwards the network is further trained on a much
smaller data set with manually strongly labelled
samples. The approach was evaluated on
various multi-lingual data sets, including the
SemEval2016 sentiment prediction benchmark (Task 4),
where it achieved state-of-the-art performance.
This model was trained on the SB10k German
Twitter sentiment corpus
A comparison of different procedures for emotion recognition on the sentence level was conducted by Aman (2007). He concluded that a combination of different automatic procedures for recording emotions is advantageous. This finding is also supported in a paper by Strapparava and Mihalcea (2010). They tested several methods for automatically detecting emotions in short texts (headlines and blog posts; 100-400 characters). Six headline advisors rated the presence of six distinct emotions as well as the valence of the texts, which were then predicted by several procedures. The study found that ”different methods have different strengths, especially with respect to individual emotions” (p. 35). Most interestingly, the 2On twitter emoticons/emojis can be used as weak labels, for instance a tweet with a smiling emoji will probably have a positive sentiment. correlation between emotions evaluated by human raters and those found by algorithms was moderate with max. r = .48 (explanation of variance max. 24%). The largest effect was found with valence analysed in a knowledge-based, areaindependent, unsupervised CLaC approach. We assume, as already mentioned above, that different measurements can cover certain aspects of the complex phenomenon emotions that others do not. Different measurements often only reveal parts of a phenomena and might sometimes even be contradictory. Thus, the combination of several measurement techniques can prove to be fruitful. In our study, we are interested in the combination between lexical analysis, sentiment analysis and self-report and specifically, if prediction of the latter improves when the former two are combined. 2.4
As discussed above, it has been known for a long
time (e.g. Freud (1891); Bu¨hler (1934) that words
can trigger emotional reactions, which more
recently has been confirmed in neurologic
As several studies have shown (e.g. Aman (2007); Strapparava and Mihalcea (2010), combinations of several methods for estimating emotions in texts lead to better predictions than one method alone. This leads to the second hypothesis.
1. The emotional valence measured by lexical analysis and by sentiment analysis each predict the subjective valence of the short texts. 2. The combination of the measurements methods (lexical analysis and sentiment analysis) increases the predictive power.
Despite the sizable amount of research in
emotion and in text analysis, we are not aware
of many studies that not only compared
The 63 analysed texts originated from a collection of 102 German texts written by 32 authors, 21 of which were German speaking students and staff of an University in Germany (mean age 26.10, SD = 10.65; gender: 85% women), and 11 of which were recruited by the first author (university staff and people recruited via social media and personal contacts; mean age 36.82, SD = 15.78; 64% women). These 63 texts are part of an international database with over 200 emotional short stories which are developed and refined within the framework of the COST initiative E-Read IS 1404 (Kaakinen et al., in preparation). The international database contains stories from Finland, France, Germany, Portugal, Spain, Switzerland, and Turkey. All stories are subjectively rated on emotional valence, emotional arousal and comprehensibility in their original language and in English. All texts have a length of 900 to 1100 characters including spaces. Texts that were not written in the first person were rewritten without changing their content and structure. The topic varies from story to story, some of them tell of joyful events and experiences (e.g. birth, love, music) or negative ones (e.g. death, abuse). A few stories are emotionally neutral, i.e. neither positive nor negative and with a medium level of emotional arousal. The stories are mostly easy to understand. Once finished, the database will be presented in a publication and made freely accessible.
The subjective valence rating of the texts was
conducted with the Self-Assessment-Manikin scale
(SAM3) by
The automatic sentiment analysis was trained on
sentences. Nevertheless, we applied it to our short
stories as a whole instead, since the subjective
ratings we wanted to predict were on a text
rather than a sentence level. For this paper, we
calculated a new overall valence variable for the
sentiment analysis data based on the negative and
positive scores (negative sentiment minus positive
sentiment), assuming that the neutral sentiment
had no influence on the positive or negative
orientation of the analysis. The reason for this decision
was that the three original variables sum up to 1
and are therefore interdependent. Consequently,
their individual effects on the subjective ratings
canceled each other out. In order to obtain values
comparable to the BAWL-R valence variable, this
new variable was created. We further analyzed
the text in terms of readability. Readability was
scored with the well established Flesch Index
Valence Subjective
Lexical Sentiment Sentiment positive negative mean 4.46 0.63 -0.44 0.14 0.29
SD
2.21
0.27
0.26
0.10
0.12
min
1.17
0.02
-0.95
0.01
0.04
max
8.61
1.17
0.20
0.47
0.84
scale
-3 - 3
1 - 9
-1 - 1
0 - 1
0 - 1
We chose to conduct our regression analyses
with a Bayesian approach, which has
important advantages over the traditional frequentist
null hypothesis significance testing. Within the
Bayesian approach, the interpretation of data is
not affected by sampling intention. In contrast to
the frequentist approach, the Bayesian approach
permits assessment of the relative credibility of
parameter values given the data and the statistical
model
The correlation between the sentiment value calculated as the difference between negative and positive sentiment - and the lexical value of valence was r = .50 (95% Credible Interval CrI = [.28; .72]). Both of them had a moderate positive correlation with the subjective valence ratings of the texts (r = .51 (95% CrI = [.28; .72] for sentiment; r = .62 (95% CrI = [.42; .82] for lexical valence). There was a weak correlation between the Flesch readability score and the other three variables (sentiment: r = − .24 (95% CrI = [-.48; .01]; lexical valence: r = − .12 95% CrI = [-.37; .12]; subjective valence: r = − .24 (95% CrI = [-.50; .01]).
A visual inspection of the MCMC chains and the R-hat diagnostic with all R-hat values < 1.02 revealed good convergence for all estimated parameters of all calculated models.
The restricted model (Model 0) including the intercepts and the level 2 variable only had a LOOic of 4969. Model 1 predicting subjective valence by sentiment had an effect of β = 4.60 (95% CrI = [2.62; 6.56]). The LOOic was 4717. Model 2 predicting subjective valence by lexical valence had an effect of β = 5.37 (95% CrI = [3.62; 7.01]) with a LOOic of 4578. To decide, which model is to prefer, we relied on the credible intervals of the LOOic. The credible intervals of the LOOic of Model 1 and 2 did not overlap with the LOOic of the restricted model 0 (see Table 2). That lead to our conclusion that both models predicted subjective valence and that the first hypothesis could be confirmed.
Model 3 predicting subjective valence of texts by sentiment and lexical valence (BAWL-R) is presented in Table 3. The design formula for model 3 was formulated as follows:
Ri is the ordered distribution (i.e. a categorical distribution that takes the vector p = { p1, p2, p3, p4, p5, p6, p7, p8} of probabilities of each subjective valence rating value below the maximum category of 9). α k is the unique intercept of each possible outcome value k, φ i is the linear model that is subtracted from each intercept, β BAW L and β Sent are the slopes of the BAWL-R (lexical analysis) and sentiment values respectively and BAW Li and Senti are the values of both predictor variables on row i.
The subjective valence was predicted by
sentiment with β = 2.07 (95% CrI = [1.58; 2.59]),
and by lexical valence with β = 3.42 (95% CrI
= [2.96; 3.89]). The LOO information criteria of
model 3 (LOOic = 4515) was smaller than that of
either of the other models. The credible interval
of model 1, but not of model 2 does not overlap
with the credible interval of the combined model
3. We conclude that sentiment and lexical analysis
predict subjective valence better than sentiment
analysis alone. Even if the credible interval of
model 3 overlaps with the credible interval of
Strapparava and Mihalcea (2010) in detecting
sentiment in headlines. The algorithm with the best
predictive power was the CLaC system that
”relies on a knowledge-based domain-independent
unsupervised approach to headline valence detection
and scoring. The system uses three main kinds
of knowledge: a list of sentiment-bearing words, a
list of valence shifters and a set of rules that
define the scope and the result of the combination of
sentiment-bearing words and valence shifters” (p.
28). This algorithm found a correlation of r = .48
for valence. The correlations with the other four
algorithms were all below r = .40. In comparison
to other studies, the sentiment analysis used in this
study revealed a rather high correlation. A more
recent study
When both measurement techniques were combined in a model, the effect of lexical valence predicting subjective valence was stronger than the effect of the sentiment analysis. The β = 2.07 in model 3 for sentiment means that an increase of 1 SD in the sentiment values corresponds to an increase of 2.07 SDs in the predicted subjective valence. Likewise, the β = 3.42 for lexical valence means that an increase of 1 SD in the lexical valence values corresponds to an increase of 3.42 SDs in the predicted subjective valence. This stronger effect of lexical analysis was also visible in the correlations of each variables with subjective valence. Both predictors correlate with each other (r = .50), and therefore share a good part of their variance. This explains that overlap between the credible intervals of model 3 (sentiment and lexical analysis as predictors) and model 2 (lexical analysis as predictors). Nevertheless, we considered the information gain of model 3 over model 2 to be large enough and therefore favour model 3.
It is known from literature that the difficulty of
texts has an impact on the emotions when reading
There are some aspects to our approach that we did not account for in this study that may be worth exploring in future studies. One such aspect is the perceived difficulty of the rating task since the subjective ratings may be biased if the task is thought to be either particularly easy or particularly difficult. This concerns both the text ratings as well as the ratings that resulted in the two analysis approaches that rely on subjective expert ratings at their very core. Another aspect worthy of inspection is the discrepancy between human ratings and both analysis approaches since they do not necessarily align at all times. Exploring under which circumstances they diverge may prove to be a promising venture. 6
The results indicate that lexical and sentiment analyses predict subjective appraisal of emotions triggered by short texts. The two methods are not redundant. It is therefore worthwhile analyzing the emotional potential of texts applying both measurement procedures. A next step is to repeat these analyses on sentence and on word level to check whether we get an even stronger predictive power. We also need to examine the integration of other text properties, including other semantic parameters, into our analysis, as done by Jacobs and Kinder (2017). The small effect gain of the Flesch-Index can be interpreted as an indication that non-emotional text properties could play a role in the perception of emotions in a text. The results found, for example, can contribute to the development of new instructional designs that measure emotional appraisals of students engaged in digital learning tasks. Positive or negative emotions in the texts or online-contributions of students can be assessed in automated form and transferred into instructional measures, and thus help to integrate automated learning support into feedback, hints or adaptive instructional design. We need even more predictive power for useful integration of such sensors, i.e. measurements of emotional or affective properties of texts in digital learning, in educational practice. From our point of view, this can be achieved by combining different measurement methods
A big ’Thank you’ to Yvonne Kammerer of the Leibnitz-Institut fu¨r Wissensmedien in Tu¨bingen. She organized the collecting of the text from Germany, and the rating of the 63 texts that were collected as part of a project in the COST Action EREAD. We also thank Mark Cieliebak and Jan Deriu for providing the sentiment prediction system and the helpful discussions, and St´ephanie McGarrity for proofreading and useful suggestions.
PJ