The ongoing digitalisation facilitates measuring emotional characteristics of texts (e.g. lexical emotional valence), and emotional face expressions (e.g. facial emotional valence). In this context, a text was lexically analysed with the revised Berlin Affective Word List (BAWL-R), and videos of 91 subjects reading this text were analysed with a facial emotion recognition software. We hypothesized that lexical emotional valence predicts readers' facial emotional valence. The result was significant but explained nearly no variance (0.3%). Detecting emotional face expressions is a well established method, which means that the mostly neutral face expressions of our participants may be a result of the non-social reading situation.
Digital reading is becoming increasingly frequent and
important. This trend is not only affecting
business correspondence (e-mail, e-collaboration) or
online shopping, but also reading fictional and
nonfictional texts. Digital reading can be defined in
contrast to “traditional” reading by the texts’ own
characteristics. Digital texts can be structured as
hypertexts permitting to navigate through multiple different
documents, but they can be linear as well. They may
contain multimedia such as sound and dynamic
visualisations. They are presented on a computer screen,
a tablet, an e-reader, a smart phone or via virtual
reality goggles. Technology permits digital reading to
become more social and interactive by sharing and
commenting on information
Humans evaluate every kind of event concerning their
actual emotional state, for instance a text or its
individual words, with respect to valence (negative /
positive), as well as novelty and relevance for an
individual’s goals or needs
Emotional valence, a result of the primary appraisal, is the experience of one’s own actual positive or negative state. It is a first dimension of the circumplex model as proposed by Russell and Barrett Feldman (1999). The second dimension of the circumplex model is emotional arousal, i.e. the subjective amount of activation or energy. Together, these two dimensions form the core affect, “the most elementary consciously accessible affective feelings (and their neurophysiological counterparts) that need not be directed at anything” (S. 806). In this paper, we analyze the emotional valence 1) with a textual analysis of a fictional text read by the participants - the lexical emotional valence - and 2) by the emotional reaction of the readers expressed on their faces while reading the text - the facial emotional valence. 2.1.1
Lexical emotional valence
on a list of Jakobson (1979). The metric
characteristics concern the structuring of a language in units
(e.g. line, verse) by means of rhythm and
articulation (e.g. rime, assonance, alliteration). The
phonological characteristics affect the function and sound of
phonemes. The syntactic characteristics pertain to the
combination of words and word groups in larger units
like sentences. The semantic characteristics concern
the meanings of sentences, parts of sentences, words,
components of words, or characters per se. The four
hierarchical levels are the sub-lexical level (i.e. the
characteristic of components of words like phonemes,
articulation), the lexical level (i.e. single words
without consideration of the textual content), the
interlexical relations between words, phrases, or sections,
and the supra-lexical level concerning whole
sentences and stories. In a structural text analysis, there
are several ways to measure different characteristics
of a text, for instance by rating the lexical emotional
valence of a text, i.e. estimating how a reader may
emotionally experience the text. In this sense, lexical
emotional valence is part of the emotional potential
of a text, a combination of “information structure,
coherence, and implicit information combined with
expressive language”
A guide for textual structure analysis is the 4x4 matrix
for text analysis by Jacobs (2015). He proposes a
classification with four text characteristics and four
hierarchical levels. The four text characteristics are based
The most commonly used tool for systematically
measuring emotions including emotional valence visible
in the face is the Facial Action Coding System (FACS)
by Ekman and Friesen (1978). FACS “is a
comprehensive, anatomically based system for measuring
all visually discernible facial movements”
We assumed that the emotional valence of words, the
lexical emotional valence, influences the reading
experience
We hypothesized that the lexical emotional valence of a text, respectively its sections, predicts the readers’ facial emotional valence. We suppose that the course of the emotional valence expressed on the reader’s face follows the course of the valence of words in a text and its sections.
In spite of a large amount of research in
emotion and in text analysis, we are aware of one other
research group
The data for this analysis stem from a reading experiment investigating the influence of cognitive load and emotional reactions on reading performance. The participants filled in several questionnaires and were instructed to read the first part of a fictional text with neutral content. When finished, they were asked how they felt just in that moment (valence, arousal) and were instructed to indicate the difficulty of the text. Then the participants had to retell the story and answered five multiple choice questions to assess what they memorized about the text. This procedure was repeated with the second and the third part of the text, one of which had negative undertones while the other had positive ones. The texts were presented on three different screen sizes simulating different reading devices (smartphone 5”, tablet 10”, and laptop 15”) and with two different levels of readability (easy vs difficult). Screen size and readability were presented by chance. The experiment was followed by a series of pictures of persons displaying different emotions and nature images (landscapes, snakes, spiders) to measure emotional reactions. 3.2
The text presented in the experiment consisted of three parts telling an overall story. The first (neutral) part describes a park in a town with a pond surrounded by a path with three people sitting on different benches (an old man, a young man and a young woman). The second (negative) part tells the story of an apparent break-up between the young couple. The text contains scenes with disgusting fantasies and describes outbursts of violence. The third (positive) part reveals the whole ordeal as a prank orchestrated by the couple’s friends and ends by describing how the two reconcile and dream of holidays by the sea. 3.3
Sample 103 students attending secondary school participated in the study. Most of the participants were women (87%) with an average age of 17.8 (SD: 1.2; range 16-21). For this analysis, the data of 91 students were usable. Data were lost due to missing videos of the respective participant’s face. There were no differences between the complete sample and the subsample in age (F[1,101]=0.56, p=.813) and gender (F[1,101]=0.19, p=.657). 3.4
The semantic lexical analysis of the text was executed
with the revised form of the Berlin Affective Word
List BAWL-R
The facial emotional valence expressed on the participants’ faces was measured objectively using the FaceReader R (version 7) software by Noldus R which is based on Ekman’s FACS and utilizes 21 of its action units. The FaceReader R rated the videos of all participants whose faces were filmed. It automatically calculates emotional valence by subtracting the intensity of the most intense negative expression (sadness, disgust, anger or fear) from the intensity of positive expression (i.e. happiness) in a specific timeframe. The values are between –1 and 1. To combine the two data sets, the reading time of each section of the three text parts was estimated with the number of characters in each section in relation to the number of characters of the whole text part (e.g. neutral). For example: the neutral text part (easy to read) contained 2618 characters (100%), its first section 654 characters, i.e. 24.98%. Therefore, each subject’s reading time for the first section of the neutral easy to read text was estimated to be 24.98% of the whole reading time of the neutral easy to read text.
Besides the above-mentioned variables we
measured subjective feelings, perceived text difficulty, and
knowledge about the texts read. In addition, the
participants filled in questionnaires assessing
sociographic issues, reading habits, visual memory span,
and actual mood
Most statistical analyses were conducted with R
The differences of the lexical emotional valence between the three text parts (neutral, negative, positive) estimated with the BAWL-R was statistically significant (F[2,1128]=22.80, p<.001). A post hoc Tukey test revealed that the lexical emotional valence of the positive text part differs significantly from the two other text parts. The lexical emotional valence did not differ between the two readability levels (easy, difficult; F[1,1129]=0.41, p=.523). The neutral text part had a lexical emotional valence of 0.66 (SD: 0.87), the negative part of 0.52 (SD: 1.18), and the positive part of 1.12 (SD: 1.12). The average facial emotional valence measured with FaceReader R did not differ between the three text parts (F[2.1595]=0.31, p=.733), nor between the two readability levels (F[1,1596]=0.02, p=.890).
The time course of the lexical emotional valence resulting from the semantic lexical analysis shows similar values for the easy and difficult texts (the upper two lines in Figure 1).
Overall, the course of the facial emotional valence on the lower two lines of Figure 1 shows values that are all in a narrow range.
We tried to predict the course of the facial emotional valence with a generalized linear mixed model (GLMM) based on the lexical emotional valence of the texts. We included fixed effects for readability (easy vs. difficult), type of text (neutral vs. negative vs. positive), and lexical emotional valence (BAWLR), including all higher order interaction terms. Furthermore, we included random intercepts and random slopes for the effect of time of each participant. Text difficulty and type of text were effect-coded in order to interpret the regression weights at the grand mean (instead of a reference category).
The GLMM yielded a significant prediction of the facial emotional valence by the lexical emotional valence (β=0.02, CI=[0.02; 0.02], p<.001). All other effects (influences of type of text and readability, and all interactions) were significant as well (all p< .001, see Table 1). However, it is important to note that all fixed effects combined only explained 0.3% of variance in the facial emotional valence (pseudo-R2 of the fixed Fixed Parts (Intercept) Lexical emotional valence Negative text Positive text Readability difficult Lex.emo.valence: difficult Lex.emo.valence: negative text Lex.emo.valence: positive text Negative text: difficult Positive text: difficult Lex.emo.valence: neg. text: difficult Lex.emo.valence: pos. text: difficult
σ2 τ00,vp ρ01 Nvp ICCvp Observations R2/Ω20
p effects).
The interaction effect between lexical valence and readability on facial valence was also significant (β=0.02, CI=[0.02; 0.02], p<.001). This effect refers to all three text parts. Visual inspection of Figure 2 revealed distinct differences between the slopes of the easy and difficult text in all three parts. Therefore, we calculated the model for all three text parts separately. All three analyses yielded significant results. The largest effect of lexical emotional valence on facial emotional valence can be found in the negative text part (β=0.09, CI=[0.08; 0.09], p<.001). But, the explained variance remained very low (1.4%; pseudoR2 of the fixed effects). The pseudo-R2 in the neutral and the positive text part were even smaller (0.8% / 0.03%). 5
Statistically, facial emotional valence is significantly predicted by the lexical emotional valence. The explained variance of the fixed effects was very low at 0.3%, challenging the possible conclusion that the facial emotional valence of readers corresponds to the lexical emotional valence of the different sections of the text. There is one result that attracts attention: The slope of the easy readable negative text suggests that facial emotional valence is predicted more strongly in this text. But as with the complete model, the explained variance of the fixed effects is very low at 1.4%. Even if this effect does not explain a large amount of the variance, it remains interesting as it fits to results of other studies. The finding that emotional reactions are suppressed with higher cognitive load (for instance a more hard-toread text) have been reported for other tasks for example in Berggren et al. (2013), and in DeFraine (2016): “cognitive load reduced the intensity of negative emotions during passive-viewing of emotional images but not during emotion maintenance” (p. 459). Van Dillen et al. (2009) found a similar result, high cognitive load “eliminated all emotional expression differences” (p. 5). It is of interest that 54% of the variance is explained when the random effects are included, i.e. also including the variance between subjects. The differences between the subjects are large and could be an indicator that the sample consists of groups with different emotional reactions, i.e. with different facial expression behavior.
There may be some problems in the research design. The estimation of the reading time for each section is based on the percentage of characters in each section. This presumes a regular reading speed. There may be some divergences that diminish the precision of the analyses, or the text sections may be too large. To get a more precise definition of the reading time of each section, or to obtain a more fine-grained analysis with a division in sentences or words we will use the eye tracking data that were collected during the experiment.
A critical aspect of the analysis model is that only the lexical emotional valence was entered without including other text characteristics as it was done by Ullrich et al. (2017) and Hsu et al. (2015). 6
Despite the significant results - the explained variance of the fixed effects is very small - we cannot conclude that the readers’ facial emotional valence corresponds to the lexical emotional valence of a given text. Reading is not mirrored in the face. Our aim to predict the emotional reactions of distance learners presenting a text with known semantic characteristics is still far away. We have to work with other measurement methods or with other methods of analysis.
We thank Dr. Ste´phanie McGarrity of the Institute for Research in Open-, Distance- and eLearning (IFeL) of the Swiss Distance University of Applied Sciences (FFHS) for proof reading and her valuable advice that greatly improved the manuscript. Also, many thanks to Dr. Fernando Benites of the ZHAW School of Engineering for his great and patient technical assistance in producing the paper.