Valence is a concept that is increasingly being used in the computational study of narrative texts. We discuss the history of the concept and show that the word has been interpreted in various ways. Then we look at a number of Dutch tools for measuring valence. We use them on sample fragments from a large collection of narrative texts and find only moderate correlations between the valences as established by the various tools. We discuss these diferences and how to handle them. We argue that the root cause of the problem is that Computational Literary Studies never properly defined the concept of valence in a narrative context.
The study of emotion and sentiment is increasingly popular in computational literary studies
[
One of the most-quoted studies in the field is the article by Reagan et al. about six basic
shapes in the emotional arcs in stories3[0]. The emotional arcs that the authors create
represent the flow of what they call ‘sentiment’, measured by the ‘Hedonometer’, a dictionary-based
tool that assigns sentiment to words1[
Bizzoni and Feldkamp 2[] do address the issue in a case study onThe Old Man and the Sea. For each sentence in the novel, two annotators rated ‘the sentiment expressed by the sentence’. They were instructed ‘to avoid rating how a sentence made them feel and to try to report only on the sentiments actually embedded in the sentence, i.e., to think about the valence of each sentence individually, without overthinking the story’s narrative to reduce contextual interpretation’. This is an interesting instruction, in that it explicitly states the sentiment is not in the reader and it shouldn’t relate to the story events. It assumes that there is such a thing as ‘the sentiment embedded in a sentence’. The paper then goes on to check whether LLM or dictionary-based sentiment models correlate with the annotators’ ratings. We will come back to this study below.
Rebora, in his survey of sentiment analysis in literary studi3es1][, also asks where the
sentiment is supposed to reside: in the text or in the reader. He notes that in literary studies,
narratologists would opt for the text, while students of reader response would look at the reader.
But he also points at a third possibility: the characters as vehicles of emotion. Nalisnick and
Baird, e.g., have applied sentiment analysis to Shakespeare’s plays in order to study the
relations between the plays characters2[7]. And there are other possibilities: the sentiment that
one finds in the text can also be used to gauge the sentiment of the author, as in Stirman and
Pennebaker’s study of suicidal poets3[
It is clear that all of the above approaches to sentiment can be enlightening. But there is a danger that we forget that sentiment in texts can have these diferent aspects, and our current tooling is certainly not able to distinguish them. In an efort to create some clarity, we will in this paper briefly recount the history of the concept of valence, focusing on the various definitions researchers have used as well as on how it was established or computed. In a second, empirical part, we look at a number of tools for computing valence in Dutch. In so far as they are dictionary-based we look at the overlap between dictionaries and the degree to which they assign the same values to their shared words. Then we use a sample of fragments from Dutch fiction to assess how the various tools compare in their assignment of valence to these fragments.1
Note that our main interest here is not in the sentiment arcs that can be derived from book
segments’ valences (see also 1[
We start our short history of the concept of valence with the 1957 booTkhe measurement of
meaning, by Osgood, Suci and Tannenbaum 2[
After averaging, this gave them, for twenty concepts and fity pairs of adjectives, 1000 measurements. A factor analysis identified the hidden dimensions underlying the measurements. A fragment of the resulting table is reproduced in Figu2r.eWe see that the results of some adjective pairs, such as good vs. bad and beautiful vs. ugly, are almost completely explained by the hidden variable I, the scores on strong vs. weak are mostly explained by hidden variable II, etc. But what are these hidden variables? Osgood et al. then write ‘The problem of labelling factors is somewhat simpler here than in the usual case. (...) The first factor is clearly identifiable asevaluative (...). The second variable identifies itself fairly well aspaotency variable (...). The third factor appears to be mainly aanctivity variable (...)’ (italics original).
Later, these dimension would become known by other names: evaluativeness as pleasure or valence, activity as arousal, potency as power or dominance. This is not the whole story, but Osgood and colleagues made a fundamental contribution to the three-factor theory of emotion. It is interesting to note that they already mention that these factor loadings depend on cultures: e.g., for Japanese and Korean respondents, the adjective pair delicate vs. rugged clearly belonged to the evaluative dimension, for U.S. respondents it did not.
We continue with a look atThe General Inquirer, one of the first text analysis programs [
The 1966 book describes among others the Harvard III dictionary, developed for use with the General Inquirer. Four groups of words were included to account for the high and low ends of the evaluativeness and potency variables found by Osgood (pp. 176, 185). We see here that what for Osgood were hidden variables, the result of a computational process that required the interpretative step of labelling, have now become measurable entities underlying texts.
A next step in our tale was set by Bradley and Lang in 1999, in their paper ‘Afective Norms for English Words’7[]. As prompts in psychological research they needed words with known values for the dimensions of (what they called) Pleasure, Arousal and Dominance. They asked subjects to rate the words using a ‘self-assessment mannikin’ (see Figur3efor an example). For pleasure, the instruction that they gave their subjects was ‘At one extreme of this scale, you are happy, pleased, satisfied, contented, hopeful. When you feel completely happy you should indicate this by bubbling in the figure at the left. The other end of the scale is when you feel completely unhappy, annoyed, unsatisfied, melancholic, despaired, or bored’ (italics ours). The valence associated with a word now equals (possibly) complete happiness or unhappiness of the subject, i.e., a subjective feeling.
The creation of ever larger dictionaries of words with associated valence and other variables
has been a constant in psycholinguistic research ever since. We mention a few: Warriner and
colleagues created a list of almost 14000 English lemmas with Valence, Arousal and Dominance
[
One study, especially relevant for the second part of this paper, was done by Moors and
colleagues [
With the appearance of consumer reviews, the problem of establishing the opinions that they expressed and the related sentiments became important subjects for marketeer2s1][. As an example of early studies we mention Hu and Liu’s work15[]. Hu and Liu find sentences in reviews that contain both a product feature and an opinion. The opinion words and their polarity (= whether they express a positive or negative sentiment) are deduced from WordNet by starting with some seedwords. Over the years, their work has produced a 6800-word opinion lexicon. In the study of consumer reviews, the question is no longer how a word strikes the reader, but what was the opinion that a writer wanted to express. That also means that the attention moves to adjectives rather than the nouns that the tradition established by Bradley [7] was typically interested in.
A Dutch sentiment dictionary with a focus on review sentiment was created by De Smedt
and Daelemans [
Under the name of SentiArt, Arthur Jacobs introduced a completely diferent way of estimating valence in the field of computational literary studies17[]. The method uses a set of positive and negative seed words in combination with a word embedding. The valence of a word is then computed summing its similarities to the positive seed words and subtracting the similarities to the negative ones. The method is based on the assumption that, in the word embedding, words with similar meanings cluster together. Because there is no need for human ratings, this seems like an objective procedure. However, the choice of the texts that are used to create the word embedding, the procedure that is used for its computation as well as the choice of seed words are to some extent arbitrary. It is also not self-evident that the procedure should work at all. Jacobs 1[7, p. 3] states that the method was able to explain 34% of the variance between words found by Warriner40[], which isn’t exactly promising.
The various approaches to the valence concept that we have discussed all assign valence at the word level. It is not self evident that it is possible to define valence at a higher level, e.g. that of sentences, paragraphs or even book chapters.
Bradley and Lang extended their work on word valence to small texts (one to a few sentences)
[
Specifically in the context of narrative, Rebora asked students to rate the sentiment in
paragraphs of a story by Pirandello32[]. Their agreement was very weak. As we saw, in 2[] two
researchers rated sentiment in sentences inThe Old Man and the Sea. Their result was better
than Rebora’s: the correlation between their ratings was strong, after detrending even very
strong. Kaakinen and colleagues report on a multilingual database of short stories (ca. 1,000
characters) [
Outside of the domains of psychology or narrative, there exists a plethora of studies on the
polarity of especially reviews and social media texts. We just mention work on the orientation
of tweets [
We have seen how the concept of valence morphed from a hidden dimension of meaning into something that could be measured in text using just a few adjectives, and further into a subjective feeling, a property of the language or a property of the things that we talk about. It is clear that in practice, these are not unrelated. If I consider peace a good thing, I’ll probably feel good about the word and I’ll use positive words in talking about it. But conceptually they are distinct, and the extent to which they agree in practice is an empirical question. Another thing that we learned in this short review is that valence judgments vary by gender, age, personality and culture. Anyone who confidently writes aboutth‘e sentiment’ of a narrative work should be aware of that.
After this brief look at the history and the operationalisation of the notion of valence, we turn to a comparison of various tools that assign valence to Dutch words and texts. Except for the approaches already discussed, we also include two transformer-based language models. These tools by themselves have no notion of valence, but have been trained to fulfill classification tasks that assign short texts to evaluative categories.
Our interest here is not in finding the tool that best approximates the ‘true’ sentiment value, if such a thing exists, or the gold labels, which we don’t have. What we are interested in is whether these tools agree or disagree and what that says about the current state of sentiment analysis for narrative, at least for Dutch.
The tools that we use are: LiLaH the sentiment mapping of LiLaH [22], a manually corrected version of an automatic translation of the NRC emotion lexicon25[], see A.2.2.
LIWC a general-purpose tool for text analys5is][based on an underlying dictionary, seAe.2.3.
xlm a transformer-based multilingual mod1e]l b[ased on Twitter, seeA.2.5.
For a further description of the tools that we will use, as well as for the settings that we apply in computing the SentiArt valences, we refer to the AppendixA(.1).
For the dictionary-based tools (that includes the tools with curated dictionaries as well as SentiArt), we first compare the dictionaries. We report a number of measures: • dictionary size; • number of shared words; • word-level agreement of assigned values; • words where the dictionaries disagree.
After the dictionary comparison, we compare the result of the tools on a sample of Dutch novels. We create the sample as follows: from a collection of 10,921 recently published Dutch books we remove non-fiction and books with less than 5000 words, then select every fith book. For every selected book (n=2087) we select a random newline character, which usually corresponds with a paragraph start. Starting from that location we select the first hundred tokens.
Then, for each of the tools mentioned in the Appendix we compute the valence assigned to the fragment. For Moors, LiLaH and SentiArt, the valence of the text fragment is the average lemma valence for all words whose lemma occurs in the relevant lexicon. Words whose lemma does not occur in the resource are ignored. For the other tools, see the Appendix for the computational procedure.
We then compute correlations between the tools’ valence assignments. For some pairs of tools we also look at fragments where the two tools produce very diferent results.
In describing correlations, we use the labels proposed by Evan13s][: 0.00 - 0.19: very weak, 0.20 - 0.39: weak, 0.40 - 0.59: moderate, 0.60 - 0.79: strong, 0.80 - 1.00: very strong.
For simplicity’s sake, in this section we ignore LIWC15, as it did much worse than LIWC07 in the analysis of the fragment valences. 4.1.1. Overlap at dictionary level 4.1.2. Agreement at dictionary level We look first at the agreement between the continuously valued (SentiArt, Moors and Pattern) and the binary-valued tools (LIWC07 and LiLaH), given in tabl1e. These look as one would expect. In all cases there is a clear distinction between the mean values of the positive and the negative words in LiLaH and LIWC07. For LIWC07, the diference between the means is somewhat larger than for LILaH. LIWC07 seems to agree better with the continuous tools than LiLaH does.
Then we look at the correlations between the continuously valued valences (Ta2b)l.eThe correlations of Moors and Pattern with SentiArt are strong, the correlation of the two curated dictionaries Moors and Pattern is very strong. We should be aware, however, that these are correlations over a relatively small number of words.
Finally, for all dictionary pairs we also checked whether there are words where the dictionaries disagree, and if so, whether there is an obvious culprit. For LIWC07 and Moors and to a lesser extent Pattern we hardly found apparent mistakes. LiLaH contains a number of clear errors, maybe due to the limited availability of the Dutch translat2o2r, p[. 154]. In the SentiArt dictionary, there are countless misclassifications. See TablAe.5 for examples. We also note here that among the top positive words in the SentiArt valences, there appears a curious group of words related to hospitality, such as dinner, hostess, sommelier, catering, service and culinary, which also raises some questions about the adequacy of the procedure.
After computing the valences assigned to the fragments we computed their correlations (see Figure4). We see that the correlations between the results of the various sentiment analysis tools that we have looked at doesn’t get better than moderate (the only strong correlation is between the two LIWC flavours). As we don’t know the ‘true’ valences, we’re not in a position to say which is the best tool, but we can say something.
1. The correlation of the XLM transformer-based model is with the others is at best weak, for the Van Roy model there is no correlation. But we already knew that these tools were very diferent than the other tools, and it confirms (if confirmation were needed) that the type of training text is really important. 2. Of the other dictionaries, the agreement of Pattern with the rest is at best moderate but mostly weak. The reason is probably Pattern’s background in consumer review analysis. 3. From the two LIWC dictionaries, LIWC15 performs noticeably less than LIWc07. This is probably due to the Dutch LIWC15 being an automatic translation of the English dictionary. 4. The remaining curated dictionaries (LiLaH, LIWC07 and Moors) and the SentiArt approach have moderate correlations with each other.
For the tools Moors, SentiArt and LIWC07 we also did an analysis of text fragments that scored high on one tool and low on another. The main causes for diferences were apparent errors in the SentiArt word valence, homonymies, the word ‘niet’ (not) in Moors being assigned a low valence, and words not being present in the curated dictionaries. Some of those are unavoidable in the context of a dictionary approach, others could be avoided by better curation. Some of the options that we used for the SentiArt computations resulted from preliminary testing with these diferently-rated fragments. See the Appendix for details.
As the main results of the empirical part of this paper we see: 1. The SentiArt procedure to assign valence to large collections of words has serious limitations, even when computed on the basis of a domain-specific word embedding. 2. These limitations can to some extent be overcome by computing distances to a centroid vector rather than to the individual seed words, by only looking at the top and bottom quartile of the resulting valence distribution, and by excluding punctuation and function words (see Appendix for details). However, while leaving out punctuation and function words from the fragment valence computation helps, it is not a real solution to the problem that apparently the word embedding-based valence assignment is producing flawed results. 3. The agreement between other tools for computing valence of Dutch narrative text are never better than moderate.
With respect to the first two items in this list, this suggests that we may have to look beyond word2vec for a better answer to questions of semantic relatedness between lemmas (e.g. to contextual text representations as provided by Transformer-based language mod1e0l]s).[ The limited number of pretty arbitrary seed words seems another limitation of the SentiArt approach. A better way of obtaining valence ratings for many more words than can be manually curated might be machine learning with as target the Moors valences, and as features (among others) the word2vec distances to some of the top and bottom Moors words.
With respect to the last item, the question is: how bad is it that these tools only agree moderately? If we knew that one of the tools is mostly correct, it wouldn’t matter, we could just stop using the others. But we suspect that this is not the case. We have seen enough limitations in all of the tools and in the dictionary approach as such that it is unlikely that any of these tools presents us with more than a rough approximation of correctness.
That might lead us to asking why we have focussed here on dictionary-based approaches. In
their survey of sentiment analysis in literary studies, Kim and Kling2e0r] [wrote that ‘much
digital humanities research (especially dealing with text) uses the methods of text analysis that
were in fashion in computational linguistics twenty years ago’. And in a direct comparison,
current machine learning tools usually perform better in predicting human valence ratings
(see e.g. [
One answer to that question could come from Teodorescu and Mohamma3d7][, who show
that, in spite of instance-level inaccuracy, dictionary-based methods work very well for larger
bins of texts (e.g. for groups of 30 or a 100 tweets). They argue that ‘[f]or applications where
simple, interpretable, low-cost, and low-carbon-footprint systems are desired, the
lexiconbased systems [...] are often more suitable’. That suggests that dictionary-based methods might
be better suited to study sentiment at the chapter than at the sentence level of a novel. Another
answer comes from Öhman 2[
We wouldn’t go so far as as to say that what dictionary-based methods can do is not sentiment analysis. But it is true that most of the work in computational linguistics has been on the detection of sentiment in the sense of stance, where the aim is to detect the view that a text’s author expresses about some object. In narrative, and especially literary narrative, the aim of the text is not to convey the author’s view about the characters or the events, and if it were, it wouldn’t necessarily be the aim of researchers to uncover that view. This doesn’t mean that we don’t need to work on well-annotated corpora of narrative on which we can apply the tools of machine learning, far from that, but it does mean that current pre-trained sentiment analysis tools have been trained on corpora so diferent from the corpora that we are interested in that they may not be very relevant to the analysis of narrative.
Returning to the question of how much of a problem we have with these moderate correlations, and assuming, for the sake of argument, that the correlation of our tools with the ‘true’ valence is about equal to the best of their mutual correlations, that is .51, what is it can we do with a measurement that misses so much information? Maybe we could look at some patterns, very carefully. But it certainly would not make sense to use these measurements as ingredients in e.g. predictive modelling or the construction of narrative arcs.
We see some ways of moving forward: 1. creating a much larger dictionary than the present curated dictionaries for Dutch along the lines sketched earlier in this section; 2. using some sort of ensemble measure, in the hope that the tools can compensate for each other’s weaknesses; 3. only using sentiment analysis in a narrative context on larger text segments; 4. starting an annotation efort for valence in narrative fragments.
All of these, however, are only stop-gap measures for what we believe is the real problem,
which is that we have as a discipline not really defined the concept of valence in a narrative
context. As we saw in our overview of the history of the concept of word valence in section
2, many completely diferent definitions and operationalisations have have been proposed. It
has been possible to get away with these diferences in the analysis of by and large simple and
straightforward texts such as social media posts. In his survey of sentiment analysis in literary
studies [
This research was funded by the Netherlands eScience Center, grant number ASDI.2020.032.
M. M. Bradley and P. J. Lang.Afective Norms for English Words (ANEW): Instruction Manual and Afective Ratings . Tech. rep. Technical report C-1, the center for research in psychophysiology, 1999. [22]
The Moors approach is based on the Moors et al. articl2e6][ discussed in the text. The valences reported by Moors vary from 1 (lowest) to 7 (highest).
A.2.1. Pattern: A Subjectivity Lexicon for Dutch Adjectives We use the De Smedt and Daelemans subjectivity dictionary discussed in the tex8t][. For the dictionary comparison, if a word occurs in the dictionary multiple times (because of multiple word senses) we take the average valence. For the computation of the fragment valence, we do not use the dictionary directly, but apply the Pattern toolset to the unlemmatised text fragment. Pattern does not just look at individual words, but uses some aspects of the context, such as the presence of intensifying adverbs (‘awfully good’).
A.2.2. LiLaH: The LiLaH Emotion Lexicon of Croatian, Dutch and Slovene
The LiLaH dictionary 2[
A.2.3. LIWC: Linguistic Inquiry and Word Count
LIWC is a general-purpose tool for text analysis created by psychologist James Pennebaker
and colleagues. Its latest version is LIWC 20225][. Underlying the tool is an English-language
dictionary that assigns words to (multiple) categories, including categories for positive and
negative emotion. There exist Dutch translations for the 20074][and 2015 [
What LIWC reports is the relative frequency in a text of words in the various categories. We compute the relative frequencies using the LIWCTools python packag3e][. For computation of the fragment valence we subtract the relative frequency of negative emotion from the relative frequency of positive emotion.
A.2.4. SentiArt: a word-embedding based computation
For SentiArt, we use a word2vec-computed23[] word embedding. We used the gensim
package [
To compute or not compute the centroids (average value) for the positive words and the negative seed words before we compute and subtract the similarities.
To use a list of noun-only seed words or to include also corresponding adjectives (and one verb, where there is no corresponding adjective).
To use or not to use the lens method [18, p. 22], which excludes the second and third quartiles of the valence distribution from the computation.
To exclude or not punctuation characters.
To exclude or not function words. We use the function words as defined by Dutch LIWC 2007.
The texts that we used for the word-embedding are the full texts of 13,210 novels taken from a larger collection of 18,467 books in Dutch.
We use a number of diferent options in the SentiArt computation of the word and the fragment valences (see tableA.1). Table A.2 gives the seed words that we use for computing the SentiArt valence. The fact that we included among the SentiArt options the possibilities to exclude punctuation and/or function words is the result of preliminary testing. We saw in these tests that punctuation and function word valences had a sizable efect on the SentiArt-assigned fragment valences, even under the ‘lens’ condition. E.g. the comma, the full stop and the indefinite article ‘een’ (‘a’) all have a SentiArt valence in the upper quartile of the distribution.
After computing the four SentiArt valence dictionaries, we look at their distributions. We also list the 50 words with the highest and lowest valence, in order to check whether the computation makes sense.
A.2.5. Transformer-based models As mentioned, we use two transformer-based models on the fragments. One model is the cardifnlp/twitter-xlm-roberta-base-sentiment model1][. This is a multilingual model, trained on tweets. It does not assign a continuous valence value, but classifies a text as positive, negative or neutral2. The other model is robbert-v2-dutch-base-hebban-reviews5. It is a model trained on Dutch book reviews from the book discussion site Hebban, and aims to predict the rating associated with the review3. For both tools, the text types that they were trained on are very diferent from the book fragments that we will use them on. For this reason, we did not expect that they would agree with the other tools, but were willing to be surprised. Table A.2 Valence seed words
Condition
Positive Basic Extended tevredenheid (contentment) blijdschap (joy) genot (pleasure) trots (pride) opluchting (relief) voldoening (satisfaction) verrassing (surprise) Basic seed words & tevreden (contented) blij (glad) genieten (enjoy) trots (proud) opgelucht (relieved) voldaan (satisfied) verrast (surprised)
Negative walging (disgust) verlegenheid (shyness) angst (anxiety) verdriet (sadness) schaamte (shame) walgend (disgusted) verlegen (shy) angstig (anxious) verdrietig (sad) beschaamd (ashamed)
We computed four SentiArt valences, as described above. Here we used only the 17,306 lemmas that occur in the novel fragments that we will analyse.
As a first check of the results, we looked at the 50 words with highest or lowest valence for each of the computations. The words with the reportedly lowest valence are for all four computations indeed words that describe very unpleasant things. As an example, here are (English translations of) the 10 words with lowest valence for the centroid - nouns and adjectives condition: fear, distraught, anger, anxious, rage, shame, misunderstood, powerlessness, anger, confounded. For the words with highest valence, the picture is somewhat diferent. There are many words that no doubt represent a positive evaluation (excellent, fantastic, great), but there also appears a curious group of words that seem somehow related to hospitality, such as dinner, hostess, sommelier, catering, service and culinary; words that certainly are far removed from the positive seed words that went into the process.
Next we look at the distribution of the computed valences. FigurAe.1 shows that the centroid-based computations, and especially the one with nouns and adjectives as seed words, have a somewhat wider distribution. That seems an attractive property, as it provides stronger distinctive power to the valence assignment.
The correlations between the SentiArt valences are very strong (TabAl.e3). We see there is a 2 to 3 percent disagreement between the centroid and non-centroid versions, and a 6 to 7 percent disagreement between the nouns versus the nouns and adjectives seed words.
For each dictionary pair, we selected 10 words that get a high valence rating in one dictionary but a low rating in another. In none of the pairs, this created word lists where intuitively we would consider one of the dictionaries wrong, except for the pair centroid and nouns / noncentroid and nouns. Here, the words that were rated high in the centroid but low in the Figure A.1: Distribution of the four SentiArt valences. ‘ct’: centroid, ‘nct’: non-centroid, ‘n’: noun labels, ‘na’: noun and adjective labels. non-centroid condition were all obviously positive: humour, eagerness, liveliness, surrender, delight, self-assurance, cheerfulness, approval, passion, lust. This provides another argument in favour of the centroid-based computation.
As explained in the previous section, for SentiArt we have five binary options, and therefore 32 diferent results. In initial testing, it appeared that the computation without centroid consistently led to results with lower correlations to the other tools than the computation with centroid. We dropped the computation without centroid from further consideration. From the remaining sixteen SentiArt valences, the best correlation with the other tools was reached with the options lens, just the original (noun) labels, and not considering punctuation and stop words (see FigureA.2 for Pearson correlation with the other dictionary-based tools.). We continue with this SentiArt valence, which in the rest of the paper we just call SentiArt.
Table A.4 Overlap in words between the tools’ dictionaries. A limitation to take into account: in LIWC07, some terms include wildcards. The number of words that it covers are therefore larger than the number reported here. The computation of the overlap does not take into account the wildcards. tool 1
tool 2 sentiart sentiart fake sorrowful rigid boring disdain- mindful forgive kiss compassion ful sucky unattractive loved ode innocence m ad , n o e ab :’
p h l ‘ t
n ,s n . io o n t
c r , ex ie d
s
no fu fo it e