Sentiment Below the Surface: Omissive and Evocative Strategies in Literature and Beyond

Abstract

As they represent one of the most complex forms of expression, literary texts continue to challenge Sentiment Analysis (SA) tools, often developed for other domains. At the same time, SA is becoming an increasingly central method in literary analysis itself, which raises the question of what are the challenges inherent to literary SA. We address this question by probing units from a variety of literary fiction texts where humans and systems diverge in their valence scoring, seeking to relate such disagreements to semantic traits central to implicit sentiment evocation in literary theory. The contribution of this study is twofold. First, we present a corpus of valence-annotated fiction - English and Danish language literary texts from the 1th9 and 20th centuries - representing diferent genres. We then test whether sentences where humans and models disagree in sentiment annotation are characterized by specific semantic traits by looking at their distribution and correlation across four diferent corpora. We find that items where humans detected significant sentiment, but where models did not, consistently employ lower levels oafrousal, dominance and interoception, and higher levels ocfoncreteness. Furthermore, we ifnd that the amount of human-model disagreement correlated with semantic aspects is linked to the interiority-exteriority continuum more than with direct sensory information. Finally, we show that this interaction of features linked to implicit sentiment varies across textual domains. Our findings conifrm that sentiment evocation exploits a more diverse and subtle set of semantic channels than those observed through simple sentiment analysis.

Keywords

• eol>sentiment expression
• literary language
• implicitness
• objective correlative
• sentiment analysis

1. Introduction

Sentiment Analysis (SA) is an increasingly central method for computational literary research [ 55 ], an especially popular application being that of gauging the ‘sentiment arcs’ of novels, i.e., the ‘shapes of stories’ [ 38, 54, 35 ]. Still, the relation between valence extracted with SA tools and the human perception of literary texts at a granular level remains an open question, as tools applied to the literary domain are primarily geared towards processing nonliterary texts.

While some recent work has examined the adequacy of available SA tools for literary analysis [ 27, 11, 56 ], the question of how to validate them – against whose judgements, and at what level (i.e., the story-level vs. sentence-level, etc.) – is a persistent concern, also due to the relative scarcity of annotated resources in the literary domain. Moreover, the observed inadequacy of SA tools for literature has raised the question of what the diference of literary texts might be in comparison to the nonliterary10[], where tools seem to perform comparably bette6r3][. How does the “literary” difer in its way of communicating sentiments to readers?

In fact, due to their textual complexity, literary texts are often said to difer from morcoemmunicative texts [ 34 ]: they are efective at multiple narrative levels6[0, 12]; creatively divergent from standard language use4[ 7, 5 ]; reliant on poetic devices1[ 6 ]; and ambiguity, efecting contesting interpretations5[ 8 ]. Moreover, literary language may exhibit various strategies for conveying emotion beyond simply using words directly associated with emotional states (e.g., “sad”). While language on, e.g., social media, may also rely on omission and subtlety, literary theorists have frequently claimed that literariness or pthoetic function of language excels in this regard and is distinct from its more directly communicative funct1ioRne.garding sentiment expression, it has recently been suggested that literary prose relies on specific semantic traits connected to afective understatement and concreteness toevoke – rather than “communicate” – sentiment in readers 1[0].2 These features are especially interesting since they are related to the seminal concept of the “objective correlative” in literary theory, which suggests that literary texts efectively convey emotions by grounding them in – concrete and objective, rather than emotional and subjective – entities and situation2s6[], while avoiding abstract and emotional language3. In this study, we further pursue the hypothesis that literary texts rely on omission and materiality, i.e., features connected to the concept of the objective correlative, to evoke sentiments, rather than simply mention them.

While divergences between human and model judgments in sentiment analysis are generally taken to indicate shortcomings in SA tools, they present an interesting case for testing the diference of literary language in expressing sentiment, assuming that SA tools are generally tuned toward more explicit forms of communication due to their development on e.g. social media – also suggested by more general studies of implicitne7s5s,[ 40 ]. Using a single novel as their data, Bizzoni and Feldkamp1[0] explored this discrepancy, finding that certain semantic elements – levels ofarousal, dominance and concreteness – are indicative of human/model disagreement. To gain further insight into sentiment expression in literary texts, we go one step further: we test the distinguishing power of these features in a much larger corpus of annotated literary texts. We also extend the list of features by an additional foiumra:geability, visuality, hapticity, and interoception, in order to further examine the connection of human/model disagreement with features that relate to the concept of the objective correlative and of omissive strategies in literary sentiment expressio4n.

We conduct two experiments. First, we test whether sentences where models and humans agree can be distinguished from sentences where they disagree based on these specific semantic traits. Secondly, we further explore the robustness of the relation between features and sentiment evocation by testing the correlation of these features to the absolute diference in 1Jakobson distinguished the “poetic function” of language from its “emotive or expressive function”, which “aims a direct expression of the speaker’s attitude toward what he is speaking about3”4[, p. 66] 2Again, these phenomena naturally extend outside the literary doma5in7][: tweets using irony or figurative language, e.g., likely efect diverging reader interpretations61[ , 67 ]. 3Besides T.S. Eliot, who coined the term, other famous proponents of this view are the Imagis5t2s][. 4All data and code for the present study is availabhleere. sentiment scoring between humans and models.

2. Related works

2.1. Literary sentiment analysis

While SA tools perform increasingly well in some doma2in,s63[], studies have pointed out the cross-domain drop in performance, as well as the lagging behind of tools for under-resourced languages [ 28, 49, 14 ]. Still, some studies have suggested that Transformer-based models might be able to bridge the gap and perform better on literary or poetic mater6i5a]l. [Assessing the performance of models on historical Danish and Norwegian literary texts, Allaith, Degn, Conroy, Pedersen, Bjerring-Hansen, and Hershcovich3][found that multilingual Transformers outperformed both fine-tuned models and classifiers based on lexical resources in the target language, which aligns with the findings of Schmidt, Dennerlein, and Wolf [ 65 ] and Schmidt and Burghardt 6[ 4 ] for historical German drama. Still, the disparity between human and model sentiment judgements in literary texts continues to be observed11[ , 56 ]. The disparity is often related to the efects of narrative, annotators generally having access to the narrative context of a sentence, rather than to difering strategies in sentiment expression across domains.

2.2. Literary sentiment expression

The concept of “implicit” expression is particularly relevant, and complex, in literary writing. Several theories of writing point to the importance of avoiding concepts or ideas (however this is intended) in a too explicit way. The widely known precept of “Show Don’t Tell” points at least partly in this direction12[]. Moreover, critics continually rely on terms like emotional “evocativeness” and “understatement” to describe writing style6s9[ , 24 ]. Literary and afect theory has also more recently foregrounded the use of materiality and sensuousness in literature – including poetry – to evoke afective reactions in readers, emphasizing the way objects and things are culturally invested with meaning and afect1[ , 17 ] and thus utilized by authors to evoke embodied, aective experiences [ 51 ].5 Despite the significance of implicit, evocative, and expressive strategies, there is little consensus on how to reliably track these in literature and whether such types of expression have recognizable linguistic markers.

The association of materiality to literary evocation is not new, and closely relates to the Modernists’ and New Critics’ valuation ocfoncreteness overabstraction [ 69 ], as well as to the idea of the ‘objective correlative’ which T.S. Elio2t6[] proposed in 1948. Eliot suggested that the effective way of expressing emotion in literature is ‘by proxy’, through an external and objective – in the sense of intersubjectively recognizeable – “set of objects, a situation, a chain of events which shall be the formula of [a] particular emotion26”][. The concept of the objective correlative suggests that literary language efectively evokes sentiments in readers by being both 5In narratology, this is close to what Fludernik has termed narrative “experential2i9ty].” B[urroway 2[0] explicitly notes that when using of nouns that evoke sense images and verbs that represent visualizeable actions “the writing comes alive”. more omissive (relying less on directly emotion-associated wor6dasn)d more concrete (relying on objects and situations).

This hypothesis has been supported by computational literary studies. Auracher and Bosch [ 6 ] found that the concreteness of literary language impacts the emotional engagement of readers and their experience of suspense, and Bizzoni and Feldkamp10[] tracked the “omissive” writing of Ernest Hemingway by looking at the amount and intensity of sentiment expressions detectable in sentences, compared to how “expressive” (in terms of sentiment) readers perceive these sentences to be. Comparing sentences where humans and models agree vs. those where they disagree, they found that arousal and dominance levels (of the NRC-VAD lexico45n])[ were indicative of omissive strategies of evocation. Moreover, the level of concreteness of the language used appeared higher in sentences with higher disagreement between humans’ and models’ valence attribution.

2.3. Semantic traits of literariness

In computational linguistics, the use of semantic traits – often derived from psycholinguistic norms – for the study of narrative is relatively freque7n0t, [ 36, 43 ]. Yet, their use to model poetic literary strategies of sentiment evocation is still relatively pioneering. Kao and Jurafsky [ 36 ] applied semantic measures to poetry: imageability to gauge “imagery”and concreteness to gauge “concrete imagery”, using the concreteness ratings of Brysbaert, Warriner, and Kuperman [18], along with objective/abstract word categories68[], as well as psycholinguistic norms [ 50 ] to model “emotional language”. They show that these features difer between “amateur” and appraised poets, where appraised poets use more concrete and imageable language and less emotional words. Conversely, Maslej, Mar, and Kuperm4a3n][find that abstraction and arousal correlate positively with the perceived interest of readers in fictional characters, which is perhaps related to the general tendency of abstract concepts to be more emotionally valenced than concrete ones3[ 9 ]. Ullrich, Aryani, Kraxenberger, Jacobs, and Conra7d0][observe how the diferences between perceived afect in poem annotations can be largely explained by lexical psycholinguistic norms, with sentiment dimensions like arousal being one of the best predictors of the perceived afective meaning of the poems. The studies show above all that imageability, concreteness and arousal strongly relate to how readers feel about literary texts. 6The advice against “sentimentalism” and abstraction in literary language is present in Eliot, though more prominent in e.g., Ezra Pounds’ literary criticism52[]. EmoBank

Letters Blog Newspaper Essays Fiction

Travel-guides Fiction4

Hymns Fairy tales Prose

Poetry

3. Data

3.1. Datasets

To probe textual features of implicit sentiment, we created a diverse corpus of fiction spanning four genres, manually annotated for valencFeic(tion4). For the second experiment of this paper – comparing more or less literary genres – we also selected other datasets to represent a diversity of genres in the literary and non-literary domains which had all been annotated for valence on a continuous scale7. As such, we include social media, journalism, and genres with various degrees of literariness (essays, letters, travel-writing, fairytales), which may also be thought to represent degrees of colloquialism (from blogs to journal8ism).

Fiction4: The new dataset presented in this study, with human annotations for valence=( 6, 300).9 It includes four diferent genres – fairy tales, hymns, prose and poetry – over theth19 and 20th century, in one high- and one low-resource language (English and Danish). The corpus was compiled with an aim toward diversity (in genre, time and language) while still aiming to include well-known and culturally significant works by both male and female auth1o0rWs. e also took into account both i) the texts’ cultural significance and ii) their narrative and poetic complexity, which may represent a particular challenge to SA tools. The authors selected were 7We have adopted a very broad understanding of genre for this paper, encompassing more or less literary genres. 8We standardized the varying valence scales of chosen corpora to a scale from -1 to 1 (negative to positive). 9For an overview table of this corpus, see Appendix. 10Note, however, that the corpus is highly skewed towards male writers, not least because of the time-period and genres covered (i.e., hymns). For a detailed overview of the corpus, see Ta7blien Appendix. Ernest Hemingway, Sylvia Plath, H.C. Andersen and the various authors of ofÏcial hymn-books (for details, see Appendix, Tabl7e).

i) Regarding their cultural significance, for the English texts, Hemingway is perhaps one of the most famous 20th century authors for his prose, and his texts are read in educat1i1onand among the public1.2 Plath, similarly, is a widely read and acclaimed poet, perhaps the best known female American poet of the 2t0h century.13

For the Danish texts, Andersen’s production is arguably the most central in Danish literary heritage [ 59 ]. While being less known internationally, the ofÏcial hymnal book is the most widely distributed “poetry book” in Denmark62[],14 used in the Danish education system at all levels, and shapes national cultural identi9t]y. [

ii) Regarding the complexity of the corpus for the sentiment annotation task, we ensured variance across genre, place and time, but also emphasized, from a literary perspective, the level of literary complexity, including texts that could be considered either particularly challenging or particularly simple. We consider HemingwaTyh’se Old Man and the Sea and Plath’s Ariel as two ideallydifÏcult cases for testing SA tools. Hemingway is known for an especially “omissive” writing style, direct and limited in its use of figurative languag3e1[], while relying on implication rather than “overt emotional display”, leaving much inference up to the reader [ 69 ]. Hemingway’s The Old Man and the Sea (1952) has been considered emblematic for this minimalist style, which may omit characteristics that models rely on in sentiment scoring. Plath’s poetry collectionAriel (1965) is complex in a slightly diferent way1.5 The so-called confessional poetry genre, of which Plath is considered emblematic, foregrounds idiosyncratic personal psychology and experiences against the “emotional vacuity of public language” and universal symbols4[ 6 ]. In Ariel Plath writes on complex and political themes in idiosyncratic style consisting of “hallucinatory images” and novel metaphor15s][, and the work has been used as a case of literature posing particular difÏculty to most reader2s5[].

Conversely, we consider Andersen and religious hymns two ideaslimlyple cases for testing SA tools. Andersen’s fairy tal1e6s are characterized by an essential simplicity, both stylistically an in their narrative progression41[ , 4 ] and their ability to engage both children and adult readers [ 41 ]. Religious hymns17 are characterized by their limited number of themes (e.g. worship, thanksgiving, etc.), which are are expressed through well-known and formalized (as well as recurring) phrases, metaphors, figurative and symbolic language48[]. The hymns’ repetitive and predictable use of language may make them more accessible to models, even though their archaic and nuanced style may present challenges.

After collecting the texts, we found that some of these simplicity/complexity assumptions are 11The Old Man and the Sea being studied in schools across the worl4d4[]. 12As of today,The Old Man and the Sea has over1 million ratings on GoodRead.s 13Plath’s prose workThe Bell Jar has around 1 million rations on GoodReads, and her poems appear among the top 250 assigned works onEnglish Literature college syll.abi 14Note that the Danish term used “lyrik” encompasses poetry and songs. 15Fiction4 includes all 40 poems inAriel. 16Fiction4 includes three of Andersen’s most known fairy tales: “The Little Mermaid” (1837), “The Ugly Duckling” (1844), and “The Shadow” (1847), in an edition where spelling has been slightly moderniz2e1d].[ 17Fiction4 include 65 hymns from three diferent ofÏcial hymnal books from the years 1798 (= 35 ), 1857 ( = 17 ), and 1873 ( = 13 ). Years refer to publication years of three ofÏcial church hymn collections, and hymns are collected at random. reflected in the correlation between human and model valence scores – at least when using our method – where Plath’s poetry shows the lowest correlation, and hymns the highest correlation (Table1).

Reference corpora and datasets: EmoBank The EmoBank is a multigenre corpus with human annotations for valenc e=( 8, 735),18 with 10 annotators per sentence 1[ 9 ].19 The corpus was composed from various categories in theManually Annotated Sub-Corpus of the American National Corpus (MASC),20 consisting of texts from 1990 and onwards3[ 3 ]. We consider theEmoBank categories: Letters, Blog, Newspaper, Essays, Fiction, and Travel guides, which are relatively balanced (Ta1b)le including both longer and shorter texts within each catego2r1y.

FB The Facebook corpus of posts (FB2)2 collected between 2009 and 2011, consists of 2,895 status updates, each by a unique user, with human annotations for valence and arousal, with 2 annotators per post 5[ 3 ]. The FB dataset difers from our other corpora, consisting of posts – not sentences. While some posts are short (e.g., “:)” and “LOL”), the average length of posts is comparable to the average sentence length in, e.gE.,moBank (Table1).

Beyond these corpora, we also include two datasets without valence annotation for comparison in terms of feature levels: Image-captions ( = 3, 334, 173 ) of the Conceptual Captions dataset 6[ 6 ], of which we consider feature values to represent a “high-water mark” (i.e. high level) of language relying on object description and visualit2y3.

Participants free emotion event descriptions ( = 6, 898 )24 of the International Survey on Emotion Antecedents and Reactions (ISEAR) 7[ 1 ],25 of which we consider feature values to represent a “high-water mark” of language dealing with interiority (i.e., relating to inside sensations and self) and emotionality. 18We exclude the heterogeneous SemEval category, as well as very short strings of sentences (noise) across the categories (length< 2). 195 annotators annotated each sentence for valence from a “reader” and “writer” perspective, i.e., 10 valence annotations per sentence. The valence scores represent a weighted average. See thdeocumentation. 20Which is in turn a subset ofthe American National Corpu.s 21The category ‘essays’, for example, comprises 8 texts, including the essay “A Brief History of Steel in Northeastern Ohio” or one on discrimination. ‘Fiction’ comprises 6 works of various genres, e.g., Richard HardiAngW’sa“sted Day” and the SciFi story “Captured Moments”. Newspapers include various short reports (e.g. “A.L. Williams Corp. was merged into Primerica Corp.” etc.) and longer reportages. Note that Travel Guides are generally written in a running prose, and includes both place-histories (e.g. “A brief history of Jerusalem”) and current-day reflections (e.g. “Dublin and the Dubliners”). See the full MASC corpuhsere. 22https://github.com/wwbp/additional_data_sets/ 23https://github.com/google-research-datasets/conceptual-captions 24To exclude noise and non-answers (e.g., “I cannot remember”), we set an arbitrary threshold of 30 tokens for a description to be included, resulting an a diminished dataset from the original 7,659 datapoints. 25https://github.com/sinmaniphel/py_isear_dataset/

4. Methods

4.1. Human and automatic sentiment annotation

Model annotation Multilingual transformer-based models have shown best performance in SA for literary texts across languages (also for historical te1x1t,s3)[ , 65 ] compared to dictionarybased approaches explicitly developed for literary texts as well as monolingual English models [ 11 ]. We, therefore, used the RoBERTa base xlm multilingual, finetuned for sentiment analysis on Twitter data2,6 which is comparable to the state-of-the-art models in a monolingual (nonliterary) setting 7[], and shows the best performance in the limited studies there are on literary prose [ 11 ].27 XML-RoBERTa28 was developed through a cross-lingual language training method, designed to boost its proficiency in comprehending and processing multiple languages by transferring skills it has acquired from one language to another.

With this model, we scored all sentences across our bilingual data2s9etTs.he model returns polaritypositive ornegative, and a neutral label. To attain more continuous, nuanced data from the transformers’ categorical output, we opted for using the same strategy as in Bizzoni and Feldkamp [ 11 ], i.e., using the confidence score of model labels as a proxy for sentiment intensity. For example, a sentence with apositive label and a confidence of, e.g., 0.75, is interpreted as a valence score of +0.75. Similarly,naegative label with confidence of 0.89, is interpreted as a valence score of -0.89. For the neutral category, confidence is disregarded and levelled to a score of 0 (midscale or “neutral”3).0 The correlation between human mean score and the transformed RoBERTa score appears high across our selected corpora (T1ab).le

As seen in Table1, RoBERTa values correlate most strongly with human annotations of the FB dataset, while correlations with annotations of fictionE(moBank and Fiction4) are much lower, possibly reflecting better development the model for certain more colloquial domains (social media, blogs, letters). 26We used this model of-the-shelf, so that the hyperparameters are as reported inh:ttps://huggingface.co/cardiff nlp/twitter-xlm-roberta-base-sentiment/blob/main/config.js.on 27Note that recent studies have tested newer, generative models for literary SA, notably Rebora, Lehmann, Heumann, Ding, and Lauer [ 56 ]. For this study, we excluded GPTs from our pool of tools. As our interest is not achieving top performance, but rather understanding the diferences between SA tools and human annotation, we sought to employ only models that were designed for sentiment analysis and that don’t depend on prompt engineering. 28https://huggingface.co/docs/transformers/en/model_doc/xlm-roberta 29For the Danish texts, we tried the model both on the original Danish, on non-validated Google translations, and on manually checked and revised Google translations. We chose to use the model’s output on validated translations, since the those valence scores correlated best with human annotations – the correlation with human mean values when applying the model on hymns and fairy tales in the original Danish w a>s .45, < .01 , on Googletranslations > .61, < .01 , and on validated English translatio n>s .65, < .01 . 30Naturally, there are caveats to transforming sentiment polarity to continuous valence scores in this way. However, the approach has been shown to outperform dictionary-based (outputting continuous scores by design) and to approximate a human continuous valence annotation in literary pro1s1e].[ Note that the distribution of transformed scores still tend to “look polar” as confidence score tend to be generally high, see6F,iAgp.pendix.

Human annotation of Fiction4 Human annotators (at least= 2 /line) read the literary texts from beginning to end, scoring each line on a 0 to 10 valence sca3l1e:0 signifying the lowest, and ten the highest valence3.2

The valence score was intended to represent the sentiment the sentence and verse expressed, and annotators were instructed to avoid rating how a sentence or verse made them feel and to try to report only on the sentiments embedded in the sentence, i.e., to think about the valence of the individual sentence and verse, without overthinking the story’s/poetry’s narrative.

It is worth noting that humans rarely reach an agreement higher than 80% (or 0.80 Krippendorf’s ) for tasks like positive/neutral/negative discrete taggin7g4[] on nonliterary texts – and have lower agreement for continuous scale polarity annotati8o]n, e[specially for literary texts [ 56 ].

4.2. Sentence subsets

For our first experiment, exploring the prevalence of semantic traits in sentences where humans/model disagree and sentences where they agree, we divided oFuirction4 corpus into two groups. First, we filtered out sentences which humans did not perceive any strong sentiment (i.e., with human valence scores between 4.5 and 5.5 on the 0-10 scale). On one hand, we then took sentences in which our chosen model did not assign any strong sentiment (below an absolute score of 0.1, i.e., between -0.1 and +0.13)3 and, on the other hand, sentences where it did. With this procedure, we distilled two groups of sentences (F1i)g: . one of sentences with humans/model disagreement, which we call the “implicit” gro u=p (1, 194 ) and one of human/model agreement, which we call the“explicit” grou p= (2, 631 ).

4.3. Features

Based on previous work (section2), we include three previously used semantic traits to examine their bearing on instances of implicit sentiment evocation: at the sentiment dimension, 31“Lines” refer to sentences in the case of prose and to verse-lines in the case of the hymns/poetry. Sentences were tokenized using thenltk tokenize package. 32Annotators were researchers, three with a background in literary studies and one in cognitive science. The two annotators of the hymns (MA and PhD of literature) had domain knowledge inth1c9entury Scandinavian literature and historical religious hymns. 33Note that the model valences range from -1 to 1 (negative to positive), where 0 represents neutral. arousal3,4 and dominance,35 and at the sensorimotor dimension, concretenes3s6, imageability, as well three additional sensory traits, visual3i7tyh,aptic,38 and interoception3.9 We use the datasets below to measure sentence semantic trait values, averaging the score per feature for each sentence.

Concreteness lexicon: The lexicon by Brysbaert, Warriner, and Kuperma1n8[] provides concreteness ratings for 37,058 English words. Annotators were recruited via Mturk (English native speakers). Each word was annotated by at least 25 annotators, on a scale from 1 (=most abstract, i.e., what cannot be experienced directly but the meaning of which is defined by other words) to 5 (=most concrete, i.e., what can be experience directly through one of the five senses). These ratings have been widely used2[ 2 ] also in the literary domain6][.

NRC-VAD lexicon The lexicon by Mohammad 4[ 5 ] provides ratings of 20,000 English words on three sentiment dimensions (valence, arousal, dominance). Annotators were recruited via CrowdFlower, and each word was annotated by at least 6 annotators with a best/worst scaling approach (e.g. most arousal vs. least arousal). The lexicon has been used widely, as well as integrated in the SA tool VADER3[ 2 ].

The Lancaster Sensorimotor Norms: The dataset provides norms of sensorimotor strength for 39,707 English words across 6 perceptual modalities (haptic, auditory, olfactory, gustatory, visual, and interoceptive)4[ 2 ]. While the dataset includes action efectors (i.e., body parts) we used only the general perceptual norms, selecting only those we deemed especially relevant to the idea of objective correlative (i.e., material, visual objects/situations): visuality, interoception and hapticity. The perceptual part of the dataset had 2,625 annotators recruited via Mturk. Each word was rated from 0 (=not experienced with sense X) to 5 (=experienced greatly with sense X). These perceptual modality ratings have been used in, e.g., metaphor detectio7n2][, and have served as a form of “embodied experience” information to enrich and improve language models [ 37 ].

Imageability The MRC Psycholinguistic Database (MRCPD) provides 26 linguistic and psycholinguistic variables for 150,837 English words – a subset of which are 9,240 words words rated for imageability2[ 3 ]. These words have been rated by annotators. Ratings reflecthow easily a word can evoke mental imagery, and fall in the range100 − 700. The lexicon has been used variously, e.g., in metaphor3[0] and literary studies3[ 6 ]. 34The degree to which a word prepares for action, captures or focuses attenti1o3n].[ 35The degree of control evoked7[ 3 ]. 36The degree to which a word denotes a perceptible entity18[]. 37The degree to which a word is experienced with the eyes42[]. 38The degree to which a word is experienced by touch42[]. 39The degree to which a word is experienced by sensations inside the bod4y2[].

5. Results

5.1. Human annotation

We report a relatively high inter-rater reliability (IRR): between annotators, we find a mean correlation (Spearman’s ) from 0.59 for poetry to 0.73 for hymns (Tabl2e).40 IRR is high, especially for hymns, considering both the fragmentariness of the verses, and that humans tend to have low agreement for continuous-scale annotation (Secti4o.n1).

5.2. Experiment 1

For our first experiment we compared the two groups of sentences iFniction4 along each of the chosen features. We report the Mann-Whitney U-test efect’s size and significance levels in Table3.41 We find that the strongest efect size is for interoceptive values (Table 3), while visually, concreteness shows two noteable “peaks” (Fig2)..42 Overall, we can confirm the diference 40As annotators operated within a continuous valence spectrum, divided into ten categories, we find that a correlation measure more clearly reflects direction and nuance of annotations (parallelity vs exactness), compared to categorical IRR measures. Therefore, we report Spearman’asnd provide Krippendorf’s for reference (the level of measurement is considereidnterval). 41For the test, we dropped sentences with NaN-values in the specific feature we were testing, the number of dropouts was < 40 in each test. 42The results of the Mann-Whitney U test are supported by a linear regression, where we sought to model the two groups by each feature. Significant results of the linear regression correspond to those indicated by the MannWhitney U-test, see Table8 in Appendix. between groups for the three features previously teste1d0][, while adding the observation of slight diferences also for language heavy in visual and haptic information, as well as a robust diference for interoceptive information in the implicit group.

For reference, we conducted the same experiment on our reference corpEomraoBank and FB, dividing the data into implicit and explicit groups of sentences as outlined in Sect4io.2n. These results are reported in tabl4e. Histograms to support this diference in feature values between groups inEmoBank can be found in Appendix, Fig.7 – as in Fiction4, levels of arousal, dominance and interoceptive are lower in the implicit group, while concreteness is higher in the implicit group. In the reference corpora (tab4l)e, the strongest efect size is tendentially, as in Fiction4, interoceptive values. Notably, interoceptive and arousal values hold significant discriminating power across all corpora, as well as concreteness if we disregarFdBtchoerpus. We see important diferences within the EmoBank, where all features appear important only in the more personal or imaginative genres, and not in newspaper and essays.

5.3. Experiment 2

In the second experiment, we check for correlations (rather than simple statistical diference) between human/model disagreement and the level of each semantic trait per sentence. This allows us to observe whether the presence of “undetected sentiment” in text has a linear relation with any of the semantic dimensions selected. For thisw, e used only sentences found in the implicit group (as outlined in Section4.2), so that we correlated the amount of disagreement between human and model with our chosen features in sentences where humans perceived

Concret. Arousal Dominance Imageab. Visual Haptic Interocept. sentiment, but models did not. We report our results in Ta5b.le

First of all, not all patterns of sentiment implicitness as seen in Experiment 1 are detectable as a correlation, suggesting that some of these features do not impact sentiment evocation linearly. On the other hand, we do see correlations that point to interesting genre diferences in how sentiment is perceived in texts. ForFiction4, we find a consistent negative correlation between human/model disagreement and arousal which aligns with the lower levels of arousal in the implicit group we saw in Experiment 1. While concreteness and interoception do not show consistent linear correlations with disagreement, efects of low interoception related to higher disagreement are evident in Andersen’s fairy tale4s3.

Within Fiction4, the role of high concreteness paired with higher dominance, and, to a lesser extent, lower arousal in sentiment disagreement (which we link to evocation) is confirmed, as well as the negative correlation of disagreement with interoception.

For comparison, we redid correlations in the reference corpora detailed in Se3c.t1i.oHnere, positive correlations are also found with concreteness: the more concrete a sentence is, the more our SA model’s sentiment judgment will difer from that of human’s. The strongest role of concreteness in sentiment disagreement appears to be not in literary texts proper, but in the travel guides and letters contained iEnmoBank, and in blogs. Interoception also holds a negative correlation with disagreement Fiction categoryEinmoBank, as it did with Fairy tales in Fiction4.

Interestingly, the negative correlation of arousal with disagreement is not as consistent in the reference corpora, where we only see a negative correlation in Blogs. We find spurious positive correlations of disagreement with dominance, and visual – notably, however, correla43The absence of a linear relation with concreteness is particularly interesting, given the results in Experiment 1.

Concreteness appears to have an efect on the evoked sentiment for human readers, but the two elements are not systematically related - the evoked sentiment does not change linearly with an increase in concreteness. FB EmoBank 2.65 ± 0.42

Letters 2.68 ± 0.40 Blog 2.61 ± 0.45 Newspaper 2.62 ± 0.32 Essays 2.49 ± 0.33 Fiction 2.69 ± 0.47 Travelguides 2.81 ± 0.43 2.72 ± 0.46 2.58 ± 0.43 2.70 ± 0.37 2.72 ± 0.36 2.90 ± 0.56 tions where they appear tend to have the same negative or positive direction across all corpora (includingFiction4), with the exception of imageability. Facebook posts inFB seem to have no significant link to many of these channels.

5.4. Genre diferences

While most datasets seem to exploit some form of trade-of between concreteness, on one side, and arousal, dominance and interoceptive on the other, relatively few show correlations with the visual and haptic semantic information, as well as with imageability. The exceptions are blogs inEmoBank, which shows a weak but positive correlation with visual and with human/model disagreements; and travel guides where disagreement has a correlation with haptic and imageability. FB and the EmoBank newspaper and letter category return non-significant correlations with most dimensions, with the exception of dominance fFoBr.

Genre diferences in the overall use of these semantic traits can be observed in Tab6l.eNote that, for example, Facebook posts seem to have high values of imageability. Still, at the same time, imageability in posts displays no correlation to the absolute disagreement between model and human (Table5). In other words, it may be that although the language of posts is highly imageable, the images are not used in a way that subtly evokes human emotion and that challenges models as much as it happens in, for example, travel guides. Similarly, literary genres (in EmoBank and Fiction4) seem to have high values for imageability and visual scores (Tab6l),e but these dimensions exhibit little correlation with human/model disagreement.

5.5. Relation between features

Through all our datasets, concreteness has a positive relation with disagreement – showing higher levels where models are unable to capture the sentiment that humans perceive – as much as interoception has a negative one. In general, the opposite strength of concreteness and interoception in all of our datasets appears to confirm our intuition that interoception works as a sort of anti-concreteness when it comes to the evocation of sentiments, as the usage of external objects and “things” to evoke sentiments in the reader will make a low recourse to the interoceptive dimension. The fact that visual, haptic and imageablity correlations, when relevant, tend to the same direction as concreteness also adds to this hypothesis.

It is intriguing that a positive correlation between human/model disagreement and concreteness tend to co-occur with a positive correlation with sensory norms. The intuition that what is concrete as something “that is perceived through the senses” or “that can be drawn” would have led us to expect correlations of, e.g., haptic and concreteness to co-occur. On the other hand, concreteness does not have to occur with explicit sensory information at all: many words likehouse, sea, orwood, do not peak on one specific sense, and yet are considered fairly concrete; and some words likemelody or rhythm might be less concrete and yet have sensory associations. The kind of concreteness that matters here might be more related to a general physical materiality rather than to a specific sensory load.

Concreteness exhibits a stronger correlation with visual, haptic, interoceptive and imageability traits (> .5, < .01 ) than with dominance and arousal (around= .2, < .01 ). When correlating terms in the concreteness dictionary with other semantic traits, we find that especially interoceptive and concreteness show an interesting correlation. Words referring to internal emotional states, also presumbaly having higher arousal (e.g., “lovesickness”, “hopelessness”), tend to cluster in the direction of high interoception and low concreteness. While concreteness has a robust negative correlation with interocepti o=n (−.52, < .01 ), i.e., high concreteness words generally are less interoceptive, we find that there is a set of words which maintain high interoception and high concreteness (upper right in Fi4g). These words may be characterized as referring to concrete objects, which are nevertheless associated with internal (vs. external) states and experiences (e.g. “bladder”, “breath”). Conversely, words in the lower right corner, with high concreteness and low interoceptive values, appear to be more of the category objects of external experience (e.g. “jewel”, “clip”, “lightswitch”), less associated with internal sensation (than is, e.g., “bee sting” or “drugs”). Considering the opposite correlation of human/model disagreement with interoception vs concreteness, we hypothesize that words used in instances of “objective correlative” would predominantly appear in the lower right corner of Fig.4). This means that they are associated with words that are not only more concrete but also “objective” in the sense of referring to external rather than subjective or internal experiences. Therefore, the “objective correlative” might be understood as “objective” in both senses: impersonal and focused on external objects.

6. Discussion and conclusion

We have examined the relation between human/model disagreement on sentiment annotations and a chosen set of semantic traits for a new corpus of literary prose, comparing them with datasets representing several other domains, and we have extended the semantic traits’ set used in previous literature to include also the sensory and interoceptive dimensions. Overall, we confirm previous results obtained on smaller data about relation of semantic traits to the presence of “undetected” or implicit sentiment in literary fiction, and we have observed similar trends also in non-fictional domains, with interesting diferences between genres.

The “undetected” sentiments are likely to be evoked, rather than stated, and this evocation seems to pass through a trade-of between several semantic traits: an increase in concreteness and a decrease in arousal and interoceptio4n4. These traits seem to align with what in literary theory has been called “objective correlative”: the strategy of conveying sentiment (or emotion) through the reference of external, material, or “objective” reality. This seems to happen together with the downplaying of semantic dimensions related to intensity and control, contributing to a subtler, less explicit form of emotional communication, which we might characterize as an omissive evocative strategy. An example of the trade-of between omission and use of objective correlative can be observed in the following sentence of Hemingway from Fiction4: ``Ay, he said aloud. There is no translation for this word and perhaps it is just a noise such as a man might make, involuntarily, feeling the nail go through his hands and into the wood'' . The sentence was consistently rated as negative by humans, and neutral by the model (see Fig5. above). Note how concreteness and interoception tend to divert in this sentence (e.g. on “feeling” or “nail”), while arousal and dominance values are sparse (Fig5.).

In the future, we intend to expand our analysis to larger and more diverse corpora, and integrate more psycholinguistic resources, seeking ultimately to contribute to the development of better tools for sentiment analysis in literary genres. We would also like to observe the relation between reader response or literary reception and the concreteness-dominance or concreteness-interoception trade-of.

Limitations

We want to underline that our corpus of fictionF(iction4) is limited, with only one author representing three of the four categories (Plath for Poetry, Hemingway for prose, and Andersen for fairy tales). Moreover, the demographic of our dataset is reduced (in terms of gender, ethnicity, age, social class, etc.). Replication of these results on a larger and more diverse corpus of fiction is needed, and our results should be interpreted with this in mind. 44Perhaps surprisingly, the concreteness’ efect did not have a strong link with existing norms of sensory information, but only with interoception, with the partial exception of literary prose.

Online Resources

See https://github.com/centre-for-humanities-computing/tleirary_evocationfor code and data.

Acknowledgments References

We want to thank everyone who contributed to this work, especially Mia Jacobsen, as well as colleagues and friends for pointing out pitfalls and sharing ideas.

M. Sandri, E. Leonardelli, S. Tonelli, and E. Jezek. “Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks”. InP:roceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Ed. by A. Vlachos and I. Augenstein. Dubrovnik, Croatia: Association for Computational Linguistics, 2023, pp. 2428–2441. doi: 10.18653/v1/2023.eacl-main.178. >2 (a) Diference between implicit/explicit groups in arousal, dominance and human annotated arousal.

We add the latter for reference since it is available in tEhme oBank corpus. Note that harousal and arousal behave similarly. (b) Diference between implicit/explicit groups in concreteness, imageability, visual, hapric and interoceptive levels.

References

[1] S. Ahmed . The cultural politics of emotion . Edinburgh Univ. Press, 2010 .

[2] H. J. Alantari , I. S. Currim , Y. Deng , and S. Singh. “ An empirical comparison of machine learning methods for text-based sentiment analysis of online consumer reviews” .InIn-: ternational Journal of Research in Marketing 39.1 ( 2022 ), pp. 1 - 19 . doi: 10 .1016/j.ijresma r. 2021 . 10 .011.

[3] A. Allaith , K. Degn , A. Conroy , B. Pedersen , J. Bjerring-Hansen , and D. Hershcovich . “ Sentiment Classification of Historical Danish and Norwegian Literary Texts” . InP:roceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa) . Ed. by T. Alumäe and M. Fishel . Tórshavn, Faroe Islands: University of Tartu Library, 2023 , pp. 324 - 334 . url: https://aclanthology.org/ 2023 .nodalida- 1 ..34

[4] C. O. Alm and R. Sproat . “ Emotional Sequencing and Development in Fairy Tales” . In: Afective Computing and Intelligent Interaction . Ed. by J. Tao , T. Tan , and R. W. Picard . Berlin, Heidelberg: Springer, 2005 , pp. 668 - 674 . do1i0 : . 1007 /11573548\_ 86 .

[5] D. Attridge . Peculiar Language. Routledge , 1988 .

[6] J. Auracher and H. Bosch . “ Showing with words: The influence of language concreteness on suspense” . In: Scientific Study of Literature 6.2 ( 2016 ), pp. 208 - 242 . doi: 10 .1075/ssol.6 .2.03aur.

[7] F. Barbieri , L. E. Anke , and J. Camacho-ColladoXs. LM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond. 2022 . doi: 10 .48550/arXiv.2104.12250.

[8] V. Batanović , M. Cvetanović , and B. Nikolić . “ A versatile framework for resource-limited sentiment articulation, annotation, and analysis of short texts” . IPnL:oS ONE 15.11 ( 2020 ). doi: 10 .1371/journal.pone. 024205 .0url: https://www.ncbi.nlm.nih.gov/pmc/articles /PMC7660500/.

[9] K. F. Baunvig . “ Forestillede faellesskabers virtuelle sangritualer: Forskningsprojekt vil kaste lys over den kulturelle betydning af den virtuelle faellessang under corona-tiden” . In: Tidsskriftet SANG 1.1 ( 2020 ), pp. 40 - 45 . doi: 10 .7146/sang.v1i1. 137029 .

[10] Y. Bizzoni and P. Feldkamp . “ Below the Sea (with the Sharks): Probing Textual Features of Implicit Sentiment in a Literary Case-study” . InP:roceedings of the Third Workshop on Understanding Implicit and Underspecified Language . Ed. by V. Pyatkin , D. Fried , E. Stengel-Eskin , A. Liu , and S. Pezzelle . Malta: Association for Computational Linguistics, 2024 , pp. 54 - 61 . url: https://aclanthology.org/ 2024 .unimplicit- 1 ..5

[11] Y. Bizzoni and P. Feldkamp . “ Comparing Transformer and Dictionary-based Sentiment Models for Literary Texts: Hemingway as a Case-study” . InP:roceedings of the 3rd International Workshop on Natural Language Processing for Digital Humanities . Tokyo, Japan: Association for Computational Linguistics, 2023 , pp. 219 - 226 . urhl:ttps://rootroo.com /downloads/nlp4dh%5C% 5Fiwclul % 5C % 5Fproceedings ..pdf [12] W. C. Booth . The Rhetoric of Fiction. 2nd edition . Chicago: University of Chicago Press, 1983 .

[13] E. Borelli , D. Crepaldi , C. A. Porro , and C. Cacciari . “ The psycholinguistic and afective structure of words conveying pain” . InP:loS one 13.6 ( 2018 ), e0199658 .

[14] K. Bowers and Q. DombrowskiK.atia and the Sentiment Snobs . 2021 . url: https://datasit tersclub.github.io/site/dsc11.htm. l

[15] C. Britzolakis . A“riel and other poems” . In:The Cambridge Companion to Sylvia Plath. Ed. by J. Gill . Cambridge University Press, 2006 , pp. 107 - 123 . do1i0 :. 1017 /ccol0521844967.

[16] C. Brooks . The well wrought urn: studies in the structure of poetry . Harcourt , 1947 .

[17] B. Brown. “ Thing Theory” . In:Critical Inquiry 28.1 , ( 2001 ), pp. 1 - 22 . url: http://www.js tor. org/stable/134425 .8 [18] M. Brysbaert , A. B. Warriner , and V. Kuperman . “ Concreteness ratings for 40 thousand generally known English word lemmas” . IBne:havior Research Methods 46.3 ( 2014 ), pp. 904 - 911 . doi: 10 .3758/s13428-013-0403-5.

[19] S. Buechel and U. Hahn . “ EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis” . IPnr:oceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics : Volume 2 , Short Papers . Ed. by M. Lapata , P. Blunsom , and A. Koller . Valencia, Spain: Association for Computational Linguistics, 2017 , pp. 578 - 585 . urlh:ttps://aclanthology.org /E17-2092.

[20] J. Burroway . Writing Fiction: A Guide to Narrative Craft . Little, Brown, 1987 .

[21] C. CCLM . Danske børn og unge har stort kendskab til H.C. Andersen . 2003 . url: https://d pu.au.dk/om-dpu/nyheder/nyhed/artikel/danske-boern -og-unge-har-stort-kendskab-ti l-hc-andersen.

[22] J. Charbonnier and C. Wartena . “ Predicting Word Concreteness and Imagery” . PI nro: - ceedings of the 13th International Conference on Computational Semantics - Long Papers . Ed. by S. Dobnik , S. Chatzikyriakidis , and V. Demberg . Gothenburg, Sweden: Association for Computational Linguistics, 2019 , pp. 176 - 187 . doi1 : 0 .18653/v1/ W19 -0415.

[23] M. Coltheart . “ The MRC Psycholinguistic Database” . InT:he Quarterly Journal of Experimental Psychology Section A 33.4 ( 1981 ), pp. 497 - 505 . doi: 10 .1080/14640748108400805.

[24] M. Daoshan and Z. Shuo . “A Discourse Study of the Iceberg Principle AinFarewell to Arms” . In: Studies in Literature and Language 8 .1 ( 2014 ), pp. 80 - 84 .

[25] A. Doche and A. S. Ross . “' Here is my shameful confession. I don't really “get” poetry': discerning reader types in responses to Sylvia PlathA'sriel on Goodreads” . In:Textual Practice 37.6 ( 2023 ), pp. 976 - 996 . doi: 10 .1080/0950236x. 2022 . 2082516 .

[26] T. Eliot .Selected Essays by T. S. Eliot . Faber & Faber , 1948 .

[27] K. Elkins . The Shapes of Stories: Sentiment Analysis for Narrative . Cambridge University Press, 2022 . doi: 10 .1017/9781009270403.

[28] H. Elsahar and M. Gallé . “To Annotate or Not? Predicting Performance Drop under Domain Shift” . In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Hong Kong , China: Association for Computational Linguistics, 2019 , pp. 2163 - 2173 . doi: 10 .18653/v1/ D19 -1222. url: https://aclanthology.org/D19-122 2 .

[29] M. Fludernik . “ Towards a 'Natural' Narratology” . JIlnse: 25.2 ( 1996 ), pp. 97 - 141 . doi: 10 .1515/jlse. 1996 . 25 .2.97.

[30] A. Gargett , J. Ruppenhofer , and J. Barnden . “ Dimensions of Metaphorical Meaning” . In: Proceedings of the 4th Workshop on Cognitive Aspects of the Lexicon (CogALex) . Ed. by M. Zock , R. Rapp , and C.-R. Huang . Dublin, Ireland: Association for Computational Linguistics and Dublin City University, 2014 , pp. 166 - 173 . doi1 : 0 .3115/v1/ W14 -4721.

[31] C. P. Heaton . “ Style inThe Old Man and the Sea” . In: Style 4.1 ( 1970 ), pp. 11 - 27 . url: https://www.jstor.org/stable/4294503.9

[32] C. Hutto and E. Gilbert. “ VADER: A parsimonious rule-based model for sentiment analysis of social media text” . In:Proceedings of the international AAAI conference on web and social media . Vol. 8 . 2014 , pp. 216 - 225 . doi: 10 .1609/icwsm.v8i1. 14550 .

[33] N. Ide , C. Baker , C. Fellbaum , and R. Passonneau . “ The Manually Annotated Sub-Corpus: A Community Resource for and by the People” . InP:roceedings of the ACL 2010 Conference Short Papers. Ed. by J . Hajič , S. Carberry , S. Clark , and J. Nivre . Uppsala, Sweden: Association for Computational Linguistics, 2010 , pp. 68 - 73 . urhl:ttps://aclanthology.o rg/P10-2013.

[34] R. Jakobson . “ Linguistics and Poetics” . In:Linguistics and Poetics. De Gruyter Mouton , 2010 ( 1981 ), pp. 18 - 51 . doi: 10 .1515/9783110802122.18.

[35] M. Jockers . A Novel Method for Detecting Plot . 2014 . url: https://www.matthewjockers .net/ 2014 /06/05/a -novel-method-for-detecting-plo . t/

[36] J. T. Kao and D. Jurafsky . “ A computational analysis of poetic style: Imagism and its influence on modern professional and amateur poetry” . InL:inguistic Issues in Language Technology 12 ( 2015 ). url: https://aclanthology.org/ 2015 .lilt- 12 . .3

[37] C. Kennington . “ Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms” . InP:roceedings of the 25th Conference on Computational Natural Language Learning . Ed. by A. Bisazza and O. Abend . Online: Association for Computational Linguistics, 2021 , pp. 148 - 157 . doi1 : 0 .18653/v1/ 2021 .conll- 1 . 11 .

[38] E. Kim and R. Klinger . “A Survey on Sentiment and Emotion Analysis for Computational Literary Studies” . In:Zeitschrift für digitale Geisteswissenschaften ( 2019 ). doi: 10 .17175/2 019\_008. url: http://arxiv.org/abs/ 1808 .0313 7.

[39] S.-T. Kousta , G. Vigliocco , D. P. Vinson , M. Andrews , and E. Del Campo. “ The representation of abstract words: Why emotion matters .” InJ:ournal of Experimental Psychology: General 140.1 ( 2011 ), pp. 14 - 34 . doi: 10 .1037/a0021446.

[40] Z. Li , Y. Zou , C. Zhang , Q. Zhang , and Z. Wei . “ Learning Implicit Sentiment in Aspectbased Sentiment Analysis with Supervised Contrastive Pre-Training” . IPnr:oceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Ed. by M.-F. Moens , X. Huang , L. Specia , and S. W.-t. Yih. Online and Punta Cana , Dominican Republic: Association for Computational Linguistics, 2021 , pp. 246 - 256 . do1i0 :. 18653 /v1/ 2021 .emnlp-main. 22 . url: https://aclanthology.org/ 2021 .emnlp- main.2.2

[41] T. Lundskaer-Nielsen. “The Language of Hans Christian Andersen's Fairy Tales - Compared with Earlier Tales” . InS:candinavistica Vilnensis 1.9 ( 2014 ), pp. 97 - 112 . doi: 10 .15 388/ScandinavisticaVilnensis. 2014 . 9 .8.url: https://www.journals.vu.lt/scandinavistica /article/view/14002.

[42] [43] D. Lynott , L. Connell , M. Brysbaert , J. Brand , and J. Carney. “ The Lancaster Sensorimotor Norms: multidimensional measures of perceptual and action strength for 40,000 English words” . In:Behavior Research Methods 52.3 ( 2020 ), pp. 1271 - 1291 . doi: 10 .3758/s13428-0 19 - 01316 -z.

M. M. Maslej , R. A. Mar , and V. Kuperman . “ The textual features of fiction that appeal to readers: Emotion and abstractness .” InP:sychology of Aesthetics, Creativity, and the Arts 15.2 ( 2021 ), pp. 272 - 283 . doi: 10 .1037/aca0000282.

[44] J. Meyers , ed. Hemingway: The Critical Heritage. Routledge , 1982 .

[45] S. Mohammad . “ Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20 ,000 English Words”. InP: roceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Melbourne, Australia: Association for Computational Linguistics, 2018 , pp. 174 - 184 . doi1 : 0 .18653/v1/ P18 -1017.

[46] C. Molesworth . “ ”With Your Own Face On”: The Origins and Consequences of Confessional Poetry” . In:Twentieth Century Literature 22.2 ( 1976 ), pp. 163 - 178 . doi: 10 .2307/44 0682.

[47] J. Mukařovský . “ Standard language and Poetic Language” . InA: Prague School Reader on Esthetics Literary Structure, and Style. Ed. by P. L. Garvin . 1932 . Georgetown University Press, 1964 , pp. 17 - 30 .

[48] M. A. Nielsen . “Salmesprog”. InD:ansk Sproghistorie Bind 4. Sprog i brug. Aarhus University Press and Society for Danish Language and Literature (DSLDK) , 2020 .

[49] B. Ohana , S. J. Delany , and B. Tierney . “A Case-Based Approach to Cross Domain Sentiment Classification” . In: Case-Based Reasoning Research and Development. Ed. by B. D. Agudo and I. Watson. Lecture Notes in Computer Science . Berlin, Heidelberg: Springer, 2012 , pp. 284 - 296 . doi: 10 .1007/978-3- 642 -32986-9\_ 22 .

[50] C. E. Osgood and G. J. Suci . “ Factor analysis of meaning .” InJ:ournal of experimental psychology 50.5 ( 1955 ), p. 325 .

[51] L. Oulanne . “Lived Things: Materialities of Agency, Afect, and Meaning in the Short Fiction of Djuna Barnes and Jean Rhys” . PhD thesis . Helsinki: University of Helsinki, 2018 . url: http:% E2%81%84%E2%81%84ethesis .helsinki.f.i

[52] E. Pound . “ A Few Don 'ts by an Imagiste” . In:Poetry 1.6 ( 1913 ), pp. 200 - 206 . url: https: //www.jstor.org/stable/2056973.0 [53] [54] D. Preoţiuc-Pietro , H. A. Schwartz , G. Park, J. Eichstaedt , M. Kern , L. Ungar , and E. Shulman. “ Modelling Valence and Arousal in Facebook posts” . PInro:ceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis .

Ed. by A. Balahur , E. van der Goot, P. Vossen, and A. Montoyo . San Diego, California: Association for Computational Linguistics, 2016 , pp. 9 - 15 . do1i : 0 .18653/v1/ W16 -0404.

url: https://aclanthology.org/W16-040.4 A. J. Reagan , L. Mitchell, D. Kiley , C. M. Danforth , and P. S. Dodds . “ The Emotional Arcs of Stories Are Dominated by Six Basic Shapes” . InE:PJ Data Science 5.1 ( 2016 ), pp. 1 - 12 .

doi: 10 .1140/epjds/s13688-016-0093-1. url: https://epjdatascience.springeropen.com/ar ticles/10.1140/epjds/s13688-016-0093-1.

[55] S. Rebora . “ Sentiment Analysis in Literary Studies. A Critical Survey” . DI nig: ital Humanities Quarterly 17.2 ( 2023 ). url: https://www.digitalhumanities.org/dhq/vol/17/2/000691 /000691.html%5C# kim - klinger2018 .b

[56] S. Rebora , M. Lehmann , A. Heumann , W. Ding , and G. Lauer. “ Comparing ChatGPT to Human Raters and Sentiment Analysis Tools for German Children's Literature”P .rIon-: ceedings of the Computational Humanities Research Conference 2023 , Paris, France, December 6- 8 , 2023 . Ed. by A. Sela , F. Jannidis , and I. Romanowska . Vol. 3558 . CEUR Workshop Proceedings. CEUR-WS.org, 2023 , pp. 333 - 343 . urlh:ttps://ceur-ws. org/ Vol- 3558 /pape r3340. pdf.

[57] V. Rentoumi , G. Giannakopoulos , V. Karkaletsis , and G. A. Vouros . “ Sentiment Analysis of Figurative Language using a Word Sense Disambiguation Approach” . IPnr:oceedings of the International Conference RANLP-2009 . Ed. by G. Angelova and R. Mitkov . Borovets, Bulgaria: Association for Computational Linguistics, 2009 , pp. 370 - 375 . urhlt:tps://acl anthology. org/R09-106 . 7

[58] I. A. Richards . Principles of Literary Criticism. Routledge , 2003 .

[59] D. Ringgaard and M. R. Thomsen , eds. Danish literature as world literature. Literatures as world literature . New York: Bloomsbury Academic, 2017 .

[60] L. M. Rosenblatt . “ The Literary Transaction: Evocation and Response” . InT:heory Into Practice 21.4 ( 1982 ), pp. 268 - 277 . url: https://www.jstor.org/stable/147635.2

[62] B. Sandstrøm . “ Salmen - fra kampsang til lovprisning” . IDna:nsk Litteraturs Historie 1100 - 1800. Ed. by V. A. Pedersen , M. Schack , and K. P. Mortensen. Gyldendal , 2007 .

[63] E. Savinova and F. Moscoso Del Prado. “ Analyzing Subjectivity Using a TransformerBased Regressor Trained on Naıv̈e Speakers' Judgements” . InP:roceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis . Ed. by J. Barnes , O. De Clercq , and R. Klinger . Toronto, Canada: Association for Computational Linguistics, 2023 , pp. 305 - 314 . doi1 : 0 .18653/v1/ 2023 .wassa- 1 . 27 .

[64] T. Schmidt and M. Burghardt . “ An Evaluation of Lexicon-based Sentiment Analysis Techniques for the Plays of Gotthold Ephraim Lessing” . InPr:oceedings of the Second Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage , Social Sciences, Humanities and Literature. Ed. by B. Alex , S. Degaetano-Ortlieb , A. Feldman , A. Kazantseva , N. Reiter , and S. Szpakowicz. Santa Fe , New Mexico: Association for Computational Linguistics, 2018 , pp. 139 - 149 . url:https://aclanthology.org/W18-451. 6

[65] T. Schmidt , K. Dennerlein , and C. Wolf . “ Using Deep Learning for Emotion Analysis of 18th and 19th Century German Plays” . InF: abrikation von Erkenntnis: Experimente in den Digital Humanities - ( 2021 ). doi: 10 .26298/melusina.8f8w -y749-udl .f

[66] P. Sharma , N. Ding , S. Goodman , and R. Soricut . “ Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning”. InP:roceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Ed. by I. Gurevych and Y. Miyao . Melbourne, Australia: Association for Computational Linguistics, 2018 , pp. 2556 - 2565 . doi: 10 .18653/v1/ P18 -1238. url: https://aclant hology. org/P18-123 . 8

[67] E. Stengel-Eskin , J. Guallar-Blasco , and B. Van Durme. “ Human-Model Divergence in the Handling of Vagueness” . In:Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language . Ed. by M. Roth , R. Tsarfaty , and Y. Goldberg . Online: Association for Computational Linguistics, 2021 , pp. 43 - 57 . do1i0 :. 18653 /v1/ 2021 .unim plicit- 1 .6.

[68] P. J. Stone , R. F. Bales , J. Z. Namenwirth , and D. M. Ogilvie . “ The general inquirer: A computer system for content analysis and retrieval based on the sentence as a unit of information” . In:Behavioral Science 7.4 ( 1962 ), p. 484 .

[69] T. Strychacz . ““ The sort of thing you should not admit”: Ernest Hemingway's Aesthetic of Emotional Restraint” . In:Boys Don't Cry? Rethinking Narratives of Masculinity and Emotion in the U .S. Ed. by M. Shamir and J. Travis . Columbia University Press, 2002 , pp. 141 - 166 . doi: 10 .7312/sham12034- 009 .

[70] S. Ullrich , A. Aryani , M. Kraxenberger , A. M. Jacobs , and M. Conrad . “ On the relation between the general afective meaning and the basic sublexical, lexical, and inter-lexical features of poetic texts- a case study using 57 poems of H.M. Enzensberger” . IFnr:ontiers in psychology 7 ( 2017 ), p. 2073 .

[71] H. G. Wallbott and K. R. Scherer . “ How universal and specific is emotional experience? Evidence from 27 countries on five continents” . In:Social Science Information 25.4 ( 1986 ), pp. 763 - 795 . doi: 10 .1177/053901886025004001.

[72] M. Wan , Q. Su , K. Ahrens , and C.-R. Huang . “ Perceptional and actional enrichment for metaphor detection with sensorimotor norms” . INn:atural Language Engineering ( 2023 ), pp. 1 - 29 . doi: 10 .1017/s135132492300044x. url: https://www.cambridge.org/core/jour nals/natural-language-engineering/article/perceptional-and -actional-enrichment-formetaphor-detection-with-sensorimotor-norms/0BA36E2578B2AD80CCCE00E6AF6969 AB .

[73] A. B. Warriner , V. Kuperman , and M. Brysbaert . “ Norms of valence, arousal, and dominance for 13,915 English lemmas” . InB:ehavior research methods 45 ( 2013 ), pp. 1191 - 1207 .

[74] T. Wilson , J. Wiebe , and P. Hofmann . “ Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis” . In:Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing . Ed. by R. Mooney , C. Brew , L.-F. Chien , and K. Kirchhof . Vancouver, British Columbia, Canada: Association for Computational Linguistics, 2005 , pp. 347 - 354 . urlh:ttps://aclanthology.org/H05-104. 4

[75] D. Zhou , J. Wang , L. Zhang , and Y. He . “ Implicit Sentiment Analysis with Event-centered Text Representation” . In:Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Ed. by M.-F. Moens , X. Huang , L. Specia , and S. W.-t. Yih. Online and Punta Cana , Dominican Republic: Association for Computational Linguistics, 2021 , pp. 6884 - 6893 . doi: 10 .18653/v1/ 2021 .emnlp-main. 551 .