Sentiment Analysis (SA) is an increasingly central method for computational literary research [55], an especially popular application being that of gauging the 'sentiment arcs' of novels, i.e., the 'shapes of stories' [38,54,35]. Still, the relation between valence extracted with SA tools and the human perception of literary texts at a granular level remains an open question, as tools applied to the literary domain are primarily geared towards processing nonliterary texts.
While some recent work has examined the adequacy of available SA tools for literary analysis [27,11,56], the question of how to validate them -against whose judgements, and at what level (i.e., the story-level vs. sentence-level, etc.) -is a persistent concern, also due to the relative scarcity of annotated resources in the literary domain. Moreover, the observed inadequacy of SA tools for literature has raised the question of what the difference of literary texts might be in comparison to the nonliterary [10], where tools seem to perform comparably better [63]. How does the "literary" differ in its way of communicating sentiments to readers?
In fact, due to their textual complexity, literary texts are often said to differ from more communicative texts [34]: they are effective at multiple narrative levels [60,12]; creatively divergent from standard language use [47,5]; reliant on poetic devices [16]; and ambiguity, effecting contesting interpretations [58]. Moreover, literary language may exhibit various strategies for conveying emotion beyond simply using words directly associated with emotional states (e.g., "sad"). While language on, e.g., social media, may also rely on omission and subtlety, literary theorists have frequently claimed that literariness or the poetic function of language excels in this regard and is distinct from its more directly communicative function. 1 Regarding sentiment expression, it has recently been suggested that literary prose relies on specific semantic traits connected to affective understatement and concreteness to evoke -rather than "communicate" -sentiment in readers [10]. 2 These features are especially interesting since they are related to the seminal concept of the "objective correlative" in literary theory, which suggests that literary texts effectively convey emotions by grounding them in -concrete and objective, rather than emotional and subjective -entities and situations [26], while avoiding abstract and emotional language. 3 In this study, we further pursue the hypothesis that literary texts rely on omission and materiality, i.e., features connected to the concept of the objective correlative, to evoke sentiments, rather than simply mention them.
While divergences between human and model judgments in sentiment analysis are generally taken to indicate shortcomings in SA tools, they present an interesting case for testing the difference of literary language in expressing sentiment, assuming that SA tools are generally tuned toward more explicit forms of communication due to their development on e.g. social media -also suggested by more general studies of implicitness [75,40]. Using a single novel as their data, Bizzoni and Feldkamp [10] explored this discrepancy, finding that certain semantic elements -levels of arousal, dominance and concreteness -are indicative of human/model disagreement. To gain further insight into sentiment expression in literary texts, we go one step further: we test the distinguishing power of these features in a much larger corpus of annotated literary texts. We also extend the list of features by an additional four: imageability, visuality, hapticity, and interoception, in order to further examine the connection of human/model disagreement with features that relate to the concept of the objective correlative and of omissive strategies in literary sentiment expression. 4We conduct two experiments. First, we test whether sentences where models and humans agree can be distinguished from sentences where they disagree based on these specific semantic traits. Secondly, we further explore the robustness of the relation between features and sentiment evocation by testing the correlation of these features to the absolute difference in sentiment scoring between humans and models.
While SA tools perform increasingly well in some domains [2,63], studies have pointed out the cross-domain drop in performance, as well as the lagging behind of tools for under-resourced languages [28,49,14]. Still, some studies have suggested that Transformer-based models might be able to bridge the gap and perform better on literary or poetic material [65]. Assessing the performance of models on historical Danish and Norwegian literary texts, Allaith, Degn, Conroy, Pedersen, Bjerring-Hansen, and Hershcovich [3] found that multilingual Transformers outperformed both fine-tuned models and classifiers based on lexical resources in the target language, which aligns with the findings of Schmidt, Dennerlein, and Wolff [65] and Schmidt and Burghardt [64] for historical German drama. Still, the disparity between human and model sentiment judgements in literary texts continues to be observed [11,56]. The disparity is often related to the effects of narrative, annotators generally having access to the narrative context of a sentence, rather than to differing strategies in sentiment expression across domains.
The concept of "implicit" expression is particularly relevant, and complex, in literary writing. Several theories of writing point to the importance of avoiding concepts or ideas (however this is intended) in a too explicit way. The widely known precept of "Show Don't Tell" points at least partly in this direction [12]. Moreover, critics continually rely on terms like emotional "evocativeness" and "understatement" to describe writing styles [69,24]. Literary and affect theory has also more recently foregrounded the use of materiality and sensuousness in literature -including poetry -to evoke affective reactions in readers, emphasizing the way objects and things are culturally invested with meaning and affect [1,17] and thus utilized by authors to evoke embodied, a ective experiences [51]. 5 Despite the significance of implicit, evocative, and expressive strategies, there is little consensus on how to reliably track these in literature and whether such types of expression have recognizable linguistic markers.
The association of materiality to literary evocation is not new, and closely relates to the Modernists' and New Critics' valuation of concreteness over abstraction [69], as well as to the idea of the 'objective correlative' which T.S. Eliot [26] proposed in 1948. Eliot suggested that the effective way of expressing emotion in literature is 'by proxy', through an external and objective -in the sense of intersubjectively recognizeable -"set of objects, a situation, a chain of events which shall be the formula of [a] particular emotion" [26]. The concept of the objective correlative suggests that literary language effectively evokes sentiments in readers by being both more omissive (relying less on directly emotion-associated words) 6 and more concrete (relying on objects and situations).
This hypothesis has been supported by computational literary studies. Auracher and Bosch [6] found that the concreteness of literary language impacts the emotional engagement of readers and their experience of suspense, and Bizzoni and Feldkamp [10] tracked the "omissive" writing of Ernest Hemingway by looking at the amount and intensity of sentiment expressions detectable in sentences, compared to how "expressive" (in terms of sentiment) readers perceive these sentences to be. Comparing sentences where humans and models agree vs. those where they disagree, they found that arousal and dominance levels (of the NRC-VAD lexicon [45]) were indicative of omissive strategies of evocation. Moreover, the level of concreteness of the language used appeared higher in sentences with higher disagreement between humans' and models' valence attribution.
In computational linguistics, the use of semantic traits -often derived from psycholinguistic norms -for the study of narrative is relatively frequent [70,36,43]. Yet, their use to model poetic literary strategies of sentiment evocation is still relatively pioneering. Kao and Jurafsky [36] applied semantic measures to poetry: imageability to gauge "imagery"and concreteness to gauge "concrete imagery", using the concreteness ratings of Brysbaert, Warriner, and Kuperman [18], along with objective/abstract word categories [68], as well as psycholinguistic norms [50] to model "emotional language". They show that these features differ between "amateur" and appraised poets, where appraised poets use more concrete and imageable language and less emotional words. Conversely, Maslej, Mar, and Kuperman [43] find that abstraction and arousal correlate positively with the perceived interest of readers in fictional characters, which is perhaps related to the general tendency of abstract concepts to be more emotionally valenced than concrete ones [39]. Ullrich, Aryani, Kraxenberger, Jacobs, and Conrad [70] observe how the differences between perceived affect in poem annotations can be largely explained by lexical psycholinguistic norms, with sentiment dimensions like arousal being one of the best predictors of the perceived affective meaning of the poems. The studies show above all that imageability, concreteness and arousal strongly relate to how readers feel about literary texts.
Used datasets with valence annotation. For all but the FB dataset, valence was annotated on a sentence basis, so the number of annotations generally indicates a number of sentences. The Spearman correlation (𝜌) between the human mean and RoBERTa scores (H/R) is provided (for all, 𝑝 < .01). Summing up, the total number of annotated lines considered in this study is 𝑛 = 19, 327. The Annotator/line indicates the number of annotators of valence reported per line in the corpus.
x
To probe textual features of implicit sentiment, we created a diverse corpus of fiction spanning four genres, manually annotated for valence (Fiction4). For the second experiment of this paper -comparing more or less literary genres -we also selected other datasets to represent a diversity of genres in the literary and non-literary domains which had all been annotated for valence on a continuous scale. 7 As such, we include social media, journalism, and genres with various degrees of literariness (essays, letters, travel-writing, fairytales), which may also be thought to represent degrees of colloquialism (from blogs to journalism).8
Fiction4: The new dataset presented in this study, with human annotations for valence (𝑛 = 6, 300). 9 It includes four different genres -fairy tales, hymns, prose and poetry -over the 19 th and 20 th century, in one high-and one low-resource language (English and Danish). The corpus was compiled with an aim toward diversity (in genre, time and language) while still aiming to include well-known and culturally significant works by both male and female authors. 10 We also took into account both i) the texts' cultural significance and ii) their narrative and poetic complexity, which may represent a particular challenge to SA tools. The authors selected were Ernest Hemingway, Sylvia Plath, H.C. Andersen and the various authors of ofÏcial hymn-books (for details, see Appendix, Table 7). i) Regarding their cultural significance, for the English texts, Hemingway is perhaps one of the most famous 20 th century authors for his prose, and his texts are read in education 11 and among the public. 12 Plath, similarly, is a widely read and acclaimed poet, perhaps the best known female American poet of the 20 th century. 13For the Danish texts, Andersen's production is arguably the most central in Danish literary heritage [59]. While being less known internationally, the ofÏcial hymnal book is the most widely distributed "poetry book" in Denmark [62], 14 used in the Danish education system at all levels, and shapes national cultural identity [9].
ii) Regarding the complexity of the corpus for the sentiment annotation task, we ensured variance across genre, place and time, but also emphasized, from a literary perspective, the level of literary complexity, including texts that could be considered either particularly challenging or particularly simple. We consider Hemingway's The Old Man and the Sea and Plath's Ariel as two ideally difÏcult cases for testing SA tools. Hemingway is known for an especially "omissive" writing style, direct and limited in its use of figurative language [31], while relying on implication rather than "overt emotional display", leaving much inference up to the reader [69]. Hemingway's The Old Man and the Sea (1952) has been considered emblematic for this minimalist style, which may omit characteristics that models rely on in sentiment scoring. Plath's poetry collection Ariel (1965) is complex in a slightly different way. 15 The so-called confessional poetry genre, of which Plath is considered emblematic, foregrounds idiosyncratic personal psychology and experiences against the "emotional vacuity of public language" and universal symbols [46]. In Ariel Plath writes on complex and political themes in idiosyncratic style consisting of "hallucinatory images" and novel metaphors [15], and the work has been used as a case of literature posing particular difÏculty to most readers [25].
Conversely, we consider Andersen and religious hymns two ideally simple cases for testing SA tools. Andersen's fairy tales 16 are characterized by an essential simplicity, both stylistically an in their narrative progression [41,4] and their ability to engage both children and adult readers [41]. Religious hymns 17 are characterized by their limited number of themes (e.g. worship, thanksgiving, etc.), which are are expressed through well-known and formalized (as well as recurring) phrases, metaphors, figurative and symbolic language [48]. The hymns' repetitive and predictable use of language may make them more accessible to models, even though their archaic and nuanced style may present challenges.
After collecting the texts, we found that some of these simplicity/complexity assumptions are reflected in the correlation between human and model valence scores -at least when using our method -where Plath's poetry shows the lowest correlation, and hymns the highest correlation (Table 1).
EmoBank The EmoBank is a multigenre corpus with human annotations for valence (𝑛 = 8, 735), 18 with 10 annotators per sentence [19]. 19 The corpus was composed from various categories in the Manually Annotated Sub-Corpus of the American National Corpus (MASC), 20 consisting of texts from 1990 and onwards [33]. We consider the EmoBank categories: Letters, Blog, Newspaper, Essays, Fiction, and Travel guides, which are relatively balanced (Table 1) including both longer and shorter texts within each category. 21FB The Facebook corpus of posts (FB) 22 collected between 2009 and 2011, consists of 2,895 status updates, each by a unique user, with human annotations for valence and arousal, with 2 annotators per post [53]. The FB dataset differs from our other corpora, consisting of postsnot sentences. While some posts are short (e.g., ":)" and "LOL"), the average length of posts is comparable to the average sentence length in, e.g., EmoBank (Table 1).
Beyond these corpora, we also include two datasets without valence annotation for comparison in terms of feature levels:
Image-captions (𝑛 = 3, 334, 173) of the Conceptual Captions dataset [66], of which we consider feature values to represent a "high-water mark" (i.e. high level) of language relying on object description and visuality. 23Participants free emotion event descriptions (𝑛 = 6, 898) 24 of the International Survey on Emotion Antecedents and Reactions (ISEAR) [71], 25 of which we consider feature values to represent a "high-water mark" of language dealing with interiority (i.e., relating to inside sensations and self) and emotionality.
Model annotation Multilingual transformer-based models have shown best performance in SA for literary texts across languages (also for historical texts) [11,3,65] compared to dictionarybased approaches explicitly developed for literary texts as well as monolingual English models [11]. We, therefore, used the RoBERTa base xlm multilingual, finetuned for sentiment analysis on Twitter data, 26 which is comparable to the state-of-the-art models in a monolingual (nonliterary) setting [7], and shows the best performance in the limited studies there are on literary prose [11]. 27 XML-RoBERTa 28 was developed through a cross-lingual language training method, designed to boost its proficiency in comprehending and processing multiple languages by transferring skills it has acquired from one language to another.
With this model, we scored all sentences across our bilingual datasets. 29 The model returns polarity positive or negative, and a neutral label. To attain more continuous, nuanced data from the transformers' categorical output, we opted for using the same strategy as in Bizzoni and Feldkamp [11], i.e., using the confidence score of model labels as a proxy for sentiment intensity. For example, a sentence with a positive label and a confidence of, e.g., 0.75, is interpreted as a valence score of +0.75. Similarly, a negative label with confidence of 0.89, is interpreted as a valence score of -0.89. For the neutral category, confidence is disregarded and levelled to a score of 0 (midscale or "neutral"). 30 The correlation between human mean score and the transformed RoBERTa score appears high across our selected corpora (Table 1).
As seen in Table 1, RoBERTa values correlate most strongly with human annotations of the FB dataset, while correlations with annotations of fiction (EmoBank and Fiction4) are much lower, possibly reflecting better development the model for certain more colloquial domains (social media, blogs, letters).
Human annotators (at least 𝑛 = 2/line) read the literary texts from beginning to end, scoring each line on a 0 to 10 valence scale: 31 0 signifying the lowest, and ten the highest valence. 32 The valence score was intended to represent the sentiment the sentence and verse expressed, and annotators were instructed to avoid rating how a sentence or verse made them feel and to try to report only on the sentiments embedded in the sentence, i.e., to think about the valence of the individual sentence and verse, without overthinking the story's/poetry's narrative.
It is worth noting that humans rarely reach an agreement higher than 80% (or 0.80 Krippendorff's 𝛼) for tasks like positive/neutral/negative discrete tagging [74] on nonliterary textsand have lower agreement for continuous scale polarity annotation [8], especially for literary texts [56].
For our first experiment, exploring the prevalence of semantic traits in sentences where humans/model disagree and sentences where they agree, we divided our Fiction4 corpus into two groups. First, we filtered out sentences which humans did not perceive any strong sentiment (i.e., with human valence scores between 4.5 and 5.5 on the 0-10 scale). On one hand, we then took sentences in which our chosen model did not assign any strong sentiment (below an absolute score of 0.1, i.e., between -0.1 and +0.1) 33 and, on the other hand, sentences where it did. With this procedure, we distilled two groups of sentences (Fig. 1): one of sentences with humans/model disagreement, which we call the "implicit" group (𝑛 = 1, 194) and one of human/model agreement, which we call the"explicit" group (𝑛 = 2, 631).
Based on previous work (section 2), we include three previously used semantic traits to examine their bearing on instances of implicit sentiment evocation: at the sentiment dimension, 31 "Lines" refer to sentences in the case of prose and to verse-lines in the case of the hymns/poetry. Sentences were tokenized using the nltk tokenize package. 32 Annotators were researchers, three with a background in literary studies and one in cognitive science. The two annotators of the hymns (MA and PhD of literature) had domain knowledge in 19 th century Scandinavian literature and historical religious hymns. 33 Note that the model valences range from -1 to 1 (negative to positive), where 0 represents neutral. arousal, 34 and dominance, 35 and at the sensorimotor dimension, concreteness, 36 imageability, as well three additional sensory traits, visuality, 37 haptic, 38 and interoception. 39 We use the datasets below to measure sentence semantic trait values, averaging the score per feature for each sentence.
The lexicon by Brysbaert, Warriner, and Kuperman [18] provides concreteness ratings for 37,058 English words. Annotators were recruited via Mturk (English native speakers). Each word was annotated by at least 25 annotators, on a scale from 1 (=most abstract, i.e., what cannot be experienced directly but the meaning of which is defined by other words) to 5 (=most concrete, i.e., what can be experience directly through one of the five senses). These ratings have been widely used [22] also in the literary domain [6].
The lexicon by Mohammad [45] provides ratings of 20,000 English words on three sentiment dimensions (valence, arousal, dominance). Annotators were recruited via CrowdFlower, and each word was annotated by at least 6 annotators with a best/worst scaling approach (e.g. most arousal vs. least arousal). The lexicon has been used widely, as well as integrated in the SA tool VADER [32].
The dataset provides norms of sensorimotor strength for 39,707 English words across 6 perceptual modalities (haptic, auditory, olfactory, gustatory, visual, and interoceptive) [42]. While the dataset includes action effectors (i.e., body parts) we used only the general perceptual norms, selecting only those we deemed especially relevant to the idea of objective correlative (i.e., material, visual objects/situations): visuality, interoception and hapticity. The perceptual part of the dataset had 2,625 annotators recruited via Mturk. Each word was rated from 0 (=not experienced with sense X) to 5 (=experienced greatly with sense X). These perceptual modality ratings have been used in, e.g., metaphor detection [72], and have served as a form of "embodied experience" information to enrich and improve language models [37].
The MRC Psycholinguistic Database (MRCPD) provides 26 linguistic and psycholinguistic variables for 150,837 English words -a subset of which are 9,240 words words rated for imageability [23]. These words have been rated by annotators. Ratings reflect how easily a word can evoke mental imagery, and fall in the range 100 − 700. The lexicon has been used variously, e.g., in metaphor [30] and literary studies [36]. 34 The degree to which a word prepares for action, captures or focuses attention [13]. 35 The degree of control evoked [73]. 36 The degree to which a word denotes a perceptible entity [18]. 37 The degree to which a word is experienced with the eyes [42]. 38 The degree to which a word is experienced by touch [42]. 39 The degree to which a word is experienced by sensations inside the body [42].
Inter Rater Reliability between annotators across literary genres, using the mean ( x ) of Spearman's 𝜌 between pairs (for all, 𝑝 < .01) -with Krippendorff's 𝛼 for reference.
We report a relatively high inter-rater reliability (IRR): between annotators, we find a mean correlation (Spearman's 𝜌) from 0.59 for poetry to 0.73 for hymns (Table 2). 40 IRR is high, especially for hymns, considering both the fragmentariness of the verses, and that humans tend to have low agreement for continuous-scale annotation (Section 4.1).
For our first experiment we compared the two groups of sentences in Fiction4 along each of the chosen features. We report the Mann-Whitney U-test effect's size and significance levels in Table 3. 41 We find that the strongest effect size is for interoceptive values (Table 3), while visually, concreteness shows two noteable "peaks" (Fig. 2). 42 Overall, we can confirm the difference 40 As annotators operated within a continuous valence spectrum, divided into ten categories, we find that a correlation measure more clearly reflects direction and nuance of annotations (parallelity vs exactness), compared to categorical IRR measures. Therefore, we report Spearman's 𝜌 and provide Krippendorff's 𝛼 for reference (the level of measurement is considered interval). 41 For the test, we dropped sentences with NaN-values in the specific feature we were testing, the number of dropouts was < 40 in each test. 42 The results of the Mann-Whitney U test are supported by a linear regression, where we sought to model the two groups by each feature. Significant results of the linear regression correspond to those indicated by the Mann-Whitney U-test, see between groups for the three features previously tested [10], while adding the observation of slight differences also for language heavy in visual and haptic information, as well as a robust difference for interoceptive information in the implicit group.
For reference, we conducted the same experiment on our reference corpora EmoBank and FB, dividing the data into implicit and explicit groups of sentences as outlined in Section 4.2. These results are reported in table 4. Histograms to support this difference in feature values between groups in EmoBank can be found in Appendix, Fig. 7 -as in Fiction4, levels of arousal, dominance and interoceptive are lower in the implicit group, while concreteness is higher in the implicit group. In the reference corpora (table 4), the strongest effect size is tendentially, as in Fiction4, interoceptive values. Notably, interoceptive and arousal values hold significant discriminating power across all corpora, as well as concreteness if we disregard the FB corpus. We see important differences within the EmoBank, where all features appear important only in the more personal or imaginative genres, and not in newspaper and essays.
In the second experiment, we check for correlations (rather than simple statistical difference) between human/model disagreement and the level of each semantic trait per sentence. This allows us to observe whether the presence of "undetected sentiment" in text has a linear relation with any of the semantic dimensions selected. For this, we used only sentences found in the implicit group (as outlined in Section 4.2), so that we correlated the amount of disagreement between human and model with our chosen features in sentences where humans perceived sentiment, but models did not. We report our results in Table 5. First of all, not all patterns of sentiment implicitness as seen in Experiment 1 are detectable as a correlation, suggesting that some of these features do not impact sentiment evocation linearly. On the other hand, we do see correlations that point to interesting genre differences in how sentiment is perceived in texts. For Fiction4, we find a consistent negative correlation between human/model disagreement and arousal which aligns with the lower levels of arousal in the implicit group we saw in Experiment 1. While concreteness and interoception do not show consistent linear correlations with disagreement, effects of low interoception related to higher disagreement are evident in Andersen's fairy tales. 43Within Fiction4, the role of high concreteness paired with higher dominance, and, to a lesser extent, lower arousal in sentiment disagreement (which we link to evocation) is confirmed, as well as the negative correlation of disagreement with interoception.
For comparison, we redid correlations in the reference corpora detailed in Section 3.1. Here, positive correlations are also found with concreteness: the more concrete a sentence is, the more our SA model's sentiment judgment will differ from that of human's. The strongest role of concreteness in sentiment disagreement appears to be not in literary texts proper, but in the travel guides and letters contained in EmoBank, and in blogs. Interoception also holds a negative correlation with disagreement Fiction category in EmoBank, as it did with Fairy tales in Fiction4.
Interestingly, the negative correlation of arousal with disagreement is not as consistent in the reference corpora, where we only see a negative correlation in Blogs. We find spurious positive correlations of disagreement with dominance, and visual -notably, however, correla- tions where they appear tend to have the same negative or positive direction across all corpora (including Fiction4), with the exception of imageability. Facebook posts in FB seem to have no significant link to many of these channels.
While most datasets seem to exploit some form of trade-off between concreteness, on one side, and arousal, dominance and interoceptive on the other, relatively few show correlations with the visual and haptic semantic information, as well as with imageability. The exceptions are blogs in EmoBank, which shows a weak but positive correlation with visual and with human/model disagreements; and travel guides where disagreement has a correlation with haptic and imageability. FB and the EmoBank newspaper and letter category return non-significant correlations with most dimensions, with the exception of dominance for FB.
Genre differences in the overall use of these semantic traits can be observed in Table 6. Note that, for example, Facebook posts seem to have high values of imageability. Still, at the same time, imageability in posts displays no correlation to the absolute disagreement between model and human (Table 5). In other words, it may be that although the language of posts is highly imageable, the images are not used in a way that subtly evokes human emotion and that challenges models as much as it happens in, for example, travel guides. Similarly, literary genres (in EmoBank and Fiction4) seem to have high values for imageability and visual scores (Table 6), but these dimensions exhibit little correlation with human/model disagreement.
Through all our datasets, concreteness has a positive relation with disagreement -showing higher levels where models are unable to capture the sentiment that humans perceive -as much as interoception has a negative one. In general, the opposite strength of concreteness and interoception in all of our datasets appears to confirm our intuition that interoception works as a sort of anti-concreteness when it comes to the evocation of sentiments, as the usage of external objects and "things" to evoke sentiments in the reader will make a low recourse to the interoceptive dimension. The fact that visual, haptic and imageablity correlations, when relevant, tend to the same direction as concreteness also adds to this hypothesis.
It is intriguing that a positive correlation between human/model disagreement and concreteness tend to co-occur with a positive correlation with sensory norms. The intuition that what is concrete as something "that is perceived through the senses" or "that can be drawn" would have led us to expect correlations of, e.g., haptic and concreteness to co-occur. On the other hand, concreteness does not have to occur with explicit sensory information at all: many words like house, sea, or wood, do not peak on one specific sense, and yet are considered fairly concrete; and some words like melody or rhythm might be less concrete and yet have sensory associations. The kind of concreteness that matters here might be more related to a general physical materiality rather than to a specific sensory load.
Concreteness exhibits a stronger correlation with visual, haptic, interoceptive and imageability traits (𝜌 > .5, 𝑝 < .01) than with dominance and arousal (around 𝜌 = .2, 𝑝 < .01). When correlating terms in the concreteness dictionary with other semantic traits, we find that especially interoceptive and concreteness show an interesting correlation. Words referring to internal emotional states, also presumbaly having higher arousal (e.g., "lovesickness", "hopelessness"), tend to cluster in the direction of high interoception and low concreteness. While concreteness has a robust negative correlation with interoception (𝜌 = −.52, 𝑝 < .01), i.e., high concreteness words generally are less interoceptive, we find that there is a set of words which maintain high interoception and high concreteness (upper right in Fig. 4). These words may be characterized as referring to concrete objects, which are nevertheless associated with internal (vs. external) states and experiences (e.g. "bladder", "breath"). Conversely, words in the lower right corner, with high concreteness and low interoceptive values, appear to be more of the category objects of external experience (e.g. "jewel", "clip", "lightswitch"), less associated with internal sensation (than is, e.g., "bee sting" or "drugs"). Considering the opposite correlation of human/model disagreement with interoception vs concreteness, we hypothesize that words used in instances of "objective correlative" would predominantly appear in the lower right corner of Fig. 4). This means that they are associated with words that are not only more concrete but also "objective" in the sense of referring to external rather than subjective or internal experiences. Therefore, the "objective correlative" might be understood as "objective" in both senses: impersonal and focused on external objects.
We have examined the relation between human/model disagreement on sentiment annotations and a chosen set of semantic traits for a new corpus of literary prose, comparing them with datasets representing several other domains, and we have extended the semantic traits' set used in previous literature to include also the sensory and interoceptive dimensions. Overall, we confirm previous results obtained on smaller data about relation of semantic traits to the presence of "undetected" or implicit sentiment in literary fiction, and we have observed similar trends also in non-fictional domains, with interesting differences between genres.
The "undetected" sentiments are likely to be evoked, rather than stated, and this evocation seems to pass through a trade-off between several semantic traits: an increase in concrete- ness and a decrease in arousal and interoception. 44 These traits seem to align with what in literary theory has been called "objective correlative": the strategy of conveying sentiment (or emotion) through the reference of external, material, or "objective" reality. This seems to happen together with the downplaying of semantic dimensions related to intensity and control, contributing to a subtler, less explicit form of emotional communication, which we might characterize as an omissive evocative strategy. An example of the trade-off between omission and use of objective correlative can be observed in the following sentence of Hemingway from Fiction4: ``Ay, he said aloud. There is no translation for this word and per- haps it is just a noise such as a man might make, involuntarily, feeling the nail go through his hands and into the wood'' . The sentence was consistently rated as negative by humans, and neutral by the model (see Fig. 5 above). Note how concreteness and interoception tend to divert in this sentence (e.g. on "feeling" or "nail"), while arousal and dominance values are sparse (Fig. 5).
In the future, we intend to expand our analysis to larger and more diverse corpora, and integrate more psycholinguistic resources, seeking ultimately to contribute to the development of better tools for sentiment analysis in literary genres. We would also like to observe the relation between reader response or literary reception and the concreteness-dominance or concreteness-interoception trade-off.
We want to underline that our corpus of fiction (Fiction4) is limited, with only one author representing three of the four categories (Plath for Poetry, Hemingway for prose, and Andersen for fairy tales). Moreover, the demographic of our dataset is reduced (in terms of gender, ethnicity, age, social class, etc.). Replication of these results on a larger and more diverse corpus of fiction is needed, and our results should be interpreted with this in mind.
Overview of texts within the four genres of the Fiction4 corpus: the total number of lines -which are verses (in the case of hymns and poetry) and sentences (in the case of fairy tales and prose)the number of words per dataset, the mean number of lines (verses/sentences) per text, the year of publication, and the number of human annotators. Note that the number of lines also represents the number of annotations, as they were done on a verse/sentence basis.
x
Effect sizes of the Mann-Whitney U test on the implicit vs. explicit group of sentences for each genre of the Fiction4 corpus and the R 2 of the linear regression (seeking to model the two groups) for reference. While R 2 coefficients are low, the idea is to show that a difference exists between the groups in terms of feature values -which nevertheless show large overlaps visually -not to model valence as such via these features. We also include the test between groups within each genre in Fiction4 separately, and the test for the FB corpus for comparison. For readability, we have divided the effect size by 100 here. Note that group sizes of the implicit/explicit sentences differ across the genre, with the smallest groups compared being fairy tales (𝑖 = 147/𝑒 = 317). Values in black: 𝑝 < .05, with*: 𝑝 < .01.
We want to thank everyone who contributed to this work, especially Mia Jacobsen, as well as colleagues and friends for pointing out pitfalls and sharing ideas.
Concret. Arousal Dominance Imag. Visual Haptic Interocept. EmoBank 2,205* 3,320* 2,940* 2,429* 2,528* 2,745 3,380* Letters 759* 1,142* 1,125* 797* 903 1,034 1,194* Fiction 2,137* 2,959* 2,362 2,223 2,312* 2,348 2,865* Blog
See https://github.com/centre-for-humanities-computing/literary_evocation for code and data.