In this paper we present a set of annotated data and the results of a number of unsupervised experiments for the analysis of sentiment in Latin poetry. More specifically, we describe a small gold standard made of eight poems by Horace, in which each sentence is labeled manually for the sentiment using a four-value classification (positive, negative, neutral and mixed). Then, we report on how this gold standard has been used to evaluate two automatic approaches for sentiment classification: one is lexicon-based and the other adopts a zero-shot transfer approach.1
The task of automatically classifying a (piece of) text according to the sentiment conveyed by it, known as Sentiment Analysis (SA), is usually performed for purposes such as monitoring contents of social media or evaluating customer experience, by analysing texts like tweets, comments, and micro-blogs.
A still under-investigated yet promising research area where developing and applying SA resources and techniques is the study of literary texts written in historical and, particularly, Classical languages (e.g. Ancient Greek and Latin). Actually, investigating the lexical properties of Classical literary texts is a century-long common practice. However, such investigation can nowadays
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
1This paper is the result of the collaboration between the
four authors. For the specific concerns of the Italian academic
attribution system, Rachele Sprugnoli is responsible for
Sections 2, 3, 4.2, 5; Marco Passarotti is responsible for Section
1; Francesco Mambrini is responsible for Section 4.1.
Giovanni Moretti developed the zero-shot classification script.
(1) lead to replicable results, (2) benefit from
techniques developed for analysing the sentiment
conveyed by any type of text and (3) be performed
with freely available lexical and textual resources.
As for the latter, the research area dedicated to
building and using linguistic resources for
Classical languages has seen a substantial growth
during the last two decades
In this paper we describe the use of LatinAffectus to perform SA of the Odes (Carmina) by Horace (65 - 8 BCE). Written between 35 and 13 BCE, the Odes are a collection of lyric poems in four books. Following the models of Greek lyrical poets like Alcaeus, Sappho, and Pindar, the Odes cover a wide range of topics related to the individual and social life in Rome during the age of Augustus, like love, friendship, religion, morality, patriotism, the uncertainty of life, the cultivation of tranquility and the observance of moderation. In spite of a rather lukewarm initial reception, the Odes quickly became a capital source of influence, in particular as a model of authorial voice and identity.3 Considering not only the importance of the Odes in the history of Latin and European literature, but also the diversity of the contents and tones of the poems collected therein, we argue that performing SA on such work can lead to interesting results and might represent a use case to open a discussion about the pros and cons of applying SA techniques and resources to literary texts written in ancient languages.
All data presented in this paper are publicly released: https://github.com/CIRCSE/La tin Sentiment Analysis . 2
The majority of linguistic resources and
applications in the field of SA involve non-literary and
non-poetic texts, such as news and user-generated
content on the web
Available annotated corpora of poems differ from each other from at least four points of view: annotation procedure (either involving experts or using crowdsourcing techniques), unit of analysis (verse, stanza, whole poem), granularity of classiifcation (from binary classes, such as positive and negative, to wide sets of emotions), foci of the emotions (annotation of the emotions as depicted in the text by the author or as felt by the reader). With respect to previous work, in this paper we chose to involve experts, to perform annotation at the sentence level (as an intermediate degree of granularity between verse and stanza), to assign four generic classes without defining the specific emotion conveyed by the text, and to focus on the sentiment as depicted by the author.
3For an orientation on the vast subject of the fortune and reception of the Odes see Baldo (2012).
4For a recent survey on sentiment and emotion analysis applied to literature, see Kim and Klinger (2018).
As for automatic classification systems, the
literature reports both lexicon-based
The Gold Standard (GS) consists of eight randomly selected odes,5 two from each of the four books that make up the work, for a total of 955 tokens, without punctuation, and 44 sentences (average sentence length: 21, standard deviation: 11). Texts were taken from the corpus prepared by the LASLA laboratory in Lie`ge.6 We performed a single-label annotation of the original Latin text by Horace at sentence level. We have chosen the sentence as unit of annotation because it represents an intermediate degree of granularity between that of the verse and that of the stanza. In fact, the limited length of a verse can hinder the full understanding of the sentiment it conveys, while a stanza, being longer, risks to contain very different content and thus, potentially, even opposite sentiments. Furthermore, not all poems can be divided into stanzas, as this depends on the metric scheme of the poem. Instead, sentences can be detected in every poem regardless of its metric scheme, and represent a unit of meaning in their own right.
In the annotation phase, we involved two experts in Latin language and literature (A1 and A2) and another annotator with basic knowledge of Latin but provided with previous experience in sentiment annotation (A3). Annotators were asked to identify the sentiment conveyed by each sentence in the GS, taking into consideration both the vocabulary used by the author and the images that are evoked in the ode. More specifically, annotators were asked to answer the following question: which of the following classes best describes how 5Book I: odes 10 and 17; Book II: odes 7 and 13; Book III: odes 13 and 23; Book IV: odes 7 and 11.
6http://web.philo.ulg.ac.be/lasla/oper a-latina/. are the emotions conveyed by the poet in the sentence under analysis? • positive: the only emotions that are conveyed at lexical level and the only images that are evoked are positive, or positive emotions are clearly prevalent; • negative: the only emotions that are conveyed at lexical level and the only images that are evoked are negative, or negative emotions are clearly prevalent; • neutral: there are no emotions conveyed by the text; • mixed: lexicon and evoked images produce opposite emotions; it is not possible to find a clearly prevailing emotion.
The annotation of the GS was organized in four
phases. In the first phase, annotators worked
together collaboratively assigning the sentiment
class to four of the eight odes (21 sentences): the
task was discussed and a common procedure was
defined. In the second phase, annotators worked
independently on the other four odes (23
sentences): A1 and A2 annotated the original Latin
text, while A3 annotated the same odes using an
Italian translation
Cohen’s k between A1 and A2 resulted in 0.5, while Fleiss’s k among the three annotators (A1A2-A3) resulted in 0.48 (both these results are considered moderate agreement). In particular, the negative class proved to be the easiest to be annotated (with a Fleiss’s k of 0.64), followed by neutral (0.57) and positive (0.45), whereas mixed was the most problematic class (0.23).
We noticed that the Italian translation was sometimes misleading, resulting in cases of disagreement: e.g., the sentence inmortalia ne speres monet annus et almum quae rapit hora diem, (ode IV, 7) is translated as ‘speranze di eterno ti vietano gli anni e le ore che involano il giorno radioso’ (literal translation of the Italian sentence into English: ‘hopes of eternity forbid you the years and the hours that steal the radiant day’). A3 marked this sentence as mixed, considering that it is impossible to identify a prevailing emotion between the negativity expressed by the verb ‘vietare’ (‘to forbid’) and the positivity of ‘giorno radioso’ (‘radiant day’). However, the translation of the Latin verb rapio is not appropriate: the Italian verb ‘involare’ (‘to steal’) does not convey the idea of the violent force inherent in rapio, which can be more correctly translated with the verb ‘to plunder’.7 3.3
Disagreements were discussed and reconciled by the three annotators: Table 1 presents the number of sentences and tokens per sentiment class. Our GS includes a majority of positive sentences (45.4%). Positive (average length: 21, standard deviation: 11), negative (average length: 24, standard deviation: 14), and mixed (average length: 25, standard deviation: 9) sentences are considerably longer than neutral ones (average length: 8, standard deviation: 3). Annotated examples are given in Table 2: English translations by Kaimowitz et al. (2008) are included for clarity. positive negative neutral mixed TOTAL The dataset for this experiment is obtained by means of a simple dictionary lookup of the lemmas in the LatinAffectus sentiment lexicon. Entries in the lexicon are assigned a score of: -1.0, -0.5 (negative polarity), 0 (neutral polarity), +0.5, +1.0 (positive polarity). The tokens in the Odes that are lemmatized under lemmas that also have an entry in the LatinAffectus are assigned the score that is found in the lexicon. For instance, the adjective malus ‘bad’ is found with a polarity value of -1.0 in LatinAffectus. All tokens lemmatized as malus (adj.) are thus given a score of -1.0. Note 7See for instance the English translation by Kaimowitz et al. (2008): “Do not hope for what’s immortal, the year warns, and the hour which plunders the day”. hic tibi copia manabit ad plenum benigno ruris honorum opulenta cornu cuncta manus auidas fugient heredis amico quae dederis animo frigora mitescunt Zephyris uer proterit aestas interitura simul pomifer autumnus fruges effuderit et mox bruma recurrit iners
Here for you will flow abundance from the horn that spills the country’s splendors All that you bestow upon your heart escapes the greedy hands of an heir With the Zephyrs cold grows mild, summer tramples springtime, soon to die, once productive autumn pours forth its fruits, and shortly lifeless winter is back Who will Venus name as master of the wine? Class positive negative mixed neutral 2.7 235 quem Venus arbitrum dicet bibendi that a score of 0.0 is assigned to both words expressly annotated as neutral in LatinAffectus and to those that do not have an entry in the lexicon.
The dictionary lookup required some manual disambiguation in cases of ambiguity due to homography. For 18 lemmas (corresponding to 49 tokens in the Odes), the sentiment lexicon provides multiple values; in most cases, as with ales ‘winged’ (adj.), but also ‘bird’ (n.), the variation is due to a different polarity attributed to the syntactic uses of the word (in the example, to the adjective and the noun). In such cases, the PoS annotation in the LASLA corpus was used to disambiguate and assign the correct score. We also reviewed those words that, although not tagged as nouns or adjectives in LASLA, still yield a match in LatinAffectus. After revision, we decided to keep the scores for a series of lemmas annotated as numerals in the corpus (simplex ‘simple, plain’, primus and primum ‘first’, prius ‘former, prior’) and the indefinite pronoun solus ‘alone, only’ that in LatinAffectus are marked as adjectives.
A sentence score (S) was computed by summing the values of all words. Thus, we attributed the label positive to all the sentences with score S > 0 and negative where S < 0. For S = 0, we attributed neutral to sentences where all words had a score of 0 and mixed where positive and negative words were equivalent. The overall accuracy of this method is 48% (macro-average F1 37, weighted macroaverage F1 44) with unbalanced scores among the four classes: 70% for positive, 42% for negative, 67% neutral, while no correct predictions were given for mixed. 4.2
We trained a language model for SA on English
and tested it on our GS by relying on two
stateof-the-art multilingual models. More specifically,
we fine-tuned Multilingual BERT (mBERT)
9For the full mapping, please see: https://github .com/google-research/google-research/blo b/master/goemotions/data/sentiment mappi ng.json.
GoEmotions social media AIT-2018 social media Poem Sentiment literary - poetry MultiEmotions-It social media AriEmozione literary - opera Horace GS literary - poetry following hyperparameters: 32 for batch size, 2e-5 for learning rate, 6 epoches, AdamW optimizer.10
We evaluated the trained model on different
datasets, including our GS. For each of the
following test sets, we randomly selected 44 texts so to
have the same number of input data as in our GS:
• GoEmotions: test set taken from the same
corpus used for training the English model.
• Poem Sentiment: collection of English verses
annotated with the same sentiment classes as
in our GS
10We adapted the following implementation: https://
gist.github.com/sayakmisra/b0cd67f406b4e
4d5972f339eb20e64a5.
• MultiEmotions-It: a multi-labeled emotion
dataset made of Italian comments posted on
YouTube and Facebook
Table 3 reports the results of mono-lingual and cross-lingual classification for the different datasets briefly described above and for the two pre-trained multilingual models. There is no clear prevalence of one model over the other: results vary greatly from one dataset to another. On the same language (thus without zero-shot transfer), we notice a drop in the performance for both mBERT and XML-RoBERTa when moving from Reddit comments, that is the same type of text as the training data, to tweets, but even more so when they are evaluated on poems. As for the zero-shot classification, results on Italian YouTube and Facebook comments are better than the ones registered on English tweets, but accuracy drops when applied to opera verses. However, the worst results are recorded for Latin with an accuracy equal to, or slightly above 30% (for mBERT: macro-average F1 29, weighted macro-average F1 35; for XML-RoBERTa: macro-average F1 24, weighted macro-average F1 26). For both mBERT and XML-RoBERTa, we register the same trend at class level: perfect accuracy for neutral, good accuracy for negative (50% with mBERT and 67% with XML-RoBERTa), low accuracy for positive (25% with mBERT and 10% with XML-RoBERTa) and no correct predictions for mixed. 5
In this paper we have presented a new GS, made of odes written by Horace, for the annotation of sentiment in Latin poetry. The extension of the manually annotated dataset is one of our future work: the goal is to have a sufficient amount of data to test supervised systems. We have also experimented two different SA approaches that do not require training data: both of them are not able to correctly identify sentences with mixed sentiments, which, in any case, are the most problematic also for human annotators. Table 4 reports a comparison in terms of precision, recall and F1score among the lexicon-based approach and the zero-shot classification experiments with both the mBERT and the XML-RoBERTa models. The former performs better on the positive class whereas the zero-shot method achieves a higher F1-score on the negative one even if this class is not the most frequent in the training data. Both mBERT and XML-RoBERTa obtain a very high precision on the sentences marked as positive (0.83 and 1.00 respectively) but the recall is extremely low (0.25 and 0.10 respectively). On the contrary, for the neutral class, the recall is perfect (1.00 for both models) but the precision is very low (0.10 and 0.11 respectively).
A manual inspection of the output of the lexicon-based method revealed two main problems of that approach: i) the limited coverage of LatinAffectus and ii) sentiment shifters are not properly taken into consideration. As for the first point, LatinAffectus covers the 43% of nominal and adjectival lemmas in the GS, leaving out lemmas with a clear sentiment orientation. To overcome this issue, we are currently working on the extension of the lexicon with additional 10,000 lemmas. Regarding the sentiment shifters, their impact is exemplified by the following sentence: cum semel occideris et de te splendida Minos fecerit arbitria non Torquate genus non te facundia non te restituet pietas (‘When you at last have died and Minos renders brillant judgement on your life, no Torquatus, not birth, not eloquence, not your devotion will bring you back.’ - ode IV, 7). Here, the sentiment score calculated by the script is very positive (3) because it does not handle the frequent negations: however, the particle non should reverses the positive polarity of facundia ‘eloquence’ and pietas ‘devotion’. This problem could be mitigated by modifying the script with rules that take into account negations and their focus.
Regarding the zero-shot classification approach, the very low performances on Latin deserve further investigation. It is possible that the problem lies in the data used to build the pre-trained models: i.e., Wikipedia for mBERT and Commoncrawl for XML-RoBERTa. Both resources were developed by relying on automatic language detection engines and are highly noisy due to the presence of languages other than Latin and of terms related to modern times. An additional improvement may also come from using for finetuning an annotated in-domain corpus in a wellresource language, that is a corpus of annotated poems: unfortunately, the currently available corpora are not big enough for such purpose.
This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme – Grant Agreement No. 769994. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Paris, France, may. European Language Resources Association (ELRA).