-

Sentiment Annotation for Lessing's Plays: Towards a Language Resource for Sentiment Analysis on German Literary Texts

Christian

0 1 2 3 4 0 Burghardt, Manuel Computational Humanities, University of Leipzig, Germany https://ch.uni-leipzig.de/ 1 Dennerlein, Katrin University of Würzburg , Germany https:// 2 Media Informatics Group, University of Regensburg , Germany 3 Schmidt, Thomas Media Informatics Group, University of Regensburg , Germany 4 Schmidt , T., Burghardt, M., Dennerlein, K. and Wolff, C.; licensed under Creative Commons License CC-BY LDK 2019 - Posters Track. Editors: Thierry Declerck and John P. McCrae

We present first results of an ongoing research project on sentiment annotation of historical plays by German playwright G. E. Lessing (1729-1781). For a subset of speeches from six of his most famous plays, we gathered sentiment annotations by two independent annotators for each play. The annotators were nine students from a Master's program of German Literature. Overall, we gathered annotations for 1,183 speeches. We report sentiment distributions and agreement metrics and put the results in the context of current research. A preliminary version of the annotated corpus of speeches is publicly available online and can be used for further investigations, evaluations and computational sentiment analysis approaches. 2012 ACM Subject Classification Document preparation → Annoation; Retrieval tasks and goals → Sentiment analysis

and phrases Sentiment Annotation Sentiment Analysis Corpus Annotation Annotation Behavior Computational Literary Studies Lessing

Sentiment analysis typically makes use of computational methods to detect and analyze sentiment (e.g. positive or negative) in written text [7, p.1]. Emotion analysis, a research area very closely related to sentiment analysis, deals with the analysis of more complex emotion categories like anger, sadness or joy in written text. More recently, sentiment and emotion analysis have also gained attention in computational literary studies [15] and are used to investigate novels [3, 4], fairy tales [ 1, 8 ] and plays [8, 9, 11, 12, 13]. However, there are only few resources available for sentiment analysis on literary texts, since sentiment annotation of literary texts has turned out to be a tedious, time-consuming, and generally challenging task [5, 14, 15], especially for historical literary texts, with their archaic language and complex plots [14]. Previous studies have shown that agreement levels among annotators of sentiment annotation are rather low, since literary texts can be understood and interpreted in a number of ways, i.e. annotations are very much dependent on the subjective understanding of the annotator [14, 15]. On the technical side, most computational approaches for sentiment analysis on literary texts currently employ heuristic and rule-based methods [4, 8, 9, 11, 12, 13]. For more advanced machine learning tasks as well as for standardized evaluations, wellcurated sentiment-annotated corpora for literary texts would be an important desideratum, but in practice they are largely missing.

We want to close this gap for the genre of German historical plays, more precisely plays of the playwright G. E. Lessing (1729-1781), by creating a corpus annotated with sentiment information. Since this is a very tedious task, we first want to investigate the suitability of different user groups with different levels of knowledge for literary works. We want to find out how important domain knowledge (experts, semi-experts and non-experts) is for this kind of task, since the acquisition of domain experts (literary scholars) is obviously more difficult than the acquisition of annotators without any formal literary training, who could possibly be hired on a large scale via a crowdsourcing platform. In this article we present first results from an annotation study with a group of semi-experts. 2

Design of the Annotation Study

We conducted the annotation study for semi-experts as part of a course in the Master’s program of German Literature at the Würzburg University. The course "Sentiment Analysis vs. Affektlehre" was focused on the plays of Lessing and revolved around the comparison of more traditional sentiment and emotion analysis approaches with recent computational methods [11, 12]. Each student had to write an essay and a presentation for a specific play. In addition, students were asked to participate in the sentiment annotation study. For this study, we chose six of the most famous plays by Lessing. Every student had to annotate 200 randomly selected speeches except for the students assigned with Damon, since this play only consists of 183 speeches. A speech is a single utterance of a character, typically separated by utterances of other characters beforehand and afterwards. A speech can consist of one or multiple sentences. For each of the six plays we asked two independent students to provide sentiment annotations. Overall, nine students participated in the annotation project. One student, who was an advanced tutor, conducted annotations for multiple plays. The entire corpus comprises 1,183 speeches and we acquired 2,366 annotations (2 annotations per speech). The corpus consists of 3,738 sentences and 44,192 tokens. A speech consists on average of 3.16 sentences and 37.36 tokens.

The overall process was very similar to [14]: Students received an annotation instruction and guidelines of three pages length, explaining the entire annotation process with examples. The annotation material was provided via Microsoft Word, which was also used as a basic annotation tool during the annotation process. Conducting annotation studies with well known products like Word or Excel is not uncommon in Digital Humanities projects, as humanities scholars are typically familiar with standard software tools [2, 14]. Every student received a file with the play-specific speeches. A speech was presented with the following information: (1) speaker name, (2) text of the speech and (3) position of the speech in the entire play. Furthermore, the predecessor and successor speech were presented to give some context information for the interpretation of the actual speech. Sentiment annotations were documented in a predefined table structure. Annotators were asked to write down the overall sentiment of an entire speech rather than the sentiment for specific parts of that speech. In case there were multiple, possibly conflicting sentiment indicators in one speech, annotators documented the most dominant sentiment for that speech.

Students were asked to annotate one of the following classes in the first table: negative, positive, neutral, mixed, uncertain, and other (we refer to this scheme as differentiated polarity). If annotators did not choose negative or positive, they were asked to provide a tendency for one of those two classes (binary polarity). Figure 1 illustrates the annotation process via an example annotation. Here the annotator chose neutral for the differentiated polarity and negative for the binary polarity.

We also gathered annotations about the reference of the sentiment and the certainty of the annotations to get more sophisticated insights about the annotation behavior. However, we will only focus on the polarity in the upcoming sections. Students had a month to complete the annotations, but other than that could freely organize the time to complete the task by themselves. In the next section, we report some of the major findings from our annotation study. 3

Results

First, we present the distributions concerning the differentiated polarity for the entire corpus and all 2,366 annotations (table 1):

Negative 734 (31%)

Positive 442 (19%)

Neutral 540 (23%)

Mixed 405 (17%)

Uncertain 235 (9%)

Other 10 (0.4%)

Most of the annotations are negative, and a significant number of annotations are mixed and uncertain. These results are in line with current research about sentiment annotation of literary texts [ 1, 14 ], which seem to have a general tendency toward more negative annotations. To analyze the agreement among annotators we used Cohen’s Kappa and the average observed agreement(AOA, number of agreements divided by the total number of annotated speeches) as basic metrics. According to [6], Kappa metrics can be interpreted like this (table 2): Kappa value <0.20 0.21-0.40 0.41-0.60 0.61-0.80 0.81-1.00

Interpretation Poor agreement Fair agreement Moderate agreement Substantial agreement

Very good agreement

As described above, we had the annotators document differentiated (6 values) as well as binary (2 values) polarity values for each of the speeches. In addition, we also inferred a third variant from those annotations, which we call threefold polarity. We basically extend the binary polarity metric by the value "neutral" in the following manner: if the value "neutral" is annotated at the differentiated polarity, this value is taken over in this variant. For any other case the value "positive" or "negative" is taken over from the binary polarity annotation. We present these metrics per play along with average values for the entire corpus (table 3):

Play

Damon Emilia Galotti

Der Freigeist Minna von Barnhelm

Nathan der Weise Miß Sara Sampson

Overall average

Annotation type Differentiated polarity

Threefold polarity