We present a case study as part of a work-in-progress project about multimodal sentiment analysis on historic German plays, taking Emilia Galotti by G. E. Lessing as our initial use case. We analyze the textual version and an audio version (audiobook). We focus on ready-to-use sentiment analysis methods: For the textual component, we implement a naive lexicon-based approach and another approach that enhances the lexicon by means of several NLP methods. For the audio analysis, we use the free version of the Vokaturi tool. We compare the results of all approaches and evaluate them against the annotations of a human expert, which serves as a gold standard. For our use case, we can show that audio and text sentiment analysis behave very differently: textual sentiment analysis tends to predict sentiment as rather negative and audio sentiment as rather positive. Compared to the gold standard, the textual sentiment analysis achieves accuracies of 56% while the accuracy for audio sentiment analysis is only 32%. We discuss possible reasons for these mediocre results and give an outlook on further steps we want to pursue in the context of multimodal sentiment analysis on historic plays.
Sentiment analysis is the area of research that deals with computational methods to
analyze and predict sentiments and emotions in written text
Sentiment analysis as a whole and especially the current research in the digital
humanities is predominately focused on the analysis of written text; other media
channels like audio, video or combinations are neglected so far. We propose several reasons
for justifying the broadening of the current focus on written text to other modalities:
First, although systematic performance evaluation in the context of narrative texts is
rare, the existing research shows that when compared to human sentiment annotations,
the accuracy can vary between 20-70% depending on the method and the type of text
Second, we also see potential for using differing media channels to improve the
annotation of sentiment for narrative texts. Literary texts have been proven to be very
difficult and tedious to annotate due to the historic and complex language
In this paper, we contribute to “multimodality in digital humanities” by presenting first work-in-progress results for multimodal sentiment analysis on narrative texts, more precisely for the specific use case of the play Emilia Galotti by G. E. Lessing. We have analyzed and compared existing text sentiment analysis approaches and a readyto-use audio speech sentiment analysis. In addition, we have evaluated the performance compared to a gold standard of annotations by a human expert. Finally, we discuss the results and the limitations of this case study but also formulate an agenda for future research in this area.
Sentiment analysis and emotion analysis are often differentiated in research. While
sentiment analysis is regarded as predicting and analyzing the overall affective state
predominantly with the classes positive, negative and neutral (we refer to these classes as
polarity), emotion analysis deals with more complex emotional categories like anger,
surprise or joy
As modalities for this study, we use the textual and an audio version (audiobook) of Emilia Galotti. We want to explore the following research questions:
RQ1: How do text sentiment analysis approaches perform compared to a ready-to-use audio sentiment analysis approach? We are focusing on simple and ready-to-use solutions, since not only is this study our first exploration in this area, but also because it is a common use case in digital humanities when the focus of research is not the development of new algorithms but the analysis of cultural artifacts. With RQ1, we want to gather first insights in the possible similarities and differences of both media channels and analysis types.
RQ2: How do text sentiment analysis approaches and the audio sentiment analysis perform against human expert annotations (with text)? We want to use the sentiment annotations of an expert as gold standard and evaluate the text and audio approach against each other. In our case study, the expert annotated the sentiment by being presented with the text. With RQ2, we want to test our assumption that audio speech conveys more precise emotional information and that audio sentiment analysis therefore achieves higher accuracies. 3
As use case for our investigations, we chose the play Emilia Galotti by G. E. Lessing (premiered 1772). The reason for this is that our recent research is focused on Lessing and Emilia Galotti is one of his most famous plays, which means that audio material for this play is available, too. All analyses are speech based, thus we are only comparing singular speeches with each other. A speech is a single utterance of a character separated by utterances of other characters beforehand and afterwards. Overall, the play consists of 835 speeches. The longest speech consists of 235 words while the shortest speech only has one word. On average, a speech in this play consists of 23 words.
As material for the textual sentiment analysis, we gathered an XML-annotated
version of the play from the platform Textgrid1. For the sentiment analysis, we employ two
lexicon-based approaches. A sentiment lexicon is a list of words annotated with
sentiment annotation. Based on simple calculation, text units can be assigned a polarity (e.g.
neutral, positive, negative). In previous work
For the audio analysis, we at first screened different available commercial and noncommercial audiobooks. However, we identified several problems: Those audiobooks use different speakers and a lot of additional sound effects and background music, which might be problematic for the sentiment analysis without larger preprocessing steps. Furthermore, audiobooks often differ strongly from the original text, leaving whole passages out or switching the order of speeches. We found the best material for this use case via semiprofessional readings of the original text on several platforms on the web. We used publicly available recordings from YouTube2. The reader is female and there are no deviations from the original text in the reading. Several piano pieces are included in the recording, but they are separate from the general reading.
For the audio sentiment analysis, we use the free version of Vokaturi3. Vokaturi is an emotion recognition software for spoken language with an easy-to-use API. GarciaGarcia et al. (2017) recommend Vokaturi as the best free software for spoken language sentiment analysis. Vokaturi is described as being language independent and it works via machine learning with two annotated databases. Vokaturi takes audio data of any length and outputs five values that range from 0 (none) – 1 (a lot) for the categories neutrality, fear, sadness, anger and happiness.
The workflow for the audio sentiment analysis is as follows: In a preprocessing step we trimmed and transformed the audio files from YouTube. We then performed forced alignment with the free Python library aeneas4. Forced alignment is a method to align text segments and audio speech and determine precise time stamps of when the text segments in the audio file start and end. As text segments we used the 835 speeches of the play. According to the time stamps, we segmented the audio file to get 835 separated audio files, which are finally used with Vokaturi. To map the output of Vokaturi to the nominal scale used for the textual sentiment analysis we employ a heuristic mathematical approach. First, we sum up all values for the negative emotions fear, sadness and anger to get a value for negative sentiment. We regard the value for happiness as positive sentiment. We then chose the maximum value of negative sentiment, positive sentiment and the value for neutrality as the final polarity of a speech. 2 The entire playlist of the files are available online:
https://www.youtube.com/playlist?list=PL06w7wmahre2eMZXDBgGcR2mgAoeI_f8k 3 Available online: https://developers.vokaturi.com/getting-started/overview 4 https://github.com/readbeyond/aeneas negative neutral positive Text: Naive
The human annotations we use were gathered with an expert literary scholar who
annotated 200 random speeches of Emilia Galotti. The annotator was presented the
speech to be annotated, the predecessor and successor speech for context and then had
to annotate the polarity as rather positive, neutral or rather negative. The annotator was
instructed to annotate the polarity he feels is most connoted with the speech. Note that
the speeches were solely presented in textual form. The annotator was a male student
of German literary studies who had to write a thesis about Emilia Galotti during the
annotation process and can therefore be regarded as an expert for this specific play. We
restricted the annotation to 200 speeches since sentiment annotation in this context has
been proven to be very tedious and challenging
We first report all results concerning the comparison of the text and audio sentiment analysis among all 835 speeches (RQ1). Table 1 shows the overall polarity distribution of all methods. The majority of the assignments of the naive approach are neutral while the majority (51%) of all speeches are assigned as positive by the audio sentiment analysis (see Table 1). Therefore the proportion of similarly assigned speeches is rather low (32%); 266
Text:
Optimized speeches are assigned the same class. Table 3 shows the same cross table but with the optimized lexicon-based approach. The optimized lexicon approach predicts that almost half of the speeches are negative while the audio sentiment analysis approach produces opposite results with half of the speeches being assigned as positive (51%; see Table 1). Consequently, a small number of 253 speeches are assigned the same class. This results in 30%.
For RQ2 we regard the 200 human annotated speeches and the performance of all approaches compared to the human annotations. First, Table 4 shows the polarity distributions of all methods on this subset of speeches. The polarity distributions for the different computational methods on the subset of the annotated corpus are similar to the distributions on the entire corpus. Most of the speeches were annotated as negative by the expert annotator (46%). With the following cross tables, we show the agreements of the computational methods and the human annotations. The human annotations are used as gold standard to evaluate the approaches. The accuracy is the proportion of correctly predicted speeches among all speeches. First, Table 5 and 6 show the text-based sentiment analysis results.
Sum 92 60 48 200 Taking the human annotation as benchmark, both textual approaches perform almost similarly: The naive approach achieves an accuracy of 52% (104 speeches) and the optimized approach 56 % (112 speeches). Both approaches are over the random baseline (approx. 36%) and slightly above the majority baseline (46%). With the human annotation as gold standard, the audio sentiment analysis achieves an accuracy of 31% with 62 speeches being predicted correctly. This is below the random baseline. The main difference is that the human annotator chose negative annotations for most of the speeches while the audio sentiment analysis in contrast predicts positive sentiment for the majority of times. 5
With the first research questions, we analyzed and compared basic text and audio sentiment analysis approaches. We identified that the sentiment analysis on both, text and audio, produces very different results with the optimized text analysis predicting the majority of speeches as negative and the audio speech sentiment analysis in contrast predicting the majority as positive, more specifically as connoted with the emotion happiness. For our specific use case, we were able to show that both channels seem to work on very different levels and with very different features, which exemplifies that the inclusion of other media channels leads to new insights, in this case even contradictory results.
We assume that there might be some specifics of the audio material that lead to
this tendency of positive sentiment assignments. Firstly, pitch plays an important factor
in the prediction process
When evaluating the computational approaches against a gold standard of a human
annotated subset of 200 speeches the general problems and limitations of sentiment
analysis on literary texts are apparent. The textual approaches as well as the
audiobased method perform rather poorly: the textual approach is just slightly above the
majority baseline, the audio approach is below the random baseline. This is in line with
current research on sentiment analysis on literary texts
Our assumption that the audio speech analysis might improve the results has been proven wrong on the chosen material. The text sentiment analysis performs far better on the gold standard. The reason for this is predominately that the human annotator as well as the text analysis assign the majority of speeches as negative while the audio sentiment analysis behaves contrarily.
To put these results in perspective one should also note that audio sentiment analysis
is in general known as being more challenging than other media channels like text and
facial expressions via video and performs often lower than those
In future work, we want to investigate how the audio speech influences the human annotation. Furthermore, bear in mind that the performance comparison is not the main goal of our efforts. In future research, we rather want to explore possibilities of combining multiple media channels to improve performance results.
Overall, there are several limitations in this case study we want to address in future research. The presented study represents only a singular use case and a work-in-progress project. Onwards, we want to analyze more examples like different audio material of one play and in general plays of other writers and eras. We also want to explore different audio sentiment analysis approaches. The usage of ready-to-use solutions like Vokaturi in general sentiment analysis research is rather rare; usually an individual machine-learning algorithm is implemented. For this specific reason, we plan several annotation studies to acquire annotated audio material on a large scale. In this context, we also want to explore how the audio or video presentation of speeches can improve the sentiment annotation process in this area.
Furthermore, we also want to include the media channel of video especially facial emotion recognition in our research via video recordings of theatrical performances of the plays. The inclusion of the oral but also facial expression of an actor provide necessary interpretation channels for a holistic understanding of the sentiment and emotion in a play. With these future plans, we want to continue our work towards a multimodal sentiment analysis and explore possibilities for the technical performance, the annotation process and the interpretation in literary studies.