We present a new task: the identification of the emotions transmitted in Italian opera arias at the verse level. This is a relevant problem for the organization of the vast repertoire of Italian Opera arias available and to enable further analyses by both musicologists and the lay public. We shape the task as a multi-class supervised problem, considering six emotions: love, joy, admiration, anger, sadness, and fear. In order to address it, we manually-annotated an opera corpus with 2.5k verses -which we release to the research community- and experimented with different classification models and representations. Our best-performing models reach macroaveraged F1 measures of ∼0.45, always considering character 3-grams representations. Such performance reflects the difficulty of the task at hand, partially caused by the size and nature of the corpus, which consists of relatively short verses written in 18thcentury Italian.
Opera lyrics have the function of expressing
the emotional state of the singing character.
In 17th- and 18th-century operas, characters
brought on stage passions induced in their
souls by the succession of events in the drama.
Musicological studies use these affects as one
of the interpretative keys of the work as a
whole
In order to do that we created the AriEmozione 1.0 corpus: a collection of 678 operas with 2.5k verses, each of which has been manually annotated with respect to emotion. We experimented with different supervised models (e.g., SVMs, neural networks) and text (e.g., character n-grams and distributed representations).
Our experiments show that, regardless of the model, character 3-grams outperform all other representations, reaching weighted macro-averaged F1 measures of ∼0.45. Underrepresented classes (e.g., fear) are the hardest to identify. Others, such as anger and sadness, being both negative, are often confused between each other.
The rest of the contribution is distributed as follows. Section 2 describes the AriEmozione 1.0 corpus. Section 3 describe the explored models and representations. Section 4 discusses the experiments and obtained results. Section 5 overviews some related work. Section 6 closes with conclusions and proposals for future work.
First of all, thank you for helping with this work. We are a group of researchers from the D. of Classical Philology and Italian Studies and the D. of Interpreting and Translation, both at UniBO. Your work will help us to produce artificial intelligence models to analyse the lyrics in music.
At this stage we are focused on opera. You will annotate arie in Italian from diverse periods, looking for the emotions that they express. Your work consists of identifying the emotion expressed in each of the verses composing an aria. You can choose among six emotions (or none of them), which are defined next: [. . . ] Each row is divided in six columns: id A unique id, tied to the verse. Do not modify it. verse A verse, inside of an aria. This is the text that you are going to analyse. emotion Here you can select the expressed emotion (or none of them) emotion sec. This is available to choose a secondary emotion, in case it is really difficult to choose just one confidence Not being 100% sure is ok. If that is the case, please let us know by choosing the right confidence level (default: “I am sure”). comments Feel free to tell us something about this instance, if you feel like. The corpus AriEmozione 1.0 is a subset of the materials collected by project CORAGO.1 AriEmozione 1.0 contains a selection of 678 operas composed between 1655 and 1765. We consider the lyrical text in the arias only. A. Zeno and P. Metastasio are among the most represented librettists in the corpus (∼ 30% of the operas); they are two of the most representative and prolific librettists of the 18th century. All texts are written in the 18th century Italian and articulated in verses and stanzas.
We labeled the emotions transmitted by
every single verse, as we observed that this is the
right granularity to obtain full text snippets
expressing one single emotion. Ren´e Descartes
wrote in 1649 “Les passions de l’ˆame”, a sort
of compendium of all possible emotions and
their possible causes
1CORAGO is the Repertoire and archive of
Italian opera librettos. It constitutes the first
implementation of the RADAMES prototype
(Repertoriazione e Archiviazione di Documenti Attinenti al
Melodramma E allo Spettacolo)
Rabbia (anger) incl. irritability, exasperation, rage, disgust, envy, torment.
Tristezza (sadness) incl. suffering, disappointment, shame, neglect, sympathy. Paura (fear) incl. horror and nervousness. An extra class nessuna (none) applies mostly to verses with non-actionable words only, neglected in the current experiments.
Two native speakers of Italian annotated all 2,473 instances independently considering the instructions displayed in Figure 1. They were asked to include (i) the emotion transmitted by the verse, (ii) an optional secondary label (in case they perceived a second emotion), and (iii) their level of confidence: total confidence, partial confidence, or very doubtful.
We measured the Cohen’s kappa
interannotator agreement
2The corpus is available at https://zenodo.org/ record/4022318.
3The code is available at https://github.com/ TinfFoil/AriEmozione. We used Sklearn for the kNN, SVM, and logistic regression models; Keras for the neural networks, and the Facebook-provided library for FastText (cf. https://scikit-learn. Model Settings k-NN L2-Norm exploring with k ∈ [1, . . . 9]. SVM RBF; both explored with c ∈ [1, 10, 100, 1000] and γ ∈ [1e − 3, 1e − 4].
Log Reg Multinomial Logistic Regression with
Newton-CG solver.
NN 2 (3) hidden layers with size ∈ [32, 64, 96, 128, 256] (∈ [8, 16, 32, 64, 96]); 20% dropout; ReLu for input/hidden layers; softmax for output layer; categorical cross-entropy loss function; Adam; epochs ∈ [1, . . . 15] FastText 300d embeddings with or without pretraining; learning rate ∈ [0.3, 0.6, 1]; epochs ∈ [1, 3, 5, 10, . . . , 100]
As for the text representations, we consider
TF–IDF vectors of both character 3-grams and
word 1-grams (no higher n values are
considered due to the corpus dimensions). For
preprocessing, we employ the spacy Italian
tokenizer4 and casefold the texts. We also explore
with dense representations, derived from the
TF–IDF vectors, by means of both LDA
4https://spacy.io/models/it model
representation kNN char 3-grams 0.38 words 0.36
LDA char 0.30 SVM–RBF char 3-grams 0.44 words 0.42
LDA char 0.28 Log reg char 3-grams words
LDA char 2-layers NN char 3-grams 0.42 words 0.42
LDA char 0.27 3-layers NN char 3-grams words
LDA char FastText char 3-grams 0.43 pre-trained chars 0.43 words 0.42 pre-trained words 0.38 test
Acc
In both cases, we target reductions to 16, 32,
and 64 dimensions. As for embeddings, we
adopted the pre-trained 300-dimensional
Italian vectors of FastText
We conducted several experiments to find the best combination of parameters and representations. Given the amount of instances available, we merged the training and development partitions and performed 10-fold cross validation. As standard, the test partition was left aside and only one prediction was carried out on it, after identifying the best configurations.
We evaluate our models on the basis of accuracy and weighted macro-averaged F1 measure to account for the class imbalance. Table 4 shows the results obtained with some interesting configurations and representations both for the cross-validation and on the test set.5 Character and word n-grams TF-IDF, LSA, and LDA were tested with all models except 5The full batch of results is available at https://docs.google.com/spreadsheets/d/ 1Ztjry2mJs6ufCZM1O5CQRyZ8pA5YDnToN0h0NGX1nW0/ edit?usp=sharing e mmariazion amore igaio paura braiba irttsezza ammirazione 0.37 0.03 0.18 0.07 0.11 0.06 amore 0.03 0.43 0.13 0.00 0.09 0.17 gioia 0.27 0.16 0.31 0.20 0.09 0.07 paura 0.10 0.03 0.00 0.40 0.02 0.07 rabbia 0.20 0.14 0.03 0.13 0.64 0.17 tristezza 0.17 0.14 0.13 0.07 0.19 0.48 for FastText, on which we test with and without pre-trained embeddings. Notice that we are not interested in combining features, but in observing their performance in isolation.
The most promising representation on crossvalidation appears to be the simple character 3-grams, with which we obtained the best results across all models; although it also features the highest variability across folds. Among all 3-gram derived representations, LDA consistently obtained the worst results across all models. Still, it is more stable across folds than the sparse 3-gram representation. As for fastText, with the same epoch number and learning rate, the character 3-gram vectors always achieved much higher accuracy than the word vectors.
Similar patterns are observed when projecting to the unseen test set. The character 3grams in general hold the best performance, while the 3-gram LDA tends to remain the worst in spite of the model used. This behavior does not hold in all cases. For instance, the logistic regression model achieves F1 =0.44 on cross-validation, but drops to 0.42 on test. This might be the result of over-fitting.
It is worth noting that all models tend to confuse rabbia and tristezza. Table 5 shows the confusion matrix for the best model on test. These two emotions get confused between each other on an average of 18% of the cases. The classifiers tend to confuse ammirazione for gioia as well, which is understandable given their semantic closeness. 5
Building on the numerous pre-existing
studies focusing on sentiment analysis
Balabantaray et al. (2012) tried to
distinguish among happy, sad, anger, disgust,
fear and surprise using WordNet
Affect
Promising work has been carried out on
news articles
To the best of our knowledge, no work in the field of either emotion or sentiment analysis has been performed on operas. 6
We addressed the novel problem of emotion classification of opera arias at the verse level. The task is interesting because of the lack of automated tools for the analysis of operas and challenging due to both the language used in 17th- and 18th-century lyrics and the complexity to produce the necessary amount of quality supervised data.
We explored with various classification models and representations. A neural network with two hidden layers fed with a simple TF-IDF character 3-gram representation is among the most promising approaches to the problem. Among the six possible emotions, the most difficult to identify are rabbia and tristezza, which tend to be confused with each other, followed by ammirazione, which is often confused by gioia. In order to foster the research on this topic, we release the AriEmozione 1.0 corpus to the community (cf. footnote 2).
As for the future work, we intend to
increase the size of the AriEmozione 1.0 corpus
by means of active learning
Another interesting alternative is the one highlighted by Zhao and Ma (2019), who adopted an efficient meta-learning approach to augment the learning ability of emotion distribution; i.e. the intensity values of a set of emotions within a single sentence, when the training dataset is small, as in the AriEmozione 1.0 corpus.
This research is carried out in the framework of CRICC: Centro di Ricerca per l’interazione con le Industrie Culturali e Creative dell’Universit`a di Bologna; a POR-FESR 2014-2020 Regione Emilia-Romagna project (https://site.unibo.it/cricc).
We thank Ilaria Gozzi and Marco Schillaci, students at Universit`a di Bologna, for their support in the manual annotation of the AriEmozione 1.0 corpus. Proceedings - First International Conference on Automated Production of Cross Media Content for Multi-channel Distribution. IEEE.