Sentiment Analysis has long been a topic of interest in natural language processing and computational literary studies, where it can be used to infer the relationships between fictional characters. Building on the dataset and results of Kim and Klinger (2019), we propose a classifier based on BERT that improves the results reported therein and show that we can use this classifier to determine the relation between characters in Harry Potter novels. Our proposed sentiment classifier yields an F1score of up to 75 % for binary classification of emotions. Aggregating these emotions over novels, we reach an F1-score of up to 68 % for the classification of a pair of characters as friendly or unfriendly.
Characters and their relations are one of the basic
building blocks of stories
Recently, the trend in NLP has been to use large
transformer models that have been pre-trained for
language modelling (or similar tasks not requiring
explicit annotations) on enormous datasets. We
follow this trend by fine-tuning BERT
Our contribution is two-fold: 1. We generally
improve results on the emotion classification tasks
from Kim and Klinger (2019). 2. We track the
emotional relations detected by our classifier over
the course of a novel and describe an easy method
to aggregate them to an overall label. We evaluate
this method on the text of the well-known Harry
Potter series
The remainder of this paper is structured as follows: After giving a short introduction, we next present related work. In chapter 3 we describe our approaches towards emotion and relation classification as well as our results. We conclude this paper with a discussion of results and some possible directions for future work Our work is situated at the intersection of sentiment analysis and social network extraction.
Character networks for works of fiction have
been studied extensively in recent years
For sentiment analysis, most work has focused
on short, self-contained texts like tweets
However, most of these works employ rather simple sentiment analysis methods (e.g., Zehe et al. (2016) rely on a simple lookup in a sentiment lexicon). Most similar to our work is Kim and Klinger (2019), which we directly build upon. The authors propose a new corpus of short pieces of text annotated with the emotional relations between characters described in these texts. They train a GRU (Cho et al., 2014) neural network to predict the emotions based on this corpus, showing promising results with F1 scores up to 67 % for undirected binary classification (positive and negative emotions) and 46 % for 5 basic emotions in the story-level evaluation as described below. We extend this work by improving the sentiment analysis model and aggregate the instance-level labels for full novels. 3
We address two tasks in this paper: mention-level emotion classification and story-level relation classification, which we see as two steps in a pipeline.
Klinger (2019), we define emotion classification as learning a classifier that, given a short piece of text (roughly one sentence) containing two characters, predicts the emotion described therein. We perform this task on different granularity levels, using either 2, 5 or 8 directed or undirected emotions.
sification as an aggregation of emotions discovered by step 1 over a novel. In this paper, we distinguish between “friendly” and “unfriendly” relations. 3.1
BERT-model
Emotion Classification For the first task, we use the dataset provided by Kim and Klinger (2019) and refer to this paper for a detailed description due to space constraints. The dataset consists of 1335 samples2, each annotated according to multiple schemes. These schemes differ in the number of emotions that are annotated (two, five or eight) and whether the emotions are directed (from a causing to an experiencing character) or undirected. Relation Classification For the second task, we have collected our own dataset. To this end, we used BookNLP on all books from the Harry Potter series to extract all interactions as described in Section 3.1. In contrast to the first dataset, we use automatically extracted characters and co-references here. We then manually annotated all pairs of characters for which we found interactions with their relationship, distinguishing between friendly and unfriendly relationships. We collected two sets of independent annotations and, where the two annotators disagreed, collected a third annotation as a tie-breaker. The tie-breaker was given the option to note that there is no (clear) relation between the two characters. This was the case in the third novel for the relation between Harry and Sirius Black (cf.
HP1 HP2 HP3 HP4 HP5 HP6 HP7
Section 4). Table 1 provides details for the resulting dataset, which we publish for future research.3 Emotion Classification We follow the evaluation setup from Kim and Klinger (2019) for emotion classification, who use multiple settings: The dataset (cf. section 3.2) provides annotations for sets of two, five and eight directed or undirected emotions. Additionally, they define different ways of representing the entities involved in the emotions, where some add a marker to entities or completely mask them (making it impossible for the model to learn that, e.g., Harry always interacts positively with Ron). We describe these schemes shortly in the following and give an example for how sentences would be represented according to each scheme:
No-indicator: Entities are represented as in the text, the model is directly fed the unmodified sentence (e.g., Alice is angry with Bob).
Role: Entities are marked as causing or experiencing (e.g., <e>Alice</e> is angry with <c>Bob</c>), where <e> marks the experiencing character and <c> the causing character.
MRole: Entities are only identified by their role (<e> is angry with <c>), <e> and <c> as above.
Entity: Entities are marked as entities with no indication as to whether they cause or experi21742 overall, but following Kim and Klinger (2019) we use only the subset annotated with a causing character 3http://professor-x.de/datasets/ harrymotions. ence the emotion (e.g., <et>Alice</et> is angry with <et>Bob</et>). MEntity: Entities are masked by entitymarkers (e.g., <et> is angry with <et>).
Relation Classification In our second experiment, we use the emotions detected in the previous step to detect overall relationships between characters in the Harry Potter series by aggregating over emotions as described in Section 3.1. In Table 3, we report macro-averaged F1-scores as well as accuracies for aggregating emotions as classified in the Entity and MEntity settings for 2 and 5 emotion classes, since we do not have role labels for the Harry Potter corpus and the emotion classification for 8 emotions did not perform well. Note that the number of emotions only pertains to the emotion classification setting, relations are always classified as friendly or unfriendly. For the 5 class setting, we define anger, disgust and sadness as negative emotions, joy as positive and anticipation as neutral. The parameter was optimised on hp1 and is set to 0:4, except for 5-MEntity ( = 0:75). Lacking a directly comparable approach, we report sampling from the true label distribution per novel as a baseline (which performs better than majority vote in our setting). We find that, on average, 2 classes lead to better results and we always outperform the baseline. 4
In this section, we discuss our findings along with some of the decisions involved in the dataset collection and provide some insight regarding the development of emotions over the course of a novel. BERT vs. GRU Our BERT-based classifier outperforms the GRU in all undirected, but not all directed settings. Specifically, in the 8-class directed evaluation, the GRU usually performs better than BERT. We hypothesise two possible reasons: a) the rather low amount of training data available for each of the 8 emotion classes, especially in the directed case. We assume that the GRU’s lower number of parameters makes it easier to tune on fewer samples. b) BERT is a bi-directional model, while the GRU used here is uni-directional. Since the GRU reads sentences in the right order, while BERT reads in both directions, it might be easier for the GRU to model directed relations.
Dataset Collection As mentioned in Section 3.2, we excluded some relations during the annotation process. This is due to two reasons: a) errors in named entity recognition and b) changing relationships. For the first category, BookNLP returned the entity “Felix Felicis”, which is a luck potion. We excluded all relationships involving the potion, but kept collective entities like ”Hogwarts”. In the second category we find the relationship between Sirius Black and most other characters in the third novel. For the majority of the book, Sirius is regarded as a villain intent on killing Harry, which is revealed to be wrong at the end of the novel, turning the relation very positive. Since the label here is unclear, we excluded it from the dataset. Developing Relations As described before, relationships can change drastically within a novel. Two prominent examples of this in the Harry Potter novels are the relations between Harry and Hermione in the first novel (where they become friends) and between Harry and Sirius Black in the third novel (see prev. paragraph). We can use the emotions detected by our classifier to plot a trajectory over the novel. The polarity for characters a and b in chapter i is then calculated as pi = pi 1 + posa;b;i nega;b;i, where pos/nega;b;i counts positive/negative emotions between a and b in chapter i, respectively, and p0 := 0. We show plots for three examples in Figure 1, using predictions from the 2-class MEntity classification. In all cases, the trajectory matches our expectation: For Harry and Hermione, the relation starts neutral with a very clear upper trend after they become friends. For Ron, the relation quickly becomes very positive. For Sirius, the relation is mostly negative, while improving clearly in the final chapters. 5
We have presented an improved approach for the classification of emotional relations between fictional characters. By aggregating sentence level emotions, we have built a classifier for novelwide character relations based on emotion analysis. While our experiments show that aggregation yields promising results, future work includes
8c 5c 2c
Directed 5c the development of a stronger classifier for storylevel relations. We also plan on investigating the influence of co-reference resolution, which is currently done automatically. Using manual labels or improved co-references resolution should further improve our results: First experiments indicate better performance for frequent characters, where resolution errors are more easily smoothed out.
Many thanks to Darleen Pappelau for helpfully providing the tie breaker annotations for the dataset. dynamic relationships between characters in literary novels. Thirtieth AAAI Conference on Artificial Intelligence.