Identifying plot structure in novels is a valuable step towards automatic processing of literary corpora. We present an approach to classify novels as either having a happy ending or not. To achieve this, we use features based on different sentiment lexica as input for an SVMclassifier, which yields an average F1-score of about 73%. In: P. Cellier, T. Charnois, A. Hotho, S. Matwin, M.-F. Moens, Y. Toussaint (Eds.): Proceedings of DMNLP, Workshop at ECML/PKDD, Riva del Garda, Italy, 2016. Copyright c by the paper's authors. Copying only for private and academic purposes.
Every child knows that stories are supposed to have a happy ending. Every adult knows that this is not always true. In fact, in the course of the 19th Century a happy ending became a sign of popular literature, while high literature was marked by a preference for the opposite. This makes happy endings an interesting point of research in the field of digital literary studies, since automatically recognizing a happy ending, as one major plot element, could help to better understand plot structures as a whole.
To achieve this, we need a representation of plot that a computer can work
with. In digital literature studies, it has been proposed to use emotional arousal
as a proxy [
In this work, we tackle the problem of identifying novels with a “happy ending” by training a classifier. Our goal is not to present the best method for doing so, but to show that it is generally possible. We introduce our proposed approach, which already yields results considerably above a random baseline, and point out some problems and their possible solutions. Our method uses sentiment lexica in order to derive features with respect to semantic polarity and basic emotions. To account for the structural dynamics of happy endings, these features are built by considering the relation of different sections of the novels. We are able to train a support vector machine (SVM) which yields an average F1 score of 0.73 on a corpus with over 200 labelled German novels. To the best of our knowledge, our work is the first to cover happy ending classification.
The remainder of this paper is structured as follows: related work and background information is presented in Sections 2 and 3. The features and data we use are described in Sections 4 and 5. Then we present our results (Section 6).
Recently, a lot of attention has been paid to sentiment analysis in the Digital Humanities community. In this section we cover publications constructing features that are useful to our task, but have not actually been used to recognize happy endings.
Matthew Jockers proposed, in a series of blog posts, to use the “analysis of the
sentiment markers” as a novel method for detecting plot [
Elsner constructs a representation of the plot in a story using sentiment
values, among other features, in [
Mohammad builds emotional representations like ours in [
In [
We refer to a novel as having a “happy ending”, if the situation of the main characters in the novel improves towards the end of the story or is constantly favourable. In this paper, we propose a method for automatically predicting whether novels have a happy ending or not, based on features derived from sentiment analysis. We start by formally defining the task of “happy ending classification” and introduce some concepts of sentiment analysis which are relevant for our features.
Happy ending classification. We formally define “happy ending classification” as a simple classification task: Given a corpus C, we aim to learn a function f : C ! {0, 1}, where f (c) = 1 iff a novel c 2 C has a “happy ending”. In this work, we use a support vector machine (SVM) to train and test the classification function f based on a labelled gold standard. The SVM model requires a feature vector for each novel (cf. Section 5). We mostly use sentiment based features as introduced in Section 4.
Sentiment analysis. Since plot construction, and in particular happy endings,
are tightly coupled with sentiments [
Polarity denotes if a word has a positive (e.g. friend) or negative (e.g. war)
connotation. It can be expressed as a ternary value ( 1, 0, or 1). A word can also
be associated with a set of basic emotions. There are many definitions for basic
emotions, as discussed in [
Generally, polarities and emotions are collected in sentiment lexica. Each lexicon contains a set of words which it associates with a number of sentiment values according to a set of dimensions (such as polarity or different emotions). In Section 4 we derive different features for each novel based on such a lexicon and in Section 5 we introduce the sentiment lexicon we use in our study. 4
For “happy ending classification” we derive feature vectors based on a set of text segments which we combine to form sections. The final feature vectors are derived based on certain characteristic values of these segments and sections (e.g. the polarity of the final segment or the difference between the polarity of the first and the last section).
Negation detection. Our features are based on sentiments. However,
sentiments can be negated (e.g. “not happy”). For considering negations, we apply
the relatively simplistic technique presented in [
Segments. Given a corpus of novels C, we first split each novel C 2 C into n segments, C = {S1, . . . , Sn}. We evenly split by word count resulting in segments of size kCnk , where kCk denotes the number of words in novel C. Note that the last segment may be shorter due to the length of the novel and the number of segments.2 Sentiment values for segments. Now we derive a set of characteristic values for each segment. Given a fixed lexicon L with several dimensions (e.g., the polarity or an emotion), let vd(w) denote the value lexicon L associates with word w according to dimension d. For example the word “death” is strongly associated with the dimension “sadness”, that is, vsadness(“death”) = 1.
For each segment Si and each dimension d in the lexicon L, we calculate the characteristic value v¯d(Si) as follows:
P vd(w) v¯d(Si) = w2 Li |Li| (1) where Li denotes the words in Si which are covered by lexicon L. 1 https://pypi.python.org/pypi/textblob-de/. The sentiment analysis in textblob is not fully ported to German and does not include basic emotions, so we did not use it directly. 2 The novels were split into words using textblob-de. Words were lemmatized iff the lexicon used in the respective experiment contained lemmatized forms. Sections. We merge consecutive segments into sections. We consider two prominent sections, main and final. The main section Smain = {S1, . . . , Sm} covers the majority (75 up to 98%, depending on the experimental setup) of the segments starting from the beginning. The final section Sfinal = {Sm+1, . . . , Sn} covers the remaining segments and represents the “ending” of the novel. Additionally, we consider a third section, the late-main section Slate = {S2m n+1, . . . , Sm}, which covers the last part of Smain. This section is introduced in order to better capture the sentiment development at the end of the novel. For example, there may be a catastrophic event shortly before the end which is then resolved, leading to a happy ending. Since, in our experiments, the late and the final section are always of the same length, all sections are defined by specifying the number of segments in the main section m.
Sentiment values for sections. For each of these sections, we calculate the characteristic value averages based on the covered segments. In particular, given the segments of a novel, C = {S1, . . . , Sn}, and a section S, we calculate the average characteristic value by extending v¯d to sections:
P v¯d(Si) v¯d(S) = Si2S |S| (2) Features. Based on these characteristic values, we finally define the features for each novel. Given n segments, a main section of size m, and a lexicon L, the feature vector contains the following values for each dimension d: (1) the characteristic value of the final section fd,final = v¯d(Sfinal), (2) the characteristic value of the last segment fd,n = v¯d(Sn), (3) the difference between the main and the final section fd,main-final = v¯d(Smain) v¯d(Sfinal) and (4) the difference between the late-main and the final section fd,late-final = v¯d(Slate) v¯d(Sfinal). The change in sentiment values towards the end of the novel is characterized by the two differences. The difference was used to ensure that generally sad novels that had a significant improvement in the final segments can still be classified as having a happy ending or, in reverse, a drop in positive emotions towards the end of a generally happy novel can be recognized as a sad ending. 5
In this section, we describe our annotated corpus, as well as the sentiment lexicon we derive our features from.
Annotated novels. Our dataset consists of 212 German novels compiled from the TextGrid Digital Library3 and the Projekt Gutenberg4, mostly written between 1750 and 1920. The number of words in the novels ranges from less than 20,000 words up to more than 300,000. These novels have been manually labelled 3 https://textgrid.de/digitale-bibliothek 4 http://gutenberg.spiegel.de by domain experts as either having a happy ending or not,5 based on sources like the Kindler,6 Wikipedia7 or by reading relevant parts of the novel. Half of the novels (106) are annotated as having a happy ending. The annotated data can be made available upon request.
Sentiment lexica. In this work, we employ the German version of the NRC
sentiment lexicon,8 which is provided by the original author of the English
version [
After removing duplicates and all-zero entries, which would not help in our
task, the lexicon contains 4597 entries, as exemplified in Table 1. We also
evaluated our approach on SentiWS [
In this section we train a support vector machine (SVM)9 for classifying happy endings of novels on the annotated corpus introduced in Section 5 using the features presented in Section 4. For the SVM we use an RBF kernel and the parameters C = 1 and = 0.01. A linear kernel gave slightly worse results, grid search for parameter selection did not lead to an improvement. We standardize the features using the sklearn StandardScaler10 before classification. All tests were run with 10-fold cross-validation.
Baselines. Since our dataset is equally divided in novels with and without a happy ending, the random baseline as well as the ZeroR classifier (which assigns every novel to the largest class) reach 50% accuracy. Dropping the notion of sections, and only using the average sentiment values of the entire novel (i.e., 5 Because this is a simple task, each novel was labelled by only a single domain expert. 6 http://www.derkindler.de 7 de.wikipedia.org 8 http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm 9 We tried some other classifiers as well, with Random Forests and Naive Bayes reaching about the same score, while k-NN and Decision Trees performed worse. 10 http://scikit-learn.org/stable/modules/preprocessing.html
happy ending precision recall f1-score support False True avg/total one value for each dimension in L) yields an F1 score of 0.54, which is slightly above the random baseline. Adding the scores of the last segment improves the F1 score to about 0.66 for the best performing segment count n = 75. This suggests that the final segment of the novel is indeed an important feature for classifying happy endings.
Best parameter configuration. Our method requires to choose a sentiment lexicon and the number of segments n, as well as the length of the main section m (in segments). We compared different configurations and found that working with the NRC as introduced in Section 5 using n = 75 segments with a main section of m = 71 segments worked best. Other lexica containing only polarity scores (cf. Section 5) performed worse, suggesting that the combination of basic emotions represents a more accurate picture of the overall mood in a novel than polarity alone. Table 2 shows the results with the best configuration, accumulated over 20 iterations.11 Influence of segmentation and section size. In this paragraph, we describe how changing the number of segments n and the percentage of segments assigned to the main section Smain, that is |Smain| , influences the results. Figure 1 shows n the average F1-score over 20 test runs based on the NRC lexicon. Each line corresponds to a segment count n. From a larger set of segment counts we chose the 4 best performing ones. The x-axis corresponds to the percentage of segments in the main section. The y-axis shows the F1-score achieved with the respective configuration. It can be seen that splitting the novels into 50 segments mostly works very well, but is outperformed by 75 segments with a main section of 71 segments (about 95%). Furthermore, most segmentations perform best when using about 5% or 10% of the segments as the final section.
Limitations. Here, we list some limitations of our work and suggest possible solutions.
One variation we tried was to limit our features to sentences containing
explicit references to the main character of the novel. The intuition behind this is
that the concept of happy ending is closely related to the fate of protagonists of
a story. Using a domain-adapted named entity recognition (NER) toolkit [
Employing more sophisticated sentiment analysis would likely improve our results. For example, while our relatively crude negation detection only led to slight improvements, considering a more advanced set of sentiment shifters should help to get better results.
We also did not take into account that some stories are not told in chronological order. Those stories are difficult to our system, as the happy ending may happen at some arbitrary point in the text. Working around this problem would require a way to identify corresponding scenes in different novels.
Finally, we are currently working on a way to choose the length of the main section individually for each novel, instead of passing it to the model as a fixed hyperparameter. 7
In this work, we have presented an SVM classifier for identifying novels with happy endings. Our approach is based on features derived from sentiment lexica and exploits structural dynamics by comparing different sections of the novels. We consider the F1-score of 0.73 to be a good starting point for future work, such as evaluating our method on more extensive labelled datasets. Additionally, it is interesting to investigate if the same parameters yield good results for different novel collections, or different languages.