We present a new dataset of sentences1 extracted from the movie Forrest Gump, annotated with the emotions perceived by a group of subjects while watching the movie. We run experiments to predict these emotions using two classifiers, one based on a Support Vector Machine with linguistic and lexical features, the other based on BERT. The experiments showed that contextual embeddings are effective in predicting human-perceived emotions.
Emotional intelligence, described as the set of skills
that contributes to the accurate appraisal,
expression and regulation of emotions in oneself and
in others
Our dataset was retrieved from studyforrest2, a research project centered around the use of the movie Forrest Gump. The project repository contains data contributions from various research groups, divided in three areas: (i) behavior and brain function, (ii) brain structure and connectivity, and (iii) movie stimulus annotations. We focused on the latter, retrieving two types of data: the speech present in the movie and the emotions that the vision of the movie elicited in a group of subjects. As for the speech, each screenplay line pronounced by the characters is transcribed in sentences and associated with two timestamps in terms of tenths of a second tbegin and tend, that respectively indicate the moment of the movie in which the character starts talking and the moment in which they stop. Emotional data comes from the contribution to the project given by Lettieri et al. (2019). A group of 12 subjects was asked to watch the movie and 2http://studyforrest.org/ report the emotions they were experiencing during the vision, among a list of six emotions (happiness, surprise, fear, sadness, anger, disgust). Emotion reporting was performed by pressing the keys of a keyboard, with which subjects could indicate the emotion they were experiencing and its intensity, within a range from 0 (no emotion) to 100. 2.1
Emotional data was collected from a continuous output z = (z1; z2; :::; zn) from the keyboard, such that each zi corresponds to an increment of 0.1 seconds in the playing time of the movie (zi = 0:1, zi+1 = 0:2, zi+2 = 0:3, ...). Each zi is associated to a list xi1; xi2; :::; xij , with xj 2 [0; 100] and j 2 [happiness, surprise, fear, sadness, anger, disgust], where each xj indicates the intensity that one emotion assumes at a given timestamp. For our purpose, this information was too detailed and it could not be mapped to textual data properly, thus we proceeded to resample emotional information. We generated new timestamps s = (s1; s2; :::; sm), such that each si corresponds to the sum of 20 consecutive zi, thus to an increment of 2 seconds in the playing time of the movie. Each si is associated to a new list of emotional values, where each new value is the average of the values associated to the summed zi.
After resampling, we aligned the text to emotional data. As one of our aims is to determine how much text is needed for accurate emotion prediction, we considered three progressively larger time windows for each sk, such that windowi = [sk m; sk], where m = (2; 4; 6). For each sentence, we retrieve its tend and align the sentence verifying if sk m tend sk, thus checking if the moment in which the sentence ends falls within the given time window. In this way, the larger the time window, the larger the amount of text that gets aligned with a specific timestamp. With this process, we created three different datasets, one for each time window. We then removed all the lines in which no text was aligned to sk. For each dataset, we end up having 898 timestamps associated with a line of text and 6 emotion declarations for each of the 12 subjects. 2.2
We first looked at the distribution of our data, examining how many times each subject declared a specific emotion. Whenever the subject assigned a value different than zero to a certain emotion, we considered that emotion as present at a given timestamp, regardless of its intensity. If all 6 emotions were zero at the same time (all xj = 0), we assigned to that case the class neutral. Furthermore, if any given emotion was declared (at least one xj 6= 0), we assigned to that case the class emotion, to indicate a generic emotional response.
As shown in Table 1, the most represented
emotions in the dataset are happiness and sadness,
while the others are underrepresented. Table 1 also
shows that emotions distribution is quite uneven
among the different subjects, as there were some
subjects that declared emotions frequently and
others that entered fewer declarations. This is due
to the fact that emotive phenomena are strongly
subjective, meaning that emotion processing is
specific to each person and that everyone experiences
emotions at a different granularity
Given the information on the agreement and on emotions distribution, we decided not to examine underrepresented emotions directly, even if their agreement was strong (i.e. surprise). In order to still account for underrepresented emotions, we relied on the general class emotion. Hence we assessed three different scenarios: (i) the presence of any kind of emotion (at least one xj 6= 0), (ii) the presence of happiness (xhappiness 6= 0) and (iii) the presence of sadness (xsadness 6= 0). Furthermore, we decided to conduct our experiments only on two subjects, subject 4 and subject 8. We focused on these specific subjects as they declared all emotions evenly, without neglecting any of them, and because the number of declarations for each emotion was quite similar between the two subjects. 3
We evaluated the three scenarios described in 2.2 in contrast to the absence of any emotion (all xj = 0), producing three binary classification tasks. We relied on two sets of features: automatically extracted linguistic and lexical features, and contextual word embeddings from a language model.
Emotion happiness surprise fear sadness anger disgust 12 11 12 12 12 8
Text I had never seen anything so beautiful in my life. She was like an angel.
Jenny! Forrest! (into radio) Ah, Jesus! My unit is down hard and hurting! 6 pulling back to the blue line, Leg Lima 6 out! Pull back! Pull back! Bubba was my best good friend.
And even I know that ain’t something you can find just around the corner. Bubba was gonna be a shrimpin’ Boat captain, But instead he died right there by that river in Vietnam.
Are you retarded, Or just plain stupid? Look, I’m Forrest Gump.
You don’t say much, do you?
For the first set of features, sentences were first
POS tagged and parsed using UDPipe
Figure 1 shows the accuracy scores for all the models, for both subjects and the three datasets. In all cases, the baseline was determined with a majority classifier. The results appear similar for both subjects.
SVM models are always outperformed by BERT ones. In any case, SVMling is the model that gave the lowest performance, remaining below or around the baseline value. On the contrary, SVMlex tends to bring a higher performance, despite remaining close to the baseline in most cases. On one side, this is due to the fact that features that look at the raw, morpho-syntactic and syntactic aspects of text, do not encode any relevant information regarding the emotional cues in the text. SVMlex always performs better than SVMling because lexical features look at patterns of words and characters that are repeated in the input text and thus record information about the lexicon of the dataset. However, as our dataset is too small, it is hard for the model to retrieve the same lexical patterns in both the training and test set.
BERT models outperform the SVM ones in both happiness and sadness prediction. In the case of emotion prediction, BERT models obtain very good results only on the 6 seconds dataset. This is due to the fact that, in this case, we have flattened all emotions into a single category, thus it may be difficult for the model to distinguish between general emotionally charged sentences and those that are not perceived as emotionally charged. When emotions are specific and clearly separated, as in happiness and sadness cases, BERT is able to infer the perLevel of Annotation
Raw Text
POS tagging Dependency
Parsing Lexical Patterns
Feature Sentence length Word length Type/Token Ratio for words and lemmas Distibution of POS Lexical density Inflectional morphology of lexical verbs and auxiliaries (Mood, Number, Person, Tense and VerbForm) Depth of the whole syntactic tree Average length of dependency links and of the longest link Average length of prepositional chains and distribution by depth Clause length (n. tokens/verbal heads) Order of subject and object Distribution of verbs by arity Distribution of verbal heads and verbal roots Distribution of dependency relations Distribution of subordinate and principal clauses Average length of subordination chains and distribution by depth Relative order of subordinate clauses Bigrams, trigrams and quadrigrams of characters, words and lemmas ceived emotions even from small amounts of text (2 and 4 seconds datasets). BERTover and BERTtransf tend to give better performances than what happens with BERTorig. In the case of BERTover, there is a very slight difference in the prediction of happiness and sadness, as in these cases the classes to be predicted were distributed quite evenly. In the case of emotion prediction, the model is helped by the higher representation of the neutral class. With BERTtransf, the performances stay in line with the ones obtained with the bare oversampling. Fine tuning the model on similar data did not add any more useful information. This is due to the fact that SemEval data were too distant from the ones of our dataset. Therefore, even though the task is similar to ours, the input text is too different from our sentences to actually make a huge difference for the prediction. We also tried another form of transfer learning, tuning the model on one subject and testing it on the other one. However, the results were too low and we did not report them. This is because emotion perception is a very personal phenomenon and it cannot be easily generalised to different individuals.
To further evaluate the results, we computed the percentage of agreement between the two models that overall had the best performances, BERTover and BERTtransf. We defined agreement as the percentage of sentences for which the models gave the same output during the classification task. Table 5 reports the results for emotion, happiness and sadness, for every timespan window, and for both subjects 4 and subject 8. The agreement is quite high in all cases, and it tends to get stronger with the amount of text on which models are trained (i.e. 6 seconds). A higher level of agreement indicates that the models have similar behaviour, thus making the same mistakes in the classification task. The lowest levels of agreement are encountered on the classification of happiness, showing that the two models work differently in this part of the task. Indeed, both BERTover and BERTtransf obtain high performances in predicting happiness, but the fact that their agreement is lower suggests that they differ in the mistakes they make in the classification. We may exploit this information to create systems that combine different classifiers, actually enhancing the classification accuracy. By doing this, it is possible to compare the cases in which two or more classifiers agree and the cases in which they make mistakes, thus choosing the best classification output accordingly. 5
In this paper, we presented a dataset of sentences extracted from the movie Forrest Gump, annotated with the emotions that a group of subjects perceived while watching the movie, and we studied how to predict these emotions. To do so, we retrieved different kinds of features from the sentences pronounced by the characters of the movie. We showed that contextual embeddings extracted from the sentences can accurately predict specific emotions, even if the amount of text used for the prediction is very little. Instead, when predicting generic emotional elicitation, a larger amount of text is required for an accurate prediction. We also show that lexical, morpho-syntactic and syntactic aspects of the sentences cannot be used to infer emotional elicitation during the view of the movie.
As emotional response is directly correlated with brain activity, we plan to add fMRI images recorded during the vision of the movie to the contextual embedding we extracted. In this way, we could verify if brain images can help to increase the accuracy in the prediction of perceived emotions.
We thank MoMiLab research group of IMT Lucca for having shared with us the data they collected on human-perceived emotions. Furthermore, we are grateful to the studyforrest project and all its contributors.