Narrative film is a composition of writing, cinematography, editing, and performance. While much computational work has focused on the writing or visual style in film, we conduct in this paper a computational exploration of acting performance. Applying speech emotion recognition models and a variationist sociolinguistic analytical framework to a corpus of popular, contemporary American film, we ifnd narrative structure, diachronic shifts, and genre- and dialogue-based constraints located in spoken performances.
Film is rich in its supply of semiotic resources, communicating meaning from the interaction
of language (encoded in a script), visuals (choices of composition, blocking, cinematography),
sound and more. Much computational work has arisen to examine slices of this semiotic field,
including measuring how gender stereotypes or plot arcs are reflected in dialogue [
As Naremore [
Much of the film revolves around these three words, overheard in a conversation between the wealthy Harlan Thrombey and his grandson, Ransom. The line is uttered by multiple characters as the film unfolds: angrily shouted by Ransom, somberly recalled by the eavesdropper, and gleefully recounted by inspector Benoit Blanc upon solving the crime. Even a single line, located within a single diegetic event, has great capacity for meaning-making in performance.
When viewed in this light, we can apply the analytical framework from variationist sociolinguistics to better understand this space of performance. Given a fixed line of dialogue (equivalent to a linguistic variable), a performance entails a choice — a selection from the set of possible variants. It is this choice, and the meaning contained within, which we study.
In this work, we design computational models to explore this form of variation by
considering the emotional range of performances in contemporary American film, exploring in
particular the tension between what characters say and how they say it. As distinct from prior work in
the computational humanities that has measured emotion from text alone1[
Using a speech emotion recognition model, we construct a parallel dataset of spoken
performances (utterances) aligned with the text of the words being spoken d(ialogue phrases).
This dataset allows us to isolate and examine how performances vary in meaning from their
paralinguistic features in addition to the textual meaning of the screenplay. We use this dataset
to carry out several case studies exploring variation in performance in American film:
1. First, we carry out a structural analysis of emotion as performed over narrative time.
Doing so allows us to characterize film as performance text, relating emotional performance
to larger narrative structure.
2. Second, we study diachronic variation by comparing emotionality of films across release
years, testing the degree to which performances have intensified over time (following
Bordwell’s theories of visual style [
In this work, we use computational methods to survey how both textual and contextual variables inform and reflect the performances rendered on screen.
In order to perform our analysis, we need to construct an aligned dataset of actor performances (utterances), the text of the words they speak (phrases), and the emotions in each utterance. We create a pipeline that takes as input a set of full-length movies, and outputs time-aligned transcriptions for utterances, their emotion labels, and groups of semantically similar phrases.
We first construct a data pipeline to segment and transcribe utterances from movie dialogues. The pipeline takes as input a set of MP4 files, where each file is one digitized film. Our analysis takes place in the speech and text modalities, so we useffmpeg to extract the audio track.
We use the pyannote1 segmentation model to detect continuous, single-speaker speech segments. Then, we use faster-whisper2 to transcribe each speech segment. Because pyannote speech segments are based on voice activity detection and silences, it can label extended, multisentence speech as a single segment. For our analysis, however, we are interested in utterances as a discursive unit. If a character makes an assessment, then poses a question, we would like to split these into two distinct utterances. As a middle ground between raw voice activity detection and segmenting discursive units, which requires complex conversational understanding, we perform a post-processing step where we further split speech segments by sentence boundaries derived from the transcriptions. Because whisper is an end-to-end model that does not produce fine-grained time alignments, we then use a speech-to-text fine-tuned wav2vec2 3 to perform word-level time alignment between the transcription and the audio, then split the audio based on sentence boundaries generated bysyntok,4 a fast, rule-based sentence segmenter.
To prevent the end credit sequences from interfering with the results, we detect when the end credits begin by performing optical character recognition (OCR) on the shots in a movie and identifying long continuous sequences of shots that contain large amounts of text. We trim the movie to the beginning of the end credits.
To perform speech emotion recognition, we use a wav2vec2 large model without any
taskspecific fine-tuning to extract audio features. Then, we train a classification head to perform
seven-way emotion classification, based on the Ekman model [
We experiment with two classification settings: an utterance-level model which makes predictions based on only the speech features of the input utterance and aconversation-level model which includes the speech features of both the input utterance and its surrounding utterances.
In both cases, we use a pretrained wav2vec2 model as a backbone model for generating vector
representations of each utterance. Because wav2vec2 creates an embedding for each audio
frame (roughly 20ms of speech), we follow prior work in computing utterance embeddings by
averaging across all timestamps within an utterance [
We implement the utterance-level model from Pepino, Riera, and Ferrer 2[
We also train an contextual model which uses a bidirectional LSTM to predict the emotion of utterances within the context of a conversation. To do so, we define conversations as groups of utterances where each occurs within 3 seconds of the next. In the MELD dataset, there are 1,478 conversations in the training split according to this criterion.
For each conversation, we predict the emotion labels of all utterances in the conversation by ifrst passing weighted activations through the biLSTM before applying the classification head to each hidden state. As before, the weights of activations are learned during training.
We expect the movie data to be similar in nature to the MELD dataset, since both consist of professionally produced and acted clips. However, to ensure that our models do not experience domain shift despite the greater range in release year and setting of the film corpus, we evaluate these models on the test split of the MELD dataset as well as a manually collected dataset consisting of 333 clips from a subset of 35 contemporary American films. Each clip was a conversation with at least 2 utterances, where conversations were identified with the same heuristic used to construct training data for the contextual model. This resulted in a final evaluation dataset of 2,157 utterances with emotion labels. The clips were labeled by two annotators: 51 clips were labeled by both annotators and 250 clips were labeled by a single annotator. The Krippendorf’s between the two annotators was 0.334, and the Fleiss’ was 0.333, which matches the agreement of the MELD dataset.
Table 1 shows the evaluation results on the MELD and Movies datasets. The models perform comparably to each other, and comparably across evaluation datasets. This performance also approaches the state of the art on MELD for audio-only models. Because the performance of the contextual model is slightly higher, we use its inference outputs for our analysis.
One powerful aspect of this dataset is that we align actor performances to the words that they
speak. To account for variation in how highly semantically similar phrases can be realized, we
cluster together phrases with high semantic similarity. We use thesentence-transformers
library to compute sentence embeddings of utterances and cluster them with the Leiden
community detection algorithm [
The above pipeline measures the emotions performed in an utterance and ties each utterance to
the text being spoken. We apply this methodology to a large corpus of contemporary, popular
American films [
Our corpus consists of the top-50 live-action, narrative films by U.S. box ofÏce from 19802022. We supplement these with films nominated for “Best Picture”-equivalent awards by one of six organizations in those years: Academy Awards, Golden Globes, British Academy of Film and Television Arts, Los Angeles Film Critics Association, National Board of Review, and National Society of Film Critics. We only include English-language films in this analysis, resulting in a total of 2,283 films.
Plantinga [
We study the distribution of emotions in utterances over the course of a movie. How do
the prevalence of emotions shift over narrative time? Similar to previous work on dialogue
in screenplays, we ask if there are emotional regularities across films [
Narrative 6T0im%e 70% 80% 90% 100% 10% 20% 30% 40% 50%
Narrative 6T0im%e 70% 80% 90% 100% (a) Emotionality
(b) Joy before looking more closely at how specific emotions are distributed temporally. We plot the average probability of an emotion label for an utterance in intervals of 5 percent, expressed as a percentage of the full run-time of the film. Specific emotions are measured as proportions of the emotional labels, excluding the neutral label.
We find that the emotional trajectories of performances are, in fact, structured over narrative
time. Figure 1a shows that emotionality increases over narrative time. We examine also the
trajectory of specific emotions across films (figs. 1b,1c,1d). We find that joyful performances
follow a U-shaped curve, with a steep increase towards the end, as movies resolve. Like Hipson
and Mohammad [
Subscribing to a particular categorization of emotions can be restrictive; in the remainder of
the paper, we explore emotional performance, but depart from analyzing specific emotional
labels. First, we study performance at a less granular level, focusing on the concept
ofemotion0.50
ality as the proportion of utterances withany emotion label.5 We measure how emotionality
has changed historically over the decades spanned by our corpus. Emotional shifts have been
identified in English fiction books: Morin and Acerbi [
When we split the data by release year, we find a mild efect that earlier films have a higher proportion of emotional utterances compared to later ones, with emotionality hitting a minimum around 2010 (see Fig. 2). However, the question remains whether the emotionalcontent is changing (as Morin and Acerbi find in literature) or if the style with which words are being uttered is changing.
To disentangle the efects of shifting content and shifting style, we consider the change in emotionality over the years within thesemantically equivalent phrase groups. If it is indeed the writing, and not the performance, that drives this shift in emotionality, we should see little change within a phrase group. However, when we look at the 511 phrases that are used in all 43 years of the dataset, we find that a fixed-efects regression shows a slightly negative, statistically significant correlation between the year and emotionality even within phrase groups ( 2 = 0.048, (1, 21461), < 0.001 ).
Though this result is seemingly at odds with Bordwell’s finding that visual style intensifies, it
is also possible that they are harmonious. In Hollywood film, the close-up shot has always been
5The model achieves an F1 score of 0.69 on the neutral label.
associated with emotional expression 2[0]. Panovsky [
Range is often said to be the mark of a great actor. Naremore writes about the importance of an
actor splitting their character “visibly into diferent aspects”, showing of emotional range [
For this analysis, we construct a general measure of emotional range across a set of utterances 1… . We characterize each utterance with a performance vector ⃗ , which is a distribution over emotions, given by the predicted probability distribution from the speech emotion recognition model. This allows us to take a more nuanced view of performances as a mixture of emotions. We model the distribution from which the vectors⃗1… are drawn as a Dirichlet, and ifnd the parameters which maximize the likelihood of the observed vectors. We define emotional range as the entropy of this distribution: a higher entropy means there is greater variance in the distribution of performances, and a lower entropy signals lower emotional range.
One criticism of the Ekman emotional model lies in its construct validity: seven discrete emotion labels may be insufÏcient to characterize the space of emotions. Ideally, we would model a continuous space of “performance”. In our previous analyses, we use these emotion labels as an intermediate between that ideal on one end, and sentiment analysis on the other. Here, our measure of emotional range is agnostic to the meaning of specific emotion labels, and serves to demonstrate how emotion classification can be a useful proxy task through which we can analyze performance in a more continuous space.
lates that some genres, like some types of comedy, have less capacity for emotional range than
others. Previous work has shown that emotionalarcs are correlated with genre [
We calculate the emotional range for each movie, and find the average score for each genre. Genre information comes from IMDB, and we exclude genres with fewer than 30 films in our dataset.6 Figure 3 shows the average entropy across genres. Thrillers, biographies, and mysteries have the least emotional range; fantasy, musicals and family films rank highest. While 6Three genres were excluded: Western (19 films), Documentary (3), and Animation (2)
Thriller Biography
Mystery
Crime Sci-Fi
Action Romance
History re Drama en Music G Comedy
Sport
War Adventure
Horror Fantasy Musical Family 11.4 11.2 11.0 10.8 10.6 10.4 10.2 10.0
Emotional Range 9.8 it is difÏcult to attribute these results to a particular property of specific genres, these findings show that some genres have more constrained or consistent emotional registers than others.
Gofman when theorizing about performance: actors draw on and play against the interactional norms with which we as audience are already familiar. We ask if this bears out in our data. Does the emotional range of dialogue phrases reflect their discursive properties?
To study this, we measure the emotional range in dialogue. Because we tie specific performances to the words that are spoken, we can identify instances across the corpus when a given phrase was uttered. We isolate the 2,656 phrase groups that are uttered at least 50 times across our dataset. For each phrase group, we calculate the emotional range of its utterances.
Table 3 shows phrases with the highest and lowest entropies. By inspecting the phrases at either end of the spectrum, we find qualitative diferences in the kinds of phrases that have higher and lower emotional range: the capacity for emotional variance reflects the discursive lfexibility of the words being spoken. Phrases with low range are functional and generally part of highly directed interactions: most phrases are either yes-or-no questions or answers to them. Phrases with high emotional range, on the other hand, mostly have more open-ended, evaluative discursive functions. In these cases, the prosody or intonation of speech can easily lend color to the statement being made. “You’re alive” can be said with joy or relief to a loved one, as Marty McFly to his mentor Doc inBack to the Future (1985), or with anger at the sight of an enemy, as Lord Norinaga greets Walker inTeenage Mutant Ninja Turtles III (1993).
With this work, we demonstrate that films can, and should, be studied as performance texts. We tie our findings to both film theory and other computational work on narratives. Here, we discuss some limitations of the current study.
Measuring emotions. We follow a vast body of previous work within natural language
processing [
First, there are doubts about the ecological validity of emotion recognition, especially as most speech emotion recognition datasets contain acted emotion as opposed to natural emotion. We note that, unlike much afective computing work, we use emotion recognition models trained on acted speech to make inference on acted speech. The professionally-produced, acted speech in the MELD dataset is well-suited to our data, which is also professionally-produced and acted. Indeed, we find that performance is similar between MELD and our in-domain evaluation data.
Another criticism lies in the cultural relativity of emotion. Though Ekman argues that the basic emotions are universal, he acknowledges there may be cultural diferences in the emotions elicited in a given context. It is reasonable to suppose that viewers’ normative knowledge also influences the interpretation of these emotions. We focus on contemporary American film in both our analysis and training data, holding at least the intended cultural audience constant. Cultural variation in performance is a ripe area for future work, as cultural diferences exist in not only the production and interpretation of emotion, but also in theories of acting.
Aside from these specific criticisms of the Ekman model, the low interannotator agreement in both our evaluation set as well as other datasets, including MELD, suggest that this model for emotion may remain too coarse to precisely describe the data. Work in both afective psychology and NLP have attempted to address this by using more fine-grained classes [8, 7] or a continuous spaces of emotion 2[7, 7]. While we used the Ekman model due to the availability of training data as well as to provide comparison with previous studies of emotion narratives, alternative emotion models may prove useful in future work.
A question of authorship. In cinema, the performance that audiences see on screen is cocreated by the actor, the director, and the editor. Baron and Carnicke2[] describe the conventional wisdom within film analysis to be that cinematic performances are made in the cutting room. “True” acting happens on the stage. Though our work studies film as performance text, it does not disentangle the processes through which the performance is constructed. It is about the performance as viewed, but not about the choices made by actors as separate from the director or editor. Our work makes the point that performance carries meaning worth studying, and opens the door for future computational work that explores its authorial roots. Embodied erformance. Finally, we examine only performance as enacted through speech. This is perhaps the modality that lies closest to the script, and allows us to apply a variationist approach to studying the relationship between performance and text, but of course performance includes not just speech but also gesture, posture, facial expression, and more. Visual description has been found to be more useful for aligning narrative events than dialogue3[7], and quantitative analysis of theater performance has found narratively meaningful patterns in movement [10]. Film is a multimodal medium that deserves analysis in all its modalities. We hope that our work examining film across the speech and text can serve as a basis for more work that examines performance as embodied visually.
In this paper, we explore the relation between film as narrative text and as performance text. Using a novel parallel dataset of speech and text from popular contemporary American film, we develop computational methods to measure how emotional prevalence and emotional range vary by both textual factors of narrative time and dialogue, as well as contextual factors of release year and genre. We hope this work inspires further multimodal studies opferformance in computational film analysis.
The research reported in this article was supported by funding from Mellon Foundation and the National Science Foundation (IIS-1942591 and DGE-2146752). We thank Jacob Lusk and Lucy Li for insightful discussion and feedback.