Evaluation and Alignment of Movie Events Extracted via Machine Learning from a Narratological Perspective Feng Zhou 0 FedericoPianzola 0 Center for Language and Cognition, University of Groningen , Oude Kijk in 't Jatstraat 26, 9712 EK Groningen , The Netherlands 49 62

We combine distant viewing and close reading to evaluate the usefulness of events extracted via machine learning from audio description of movies. To do this, we manually annotate events from Wikipedia summaries for three movies and align them to ML-extracted events. Our exploration suggests that computational narratology should combine datasets with events extracted from multimodal data sources that take into account both visual and verbal cues when detecting events.

 eol>movie events narrative events computational narratology audio description movie summaries 1. Introduction 2. Related Research

Narrative theory de昀椀nes events as the minimal units of narrative, being states, processes in time, and changes of state [ 22, 32 ]. Moreover, narrative events are o昀琀en analyzed along the dimensions of their duration and frequency in chronological terms, and span of their representation through a medium1[0]. In the case of movies, events are represented as scenes, typically taking place at a location, involving a set of characters, and spanning a continuous period of time [ 8 ]. In addition, sub-scenes occur within a single location and time frame, are usually shorter, contain several shots that do not constitute a complete semantic unit, and may lack a clear beginning or end8[]. As scenes are sequences of sub-scenes, the shots contained within a series of sub-scenes can be used to create narrative sub-events or whole even7ts, 1[ 8 ].

To cope with such granularity, most ML methods extract events through the cross-modal understanding of audio, visual, and video-related text: e.g. alignment between script and timestamped subtitle [ 15 ], retrieval via semantic graphs of video clips and plot summari3e4s, [ 35 ], segmentation by detecting audio descriptions between character dialogue3s1,[ 23 ]. The evaluation of ML methods is done via manual annotation of movie events and aligning sentences in movie scripts or summaries with video clip3s0[ , 35, 36 ]. User-generated content platforms like Wikipedia and IMDb provide a vast amount of movie-related textual da3ta, 1[ 3 ] that can provide high-level semantic annotation of movies, such as describing speci昀椀c scenes and events using text [ 6 ], and serve as gold standards to evaluate the ability of machine learning methods for event extraction.

Despite the variety of methods, the relevance of ML-extracted events for summaries that align with human expectations has not been explored in detail. Therefore, we compared a dataset of automatically extracted events to events manually extracted from movie summaries, focusing on alignment, duration, length, and type of events.

 3. Data 3.1. Machine Learning Extracted Events from Audio Description

Audio description (AD), also known as Descriptive Video Service (DVS), is conceived for visually impaired audiences and usually contains the most important visual elements of a movie frame, such as scene changes, characters’ appearance, actions, and interactions between characters [31]. AD uses short and precise language inserted between characters’ dialogue and mixed with the original soundtrack of the movie31[ , 23 ]. AD has been used to extract movie events and research the narrative structure of visual materials because of the high quality of the movie description text and the high degree of alignment with visual conten3t1][. We used the Montreal Video Annotation Dataset (M-VAD)3[ 1 ]1 to evaluate this type of story representation with respect to narrative event granularity and plot covera2g8e].[ Three movies of 1Since AD appears between character dialogues, the curators of the M-VAD dataset 昀椀rst separated the AD soundtrack from the movie soundtrack, then they automatically segmented the movie into di昀erent events by detecting the pauses between ADs, 昀椀nally they transcribed the AD of the events into text using a professional transcription service. The data including the movie event description text and the corresponding video clips’ timestamps are obtained from the M-VAD Names GitHub repository: https://github.com/aimagelab/mvad-namesdataset/releases/tag/1.0. The subset used in this article and the manually-annotated data are available in the reposdi昀erent genres were selected from the M-VAD dataset:500 Days Of Summer (2009, romantic comedy, 95 minutes), The Social Network (2010, biographical drama, 120 minutes), andFlight (2012, thriller drama, 138 minutes).

3.2. Manually Annotated Events from Movie Summaries

Wikipedia crowdsourced summaries can be considered a proxy of the events that people 昀椀nd important in a movie, spanning the whole story arc28[]. They are also used as gold standard in NLP because they have an event granularity that balances e昀케cient plot recognition and information loss 2[ 8, 37 ]. We linked these gold-standard events to the respective video clips by identifying the timestamps corresponding to the events’ occurrences in the movie. These events manually extracted from summaries allowed us to evaluate the ML movie event dataset with respect to key content coverage (condensing important parts of informatio2n9)][and the temporal localization of such events in the plot. Additionally, events can be classi昀椀ed by type [ 9 ] and used to explore the criteria underlying the summarization process and the plot structure [ 12 ]. We annotated events in summary texts based on the main verb in sentences, according to four types: stative, process, change of state in the story world, or none of the previo3u2s].[Two annotators independently annotated events from Wikipedia summary texts using the so昀琀ware CATMA [ 11 ] and then marked the start and end timestamps of the events’ corresponding video clips while watching the movies.

4. Methods

To test whether Wikipedia summaries’ events cover the semantic content of most video clips describing key events in the movie, we 昀椀rst rescaled between 0 and 1 the timestamp (for video clip index) and annotation’s 昀椀rst character’s position (for plot sentence index) of all annotated events in the summary text that we were able to align to corresponding video clips. In this way, we obtained a comparable plot sentence index and video clip inde1x].[2 Second, we manually aligned human annotated events with the ML-extracted events based on the respective timestamps to examine whether the ML-extracted events have the same event description granularity as Wikipedia summary events from text and video perspectiv2e8s][. The criterion used for the manual alignment is whether the overlapping time interval between one or several corresponding ML- and human-extracted events is at least 75% of either extracted event(s) duration [24]. Additionally, a human evaluation was conducted to check whether the events extracted via ML can represent the semantics of manually annotated events. Subsequently, to assess the extent of the alignment between ML-extracted and human-annotated events, we counted the number of events, and compared their time duration and distribution throughout the movie [ 14 ]. itory: https://github.com/Marina-Zhou/Movie_Event_Alignment 2To avoid altering the depiction of event distribution in the movie summary, the few summary events without a timestamp have not been plotted in2. Additionally, there are instances where a manually extracted event relates to a number of movie clips. Only the segment with the earliest chronological occurrence is kept, the others are discarded for the sake of clarity of the visualization. The full list of events is anyway available in the released dataset.

5. Results and Discussion

[and with whom Whip had spent the night before the incident].

Another notable phenomenon occurs at the end of the summaries, where many events of the movie ending are reported. The distribution histograms show that the majority of the events in the summaries correspond to events from the last third of the movie. A possible reason is that the end is where usually the most important events of a story occur, sometimes being the con昀氀uence of multiple events [ 5 ]. But it could also be due to the interpretative function of summaries, which need to provide a sense of closure, therefore the short summary text describes more video clip events at the end of the 昀椀lm to create such e昀ect of closure2[ 7 ], even though this may not be present in the story.

Figure 3 shows that ML-extracted events cover almost the entire duration of the movie, whereas events in summaries have notable gaps. The width of the bands re昀氀ects the length of the reported events and, in Figur4e, it is visible how several ML-extracted events can be encompassed by a longer summary event. There are also some successfully aligned events belonging to sub-scenes about secondary characters that appear in parallel narrati2v,e7s].[

Figure5 shows that only part of the ML-extracted events not aligned with summary events occur within the time spans corresponding to manually-annotated events. These events cannot be aligned to a summary event either because the overlapping range with summary events is too small (less than 75%) or because the number of ML-events is insu昀케cient to fully re昀氀ect the semantics of the manually-annotated events. For example, in the movFielight, the simple ML-extracted event [Whip turns o昀 the news] overlaps with the much more informative manually-annotated event [he stays with Charlie until the NTSB hearing]. This is due to the mere descriptive function of AD.

The other type of ML-extracted events not aligned with summary events occur outside the time spans covered by summary events. To some extent, these events can be considered progressive scenes that set the background for a summary event and serve as event separator1s6][. However, most of them are unimportant events that do not contribute to the plot development. They primarily describe the appearance and secondary actions of the characters, and the visual elements in scenes.

Our annotated data indicates that events extracted from AD more easily align with manuallyannotated events when they contain descriptions of visual elements, especially direct descriptions of character actions, rather than information captured through dialogue or background sounds. These ML-extracted events are located within manually-annotated events with longer time duration and may not exhibit direct semantic relevance for such manuallyannotated events. Nevertheless, they constitute an integral component of the broader manually-annotated events because they are progressive scenes that separate sub-events. As such, they have a role in the narrative progression because they introduce new situations where subsequent events happen. For instance, in the movieFlight, the ML-extracted event [They reach a clearing, through clouds and bright sunlight, as Whip levels out] serves as a transition, enabling the audience to experience the severe turbulence of the plane and facilitating a smoother acceptance of new information and emotions during a plot transition.

Most of the manually-annotated events are of type “process” (67~78%) and “change of state” (14~16%). Since ML-extracted events contain the description of the characters’ actions, it is easier to align them with manually-annotated “process” and “change of state” events, than with “stative” events (see Figure6). Notably, most of the “change of state” events in the summary are aligned with ML-extracted events, suggesting that these are indeed important plot turns [ 19 ]. However, we were not able to align some “change of state” events from the movi5e0s0 Days of Summer and The Social Network (see Appendix A). This suggests that AD events could still miss information that is important for the plot. 6. Limitations • The dataset used in this research only contains data from three di昀erent movies. Therefore, clear generalization about the usefulness of events extracted from AD via ML for the understanding of narrative cannot be drawn. • The source of ML-extracted events are AD scripts, which are written by humans. Therefore, this type of data cannot be considered completely machine-generated. A comparison with captions generated by multimodal Large Language Models could provide a

better case study. • AD only describes the visual elements of the movie, not the events performed through the character’s dialogue, e.g someone angrily yelling at someone else could be a highly relevant event, maybe a break-up between two partners. Therefore, we suggest to combine AD with data sourced from subtitles or movie scripts. • The amount of manually-annotated data is quite small (only three movies), so for all the analyses we retained all data annotated by one annotator. The inter-annotator agreement is reported only for reference. Therefore, some subjectivity is unavoidable. • There is a 1-2 seconds delay between ADs and the corresponding scenes in the movie, so the curators of the M-VAD dataset added 2 seconds at the end of the video to compensate for this deviation 3[ 1 ]. However, this method may also cause inaccurate timestamps. For example, [He takes Whip’s hand] in the movieFlight has the end timestamp 01:26:56 but the event actually ends 01:26:54. Moreover, in the M-VAD dataset, the 昀椀rst 3 minutes of the movie are discarded. All these choices slightly a昀ect the alignment with our manually-annotated events. • We removed some manually-annotated events not corresponding to any video clip in the movie when plotting the visualization, e.g. an event reported by a character but not shown on screen.

7. Conclusions

One one hand, most of the events extracted by machine learning from AD can be considered sub-events rather than narrative events. Because their time duration is very short, they cannot be considered as complete semantic units and need to be understood in conjunction with the video context and dialogue. These events o昀琀en include descriptions of visual elements in the 椀昀lm and are placed between character dialogues. Some of these events serve as progressive scenes, both inside and outside the automatically extracted event, and are used as separators for other sub-events and events. On the other hand, the events extracted manually from the summary are a high-level synthesis of the dialogue and actions of the characters in the movie, and cover longer time spans that may contain multiple sequential events and sub-events. Moreover, verbs in summary events o昀琀en refer to one of the four categories of narratological events and are highly relevant for the understanding of the plot, but some verbs in AD events describe the settings and cannot be considered narratological events.

Computational narratology can certainly make use of ML-extracted events based on audio description but there are strong limitations that suggest using them in combination with other data sources that take into account both visual and verbal cues when detecting events. The multimodal communicative nature of movies seems to pose a big challenge for computational narratology because existing datasets used for ML can only provide an extremely simpli昀椀ed notion of narrative event.

In order to train ML models to extract narrative events, there is a need for movie events datasets that include audiovisual information, subtitles, and AD sentences. The annotation guidelines should also take into account that the narratological concept of event, as unit of a story, should also be related to other conceptualization of events, namely sub-events as those observed in AD and macro-events as those observed in movie summaries.

Since events and causal relationship between events play a key role in narrative understanding [ 28 ], our multimodal human evaluation method on the M-VAD movie event dataset should be replicated to evaluate the ability of other automatic methods for the extraction of narrative events. This could help to understand how summary events are constructed by smaller units such as events displayed in the story and sub-events.

In addition, we have not yet conducted research on di昀erent types of events extracted by ML and the story arcs that they form. In the future, we could also explore whether movie events extracted via di昀erent ML methods contain important narrative events (key plot turns) in the story arc [ 12 ].

A. Appendix [1] M. Bain , A. Nagrani , A. Brown , and A. Zisserman . “Condensed Movies: Story Based Retrieval with Contextual Embeddings” . InC: omputer Vision - ACCV 2020: 15th Asian Conference on Computer Vision , Kyoto, Japan, November 30 - December 4 , 2020 , Revised Selected Papers , Part V. Kyoto , Japan: Springer-Verlag, 2020 , pp. 460 - 479 . doi1 : 0 .1007/9 78- 3 - 030 -69541-5\_ 28 . [2] D. Bordwell , K. Thompson , and J. SmithF. ilm art: An introduction . Vol. 8 . McGraw-Hill New York, 2008 . [3] M. Burghardt , A. He昀琀berger, J. Pause, N.-O. Walkowski , and M. Zeppelzauer . “ Film and video analysis in the digital humanities-an interdisciplinary dialog” D.Iing:ital Humanities Quarterly 14.4 ( 2020 ). url: http://digitalhumanities.org:8081/dhq/vol/14/4/000532/0 00532.html. [4] S. Chaturvedi , S. Srivastava , and D. Roth . “ Where Have I Heard This Story Before? Identifying Narrative Similarity in Movie Remakes” . InP:roceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , Volume 2 ( Short Papers). New Orleans , Louisiana: Association for Computational Linguistics, 2018 , pp. 673 - 678 . doih:ttps://doi.org/10.18653/v1/ N18 -210 6 . [5] N. Cohn . “ Visual Narrative Structure” . InC:ognitive Science 37.3 ( 2013 ), pp. 413 - 452 . doi: https://doi.org/10.1111/cogs.12016. [6] A. Cooper , F. Nascimento , and D. Francis . “ Exploring Film Language with a Digital Analysis Tool: the Case of Kinolab .” InD:HQ: Digital Humanities Quarterly 15.1 ( 2021 ). url: https://www.digitalhumanities.org/dhq/vol/15/1/000515/000515.ht m.l [7] J. E. Cutting. “ Event segmentation and seven types of narrative discontinuity in popular movies” . In: Acta Psychologica 149 ( 2014 ), pp. 69 - 77 . doi: https://doi.org/10.1016/j.actps y. 2014 . 03 .003. [8] J. E. Cutting. “ Narrative theory and the dynamics of popular movies” . IPns:ychonomic Bulletin & Review 23.6 ( 2016 ), pp. 1713 - 1743 . doi: https://doi.org/10.3758/s13423-016-10 51 - 4 . [9] S. Dunn and M. Schumacher . “Explaining Events to Computers: Critical Quanti昀椀cation, Multiplicity and Narratives in Cultural Heritage.” DInH:Q: Digital Humanities Quarterly 10.3 ( 2016 ). url: http://www.digitalhumanities.org/dhq/vol/10/3/000262/000262.ht m.l [10] G. Genette . Narrative discourse: an essay in method . Ithaca, N.Y: Cornell University Press, 1980 . [11] E. Gius , J. C. Meister , M. Meister , M. Petris , C. Bruck , J. Jacke , M. Schumacher , D. Gerstorfer , M. Flüh , and J. HorstmannC. atma. 2022 . doi: 10 .5281/zenodo.6419805. [12] E. Gius and M. Vauth . “ Towards an Event Based Plot Model. A Computational Narratology Approach” . In:Journal of Computational Literary Studies 1 .1 ( 2022 ). doi: https://doi .org/10.48694/jcls.11 0. [13] S. Jänicke , G. Franzini, M. F. Cheema , and G. Scheuermann. “ On Close and Distant Reading in Digital Humanities: A Survey and Future Challenges” . IEnu:rographics Conference on Visualization (EuroVis) - STARs ( 2015 ). Ed. by R. Borgo , F. Ganovelli , and I. Viola , pp. 83 - 103 . doi: 10 .2312/eurovisstar.20151113. [14] N. W. Kim , B. Bach , H. Im , S. Schriber , M. Gross , and H. P昀椀ster. “ Visualizing Nonlinear Narratives with Story Curves” . InIE:EE Transactions on Visualization and Computer Graphics 24.1 ( 2018 ), pp. 595 - 604 . doi: https://doi.org/10.1109/TVCG. 2017 . 274411 8. [15] I. Laptev , M. Marszalek , C. Schmid , and B. Rozenfeld . “ Learning realistic human actions from movies” . In:2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, AK , USA: Ieee, 2008 , pp. 1 - 8 . doi:https://doi.org/10.1109/CVPR. 2008 . 458775 6. [16] Y. Li , S. Narayanan , and C. Kuo . “ Content-Based Movie Analysis and Indexing Based on AudioVisual Cues” . In:IEEE Transactions on Circuits and Systems for Video Technology 14.8 ( 2004 ), pp. 1073 - 1085 . doi: https://doi.org/10.1109/TCSVT. 2004 . 83196 8. [17] C. Liu , A. Shmilovici , and M. Last . “ MND: A New Dataset and Benchmark of Movie Scenes Classi昀椀ed by Their Narrative Function” . In:Computer Vision - ECCV 2022 Workshops. Ed. by L. Karlinsky , T. Michaeli , and K. Nishino . Vol. 13804 . Cham: Springer Nature Switzerland, 2023 , pp. 610 - 626 . doi:https://doi.org/10.1007/978-3- 031 -25069-9\_3 9 . [18] J. P. Magliano and J. M. Zacks . “ The Impact of Continuity Editing in Narrative Film on Event Segmentation: Cognitive Science” . In:Cognitive Science 35.8 ( 2011 ), pp. 1489 - 1517 . doi: https://doi.org/10.1111/j.1551- 6709 . 2011 . 01202 . x. [19] P. Papalampidi , F. Keller, and M. Lapata . “ Movie Plot Analysis via Turning Point Identi昀椀- cation” . In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) . Hong Kong , China: Association for Computational Linguistics, 2019 , pp. 1707 - 1717 . doi: 10 .18653/v1/ D19 -1180. [20] V. F. Perkins . “ Where is the world? The horizon of events in movie 昀椀ction” . InS:tyle and meaning: Studies in the detailed analysis of 昀椀lm ( 2005 ), pp. 16 - 41 . [21] J. Phelan and P. J. Rabinowitz, eds.Understanding narrative. The theory and interpretation of narrative series . Columbus: Ohio State University Press, 1994 . [22] G. Prince . A Grammar of Stories: An Introduction . Berlin, Boston: De Gruyter, 1974 . doi: https://doi.org/10.1515/978311081590 0. [23] [24] [25] A. Rohrbach , M. Rohrbach , N. Tandon , and B. Schiele . “A Dataset for Movie Description” . In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2015 , pp. 3202 - 3212 . doi: https://doi.org/10.1109/CVPR. 2015 . 729894 0. A. Rohrbach , A. Torabi , M. Rohrbach , N. Tandon , C. Pal , H. Larochelle , A. Courville , and B. Schiele . “Movie Description”. InI:nternational Journal of Computer Vision 123 .1 ( 2017 ), pp. 94 - 120 . doi: https://doi.org/10.1007/s11263-016-0987- 1. A. Salway . “ A corpus-based analysis of audio description” . In: Leiden, The Netherlands: Brill , 2007 , pp. 151 - 174 . doih:ttps://doi.org/10.1163/9789401209564\_01 2. [26] J. Sang and C. Xu . “ Character-based movie summarization” . InP:roceedings of the 18th ACM international conference on Multimedia. Firenze Italy: Acm , 2010 , pp. 855 - 858 . doi: https://doi.org/10.1145/1873951.187409 6. [27] E. Segal . Beginnings and Endings . 2019 . doi: https://doi.org/10.1093/acrefore/978019020 1098.013.1051. [28] Y. Sun , Q. Chao , Y. Ji , and B. Li . Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding . 2022 . url: http://arxiv.org/abs/2203.0571 1. [29] S. Syed , T. Yousef , K. Al Khatib , S. Jänicke , and M. Potthast . “ Summary Explorer: Visualizing the State of the Art in Text Summarization” . InP: roceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Online and Punta Cana , Dominican Republic: Association for Computational Linguistics, 2021 , pp. 185 - 194 . doi: https://doi.org/10.18653/v1/ 2021 .emnlp-demo. 2 .2 [30] [31] M. Tapaswi , Y. Zhu , R. Stiefelhagen , A. Torralba , R. Urtasun , and S. Fidler . “ MovieQA: Understanding Stories in Movies through Question-Answering” . I2n0:16 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 2016 , pp. 4631 - 4640 . doi: https://do i. org/10 .1109/CVPR. 2016 . 501 . A. Torabi , C. Pal , H. Larochelle , and A. CourvilUlsei.ng Descriptive Video Services to Create a Large Data Source for Video Annotation Research . 2015 . arXiv: 1503 .01070 [cs.CV]. M. Vauth , H. O. Hatzel , E. Gius, and C. Biemann . “ Automated Event Annotation in Literary Texts.” In:Chr . 2021 , pp. 333 - 345 . url: https://ceur-ws. org/ Vol- 2989 /short%5C% 5 Fpaper18 .pdf. [33] G. Vercauteren. “ A narratological approach to content selection in audio description: towards a strategy for the description of narratological time”M.Inon:TI . Monografıás de Traducción e Interpretación 4 ( 2012 ), pp. 207 - 231 . doi: https://doi.org/10.6035/MonTI.20 12.4.9. [34] P. Vicol , M. Tapaswi , L. Castrejon , and S. Fidler . “ MovieGraphs: Towards Understanding Human-Centric Situations from Videos” . In2:018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2018 , pp. 8581 - 8590 . doi: https://doi.org/10.1109/CVPR.20 18 .00895. [35] Y. Xiong , Q. Huang , L. Guo , H. Zhou , B. Zhou , and D. Lin . “A Graph-Based Framework to Bridge Movies and Synopses” . In2:019 IEEE/CVF International Conference on Computer Vision (ICCV). 2019 , pp. 4591 - 4600 . doi: https://doi.org/10.1109/ICCV. 2019 . 0046 9. [36] Q. Yi , G. Zhang, J. Liu, and S. Zhang. “ Movie Scene Event Extraction with Graph Attention Network Based on Argument Correlation Information” . SInen:sors 23.4 ( 2023 ), p. 2285 . doi: https://doi.org/10.3390/s2304228 5. [37] J. M. Zacks and B. Tversky . “ Event structure in perception and conception .” InPs:ychological Bulletin 127.1 ( 2001 ), pp. 3 - 21 . doi: http://doi.apa.org/getdoi.cfm?doi=10.1037/00 33- 2909 . 127 . 1 .3.