The plot lines of modern ctional movies often involve complex transitions in time that results from presenting the narrative material of the plot in an order radically di erent from that in which events took place. Examples such as Pulp Fiction or Memento have accustomed viewers to dealing with plot storylines presented with very severe disruptions relative to the original chronology of the story being told. Such breaks in chronology are sometimes just presented seamlessly and left for the viewer to identify and sometimes presented as ashbacks initiated by a character in the story telling other characters about events in the past. This is an occurrence of embedded stories, which introduces di erent narrative levels: the frame story in which the telling character appears, and the embedded story that the telling character tells. The situation becomes even more di cult when such mechanisms are combined with further complexity arising from stories that occur in more than one world, i.e, stories that combine characters in the real world either travelling to or telling about a ctional world that is di erent from the real one.
The challenge of correctly representing the plot lines of this type of movie has to address a number of aspects. First, a mechanism needs to be provided to represent relative chronology of events along a particular timeline. Second, it needs to consider explicitly two separate timelines: the one for when events are described in the discourse presented to the viewer and the one for when events happen in the storyworld being described. Third, it needs to allow the explicit representation of di erent worlds. Each di erent world will have its own timeline independent of other worlds. Fourth, it needs to consider that di erent narrative levels are likely to involve di erent timelines. These will either be timelines for a di erent world-if the embedded story refers to a di erent world than the frame story-or for the same world but at a di erent time relative to the point of the frame story in which the embedded story appears-if the embedded story is for instance a ashback. Finally, if embedded discourses introduced by modal verbs are involved-subclauses of a sentence describing possibilities, wishes, intentions, plans . . . -the representation needs to distinguish between fragments of a particular world that are told to the viewer as being true in the world and those that are not. When events are not being told as true, the modality involved (wish, plan, intention. . . ) needs to be identi ed.
The present paper describes a proposal for a representation that addresses these requirements. This proposal arose in response to a need of a representation expressive enough to capture the complexities of movie plots in annotations intended for a corpus of movie plots annotated with elements relevant to narrative structure. The idea of building such a corpus came about to inform a process of automated processing of narrative to build explicit representations of narrative structure.
Four di erent topics need to be covered to provide background for the paper: the di erent layers in narrative that may result in di erent chronologies for the same story, and existing e orts to represent time, narrative levels and narrative structure.
An important distinction to be made is the one between a sequence of events and a discourse that orders and presents events, as suggested by [1]. Even then the actual assignment of speci c terms to each these two concepts varies depending on the particular theorist. The sequence of events told more or less alligns with what classical narratologists refer to as fabula or histoire, the discourse that presents the events in a particular order similarly alligns with what is traditionally referred to as sjuzhet or discours. To reduce the risk of confusion, we will refer in this paper to the actual sequence of events as fabula and to the order of presentation of events in a given telling as discourse. An important challenge to address will be to capture the di erence in relative order of appearance of events between fabula and discourse.
The challenge of identifying di erences in relative chronology for the events of the story between the order in which they appear in the fabula and the order in which they appear in the discourse has been identi ed by del Monte and Marchesini [2]. They propose a manual method for identifying in the text elementary units of narrative structure, "which are processes or event-sequences of clearly demarcated duration", called narremes after Collier [3], and then reorganizing these narremes chronologically into a representation of the fabula of a novel.
Even though the basic mode of communication using language is to assume that statements are intended to be true descriptions of the world, speakers may also refer to events by assigning to them a di erent attitude with respect to their truth value. This di erent attitude towards the relation between the given statement and reality or truth is usually marked explicitly by modality [4]. Di erent languages address this task in di erent ways. In English, the most usual means for expressing modality are either auxiliaries-such as may or can -or speci c verbs that carry the corresponding meaning lexically -such as want or need. When attempting to establish relative chronology between events in a discourse, events within the scope of a modal statement must be considered speci cally.
A similar situation arises with respect to reported discourse, where a speaker reports an utterance or belief originally contributed by someone else [5]. This can take the form of direct speech-the exact words of the reported agent are given verbatim-or as indirect speech-what the other person said is paraphrased. These di erent forms are usually marked typographically. In English, direct speech is marked by enclosing it between quotes, and indirect speech is usually introduced by a verb of speech followed by complementizer that. Instances of reported speech involve two di erent times: the time when the speech is reported and the time when the reported speech took place.
A related phenomenon is that of embedded stories. These occur when a speaker within a given discourse tells a story as part of her discourse. In narratological terms, the original discourse of the speaker acts is considered the frame story, and the story that she tells is known as the embedded story [6]. Because similar situations may also take place within the story being told, the phenomenon is recursive. Each time an embedded story is told, a new narrative level is being introduced. Each di erent narrative level is likely to have a di erent chronology and may even have a di erent modality, depending on whether the story is factual or ctional [7].
The annotation of narrative levels was the subject of a workshop that established a set of guidelines and an annotated corpus [8]. The set of guidelines was further re ned in [9]. These guidelines opt for a system of identify narrative levels in terms of relative depth, with number 1 assigned to "a narrative that is not embedded in any other narrative", and then numbering successive embeddings with increasing numbers. In each case, the number only indicates a similar depth of embedding, and it does not imply any type of relation between stories embedded at a similar depth. The guidelines outline the di erence between story time-actual series of events that occur-and narrative time-series of events as presented. They rely on Genette [6] for their de nition of anachrony -"all forms of discordance between the two temporal orders of story and narrative". The guidelines are designed to annotate spans of text that correspond to di erent narrative levels. The annotation begins "at the point in the text where the narrative transitions either to a higher or lower level" and ends "when the phenomenon ends and the narration returns to the level it had previously been at". The guidelines indicate that such changes may ocurr in the middle of a sentence and that the new narrative level may carry on into another sentence, paragraph or even chapter. Changes in relative chronology are only marked either as prolepsis-evoking in advance an event that will take place later-or analepsis-evocation after the fact of an event that took place earlier.
Further re nements of the annotation guidelines [10] argue that when a character's thoughts are presented vividly this may suggest an additional change in narrative level, in the sense that a di erent world is being considered [11]. They argue that the same narrative level pertains, and only the world being referred to changes.
The need to represent separately di erent narrative levels when news articles include reported speech is defended by Silvano et al. [12]. This proposal includes means for annotating the instances of reported speech, and clustering the set of events so reported as connected by a temporal link of type Identity. The authors argue that this solution addresses the problem that news articles do not usually allow the establishment of an accurate relative temporal order between the events reported.
It has long been accepted that narrative involves telling about events that happened in time and that telling itself takes time [13]. In computational terms, the fundamental reference as to how to represent time is Allen's temporal algebra [14], which outlines the set of possible temporal relations between events that themselves take a nite amount of time (duration) to happen. Allen outlines that events are not just set at particular points on a timeline, but that they should be considered as intervals that can also overlap (or not), be contiguous (or not), or be contained within one another (or not). He proposes a set of mathematical relations to describe all possible relations between such events.
The speci c challenges of representing time in narrative are addressed by Mani [15]. Mani argues that narrative relies on temporal abstractions that may involve considering time either in vague terms, or with imprecise information or at di erent granularities. To deal with these di culties, he proposes the concept of chronoscope, an underspeci ed representation of ambiguity based on factoring out common elements. By relying on temporal abstractions chronoscopes allow annotators to collapse temporal relations, or to zoom the representation to di erent time granularities.
An e ort to annotate texts with temporal information is TimeML [16]. The annotation guidelines for TimeML provide de nitions for how to identify events-which may be introduced by verbs, nominalizations, adjectives, predicative clauses or prepositional phrases-and how to mark the corresponding spans of text with XML-type tags. Such types allow the introduction of attributes, including an event ID. It also allows the introduction of link tags that serve to annotate relations between events. Events are represented in a link tag by its ID. TLINKS are speci c link tag intended to annotate temporal relations between events.
The importance of annotating relative temporal information in narratives is highlighted by Macovei [17]. Macovei postulates the concept of time tracks built out of time segments, which are groups of events presented in order. The temporal relations between time tracks and time segments can be used to analyse the chronology of stories. Macovei marks here time tracks and time segments using TimeML. The goal of having time tracks is to allow the visualization of the adventures of a particular character, separate from the rest of the story.
The annotation of temporal relations over narrative texts is taken to the extreme in Narra-tiveTime, the rst timeline-based framework for full temporal annotation [18]. NarrativeTime builds a dynamic timeline which compiles and visualizes all the relative temporal information contained in the full set of all possible TLINKS in the text. It also allows for branching of the timeline to represent uncertainties in temporal information as reported in text.
A related e ort is SceneML [19], a framework for annotating scenes in narratives. SceneML relies on the concept of scene-a span of text that exhibit continuity of time, location and characteras basic unit of annotation for narratives. SceneML allows for the explicit representation of entities-scenes, scene descriptions segments (SDSs), characters, times and locations-and relations-scene-scene narrative progression links, relational links connecting times, characters, locations to the scenes in which they are participant elements and connecting SDSs to the scenes they comprise. Narrative progression links are used to indicate relations between scenes that are adjacent. Relations considered are: sequence-change of scene involves only character or location-, analepsis-there is a ashback in one scene to a scene in the past-, prolepsisashforward to a scene yet to come-and concurrent -the transition takes us to di erent characters and a di erent place but at the same time.
The main contribution of this paper is a proposal for a method to represent the set of events narrated in a story in such a way that it provides an informative representation of their relative chronology. This representation is intended to allow the explicit capture the in uence on the relative chronology of events of the aspects outlined in the introduction, namely, transitions between di erent universes, di erent narrative levels, and di erent types of modality of reported speech.
In order to achieve this, the events narrated in the story are represented explicitly as action units. An action unit is a representation of an event that includes the following attributes:
• Universe: the world in which the event takes place • Narrative Level: the depth of narrative level at which the event is narrated (with respect to the primary frame story, which is numbered as 0) • Action Unit ID: a univocal identi er for the action unit • Mode: identi es events that di er in the degree of truth assigned to them by the speaker (captures issues such as modality) • Relative Order: describes temporal relation of the action unit with other action units, using Allen's temporal algebra (number refers to each of the types of relations proposed by Allen) • Content transcription: a number of elds that capture the informational content of the action unit, including: type of action, subject of the action, direct object of the action, indirect object of the action (if present), location of the action, and a set of action units that may be referred to when the action involves a verb of speech Whenever verbs of speech are involved, the clause corresponding to the reporting of the speech and the clause corresponding to the reported speech are encoded as separate action units, and the reported clause is referred by its Action Unit ID as direct object of the reporting clause. This allows both for the explicit notation of di erences between the two clauses in terms of world, narrative level and chronology, and for the relation of embedding between them.
The proposed scheme for representation is illustrated with examples from the lm "The Princess Bride" (Rob Reiner, 1987). An example of this situation is shown in Figure 1. The scene describes how the mother enters the room while the boy is playing video games and tells the boy that his grandfather has arrived. The telling is divided into action unit 2 (the telling, an event that actually happens) and action unit 4 (the information told, reference to grandfather's arrival, an event that has already happened). The representation for action unit 2 states explicitly that action unit 4 is told as direct object of action 2 (encoded by referring to the Action Unit ID in the Direct Object column). Because the event referred to by action unit 4 is mentioned rather than presented as actually happening, a di erent narrative level is assigned to it (1 instead of 0). The type of reference is indicated in the Mode column, to show that this event is mentioned as something that has happened in the past. The temporal relation between action unit 2 and action unit 4 is described (in terms of Allen's relations) in the column for Relative Order, stating that 4 precedes 2. This implies that 2 happens after 4 but not necessarily contiguously. This type of relation allows us to capture a certain vagueness in the inferred temporal relations. The entrance of grandfather into the room, which happens a moment later, is represented as action 5. The scene captures two separate streams of events: the grandfather's arrival at the house (action unit 4), and the stream of actions we actually see on the screen (1, 105, 106 and 5). Action 4 is assigned to narrative level 1 (event being told), whereas all the other actions are assigned to narrative level 0. The relative temporal order within the stream of action units 1, 105, 106 and 5 is de ned as X meets Y, because each action unit does takes place immediately after the preceding one in that stream. In this example, all the action units take place in the same story world, so the value for the column Universe remains constant (A, referring to contemporary USA).
The representation of narrative levels involving a change of world can be shown in the example in Figure 2. In this case the discourse alternates between events happening in the real world (grandfather reading the story to the child, action units 7, 7a, 10 and 10a) and events happening in the ctional world (the story of Wesley and Buttercup, action units 8, 9, 11 and 12). In this case, the event of reading particular segments of the book has been encoded as two separate action units, one describing the physical act of reading from the book (action unit 7) and one that describes the act of reading out aloud the events that the book tells (action unit 7a). Allen's equals relation has been used to indicate the fact that these acts overlap in time. The same considerations apply to action units 10 and 10a. It is important to note that in this case both the world in which the action units happen and the narrative level change between these two streams of events: action units 7 and 10 take place in universe A and narrative level 0, and action units 8, 9 , 11 and 12 take place in universe B and narrative level 1. The actual nesting between reporting and reported events is represented explicitly to show that action unit 7 involves the telling of action units 8 and 9 (action unit IDs 8,9 indicated as Direct Object for action unit 7), and action unit 10 involves the telling of action units 11 and 12 (action unit IDs 11,12 indicated as Direct Object for action unit 10).
The temporal relations in this example are particularly interesting because they illustrate the challenges that the task faces. The temporal relations shown between events that take place in universe A show a continuous timeline: 7 is followed by 10. However, the temporal relations shown shown between events that take place in universe B obey a separate timeline: 8 is followed by 9 which is followed by 11, which starts 12. Whenever the narrative involves a change in universe, a separate timeline needs to be considered. Action unit 12 di ers from the preceding ones in that it describes not an event of speci c duration but the setting in motion of a process that will develop over time. For this reason its temporal relation is de ned as starting after event 11. This use of Allen's relation allows us to capture some of the underde ned nature of relative temporal relations in narrative discourse.
The representation of narrative levels that involve a longer story but not involving a change of world can be shown in the example in Figure 3 In this case, Iñigo Montoya tells (action unit 43) the Man in Black about his father's death (action units 45 to 51) before challenging the Man in Black to a duel (action unit 52). All the action units happen in the same world (universe B), but at di erent narrative levels. The story being told by the grandfather to the boy corresponds to narrative level 1, and the story told by Iñigo Montoya to the Man in Black corresponds to narrative level 2. The change of location between the two narrative levels is also represented: narrative level 1 takes place in the ruins atop the cli , narrative level 2 takes place in the smithy belonging to Iñigo's father. In terms of chronology, the transcription indicates that the death of Domingo Montoya precedes (in the ctional world) the rst action that we have been told of the story (the rst event of the story, action unit 45 precedes action unit 8, Wesley's rst interaction with Buttercup). The relative limits of Iñígo's telling of the story are represented by indicating that the event immediately before the reporting starts the telling (action unit 43). and the event immediately after the telling (the challenge that is action unit 52) nishes it. The set of action unit IDs for the events in Iñigo's story is indicated as direct object of the telling (action unit 43).
A further example in Figure 4 shows how modal events are represented. In this scene, following the kidnapping of Buttercup (action unit 26) Vizzini tells (action unit 27) his confederates about this plan (action units 28-35). In this case a similar structure as in the prior example is used to represent the telling. The kidnapping of Buttercup (action unit 26) starts the telling and the departure on a ship (action 36) nishes it.
However, in this case, the set of events told is presented as a plan, which implies a speci c modality: this is not something that has happened but something that Vizzini intends to happen in the future. This is indicated in the Mode column by assigning the plan value to these events.
The play of narrative levels is more interesting here than in the preceding examples. Again, the main reference here is narrative level 1 (the story the grandfather is telling the boy), and Vizzini's telling of this plan becomes narrative level 2. However, the plan itself involves two further instances of nesting. These further nestings involve a certain modality. First, Vizzini explains (as part of his telling in action unit 27) that Humperdinck will suspect (action unit 29) that the Guilderians have kidnapped Buttercup (action unit 30). The relations between these events are complex. Humperdinck's suspicions (action unit 29) are only part of Vizzini's plan. The event that Humperdinck (may) suspect (action unit 30) also carries a certain modality: even if Vizzini's plan succeeds, this is only a suspicion (so it is marked as such using the suspicion label). The fact that these events are nested within one another implies that action unit 30 has a layered modality: a suspicion that is part of a plan. This fact must be taken into account when making inferences related to it or based on it. Second, Vizzini explains (as part of his telling in action unit 27) that-according to his plan-Humperdinck will, on nding Buttercup's body (action unit 33) believe (action unit 34) that the Guilderians have killed Buttercup (action unit 35). The relations between these events are similarly complex. Humperdinck's belief (action unit 34) is part of Vizzini's plan. The event that Humperdinck (may) believe (action unit 35) also carries a certain modality: even if Vizzini's plan succeeds, this is only a belief (so it is marked as such using the belief label). The fact that these events are nested within one another implies that action unit 35 has a layered modality: a belief that is part of a plan. Again, this must be taken into account when making inferences related to it or based on it.
The modal nature of some of these statements implies that, even though they are predicated on the same world as the events that they are nested in, they may not be considered on the same relative timeline. The temporal relations that can be inferred are relatively clear: both Humperdinck's suspicion and his belief refer to events that have happened in the past. However, their modality implies that, while we can establish a relative temporal order between the moment when Humperdinck holds the corresponding thoughts and the suspected or believed events, we can not actually place them in a speci c relative order with the non-modal events. This is true not only with respect to the events that have already happened (because the events in the plan have not themselves happened) but also with respect to the events in Vizzini's plan (both the suspicion and the belief are in fact in contradiction with the events that Vizzini plans). This suggests that more complex representation mechanisms may be required to deal with these cases.
It is important to note, that, in contrast with many of the approaches reviewed in Section 2, the proposal presented in this paper does not involve directly annotating over the text that conveys the narrative, but rather it builds a parallel representation that explicitly captures the relevant information gleaned from the text. This representation is intended to allow the explicit capture the in uence on the relative chronology of the issues outlined in the introduction. Nevertheless, as similar phenomena need to be captured, the proposed representation can be compared to the methods for representing time used in existing annotation processes.
The approach followed in the proposed schema with respect to the representation of narrative levels di ers from the one proposed by Kearns [9] in that rather than marking that start and end of the span of text corresponding to a narrative level it represents individual events as separate representational units, and it assigns a narrative level number to each event. However, the spans of events for a given narrative level may be reconstructed by clustering the events with the same narrative level that are contiguous in the representation. When annotating, Kearn's approach only captures changes in narrative level, whereas our approach also captures changes in world that is being described, whenever a change in narrative level also implies a change to narrative events that take place in a di erent world. This is frequent when the embedded story occurs in a ctional world. This richer representation allows a ner grain of analysis of ctional texts, where changes in actual world are frequent, and of modal discourses, which involve a change in possible world even when the actual world is taken as reference. The representation of temporal information associated with narrative levels in Kearns' approach is restricted to whether the change in level involves a change forward or backward relative to story time. Our proposal allows for more detailed representation of relative temporal relations based on Allen's temporal algebra.
With respect to the proposal in Silvano et al. [12], the solution proposed here attempts to explore in more detail what relative temporal relations might be inferred between the events. The set of relative temporal relations that may be inferred is limited: in some cases the relations are vague and in some cases there is insu cient information to establish speci c relations.
The detailed notation for temporal abstractions proposed by Mani [15] for would allow a more ne-grained representation of temporal relations than is a orded by the current solution based on Allen's temporal algebra [14], as it does not require precise information about temporal relation. An extension of the proposal in this paper to allow the use of chronoscopes will be considered as further work.
The notation for clauses introduced in this paper alligns with the elements of text that are associated to events in TimeML [16]. They are also assigned an ID, and relative temporal relations with other events are explicitly in the X column in a way similar to the use of TLINKS.
The scheme we propose allows explicit representation of characters as in SceneML [19] but it presents an advantage of more detailed representation of temporal relations between events using Allen's temporal algebra, as discussed in the comparison with Kearns' approach.
The scheme proposed in this paper allows, by ltering the set of events based on the values for particular columns, for the visualization of the adventures of particular characters as intended by Macovei with the use of time tracks [17]. The importance of being able to consult a full picture of the relative temporal timeline for the various events in a story is also advocated by NarrativeTime [18]. With respect to these approaches, our proposal adds explicit representation of di erent worlds that might be involved in di erent segments of the text, as in texts that combine ctional and non-ctional worlds, or texts that refer to di erent historical periods.
The paper proposes a representation for the events in a ctional movie plot that captures the mention of di erent worlds, transitions between narrative levels arising from the telling of stories embedded in other stories, modal statements and relative chronologies between the various timelines involved (for di erent worlds, di erent modalities, or di erent temporal segments within the same world). The examples presented suggest that existing annotation schemas may need to be extended with additional representational mechanisms to account for the set of phenomena discussed in the paper. As future work we intend to explore how the proposed representation can be applied to the plots of a number of movies.
This paper has been partially supported by the CANTOR project (PID2019-108927RB-I00) funded by the Spanish Ministry of Science and Innovation.