We present the first publicly available dataset of English narrative texts annotated in compliance with SceneML, a framework for annotating scenes in narrative text. The dataset is composed of selected chapters from six narrative texts -two children's stories and four novels from Project Gutenberg. We give a brief overview of SceneML, describe the corpus sources and the annotation process and provide details of the resulting annotations and inter-annotator agreement.
Narrative, or storytelling, is a fundamental mode of human discourse, found across all cultures and all times, and in many different forms, including writing (both fiction and non-fiction), spoken storytelling, film, video games, and so on [1]. A basic structural unit of narrative is the scene, "a unit of a story in which the elements of time, location, and main characters are constant" [2]. Narratives tend to progress as a sequence of scenes, though of course the sequence of scenes in a narration need not be the same as the temporal sequence of the narrated events in the storyworld [3] the narrative is describing. Furthermore, one scene may be expressed in multiple non-contiguous text segments in the narration. Additionally, what are generally deemed narrative texts may include various non-narrative elements, e.g. authorial comment. Thus, the task of identifying those chunks in a narrative text which correspond to scenes in the storyworld and temporally ordering these chunks is a non-trivial challenge. It is an important challenge both for the insights it gives us into the structure of narratives and for possible applications, which include automatic story illustration, aligning books and movies, automatic generation of image descriptions and automatic generation of narratives.
In previous work we have introduced SceneML as a framework for annotating scenes in narrative text [4] and discussed issues arising in a pilot annotation exercise which focussed on the scene identification task [2]. In this paper we present ScANT, the first publicly available dataset of English narrative texts annotated in compliance with SceneML. While the corpus is small -just 14 chapters from 6 narrative sources -our hope is that the wider community will find this useful both for converging on annotation standards for scene identification and for initial training and testing of automatic scene identification algorithms. The corpus and annotation guidelines are available at https://doi.org/10.15131/shef.data.21517908 and are made available under the CC By-NC 4.0 licence 1 .
There has been a growing interest in computational analysis of narrative. Ranade et al. [5] provide a thorough overview of recent work on computational understanding of narrative and Santana et al. [6] provide an extended survey on narrative extraction from textual data. However, neither of these addresses the issue of identifying scenes in narrative text. The only other work on annotation of scenes in narrative texts of which we are aware is that of Zehe et al. [7]. Their work differs from ours in several respects. First, their definition of scene states that a scene is a segment in a narrative in which the time, place and characters remain constant and which centres around one action. This contrasts with our definition that does not take into account the actions in a scene and allows multiple actions to happen in one scene (see Gaizaukas and Alrashid [4] for discussion around our choice of definition). Secondly, their scheme is less comprehensive -it does not define narrative progression links between scenes or scene transition segments, and only distinguishes scene and non-scene segments. Thirdly, they follow a container principle (small places make up larger places) to detect a change in place, e.g. if the action of characters moves from a corridor to dining room that will not indicate a change in place as they both part of a hotel, where as our definition counts these as two different places. Finally, they work on German texts while we are working on English texts.
SceneML is an evolving framework for annotating scenes in narrative text. The latest specification and annotation guidelines are available along with the corpus at the DOI referenced above. Here we summarise the core concepts in SceneML.
A scene is defined as a unit of narrative in which the time, location and principal characters are constant and in which specific events which constitute the narrative are recounted. Any change in time, location or characters indicates a change in the scene. A scene is realised in text (for written forms of narrative) through one or more scene description segments (SDSs). The SDS mechanism allows for the relation of one scene in a narrative to be embedded within another, as for example, in flashback or flashforward. The task of scene identification thus becomes the task of identifying the boundaries of SDSs, and linking SDSs for the same scene together2 . SceneML also specifies a set of four narrative progression relations (sequence, analepsis, prolepsis and concurrence) that are used to capture the temporal relations between scenes.
Typically, not all text in a narrative is part of a scene description. Some passages describe not one scene or another but rather the transition between scenes. For example, in Conan Doyle's The Man With The Twisted Lip the first scene takes place in Watson's house and the second in the East End of London, where Watson goes to seek a missing man. Between the two we have the short passage: "And so in ten minutes I had left my armchair and cheery sitting-room behind me, and was speeding eastward in a hansom on a strange errand...". Such elements SceneML refers to as scene transition segments (STSs). Other sorts of non-scene elements are also present in narrative. These include general philosophising or opinion segments, background information segments, and narrative summary or narrative catchup. These passages serve a variety of functions but do not relate specific, situated events involving protagonists in the story. All such passages SceneML designates as non-scene elements.
The dataset is composed of selected chapters from children's stories and from out-of-copyright adult novels. The former were hypothesised as likely to have a simpler narrative structure and hence to be a good place to trial our approach; the latter as likely to possess more complex narrative structure and hence pose a more challenging test to our approach. The sources are:
(1) Bunnies from the Future, a middle grade children's story by Joe Corcoran 3 . The author has personally granted permission for us to release annotated chapters of this work. ( 2) The Wonderful Wizard of Oz, originally released as part of the Brown Corpus 4 and free for noncommercial purposes. (3) Pride and Prejudice, A Tale of Two Cities, The Adventures of Sherlock Holmes and The Great Gatsby from Project Gutenberg5 . These are out of copyright in the US and UK and freely re-distributable subject to Project Gutenberg's terms and conditions.
In an earlier pilot study [2] we investigated how well-defined the SceneML definitions and annotation framework were with respect to scene boundary identification. Analysis of the annotations in that study revealed several causes of observed disagreement: (1) lack of understanding of the guidelines and task, (2) lack of clarity or specificity in the guidelines, (3) failure of non-native English speakers to fully grasp the meaning of certain expressions (e.g. idioms). We have addressed these issues in the construction of ScANT through the following steps: (1) A more thorough training process that included both an initial training session with a presentation, demonstration and hands-on exercise for the trainees, plus a follow-on take-away exercise that was scored against gold-standard annotations produced by the authors and then discussed with the trainees, (2) Improvement of the initial guidelines to remove sources of confusion revealed in the earlier pilot, (3) recruitment of native English-speaking annotators with sensitivity to text analysis (two PhD students, one in English Literature and one in Computational Linguistics).
The annotation process was carried out through a web-based interface to a local instance of Brat Annotation Tool 6 . Annotators used swipe and click operations to annotate SDSs and STs. Multiple SDSs that are part of the same scene were linked using the Brat relation annotation tool to signal that a same-scene-as relation holds between them. The annotated data is stored and made available in BRAT standoff annotation format 7 .
The corpus consists of fourteen chapters from six different narrative sources 8 . In each chapter SDSs, STs and same-scene-as relations were annotated by two annotators and saved in a separate text file. Both annotators' annotations are supplied with the corpus. Further annotations together with a consensus annotation may be made available in the future.
Table 1 shows summary statistics for the ScANT corpus and associated annotations. Note that the relation between scenes and SDSs is largely one-to-one. With one exception this is always true for the children's stories, while there is somewhat more variation, suggesting more complex narrative form, in the adult novels.
The variation between annotators A1 and A2 is relatively small in terms of SDSs and Scenes. However, they are far apart regarding both scene transition segments and non-scene segments. We discuss this further below in Section 4.3. First, however, we examine inter-annotator agreement regarding SDSs in more detail.
Table 2 shows inter-annotator agreement results for SDSs using Cohen's Kappa [8]. To calculate Kappa, each sentence is given a tag, 1 for sentences on the boundary of an SDS (either beginning or end) and 0 otherwise. Boundaries of STs are ignored as it is clear the two annotators' understanding of the task is so different that precise quantitative analysis is not merited. Aside from calculating only exact matches as agreement (๐ = 0 in Table 2 ) we also investigated a more lenient approach to calculate the agreement in which annotators are deemed to agree if they place a sentence boundary within N sentences. This was prompted by the observation that in many cases annotators seemed to be placing SDS boundaries relatively close to each other, but not exactly in the same place. Kappa scores have been calculated for various N sizes: ๐ = 30% of the median SDS sentence length in each chapter, N=1, N=3 and N=5. We have omitted one chapter from Table 2 -A Tale of Two Cities, chapter 1, because one annotator believed it contained nothing but non-scene segments, while the other thought it contained 5 scenes. This gave a kappa score of 0, which skewed the rest of the results.
Regarding differences between our annotators, it is clear that the two annotators have a different conception of what STs and NSSs are. This is probably due to the fact that the children's stories which we used as training materials contain very few of either of these, particulary of NSSs (in fact this appears to be an interesting difference between children's and adult narrative).
Any future annotation effort should ensure that these concepts are more clearly understood by annotators. On examination our view is that A1 has followed the guidelines much more closely regarding STs and NSSs and therefore, if one is to train or test a classifier on these materials, our recommendation would be to use the A1 annotations only. However recent work such as that reported in Uma et al. [9] highlights the potential value of learning from disagreement, so we have included both sets of annotations in the corpus.
Concerning the kappa scores for SDS agreement, they fall in the range that has been interpreted as "fair" or "fair to good". However, kappa scores are known to be lower when there are fewer labels (just two in our case) and where the labels are not equiprobably occurring (also true in our case, since 0 labels are much more frequent than 1's) so results should be viewed in this light [10]. Percentage agreement scores for SDS annotations are around 90%. Note that kappa scores rise significantly if we are prepared to allow some leniency in terms of non-exact matching. How legitimate this is needs further examination to determine whether the improvement is a reflection of genuine uncertainty about the precise boundary between what the annotators clearly agree are distinct scenes or whether it is the result of conflating separate scenes, according to the different annotators' perceptions.
While the corpus is too small to start making generalisations about stylistic differences between different authors, it is worth noting that the amount of non-scene content in the adult novels (6.21% of the total sentences if we accept A1's NSS annotations, which we believe are more accurate) is vastly greater than that in the children's stories (0.14%), suggesting that much beyond simple event narration goes on in adult fiction.
We have presented the first dataset of English narrative texts annotated in compliance with SceneML. The dataset consists of fourteen chapters of novels and children's stories annotated for scene description segments and scene transitions segments as defined in SceneML. A total of almost 200 scenes have been annotated.
Future work plans include various activities. These include: 1. gathering further annotations for the ScANT source texts to increase robustness of the annotations and guidelines; 2. extending the annotation to include SceneML narrative progression links; 3. training a model on the corpus to investigate automating the task of scene boundary detection and to ascertain the sufficiency of ScANT for this task; 4. adding other text types to the annotated dataset, such as biography, plays and film scripts; 5. expanding the dataset to include texts in other languages. 6. exploring whether there is interest in a shared task challenge on scene boundary detection.
We hope the community finds ScANT of use and welcome comment on our work.
The authors thank the Text2Story reviewers for their helpful comments. The first author acknowledges support from the University of Jeddah in the form of a PhD studentship.