We present the first publicly available dataset of English narrative texts annotated in compliance with SceneML, a framework for annotating scenes in narrative text. The dataset is composed of selected chapters from six narrative texts - two children's stories and four novels from Project Gutenberg. We give a brief overview of SceneML, describe the corpus sources and the annotation process and provide details of the resulting annotations and inter-annotator agreement.
Narrative, or storytelling, is a fundamental mode of human discourse, found across all cultures
and all times, and in many diferent forms, including writing (both fiction and non-fiction),
spoken storytelling, film, video games, and so on [
In previous work we have introduced SceneML as a framework for annotating scenes in
narrative text [
There has been a growing interest in computational analysis of narrative. Ranade et al. [
SceneML is an evolving framework for annotating scenes in narrative text. The latest specification and annotation guidelines are available along with the corpus at the DOI referenced above. Here we summarise the core concepts in SceneML.
A scene is defined as a unit of narrative in which the time, location and principal characters are constant and in which specific events which constitute the narrative are recounted. Any change in time, location or characters indicates a change in the scene. A scene is realised in text (for written forms of narrative) through one or more scene description segments (SDSs). The SDS mechanism allows for the relation of one scene in a narrative to be embedded within another, as for example, in flashback or flashforward. The task of scene identification thus becomes the task of identifying the boundaries of SDSs, and linking SDSs for the same scene together 2. SceneML also specifies a set of four narrative progression relations (sequence, analepsis, prolepsis and concurrence) that are used to capture the temporal relations between scenes.
Typically, not all text in a narrative is part of a scene description. Some passages describe not one scene or another but rather the transition between scenes. For example, in Conan Doyle’s The Man With The Twisted Lip the first scene takes place in Watson’s house and the second 1https://creativecommons.org/licenses/by-nc/4.0/. 2Full scene annotation in SceneML also involves annotating the time, place and characters (named entities) in the scene, using existing annotation standards (ISO-TimeML,ISO-Space and the ACE NE guidelines). However, in ScANT we focus on scene segmentation only in the East End of London, where Watson goes to seek a missing man. Between the two we have the short passage: “And so in ten minutes I had left my armchair and cheery sitting-room behind me, and was speeding eastward in a hansom on a strange errand...”. Such elements SceneML refers to as scene transition segments (STSs). Other sorts of non-scene elements are also present in narrative. These include general philosophising or opinion segments, background information segments, and narrative summary or narrative catchup. These passages serve a variety of functions but do not relate specific, situated events involving protagonists in the story. All such passages SceneML designates as non-scene elements.
The dataset is composed of selected chapters from children’s stories and from out-of-copyright adult novels. The former were hypothesised as likely to have a simpler narrative structure and hence to be a good place to trial our approach; the latter as likely to possess more complex narrative structure and hence pose a more challenging test to our approach. The sources are: (1) Bunnies from the Future, a middle grade children’s story by Joe Corcoran 3. The author has personally granted permission for us to release annotated chapters of this work. (2) The Wonderful Wizard of Oz, originally released as part of the Brown Corpus 4 and free for noncommercial purposes. (3) Pride and Prejudice, A Tale of Two Cities, The Adventures of Sherlock Holmes and The Great Gatsby from Project Gutenberg 5. These are out of copyright in the US and UK and freely re-distributable subject to Project Gutenberg’s terms and conditions.
In an earlier pilot study [
The annotation process was carried out through a web-based interface to a local instance of Brat Annotation Tool 6. Annotators used swipe and click operations to annotate SDSs and STs. Multiple SDSs that are part of the same scene were linked using the Brat relation annotation 3https://freekidsbooks.org/author/joe-corcoran/ 4https://www.nltk.org/nltk_data/ 5https://www.gutenberg.org 6https://brat.nlplab.org tool to signal that a same-scene-as relation holds between them. The annotated data is stored and made available in BRAT standof annotation format 7.
The corpus consists of fourteen chapters from six diferent narrative sources 8. In each chapter SDSs, STs and same-scene-as relations were annotated by two annotators and saved in a separate text file. Both annotators’ annotations are supplied with the corpus. Further annotations together with a consensus annotation may be made available in the future.
7It can be converted to JSONL format using the tool at: https://github.com/astutic/brat-standoff-to-json/. 8As one of the chapters is quite long it has been divided into three parts for analysis.
Table 2 shows inter-annotator agreement results for SDSs using Cohen’s Kappa [
Aside from calculating only exact matches as agreement ( = 0 in Table 2 ) we also investigated a more lenient approach to calculate the agreement in which annotators are deemed to agree if they place a sentence boundary within N sentences. This was prompted by the observation that in many cases annotators seemed to be placing SDS boundaries relatively close to each other, but not exactly in the same place. Kappa scores have been calculated for various N sizes: = 30% of the median SDS sentence length in each chapter, N=1, N=3 and N=5. We have omitted one chapter from Table 2 – A Tale of Two Cities, chapter 1, because one annotator believed it contained nothing but non-scene segments, while the other thought it contained 5 scenes. This gave a kappa score of 0, which skewed the rest of the results.
Regarding diferences between our annotators, it is clear that the two annotators have a diferent
conception of what STs and NSSs are. This is probably due to the fact that the children’s stories
which we used as training materials contain very few of either of these, particulary of NSSs
(in fact this appears to be an interesting diference between children’s and adult narrative).
Any future annotation efort should ensure that these concepts are more clearly understood by
annotators. On examination our view is that A1 has followed the guidelines much more closely
regarding STs and NSSs and therefore, if one is to train or test a classifier on these materials,
our recommendation would be to use the A1 annotations only. However recent work such as
that reported in Uma et al. [
Concerning the kappa scores for SDS agreement, they fall in the range that has been
interpreted as ”fair” or ”fair to good”. However, kappa scores are known to be lower when there
are fewer labels (just two in our case) and where the labels are not equiprobably occurring
(also true in our case, since 0 labels are much more frequent than 1’s) so results should be
viewed in this light [
While the corpus is too small to start making generalisations about stylistic diferences between diferent authors, it is worth noting that the amount of non-scene content in the adult novels (6.21% of the total sentences if we accept A1’s NSS annotations, which we believe are more accurate) is vastly greater than that in the children’s stories (0.14%), suggesting that much beyond simple event narration goes on in adult fiction.
We have presented the first dataset of English narrative texts annotated in compliance with SceneML. The dataset consists of fourteen chapters of novels and children’s stories annotated for scene description segments and scene transitions segments as defined in SceneML. A total of almost 200 scenes have been annotated.
Future work plans include various activities. These include: 1. gathering further annotations for the ScANT source texts to increase robustness of the annotations and guidelines; 2. extending the annotation to include SceneML narrative progression links; 3. training a model on the corpus to investigate automating the task of scene boundary detection and to ascertain the suficiency of ScANT for this task; 4. adding other text types to the annotated dataset, such as biography, plays and film scripts; 5. expanding the dataset to include texts in other languages.
6. exploring whether there is interest in a shared task challenge on scene boundary detection. We hope the community finds ScANT of use and welcome comment on our work.
The authors thank the Text2Story reviewers for their helpful comments. The first author acknowledges support from the University of Jeddah in the form of a PhD studentship.