As part of developing aComputational Narrative Understanding, modeling events within stories has recently received significant attention within the digital humanities community. Most of the current research aims at good performance when predicting events. By contrast, we explore a focused approach based on qualitative observations. We attempt to trace the role of structural elements - more specifically, temporal function words - that may be characteristic of a narrative's turning point. We draw on a corpus of UFO sighting reports in which authors employ a prototypical narrative structure that relies on a turning point at which the extraordinary intrudes the ordinary. Using binary logistic regression, we can identify structural properties which are indicative of turning points in our data, showcasing that a focus on detail can fruitfully complement NLP models in gaining a quantitatively informed understanding of narratives.
1. Introduction (1) I was in my room of a paying guest flat, 5th floor, and was about to go for my bath and then I suddenly noticed from my window an object glowing/flashing over a jungle area more than 1 km away from my apartment. (Report 76519) (2) As we drove north, 2 out of four of us saw a big bright blue ball of fire that looked as if it got brighter the closer it got to the ground. (Report 65963) (3) I was getting in my car, when all four of us – my grandson, my grandson’s tutor, my granddaughter, and myself – noticed a low, slow-moving, sideways teardrop-shaped object moving from north to south through the San Gabriel Mountains. (Report 4061) When people tell the story of something extraordinary which has happened to them, they use a particular kind of language. This is especially true for recounting moments when the Author 1 and 2 contributed 40 % each to this work, author 3 contributed 20 %. ‡ extraordinary intrudes the ordinary. The excerpts above are sentences stemming from texts about alleged UFO sightings which were collected online. Within these narratives, the first appearance of something out of the ordinary marks an importatnutrning point. When looking at these sentences, a pattern emerges: While they might difer from other parts of the text contentwise, they also tend to stand out structurally. More precisely, they typically include an adverbial which temporally grounds the event in relation to other parts of the narrative.
Concepts related to what we just introduced atsurning points are, among others, Labov’s
most reportable event [
Following observations such a(s1) – (3), we opted to focus onthen_ADV, as_SCONJ, and when_ADV as function words which frequently introduce temporal adverbials and seem to characterize turning point sentences. We test our hypothesis by specifying a model that predicts whether a sentence is a turning point or not, assessing whether the selected words are associated with a higher probability. By using a limited number of linguistic factors as predictors in our model, we aim to contribute to a better understanding of turning points as simpler models help to keep the impact of individual variables more transparent.
The computational modeling of narratives – both their constituent elements (e.g., characters
or events) and their overall structure – is a vibrant field of research1, 2[
While these studies broadly investigate the same phenomenon, they difer from our approach
in that they are predominantly concerned with predicting events while we aim at identifying
certain characteristics of those events we consider turning points. While, e.g., approaches like
the one by Ouyang/McKeown [
The data stem from a larger corpus of approximately 110,000 reports of UFO sightings submitted via the online platformUFO Stalker (https://ufostalker.com)and scraped by one of the authors.1 Texts are mostly written in English, presumably by people from the U.S., even though these metadata cannot be verified. The reports’ narrative shape is typically as follows: In a short exposition or staging phase, authors describe the – usually mundane – situation they say they were in and quite often who they were with at the time (cf. (4)). Then something happens: Most often, authors report that they suddenly see a strange light moving in the sky and such the ‘actual’ reporting of the sighting unfolds. This reporting is mostly an account of the author’s cognitive processes. Reports typically end without them reaching a definitive conclusion with regard to what it was that they saw. This puzzlement can be seen as the prototypical resolution of these narratives.
(4) I was inside my house with my wife, brother and his daughter. i thought i’d go out into my backyard. so i opened my door which faces east and walked out. i stopped about 6 feet from the door and felt like i needed to look up in the south direction. so i did and then i saw it right there in front of a low cloud. it like came out and went down about 25 feet then left about 25 feet then back then up and back again, then stopped and sat there. i was yelling at my wife and brother to come out here fast now! my wife was the first one yet could not see it cause she did not have her glasses on! then my brother came out and before he could actually look at it, it went into another cloud next to it. the funny thing is these clouds were kind of transparent so i do not know where it went it just went into it and vanished. at the time i saw it, it was a circular object just like a ufo to say. it was of a dark color yet you could see the sun hitting it. so it was there! but the moves were just to quick. it went from a straight down to a left turn in just a split second and did what i said above in the same time. but i did get to see it for the time mentioned above. it makes me wonder if it intended for me to see it. as when i walked out the door i had the urge to look in that direction. but who knows. this was what i saw and was amazed at what it did. (Report 60500)
We sampled 496 texts from the larger corpus of reports. These texts were preprocessed
using Stanza (Version 1.8.1) [
100
point which we operationalized as the one sentence where it becomes clear that the narrative is about a UFO sighting, i.e., we only annotated one turning point per text. Inter-annotator agreement was good (ICC(2,1) = 0.808, 95%-CI [0.766, 0.843]; ICC(3,1) = 0.81, 95%-CI [0.769, 0.845]). Disagreement was resolved via discussion. Reports that consisted of fewer than three sentences were discarded, in line with the data preparation done by Ouyang/McKeown9] [ following Prince’s definition of a minimal narrative12[]. Also excluded were reports that were written in languages other than English, described something other than a UFO sighting, were a mere description of photos or videos that were provided along with the report, did not feature any narrativity or did not include a discernible turning point. Finally, 352 reports consisting of 5,346 sentences were included in the analysis. Texts contained up to 81 sentences (Median = 12, IQR = 10).
To test the hypothesis laid out above, we fit binary logistic regression models with the probability of a sentence being the turning point as the outcome variable. As predictor variables we used dummy variables coding for whether the worwdshen_ADV, then_ADV oras_SCONJ occurred in a given sentence. Further, we opted to include two more structural variables. First, we added the sentence’s relative position within the text (the sentence’s index divided by the text’s length) as a percentage. Since we assumed a certain narrative structure, we knew that position within the narrative would play a role: We observed beforehand that the turning point is 60% 40% it n o P g n i n r u T y iilt b a b o rP20% 0% Length 8 16 32 0% 25%
50%
75%
100%
usually located toward the beginning of the story. Second, we included logged sentence length
as a predictor. Sap et al. found that what they camllajor events are usually expressed in longer
sentences [
Looking at descriptive evidence, the three selected words do exhibit diferent occurrence distributions depending on whether sentences are turning points or not (F1ig).. The percentage of sentences that includewhen_ADV is four times higher for turning points than for non-turning points. The same tendency, though less pronounced can be observed for the woarsd_SCONJ, whereas then_ADV occurs in a similarly sized share in both subsets of the corpus. Turning point sentences have a median relative position within the text of 16.7 (IQR = 19.4), so they are usually present in the earlier parts of the narrative (Fig. 2). Turning point sentences are also longer than non-turning point sentences in our data (MediTaPn= 25 vs. Mediannon-TP = 17; 60% it n o P g inn40% r u T y iilt b a b o r P 20% 0% 0% 25%
50%
75% 100%
We fit three separate models. In a first step, regressing the probability of a sentence being a turning point on a sentence’s relative position within the text, we estimate a negative association. This model described a small amount of variance (Tju r2’s= 0.113). In a second step, we added the logged sentence length (Tjur’s2 = 0.183). Fig. 4 plots the predicted probabilities of a sentence being a turning point against its relative position within the text (a percentage value close to 0 for the very beginning and 100 for the end of a text) for diferent sentence lengths. As can be seen, the model assigns very low probability to sentences after half of the narrative has passed whereas sentences that lie in the first quarter of the text are assigned probabilities between around 0.15 and 0.04 for shorter sentences and between 0.54 and 0.20 for very long sentences.
Adding our main variables of interest, namely the occurrence of temporal markers resulted in improved model fit (Tjur’s 2 = 0.213; for full model comparison, sDeeata Availability). The estimated coefÏcients for as_SCONJ and then_ADV were not statistically significant, which is in accordance with the descriptive patterns presented above. The worwdhen_ADV, however, was associated with an increased probability of a sentence being a turning point (Fi5g). Again, diferent sentence lengths predict diferent probabilities for a sentence being the turning point with longer sentences being associated with higher probabilities (Fi6g).
It is important to note that adding content words which we know a priori to be discriminative 80% 60% it n o P g n i n r ilityuT40% b a b o r P 20% 0%
Predicted Probabilities of Turning Point
Length = 8
Length = 16
Length = 32 0% 25% 50% of turning points for our specific genre of text would also result in better model fit – e.g., adding a binary variable that captures whether the worsdky appears in a given sentence (which is presumably typical of turning points in our texts since that is the locus of the extraordinary event) improves model fit (Tjur’s 2 = 0.242). While it is clear that including more or even all word occurrences into the model will result in better model fit or predictive power, respectively, it was not our aim to design a model that discriminates turning point sentences and non-turning point sentences perfectly – i.e., solve a classification problem – but rather test the theoretical question laid out above in a very specific exemplary genre of texts.
Our investigation of turning points in UFO sighting narratives was driven by a hypothesis on the role of certain content-independent characteristics of turning points and had a relatively narrow scope: Not only does our corpus consist of a very particular and, possibly, idiosyncratic genre of texts. Also, our study only used a small hand-annotated sample and focused on few variables that were situated at diferent levels: Position within texts and sentence length did already account for some variation in the probability of sentences being turning points. Regarding the role of temporal function words, we found that whwilheen_ADV is predictive of turning points,then_ADV and as_SCONJ are not. Thus, we were not able to identify a whole group or class of words that are used to mark turning points, but we did corroborate that the use ofwhen_ADV is predictive of a turning point. This finding supports our general hypothesis that turning points are characterized not only by their content, but also by structural properties such as temporal adverbials. Whether this also holds true for other types of narratives remains subject to further investigation.
Using state-of-the-art NLP methodology, there may be huge advances in the prediction of event types in narrative texts over the next few years. Another question, however, is how well these NLP models will serve us to understand what makes a turning point a turning point (or an event an event, for that matter). On a theoretical level, one can think about approaches like ours as modeling the reader, but also as modeling the author: What hints enable readers to place the content of a given sentence within the greater narrative? What hints does the author deem viable to trigger said interpretation? Do these cues vary between diferent genres that feature diferent narrative structures or schemas? These and many more questions should be addressed by future research from the vantage point of diferent disciplines – such as literary studies, linguistics, and psychology. This will help us gain a quantitatively informed understanding of (literary) narratives. We hope to have exemplified with this study how focusing on individual linguistic characteristics can complement prediction-focused approaches, aiding the development of a more thorough, corpus-based understanding of narrativity.
The data and code for our analysis are available hat:tps://osf.io/vd9pu/.
Constant Tjur’s 2 Observations Akaike Inf. Crit. (1) −0.06∗∗∗ (−0.07, −0.05)
TurningPoint
(2) −0.06∗∗∗ (−0.07, −0.06)
∗p<0.1; ∗∗p<0.05; ∗∗∗p<0.01 (3) −0.06∗∗∗ (−0.07, −0.06)