(1) I was in my room of a paying guest flat, 5th floor, and was about to go for my bath and then I suddenly noticed from my window an object glowing/flashing over a jungle area more than 1 km away from my apartment. (Report 76519)

(2) As we drove north, 2 out of four of us saw a big bright blue ball of fire that looked as if it got brighter the closer it got to the ground. (Report 65963)

(3) I was getting in my car, when all four of us -my grandson, my grandson's tutor, my granddaughter, and myself -noticed a low, slow-moving, sideways teardrop-shaped object moving from north to south through the San Gabriel Mountains. (Report 4061)

When people tell the story of something extraordinary which has happened to them, they use a particular kind of language. This is especially true for recounting moments when the extraordinary intrudes the ordinary. The excerpts above are sentences stemming from texts about alleged UFO sightings which were collected online. Within these narratives, the first appearance of something out of the ordinary marks an important turning point. When looking at these sentences, a pattern emerges: While they might differ from other parts of the text contentwise, they also tend to stand out structurally. More precisely, they typically include an adverbial which temporally grounds the event in relation to other parts of the narrative.

Concepts related to what we just introduced as turning points are, among others, Labov's most reportable event [6], the disruptive event in the narrative theory of Todorov [19] or Field's plot point [3]. Hühn [5] distinguishes between type-I-events and type-II-events: Whereas every change of state in a story marks a type-I-event, a type-II-event is characterized by further differentiating traits, such as its unpredictability and its deviation from the norm. We see a turning point as a type-II-event which has a particular function and prototypical position in narratives. Hühn argues that type-II-events can only be identified hermeneutically [5]. However, he also notes following Schmid [17,16] that there are criteria which hint at the presence of a type-II-event in a sentence such as, e.g., the non-iterativity of an action. In our context, this should entail a higher frequency of temporal function words such as the highlighted ones in (1) -( 3) in sentences recounting a turning point compared to other sentences, as these words typically hint at a singular event. In this short paper, we aim at testing whether there is a systematic association between sentences containing a turning point and the use of certain context-independent markers of temporality.

Following observations such as ( 1) -( 3), we opted to focus on then_ADV, as_SCONJ, and when_ADV as function words which frequently introduce temporal adverbials and seem to characterize turning point sentences. We test our hypothesis by specifying a model that predicts whether a sentence is a turning point or not, assessing whether the selected words are associated with a higher probability. By using a limited number of linguistic factors as predictors in our model, we aim to contribute to a better understanding of turning points as simpler models help to keep the impact of individual variables more transparent.

The computational modeling of narratives -both their constituent elements (e.g., characters or events) and their overall structure -is a vibrant field of research [1,2,14] and can be seen as part of a project that strives to develop what Piper calls a Computational Narrative Understanding [10]. Literary event detection is a key element in this enterprise [18,10,11]. NLP research on how to best predict events in a narrative has yielded models that have been tested on various datasets with recent approaches reporting good performance [20,7,8]. Not all of these studies try and measure the same theoretical construct, since event and related concepts are not defined consistently [4]. Also, events can be measured on different levels, e.g., sentence vs. word. Nevertheless, all approaches have in common that they aim to extract those parts of a narrative that are distinguished from other parts in the way they contribute to the development of the story.

While these studies broadly investigate the same phenomenon, they differ from our approach in that they are predominantly concerned with predicting events while we aim at identifying certain characteristics of those events we consider turning points. While, e.g., approaches like the one by Ouyang/McKeown [9] make extensive use of prior findings from linguistics and literary studies when selecting features for their models, they are still aimed at good predictive performance. This is typically achieved by including a myriad of different factors which, on the downside, hampers disentangling variables that contribute to what constitutes a turning point. In contrast, to be able to better interpret results, we aim to keep our model as simple as possible when estimating turning point probabilities.

The data stem from a larger corpus of approximately 110,000 reports of UFO sightings submitted via the online platform UFO Stalker (https://ufostalker.com) and scraped by one of the authors. 1 Texts are mostly written in English, presumably by people from the U.S., even though these metadata cannot be verified. The reports' narrative shape is typically as follows: In a short exposition or staging phase, authors describe the -usually mundane -situation they say they were in and quite often who they were with at the time (cf. ( 4)). Then something happens: Most often, authors report that they suddenly see a strange light moving in the sky and such the 'actual' reporting of the sighting unfolds. This reporting is mostly an account of the author's cognitive processes. Reports typically end without them reaching a definitive conclusion with regard to what it was that they saw. This puzzlement can be seen as the prototypical resolution of these narratives. We sampled 496 texts from the larger corpus of reports. These texts were preprocessed using Stanza (Version 1.8.1) [13] for tokenization, sentence segmentation and part-of-speech tagging. Two of the authors annotated which of each report's sentences marks the turning point which we operationalized as the one sentence where it becomes clear that the narrative is about a UFO sighting, i.e., we only annotated one turning point per text. Inter-annotator agreement was good (ICC(2,1) = 0.808, 95%-CI [0.766, 0.843]; ICC(3,1) = 0.81, 95%-CI [0.769, 0.845]). Disagreement was resolved via discussion. Reports that consisted of fewer than three sentences were discarded, in line with the data preparation done by Ouyang/McKeown [9] following Prince's definition of a minimal narrative [12]. Also excluded were reports that were written in languages other than English, described something other than a UFO sighting, were a mere description of photos or videos that were provided along with the report, did not feature any narrativity or did not include a discernible turning point. Finally, 352 reports consisting of 5,346 sentences were included in the analysis. Texts contained up to 81 sentences (Median = 12, IQR = 10).

To test the hypothesis laid out above, we fit binary logistic regression models with the probability of a sentence being the turning point as the outcome variable. As predictor variables we used dummy variables coding for whether the words when_ADV, then_ADV or as_SCONJ occurred in a given sentence. Further, we opted to include two more structural variables. First, we added the sentence's relative position within the text (the sentence's index divided by the text's length) as a percentage. Since we assumed a certain narrative structure, we knew that position within the narrative would play a role: We observed beforehand that the turning point is usually located toward the beginning of the story. Second, we included logged sentence length as a predictor. Sap et al. found that what they call major events are usually expressed in longer sentences [15] and a similar pattern has been observed by Ouyang/McKeown [9]. Importantly, the context of sentences is not included in any way -no information on what was written in the preceding or following sentence was used in the model, i.e., sentences were assumed independent. Thus, we do not measure any kind of change from one sentence to the next (like, e.g., Ouyang/McKeown do [9]), but rather compare 'global' differences between turning points and non-turning points. Note that even though sentences are naturally clustered at the text level, a multilevel model is not warranted in our case since we decided to only select one sentence per text as the turning point. Thus, the turning point probability does not vary between texts. Looking at descriptive evidence, the three selected words do exhibit different occurrence distributions depending on whether sentences are turning points or not (Fig. 1). The percentage of sentences that include when_ADV is four times higher for turning points than for non-turning points. The same tendency, though less pronounced can be observed for the word as_SCONJ, whereas then_ADV occurs in a similarly sized share in both subsets of the corpus. Turning point sentences have a median relative position within the text of 16.7 (IQR = 19.4), so they are usually present in the earlier parts of the narrative (Fig. 2). Turning point sentences are also longer than non-turning point sentences in our data (Median TP = 25 vs. Median non-TP = 17; Fig. 3).

We fit three separate models. In a first step, regressing the probability of a sentence being a turning point on a sentence's relative position within the text, we estimate a negative association. This model described a small amount of variance (Tjur's 𝑅 2 = 0.113). In a second step, we added the logged sentence length (Tjur's 𝑅 2 = 0.183). Fig. 4 plots the predicted probabilities of a sentence being a turning point against its relative position within the text (a percentage value close to 0 for the very beginning and 100 for the end of a text) for different sentence lengths.

As can be seen, the model assigns very low probability to sentences after half of the narrative has passed whereas sentences that lie in the first quarter of the text are assigned probabilities between around 0.15 and 0.04 for shorter sentences and between 0.54 and 0.20 for very long sentences.

Adding our main variables of interest, namely the occurrence of temporal markers resulted in improved model fit (Tjur's 𝑅 2 = 0.213; for full model comparison, see Data Availability). The estimated coefÏcients for as_SCONJ and then_ADV were not statistically significant, which is in accordance with the descriptive patterns presented above. The word when_ADV, however, was associated with an increased probability of a sentence being a turning point (Fig. 5). Again, different sentence lengths predict different probabilities for a sentence being the turning point with longer sentences being associated with higher probabilities (Fig. 6).

It is important to note that adding content words which we know a priori to be discriminative of turning points for our specific genre of text would also result in better model fit -e.g., adding a binary variable that captures whether the word sky appears in a given sentence (which is presumably typical of turning points in our texts since that is the locus of the extraordinary event) improves model fit (Tjur's 𝑅 2 = 0.242). While it is clear that including more or even all word occurrences into the model will result in better model fit or predictive power, respectively, it was not our aim to design a model that discriminates turning point sentences and non-turning point sentences perfectly -i.e., solve a classification problem -but rather test the theoretical question laid out above in a very specific exemplary genre of texts.

Our investigation of turning points in UFO sighting narratives was driven by a hypothesis on the role of certain content-independent characteristics of turning points and had a relatively narrow scope: Not only does our corpus consist of a very particular and, possibly, idiosyncratic genre of texts. Also, our study only used a small hand-annotated sample and focused on few variables that were situated at different levels: Position within texts and sentence length did already account for some variation in the probability of sentences being turning points. Regarding the role of temporal function words, we found that while when_ADV is predictive of turning points, then_ADV and as_SCONJ are not. Thus, we were not able to identify a whole group or class of words that are used to mark turning points, but we did corroborate that the use of when_ADV is predictive of a turning point. This finding supports our general hypothesis that turning points are characterized not only by their content, but also by structural properties such as temporal adverbials. Whether this also holds true for other types of narratives remains subject to further investigation. Using state-of-the-art NLP methodology, there may be huge advances in the prediction of event types in narrative texts over the next few years. Another question, however, is how well these NLP models will serve us to understand what makes a turning point a turning point (or an event an event, for that matter). On a theoretical level, one can think about approaches like ours as modeling the reader, but also as modeling the author: What hints enable readers to place the content of a given sentence within the greater narrative? What hints does the author deem viable to trigger said interpretation? Do these cues vary between different genres that feature different narrative structures or schemas? These and many more questions should be addressed by future research from the vantage point of different disciplines -such as literary studies, linguistics, and psychology. This will help us gain a quantitatively informed understanding of (literary) narratives. We hope to have exemplified with this study how focusing on individual linguistic characteristics can complement prediction-focused approaches, aiding the development of a more thorough, corpus-based understanding of narrativity.