We approach the modeling of event structure of literary texts with narratological event concepts. A manually annotated corpus of 4 prose texts with event categories allows us to learn to automatically classify events using a transformer model, relying on a rule-based system in conjunction with a pretrained parser to identify events. For the evaluation of both manual and automated annotation, we use narrativity graphs, which capture the change in narrativity over the course of the text. In an exploratory analysis, we apply the event classifier in conjunction with graph-based narrativity metrics to a large literary corpus. We find that text length does neither influence the length of eventful passages nor the number of eventful passages in the beginnings of texts.
Change of state: “(1) a temporal structure with at least two states, the initial situation and the final situation; and (2) the equivalence of the initial and final situations, that is, the presence of a similarity and a contrast between the states, or, more precisely, the identity and diference of the properties of those states”
patient but not an animated agent”
Change of physical states: “deliberate actions and accidental happenings”
dictable, persistent, irreversible, iterative) change of state unprenon
a voluntary human or human-like agent”
Most concepts of narrative theory are based on the distinction between story (histoire) and
textual representation (discourse) [
Table 1 shows common conceptions of event in narrative theory. Most of these are based on a notion of change of state and introduce additional parameters to diferentiate event types. For example, Schmid [24] introduces five event properties to evaluate events with regard to their narrativity. Following this proposal, a change of state’s narrativity rises with its relevance for the narrated story, with its unpredictability in regard to the past course of the story, with its persistence for the following story, with its irreversibility or with its singularity. Other concepts take into account whether a change of state is the result of an intentional act by an anthropomorphic agent. With Prince [21] and his concepts of stative and active events, there is only one theorist who does not define events as a change of states. This grounds on a textual definition of events: Prince has suggested, “to call event in a story any part of that story which can be expressed by a sentence, where sentence is taken to be the transform of at least one, but less than two, discrete elementary strings” [21, p. 17].
In natural language processing (NLP), events are typically understood as anything that is
“happening during a particular interval of time” [
In the literature domain, Sims et al. [25] diferentiate between realis and irrealis events, thereby distinguishing between events actually happening in the narrated world and those that, for example, only happen in the thoughts of characters. They make use of event detection to establish a correlation between the frequency of realis events in a text and its reception. Recognizing the limitations of fixed event categories in the more open domain of literature, they adopt an open approach to event categories.
However, while the distinction of events and non-events is typically made, the impact of event types, or categories, on the narrative is not explicitly modeled. We limit the event trigger detection to verbs rather than including nouns and adjectives.
While most definitions of events in narrative theory rely on change of states (see Table 1), we adopted a more fine grained approach. Our classification of events is based on core features of events in narrative theory (i.e., being a state, a process in time and change of state). This also allows us to include processes of speaking, thinking and movement.1
Additionally, we adopted Prince’s [20] idea that events can be presented by a single sentence by analyzing verbal phrases.
Annotation Spans. We annotate each verbal phrase. In cases where a text segment cannot be assigned to a verbal phrase, it is annotated as <non_event> (see below). To avoid overlapping annotations, an annotation span is defined by an inflected verb and its direct subordinated dependencies. Subordinated verbal phrases do not lead to overlapping annotations: 1. [1As Gregor Samsa one morning from uneasy dreams awoke]1, [2found he himself in his bed into a monstrous insect-like creature transformed]2.2 Event Types. Our annotation scheme provides four event types: changes of state, process events, stative events and non-events. A given verbal phrase’s event type is defined by the act inside the narrated world that the full verb signifies.
1These do not imply a change of state as defined in narratological event concepts, however they are part of
Chatman’s [
2This and all following examples are taken from Franz Kafka’s narration Die Verwandlung (Translations
taken from Kafka [
The event type <change_of_state> covers physical and mental state changes of animate and inanimate entities. The change of state needs to be expressed in a single verbal phrase. An example of a (mental) change of state is the process of waking up, described by the first verbal phrase in Example 1.
The event type <process_event> covers actions and happenings that do not lead to a change of state, such as processes of moving, talking, thinking and feeling. For example, the second verbal phrase in Example 1 is a process of perception (“found […] himself […] transformed”) and, therefore, has to be annotated as a process event.3
The event type <stative_event> covers all verbal phrases that refer to the state of an animate or inanimate entity. This also includes physical and mental states. Unlike changes of state and processes, stative events do never refer to temporal processes. As Example 2 shows, stative events are often descriptions of places, persons or objects.
2. [1There stood a bowl filled with sweet milk ]1 [2in which swam small bits of white bread.]2 The event type <non_event> finally covers verbal phrases that do not refer to a fact in the narrated world. The most common variants of non-events are questions or modalized and generic statements (Example 3). Non-events are a necessary addition allowing us to annotate all text passages not just those referring to events, thereby simplifying the annotation schema.
3. [1A man must have his sleep.]1
So far, we have double annotated four German prose texts with 151,000 tokens in total: Besides Kafka’s Metamorphosis, these are Theodor Fontane’s Effi Briest (partially; 14 of 36 chapters), Marie von Ebner-Eschenbach’s Krambambuli and Heinrich von Kleist’s Das Erdbeben in Chili.4 After some training, the annotators (students in literary studies) reached an agreement of 0.73 Krippendorf’s α.5 Figure 1 shows a confusion matrix for two of the texts. The agreement difers both depending on the categories / event types and the annotated text. We suspect that the agreement is influenced by the linguistic style of the respective authors.
Our manual annotations show that the first three event types are the most common: nonevents, stative events and process events are the most common types as Table 2 shows. For the three shorter texts, Die Verwandlung, Das Erdbeben in Chili and Krambambuli, even the 3This example shows why it is important to use the full verb as an annotation criterion. The transformation is only described as an object of perception and not as an objective fact in the narrated world.
4These annotations are quite time-consuming. For the four sections or the 14 chapters of Effi Briest , for example, each of our annotators needed over 60 working hours.
5The annotation’s span overlap average is 97%. (a) Erdbeben in Chili 1.0 proportion between the first three types is similar. Only in Theodor Fontane’s novel Effi Briest non-events are the most common event type. This is due to the high proportion (here: over 60% of the text) of character speech, which generally includes mostly non-events.
Our event types instantiate diferent degrees of eventfulness, ranging from no (non-event) over little (stative events) to more narrativity (process events and changes of states). Therefore, we could create a simple metric: Non-events are given a narrative value of 0, stative events a value of 2, process events a value of 5 and change of states a value of 7.6 Based on this metric, we generated narrativity graphs for each manually annotated text.
We did this, as shown in Figure 2, by using cosine weighted smoothing (smoothing window = 50 events). We assume that the narrativity of a section of text results from the narrativity of longer sequences of sentences. This also allows us to evaluate the narrativity graphs, as we will show later.
The graph in Figure 2 shows that we detect highly eventful scenes in the text. Whenever the narrativity value drops sharply in the course of the text, this is due to introspective-reflexive passages.
Additionally, we compare diferent annotators using narrativity graphs. Figure 3 shows the four parts of Fontane’s Effi Briest . Even though there are diferences between the two annotators, the graphs are structurally very similar and identify the same passages as peaks 6We tested diferent values without changing the ordering of event types with respect to their narrativity score. Structurally, this seems to have limited impact on the narrativity graphs, as long as said ordering stays intact but further evaluation is required. 1.5 1 2 60000
Text Course (Characters) 1. After the metamorphosis, Gregor exposes himself for the first time to his family and colleague. 2. Gregor leaves his room, his mother loses consciousness, the colleague flees and his father forces him back into his room. 3. Gregor’s father throws apples at him. Gregor gets seriously wounded. Escalation of the father-son conflict. 4. Three tenants move into the family’s flat. 5. Gregor shows himself to the tenants, who then flee.
6. Gregor dies. of narrativity.
Against this background, we would consider the automation of event annotation successful if it also allows as to create structurally similar graphs, without necessarily classifying each individual event correctly.
Automated event annotation is approached using a two-step process. In the first step, we extract verbal phrases, annotating them with regard to their event type in a second step.
Following the annotation guidelines, we automatically annotate events based on full verbs
and their verbal phrases. We rely on a pre-trained tagger and parser [
Additional rules aim to replicate the annotation guideline’s approach to the inclusion of punctuation. For the span evaluation, we exclude all special characters, thereby disregarding errors only related to punctuation or white space. In this scenario, our rule-based tree-walking system yields an F1-Score of 0.71 on Die Verwandlung on a per-span basis (meaning each of potentially multiple spans in an annotation is handled individually).
Automated classification of event types operates on, sometimes non-continuous, spans of text
produced by the first step. Our transformer-based architecture [
570000 575000 Text Course: Characters 580000 585000 590000 of the same event are automatically marked using special tags in the transformer’s input vocabulary7, thereby allowing the model to focus on the verb of concern if multiple verbs are present in the input sequence. The following example illustrates our input formatting for an event with two text spans.
4. “<EVENT>Vielmehr trieb er</EVENT>, als gäbe es kein Hindernis, <EVENT>Gregor jetzt unter besonderem Lärm vorwärts</EVENT>”8
More specifically we rely on an ELECTRA model [
Figure 4a illustrates good performance in recognizing all but one of the classes. Our model performs poorly in distinguishing the process and change-of-state classes. It is important to note that in the development set, the change-of-state class is only found in 4 of 732 events. This can, in part, be attributed to the small number of training examples for the change-of-state 7The special tags are placed based on the output of our rule-based processing of the parse tree or, in the case of the classification evaluation, based on spans annotated by human annotators.
8“On the contrary, as if there were no obstacle and with a peculiar noise, he now drove Gregor forwards.” 9https://huggingface.co/german-nlp-group/electra-base-german-uncased 10For our initial experiments, we use a 80-10-10 train/development/test split while a 90/10 train/development split with a held-out document for testing is employed in the out-of-distribution setup. 1.0 0.8
itttvvaeeEnS trsscvPoeeEn Predicted Labels (a) Early Stopping Development Set (b) Out Of Distribution Data class. As a result of the rarity of the change of state class, and given that its members are often attributed to the class with the second-highest narrativity score (limiting the impact on the smoothed narrativity graph), we do not see this result as detrimental to our eforts. Depending on the specific goal, further improvements could be made. Specifically, loss weighting by each class’s narrativity score might improve the model with regards to their application to narrativity graphs. If a more balanced performance for each individual class was desired, early stopping on macro average F1-scores rather than weighted averages would also be an option.
While we have established sufficient performance in classifying events from the training distribution, the application to unseen literary works comes with various challenges, including stylistic diferences and a change in the distribution of event types.
To validate the model’s performance on such data, we hold out the text Die Verwandlung during training, performing early stopping on a subset of events from the remaining documents. Accordingly, the resulting classification performance is broadly applicable to many documents.
While Figure 4b shows no generalization error to out of distribution data when compared to Figure 4a, a limitation of this evaluation remains the impact of propagated errors from the initial span detection step.
To assess the efectiveness of our classification pipeline with regard to narrative graphs, we compare the annotations created by the out-of-distribution model, with those produced by the original annotators. The central characteristics of the graph based on annotations are also to be found in our automatically created graph (see Figure 5). In comparison to the graphs in 5 it may appear as though the model outperforms human annotators. While this can in 4.0
60000 Text Course (Characters) 80000 part be attributed to our system’s good performance, we hypothesize that Die Verwandlung is inherently easier to annotate, a theory supported by the reports of our annotators. As discussed previously, we reach an F1-Score of 0.71 in selecting the correct spans; our classification reaches an F1-Score of 0.78 but due to the heavy smoothing employed in the graph, individual errors may not have a large impact on our results. The comparison of our model’s output with manually annotated data gives us confidence in the general applicability of the model to unseen data, justifying its application in corpus analysis.
We used the trained model to annotate the 2528 texts in the d-Prose corpus [
For illustration, Figure 6 shows the peak properties for the text beginnings of Krambambouli and Das Erdbeben in Chili.
With the first evaluation of the narrativity graphs in Figure 7, we test the influence of text
length on the width of the narrative peaks. We are interested in seeing whether longer texts are
11To compare the narrativity graphs we first tested dynamic time warping (DTW), a technique originally
proposed for speech recognition [
12We used 4 diferent relative thresholds. Whenever the narrativity graph exceeds 60, 70, 80, or 90 % of the text’s maximum narrativity value, it is considered to be the beginning of a new peak.
13The width is the count of events included in a single peak. 14The index of the highest value in a peak defines its position. Narrativity Graph Peak Width Peak Height Peak Position 100 200 300
Text Course: Events 400 500 100 200 300
Text Course: Events 400 500 0 7.65 0.0 100 200 Average Peak Width 50 100 150 Average Peak Width
50 75 100 Average Peak Width
50 100 Average Peak Width associated with larger peak widths, which from a narratological perspective would be expected. In longer texts, individual passages are more detailed, accordingly one would expect them to contain longer narrative passages.
However, the average width of the peaks is not influenced by the length of the texts (correlation for all thresholds r < 0.2). This indicates that the alternation of strongly and weakly eventful passages follows similar principles in long and short forms. Only at a threshold of 4 there is a stronger correlation (r = 0.19) between the number of events in a text and the average peak width.
To check which text parts are particularly narrative, we analyze the position of peaks for each of the four thresholds in Figure 8. For that purpose, we divided the corpus into four subcorpora based on each text’s event count: • 339 texts with more than 500 and less than 1,000 events.15 • 634 texts with more than 1,000 and less than 5,000 events. • 329 texts with more than 5,000 and less than 10,000 events.
15Due to smoothing in the generation of the narrativity graphs, we cannot examine shorter text with our method.
tn100 u o C k a e P 0 t n ouC250 k a e P 0 tnu500 o C k a e P 0 t n ouC500 k a e P 0 50 0 250 500 0 0 0 500 50 0 200 250 500 0 0 0 0.0
0.5 Normalized Text Position
• 258 texts with more than 10,000 and less than 100,000 events.
This work was supported by the DFG through the project “Evaluating Events in Narrative Theory (EvENT)” (grants BI 1544/11-1 and GI 1105/3-1) as part of the priority program “Computational Literary Studies (CLS)” (SPP 2207). We thank Gina Maria Sachse and Michael Weiland for their annotation work. 16In this context, it is noteworthy that the narrativity of a text’s beginning is highly correlated with its overall average narrativity. (For each the four subcorpora r > 0.64). . Paperback