Storytelling is a universal human practice that serves as a key site of education, collective memory, fostering social belief systems, and furthering human creativity. It can occur in diferent discursive domains for diferent social purposes with difering degrees of intensity. In this project, we develop computational methods for measuring the degree of narrativity in over 335,000 text passages distributed across two- to three-hundred years of history and four separate discursive domains (fiction, non-fiction, science, and poetry). We show how these domains are strongly diferentiated according to their degree of narrative communication and, second, how truth-based discourse has declined considerably in its utilization of narrative communication. These findings suggest that there has been a long-term historical diferentiation between the practices of knowing and telling, which raises important questions with respect to the social acceptance of both science and the arts.
in the text, just as ostensibly non-narrative documents, such as scientific reports, may also
exhibit degrees of narrativity. Herman [
While a wealth of recent work in the field of natural language processing has engaged with
the detection of diferent dimensions of narrativity (such as causal and temporal relations
[
In this paper, we develop computational models to detect “narrativity” as a local,
multidimensional textual quality across four diferent discursive domains over a two- to
threehundred year time-period. Our aim in doing so is to test the relationship between narrative
communication and the process of functional diferentiation among social systems as theorized
by the sociologist Niklas Luhmann [
The question we wish to test here is whether the practice of narrative participates in this
process of functional diferentiation between the social systems Luhmann labels “art” and
“science.” According to Luhmann, art’s function lies in its ability to communicate the sensory
process of observation, to “allow a world to appear within the world” (p. 241), whereas the
function of science is to “structure the field of possible statements [about the world] with the
help of the code true/untrue” (p. 227) [
And yet according to the data and models used here, we can observe a very clear historical
trajectory of the de-narrativization of truth-based discourse. Our findings bring to light
longstanding and growing tensions between what Hayden White first introduced as the relationship
between knowing and telling [
Our findings, preliminary as they are, suggest the need for further research into this disassociation of science and narration and its potential social efects. Is the growing public distrust in science related to the denarrativization of scientific communication? Do eforts of “public science” or science journalism have a positive efect on reversing public distrust and are such efects related to their degree of narrativity? If narration is increasingly seen as belonging to the domain of art, has this diferentiation from science unintentionally contributed to the devaluation of the arts (or their study)? How might the arts instead participate more explicitly in the process of knowledge transfer, i.e. help “recouple” in Luhmann’s terms the practices of knowing and telling?
In order to detect narrativity in our historical collections, we undertake the following steps, which we describe in greater detail in the following sections. First, we construct a data set of 335,245 documents to represent our two primary social systems of art and science, which we subdivide into four domains of fiction, poetry, science, and non-fiction. We then develop a working theory of “narrativity” drawn from existing theoretical literature that informs our manual annotation of the data. Building a team of three trained student annotators, we handannotate 401 passages according to a scalar understanding of narrativity derived through numerous meetings and discussion. This data is then used to train and test our machine learning models, which we describe in Section 3. We present our results in Section 4 and include a discussion of their potential implications as well as limitations (Section 5). Finally, we conclude with a brief discussion of where future work in computational narrative studies might lead.
In order to annotate training data for the presence of narrativity, we rely on the following
theoretical schema developed by Herman [
2. Event sequencing: narrativity depends on temporally ordered events
3. World making: narrativity depends on the fact of disequilibrium such that we can observe
a change in the world
4. Feltness: narrativity captures the experience of events, i.e. “what it is like”
Herman’s categories can be seen as syntheses of previous narratological frameworks,
capturing a good degree of consensus in the field. The emphasis on feltness, for example, is strongly
indebted to the argument by Fludernik [7] that “Experientiality reflects a cognitive schema
of embodiedness that relates to human existence and human concerns” (p. 9). Similarly,
event-sequencing is strongly indebted to the work of theorists like Genette [9], Sternberg [
In general then, Herman’s model is guided by the notion that, “Narrative roots itself in the lived, felt experience of human or human-like agents interacting in an ongoing way with their cohorts and surrounding environment” (our emphasis). Thus for Herman what matters most about narrativity is: a) the centralization of one or more agents; b) the sequencing of events and thus time; and finally, c) the idea of “lived experience in an environment”, i.e. a sense of world building.
Based on this theoretical framework, we hand-annotate 401 passages drawn from the experimental data using the following steps:
First, we assembled a team of three annotators who all have majors in the humanities. These are readers who have high levels of education and exposure to training in textual analysis. Second, over the course of several weeks we engaged in discussions and experiments regarding the concept of “narrativity” with respect to the theoretical framework discussed above as well as diferent kinds of text passages. These discussions culminated in a codebook, which is included in the supplementary material.1 Annotators were then asked to code a given passage across three dimensions of narrativity, which were defined for the annotators as “agency,” “event sequencing,” and “world making.” Note how we translated Herman’s “feltness” into “agency” to better account for the idea of experientiality at the heart of most major narrative theories.
For each passage, readers were asked to respond to the following statements using a five-point Likert scale: • “This passage foregrounds the lived experience of particular agents.” (Agency) • “This passage is organized around sequences of events that occur over time.” (Event sequences) • “This passage creates a world that I can see and feel.” (World making)
2. Somewhat disagree 3. Unsure 4. Somewhat agree 5. Strongly agree
Notice how we do not expressly ask if readers felt that the passage was “narrative” or not. Rather, we ask them to consider their feelings with respect to these three primary narrative dimensions, which we then average into a single “narrativity” score. We found that this increased reader agreement and allowed for more nuanced understandings of narrative communication. For example, it was not uncommon for some types of discourse to emphasize sequential events but lack an emphasis on agency or building a world.
We provide a few sample passages that received low and high average narrativity scores by readers. Note that passages have been truncated from their actual length.
Non-Fiction - Average Reader Score 1.2 The employment of the uninterpretable symbol in the intermediate processes of trigonometry furnishes an illustration of what has been said. Lapprehend that there is no mode of explaining that application which does not covertly assume the very principle in question. But that principle, though not, as I conceive, warranted by formal reasoning based upon other grounds, seems to deserve a place among those axiomatic 1Note that we provide the reader-annotated data, annotator’s codebook, metadata, code for all models and concrete implementation details of custom features from Table 1 in the Supplementary Material. It is available at https://doi.org/10.7910/DVN/DAWVME truths which constitute in some sense the foundation of general knowledge, and which may properly be regarded as expressions of the mind’s own laws and constitution. Fiction - Average Reader Score 1.44 It is too weak for a shield, too transparent for a screen, too thin for a shelter, too light for gravity, and too threadbare for a jest. The wearer would be naught indeed who should misbeseem such a wedding garment. But wherefore does the sheep wear wool? That he in season sheared may be, And the shepherd be warm though his flock be cool.
Science - Average Reader Score 4.55 I assisted at the opening of her Body, and having found in the matrix a little round mass of the bigness of a great black Cherry, I took the husband aside, and asked him, Num a tempore fluxus menstruorum uxorem cognevisset? And having received for answer, that he had, I prayed him to let me carry home with me this little ball, which I had found in her womb. I was no sooner come home but I opened it, and found, that nature had wrought with so much activity in so small a time...
Fiction - Average Reader Score 4.55 Whereupon a sudden outcry arose within the house, and a head popped angrily out of the aperture so suddenly created. But as instantly it returned within. For Jorian tossed the lattice to the ground by the door and thrust his spear-head into the cravat of red which the man had about his throat, shouting to him all the while in the name of the Prince, of the Duke, of the Emperor, of the Archbishop, of all potentates, lay and secular, to come down and open the gates.
Because our annotations use a multi-point scale, we assess inter-rater reliability (IRR) using
the average deviation index as discussed in Burke, Finkelstein, and Dusig [
Our experimental data consists of five separate collections that are designed to capture the
two social systems of “art” and “science,” which we represent as the four discursive domains
of fiction, non-fiction, poetry, and science. Doing so allows us to see aggregate behavior across
the two systems as well as potential internal diferences based on discourse type. Our data
consists of:
• Fiction & Non-Fiction. This data is derived from the Hathi Trust Digital Library and
is drawn from Piper and Bagga [
of the number of passages per year is indicated in Figure 2. Years represent year of
publication, not year of composition or first printing. Our data reflects reading material
available in a given year as archived by academic libraries.
• Poetry. This data is drawn from the Literature Online Poetry database. It consists
of 73,077 poems by 857 authors who wrote in English and who were alive during the
nineteenth and twentieth centuries. To estimate year of publication, we use the author’s
birth-date plus 35 years to capture an estimated career midpoint. Because of the
relatively small number of poets in our dataset, we are not able to capture a consistent
number of poems per year.
• Science. To represent the domain of scientific writing, we use two diferent data sets. The
ifrst is drawn from the Royal Society Corpus (RSC 4.0) based on the first two centuries of
the Philosophical Transactions of the Royal Society of London from its beginning in 1665
to 1869 [
Because our interest is in local narrativity, i.e. the extent to which a span of tokens expresses
narrative communication, we represent our documents as randomly selected sequences of 5
sentences in length. This number has been indicated in prior work as a reasonable frame in
which completed narratives can transpire [
For the purposes of our project, we use the predicted probability of a passage’s “narrativity” as an indicator of the degree of narrative communication present in that passage. In order to build a model to predict a passage’s narrativity, we train and validate our models using our reader annotated data. We experiment with three widely-used algorithms (Logistic Regression, Support Vector Machines, and Random Forests) and multiple combinations of diferent features to identify the best-performing model. We present our feature components in Table 12 and present the performance of each model using diferent feature combinations in Figure 3. As can be seen in Figure 3, Random Forest performs the best out of the three learning algorithms. Table 2 presents a brief overview of the top-5 performing models using Random Forest.
We assess model performance according to Pearson’s correlation coefficient rather than the more traditional F1 score (although we also report traditional classification metrics in Table 2). Because our metric of narrativity is predicted probability and not a binary classification, the question we want to address is how well our models correlate with the scalar nature of reader judgments.
To construct our experimental feature spaces, we aggregate our features into three general categories: lexical features (ngrams), syntactical features (part-of-speech and dependency relationships), and higher-level custom features designed to capture specific narratological theories, including time, concreteness, animate entities and perceptuality. For a full discussion of the custom features, see the supplementary material.
As we can see in Figure 3, all three classifiers behave similarly and achieve their maximum
performance on a variety of feature combinations. Interestingly, unigrams tend to perform
2Note that the results shown in Figure 3 correspond to a maximum of 100 features per category. This
is why # Features for word-bigrams, for example, is 100 although the complete feature space involved 25,434
word-bigrams. Experiments with other values of max-features yielded similar results.
better than the limited sets of bi- or trigrams for lexemes, pos, and dependency tags. The
best performing model consists of part-of-speech unigrams, % dialog and custom-built features
that aim to capture “event sequences”, “world building”, and “agency” for which we use the
categories tense, mood, and voice. There appears to be a strong grammatical signature to
narrativity that marginally grows in strength when we add in features that capture the notion
of “environment” emphasized in Herman’s theory above [
As we can see in Figure 4, the correlation between reader judgments and predicted probability is approximately linear and indicates a reasonable level of agreement (r = 0.742). We observe higher levels of variability in the middle range of annotations between 2.0 and 3.0, which is to be expected. As readers’ judgments become less certain, so too do we observe more variability in our models’ predictions. Future work will want to explore the extent to which more annotations lead to higher levels of correlation or whether we achieve some kind of maximum level of correlation between computational models and human judgments in this area.
Applying our best performing model (Random Forest with POS-unigrams + Pct-Quoted +
tense, mood and voice features) on the experimental data described above, we generate the
average yearly predicted probability of narrativity across all four domains as shown in Figure
5. According to our models the four domains behave in distinctive fashion with respect to
narrativity, providing support for the idea that narrativity may be another facet underlying
Luhmann’s thesis about functional diferentiation [
In terms of our two “literary” domains, we see little change over time, suggesting relative stability of these domains’ relationship to narrativity. While this does not run counter to expectations with respect to fiction, theorists of poetry might be surprised to see such continuity given the popularity of long narrative poems in the nineteenth century (for example in the work of Walter Scott or Longfellow to name two prominent examples). Future work will want to explore in greater depth whether there is a meaningful break with respect to poetic narrativity for authors born after the late nineteenth-century that then potentially reverses course for younger poets born closer to the end of the twentieth-century as indicated in Figure 5. More domain-specific training data would be needed along with more careful sampling techniques to gain confidence about any such shifts given how slight they are with respect to our models.
Taken altogether, our models suggest that narrativity is strongly socially diferentiated across diferent discursive domains and that at least with respect to fact-based discourses this diferentiation is increasing strongly over time as both non-fiction writing and specifically scientific writing exhibit declines in their reliance on narrative communication. We take up the implications of these findings in our discussion.
Our study raises a number of questions for future research. Representing the social systems of “art” and “science” is a challenging task. In our work, we have tried to capture at least two larger subdomains of writing within these systems to better understand the kind of internal diferentiation may be at work. While future work will want to experiment with diferent samples dependent on diferent archival resources, we do observe interesting diferences with respect to the narrative behavior of our subdomains. For example, we see how the narrativity of poetry is not only considerably lower than prose fiction, it consistently hovers around the iffty-percent mark, suggesting that one of the potential social functions of poetry as a genre could be its ability to communicate narrative ambiguity. Such ambiguity is corroborated by the higher average deviation among our annotators with respect to the poetry training data. This suggests an interesting potential theory one could pursue for the future study of poetry in a larger social context along with the potential increase in narrativity that we observe for poets born after the 1960s.
Because our models have been trained to understand cross-domain behavior of narrativity, our work cannot however speak to within-domain distinctions with respect to narrativity. For example, an interesting question to be pursued in the future is the intensity and extent of narrativity at the document level within our diferent discourses. When do we see novels for instance engage in more explicitly “narrative” communication, are there reliable patterns of the rise and fall of narrativity, or finally what kinds of novels (genres) indicate greater degrees of narrativity overall? Similarly, for science documents while we observe an overall process of denarrativization of scientific documents, are there still portions of articles that engage in more narrative-like behavior or portions of the field (i.e. disciplines) that engage in more narrative communication than others? These questions would help provide insights into the relationship between categories like genre, discipline and narration.
Further reflection could also be given to our framework of “truth-based discourse,” which is not exactly synonymous with “science,” which is one of the reasons we also model “nonifctional” writing as well. Scientific writing is one kind of communication that makes truth claims, but there are numerous others that belong to diferent institutional frameworks. We note that in a random sample of two-hundred passages drawn from our non-fiction experimental data that the number of “scientific” texts moves from 6 in the nineteenth century to 10 in the twentieth. While this represents a large increase, it is still a very small fraction of all writing in our non-fiction sample suggesting that the decline in narrativity in non-fiction is not strongly related to a rise of scientific writing in Hathi Trust. In other words, writing classified as nonifction is exhibiting similar trends to science but is being produced in diferent institutional contexts. Future work could explore more deeply why we see this denarrativization of nonifction along with science writing.
At the level of data annotation, while we demonstrate solid agreement between readers and reasonable model correlation with reader judgments, we are not able to annotate large amounts of data to better calibrate our models. Hand-annotation is a slow and expensive process and future work will want to explore mechanisms that allow for scaling annotation while maintaining quality. We assume model accuracy will increase with increased amounts of annotated data, which may or may not have a bearing on the historical trends we observe here. The observed declines in our science and non-fiction corpora are so steep and consistent that we would be surprised if future work indicated significant changes in this regard.
In terms of our theoretical framework, it is important to underscore that our approximation of narrativity is just that. While we do not observe significant shifts in the distribution of narrativity according to feature-space selection, our models are still guided by a particular theoretical framework with respect to narrativity. Future work will want to explore alternative theories and feature representations of narrativity to see if the historical trends we are observing continue to emerge.
Finally, future work will want to explore the extent to which our findings are or are not culturally specific, i.e. the extent to which they hold in other language communities and the extent to which those correlations are driven by social factors such as national wealth or education levels. Just how universal is this process of functional diferentiation and denarrativization with respect to truth-based discourses that we have observed here? Is this indeed a marker of “modernization”?
Our work has demonstrated that narrative communication is a detectable linguistic quality of texts from the perspective of human readers and machine learning. Even with a small set of training data we can achieve reasonable levels of predictive accuracy and correlation with trained reader judgments across very diferent kinds of texts over relatively long historical time spans. We also show that with sufficient training readers can agree quite well regarding the intensity of a passage’s narrativity.
Being able to identify the degree of narrativity in large-scale historical document collections allows us to gain a better understanding of the distribution of narrative communication across documents that serve diferent social functions. Modeling narrativity at the computational level suggests that narrative is a form of communication that participates in Luhmann’s theory of functional diferentiation, at least with respect to the social systems of art and science. While culturally and historically universal – narration is present in all linguistic communities and recorded time periods – narration is far from being socially universal. Indeed, it appears that in modern, highly diferentiated societies narration is increasingly aligned with the particular social system of the arts as truth-based discourse becomes less and less narrativized over time. How this may impact urgent large-scale questions such as trust in science or particular collective responses to social problems such as climate change or health pandemics remains an open, yet important question for future research.