With the ever-increasing amount of digital content, users face the challenge of coping with enormous quantities of information. This challenge is especially true for digital content curators, i. e., knowledge workers such as, among others, journalists [1], television producers [2], designers [3], librarians [4], or academics [5]. These and other professional profiles have in common that they monitor and process existing or incoming content to produce new content. In several projects [6,5], we have been developing technical approaches to support knowledge workers in their day-to-day jobs, curating large amounts of mostly textual content more efficiently and effectively. One of our focus areas is the generation of storylines. This includes the (semi)automatic creation of new content as well as helpful presentation and visualisation techniques. We call the approach Semantic Storytelling [7,8,1,9,2,10,11].
Recently, storytelling has mostly been interpreted as a language generation task [see, e. g., 12,13,14,15], where the goal is to generate texts. We interpret the concept differently by concentrating on the extraction and presentation of stories and their parts, contained in content streams, e. g., document collections or social media feeds. We see storylines as sets of building blocks, which, depending on their combination (temporal, geographical, causal etc.), can be assembled into a story in various ways. Our goal is the recognition of atomic pieces of information (e. g., facts, propositions, events) in document collections and the identification of semantic relations between these pieces. Applications can be conceptualised, e. g., as information systems (for the retrieval of existing content) or recommender systems (for creating new content).
We define Semantic Storytelling as the (semi-)automatic generation of storylines based on information extracted from documents or social media streams which are processed, classified, annotated and visualised, typically in an interactive way. This section describes the existing components and utilized previous work, followed by an architecture description conceptualized to meet the requirements of our Semantic Storytelling approach.
Three groups of text analytics components are the foundation of our architecture [see, e. g., 7, 16, 8, 2]: (1) services that analyse documents or document collections to provide documentlevel metadata, (2) services that extract, annotate and enrich specific parts of the incoming content, and (3) services that transform the content, e. g., via summarization or translation.
The tools are in different stages of technical maturity. They are orchestrated using a workflow manager [17]. Named entity recognition and linking as well as time expression analysis are performed to identify named entities of various types and classes. The integration of time expression analysis allows reasoning over temporal expressions and anchoring entities and events to a timeline. We use topic detection to assign abstract topics to individual sentences, paragraphs, chapters, and documents. For the annotations, we use NLP Interchange Format [NIF,18], which allows the exploitation of the Linked Data paradigm and Linked Open Data resources. We distinguish between different classes or genres of documents, i. e., we experiment with different approaches for identifying document structures [10] and, on top of this, document genres [19]. An ontology to represent a heterogeneous set of document characteristics, tying together the different parts of annotations mentioned above, is currently under development. We implemented a number of experimental prototypes and user interfaces [7,9,8,2]. On top of the semantic analysis of documents, we map the extracted information, whenever possible, to Linked Open Data and visualise the result [20,3]. By providing feedback to the output of certain semantic services, content curators have control over the workflow. The storytelling UIs involve the dynamic and interactive recomposition and visualisation of extracted information. This involves arranging content elements (documents, paragraphs, sentences, events) on a dynamic timeline or as a graph. Figure 1 shows an example that visualises a story as a graph in which individual story units (content pieces) are represented as nodes. Different content types such as topics and text segments are displayed in different colors. Edges represent specific relations.
At the core of the Semantic Storytelling approach are three processing steps. Figure 2 and the following paragraphs illustrate these steps in more detail.
We start with a topic 𝑇, instantiated through a text segment, e. g., a named entity, headline or document. To identify content pieces relevant for 𝑇, we process an incoming content stream and decide for each piece whether it is relevant for 𝑇. Relevance can be computed in various ways, e. g., text similarity measures, or we can compute the overlap in terms of named entities.
The accuracy depends on the length of the segments. For example, we can perform pairwise comparisons of document similarity starting with the seed document 𝑑 𝑠 of which we know that it represents topic 𝑇 and measure its similarity to other candidate documents. Document pairs with a high similarity score are assumed to cover the same topic. If an incoming segment, e. g., a news article, is relevant to 𝑇 or its seed document 𝑑 𝑠 , the next steps involve identifying the important atomic segments (i. e., sentences or paragraphs) and determining the relations that hold between these atomic segments and 𝑇.
Given a document 𝑑 related to 𝑇, we need to determine the importance of 𝑑 (and its segments) with regard to 𝑇. Various cues and indicators can be exploited, e. g., an incoming news piece on 𝑇 that was published only seconds ago and that includes the cue "BREAKING" in its title. One way of determining the topical importance of an individual segment is to treat it as a segment-level question answering task. Given a document 𝑑 that consists of a sequence of segments (𝑡 1 , 𝑡 2 , … 𝑡 𝑛 ), the aim is to find the segment 𝑡 𝑖 that contains the answer to the input question. In our context, the input would be the topic 𝑇 instead of a question. A weighting schema could be applied such that, e. g., novel news pieces are preferred over old ones.
Relations can exist between two segments on different levels. While we are primarily looking at discourse or coherence relations, relations can also relate to more simple levels such as segment order. The modeling of semantic, discourse, or coherence relations in textual content, where propositions, statements or events are the individual units, is at the core of discourse parsing frameworks. These analyse a text for, typically, intra-textual but not inter-sentential, relations. We borrow from discourse parsing and experiment with PDTB2 annotations [21] (see Section 3.1), but there are several added challenges. Crucially, our system needs to be able to robustly find (discourse) relations of short segments extracted from different texts while we have ample evidence from step 1 that the two texts are relevant to each other.
While steps 1 and 2 can be considered common NLP tasks, step 3 is less standardized with only little related work in the literature, which is why we elaborate on the identification of relations between text segments in more detail. We summarise three separately published experiments that represent different approaches to the task: PDTB2 discourse classification [22], Wikipedia article relations [23], and text segment ordering [24].
The goal of this experiment [22] is to get preliminary results regarding the (ambitious) task of identifying discourse relations between two arbitrary text segments that are relevant to each other and that both relate to the same topic. The experiments are based on PDTB2 [21], which [25]. For the relation classification, we use a Siamese architecture.
Table 1 shows the results of the classifier which performs best for Expansion, by far the most frequent class. The performance is lower than state-of-the-art approaches, but a comparison is not straightforward. First, our classification performs considerably lower than other approaches because we have not implemented features relating to the connective. Second, we only use the four PDTB2 top level classes. Most other approaches use a more fine-grained set, resulting in many more classes and lower performance. See for example [26], who report an F1-score of 40.70 for implicit relations. For our experiments, however, we argue that a coarse classification, with more training examples and higher accuracy, is better suited.
Results for Wikipedia relation classification as micro avg. F1-scores.
Std.
Avg. GloVe [27] 0.875 ± 0.0036 Paragraph Vectors [28] 0.845 ± 0.0019 Siamese BERT [29,30] 0.870 ± 0.0067 Siamese XLNet [31,30] 0.864 ± 0.0096 BERT [29] 0.933 ± 0.0039 XLNet [31] 0.926 ± 0.0016
In this experiment [23], we transfer the relation classification from a sentence-level to a document-level and from PDTB relations to more generic ones. Given a seed document 𝑑 𝑠 , we are interested in finding a target document 𝑑 𝑡 that shares the semantic relation 𝑟 𝑖 with 𝑑 𝑠 . We model the task of finding the relation 𝑟 of a document pair (𝑑 𝑠 , 𝑑 𝑡 ) as a pairwise multi-class document classification problem. Wikipedia articles are utilized as documents and Wikidata properties [32] define their semantic relations. For example, the Wikipedia article on Albert Einstein and its Wikidata item are connected to German Empire through the property country of citizenship. The Wikidata property acts as the relation between the article pair and the class label in the training data for this pair of documents. We selected nine relations ranging from country of citizenship to facet of to opposite of. Besides the number of available Wikipedia article pairs, diversity was also a criterion in our selection with respect to the different semantic meanings of properties. We evaluate six different methods: Document vectors from average GloVe word vectors [27], Paragraph Vectors [28], BERT [29], XLNet [31], and Siamese variations of BERT and XLNet [33,30].
Section 2 presents the empirical results. BERT yields the best micro average F1-score with 0.933, followed by XLNet with 0.926 F1. The vanilla Transformers, BERT and XLNet, generally outperform their Siamese counterparts. The shared contextual information during the encoding of document pairs most likely yields the better performance for vanilla Transformers. Even abstract relations, like facet of, yield a considerable high F1-score (0.91 for BERT). Siamese BERT (0.870 F1) and Siamese XLNet (0.870 F1) are even outperformed by AvgGloVe (0.875 F1) despite AvgGloVe requiring only a fraction of the computing resources compared to the Transformer models. A qualitative evaluation and detailed analysis is presented in [23]. Our results suggest that pairwise classification is suitable for classifying semantic relations between documents. In another study [34], we confirm this finding also for the domain of research papers.
In the last experiment, we study the relation of coherence between text segments [24]. When segments are related to the same topic, we want to determine which of the segments (sentences or paragraphs) should precede the other in order to maximise discourse coherence. In the best case, the predicted order would correspond to a complete and coherent story line although being composed of individual segments. As opposed to pairwise ordering (similar to pairwise classification, Section 3.2), we apply direct ordering of all segments using a Pointer network [35] combined with the pre-trained encoder-decoder model BART [36]. As baseline, we rely on Hierarchical Attention Networks (HAN) inspired by [35] that uses a Pointer network, Multi-Head Attention and LSTMs for sequence representations. We evaluate the BART-Pointer and the HAN baseline on two new paragraph ordering datasets tailored to Semantic Storytelling. As opposed to common segment ordering datasets using only single sentences as segments, for Semantic Storytelling, we are also interested in paragraphs with more than one sentence. Hence, we construct two new datasets for paragraph ordering based on the CNN DailyMail (CNN-DM) dataset and on Wikipedia.
Our model outperforms the HAN baseline on both datasets. With a Perfect Match Ratio (PMR) of 0.3699 for Wikipedia and 0.0171 for CNN-DM, the BART-Pointer combination is significantly better than the baseline, which yields 0.2100 PMR for Wikipedia and 0.0049 PMR for CNN-DM. An evaluation with Kendall's Tau metric (𝜏) shows that for 36.99% of the test samples our model is able to perfectly order the shuffled paragraphs from the introduction of Wikipedia articles while only 1.71% of the CNN-DM articles can be ordered perfectly. CNN-DM seems to be a greater challenge to the model by the number of paragraphs with an average of 14.5 sequences to order, whereas the Wikipedia dataset has an average of 6.29 paragraphs. We assume that the introductions in Wikipedia articles is often more consistent than those in CNN-DM, thus, the order is easier to learn. To sum up, we evaluate the BART-Pointer combination as suitable for ordering paragraphs to create a coherent text from an unordered collection of segments.
The (semi)automatic identification and generation of storylines from text segments is still in its infancy. In this paper, we focus on one crucial step of our Semantic Storytelling approach, i. e., the identification of the relation between text segments. We approach the task from different angles, i.e., the notion of relation is defined as PDTB2 discourse relation, Wikidata properties, or order relation. The results of the PDTB2 experiments (Section 3.1) reveal a below state-of-art performance of the Siamese BERT approach. We attribute the low performance to the inability of the Siamese network to encode relational information of the two text segments.
This assumption is confirmed by the method evaluation of the second experiment (Section 3.2) carried out as a pairwise multi-class document classification task with more training data and on a variety of non-discourse relations. The methods from the experiment yield substantially higher accuracy scores compared to experiment 1. On a methodological level, vanilla Transformer models turn out to be more suitable for the relation classification task as their Siamese counterparts. We find that also presumable difficult relations like facet of achieve promising results that would be suitable for our use cases. However, the pairwise document classification approach has one drawback: the approach only classifies a single pair of segments while stories consists of multiple segments.
To address this, experiment 3 explores relation identification as a paragraph ordering task (Section 3.3). The order relation ensures a coherent story generation, i. e., more than two segments are arranged in a meaningful manner. In our experiment, we demonstrate that a Pointer network in combination with an encoder-decoder model like BART, is capable of not only ordering sentences but also text segments of paragraph length. Given the difficulty of this task, our results are very encouraging.
This brief overview of related work refers to several areas including narratology, discourse theory, as well as applied work in computational linguistics and language technology.
Several approaches grounded in narratology address storytelling as a way of automatising the detection of instances of story grammars [37], especially events, in texts. Caselli and Vossen [38] present a data set for the detection of temporal and causal relations and use a plot structure [39] to order events found in narratives or text documents, chronologically and logically. According to Bal [39], narratives follow a plot structure that consists of ordered events, told by an agent or author and caused or experienced by actors. Yarlott and Finlayson [40] use Propp's (1968) morphology of Russian hero tales for story detection and generation systems. His book, first published in 1928, Propp analyzes the structural elements of Russian folk tales, which always occur in a fixed, consecutive order. Yan et al. [42] describe a system that learns "functional story schemas" as sets of functional structures (e. g., character introduction, conflict setup, etc.) in social media narratives. They extract patterns of functional structures. Afterwards, their formation in a story is analyzed across all stories to find schematic structures. Vice versa, Gordon et al. [43] use stories from blog articles to perform automated causal reasoning. Bois et al. [44] recommend articles based on simple lexical similarity. They link news articles in the form of a graph and label links to inform users on the nature of the relation between two news pieces. Ribeiro et al. [45] cluster news articles based on identified event instances and word alignment. They attempt to form clusters of online articles that deal with a certain event type. Nie et al. [46] use dependency parsing and discourse relations to determine sentence relations by learning vector representations. Yarlott et al. [47] apply the discourse theory by Dijk [48] to examine how paragraphs behave when used as discourse structure units in news articles.
After laying the groundwork for Semantic Storytelling through various experiments, we are now approaching the final phase, in which we attempt to combine the components described in Section 2.1, including text analytics and enrichment services that operate on documents, document collections or text segments [10], with the emerging set of technologies described in Section 2. To this end, we identified the key building blocks for Semantic Storytelling technologies. While step 1 can be implemented using one of several known approaches, steps 2 and 3 are much more challenging (Section 2). Our approach is grounded in the assumption that different texts that deal with the same topic but that are from different authors and different sources can be interconnected in a meaningful way through relations, which we attempt to extract automatically. We want to support content curators and make use of these relations holding between two segments by exposing them explicitly and exploiting them in the construction of storylines in a semiautomatic or fully automatic way. While our experiments are promising [49,24,23] they also show that additional research is needed before we can integrate the technologies into prototypes. Data sets annotated for rhetorical or discourse structure are still rather limited both in availability and in size. Our future work will focus on expanding our setup, especially with regard to the analysis and classification of discourse relations and more sophisticated processing of connectives. We will integrate a more flexible approach with regard to the processing of single documents by concentrating on larger parts of a document including longer summaries and paraphrased variants to increase coverage. Taking into account explicit ontological knowledge to identify semantic relations between texts will also be an important next step towards the completion of the envisaged Semantic Storytelling prototype [10].
The work presented in this paper has received funding from the German Federal Ministry of Education and Research (BMBF) through the projects QURATOR (Wachstumskern no. 03WKDA1A) and PANQURA (no. 03COV03E).