Narratives wield a profound in uence, shaping perceptions, beliefs, and decision-making processes. Although contemporary pre-trained language models have showcased impressive capabilities in text generation and question-answering tasks, they grapple with inherent limitations in knowledge coverage and exhibit vulnerability to societal biases. This work endeavors to forge a methodology that applies Knowledge Graphs in narrative construction. Rather than solely focusing on fundamental aspects such as the 4W (who, what, when, where) and general relationships, our approach comprises nely detailed semantic relations, delineating precise type of causality such as an event preventing, intending-to-cause, causing, or enabling another event. Applying state-of-art methods to predict such rich information, we demonstrate that it is possible to obtain automatically generated narratives of better grammatical and semantic accuracy.
Narratives stand at the heart of our societal fabric, serving our understanding and facilitating the exchange and preservation of knowledge. These narratives lter through our everyday lives, appearing in diverse forms such as commercials, political campaigns, news broadcasts, and more, each with its unique purpose and signi cance. Stories hold immense power to shape our thoughts, beliefs, and actions, making them captivating and transformative [1]. Consequently, the quest to innovate in the realm of complex narrative generation holds the potential to usher in a new era of AI systems that are intricately attuned to human sensibilities. Building upon the profound role of narratives in our society, it becomes evident that our means of narrative generation and comprehension are intertwined with the capabilities of modern AI. Pre-trained language models (PLM), exempli ed by models such as BERT [2], GPT-3 [3], and the more recent ChatGPT (GPT-3.5) 1 , have showcased remarkable progress in text generation, and conversational tasks. Yet, these models, shaped by training on extensive datasets drawn from undisclosed and diverse sources, bear intrinsic limitations, including knowledge gaps, inaccuracies, and societal biases [3,4]. Their challenges in maintaining semantic coherence and capturing long-term dependencies within text generation further underscore the need for innovation in narrative crafting [5,6].
Knowledge Graphs (KGs) are proven to be suitable structures for human knowledge, designed for machine-readability and adaptability, while several experiments of text generation from KGs are present in the literature [7]. Several KGs are available as data sources for the automatic generation of narratives. For example, EventKG [8] is a knowledge graph that consolidates and links events extracted from diverse sources, including Wikidata and YAGO [9]. This knowledge graph comprises more than 1.3 million events, each associated with its respective spatial and temporal coordinates. However, EventKG primarily focuses on representing events attributed and relationships between sub-events and super-events. While the value of such a knowledge graph is undeniable, its limitation to speci c event properties, notably the sub(super)events or the 4W, results in succinct and somewhat incomplete narratives.
Instead, the FARO dataset [10,11] encompasses a broader spectrum of semantically precise relationships. This includes event-related connections such as Prevention, Enabling, Causality, and Intention. In this work, we propose to enhance the WebNLG dataset [12], by incorporating the FARO dataset. This augmentation aims to generate text with more detailed semantics, particularly focusing on causal, preventive, intentional, and enabling relationships within a speci ed subgraph of events. You can locate the implementation code, and the appendix at https://github.com/ANR-kFLOW/KG2Narrative.
The remainder of this paper is structured as follows: we rst review the prior research pertaining to narratives and the extraction of relevant information from KGs (Section 2). We present datasets in Section 3, and we detail our approach for KG summarization, which encompasses an initial information selection step before text generation in Section 4. We then present both qualitative and quantitative results in Section 5. We conclude and outline some future work in Section 6.
A narrative graph [13] incorporates two main components: the individual representation of events, including the "four W" aspects (who, what, when, where) and the interconnection of these events through temporal and causal relationships. The Simple Event Model (SEM) [14] provides a foundation for modeling events, but is still insu cient to link disparate events or classes of the same type. To address this limitation, Blin [13] suggests enriching the event relation types: temporal or causal links from Allen [15] and dbo:alongside links between classes of the same type. Furthermore, the FARO ontology2 [10] covers most of the existing event relations in the literature, from temporal relation to causal and more ne-grained ones such as prevention.
KG summarization is an initial step of information retrieval and selection. To acquire the essential nodes for event description, an e ective approach involves ranking techniques that assign signi cance to nodes based on the relationships they possess. Various methods can be used such as entity ranking, relationship ranking, and semantic document ranking [16]. Blin et al. propose a system that can identify relevant information needed to build a narrative graph, by using an informed graph search traversal strategy [17]. To determine which information is considered 'relevant' the method uses lters to prune the search space with respect to the Simple Event Model (What, Who, Where, When).
On the other hand, di erent methods for generating texts from knowledge graphs have been proposed. In [18], triples are extracted to ne-tune a GPT-2 model [19], making the model dependent on the input triples. A similar approach is introduced in [20], involving BART [21] and T5 [22]. This approach obtained state-of-the-art performances on the AGENDA dataset [23] but not on the WebNLG dataset. Both found that Pre-trained Language Models (PLM) work well on unordered representations of the graph. JointGT [24] uses BART and T5, and exploits new pre-training methods to explicitly preserve the input graph's structural information. JointGT outperforms the other mentioned technique on WebNLG, which might indicate that including the topology of the graph lead to better results. A di erent approach [25] uses a transformer encoding structure to encode both the global information and the local topology information, and feeds a transformer to decode and generate text. However, this did not work as well as the previously mentioned technique [20], which used a PLM model without encoding. This might indicate that PLMs obtain better results than self trained transformer models.
In this section, we present the datasets that we used to train our method: WebNLG [26] and the FARO dataset [11] (Table 1). For evaluation, we use two evaluation datasets: the FARO test set and the ASRAEL KG [27]. ASRAEL is a knowledge graph that includes various event-related articles and their interconnections, including the 4W relations. Before our evaluation, ASRAEL lacked precise semantic relations. Therefore, we had to extract these relations from the event articles (linked to the KG) to conduct the assessment. We enhanced the ASRAEL KG with these extracted additional relations (similarly to the ones in FARO), resulting in a more dense and comprehensive knowledge graph. To achieve this objective, we used a pre-trained REBEL model [28] to extract events and relations (cause, enable, prevent, and enable). Furthermore, we leverage an existing event co-reference resolution model [29] to perform the task within the KG. This model creates clusters of mentions, computes similarity scores for each cluster, merges those with the highest score, and repeats this process until the score fell below a de ned threshold, which we empirically set to 0.95. This clustering process resulted in a graph primarily composed of clusters with a single mention, which are due to not nding a similar match. According to our manual assessment, the algorithm matched correctly a large number of syntactic matches, which makes it trustworthy. In total, we successfully clustered 45,031 mentions, with 36,057 being unique. The resulting narrative graph 3 provides a RDF representation of event co-references and relationships, enriched with ontologies such as NIF (NLP Interchange Format4 ), SEM and FARO to describe the relations between triples, further enhancing the context and meaning of our knowledge graph. A global overview of a narrative graph, and a concrete example can be found in the appendix.
Knowledge Graph summarization comprises two tasks: the selection of pertinent information from the knowledge graph, and the text generation based on the extracted data.
A SPARQL query has been written to extract the essential nodes, such as persons, places, and times, crucial for narrative construction from a main event within the ASRAEL KG. This query prioritizes the selection of events involving the 4W nodes with higher frequencies of incoming edges. Mentions are selected similarly; the larger the cluster of co-referent mentions (formed by the event co-reference model) is, the higher the priority of said cluster. Since we face a limitation on the number of input tokens of the text generation model, up to three mentions are selected from the same cluster.
The quality of the output depends largely on the quality the output of previous steps (relation extraction and co-reference resolution). Future work aims to enhance the accuracy of both these tasks and explore methods for identifying indirectly linked relevant nodes to selected events.
As anticipated in Section 2, using a PLM instead of training a language model from scratch can lead to better results. Furthermore, incorporating the graph's topology into the model has been shown to generate better natural text. JointGT [24] incorporates both of these characteristics, hence, we adopted this method. The authors pre-trained this model on the KGText dataset [30], consisting of 7 million graph-text pairs extracted from English Wikipedia dump. 5 It includes around 1.8 million entities and 1,210 relations.
The WebNLG dataset does not contain any of the FARO relations. Therefore, we ne-tuned the model on a merged dataset, combining the WebNLG and FARO, as in Table 2 without making changes to the model itself. The creation of this combined dataset involves the following multistep process. Initially, entities and their respective encodings are extracted from the WebNLG dataset. Subsequently, entities from the FARO dataset are encoded utilizing the extracted encodings from WebNLG. Finally, the resulting encodings and their relations are integrated into the original WebNLG dataset, thereby producing the combined dataset. The model undergoes ne-tuning on the WebNLG dataset. We refer to the original model as base model, and the model ne-tuned on the combined dataset as combined model. 6 Table 3 provides crucial insights into the model's performance, measured by the BLEU, METEOR, and ROUGE metrics. BLEU emphasizes precision, indicating how accurately the generated text aligns with the reference text. On the other hand, ROUGE focuses on recall, gauging the extent to which the reference text is captured in the generated output. METEOR combines elements of both precision and recall, and its e ectiveness can be further enhanced by incorporating improved word matching strategies. ROUGE suggests a high level of alignment with reference texts in conveying information, while BLEU shows minor word deviations from references. The lower METEOR score might stem from alignment nuances in score calculation. Notably, the base model's test performance closely mirrors the results outlined in the original JointGT paper [24]. The model that was trained on the combined dataset performed slightly worse for all three metrics than the model that was trained on the base WebNLG data. This can be explained by two considerations. First, it is evident in Table 3 that tests on FARO have very low performances. Secondly, the FARO dataset only accounts for a relatively small proportion in the combined dataset (Table 2). To better understand the reasons, a qualitative analysis is proposed in the next section.
We examine instances from WebNLG and FARO datasets to analyze the base and combined model's performance. Observing Tables 4 and 5, the text generated by the combined datasettrained model appears more semantically robust. The base model's generated text for FARO triples (Table 4, column Base generated) is notably brief, often mirroring the triples with semantic inaccuracies. Conversely, the combined model produces more coherent and accurate sentences in the same dataset (column Combined generated), maintaining triple direction. However, it's important to note that while the generated content respects triple and semantic accuracy, it may still have limitations in altering the original label's content.
We also get a sight why the quantitative results are slightly worst for the combined model. The WebNLG data (Table 5) contains multiple triples per instance, giving more information about the text, and contains multiple labels. The FARO data (Table 4) contains only one triple per instance, together with one target sentence (label). Therefore, the model has less information about what to generate, and less chances to match the target label. Looking at the FARO input triples and the target label, it can be seen that the relationship (predicate) is often not explicitly represented by a particular word in the target sentence (implicit relation), making the evaluation with matching words harder. We provide additional insights in the appendix.
User Evaluation on ASRAEL To evaluate the system's performance, seven events from the ASRAEL dataset have been selected based on several criteria: values for the 4W properties, linking to a minimal number of articles, etc. The two largest (in terms of having the most articles) events in ASRAEL having all of the 4W properties are selected for evaluation: "Operation Breaking Dawn", and "2021 storming of the United States Capitol". The rationality behind this is to ensure that the information selection method is challenged by having an extensive amount of information to choose from. Among the remaining events in ASRAEL that include information about the place and time, ve additional events are selected, bringing the total to seven.
The information selection method is used to select time, place, actor, and up to three mentions from the seven selected events. The base and combined models are used to generate text from the selected information. This information per event can be found in the appendix, together with the generated text. A manual evaluation was needed due to the absence reference text for automated metrics. Three annotators with a pro cient level of English uency determined which text was better for each event, by using either "win", "lose", or "tie", assessing uency (grammatical correctness) and adequacy (correct integration of triples). This method aligns with the approach in [24]. Majority voting determined the winner or equality between models, followed by a nonparametric sign test at a signi cance level of ↵ = 0.05 to establish superiority. The non-parametric statistical sign-test is used to compare data. It assesses whether the median di erence between observations di ers signi cantly from zero, providing a p-value that indicates the probability of observing the given di erence or a more extreme di erence if the null hypothesis (no di erence) were true. The signi cance level, denoted by alpha ↵, is a predetermined threshold set at 0.05, Table 4: Sample of the FARO test-set and the generated output of the base and combined model.
(o er, cause, reimburse) (The directors said if Messrs. Drabinsky and Gottlieb mail an o er to shareholders by Nov. 22, it will reimburse them a maximum of C$8.5 million for expenses related to a bid.)
The cause of the o er is to reimburse .
The company has also announced that it will o er a new credit facility to small businesses, in an e ort to reimburse them for the cost of capital expenditures. against which the p-value is compared to determine statistical signi cance. Results of this annotation are accessible in Table 6.
The combined model produces better uent text than the base model in 71.4% of the cases. The non-parametric "sign test" was performed to measure a signi cant di erence in the uency of the text. With a p-value of 0.11, no signi cant di erence was found. The same was done to gauge the text's adequacy. With a p-value of 0.25, no signi cant di erence was found.
To demonstrate whether the obtained results are consistent independently from the quality of the information extraction output, we decided to perform a user evaluation on a single article (sample), which has been manually annotated by handcrafting the resulting subgraph. This subgraph has been processed with both the combined and base model, and then evaluated using either "win", "lose", or "tie", in the same way as described in the previous section. The percentage of wins, losses and ties for the combined model, together with the Fleiss' kappa are reported in BLEU, METEOR, and ROUGE metrics have been computed using the sentences from the article as "reference label". These scores are detailed in Table 7. This illustrates that the base model performs slightly better than the model that was trained on the combined data. A reason for this could be formulated by looking at the generated texts, which can be found in the appendix. More often than the combined model, the base model will output parts of the triple without taking the relationship between them into account. This will result in a badly formed sentence, but higher metrics, since more triples are incorporated. This is also re ected in the scores in Table 8, where the combined model is commonly noted for producing more uent texts. Furthermore, the scores in Table 7 (computed on a single annotated article) are much lower then those computed on the whole WebNLG test set (Table 3). This outcome could be expected, considering that some of the triples extracted from the article are not, or to a limited extend, present in the original WebNLG data used to pre-train the JointGT model.
The primary goal of this research is to investigate how to build complex narratives in the form of graphs of events, generating text with good level of complexity and semantic richness, expecting the system to generate answers beyond only What (event), Who (actor), Where (location), and When (time).
We enhanced the WebNLG dataset through the incorporation of the FARO dataset, aimed at re ning the semantic depth of event relations. The expanded dataset now encompasses intricate relations including causality, prevention, intention, and enabling. Even if the metrics show not clear improvement, from qualitative analysis, we can state that training on precise event relations produces more complete generated sentences, while no statistically signi cant di erence was observed on uency. Future work will experiment on more data to draw nal conclusions. Our information selection from the graph focuses solely on the main event, disregarding pertinent details from interconnected events. Additionally, the data used for ne-tuning di ers from the original dataset in terms of triple counts and instances, potentially impacting model evaluation. Future research could explore selectively extracting sub-events and relations at the document level to enhance clustering. Moreover, augmenting the dataset through NLP techniques could signi cantly improve its quality and comprehensiveness.
This work has been partially supported by the French National Research Agency (ANR) within the kFLOW project (Grant n°ANR-21-CE23-0028) and the European Union Horizon 2020 research and innovation programme within the MUHAI project (Grant n°951846).