In this paper we present a rst step towards a system to extract formal narratives from text. This work is part of a wider research on the introduction of narratives in Digital Libraries. We represent narratives as networks of events, each set in space and time, endowed with factual components, and linked to each other through semantic relations. In order to extract a narrative from text, the rst step is to automatically detect and classify the events in the text. We present a software we developed that uses neural networks for event detection and classi cation. It was trained on a dataset of annotated biographies of writers and artists from the English Wikipedia and on the ACE 2005 training corpus. We tested the software on the biography of Florentine poet Dante Alighieri. This software constitutes the rst component of a broader system for narrative extraction from natural language text.
On top of the ontology, we developed the Narrative Building and Visualising Tool (NBVT), a semi-automatic software that allows users to construct and visualize narratives through a Web interface [Met16]. The tool is able to import knowledge (e.g. locations, people, organisations) from the Wikidata knowledge base [VK14], thereby facilitating the user in the construction of the narrative. The tool has been used to create four narratives2 about di erent subjects: the life of Florentine poet Dante Alighieri, the life of Austrian painter Gustav Klimt, the history of the giant squid, and the history of climate change.
In order to improve the narrative building process, it would be particularly useful to endow the tool with a functionality to automatically extract elements of narrative from natural language text. For instance, the tool could identify the main events in the text, their factual components, and the relations existing among them. In this paper we present the results of an initial experiment towards this goal, focusing on event detection and classi cation, and we discuss how we plan to proceed further.
Section 2 describes the requirements we identi ed for the task of narrative extraction from text. Section 3 reports some previous studies that we have found useful when designing our event detection and classi cation system. Section 4 describes the implementation of our experimental system, including its implementation and evaluation. Section 5 describes our initial work on a user interface for building narratives based on the extracted data. Finally, Section 6 reports conclusions and future works. 2
In this section we present some technical requirements for narrative extraction that we have identi ed. We want to be able to extract as much knowledge as possible from natural language text, in order to aid the user in the narrative building process. The requirements are: 1. Event Detection. First of all, events need to be detected in the text. We plan to do so using deep learning [LBH15] techniques allowing us to identify event triggers, i.e. words that express the occurrence of an event. 2. Event Classi cation. Events that are detected in the text need to be classi ed, i.e. categorized into a set of event classes (for instance, World War II could be classi ed in a class called Con ict ). 3. Named Entity Recognition. We need to extract from the text the named entities that identify the factual components of the events. 4. Event Component Extraction. After we have found all entities and events in the text, we need to identify which entities act as arguments of the events. 5. Temporal Entity Extraction. Temporal entities such as dates, years, and other indications of time also need to be extracted. These entities are needed to correctly place the event in the narrative's timeline. 6. Relation Extraction. We want to recognize the relations existing between events and also between events and their components (e.g. the role that a person plays in the event). 7. Entity and Event Linking. It is useful to perform entity linking and event linking to connect them to an external knowledge base. 8. Narrative Construction. Once we have identi ed all entities and all events, and linked them to each other, we have to construct the fabula and plot of the narrative.
In this paper we focus on the rst two requirements, i.e. event detection and classi cation, and describe the system we developed to achieve this goal. The satisfaction of the remaining requirements will be tackled in a future publication. 3
In this section we report the most relevant related works for our present study about event extraction and narrative extraction from natural language text.
Since the 1990s, machine learning has been applied to the task of event extraction from text, rst through rule-based systems [Chi98, LCF+92] and later through supervised classi ers [Fre98, CN02]. Starting in 2004, 2https://dlnarratives.eu/narratives.html the ACE program has focused on recognition of entities, values, temporal expressions, relations, and events from natural language text [DMP+04]. In particular, the ACE 2005 evaluation involved the extraction of several classes of events from an extensive annotated corpus [WM05].
In the 2010s, research in event extraction made signi cant progress, with the development of new featurebased and pattern-based approaches [Cao17] and a gradual shift from traditional dependency parsing [LJH13] to deep learning [LBH15]. In particular, convolutional neural networks are often able to identify the structural features of a sentence in a deeper way than feature-based approaches [NG15], but they also have shortcomings, e.g. they nd di culty with sentences containing multiple events [CXL+15]. Recurrent neural networks, and in particular bidirectional LSTMs [SP97], are better able to capture long-term dependencies in natural language text [NCG16, FQL18].
The application of event extraction techniques to narrative representation has been limited. These works are particularly relevant to our research topic since they share our goal of extracting not just events, but narratives from text. Chambers & Jurafsky were the rst to apply event extraction algorithms to narrative extraction from text [CJ08, CJ09]. Another relevant approach is that of Elson [EDM10], who extracted social networks from a set of 19th Century literary texts. More recently, narratives were applied to information extraction from news streams [VCK15] in the context of the NewsReader project [VRS+14].
At the same time, research has been very active on the extraction of events from social media streams [SOM10, BNG11, WL11] and the subsequent development of a knowledge graph based on the detected events [SST+18]. Event extraction has also been researched extensively in the biomedical eld [YTMT00, BS11]. 4
As an initial experiment, we implemented a system to perform the rst two tasks that we identi ed as requirements in section 2, i.e. event extraction and classi cation. After studying the main event extraction systems reported in Section 3, we decided to adopt a recurrent neural network architecture based on bidirectional Long Short-Term Memory (LSTM) [SP97, HS97], since the results obtained through this approach are promising [FQL18]. In this phase of our research, the goal is not to develop a system with better performance on event extraction from text in comparison with the previous ones, but only to investigate if this approach can be satisfactory in order to be integrated in our architecture to build and visualize narratives.
In order to evaluate if the LSTM could be successfully applied for our aims, we selected a case study to test this approach. We chose the narrative of the life of Dante Alighieri as case study, focusing on 12 classes of event that were especially relevant in this case study. The classes are reported in Table 1.
4.1
Annotation of Training and Test Sets We analyzed several corpora containing annotated events, such as the ACE 2005 corpus,3 ASTRE [NTFB16], and MEANTIME [MSU+16]. Among these, we decided to adopt the ACE 2005 corpus since it contains several event classes that would be useful for our purposes. Unfortunately, some important classes that we need to extract are not present in the corpus (e.g. Creation, Education, Residence, Membership).4 Furthermore, some of the classes that are present in the corpus are under-represented (e.g. Birth). For these reasons, we decided to supplement the ACE 2005 dataset with an additional training dataset of our own development.
Given that our case study about the life of Dante Alighieri is based on the English Wikipedia page about the poet, we selected 10 biographies of writers and artists from Wikipedia as additional training set. Our aim was to collect texts containing events that describe the life of a person. We included biographies of people who lived in a wide range of time, from the Middle Ages to the 20th century. We developed a simple annotation interface that we used to tag all event triggers found in the text. The interface, shown in Figure 4.1, allows the annotation of any number of classes de ned by the user, and is able to export the annotated dataset in JSON5 format. The annotation of the training set was performed by two annotators. The inter-annotator agreement, measured using Cohen's kappa statistic [Coh60], is reported in Table 2.
The total number of annotated events in this training set is 604, with an average of about 50 events per class. Given the fact that the subjects of the Wikipedia pages are all writers and artists, the most represented event class is Creation (183 events). The least represented classes are Membership (8) and Sentence (9). All other classes appear at least 15 times in the set.
As test set, we annotated the English Wikipedia page about Dante Alighieri using the same annotation
interface that we developed for the training corpus and following the same annotation methodology. The resulting
annotation contains 88 events, with a minimum of 3 instances, and an average of about 7 instances, for each
event class. The most represented event class is Con ict (
The software takes as input the annotated sentences, tokenizes them, and applies the Komninos dependencybased word embeddings [KM16]. The embeddings are given as input to a convolutional neural network (CNN), after which pooling is applied. The results are fed to two LSTM layers of di erent sizes, in sequence. Dropout is applied both to the CNN and to each LSTM layer. Finally, a hidden dense layer computes the nal results by applying a softmax function [Bri90]. 4.3
Evaluation Given the heterogeneity of the two training sets, we decided to run the neural network model twice, rst on the ACE 2005 dataset and then on the set of 10 Wikipedia biographies we annotated. In both cases, we performed validation on 10% of the data, which was held out from the two training sets. After running the model on the two datasets, we combined the results using a weighting function. We performed some tests assigning di erent weights to the two datasets in order to identify the optimal weights to maximize the F1-score. Based on the results of our tests, we decided to assign a 0.33 weight to the ACE dataset and a 0.66 weight to the Wikipedia dataset. Finally, we computed precision, recall, and F1-score using as test set our annotation of the Wikipedia page about Dante Alighieri. The results of the evaluation are reported in Table 3.
6https://github.com/UKPLab/emnlp2017-bilstm-cnn-crf 6Computed as the unweighted macro-average among the 12 event classes.
Our results cannot be compared directly to the baselines for the ACE 2005 dataset reported in the publications cited in the Related Works section, since we introduced a di erent corpus with a di erent set of classes. As baseline for the detection task, we chose the results of the event extraction component of the NewsReader software [VRS+14]. NewsReader is provided as a service by the NLPHub of the DataMiner cloud computing system, based on the D4Science e-Infrastructure [CPSP17]. On our test set, NewsReader was able to detect events with 71.9% precision and 61.4% recall, achieving an F1-Score of 66.2%. Our software constitutes an improvement over this baseline (see Table 3). We plan to conduct a more detailed evaluation in the near future. 5
After completing our experiment on event extraction and classi cation, we started developing a web interface that we plan to integrate into our Narrative Building and Visualising Tool. The interface allows the user to build events starting from the ones identi ed as candidate events by the event extraction system.
In addition to events, the current version of the interface also shows named entities (extracted using Stanford CoreNLP [MSB+14]). When possible, the named entities are linked to Wikipedia by using the WAT [PF14] entity linking service, and subsequently to Wikidata. Since at this stage the system does not perform relation extraction, the interface allows the user to manually link the entities that are factual components of the events to the identi ed events through drag-and-drop functionality. The interface is shown in Figure 5. 6
In this paper we have presented an initial study on narrative extraction from text, focusing on event detection and classi cation. To detect and classify events, we have developed a software based on recurrent neural networks. We have built a training set based on English Wikipedia pages of writers and artists, and tested the system on the biography of Florentine poet Dante Alighieri. Based on the current training set, the system has achieved an F1score of 73.0 on event detection and an F1-score of 70.3 on event classi cation. We have also started development of a full- edged narrative extraction interface. Currently, the interface allows the user to automatically detect events and named entities in the text, and manually link them to each other.
As future works, we plan to build a complete system for narrative extraction from text, structured in several components. In addition to the initial event detection and classi cation component we presented in this paper, the system will be endowed with components for further automating the narrative extraction process, e.g. by extracting relations between events and their related entities, and linking both entities and events to an existing knowledge base. [Bal97] Jimmy Lei Ba, Jamie Ryan Kiros, and Geo rey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Valentina Bartalesi, Carlo Meghini, and Daniele Metilli. A conceptualisation of narratives and its expression in the CRM. International Journal of Metadata, Semantics and Ontologies, 12(1):35{46, 2017. [Doe03]
Xiaocheng Feng, Bing Qin, and Ting Liu. A language-independent neural network for event detection.
Science China Information Sciences, 61(9):092106, 2018.
Dayne Freitag. Multistrategy learning for information extraction. In Proceedings of the International
David Herman. Basic elements of narrative. John Wiley & Sons, 2011. Sepp Hochreiter and Jurgen Schmidhuber. 9(8):1735{1780, 1997.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188, 2014.
Alexandros Komninos and Suresh Manandhar. Dependency based embeddings for sentence classi cation tasks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1490{1500, 2016. Yann LeCun, Yoshua Bengio, and Geo rey Hinton. Deep learning. nature, 521(7553):436, 2015. Wendy Lehnert, Claire Cardie, David Fisher, John McCarthy, Ellen Rilo , and Stephen Soderland.
151{158. Association for Computational Linguistics, 1992.
Qi Li, Heng Ji, and Liang Huang. Joint event extraction via structured prediction with global features. In Proceedings of of the Annual Meeting of the Association for Computational Linguistics, volume 1, pages 73{82, 2013.
Daniele Metilli. A Wikidata-based tool for the creation of narratives. Master's thesis, University of Pisa, 2016. Available from https://etd.adm.unipi.it. [NTFB16] Kiem-Hieu Nguyen, Xavier Tannier, Olivier Ferret, and Romaric Besancon. A dataset for open event extraction in english. In LREC, 2016.
Francesco Piccinno and Paolo Ferragina. From TagME to WAT: a new entity annotator. In Proceedings of the First International Workshop on Entity Recognition & Disambiguation, pages 55{62. ACM, 2014.
Vladimir Propp. Morphology of the Folktale. University of Texas Press, 2010.
Nils Reimers and Iryna Gurevych. Reporting score distributions makes a di erence: Performance study of lstm-networks for sequence tagging. arXiv preprint arXiv:1707.09861, 2017.
Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. Earthquake shakes Twitter users: real-time event detection by social sensors. In Proceedings of the 19th International Conference on World Wide Web, pages 851{860. ACM, 2010.
Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673{2681, 1997. [MSB+14] Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Proceedings of of the Annual
[MSU+16] A-L Minard, Manuela Speranza, Ruben Urizar, Begona Altuna, MGJ van Erp, AM Schoen, CM van Son, et al. Meantime, the newsreader multilingual event and time corpus. In Proceedings of the 10th
Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. Joint event extraction via recurrent neural networks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 300{309, 2016.
Thien Huu Nguyen and Ralph Grishman. Event detection and domain adaptation with convolutional neural networks. In Proceedings of of the Annual Meeting of the Association for Computational Linguistics, volume 2, pages 365{371, 2015.
Piek Vossen, Tommaso Caselli, and Yiota Kontzopoulou. Storylines for structuring massive streams of news. In Proceedings of the First Workshop on Computing News Storylines, pages 40{49, 2015.