The recent development of Large Language Models (LLMs) poses significant promise for advancing several natural language-based tasks, including event extraction from lengthy text. LLMs such as GPT models [1] have demonstrated remarkable capabilities in understanding and generating natural language text. The application of LLMs for event extraction offers several advantages. Firstly, these models can process vast amounts of text data, enabling comprehensive analysis of events described in natural language. Secondly, LLMs can capture complex linguistic structures and contextual nuances typical of different kinds of documents, enhancing the accuracy of extracted event details. The continuous learning ability of LLMs allows them to adapt to different writing styles and language conventions. However, challenges persist in leveraging LLMs for event extraction in languages other than English, particularly in languages with limited available resources such as Italian. Fine-tuning requires curated datasets that accurately represent the diversity of language and scenarios, and the annotation of different event-related data.
Despite these challenges, the potential of LLMs to revolutionize event extraction is substantial. For instance, Question Answering (QA) models can facilitate rapid and efficient access to relevant information by automatically identifying text spans containing the desired answers to specific questions. While other models can be provided with detailed instructions to extract specific data from the text. Integrating these models into NLP pipelines can streamline the process of real-time event analysis, allowing for timely and efficient extraction of event-related information from textual data. This paper explores the role of LLMs in advancing event extraction from lengthy text. In particular, we focus on the Italian language and we explore the resources available for adapting and evaluating LLMs to event extraction on Italian documents. In the end, we define possible future directions for research in this dynamic field.
Event extraction aims at identifying and categorizing events described within a text, including the recognition of the entities involved in the event (such as individuals, organizations, or locations), and the extraction of temporal references and any elements that are relevant for the event. This task has gained significant popularity in recent years due to its broad applicability and practical utility in various real-world scenarios. Figure 1 shows an example of the results of event extraction from a document describing an air crash. In addition to the identification of the event type, different event roles have been annotated, e.g., the date of the event occurrence, the aircraft agency.
Due to the complexity of the natural language, event extraction poses several challenges that require sophisticated techniques to address effectively.
The first challenge consists of detecting multiple events described in the same document and understanding which are the references to each event. Natural language often contains ambiguous expressions that can refer to multiple events or entities. This ambiguity, along with the use of coreference, further complicates the task of accurately extracting event data from text since resolving ambiguity requires contextual understanding and disambiguation techniques.
Identifying relevant elements for each event requires distinguishing between event triggers (words or phrases that indicate the occurrence of an event) and background information and noise. Another complexity is given by the variability in language usage, writing styles, syntactic structures, and document length. Indeed, event extraction can be performed on short text like tweets, longer documents such as news articles, and lengthy documents such as investigative reports or government documents. All these factors require the use of techniques able of accommodating these variations to achieve accurate and reliable results across diverse text types and genres.
Two of the key aspects of events are the time and the space, i.e., when the event took place and where. The recognition and standardization of temporal and spatial expressions could be complex since temporal reference can be expressed in various formats (such as dates, times, part of the day). In addition, a document describing an event can refer to the location of the event providing information at different granularity, for example indicating the name of the city, specifying the address, and/or describing the type of the place like an apartment, a shop, or a park. During event extraction, the references to all these locations should be identified.
Several approaches have been proposed for event extraction in recent surveys, from traditional methods which rely on the use of linguistic rules for pattern identification within the text to more advanced solutions such as machine learning and deep learning algorithms able to learn patterns after training on annotated data, and the use of pre-trained language models [2,3]. LLMs based approaches have emerged as a promising avenue for event extraction in recent years. These models leverage the power of machine learning and deep learning algorithms as they are pre-trained on vast amounts of text data and then fine-tuned for specific tasks. By encoding contextual information and capturing semantic relationships within the text, LLMs seem to be promising in identifying and extracting events from various sources. We identified three main approaches based on the use of LLMs that could reach good performance in event extraction: sequence labeling models, extractive Question Answering (QA) models and instruction-tuned models.
Sequence Labeling models In Sequence labeling each token in a sequence is assigned a label based on its role or category within the context of the sequence. Sequence labeling models can be used to identify those text spans reporting relevant information within a text. Therefore, it is widely employed for several classical NLP tasks like part-of-speech (POS) tagging, named entity recognition (NER), text chunking.
Sequence labeling models are suitable for the scenario of event extraction, where they can identify and classify those parts of text reporting information about events. Indeed, some works in literature have already treated event extraction as a sequence labeling or NER problem, [4,5], also for Italian Language [6].
Extractive Question Answering The goal of extractive QA models is to understand an input question in natural language and extract the answer as a span from an input text. QA models can facilitate rapid and efficient access to event-related information by automatically identifying text spans containing the desired answers to specific questions. For instance, the question "When did the event take place?" (Q1) can be formulated to retrieve the date of the event.
The results of these models depend significantly on the quality of the input documents, as well as the structure of the questions provided to the models. Prior knowledge about the kind of event described in the document allows to formulate ad hoc questions. For instance, considering the document in Figure 1, the question "When did the air crash take place?" (Q2) should provide more accurate answers than Q1. In addition, questions should be enriched by other details about the event after a partial process of event extraction. For example, the question "When did the Flight 345 crash?" (Q3) contain the reference to the flight number and should help the QA models to select the correct context for the extraction of the date.
Within the QA models, distinctions arise between Single-Span QA (SQA) and Multi-Span QA (MQA). While the former identifies a single text segment for each question, the latter locates answers even when distributed across non-consecutive text segments, potentially located far apart within a document. Given the prevalence of such scenarios, especially in complex inquiries and detailed documents, the limitations of SQA models are evident. An example is the annotation of "causalities and losses" in Figure 1. The recent surge in MQA model development [7,8,9] underscores a notable interest.
In the current state-of-the-art, the only Italian dataset properly designed for training QA models is SQuAD-it [10], derived from the automatic translation of the English SQuAD dataset, consisting of a list of pairs questionanswer. However, this dataset can be used only for SQA, therefore it is unsuitable for complex tasks like event extraction which requires the ability to retrieve multiple spans for one question.
Instruction-Tuned models Among LLMs, Auto-Regressive models such as GPT [1] or Llama [11] series stand out. These models leverage advanced deep learning techniques to predict the subsequent word based on an input text. This prediction process is repeated multiple times, with each predicted word being added to the original text. By training on vast amounts of text data, Auto-Regressive LLMs effectively capture complex patterns and structures in language, leading them to generate full and coherent text which is contextually relevant to input text.
The research in recent years has led to the development of instruction tuning [12] to bridge the gap between the next-word prediction objective of LLMs and the users' objective of following their instructions helpfully and safely. Instruction-tuning involves a fine-tuning of Auto-Regressive LLMs with input-output pairs, where input denotes the human instructions, and output denotes the desired output that follows the instruction. The results of this process are the Instruction-Tuned LLMs, designed specifically to provide appropriate results based on instruction inputs. This ability is also enhanced as a crosstask generalization, leading Instruction-Tuned LLMs to better performances on novel tasks.
Instruction-Tuned LLMs can be employed to solve a wide range of NLP tasks through various techniques of prompt engineering [13], i.e., the process of designing task-specific instructions to guide model output. Therefore, the utilization of these models can also yield benefits for event extraction.
Currently, there are several Instruction-Tuned LLMs capable of understanding and generating text. For those, Italian represents a minority percentage in the training data compared to more widely used languages on the web such as English. Among these, there are proprietary models like GPT-3.5 and GPT-4 from OpenAI, Gemini from Google, and open-source families of LLMs like Mistral [14] and Mixtral [15] from Mistral AI and Llama [11] and Llama 2 [16] from Meta. From this last family, Llamantino [17] has been derived through a language adaptation process to the Italian Language.
Currently, there are few Italian datasets suitable for event extraction. Some of them provide a comprehensive annotation of event-related data, while in other cases, only one type of information (e.g., the temporal references) is annotated.
The EVENTI1 corpus was built in 2014 for the evaluation of Temporal Information Processing systems of the EVENTI evaluation exercise [18] in the EVALITA workshop. The corpus consists of three datasets: the Main task training data (274 documents) and test data (92 documents) of contemporary news articles and the Pilot task (10 documents) test data of historical news articles. The annotation guidelines involve the use of four tags to annotate different elements within news texts: the EVENT tag is used to annotate all the mentions of events including verbs, nouns, prepositional phrases and adjectives; the TIMEX3 tag is used for temporal expressions; the SIGNAL tag identifies textual items which encode a relation either between EVENTs, or TIMEX3s or both; the TLINK tag is used for temporal dependencies between EVENTs and/or Temporal Expressions.
The NewsReader MEANTIME (Multilingual Event ANd TIME) is a multilingual semantically annotated corpus of 480 Wikinews articles in four languages: English, Italian, Spanish, and Dutch [19]. The corpus was released in 2016 and derives from the NewsReader Project2 [20] which aims at extracting information about what happened to whom, when, and where, processing a large volume of financial and economic data. The corpus is enriched with annotations that span multiple levels, including entities, entity mentions, events, temporal information, semantic roles, and intra-document and cross-document event and entity coreference.
The De Gasperi corpus [21] is a collection of historical documents by Alcide De Gasperi, the first Prime Minister of the Italian Republic. The corpus was released in 2019 and includes 2,762 documents published between 1901 and 1954, originally released in an oral or written form. In addition to the raw text, a set of meta-data and additional semiautomatically annotated information are available. The corpus contains different kinds of documents, like daily press written by De Gasperi when he worked as a journalist for newspapers in Trentino, and speeches in institutional venues when he was a Member of the Italian Parliament. In each document, references to persons and places are annotated.
DICE3 [22] is a collection of 10,395 Italian news articles describing crime events that happened in the Modena province between 2011 and 2021. The news articles are extracted from one of the most popular local newspapers, "Gazzetta di Modena", following the approach described in [23]. Thanks to an agreement between the University of Modena and Reggio Emilia and the Gazzetta di Modena, DICE was released online in 2023, free to redistribute and transform without encountering legal copyright issues under an Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).
Along with the data related to the title, the text, and the publication date of each news article that are crawled from the newspaper's webpage, several annotations are available on the data. The crime event category (e.g., theft, robbery) is assigned to each news article using text categorization approaches based on word embeddings [24,25]. The news articles underwent automated NLP processes to extract temporal references, entities, and corresponding DBpedia resources. Duplicates are annotated to identify news articles referring to the same crime event. The theft-related news articles are annotated manually following a sophisticated annotation schema to identify stolen items (What), crime locations (Where), references to authors and victims, and their sociodemographic characteristics (Who). The annotation provided in the dataset is multi-span since it involves identifying and linking multiple text spans within the document.
EventNet-ITA4 [26] is an Italian corpus for Frame Parsing applied to events released in 2024. Semantic Frame Parsing is a task which aims at identifying semantic frames within textual data. A semantic frame [27] is a cognitive structure that organizes and represents knowledge about a concept or situation. It consists of a set of interconnected elements such as roles, attributes, and relations, which collectively define the meaning and typical features of that concept or situation. Frames help humans understand and interpret language by providing a mental framework for comprehending and categorizing information.
EventNet-ITA is built upon the idea of enabling frame parsing for event extraction. It is composed of 53,854 sentences manually annotated with 205 semantic frames of events and covers different domains, like conflictual, social, communication, legal, geopolitical, economic and biographical events.
Automated information extraction from documents continues to captivate the scientific community due to its manifold advantages, facilitating improved information accessibility across various domains. By leveraging LLMs and exploiting annotated datasets, researchers can develop robust event extraction systems capable of achieving high accuracy and efficiency across a wide range of text sources. As the field continues to advance, further research into LLMs and their applications in event extraction is expected to drive continued innovation and progress in this area.
Future directions will focus on three key aspects:
• Definition of an Italian benchmark: while we have identified five Italian datasets suitable for event extraction, further efforts are needed to expand their annotation and support comprehensive event extraction tasks. This entails defining a standardized benchmark for evaluating event extraction systems. Such a benchmark would serve as a common evaluation dataset, enabling comparisons between different approaches and fostering the development of more accurate and reliable event extraction models. • Evaluation of LLMs on the benchmark: despite the limited literature on Italian event extraction, our preliminary evaluation of three BERTbased QA models on the DICE dataset revealed promising results [22]. However, challenges persist, particularly related to the size and quality of the annotated data. Once the benchmark is defined, future efforts will focus on evaluating and comparing various approaches outlined in Section 2.3. The evaluation will include the recent Minerva models that represent the first family of LLMs trained from scratch on Italian documents developed by Sapienza NLP. • Creation of a synthetic annotated dataset: since manual annotation is a time and resourceconsuming process, new strategies will be studied to automate the process of annotation. Employing LLMs for data augmentation (i.e., to expand the annotated dataset) is now the most promising approach, especially focusing on text generation models. Given a list of desired annotations, i.e., the spans to extract from the text (like "May 14th" as the date of the event), the LLM is asked to create a document with that span with the expected role in the event described (like "create a document describing an event that occurred on May 14th"). This methodology allows for obtaining a synthetic dataset that is also already annotated. Furthermore, this approach offers control over text generation and ensures fairness in dataset composition, ultimately contributing to the development of balanced and unbiased datasets essential for training accurate and equitable AI models.