Oficial crime statistics are often released with significant delays and in an aggregated form that limits their usefulness for real-time monitoring and detailed analysis. In diferent countries, as happens in Italy, accessing to detailed crime records is further restricted by the confidential nature of police reports. Conversely, online news articles ofer an alternative source of information that is rich and timely. However, their unstructured textual content poses challenges for automated data extraction processing. In this paper, we present a Natural Language Processing (NLP) pipeline designed to extract structured information from Italian crime news articles, which has been evaluated on a semi-automatically annotated dataset. Despite the limited amount of annotated data, the proposed approach showed promising results: the topic modeling component, included into the pipeline, facilitated targeted crime analysis, and the Named Entity Recognition (NER) module achieved an overall F1-score of 74%. These results demonstrate the efectiveness of the methodology in low-resource settings and highlight its potential for supporting automated crime analysis based on unstructured news data.
Crime is a complex phenomenon, and the ways in which it is addressed significantly vary depending on the legal and institutional framework of each country. In particular, the processes involved in collecting, managing and disseminating crime-related information can widely difer from one country to another. While many governments publish oficial statistics on the location, date and type of crimes, this data is often released with considerable delays, which limits timely monitoring. Furthermore, richer, more detailed sources, such as police records, usually remain confidential and are not publicly accessible.
This is the case in Italy, where the only reliable data source in this domain is the National Institute of Statistics (ISTAT)1. However, the information it provides is aggregated and typically becomes publicly available only at least one year after the incidents have occurred. This highlights the need to explore alternative sources of information that can complement or, in some cases, anticipate the evidence provided by institutional channels. One example of such a source is online crime news articles, which are often published in a timely manner when a crime occurs. This can help to reduce the information gap. Nevertheless, although Natural Language Processing (NLP) techniques make it possible to extract specific, detailed information about crimes, dealing with textual corpora is always dificult due to ambiguous sentences and diferent writing styles. It is also challenging to guarantee a proper characterization of domain knowledge.
To address this need, we developed a pipeline consisting of three main steps: web scraping on Italian
crime news articles, crime categorization using BERTopic [
From this pipeline and the obtained results, this paper aims to investigate some key questions allowing the evaluation of the efectiveness of the proposed approach. Specifically, in this paper we try to provide answers to the following Research Questions (RQs): RQ1: What is the most compliant evaluation scheme to use in the context of crime? RQ2: How efective is the model with respect to the diferent categories of entities? RQ3: What types of errors emerge in the identification of individual entities?
The remainder of the paper is organized as follows: Section 2 discusses diferent work related to the extraction of key information from crime cases; Section 3 describes the pipeline and its various stages; Section 4 discusses the performed experimental evaluation phase; and finally, Section 5 outlines conclusions about the work done and possible future developments.
This section addresses the problem of extracting key information from crime-related cases and explores the most common approaches used in this domain. Based on a review of the literature, three main categories of approaches are identified: • Rule-based approaches that rely on predefined syntactic and semantic rules to identify and classify information. They are accurate but lack adaptability to new or unexpected contexts. • Machine Learning (ML) based approaches that use statistical models trained on annotated data to recognize patterns and structures. They are more flexible than rule-based systems but require large amounts of labeled data. • Large Language Model (LLM) based approaches that leverage pre-trained language models (e.g., GPT or BERT) to understand and generate complex text. They are highly efective with unstructured data and complex scenarios.
Rule-based Approaches. These methods are efective for extracting basic information such as
location, date, and names from crime news articles. Rahma et al. [
Although precise, these approaches are limited in generalizing across linguistic patterns and require expert knowledge for the definition of rule.
Machine Learning Approaches. ML-based techniques often deal with the task as NER. Arulanandam
et al. [
These approaches ofer flexibility but struggle with context-dependent information and heavily rely on large annotated datasets.
LLM-Based Approaches. These methods ofer greater flexibility and generalization. Their subtypes
include Sequence Labeling, Extractive Question Answering (QA), and Instruction-Tuned Models.
In Sequence Labeling, each token in the input sequence is assigned to a label, enabling the model to
identify and extract structured information, such as named entities or metadata fields. Pongpaichet et
al. [
An Instruction-Tuned Model is a LLM that has been further trained on a dataset of instructions and corresponding outputs in order to improve its ability to follow instructions and perform specific tasks. Park et al. [11] used GPT-3.5 for prompt engineering to generate a labeled Korean crime dataset, then several fine-tuned models (e.g., KLUE-BERT, XLM-RoBERTa) use both token and sequence classification. They achieved an F1-score of 87%.
Italian Language Approaches. Although most of the studies are not focused on Italian, as described above, similar approaches have been used in other domains, such as the medical one. In fact, Crema et al. [12] developed the NLP Extraction and Management Tool (NEMT), a semiautomatic tool based on a QA bot fine-tuned on clinical QA datasets, achieving an F1-score of 84.7%. Buonocore et al. [13] proposed a BERT-based NER system to extract events from cardiology reports, using an IOB2-tagged dataset. They also included an extractive QA component for clinical registry tasks.
Nevertheless, there is the need of exploiting (semi-)automatic pipelines capable of properly analyze texts, extract information related to the specific domain of crime, and possibly relate the information extraction step with other proper variables, as in the case of the crime category. The proposed methodology represent one of the first strategies following this direction.
The proposed approach is based on an NLP pipeline, developed to extract structured information from Italian crime-related news articles (see Figure 1). The system is organized into three main phases: 1. Data collection and processing (Web scraping) that uses libraries like Requests2 and BeautifoulSoup3 to fetch and parse articles in order to extract useful information like the main text; 2. Topic Identification (Categorization) that uses the BERTopic model, trained on the input articles in order to generate diferent crimes categories; and 3. Information Extraction that, as main phase, aims to annotate the dataset for training of diferent BERT models and to extract domain related entities (i.e., through a NER process) through the spaCy4 library.
For demonstration purposes, the presented pipeline is customized on crime articles involving weapons.
Due to the lack of oficial and timely crime data, we considered online crime news, collecting them to compose a dataset that consists of roughly 4, 500 of italian articles published between January 2023
Requests, BeautifulSoup
Fetch using
Requests
Extract news text using BeautifoulSoup ONLINE NEWSPAPER
BERTopic Sentence splitting Generate emdeddings Dimensionality reduction
Clustering CountVectorizer c-TF-IDF
Fine-tune representations
Doccano, spaCy Use of weapons Manual revision Final dataset gpt-4o-mini labeling: CRIME LOCATION CRIME DATE CRIMA AUTHOR CRIME VICTIM WEAPON EMERGENCY BERT-baseitalian-cased BERT-basemultilingual xlm-RoBerta
RoBertaitalian
CNN-spaCy
Fine-tuning
Fires Assaults
Road accidents Use of weapons
Suicides
Water accidents and February 2025. The source of online newspaper was Virgilio Notizie website5 (see Figure 1 - Web scraping). The collected newspaper were processed to extract meaningful information, such as title, publication date, description, url, and main text. All these information was then saved in a JSON format for further analysis.
To group articles according to their content and categorize them in diferent crime categories, we
employed BERTopic [
Once the clusters were defined, the most representative terms for each topic were extracted using
the c-TF-IDF algorithm [
As shown in Figure 1, in the final phase, i.e., Information Extraction, the goal is to extract six specific types of entities from the crimes involving the use of weapons. The extracted entities are: author, victim, used weapon, location and date of the crime, and any involved emergency services. Since no annotated dataset for this domain was readily available, we adopted a semi-automatic annotation strategy. Initial annotations were produced with GPT-4o-mini and then manually reviewed and corrected by using the Doccano annotation tool10 (see Figure 2).
This Section presents the experiments performed on the diferent NER models evaluated in the proposed approach. The evaluation is organized according to three research questions (RQ1–RQ3). RQ1 investigates which evaluation schema best captures performance in this domain, recognizing that traditional metrics at token level may not be the best approach. RQ2 explores the model’s performance across diferent entity types to identify strengths and weaknesses in recognizing specific categories. Finally, RQ3 examines the nature and sources of detected errors, aiming to uncover persistent issues, such as class imbalance and context ambiguity that impact the model accuracy.
Technical Settings. We collected and semi-automatically annotated a dataset of around 100 samples, which has been used to fine-tune diferent BERT-based models on a NER task. The models are: BERTbase-italian-cased11, BERT-base-multilingual12, xlm-RoBerta13, RoBerta-italian14 and CNN-spaCy model. All models were fine-tuned for up to 80 epochs, with a learning rate of 5 × (10 −5 ), using Adam as optimizer, while the smart batching technique was used with a dynamic batch size. In addition, a strided-spans technique was used to handle texts that exceed the BERT context limit. Preliminary results. To evaluate and select the best model, we used the F1-score metric on the results obtained from the validation set. As shown in Figure 3, the BERT-base-italian-cased model demonstrated 10https://github.com/doccano/doccano 11https://huggingface.co/dbmdz/bert-base-italian-cased 12https://huggingface.co/google-bert/bert-base-multilingual-uncased 13https://huggingface.co/FacebookAI/xlm-roberta-base 14https://huggingface.co/osiria/roberta-base-italian the best performance compared to the other models, with an F1-score of 68%. Subsequently, the selected model was evaluated on an unseen test set to assess its generalization capability, achieving a strict F1-score of 65% (see Table 1).
RQ1. What is the most suitable evaluation schema for the crime domain? To identify the most appropriate scoring strategy, for evaluation schemas from SemEval 2013 were compared (see Table 1): • strict: exact string match and entity type; • exact: exact string match regardless of the type; • partial: partial string match regardless of the type; • ent_type: some overlap between the system tagged entity and the gold annotation. The results indicated that the ent_type schema is the most appropriate in the context of crime-related entity extraction, as it accounts for entity type correctness even when the span is not perfectly matched (Table 2 shows several examples of unrecognized matching through the strict evaluation schema). In fact, using the ent_type schema, the best model (i.e., BERT-base-italian-cased) achieved an overall F1-score of 74%, compared to 65% with the strict schema.
RQ2. How efective is the model with respect to the diferent categories of entities? The performance analysis was conducted per entity type (see Figure 4). The model showed high performance on specific categories, especially for weapon (F1-score: 93%) and crime date (F1-score: 87%). Intermediate performance was observed for emergency response (F1-score: 79%), victim (F1-score: 70%), and crime location (F1-score: 69%). The lowest result was obtained for the author entity, which reached only 43%, suggesting dificulties in detecting semantically ambiguous or context-dependent references to the author of the crime.
ent_type partial strict exact correct 116 103 101 103 incorrect 4 0 19 17 partial 0 17 0 0 missed 29 29 29 29 Incorrect type error analysis for the strict evaluation schema.
RQ3. What types of errors emerge in the entity recognition process? As shown in Table 3, an analysis process concerning the model-prediction errors revealed that missed entities (i.e., false negatives) for the author class and spurious entities (i.e., false positives) for the crime location class. This behavior is due to the lack of representative examples in the training data, particularly for the author class, which negatively impacted the model generalization. On the other hand, many false positives for the crime location entity were caused by the presence of secondary locations in the articles. These observations indicate that class imbalance and entity context disambiguation remain key challenges to address in the future.
Discussion.
The NER module demonstrated good generalization capabilities on some specific entities, such as weapon (F1-score 93%) and crime data (F1-score 87%), while it encountered greater dificulties on more ambiguous entities such as author (F1-score 43%), where semantic ambiguity, small errors in the annotation phase and the scarcity of representative examples had a negative impact on performance. Result distributions and performance values per entity-type following the ent_type schema. These results confirm both the validity of the adopted approach and the crucial importance of the quality of the annotated dataset.
In this paper, we propose an automated methodology for extracting key concepts from Italian-language crime news articles. The aim is to fill the information gaps left by institutional sources, which are often aggregated or accessed with a delay. To this end, we developed a pipeline consisting of three main modules: web scraping for collecting online news, categorizing news using topic modelling (BERTopic) and extracting structured entities through NER, based on fine-tuned BERT models. Despite the limited availability of annotated data (about 100 samples), the overall results were promising, with high performance on well-defined entities, such as weapon and data crime, and the presence of a few critical entities to recognize, such as the author. This confirms the efectiveness of the approach and highlights the importance of the quality of annotated datasets.
In the future, we would like to integrate the descibed NLP pipeline into a tool. It would allow the automatic identification of entities within texts and/or articles. Then, through a user parameterization schema, it would be possible to select the entities of interest and automatically generate structured datasets in order to be used to conduct further analyses in the domain of crime.
This study was funded by Ministero dell’Università e della Ricerca (MUR) of Italy in the context of project denoted as BLOODSTAIN in the program PRIN 2022 (grant number D53D23008660006).
[8] F. Rollo, L. Po, G. Bonisoli, et al., Online news event extraction for crime analysis., in: Italian
Symposium on Advanced Database Systems (SEBD), 2022, pp. 223–230. [9] Y. Cao, Y. Lan, F. Zhai, P. Li, 5w1h extraction with large language models, in: International Joint
Conference on Neural Networks (IJCNN), IEEE, 2024, pp. 1–8. [10] T. Sellam, D. Das, A. Parikh, BLEURT: Learning robust metrics for text generation, in: D. Jurafsky, J. Chai, N. Schluter, J. Tetreault (Eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2020, pp. 7881–7892. [11] Y. Park, R. S. Park, H. Kim, Key information extraction for crime investigation by hybrid classification model, Electronics 13 (2024) 1525. [12] C. Crema, F. Verde, P. Tiraboschi, C. Marra, A. Arighi, S. Fostinelli, G. M. Giufré, V. P. Dal Maschio, F. L’abbate, F. Solca, et al., Medical information extraction with nlp-powered qabots: A real-world scenario, IEEE Journal of Biomedical and Health Informatics 28 (2024) 6906–6917. [13] T. M. Buonocore, E. Parimbelli, V. Tibollo, C. Napolitano, S. Priori, R. Bellazzi, A rule-free approach for cardiological registry filling from italian clinical notes with question answering transformers, in: International Conference on Artificial Intelligence in Medicine, Springer, 2023, pp. 153–162. [14] L. McInnes, J. Healy, J. Melville, Umap: Uniform manifold approximation and projection for dimension reduction, arXiv (2018). [15] L. McInnes, J. Healy, S. Astels, et al., hdbscan: Hierarchical density based clustering., J. Open Source Softw. 2 (2017) 205.