The utilisation of Artificial Intelligence in investigative contexts facilitates the optimisation of the analysis of digital artefacts, which can be collected in a variety of formats, including text, printouts, images, and videos, as well as from diverse sources, such as datasets, networks, and devices. More recently, while the adoption of LLM (Large Language Models) appears to hold significant potentials, it is also accompanied by emerging concerns relating to its technological limitations, including distortions and hallucinations, which can hinder law enforcement and compromise the balance between the rights of the accused and the power of the judicial authorities. Notwithstanding the activities prohibited by Art. 5 of the AI ACT, the fact is that LLM technologies are already assuming a pivotal role in investigations, yet are still inadequate the principles, methodologies and tools allowing courts to assess accuracy and precision of the language models before implementing them. The objective of this research project is to develop a methodology that can be adopted for the assessment and comparison of LLMs in performing analytical processes. To this end, a synthetic dataset of electronic evidence is generated from a fictitious criminal scenario and fed to the agent with engineered prompting techniques.
model [
On this regard, it is important to remember that international standards have been developed for the sharing of digital evidence, as well as legal ontologies specifically dedicated to this sector, such as CASE1, which contribute to spreading the adoption of AI in the forensic field and increasing the level of reliability.
However, if it is widely accepted that “data” and “algorithms” are two sides of the same coin, which we can generically name "information", their “quality” must be assessed using diferent criteria. First of all – but we will see this later on – very often assessments can only be made indirectly, because the technologies used do not ofer transparency in their processing. Secondly, it should be noted that AI is, on the one hand, an “object”, similar to the digital data being observed, and on the other hand, an “agent”, like the human being who observes it. The automatism of the activity that AI is called upon to perform reveals not only the essence of its instrumentality – the fact that it is used to make data processing more eficient – but also the seed of issues concerning the balance between the operation of the artificial agent and the surveillance of the human being. In other words, AI can perform its tasks with "varying degrees of autonomy" – as also found in the regulatory definition of Art. 3(1) of the “AI ACT” Regulation2 – and, consequently, external expectations – in terms of “quality”, which is what interests us most specifically here – depend on a very heterogeneous set of factors, including, in particular, the level of “trustworthiness” established by the human being with whom it interacts or in the reference context. In this sense, it becomes crucial to establish criteria for determining also the "quality" of algorithms.
In this work, we intend to present a research proposal that addresses the "Information Quality" in the implementation of Artificial Intelligence in Digital Forensics, with the aim of establishing an AIQA (Artificial Intelligence Quality Assessment) framework. To this end, we created a pilot experiment in which a "criminal scenario" is created based on oficial reports, and it is used to generate a synthetic dataset, which has been fed to LLM agents in order to assess their performances according to given benchmarks.
Recent contributions have explored the possibility of using LLMs in the forensic field for purposes other
than strictly evidentiary or investigative ones, especially for for training and educational purposes [
The narrative used as a starting point for this work refers to online fraud. This is a constantly growing
phenomenon in which fraud is perpetrated by ofering the opportunity to invest in funds or virtual
currencies [
Fraudulent activities are perpetrated using a variety of methods, such as fake phone numbers (phone spoofing), forged identity documents or professional qualification certificates (e.g. “financial advisor”), sometimes issued by non-existent organizations, as well as web browsing techniques that allow them to mask their IP connection while interacting with their victims and each other(TOR, VPN). In addition, victims are often persuaded to grant remote access to their "consultants" to their computers, allowing them to transfer funds overseas very easily. In similar cases, law enforcement oficers are expected to perform several tasks, such as: 1. Forensic acquisition of digital evidence. Cloning the victim’s devices, acquiring chats, emails, screenshots and files), while preserving data in accordance with international forensic standards (e.g. ISO/IEC 27037)4. 2. Verification of the integrity of communications. Extracting and decoding data, verifying links to the fraudulent platform, analysing metadata and recovering deleted items). 3. Tracing cryptocurrency flows. Following the path of cryptocurrency funds and identify any wallets linked to exchanges or mixers, using blockchain forensic tools and specialised software (e.g. Chainalysis, Elliptic).
4. OSINT. Linking suspicious wallets, web domains, and digital identities.
The fictitious criminal organization we created is composed of four individuals located in the European Union (Italy, Austria, Croatia, Slovenia) with high operational capabilities and advanced technological skills, capable of creating shell companies and counterfeited ID cards, or to use real documents stolen from previous victims, managing spoofing techniques (both for emails and for telephone numbers), as well as of setting up fraudulent websites, producing "deep fakes", and generating international financial lfows through digital platforms and cryptocurrencies. They supposedly operated in a specific time period (from 1 January 2024 to 30 June 2025), luring one hundred victims and recruiting five “straw men” – some unaware, others accomplices – spread in their respective Countries, generating a total revenue of at least five million euros.
The LLM was instructed to generate various kinds of digital artefacts produced by this activity, i.e. records of telephone conversations, texts (in diferent languages), and metadata from electronic communications (among criminals, or towards victims and with "straw men"), as well as financial transactions.
In creating the dataset, some simplifications were introduced to optimize Prompt and output lengths in accordance with the limitations of current LLM providers: telephone numbers are not falsified; the IP addresses of the connections are not masked; it was assumed that the exchanges complied with KYC (Know Your Customer) regulation and cooperated with the police authorities.
The prompt used, the dataset containing the “synthetic data” generated by the LLM and the outcomes are published in open format on a freely accessible dataset5.
As shown in Figure 1, our workflow can be divided into three phases: (1) Dataset preparation; (2) LLMs testing; (3) LLMs benchmarking. In the following paragraphs, we ofer a few clarifications for each of them. 3https://www.milanofinanza.it/news/trufe-online-falsi-video-di-giorgia-meloni-promettono-un-guadagno-di-50-000-eurocon-un-investimento-202412121252369462. 4https://www.iso.org/standard/44381.html. 5the LLM adopted was ChatGPT5, the dataset is available at https://zenodo.org/records/16927274.
In the training phase, the choice of examples that are as realistic as possible to provide to the model is a critical aspect, since the quality of the outcome will depend on how the data are selected, classified, labeled, and processed. Biases can also present themselves in the forensic field and are very dangerous.
The methodology proposed here takes into account that inputs can be biased, and tries to reduce such risks by adopting separate measures for each step.
1. Criminal scenario design. As explained above, we created a simulation. In other worlds, our intention is not to incriminate real individuals using predictive policing approaches but to generate fictitious criminals enacting consolidated strategies. 2. Prompt design. The purpose is to generate a synthetic data set that can represent - as a whole the narrative behind the data. 3. LLM1 prompting. Using an AI tool (preferably diferent from the one being evaluated), synthetic data are generated. For example, for an investigation into a cryptocurrency scam conducted via email and WhatsApp, sequences of messages will be produced that are consistent, appropriately distributed over time, and similar to those actually received by real victims. 4. Criminal scenario data set. The data set based on the criminal scenario designed. Being ifctitious, it is by design anonymous. 5. Reference data set. This data set contains public domain, hence neutral, references6. This ensures also that the format of the evidences generated by the agent (e.g. bank account reports, telephone call logs, Whatspp messages) is consistent with that used in real-life law enforcement (e.g. ETSI Charging Data Records). 6. Synthetic data set. The synthetic data is added to the reference dataset to construct what we will call the synthetic dataset. This represents a realistic and consistent example of a set of data that could be found on one or more devices belonging to a possible perpetrator.
Detection, classification and analytical functions are applied to our datasets by the LLM to be evaluated, and the results obtained are compared with those expected (i.e. with the information available a priori). 1. Prompt design. By applying appropriate metrics, the reliability of the tool in the specific context is then determined objectively. 2. LLM2 - LLMn prompting. By applying the same procedure to diferent LLMs, it is possible to compare them, to evaluate the best performer, or to obtain confidence indices with which to measure the results, or even to deploy multiple LLMs for the same case. 3. LLM2-LLMn extracted criminal dataset. This sequence of operations can be repeated with diferent synthetic datasets to average the measurements and obtain a higher level of confidence.
Inevitably, discrepancies will emerge between the results obtained and the original synthetic data; therefore, the metrics described below are applied to quantitatively measure the performance of the tool in the context under consideration.
To define the metrics, it is necessary to consider the objectives of using the AI tool. Since, in the course of judicial investigations, law enforcers cannot avoid directly analysing the original data, the AI tool can be used to identify data of interest within device memories, which may contain – as typically do – amounts of information that make manual searching and selection dificult or impossible.
The metrics for evaluating the efectiveness of the AI tool in question must be defined in such a way as to measure how close the automatic processing is to the ideal result.
Some criteria for their definition are as follows:
1. Eficiency. For each type of message, the ratio between the number of relevant messages found and the total number of messages of the same type in the synthetic data. 2. False negatives. For each type of message, the ratio between the number of relevant messages not found and the total number of messages of the same type in the synthetic data. 3. False positives. For each type of message, the ratio between the number of irrelevant messages classified as relevant and the number of messages of the same type in the summary data. 4. Summary quality. The LLM can provide a summary of the content of messages identified as relevant and recognise regular patterns, timing, operating modes, etc. The evaluation in this case will be represented by a score, for example obtained as the average of the scores assigned by the participants in the evaluation.
This methodology does not aim in solving the explainability issues, but just to ofer a tool for assessing the trustworthiness of artificial agents. We believe that AIQA, possibly in synergy with IQA and sector ontologies such as CASE, can be a useful tool with a variety of possible applications. For example, it could be possible not only to train a single artificial agent – optimising its model – but also to compare the quality of diferent agents used within a proceeding – for example, those of the prosecution, the defence and the civil party – in an easily understandable and communicable way.
During the preparation of this work, the authors employed Generative AI tools (ChatGPT 5.1 Plus, Overleaf Writefull and Deepl) in supporting text polishing and translation. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.