Nowadays, the spread of disinformation poses a major challenge for society. Citizens find themselves immersed in a complex and data-saturated digital context that hinders their ability to critically discern between truthful information and disinformation. The task is complex because disinformation often appears in the form of rhetorical manipulations, logical fallacies, or distortions of the truth. This project aims to develop a system for detecting disinformation based on computational argumentation techniques and large language models, which promotes critical thinking and media literacy in society. The implemented web tool analyzes the patterns of human reasoning used in argumentation, classifying arguments into argumentation schemes defined by argumentation theory. After identifying the argumentation scheme, the system thoroughly examines the reasoning presented in the argument and uses a set of critical questions to question its validity. Using a large language model, enhanced with external contextualization from various information sources, the system is guided in the process of evaluating the truthfulness of the argument. The final response includes both a qualitative and quantitative justification of the level of truthfulness, providing links and references to the information sources used in the evaluation.
This project presents a practical application of computational argumentation in a specific domain of natural argumentation, such as the dissemination and exchange of information in social networks, media, and other communication channels, where the fight against disinformation is a complex and crucial task. In this context and recognizing that disinformation often involves logical fallacies and rhetorical manipulation, this work investigates how Articfiial Intelligence, particularly computational argumentation and large language models, can be leveraged to develop a system that enhances critical thinking and media literacy.
An extensive review of the current state of the art reveals that disinformation detection is highly complex
due to natural language’s inherent ambiguities, vagueness, enthymemes1, and dialectical variations,
which can easily lead to deception or fallacies. As a fundamental natural language processing (NLP) task,
disinformation detection involves analyzing text and speech. Before applying specialized Argument
Mining techniques, it is crucial to first address key NLP aspects such as text preprocessing, feature
extraction, and numerical representation, as discussed in [
However, the primary challenge in performing NLP tasks is preserving the order of words within
their original context, as losing this information can degrade model performance. Capturing long-term
dependencies is challenging, but sequential models like Recurrent Neural Networks (RNNs)[
Transformer-based models outperform traditional architectures like RNNs due to their attention mechanisms, which efectively capture long-range relationships without being constrained by word dependency length or complexity. Consequently, large language models (LLMs) based on Transformers represent the state-of-the-art in NLP tasks for disinformation detection. However, it has been observed that LLMs have certain limitations when applied to this area of study.
LLMs consist of millions of parameters, leading to high computational costs, substantial economic investment, and a significant environmental impact due to their carbon footprint. Although techniques like fine-tuning can refine these models, they still require powerful GPUs. Therefore, finding methods to reduce computational expenses without sacrificing performance is essential, making prompt engineering a promising approach.
These architectures often have dificulty retrieving specific information if it wasn’t part of their
training data, leading to "hallucinations," where the model generates information that seems plausible
but is actually incorrect. Furthermore, it becomes evident that these models lack a logical layer. In other
words, they do not consider the underlying logic that makes a text or argument truthful; instead, they
simply detect and classify content based on its structure without paying attention to the reasoning
involved, as seen in [
As outlined in [
A key concept used throughout this project is argumentation schemes [
The disinformation detection system2 comprises two interconnected modules (Figure 2), as follows: 2Link to the implemented web tool: http://desinformacion.gti-ia.upv.es/ • Module 1: "Classifier System in Argumentation Schemes" . This module takes a sentence (written in English) representing an argument as input and classifies it according to an argumentation scheme from argumentation theory. This module uses RASA3, an open-source framework for developing conversational systems. • Module 2: "Veracity Level Generator and Evaluator System". This module receives the output from Module 1. Based on the argumentation scheme and a set of critical questions, the system uses a LLM with external contextualization to finally provide a qualitative justification and a quantitative value indicating the argument’s level of truthfulness. The LLAMA4 family was investigated, with the LLAMA 3 70B model used as the LLM for this module.
The NLAS-MULTI corpus [
Initially, a single-layer classifier was implemented to identify the argumentation scheme of an input sentence from 19 possible classes. However, due to sub-optimal results (discussed in section 5.1), a two-layer classifier was adopted, as Figure 3 illustrates. This new approach, based on Walton’s theory, divides the 19 schemes into four groups, creating four classifiers in RASA, distributed across two layers. The first layer classifies the sentence into one of these groups, and the second layer activates the corresponding classifier to determine the specific argumentation scheme.
To implement these classifiers, the RASA NLU module based on intent detection was developed, with
each class representing an intent defined by multiple examples. A consistent configuration was used
across all four classifiers, including the default pipeline: the WhitespaceTokenizer [
In this module, the quantized LLAMA 3 model with 70 billion parameters is employed using an onpremise deployment of the model. The process begins with the creation of a client using the OpenAI 3https://rasa.com/ 4https://llama.meta.com/ Python library, enabling interaction with the model through chat completions. Prompt engineering is utilized to craft requests in JSON format, which instructs the model on the specific tasks to be performed.
As shown in Figure 3, two workflows are defined depending on the previous classification—either an argumentation scheme or a non-schema type. If the input sentence does not correspond to any argumentation scheme, a context search is conducted using the sentence’s inherent information. On the contrary, if the sentence aligns with an argumentation scheme, critical questions are generated and tailored to the content of the sentence. A parallel context search then retrieves information from three external sources (Google, Wikipedia, and Bing) for each critical question. Finally, with the gathered context, the model addresses the defined critical questions and synthesizes all collected information to provide a structured evaluation of the input argument’s veracity, acting as an expert assistant in computational argumentation.
In Module 1, a comparison of the unilayer and bilayer architectures reveals that the two-layer model consistently outperforms the single-layer model across all metrics, including accuracy, recall, precision, and F1-score, achieving values between 85% and 87% (Figure 4). The primary issue with the unilayer approach was the error introduced by the non-schema class. This is evident from the confusion matrix (Figure 5), where a prominent column in the non-schema class area indicates a tendency of the classifier to misclassify arguments into this category. The two-layer classifier addresses this problem efectively, reducing the noise associated with the non-schema class and improving the precision for this class ("group0") from 20.19% to 72.22%, as shown in Figure 4.
To evaluate the performance of module 2, a survey was conducted with the answers generated by the system for 20 examples of arguments, covering the 19 classes of argumentation schemes. Responses from a total of 80 adults between the ages of 18 and 65 were collected and analyzed. Each respondent evaluated three specific aspects for each example: the quantitative justification of the level of truthfulness, the qualitative justification of the level of truthfulness, and the adequacy of the sources.
Figure 6 illustrates that all aspects of the system received similarly positive ratings. In terms of overall satisfaction (Figure 6), a significant majority (83.8%) rated the system’s responses as satisfactory with a high degree of contentment, indicating consistent coherence across all areas.
In conclusion, this work successfully developed a tool for detecting disinformation in societal arguments. This research has met its objectives through the creation of a web-based tool comprising two interconnected modules that integrate computational argumentation techniques with LLMs.
Looking ahead, for scaling the system, future work could involve integrating intelligent agents. Implementing domain-specific expert agents or agents with access to diverse information sources could enhance the system’s final responses through the use of advanced technologies and collaborative approaches. Additionally, the veracity of the sources used to generate context should also be studied.