This paper addresses the lack of automated contradiction detection systems for the Spanish language. The ES-Contradiction dataset was created and contains examples with two pieces of information classiifed as Compatible, Contradiction, or Unrelated. To the author's knowledge, a Spanish-language contradiction dataset is non-existent and therefore, the ES-Contradiction dataset fills an important research gap, given Spanish being one of the most widely spoken languages. Moreover, the dataset built includes a fine-grained annotation of the diferent types of contradictions in the dataset. A baseline system was designed to validate the efectiveness of the dataset. The BETO transformer model was used to build this baseline system, which obtained a good result to detect the three class labels Compatible, Contradiction, or Unrelated.
A huge amount of fake news is generated and distributed by digital media every day. Hence,
the manual evaluation of its veracity is practically impossible in a reasonable time frame [
Fake news has existed for a long time [
For this purpose, a deep search for resources was carried out on contradiction detection
from which we concluded that most of the resources and systems for contradiction detection
are developed in English [
The research presented in this paper is focused on contradiction detection by using computational models.
The most common approaches to contradiction detection in texts use the linguistic features
extracted from texts to build a classifier by training from the annotated examples, such as
the works in [
Currently, the availability of large annotated datasets for contradiction detection are mainly
present in English [
There are a few studies that address the detection of contradictions in languages other than
English, such as:
1. Machine translation of SNLI from English into German. A model was built using the
German version of SNLI and the results of the predictions are very similar to the same
model trained on the original SNLI version in English [
Our dataset (ES-Contradiction) is focused on contradictions that are likely to appear in traditional
news items written in the Spanish language. Unlike other datasets, for the dataset proposed in
this work, contradictions are annotated by distinguishing the type of contradiction according
to its specific characteristics [
In order to create the ES-Contradiction dataset, news articles from a renowned Spanish source
were automatically collected, including the headline and body text. According to the journalistic
structure of a news item, the headline is the title of the news article, and it provides the main
idea of the story. A headline is expected to be as efective as possible, without losing accuracy
or becoming misleading [
The ES-Contradiction dataset is annotated according to the following classes: • Compatible (two pieces of texts that address the same fact contain compatible information at the same time frame) • Contradiction (two texts that address the same fact contain incompatible information or two texts that address antonym facts contain compatible information at the same time) • Unrelated (two texts that address diferent facts)
The dataset was built in four stages, subsequently outlined and detailed: 1. Extracting information from data source: The headline, body text, and date of the news item are extracted from a reliable data source. In this case, the news agency EFE 2. 1 . The news are extracted assuming that the headlines and body texts are compatible, although in the third stage this relationship is verified. 3. Modifying news headlines: The aim of this stage is to make the news headline contradictory to the body text by including simple alterations to the headline structure. These alterations will change the semantic content of the sentence, making it contradictory to the previous headline and body text. The changes to the headline together with some examples are given as follows: • NEGATION (Con_Neg): This alteration consists of negating the headline of the news item.
a) Original headline: “El Gobierno pide permanecer en los domicilios porque la situación es grave” (“The government asks people to stay home as the situation is serious”) b) Modified headline: “El Gobierno no pide permanecer en los domicilios porque la situación no es grave” (“The government doesn’t ask people to stay home as the situation isn’t serious”) • ANTONYM (Con_Ant): This transformation consists of replacing the verb denoting the main event of the headline with an antonym.
a) Original headline: “Las licencias VTC aumentan un 19% en lo que va de año” (“VTC licences increase by 19% this year”) b) Modified headline: “Las licencias VTC caen un 19% en lo que va de año” (“VTC licences fall by 19% this year”) 1https://www.efe.com/efe/espana/1, accessed on 15 June 2021 • NUMERIC (Con_Num): This amendment consists of changing numbers, dates, or times appearing in the headline.
a) Original headline: “Accionistas de Nissan aprueban la incorporación de cuatro nuevos consejeros” (“Nissan shareholders approve the appointment of four new board directors”) b) Modified headline: “Accionistas de Nissan aprueban la incorporación de diez nuevos consejeros” (“Nissan shareholders approve the appointment of ten new board directors”’) • STRUCTURE (Con_Str): This modification consists of changing the position of one word for another or substituting words in the sentence.
a) Original headline: “Tokio avanza optimista por el diálogo entre EEUU y China” (“Tokyo is optimistic about the dialogue between USA and China”) b) Modified headline: “Tokio avanza optimista por el diálogo entre EEUU y UE” (“Tokyo is optimistic about the dialogue between USA and EU”) 4. Classifying the relationship between the headline and the body text: The semantic relationship between the headline and the body text was annotated in two phases: The first phase consisted of classifying the information into Compatible (compatible information) or Contradiction (contradictory information). In the second phase, in the case of Contradiction, the type of contradiction was also annotated (Negation, Antonym, Numeric, Structure). 5. Aleatory mixing headline and body text: The news items reserved in the first stage were used to generate unrelated examples (Unrelated). The headline was separated from the corresponding body text and all the headlines were randomly mixed with the body texts. In the mixing process, it was verified that the headline is not mixed with the corresponding body text. This step was done automatically without the intervention of the annotators.
The dataset consists of 7403 news items, of which 2431 contain Compatible headline–body news items, 2473 contain Contradiction headline–body news items, and 2499 are Unrelated headline–body news items. This represents a balanced dataset with three main classification items. The dataset split sizes for each annotated class are presented in Table 1. We partitioned the annotated news items into training and test sets. The dataset is available at Zenodo2.
As can be seen in Table 2, our dataset contains examples of each type of contradiction. However, it is important to clarify that there are few examples of structure contradiction, given the complexity of finding sentences that allow for this type of modification.
Due to the particularities of the dataset annotation process, it was necessary to validate the second and third stages of the process. For the second stage, a super-annotator validation 1755 744 2499 was conducted, while for the third stage, an inter-annotator agreement was carried out. We randomly selected 4% of the Compatible and Contradiction pairs (n = 200) to carry out the dataset validations.
For the second stage, it was not possible to make an inter-annotator agreement because this stage consists of headline modifications and the possible variations are infinite. In this case, a manual review of the modified headlines is performed by the Super-Annotator to detect inconsistencies with the indications in the annotation guide. Only 2% of the analyzed examples present inconsistencies with the annotation guide, corroborating the validity of this stage.
In order to measure the quality of the third stage annotation, an inter-annotator agreement
between two annotators was performed. In cases where there was no agreement, a consensus
process was carried out among the annotators. Using Cohen’s [
To test the validity of the newly created Spanish contradiction dataset in this task, a baseline
system was created that is based on the BETO3 model described in [
The main objective of the experimentation proposed in this research is to demonstrate that a model is able to learn how to automatically detect contradiction types and contradictions with high accuracy from the ES-Contradiction dataset.
This experiment is performed on the entire dataset to predict Compatible, Contradiction, or Unrelated for each example. The system created is capable of detecting the Unrelated class with a high level of precision and achieves significantly good results in the Compatible and Contradiction classes. Table 3 presents the results.
The results obtained in the Unrelated class indicate that the system is capable of detecting
with excellent 1 these types of examples, corroborating the results obtained in the literature
on this type of semantic relation between texts [
In this experiment, the Unrelated class is removed from the ES-Contradiction dataset to measure the accuracy of the approach in terms of distinguishing between compatible or contradiction information, assuming that the information is related. The results are shown in Table 4. The approach obtains similar results in both predicted classes. This is due to the quality of the training examples and the balanced number of examples from each class in this dataset.
This experiment aims to analyze the detection capability of the approach by contradiction types. Table 5 shows the results obtained exclusively for the detection of contradiction types.
The structural Contradiction class (Con_Str) is the one that obtains the lowest accuracy results
and 1. This contradiction type is considered one of the most complicated contradictions to
detect compared with the other contradictions [
The experiments carried out have validated the efectiveness of a contradiction detection system using the ES-Contradiction dataset. However, the experiments evidenced one of the deficiencies in the ES-Contradiction dataset namely, the lack of examples of the Con_Str class. It would also be interesting to include other types of contradictions that are not being taken into account in this dataset.
For the purpose of improving the results of the Contradiction class, we can test by including
resources that detect antonyms and synonyms in line with [
It is highly likely that contradictions such as the structure contradiction need external
semantic knowledge to improve detection results, similar to the introduction of SRL [
This work has built the ES-Contradiction dataset, a new Spanish language dataset that contains Contradiction, Compatible, and Unrelated information. Unlike other datasets, in the ESContradiction dataset, contradictions are annotated with a fine-grained annotation. We used the BETO model to create an automated contradiction detection system.
The results obtained by our system show that the created Spanish contradictions dataset is a good option for generating a language model that is able to detect contradictions in the Spanish language. In order to create a powerful contradiction detection system in Spanish, it is necessary to extend our dataset with other types of contradictions and add specific features.
Finally, the creation of this dataset will make it possible to validate the efectiveness of a contradiction detection architecture in the Spanish language that will be created in future works.
This research work has been partially funded by Generalitat Valenciana through project “SIIA: Tecnologias del lenguaje humano para una sociedad inclusiva, igualitaria, y accesible” with grant reference PROMETEU/2018/089, by the Spanish Government through project RTI2018-094653B-C22: “Modelang: Modeling the behavior of digital entities by Human Language Technologies”, as well as being partially supported by a grant from the Fondo Europeo de Desarrollo Regional (FEDER) and the LIVING-LANG project (RTI2018-094653-B-C21) from the Spanish Government.