We describe the fourth edition of the CheckThat! Lab, part of the 2021 Conference and Labs of the Evaluation Forum (CLEF). The lab evaluates technology supporting three tasks related to factuality, and it covers Arabic, Bulgarian, English, Spanish, and Turkish. Here, we present the task 2, which asks to detect previously fact-checked claims (in two languages). A total of four teams participated in this task, submitted a total of sixteen runs, and most submissions managed to achieve sizable improvements over the baselines using transformer based models such as BERT, RoBERTa. In this paper, we describe the process of data collection and the task setup, including the evaluation measures used, and we give a brief overview of the participating systems. Last but not least, we release to the research community all datasets from the lab as well as the evaluation scripts, which should enable further research in detecting previously fact-checked claims.
such systems sufer from credibility issues. Hence, it is important to reduce the manual efort
by detecting when a claim has already been fact-checked. Work in this direction includes [
To deal with this problem, we launched the CheckThat! Lab, which features a number of tasks aiming to help automate the fact-checking process and to reduce the spread of disinformation and misinformation. The CheckThat! lab2 was run for the fourth time in the framework of CLEF 2021. The purpose of the 2021 edition of the lab was to foster the development of technology that would enable finding check-worthy claims, finding claims that have been previously fact-checked, and predicting the veracity of a news article and its topic. Thus, the lab focuses on three types of content: (i) tweets, (ii) political debates and speeches, and (iii) news articles.
In this paper, we describe in detail the second task, detecting previously fact-checked claims, of the CheckThat! lab tasks.3 Figure 1 shows the full CheckThat! identification and verification pipeline, including the tasks on detecting check-worthy claims, detecting previously factchecked claims, and veracity and topic detection of news articles. The second task is defined as follows: “given a check-worthy input claim and a set of verified claims, rank the previously verified claims in order of usefulness to fact-check the input claim.” It consists of the following two subtasks:
Subtask 2A: Detecting previously fact-checked claims in tweets. Given a tweet, detect whether the claim it makes was previously fact-checked with respect to a collection of fact-checked claims. This is a ranking task, ofered in Arabic and English, where the systems need to return a list of top- candidates.
Subtask 2B: Detecting previously fact-checked claims in political debates or speeches.
participants were free to work on any language(s) of their interest, and they could also use multilingual approaches that make use of all datasets for training. Subtask 2A attracted four teams, and the most successful approaches used transformers or a combination of embeddings, manually engineered features, and neural networks. Section 3 ofers more details.
the main data source. The task attracted three teams, and a combination of transformers, prepossessing, and augmentation approaches performed the best. Section 4 gives more details.
11, 12]. This includes datasets [
annotation setup and one of their datasets: PolitiFact. Previous work has mentioned the task as
an integral step of an end-to-end automated fact-checking pipeline, but there was very little
detail provided about this component and it was not evaluated [
search a number of fact-checking websites. However, the tool cannot handle a complex claim, as it uses the standard Google search funcionality, which is not optimized for semantic matching of long claims.
from multiple sources, organized into a knowledge graph (KG). The system can perform data exploration, e.g., it can find all claims that contain a certain named entity or keyphrase. In contrast, we are interested in detecting whether a claim was previously fact-checked.
[
checked with respect to a collection of fact-checked claims. The task is ofered in Arabic and
3.1. Dataset
AraFacts [
check that the selected tweets were compliant with the guidelines. They further resolved any disagreements by discussing the reasons behind their choice, and excluded the claims where the disagreement remains. We ended up with 858 tweet–VerClaim pairs.
platforms, and since a claim can be verified by multiple sources, we had to check whether the annotated tweets can be paired with more than one claim from our verified claims collection.
verified by multiple sources. For each given verified claim, we selected all the claims that had a
any non-similar claims. Given similar claims to the ones in our qrels (query relevances), we constructed new tweet–VerClaim pairs.
To further verify any missing similar claims, we used Pyserini [
3.2. Evaluation
Mean Average Precision (MAP), reciprocal rank, Precision@ ( @) and MAP@ for P t1, 3, 5, 10, 20, 30u. We used MAP@5 as the oficial evaluation measure.
Sen. Mitch McConnell: “As recently as October, now-President Biden said you can’t legislate by executive action unless you are a dictator. Well, in one week, he signed more than 30 unilateral actions.” pic.twitter.com/PYQKe9Geez — Forbes (@Forbes) January 28, 2021 When he was still a candidate for the presidency in October 2020, U.S. President Joe Biden said, “You can’t legislate by executive order unless you’re a dictator.” Photographs you post on Snapchat can now be used as evidence in legal cases unless you opt out.
U.S. Sen. Mitch McConnell said he would not participate in 2020 election debates that include female moderators.
U.S. Sen. Majority Leader Mitch McConnell said that U.S. President Trump "provoked" the attack on the Capitol.
President Joe Biden signed an executive order in 2021 allowing the U.S. to fund abortions abroad.
A supporter of President Donald Trump carries a Confederate battle flag on the second floor of the U.S. Capitol near the entrance to the Senate after breaching security defenses, in Washington, January 6, 2021.
Photo by Mike Theiler pic.twitter.com/pbhwfAVsUX — corinne_perkins (@corinne_perkins) January 6, 2021 In January 2021, Hillary Clinton suggested U.S. President Donald Trump spoke by phone with Vladimir Putin on the day of an attack on the U.S.
Capitol, Jan. 6, 2021.
In January 2021, OnlyFans removed Donald Trump’s account in the aftermath of the Jan. 6 attack on the U.S. Capitol.
A Confederate flag was spotted inside and outside the U.S. Capitol as a pro-Trump mob stormed the building.
A pro-Trump mob chanted “Hang Mike Pence” as they stormed the U.S.
Capitol on Jan. 6, 2021.
Kevin Seefried, who carried a Confederate flag into the U.S. Capitol during the attack on the building in January 2021, is registered as a Democrat in Delaware. ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗ ✗ 3.3. Overview of the Systems
it to predict the relevance score for a given tweet–VerClaim pair. The fine-tuned model was used to re-rank the candidate claims based on the predicted relevance scores. Verified claims (to match against)
re-ranking LambdaMART model. The system is evaluated on the English version of the dataset collected from tweets.
ering the problem as a ranking task.
the cosine similarity for each pair of an input tweet and a verified claim. The prediction was made by passing a sorted list of cosine similarities to a neural network. 3.4. Results
Arabic A single system was submitted for this task by the bigIR team. They used AraBERT to re-rank a list of candidates retrieved by a BM25 model. They first constructed a balanced training dataset where the positive examples correspond to the query relevance (qrels) provided by the organizers, while the negative examples were selected from the top retrieved candidates by BM25 such that they are not already labeled as positive. Second, they fine-tuned AraBERT to predict the relevance score for a given tweet–VerClaim pair. They added two neural network layers on top of AraBERT to perform the classification. Finally, at inference time, they used BM25 to retrieve the top 20 candidate verified-claims. Then, they fed each tweet–VerClaim pair to the ifne-tuned model to obtain a relevance score and to re-rank the candidate claims accordingly.
0.910 0.799 English Three teams participated for English, submitting a total of ten runs. All of them managed to improve over the Elastic Search (ES) baseline by a large margin. Team Aschern performed best; they used TF.IDF, fine-tuned pre-trained sentence-BERT, and LambdaMART for re-ranking, and scored 13.4 (MAP@5) points above the baseline. The second-best system was submitted by the NLytics team, which fine-tuned RoBERTa, improving by 5 (MAP@5) points absolute over the baseline. 4. Subtask 2B: Detecting Previously Fact-Checked Claims in
Political Debates or Speeches
been previously fact-checked with respect to a collection of previously fact-checked claims.
4.1. Dataset 4.2. Evaluation
claims match more than one verified claim) in a collection of 19,250 verified claims in PolitiFact.
0.339 0.313 0.308 0.210
MAP 0.351 0.338 0.332 0.219 0.304 0.266 0.253 0.165
Precision 4.3. Overview of the Systems
for all claims (input and verified), then computed a cosine similarity for each pair of an input claim and a verified claim. Finally, they made a prediction by passing a sorted list of cosine similarities to a neural network.
used the scores predicted by that model along with BM25 scores as features to train a rankSVM re-ranker. They further studied the impact of applying online mining of triplets, and they performed some experiments to augment the dataset automatically.
treating the problem as a ranking task. 4.4. Results
on detecting previously fact-checked claims in tweets (Subtask 2A), and in political debates or speeches (Subtask 2B). Inline with the general mission of CLEF, we promoted multi-linguality by ofering the task in two diferent languages: Arabic and English. The participating systems ifne-tuned transformer models (such as BERT and RoBERTa) and some tried data augmentation.
11S-1204-170060 from the Qatar National Research Fund (a member of Qatar Foundation). The work of Fatima Haouari was supported by GSRA grant #GSRA6-1-0611-19074 from the Qatar