This extended abstract presents results from two recent studies [1, 2] aimed at enhancing the practical application and efectiveness of fact-checking systems. La Barbera et al. [ 1] detail the implementation of crowdsourcing in fact-checking, demonstrating its practical viability through experimental evaluation using a dataset of political public statements. Zeng et al. [2] build on this foundation by integrating crowdsourced data with Large Language Models, proposing the first hybrid system that combines human insights and AI capabilities.
1. Introduction
The rapid proliferation of misinformation across digital platforms poses significant challenges
to societal trust and public safety. Traditional fact-checking methods, predominantly based
on experts, are unable to cope with the ever-increasing volume and speed of misinformation
dissemination. This has created interest in the development of more scalable solutions that are
able to enhance the accuracy and eficiency of misinformation detection methods, addressing
the problem at scale. Crowdsourcing has emerged as a powerful tool in this domain [
In this work, we present results from two recent studies in the field of crowdsourced
factchecking [
Mostly True True False True
2. Crowdsourced Fact-checking: Does It Actually Work?
Objective and Methodology. The primary objective of La Barbera et al. [
Results. This study highlighted the efectiveness of crowdsourcing for fact-checking,
demonstrating higher efectiveness when compared to previous studies [
S6 Hard Vote
Soft Vote
Mean Median
Meta Vote
Model First
Hard Vote
Soft Vote
Mean Median
Meta Vote
Model First
First
S6 3.0 2.5 2.0 SEM 1.5 1.0
Discussion. The results of La Barbera et al. [
The notable diference in La Barbera et al. [
Hybrid Human-AI Misinformation Detection
Objective and Methodology. The primary objective of Zeng et al. [
To integrate human and AI insights, several combination strategies were explored, including simple averaging, weighted averaging based on confidence scores, and more complex ensemble methods tailored to the specific characteristics of the data. The performance of the hybrid system was evaluated using standard metrics like accuracy, precision, recall, and F1-score, supplemented by detailed error analysis to fine-tune the integration approach and ensure high-quality fact-checking.
Results. The integration of crowdsourced data with LLMs provided significant insights into the potential of hybrid systems for misinformation detection. The performance was quantitatively evaluated using two scales, S2 and S6, reflecting simpler and more complex judgment scales, respectively.
Figure 2 provide a detailed comparison of accuracy and Mean Squared Error (MSE) across individual models, crowdsourced data, and their integration. For the S2 scale, the results demonstrate consistent model performance with an accuracy around 0.7. In contrast, crowdsourced judgments maintained a higher accuracy of 0.816, surpassing individual model performances across all aggregation methods except soft-voting. For the S6 scale, the models achieved the best accuracy using hard-voting, soft-voting, and median aggregation methods (up to 0.441). The most efective hybrid combination on the S2 scale was achieved using the Meta Vote method, which delivered an outstanding accuracy of 0.875 and the lowest error rates (MSE 0.125). For the S6 scale, the median aggregation provided the best accuracy (0.441), indicating that simpler aggregation methods might be more efective for tasks with a more fine-grained scale.
Further analysis involved evaluating classification diferences across various truthfulness
levels using confusion matrices (not shown), which revealed that models were more consistent in
classifying middle scale values like Mostly-False, Half-True, and Mostly-True but struggled with
extreme categories such as Pants-On-Fire and True. Conversely, crowdsourced data showed
a better capability to identify these extremes, indicating a deeper contextual understanding
of the statements. The hybrid combinations provided more balanced judgments across the
truthfulness spectrum of the S6 scale. While these combinations did not always outperform
other methods in terms of raw accuracy or error rates, they ofered a more subtle and robust
classification, crucial for tasks requiring a sophisticated understanding of truthfulness.
Discussion. The integration of crowdsourced data with LLMs proposed by Zeng et al. [
Zeng et al. [
Acknowledgments. This research is partially supported by the European Union’s NextGenerationEU PNRR M4.C2.1.1 – PRIN 2022 project “20227F2ZN3 MoT–The Measure of Truth: An Evaluation-Centered Machine-Human Hybrid Framework for Assessing Information Truthfulness” - 20227F2ZN3_001 – CUP G53D23002800006, and by the Strategic Plan of the University of Udine–Interdepartmental Project on Artificial Intelligence (2020-25).