In this paper, we present our journey in exploring the use of crowdsourcing for fact-checking. We discuss our early experiments aimed towards the identification of the best possible setting for misinformation assessment using crowdsourcing. Our results indicate that the crowd can efectively address misinformation at scale, showing some degree of correlation with experts. We also highlight the influence of worker background on the quality of truthfulness assessments.
Fact-checking is crucial as online misinformation erodes trust in traditional sources, leading
to real-world consequences. Experts’ capacity to keep up with social media’s information
overload is challenged, requiring eficient, scalable alternatives to mitigate misinformation. One
promising approach in this direction is the use of crowdsourcing, which taps into the collective
intelligence of a large group of people, and has been successfully used in various domains [
This paper summarizes our research on using crowdsourcing for fact-checking [
Crowdsourcing has been efective in various information credibility research contexts, such as
news quality evaluation [
It is thus reasonable to hypothesize good performances of the wisdom of the crowd approach also when assessing the truthfulness of given information items. In theory, crowdsourcing ofers several advantages when it comes to fact-checking and identifying misinformation. Firstly, it allows for rapid and cost-efective evaluation of a large volume of information, making it scalable. Secondly, the diversity of the worker pool can provide multiple perspectives for a comprehensive assessment. Finally, the use of crowdsourcing can enhance transparency by relying on input from multiple individuals, rather than a single expert’s opinion. To experimentally verify whether a crowd of non-expert human judges can detect and objectively categorize online (mis)information thus supporting the fact-checking activity, we performed experiments with a pipeline built to leverage crowdsourcing for the misinformation assessment task. We discuss the developed experiments along with their results in the following.
We first conducted an experiment to evaluate the truthfulness of a subset of 120 statements
made by US politicians [
This experiment demonstrated that crowd workers can efectively classify misinformation using the proposed truthfulness scales, as indicated by the results of the aggregation functions tested and reported by La Barbera et al. [4, Figure 4]. The arithmetic mean consistently produced the most accurate final truthfulness score among workers. However, the use of a 100-level scale exhibited leniency towards values multiples of 10, suggesting that less coarse-grained scales may be preferable be workers. Additionally, political bias was observed in workers’ assessments, with individuals tending to be more indulgent towards statements made by politicians of their own political afiliation. This experiment also highlighted potential systematic biases, as over 70% of the evidence sources cited by workers originated from the PolitiFact domain.
To enhance the robustness and generalizability of our findings, we also extended the previous experiment by incorporating a 3-level scale in addition to the 6 and 100-level ones along with statements from ABC News2 as a second source for statements in addition to PolitiFact. Each worker was asked to assess six statements from PolitiFact, three from ABC, and two gold standard questions. To prevent search bias observed in the previous experiment, we excluded ABC and PolitiFact search results from worker web searches. Furthermore, to measure worker’s ability to override their initial “gut” response and engage in further reflection to find the correct answer, we included a Cognitive Reflection Test (CRT) questionnaire.
In this second experiment, we confirmed and extended the findings of the first, demonstrating worker agreement with experts across various scales, with a preference for less fine-grained scales [5, Figure 2]. Our results also revealed worker bias, with political background and demographic factors afecting judgment accuracy [ 5, Table 5]. These findings support the use of crowdsourced truthfulness judgments to assess misinformation on a large scale. However, we found several unresolved issues concerning the statements evaluated in our experiments, as some were not recent and were made by public figures from a country other than that of the workers.
To improve previous research findings, we conducted a study on the COVID-19 pandemic, a
major source of daily misinformation. Our objective was to assess the feasibility and reliability
of crowdsourcing as a method for evaluating the truthfulness of subjective and time-sensitive
misinformation, as compared to expert judgment. We initiated our study by requesting crowd
workers to evaluate the veracity of COVID-19-related statements during the pandemic and
provide corroborating evidence [
Our study found that the crowd can assess the truthfulness of pandemic-related statements, in agreement with previous experiments (see [6, Figure 2]). Our results on worker behavior, agreement, scale transformations, aggregation functions, worker background and bias align with previously collected data. Our longitudinal study further confirmed the presence of worker biases observed in previous experiments (see [6, Table 2]). We also found experienced workers exhibiting diferent behaviors than novices; experienced workers spent more time evaluating each statement and their quality was similar to or higher than other workers. Additionally, we found that as the number of batches performed by workers increased, the average time spent on all documents decreased substantially, indicating a significant impact of time span on judgment quality for both novice and experienced workers. 2https://apo.org.au/collection/302996/rmit-abc-fact-check
Previous experiments [
Building on previous research [
This paper presents our research on using crowdsourcing for fact-checking and provides a summary of our findings. While our results suggest that crowdsourcing has the potential to be a reliable way to combat misinformation at scale, more work is needed to improve its reliability for use in real settings. Future studies should explore strategies to enhance worker efectiveness by investigating additional truthfulness dimensions, worker motivation and incentives, more complex aggregation functions, and worker behavioral signals. Crowdsourcing has the potential to evaluate large volumes of information from diverse perspectives, making it a valuable resource for fact-checking. Conference on Healthcare Informatics, 2017, pp. 518–518. doi:10.1109/ICHI.2017.58. [13] W. Y. Wang, "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection, in: R. Barzilay, M. Kan (Eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, volume 4, Association for Computational Linguistics, 2017, pp. 422–426. doi:10.18653/v1/P17-2067. [14] International Organization for Standardization, ISO/IEC 25012:2008 Software engineering — Software product Quality Requirements and Evaluation (SQuaRE) — Data quality model, Technical Report, ISO, 2008. URL: https://www.iso.org/standard/35736.html. [15] B. K. Kahn, D. M. Strong, R. Y. Wang, Information Quality Benchmarks: Product and Service Performance, Communications of the ACM 45 (2002) 184–192. doi:10.1145/ 505248.506007. [16] D. Ceolin, J. Noordegraaf, L. Aroyo, Capturing the Inefable: Collecting, Analysing, and Automating Web Document Quality Assessments, in: E. Blomqvist, P. Ciancarini, F. Poggi, F. Vitali (Eds.), Knowledge Engineer, Springer International Publishing, Cham, 2016, pp. 83–97. doi:10.1007/978-3-319-49004-5_6.