We report on collecting truthfulness values (i) by means of crowdsourcing and (ii) using negrained scales. In our experiment we collect truthfulness values using a bounded and discrete scale with 100 levels as well as a magnitude estimation scale, which is unbounded, continuous and has in nite amount of levels. We compare the two scales and discuss the agreement with a ground truth provided by experts on a six-level scale.
Checking the validity of statements is an important task to support the detection of rumors and fake news in social media. One of the challenges is the ability to scale the collection of validity labels for a large number of statements.
Fact-checking has been shown as a task di cult to be performed in crowdsourcing platforms.1 However, crowdworkers are often asked to annotate truthfulness of statements using a few discrete values (e.g., true/false labels).
Recent work in information retrieval [Roi+18; Mad+17] has shown that using more ne-grained 2
Recent work looked at the methods to automatically detect fake news and fact-check. Kriplean et al. [Kri+14] look at the use of volunteer crowdsourcing to fact-check embedded into a socio-technical system similar to the democratic process. As compared to them, we look at the more systematic involvement of humans in the loop to quantitatively assess the truthfulness of statements.
Our work looks at experimentally comparing di erent schemes to collect labelled data for truthful facts. Related to this, Medo and Wakeling [MW10] investigate how the discretization of ratings a ects the codetermination procedure, i.e., where estimates of user and object reputation are re ned iteratively together.
Zubiaga et al. [Zub+18] and Zubiaga and Ji [ZJ14] look at how humans assess credibility of information and, by means of a human study, identify key credibility perception features to be used for automatic detection of credible tweets. As compared to them, we also look at the human dimension of credibility checking but rather focus on which is the most appropriate scale for human assessors to make such assessment.
Kochkina, Liakata, and Zubiaga [KLZ18b] and Kochkina, Liakata, and Zubiaga [KLZ18a] look at rumour veri cation by proposing a supervised machine learning model to automatically perform such a task. As compared to them, we focus on understanding the most e ective scale used to collect training data to then build such models.
Besides the dataset we used for our experiments in this paper, other datasets related to fact checking and the truthfulness assessment of statements have been created. The Fake News Challenge2 addresses the the task of stance detection: estimate the stance of a body text from a news article relative to a headline. Specifically, the body text may agree, disagree, discuss or be unrelated to the headline. Fact-checking Lab at CLEF 2018 [Nak+18] addresses a ranking task, i.e., to rank sentences in a political debate according to their worthiness for fact-checking, and a classi cation task, i.e., given a sentence that is worth checking, to decide whether the claim is true, false or unsure of its factuality. In our work we use the dataset rst proposed by Wang [Wan17] as it has been created using six-level labels which is in-line with our research question about how many levels are most appropriate for such labelling task. 3 3.1
We use a sample of statements from the dataset detailed by Wang [Wan17]. The dataset consists of a collection of 12,836 labelled statements; each statement is accompanied by some meta-data specifying its \speaker", \speaker's job", and \context" (i.e., the context in which the statement has been said) information, as well as the the truth label made by experts on a six-level scale: pants- re (i.e., lie), false, barely-true, half-true, mostly-true, and true.
For our re-assessment, we perform a strati ed random sampling to select 10 statements for each of the six categories, obtaining a total of 60 statements. The screenshot in Figure 1 shows one of the statements included in our sample. 3.2
We obtain for each statement a crowdsourced truth label by 10 di erent workers. Each worker judges six statements (one for each category) plus two additional \gold" statements used for quality checks. We also ask each worker to provide a justi cation for the truth value he/she provide.
We pay the workers 0.2$ for each set of 8 judgments (i.e., one Human Intelligent Task, or HIT). Workers are allowed to do one HIT for each scale only, but they are allowed to provide judgments for both scales.
We use randomized statement ordering to avoid any possible document-ordering e ect/bias.
To ensure a good quality dataset, we use the following quality checks in the crowdsourcing phase: the truth value of the two gold statements (one patently false and the other one patently true) has to be consistent; the time spent to judge each statement has to be greater than 8 seconds; each worker has two attempts to complete the task; at the third unsuccessful attempt of submitting the task the user is prevented to continue further.
We collected the data using the Figure-Eight platform.3 3.3
We consider two di erent truth scales, keeping the same experimental setting (i.e., quality checks, HITs, etc.): 1. a scale in the [0; 100] range, denoted as S100; 2. the Magnitude Estimation [Mos77] scale in the (0; 1) range, denoted as ME1.
The e ects and bene ts of using the two scales in the setting of assessing document relevance for information retrieval evaluation has been explored by Maddalena et al. [Mad+17] and Roitero et al. [Roi+18].
Overall, we collect 800 truth labels for each scale, so 1,600 in total, for a total cost of 48$ including fees. 2http://www.fakenewschallenge.org/ 3https://www.figure-eight.com/. 0 20 40 60 80
100 Score 20 40 60
80 Score
While the raw scores obtained with the S100 scale are ready to use, the scores from ME1 need a normalization phase (since each worker will use a personal, and potentially di erent, \inner scale factor" due to the absence of scale boundaries); we computed the normalized scores for the ME1 scale following the standard normalization approach for such a scale, namely geometric averaging [Ges97; McG03; Mos77]: higher values, i.e., the right of the plot, and there is a clear tendency of giving scores which are multiple of ten (an e ect that is consistent with the ndings by Roitero et al. [Roi+18]).
For the ME1 scale, we see that the normalized scores are almost normally-distributed (which is consistent with the property that scores collected on a ratio scale like ME1 should be log-normal), although the distribution is slightly skewed towards lower values (i.e., left part of the plot). s = exp log s
H (log s) + (log s) ; 4.2
where s is is the raw score, H (log s) is the mean value of the log s within a HIT, and (log s) is the mean of the logarithm of all ME1 scores.
Figure 2 shows the individual scores distributions: for S100 (left) the raw scores are reported and for ME1 (right) the normalized scores. The x-axis represents the score, while the y-axis its absolute frequency; the cumulative distribution is denoted by the red line. As we can see, for S100 the distribution is skewed towards Next, we compute the aggregated scores for both scales: we aggregate the scores of the ten workers judging the same statement. Following the standard practices, we aggregate the S100 values using the arithmetic mean, as done by Roitero et al. [Roi+18], and the ME1 values using the median, as done by Maddalena et al. [Mad+17] and Roitero et al. [Roi+18]. Figure 3 shows the aggregated scores; comparing with Figure 2, we notice that for S100 the distribution is 30 20 more balanced, although it can not be said to be bellshaped, and the decimal tendency e ect disappears; furthermore, the most common value is not 100 (i.e., the limit of the scale) anymore. Concerning ME1, we see that the scores are still roughly normally distributed.4 However, the x-range is more limited; this is an e ect of the aggregation function, which tends to remove the outlier scores. 4.3
We now turn to compare with the ground truth our truth levels obtained by crowdsourcing. Figure 4 shows the comparison between the S100 and ME1 (normalized and) aggregated scores with the six-level ground truth. In each of the two charts, each box-plot represents the corresponding scores distribution. We also report the individual (normalized and) aggregated scores as colored dots with some random horizontal jitter. We can see that, even with a small number of documents (i.e., ten for each category), the median values of the box-plots are increasing; this is always the case for S100, and true for most of the cases for ME1 (where there is only one case in which this is untrue, for the two adjacent categories \Lie" and \False"). This behavior suggests that both the S100 and ME1 scales allow to collect truth levels that are overall consistent with the ground truth, and that the S100 scale leads to a slightly higher level of agreement with the expert judges than the ME1 scale. We analyze agreement in more detail in the following.
4Running the omnibus test of normality implemented in scipy.stats.normaltest [DP73], we cannot reject the null hypothesis, i.e., p > :001 for both the aggregated and raw normalized scores. Although not rejecting the null hypothesis does not necessary tell us that they follow a normal distribution, we can say we are pretty con dent they came from a normal distribution. 4.4
Figure 5 shows the inter-assessor agreement of the workers, namely the agreement among all the ten workers judging the same statement. Agreement is computed using Krippendor 's [Kri07] and Common Agreement [Che+17] measures; as already pointed out by Checco et al. [Che+17], and measure substantially di erent notions of agreement. As we can see, while the two agreement measures show some degree of similarity for S100, for ME1 the agreement computed is substantially di erent: while has values close to zero (i.e., no agreement), shows a high agreement level, on average around 0:8. Checco et al. [Che+17] show that can have an agreement value of zero even when the agreement is actually present in the data. Although agreement values seem higher for ME1, especially when using , it is di cult to clearly prefer one of the two scales from these results. 4.5
We also measure the agreement within one unit. We use the de nition of pairwise agreement by Roitero et al. [Roi+18, Section 4.2.1] that allows to compare (S100 and ME1) scores with a ground truth on di erent scales (six levels). Figure 6 shows that the pairwise agreement with the experts of the scores collected using the two scales is similar. 4.6
As a last result, we note that the two scales measure something di erent, as shown by the scatter-plot in Figure 7. Each dot is one statement and the two coordinates are its aggregated scores on the two scales. Although Pearson's correlation between the two scales is positive and signi cant, it is clear that there are some di erences, that we plan to study in future work. 100 80 . 60 q e rF 40 We performed a crowdsourcing experiment to analyze the impact of using di erent ne-grained labeling scales when asking crowdworkers to annotate truthfulness of statements. In particular, we tested two labeling scales: S100 [RMM17] and ME1 [Mad+17]. Our preliminary results with a small sample of statements from Wang's dataset [Wan17] suggest that: Crowdworkers annotate truthfulness of statements in a way that is overall consistent with the agr_measure alpha phi S100 ME experts ground truth collected on a six-levels scale (see Figure 4), thus it seems viable to crowdsource truthfulness of statements.
Also due to the limited size of our sample (10 statements), we cannot quantify which is the best scale to be used in this scenario: we plan to further address this issue in future work. In this respect, we remark that whereas the reliability of the S100 scale is perhaps expected, it is worth noticing that the ME1 scale, for sure less familiar, leads anyway to truthfulness values that are of comparable quality to the ones collected by means of the S100 scale.
The scale used has anyway some e ect, as it is shown by the di erences in Figure 4, the di erent agreement values in Figure 5, and the rather low agreement between S100 and ME1 in Figure 7. S100 and ME1 scales seems to lead to similar agreement with expert judges (Figure 6).
For space limits, we do not report on other data like, for example, the justi cations provided by the workers or the time taken to complete the job. We plan to do so in future work.
Our preliminary experiment is an enabling step to further explore the impact of di erent ne-grained labeling scales for fact-checking in crowdsourcing scenarios. We plan to extend the experiment with more and more diverse statements, also from other datasets, which will allow us to perform further analyses. We plan in particular to understand in more detail the di erences between the two scales highlighted in Figure 7. [Che+17]
Preslav Nakov, Alberto Barron-Ceden~o, Tamer Elsayed, Reem Suwaileh, Llu s Marquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. \Overview of the CLEF-2018 CheckThat! Lab on Automatic Identi cation and Veri cation of Political Claims". In: Proc. CLEF. 2018.