Crowdsourcing methodologies have recently emerged as a cheap and fast alternative to the traditional document assessment process for ground truth creation. Early approaches make use of voting and/or classification methodologies to combine crowd judgements into a merged pool, used as reference in the evaluation process. A measure-based approach has instead been used in Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) [3], focusing in optimizing the final evaluation measure without merging judgements at pool level. s-AWARE extends AWARE with a set of supervised methods. We rely on several TREC collections to evaluate s-AWARE and we show that it outperforms state-of-the-art methods. Moreover, our results show that when moving to the real case scenario where a crowd-assessor only judges a portion of the dataset, s-AWARE is quite an effective approach.
Document assessment for ground-truth creation is one of the most demanding tasks in preparing an experimental collection in both terms of time and costs, and it has traditionally been performed by relying on expert assessors [8]. Crowdsourcing methodologies [2] have been recently exploited for a faster and cheaper collection of multiple, even less qualified, document assessments. These judgements are used together in the evaluation process with the objective of achieving a proficient evaluation, comparable to the traditional one. The most common way to use crowd-judgements is to create a merged pool to be used as the gold standard for evaluation. Since errors in the merged pool can unfairly affect evaluation measures, in our work we moved the merging process at measure level, as firstly proposed in Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) [3]. Performance measures are firstly computed based on each crowd-assessor judgements and then merged weighting by an estimate of each assessor accuracy, computed making use of unsupervised estimators. s-AWARE extends AWARE and uses supervised estimators based on the closeness between each assessor and the gold standard in a small set of training topics. We evaluated our s-AWARE against the state-of-the-art supervised and unsupervised methods by using several TREC datasets, achieving promising results.
This extended abstract will describe s-AWARE methodology and performance, presenting some related works (Section 2), the description of the approaches (Section 3), the performed experiments (Section 4) and possible future extensions (Section 5).
One of the first developed crowd-assessor merging approach is Majority Vote (MV) [11], that assigns to each document the judgement proposed by the majority of crowd-assessors; Weighted versions of MV have been proposed do boost proficient assessors, e.g. [12,11]. Expectation Maximization (EM) algorithms optimize the probability of relevance of each document in an unsupervised [6] or semi-supervised way [10] and then assign to each document the most probable judgement . Another EM alternative [5] uses a variant of the same algorithm to optimize assessor reliability to be used to weight crowd judgements. One weakness of the above described pool merging strategies is the possibility to propagate mislabelling errors to evaluation measures. Different measures could even be differently affected by the same pool error. Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) tries to overcome this problem by performing evaluation on the judgements given by every crowd-assessor and combining the obtained measures weighting each assessor with his accuracy, estimated in an unsupervised way favouring assessors behaving differently from some fake random assessors:
where m is the number of crowd-assessors to merge, µ rk t is the value of the performance measure computed on run r for topic t according to the kth crowd-assessor, and a k is the accuracy of the k-th crowd-assessor. AWARE computes accuracies as a function of the distance from random assessors: the more a crowd-assessor is far from a set of random assessors, the better it is.
To describe s-AWARE accuracy estimation, we consider the matrix M k containing the measures computed for a set S of systems and a set T of topics based on the judgements issued by the k-th crowd assessor, and we define M * as the gold standard measures matrix. The idea behind s-AWARE is to assign an higher accuracy to assessors that behaved similarly to the gold standard on a set of training topics. We consider the two best performing approaches used in AWARE to quantify the "closeness" C k to the gold standard:
-Measure closeness: we consider the Root Mean Square Error (RMSE) between the crowd-measure and the gold standard one
where M (•, s) indicates the average measure by topic -Ranking of Systems closeness: we use the Kendall's τ correlation between the ranking of systems based on the crowd-measures and the gold standard one
where A is the number of system pairs ranked in the same order in M k (•, S) and M * (•, S), and D is the number of discordant pairs.
C k s are then normalized in the [0,1] range, obtaining normalized C k equal to 1 with gold standard behaviour. Squared and cubed C k are also considered to sharpen the distinction between good and bad assessors.
We compared s-AWARE approaches against MV, EM with MV seeding [6], AWARE with uniform accuracy scores (uniform), unsupervised AWARE (u-AWARE) unsup rmse tpc and unsup tau tpc approaches (using respectively RMSE and Kendall's τ GAP computation), Georgescu Zhu EM method (hard labels, PN discrimination, no boost version) (emGZ) [5] and semi-supervised EM (using 30% of the documents as training set)(emsemi) [10].
In our evaluation, for each approach, we evaluated the systems with Average Precision (AP), and we evaluated each approach performance by computing the AP Correlation (APC) [15] between the ranking induced by AP values and the gold standard ranking. We used two different collections, using the NIST judgments as gold standard:
-TREC 2012 Crowdsourcing track [9]: 31 complete pools of judgements on 10 topics common to TREC 08 Adhoc track (T08) [14] and TREC 13 Robust track(T13) [13]. We used the 129 runs from T08 and the 110 runs from T13. -TREC 2017 Common Core track (T26) [1]: real crowdsourced judgements gathered by Inel Et al. [7] from 406 crowd assessors on short documents (≤ 1000 words) within the NYTimes corpus. Judgements refer to 50 topics, having exactly 7 judgements for each (topic, document) pair. We used the 75 runs from T26.
We tested s-AWARE using only the 30% of the topics as training set. We considered k-tuples from 2 to 7 crowd-assessors. We validated the results by repeating both topic and assessor sampling 100 times for each k-tuples size.
We performed experiments under two possible scenarios, considering Whole Assessors and Partitioned Assessors. In the Whole Assessors case (most favorable to supervised approaches but quite unrealistic) each crowd-assessor completely judges all the topics. Whole Assessors data is available only for the T08 and T13 tracks. In the Partitioned Assessors case (real case scenario, more challenging for supervised approaches), each crowd-assessor judges just a portion of the documents for a portion of the topics. Therefore, to get the complete pools assigned to each Partitioned Assessor we group judgements coming from different crowd-assessors. This is the case of T26 track, but we also simulated this configuration on the T08 and T13 tracks, by assembling the judgments coming from more participants into each topic.
Table 1 reports the AP Correlation results in the tested configurations on the test portion of the dataset (70% of the documents from 70% of the topics, the common subset of documents unseen by both s-AWARE and emsemi). The best performing approach in the Whole Assessors case is our sup tau cubed, constantly achieving better performance with respect to all the other approaches. More in general, as expected, s-AWARE approaches generally outperform the baselines and the corresponding unsupervised u-AWARE approach, that anyway significantly outperform the baselines. We notice a very poor performance of emGZ and only a little improvement of emsemi with respect to emmv. This is probably due to the very limited amount of training data, more effectively exploited by s-AWARE. In Partitioned Assessors case we face up a different situation, where s-AWARE advantage is limited with respect to u-AWARE approaches. On T08 and T13, unsup rmse tpc u-AWARE method performs generally better than s-AWARE, but s-AWARE still outperform the other u-AWARE approaches and the baselines. This narrower gap supports the idea that the Partitioned Assessors case is less favorable to supervised approaches, since the training phase reflects less what happens in the test phase; In general, we can observe that s-AWARE still performs remarkably better than emsemi. Looking to T26, s-AWARE approaches always outperform all the other approaches, with sup tau cubed achieving the best performance for all k-tuples. This is very promising since, while Partitioned assessors for T08 and T13 are simulated, T26 is the only dataset obtained by real crowd assessors, showing a good performance in a real case scenario. In fact, we hypothesize that bad performance on T08 and T13 can be due to the little more fragmentation of the simulated partitioned assessors, i.e. smaller pieces from more crowd-assessors, with respect to the the T26 ones. In all our results, Kendall's τ performs better than RMSE as s-AWARE "closeness" accuracy computation, and cubed and squared s-AWARE approaches achieve, in general, better performance than the basic closeness approach, since they emphasize more sharply the difference between good and bad assessors. Moreover, results highlight that s-AWARE approaches can obtain good results even with small k-tuple size.
We presented s-AWARE [4] a methodology for merging crowd-assessors, that extends AWARE approach to supervised techniques. We tested s-AWARE against a set of unsupervised and supervised baselines, highlighting the effectiveness of s-AWARE in the very challenging real scenario situation where only 30% of the documents were used for training. s-AWARE outperform all the others in the Whole Assessors case and is still quite robust in the Partitioned Assessors case.
In the future, we plan to extend AWARE framework to better deal with partial assessments, assigning an accuracy score to each real crowd assessor, avoiding the need to group judgements as done in Partitioned assessor case.