Crowdsourcing methodologies have recently emerged as a cheap and fast alternative to the traditional document assessment process for ground truth creation. Early approaches make use of voting and/or classi cation methodologies to combine crowd judgements into a merged pool, used as reference in the evaluation process. A measure-based approach has instead been used in Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) [3], focusing in optimizing the nal evaluation measure without merging judgements at pool level. s-AWARE extends AWARE with a set of supervised methods. We rely on several TREC collections to evaluate s-AWARE and we show that it outperforms state-of-the-art methods. Moreover, our results show that when moving to the real case scenario where a crowd-assessor only judges a portion of the dataset, s-AWARE is quite an e ective approach.
Document assessment for ground-truth creation is one of the most demanding
tasks in preparing an experimental collection in both terms of time and costs,
and it has traditionally been performed by relying on expert assessors [
Copyright c 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). This volume is published and copyrighted by its editors. IRCDL 2021, February 18-19, 2021, Padua, Italy. based on each crowd-assessor judgements and then merged weighting by an estimate of each assessor accuracy, computed making use of unsupervised estimators. s-AWARE extends AWARE and uses supervised estimators based on the closeness between each assessor and the gold standard in a small set of training topics. We evaluated our s-AWARE against the state-of-the-art supervised and unsupervised methods by using several TREC datasets, achieving promising results.
This extended abstract will describe s-AWARE methodology and performance, presenting some related works (Section 2), the description of the approaches (Section 3), the performed experiments (Section 4) and possible future extensions (Section 5). 2
One of the rst developed crowd-assessor merging approach is Majority Vote
(MV) [
Expectation Maximization (EM) algorithms optimize the probability of
relevance of each document in an unsupervised [
One weakness of the above described pool merging strategies is the possibility to propagate mislabelling errors to evaluation measures. Di erent measures could even be di erently a ected by the same pool error. Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) tries to overcome this problem by performing evaluation on the judgements given by every crowd-assessor and combining the obtained measures weighting each assessor with his accuracy, estimated in an unsupervised way favouring assessors behaving di erently from some fake random assessors:
m aware (rt) = X k=1 r^tk
ak(t) Pm h=1 ah(t) where m is the number of crowd-assessors to merge, r^tk is the value of the performance measure computed on run r for topic t according to the kth crowd-assessor, and ak is the accuracy of the k-th crowd-assessor. AWARE computes accuracies as a function of the distance from random assessors: the more a crowd-assessor is far from a set of random assessors, the better it is. 3 s-AWARE
To describe s-AWARE accuracy estimation, we consider the matrix Mk containing the measures computed for a set S of systems and a set T of topics based on the judgements issued by the k-th crowd assessor, and we de ne M as the gold standard measures matrix. The idea behind s-AWARE is to assign an higher accuracy to assessors that behaved similarly to the gold standard on a set of training topics. We consider the two best performing approaches used in AWARE to quantify the \closeness" Ck to the gold standard: { Measure closeness: we consider the Root Mean Square Error (RMSE) between the crowd-measure and the gold standard one
Ck = RM SE M k( ; S)
M ( ; S) = uutuu XjSj v s=1
M k( ; s)
M ( ; s) j S j 2 where M ( ; s) indicates the average measure by topic { Ranking of Systems closeness: we use the Kendall's correlation between the ranking of systems based on the crowd-measures and the gold standard one Ck =
M k( ; S); M ( ; S) =
A D j S j (j S j 1)=2 where A is the number of system pairs ranked in the same order in M k( ; S) and M ( ; S), and D is the number of discordant pairs.
Cks are then normalized in the [
We compared s-AWARE approaches against MV, EM with MV seeding[
In our evaluation, for each approach, we evaluated the systems with Average Precision (AP), and we evaluated each approach performance by computing the AP Correlation (APC) [15] between the ranking induced by AP values and the gold standard ranking.
We used two di erent collections, using the NIST judgments as gold standard:
{ TREC 2012 Crowdsourcing track [
We tested s-AWARE using only the 30% of the topics as training set. We considered k-tuples from 2 to 7 crowd-assessors. We validated the results by repeating both topic and assessor sampling 100 times for each k-tuples size.
We performed experiments under two possible scenarios, considering Whole Assessors and Partitioned Assessors. In the Whole Assessors case (most favorable to supervised approaches but quite unrealistic) each crowd-assessor completely judges all the topics. Whole Assessors data is available only for the T08 and T13 tracks. In the Partitioned Assessors case (real case scenario, more challenging for supervised approaches), each crowd-assessor judges just a portion of the documents for a portion of the topics. Therefore, to get the complete pools assigned to each Partitioned Assessor we group judgements coming from di erent crowd-assessors. This is the case of T26 track, but we also simulated this con guration on the T08 and T13 tracks, by assembling the judgments coming from more participants into each topic. 4.2
Table 1 reports the AP Correlation results in the tested con gurations on the test portion of the dataset (70% of the documents from 70% of the topics, the common subset of documents unseen by both s-AWARE and emsemi). The best performing approach in the Whole Assessors case is our sup tau cubed, constantly achieving better performance with respect to all the other approaches. More in general, as expected, s-AWARE approaches generally outperform the baselines and the corresponding unsupervised u-AWARE approach, that anyway signi cantly outperform the baselines.
We notice a very poor performance of emGZ and only a little improvement of emsemi with respect to emmv. This is probably due to the very limited amount of training data, more e ectively exploited by s-AWARE.
In Partitioned Assessors case we face up a di erent situation, where s-AWARE advantage is limited with respect to u-AWARE approaches. On T08 and T13, unsup rmse tpc u-AWARE method performs generally better than s-AWARE, but s-AWARE still outperform the other u-AWARE approaches and the baselines. This narrower gap supports the idea that the Partitioned Assessors case is less favorable to supervised approaches, since the training phase re ects less what happens in the test phase; In general, we can observe that s-AWARE still performs remarkably better than emsemi.
Looking to T26, s-AWARE approaches always outperform all the other approaches, with sup tau cubed achieving the best performance for all k-tuples. This is very promising since, while Partitioned assessors for T08 and T13 are simulated, T26 is the only dataset obtained by real crowd assessors, showing a good performance in a real case scenario. In fact, we hypothesize that bad performance on T08 and T13 can be due to the little more fragmentation of the simulated partitioned assessors, i.e. smaller pieces from more crowd-assessors, with respect to the the T26 ones.
In all our results, Kendall's performs better than RMSE as s-AWARE \closeness" accuracy computation, and cubed and squared s-AWARE approaches achieve, in general, better performance than the basic closeness approach, since they emphasize more sharply the di erence between good and bad assessors. Moreover, results highlight that s-AWARE approaches can obtain good results even with small k-tuple size.
We presented s-AWARE [