=Paper=
{{Paper
|id=Vol-2816/short4
|storemode=property
|title=s-AWARE: Using Crowd Judgements in Supervised Measure-Based Methods for IR Evaluation
|pdfUrl=https://ceur-ws.org/Vol-2816/short4.pdf
|volume=Vol-2816
|authors=Marco Ferrante,Nicola Ferro,Luca Piazzon
|dblpUrl=https://dblp.org/rec/conf/ircdl/Ferrante0P21
}}
==s-AWARE: Using Crowd Judgements in Supervised Measure-Based Methods for IR Evaluation==
https://ceur-ws.org/Vol-2816/short4.pdf
s-AWARE: Using Crowd Judgements in
Supervised Measure-Based Methods for IR
Evaluation?
Marco Ferrante1 , Nicola Ferro2 , and Luca Piazzon2
1
Department of Mathematics “Tullio Levi-Civita”, University of Padua, Italy
ferrante@math.unipd.it
2
Department of Information Engineering, University of Padua, Italy
{ferro,piazzonl}@dei.unipd.it
Abstract. Crowdsourcing methodologies have recently emerged as a
cheap and fast alternative to the traditional document assessment pro-
cess for ground truth creation. Early approaches make use of voting
and/or classification methodologies to combine crowd judgements into a
merged pool, used as reference in the evaluation process.
A measure-based approach has instead been used in Assessor-driven
Weighted Averages for Retrieval Evaluation (AWARE) [3], focusing in
optimizing the final evaluation measure without merging judgements at
pool level.
s-AWARE extends AWARE with a set of supervised methods. We rely
on several TREC collections to evaluate s-AWARE and we show that it
outperforms state-of-the-art methods. Moreover, our results show that
when moving to the real case scenario where a crowd-assessor only judges
a portion of the dataset, s-AWARE is quite an effective approach.
1 Introduction
Document assessment for ground-truth creation is one of the most demanding
tasks in preparing an experimental collection in both terms of time and costs,
and it has traditionally been performed by relying on expert assessors [8].
Crowdsourcing methodologies [2] have been recently exploited for a faster and
cheaper collection of multiple, even less qualified, document assessments. These
judgements are used together in the evaluation process with the objective of
achieving a proficient evaluation, comparable to the traditional one. The most
common way to use crowd-judgements is to create a merged pool to be used
as the gold standard for evaluation. Since errors in the merged pool can un-
fairly affect evaluation measures, in our work we moved the merging process
at measure level, as firstly proposed in Assessor-driven Weighted Averages for
Retrieval Evaluation (AWARE)[3]. Performance measures are firstly computed
?
Extended abstract of [4].
Copyright c 2021 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0). This volume is published
and copyrighted by its editors. IRCDL 2021, February 18-19, 2021, Padua, Italy.
based on each crowd-assessor judgements and then merged weighting by an es-
timate of each assessor accuracy, computed making use of unsupervised estima-
tors. s-AWARE extends AWARE and uses supervised estimators based on the
closeness between each assessor and the gold standard in a small set of train-
ing topics. We evaluated our s-AWARE against the state-of-the-art supervised
and unsupervised methods by using several TREC datasets, achieving promising
results.
This extended abstract will describe s-AWARE methodology and perfor-
mance, presenting some related works (Section 2), the description of the ap-
proaches (Section 3), the performed experiments (Section 4) and possible future
extensions (Section 5).
2 Related Works
One of the first developed crowd-assessor merging approach is Majority Vote
(MV) [11], that assigns to each document the judgement proposed by the ma-
jority of crowd-assessors; Weighted versions of MV have been proposed do boost
proficient assessors, e.g. [12,11].
Expectation Maximization (EM) algorithms optimize the probability of rele-
vance of each document in an unsupervised [6] or semi-supervised way [10] and
then assign to each document the most probable judgement . Another EM al-
ternative [5] uses a variant of the same algorithm to optimize assessor reliability
to be used to weight crowd judgements.
One weakness of the above described pool merging strategies is the possibility to
propagate mislabelling errors to evaluation measures. Different measures could
even be differently affected by the same pool error. Assessor-driven Weighted
Averages for Retrieval Evaluation (AWARE) tries to overcome this problem
by performing evaluation on the judgements given by every crowd-assessor and
combining the obtained measures weighting each assessor with his accuracy, es-
timated in an unsupervised way favouring assessors behaving differently from
some fake random assessors:
m
X ak (t)
aware µ(rt ) = µ r̂tk Pm
k=1 h=1 ah (t)
where m is the number of crowd-assessors to merge, µ r̂tk is the value of
the performance measure computed on run r for topic t according to the k-
th crowd-assessor, and ak is the accuracy of the k-th crowd-assessor. AWARE
computes accuracies as a function of the distance from random assessors: the
more a crowd-assessor is far from a set of random assessors, the better it is.
3 s-AWARE Methodology
To describe s-AWARE accuracy estimation, we consider the matrix Mk con-
taining the measures computed for a set S of systems and a set T of topics
based on the judgements issued by the k-th crowd assessor, and we define M ∗
as the gold standard measures matrix. The idea behind s-AWARE is to assign
an higher accuracy to assessors that behaved similarly to the gold standard on
a set of training topics. We consider the two best performing approaches used
in AWARE to quantify the “closeness” Ck to the gold standard:
– Measure closeness: we consider the Root Mean Square Error (RMSE) between
the crowd-measure and the gold standard one
v
u
u |S| ∗
2
∗
uX M k (·, s) − M (·, s)
Ck = RM SE M k (·, S) − M (·, S) = u
|S|
t
s=1
where M (·, s) indicates the average measure by topic
– Ranking of Systems closeness: we use the Kendall’s τ correlation between the
ranking of systems based on the crowd-measures and the gold standard one
∗
A−D
Ck = τ M k (·, S), M (·, S) =
| S | (| S | −1)/2
where A is the number of system pairs ranked in the same order in M k (·, S)
∗
and M (·, S), and D is the number of discordant pairs.
Ck s are then normalized in the [0,1] range, obtaining normalized Ck equal to
1 with gold standard behaviour. Squared and cubed Ck are also considered to
sharpen the distinction between good and bad assessors.
4 Experiments
4.1 Setup
We compared s-AWARE approaches against MV, EM with MV seeding[6],
AWARE with uniform accuracy scores (uniform), unsupervised AWARE (u-
AWARE) unsup rmse tpc and unsup tau tpc approaches (using respectively
RMSE and Kendall’s τ GAP computation), Georgescu Zhu EM method (hard
labels, PN discrimination, no boost version) (emGZ) [5] and semi-supervised EM
(using 30% of the documents as training set)(emsemi) [10].
In our evaluation, for each approach, we evaluated the systems with Average
Precision (AP), and we evaluated each approach performance by computing the
AP Correlation (APC) [15] between the ranking induced by AP values and the
gold standard ranking.
We used two different collections, using the NIST judgments as gold standard:
– TREC 2012 Crowdsourcing track [9]: 31 complete pools of judgements on 10
topics common to TREC 08 Adhoc track (T08) [14] and TREC 13 Robust
track(T13) [13]. We used the 129 runs from T08 and the 110 runs from T13.
– TREC 2017 Common Core track (T26) [1]: real crowdsourced judgements
gathered by Inel Et al. [7] from 406 crowd assessors on short documents
(≤ 1000 words) within the NYTimes corpus. Judgements refer to 50 topics,
having exactly 7 judgements for each (topic, document) pair. We used the
75 runs from T26.
We tested s-AWARE using only the 30% of the topics as training set. We
considered k-tuples from 2 to 7 crowd-assessors. We validated the results by
repeating both topic and assessor sampling 100 times for each k-tuples size.
We performed experiments under two possible scenarios, considering Whole
Assessors and Partitioned Assessors. In the Whole Assessors case (most favor-
able to supervised approaches but quite unrealistic) each crowd-assessor com-
pletely judges all the topics. Whole Assessors data is available only for the T08
and T13 tracks. In the Partitioned Assessors case (real case scenario, more chal-
lenging for supervised approaches), each crowd-assessor judges just a portion of
the documents for a portion of the topics. Therefore, to get the complete pools
assigned to each Partitioned Assessor we group judgements coming from differ-
ent crowd-assessors. This is the case of T26 track, but we also simulated this
configuration on the T08 and T13 tracks, by assembling the judgments coming
from more participants into each topic.
4.2 Main Results
Table 1 reports the AP Correlation results in the tested configurations on the
test portion of the dataset (70% of the documents from 70% of the topics, the
common subset of documents unseen by both s-AWARE and emsemi). The best
performing approach in the Whole Assessors case is our sup tau cubed, con-
stantly achieving better performance with respect to all the other approaches.
More in general, as expected, s-AWARE approaches generally outperform the
baselines and the corresponding unsupervised u-AWARE approach, that any-
way significantly outperform the baselines.
We notice a very poor performance of emGZ and only a little improvement of
emsemi with respect to emmv. This is probably due to the very limited amount
of training data, more effectively exploited by s-AWARE.
In Partitioned Assessors case we face up a different situation, where s-AWARE
advantage is limited with respect to u-AWARE approaches. On T08 and T13,
unsup rmse tpc u-AWARE method performs generally better than s-AWARE,
but s-AWARE still outperform the other u-AWARE approaches and the base-
lines. This narrower gap supports the idea that the Partitioned Assessors case
is less favorable to supervised approaches, since the training phase reflects less
what happens in the test phase; In general, we can observe that s-AWARE still
performs remarkably better than emsemi.
Looking to T26, s-AWARE approaches always outperform all the other ap-
proaches, with sup tau cubed achieving the best performance for all k-tuples.
This is very promising since, while Partitioned assessors for T08 and T13 are
simulated, T26 is the only dataset obtained by real crowd assessors, showing
a good performance in a real case scenario. In fact, we hypothesize that bad
performance on T08 and T13 can be due to the little more fragmentation of the
simulated partitioned assessors, i.e. smaller pieces from more crowd-assessors,
with respect to the the T26 ones.
In all our results, Kendall’s τ performs better than RMSE as s-AWARE “close-
ness” accuracy computation, and cubed and squared s-AWARE approaches achieve,
in general, better performance than the basic closeness approach, since they em-
phasize more sharply the difference between good and bad assessors. Moreover,
results highlight that s-AWARE approaches can obtain good results even with
small k-tuple size.
sup rmse squared
sup tau squared
sup rmse cubed
unsup rmse tpc
sup tau cubed
unsup tau tpc
sup rmse
uniform
sup tau
emsemi
emGZ
emmv
mv
k02 0.6048 0.6184 0.6086 0.6278 0.6120 0.6326 0.6075 0.6031 0.6008 0.5326 0.5183 0.5455 0.5470
T08-whole
k03 0.6317 0.6499 0.6366 0.6659 0.6414 0.6766 0.6324 0.6298 0.6265 0.6099 0.6025 0.5413 0.6097
k04 0.6492 0.6707 0.6546 0.6905 0.6598 0.7045 0.6422 0.6501 0.6436 0.6147 0.6154 0.5562 0.6329
k05 0.6689 0.6958 0.6751 0.7221 0.6812 0.7409 0.6808 0.6732 0.6625 0.6569 0.6512 0.5445 0.6535
k06 0.6555 0.6833 0.6620 0.7120 0.6685 0.7340 0.6622 0.6651 0.6492 0.6163 0.5918 0.5095 0.5963
k07 0.6719 0.6998 0.6782 0.7274 0.6845 0.7482 0.6709 0.6834 0.6657 0.6696 0.6396 0.5028 0.6443
k02 0.6111 0.6192 0.6139 0.6238 0.6162 0.6254 0.6005 0.6078 0.6079 0.5410 0.4974 0.5012 0.5186
T13-whole
k03 0.6526 0.6616 0.6562 0.6692 0.6594 0.6733 0.6254 0.6548 0.6486 0.6088 0.5926 0.4770 0.6085
k04 0.6687 0.6825 0.6728 0.6941 0.6765 0.7008 0.6250 0.6823 0.6641 0.6214 0.6119 0.4910 0.6241
k05 0.7061 0.7237 0.7106 0.7387 0.7148 0.7478 0.6797 0.7209 0.7011 0.6613 0.6491 0.4478 0.6497
k06 0.6872 0.7068 0.6923 0.7253 0.6971 0.7379 0.6502 0.7151 0.6818 0.6197 0.5913 0.4289 0.5919
k07 0.7045 0.7232 0.7092 0.7402 0.7135 0.7515 0.6552 0.7330 0.6996 0.6708 0.6452 0.4062 0.6476
k02 0.5314 0.5390 0.5332 0.5456 0.5350 0.5500 0.5508 0.5317 0.5294 0.4919 0.4944 0.5024 0.4913
T08-part
k03 0.5466 0.5587 0.5497 0.5700 0.5526 0.5783 0.5831 0.5457 0.5436 0.5171 0.5292 0.5050 0.5321
k04 0.5549 0.5690 0.5584 0.5830 0.5621 0.5935 0.6037 0.5553 0.5512 0.5153 0.4967 0.4992 0.5191
k05 0.5564 0.5725 0.5604 0.5891 0.5645 0.6019 0.6168 0.5599 0.5523 0.5368 0.4804 0.4914 0.5118
k06 0.5683 0.5863 0.5729 0.6064 0.5775 0.6226 0.6552 0.5692 0.5638 0.5287 0.4785 0.4782 0.4962
k07 0.5672 0.5900 0.5737 0.6150 0.5797 0.6333 0.6872 0.5696 0.5615 0.5373 0.4774 0.4639 0.4776
k02 0.5842 0.5959 0.5862 0.6038 0.5879 0.6078 0.5998 0.5767 0.5820 0.5406 0.5052 0.4945 0.4847
T13-part
k03 0.6155 0.6299 0.6181 0.6406 0.6206 0.6474 0.6412 0.6015 0.6126 0.5728 0.5854 0.4611 0.5742
k04 0.6372 0.6528 0.6402 0.6647 0.6430 0.6722 0.6706 0.6270 0.6340 0.5848 0.5757 0.4157 0.5838
k05 0.6481 0.6641 0.6515 0.6773 0.6549 0.6862 0.6929 0.6508 0.6444 0.6079 0.5619 0.3521 0.6009
k06 0.6616 0.6776 0.6653 0.6914 0.6691 0.7015 0.7211 0.6663 0.6579 0.6165 0.5573 0.3044 0.5840
k07 0.6560 0.6728 0.6603 0.6884 0.6642 0.7006 0.7306 0.6412 0.6512 0.6209 0.5332 0.1963 0.5568
k02 0.3817 0.4008 0.3796 0.4084 0.3774 0.4124 0.3531 0.3928 0.3837 0.3731 0.3362 0.3506 0.3625
T26-part
k03 0.3863 0.4067 0.3839 0.4151 0.3815 0.4191 0.3522 0.4028 0.3886 0.3783 0.3512 0.3753 0.3680
k04 0.3824 0.4072 0.3795 0.4179 0.3767 0.4236 0.3421 0.4029 0.3853 0.3791 0.3525 0.3688 0.3625
k05 0.3832 0.4102 0.3796 0.4228 0.3761 0.4295 0.3396 0.4077 0.3866 0.3785 0.3602 0.3648 0.3729
k06 0.3926 0.4232 0.3896 0.4366 0.3870 0.4441 0.3568 0.4207 0.3961 0.3781 0.3584 0.3466 0.3737
k07 0.4534 0.4787 0.4521 0.4918 0.4507 0.4980 0.4171 0.4841 0.4561 0.4400 0.4302 0.3715 0.4239
Table 1: AP Correlation results. Baseline approaches are in blue, u-AWARE in green, s-AWARE in
orange. Darker color indicate better performance. Best performing approaches for each k-tuple size
are in bold.
5 Conclusions and Future Work
We presented s-AWARE [4] a methodology for merging crowd-assessors, that ex-
tends AWARE approach to supervised techniques. We tested s-AWARE against
a set of unsupervised and supervised baselines, highlighting the effectiveness of
s-AWARE in the very challenging real scenario situation where only 30% of the
documents were used for training. s-AWARE outperform all the others in the
Whole Assessors case and is still quite robust in the Partitioned Assessors case.
In the future, we plan to extend AWARE framework to better deal with partial
assessments, assigning an accuracy score to each real crowd assessor, avoiding
the need to group judgements as done in Partitioned assessor case.
References
1. Allan, J., Harman, D.K., Kanoulas, E., Li, D., Van Gysel, C., Voorhees, E.M.:
TREC 2017 Common Core Track Overview. In: Voorhees, E.M., Ellis, A. (eds.)
The Twenty-Sixth Text REtrieval Conference Proceedings (TREC 2017). National
Institute of Standards and Technology (NIST), Special Publication 500-324, Wash-
ington, USA (2018)
2. Alonso, O.: The Practice of Crowdsourcing. Morgan & Claypool Publishers, USA
(May 2019)
3. Ferrante, M., Ferro, N., Maistro, M.: AWARE: Exploiting Evaluation Measures to
Combine Multiple Assessors. ACM Transactions on Information Systems 36(2),
1–38 (aug 2017). https://doi.org/10.1145/3110217, https://doi.org/10.1145%
2F3110217
4. Ferrante, M., Ferro, N., Piazzon, L.: s-aware: Supervised measure-based methods
for crowd-assessors combination. In: Arampatzis, A., Kanoulas, E., Tsikrika, T.,
Vrochidis, S., Joho, H., Lioma, C., Eickhoff, C., Névéol, A., Cappellato, L., Ferro,
N. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction.
pp. 16–27. Springer International Publishing, Cham (2020)
5. Georgescu, M., Zhu, X.: Aggregation of crowdsourced labels based on worker his-
tory. In: Proceedings of the 4th International Conference on Web Intelligence,
Mining and Semantics (WIMS14). WIMS ’14, Association for Computing Ma-
chinery, New York, NY, USA (2014). https://doi.org/10.1145/2611040.2611074,
https://doi.org/10.1145/2611040.2611074
6. Hosseini, M., Cox, I.J., Milić-Frayling, N., Kazai, G., Vinay, V.: On aggregating
labels from multiple crowd workers to infer relevance of documents. In: Proceedings
of the 34th European Conference on Advances in Information Retrieval. pp. 182–
194. ECIR’12 (2012)
7. Inel, O., Haralabopoulos, G., Li, D., Van Gysel, C., Szlávik, Z., Simperl, E.,
Kanoulas, E., Aroyo, L.: Studying Topical Relevance with Evidence-based Crowd-
sourcing. In: Cuzzocrea, A., Allan, J., Paton, N.W., Srivastava, D., Agrawal, R.,
Broder, A., Zaki, M.J., Candan, S., Labrinidis, A., Schuster, A., Wang, H. (eds.)
Proc. 27th International Conference on Information and Knowledge Management
(CIKM 2018). pp. 1253–1262. ACM Press, New York, USA (2018)
8. Sanderson, M.: Test Collection Based Evaluation of Information Retrieval Systems.
Foundations and Trends in Information Retrieval (FnTIR) 4(4), 247–375 (2010)
9. Smucker, M.D., Kazai, G., Lease, M.: Overview of the TREC 2012 Crowdsourcing
Track. In: Voorhees, E.M., Buckland, L.P. (eds.) The Twenty-First Text REtrieval
Conference Proceedings (TREC 2012). National Institute of Standards and Tech-
nology (NIST), Special Publication 500-298, Washington, USA (2013)
10. Tang, W., Lease, M.: Semi-supervised consensus labeling for crowdsourcing. In:
Proceedings of the SIGIR 2011 workshop on crowdsourcing for information retrieval
(CIR). pp. 36–41. Association for Computing Machinery, New York, NY, USA
(2011)
11. Tao, D., Cheng, J., Yu, Z., Yue, K., Wang, L.: Domain-weighted majority voting
for crowdsourcing. IEEE Transactions on Neural Networks and Learning Systems
30(1), 163–174 (jan 2019). https://doi.org/10.1109/tnnls.2018.2836969, https://
doi.org/10.1109%2Ftnnls.2018.2836969
12. Tian, T., Zhu, J., Qiaoben, Y.: Max-margin majority voting for learning from
crowds. IEEE Transactions on Pattern Analysis and Machine Intelligence 41(10),
2480–2494 (oct 2019). https://doi.org/10.1109/tpami.2018.2860987, https://doi.
org/10.1109%2Ftpami.2018.2860987
13. Voorhees, E.M.: Overview of the TREC 2004 Robust Track. In: Voorhees, E.M.,
Buckland, L.P. (eds.) The Thirteenth Text REtrieval Conference Proceedings
(TREC 2004). National Institute of Standards and Technology (NIST), Special
Publication 500-261, Washington, USA (2004)
14. Voorhees, E.M., Harman, D.K.: Overview of the Eigth Text REtrieval Conference
(TREC-8). In: Voorhees, E.M., Harman, D.K. (eds.) The Eighth Text REtrieval
Conference (TREC-8). pp. 1–24. National Institute of Standards and Technology
(NIST), Special Publication 500-246, Washington, USA (1999)
15. Yilmaz, E., Aslam, J.A., Robertson, S.E.: A New Rank Correlation Coefficient for
Information Retrieval. In: Chua, T.S., Leong, M.K., Oard, D.W., Sebastiani, F.
(eds.) Proc. 31st Annual International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR 2008). pp. 587–594. ACM Press,
New York, USA (2008)