is paper reports on a case study on the inter-assessor disagreements in the English NTCIR-13 We Want Web (WWW) collection. For each of our 50 topics, pooled documents were independently judged by three assessors: two “lancers” and one Waseda University student. A lancer is a worker hired through a Japanese part time job matching website, where the hirer is required to rate the quality of the lancer's work upon task completion and therefore the lancer has a reputation to maintain. Nine lancers and ve students were hired in total; the hourly pay was the same for all assessors. On the whole, the inter-assessor agreement between two lancers is statistically signicantly higher than that between a lancer and a student. We then compared the system rankings and statistical signicance test results according to dierent qrels versions created by changing which asessors to rely on: overall, the outcomes do dier according to the qrels versions, and those that rely on multiple assessors have a higher discriminative power than those that rely on a single assessor. Furthermore, we consider removing topics with relatively low inter-assessor agreements from the original topic set: we thus rank systems using 27 high-agreement topics, aer removing 23 low-agreement topics. While the system ranking with the full topic set and that with the high-agreement set are statistically equivalent, the ranking with the high-agreement set and that with the low-agreement set are not. Moreover, the low-agreement set substantially underperforms the full and the high-agreement sets in terms of discriminative power. Hence, from a statistical point of view, our results suggest that a high-agreement topic set is more useful for nding concrete research conclusions than a low-agreement one.
•Information systems ! Retrieval eectiveness; inter-assessor agreement; p-values; relevance assessments; statistical signicance
While IR researchers oen view laboratory IR evaluation results as something objective, at the core of any laboratory IR experiments lie the relevance assessments, which are the result of subjective judgements of documents by a person, or multiple persons, based on a particular (intepretation of an) information need. Hence it is of utmost importance for IR researchers to understand the eects Copying permied for private and academic purposes.
EVIA 2017, co-located with NTCIR-13, Tokyo, Japan. © 2017 Copyright held by the author. of the subjective nature of the relevance assessment process on the nal IR evaluation results.
is paper reports on a case study on the inter-assessor
disagreements in a recently-constructed ad hoc web search test
collection, namely, the English NTCIR-13 We Want Web (WWW)
collection [
Studies on the eect of inter-assessor (dis)agreement on IR system
evaluation have a long history; Bailey et al. [
Voorhees [
e present study diers from that of Voorhees in the following aspects at least:
We use a new English web search test collection contructed for the NTCIR-13 WWW task, with depth-30 pools.
For each of our 50 topics, the same pool was completely independently judged by three assessors. Nine assessors were hired through the lancers website, and an additional ve assessors were hired at Waseda University, so that each topic was judged by two lancers and one student. We collected graded relevance assessments from each assessor: highly relevant (2 points), relevant (1 point), nonrelevant (0) and error (0) for cases where the web pages to judge could not be displayed. When consolidating the multiple assessments, the raw scores were added to form more ne-grained graded relevance data.
We use graded relevance measures at cuto 10
(representing the quality of the rst search engine result page),
namely nDCG@10, Q@10, and nERR@10 [
As our topics were sampled from a query log, none of our
assessors are the topic originators (or “primary” [
We discuss inter-assessor agreeement and system ranking
agreement using stastical tools, namely, linear weighted κ
with 95%CIs (which, unlike raw overlap measures, takes
chance agreement into account [
While the recent work of Maddalena et al. [
Krippendorph’s α disregards which assessments came from which assessors, as it is a measure of the overall reliability of the data. In the present study, where we only have three assessors, we are more interested in the agreement between every pair of assessors and hence utilise Cohen’s linear weighted κ. Hence our denition of a high/low-agreement topic diers from that of Maddalena et al.: according to our denition, a topic is in high agreement if the κ is statistically signicantly positive for every pair of assessors. While Maddalena et al. discussed the absolute eectiveness scores and system rankings only, the present study discusses statistical signicance testing aer replacing the full topic set with the high/low-agreement set.
Maddalena et al. focussed on nDCG; we discuss the three
aforementioned ocial measures of the NTCIR-13 WWW
task [
DATA e NTCIR-13 WWW English subtask created 100 test topics; 13 runs were submied from three teams. We acknowledge that this is a clear limitation of the present study: we would have liked a larger number of runs from a larger number of teams. However, we claim that this limitation does not invalidate neither our approach to analysing inter-assessor disagreement nor the actual results on the system ranking and statistical signicance. We hope to repeat the same analysis on a larger set of runs in the next round of the WWW task.
For evaluating the 13 runs submied to the WWW task, we created a depth-30 pool for each of the 100 topics and this resulted in a total of 22,912 documents to judge. We hired nine lancers who speak English through the lancers website: the job call and the relevance assessment instructions were published on the website in English. None of them had any prior experience in relevance assessments. Topics were assigned at random to the nine assessors so that each topic had two independent judgments from two lancers. e ocial relevance assessments of the WWW task were formed by consolidating the two lancer scores: since each lancer gave 0, 1, or 2, the nal relevance levels were L0-L4.
For the present study, we focus on a subset of the above test set, which contains 50 topics whose topic IDs are odd numbers. e number of pooled documents for this topic set is 11,214. We then hired ve students from the Department of Computer Science and Engineering, Waseda University, to provide a third set of judgments for each of the 50 topics. e instructions given to them were identical to those given to the lancers. e students also did not have any prior experience in relevance assessments. Moreover, lancers and students all received an hourly pay of 1,200 Japanese Yen. However, hiring lancers is more expensive, because we have to pay about 20% to Lancers the company on top of what we pay to the individual lancers. e purpose of collecting the third set of assessments was to compare the lancer-lancer inter-assessor agreement with the lancer-student agreement, which should shed some light on the reliability of the dierent assessor types. All of the assessors completed the work in about one month.
It should be noted that all of our assessors are “bronze”
according to the denition by Bailey et al. [
To quantify inter-assessor agreement, we compute Cohen’s linear weighted κ for every assessor pair, where, for example, the weight for a ¹0; 2º disagreement is 2 and that for a ¹0; 1º or ¹1; 2º disagreement is 15. It should be noted that κ represents how much agreement there is beyond chance.
Table 1 summarises the inter-assessor agreement results. e “raw scores” section shows the 3 3 confusion matrices for each pair of assessors; the counts were summed across topics, although lancer1, lancer2, and student are actually not single persons. e linear weighted κ’s were computed based on these matrices. It can be observed that the lancer-lancer κ is statistically signicantly higher than the lancer-student κ’s, which means that the lancers agree with each other more than they do with students. While the lack of gold data prevents us from concluding that lancers are more reliable than students, it does suggest that lancers are worth hiring if we are looking for high inter-assessor agreement. As we shall see in Section 4.1, the discriminative power results discussed in also support this observation.
We also computed per-topic linear weighted κ’s so that the assessments of exactly three individuals are compared against one another: the mean, minimum and maximum values are also provided in Table 1. It can be observed that the lowest per-topic κ observed is 0:128, for “lancer1 vs. student”; the 95%CI for this instance was » 0:262; 0:0006¼, suggesting the lack of agreement beyond chance6.
Table 1 also shows the number of topics for which the per-topic
κ’s were not statistically signicantly positive, that is, the 95%CI
lower limits were not positive, as exemplied by the above instance.
ese numbers indicate that the lancer-lancer agreements were
statistically signicantly positive for 50 8 = 42 topics while the
lancer-student agreements were statistically signicantly positive
for only 35 (31) topics. Again, the lancers agree with each other
more than they do with students.
5 Fleiss’ κ [
ere were 27 topics where all three per-topic κ’s were statistically signicantly positive; for the remaining 23 topics, at least one per-topic κ was not. We shall refer to the set of the former 27 topics as the high-agreement set and the laer as the low-agreement set. We shall utilise these subsets in Section 4.2.
Also in Table 1, the binary Cohen’s κ row shows the κ values aer collapsing the 3 3 matrices into 2 2 matrices by treating highly relevant and relevant as just relevant. Again, the lancerlancer κ is statistically signicantly higher than the lancer-student κ’s. Finally, the table shows the raw agreement based on the 2 2 confusion matrices: the counts of ¹0; 0º and ¹1; 1º are divided by those of ¹0; 0º, ¹0; 1º, ¹1; 0º, and ¹1; 1º. It can be observed that only the lancer-lancer agreement exceeds 70%.
We summed the raw scores of lancer1, lancer2, and student to form a qrels set which we call all3; we also summed the raw scores of lancer1 and lancer2 to form a qrels set which we call 2lancers. Table 2 shows the distribution of documents across the relevance levels. Note that all3 and 2lancers are on 7-point and 5-point scales, respectively, while the others are on a 3-point scale. In this way, we preserve the views of individual assessors instead of collapsing the assessments into binary or to force them to reach a consensus. Note that nDCG@10, Q@10, and nERR@10 can fully utilise the rich relevance assessments. As we shall see in the next section, this approach to combining multiple relevance assessments is benecial.
For alternatives to simply summing up the raw assessor scores,
we refer the reader to Maddalena et al. [
Dierent Qrels Versions e previous section showed that assessors do disagree, and that the lancers agree with each other more than they do with students. is section investigates the eect of inter-assessor disagreement on system ranking and statistical signicance through comparisons across the ve qrels versions: all3, 2lancers, lancer1, lancer2, and student.
4.1.1 System Ranking. Figure 1 visualises the system rankings
and the actual mean eectiveness scores according to the ve
different qrels, for nDCG@10, Q@10, and nERR@10. In each graph,
the runs have been sorted by the all3 scores, and therefore if every
curve is monotonically decreasing, that means all the qrels versions
produce system rankings that are identical to the one based on all3.
First, it can be observed that the absolute eectiveness scores dier
depending on the qrels version used, just as Voorhees [
4.1.2 Statistically Significant Dierences across Systems. e
next and perhaps more important question is: how do the dierent
qrels versions aect pairwise statistical signicance test results? If
the researcher is interested in the dierence between every system
pair, a proper multiple comparison procedure should be employed
to ensure that the familywise error rate is bounded above by the
signicance criterion α [
Table 4 shows the results of comparing the outcomes of signicance test results at α = 0:05: for example, Table 4(a) shows that, in terms of nDCG@10, all3 obtained 2 + 29 = 31 statistically signicantly dierent run pairs, while 2lancers obtained 29 + 2 = 31, and that the two qrels versions had 29 pairs in common. us the Statistical Signicance Overlap (SSO) is 29¹2 + 29 + 2º = 87:9%. e table shows that the student results disagree relatively oen with the others: for example, in Table 4(b), Q@10 with lancer1 have 11 statistically signicantly dierent run pairs that are not statistically signicantly dierent according to the same measure with student, while the opposite is true for ve pairs. e two qrels versions have 15 pairs in common and the SSO is only 48.4%. us, dierent qrels versions can lead to dierent research conclusions.
Table 5 shows the number of statistically signicantly dierent
run pairs (i.e., discriminative power [
In Section 3, we dened a high-agreement set containing 27 topics and a low-agreement set containg 23 topics. A high-agreement topic is one for which every assessor pair “statistically agreed,” in the sense that the 95%CI lower limit of the per-topic κ was positive. A low-agreement topic is one for which at least one assessor pair did not show any agreement beyond chance, and therefore deemed unreliable. While to the best of our knowledge this kind of close analysis of inter-assessor agreement is rarely done prior to evaluating the submied runs, removing such topics at an early stage may be a useful practice for ensuring test collection reliability. Hence, in this section, we focus on the all3 qrels, and compare the evaluation outcomes when the full topic set (50 topics) is replaced with just the high-agreement set or even just the lowagreement set. e fact that the high-agreement and low-agreement sets are similar in sample size is highly convenient for comparing them in terms of discriminative power.
4.2.1 System Ranking. Figure 2 visualises the system rankings and the actual mean eectiveness scores according to the three topic sets, for nDCG@10, Q@10, and nERR@10. Again, in each graph, the runs have been sorted by the all3 scores (mean over 50 topics). Table 6 compares the system rankings in terms of Kendall’s τ with 95%CIs. e values in bold indicate the cases where the two rankings are statistically not equivalent. It can be observed that while the system rankings by the full set and the high-agreement set are statistically equivalent, those by the full set and the lowagreement set are not. us, the properties of the high-agreement topics appear to be dominant in the full topic set. 4.2.2 Statistically Significanct Dierences across Systems. Table 7 compares the outcomes of statistical signicance test results (Randomised Tukey HSD with B = 10; 000 trials) across the three topic sets in a way similar to Table 4. Note that the two subsets are inherently less discriminative than the full set as the sample sizes are about half that of the full set. It can be observed that the set of statistically signifantly dierent pairs according to the high-agreement (low-agreement) set is always a subset of the set of statistically signifantly dierent pairs according to the full topic set. More interestingly, the set of statistically signifantly dierent pairs according to the low-agreement set is almost a subet of the set of statistically signifantly dierent pairs according to the highagreement set: for example, Table 7(b) shows that there is only one system pair for which the low-agreement set obtained a statistically signicant dierence while the high-agreement set did not in terms of Q@10. all3 (27 good topics) all3 (23 bad topics)
CONCLUSIONS AND FUTURE WORK is paper reported on a case study involving only 13 runs contributed from only three teams. Hence we do not claim that our nding will generalise; we merely hope to apply the same methodology to test collections that will be created for the future rounds of the WWW task and possibly even other tasks. Our main ndings using the English NTCIR-13 WWW test collection are as follows: Lancer-lancer inter-assessor agreements are statistically signicantly higher than lancer-student agreements. e student qrels is less discriminative than the lancers qrels. While the lack of gold data prevents us from concluding which type of assessors is more reliable, these results suggest that hiring lancers has some merit despite the extra cost.
Dierent qrels versions based on dierent (combinations of) assessors can lead to somewhat dierent system rankings and statistical signicance test results. Combining multiple assessors’ labels to form ne-grained relevance levels is benecial in terms of discriminative power.
Removing 23 low-agreement topics (in terms of inter-assessor agreement) from the full set of 50 topics prior to evaluating runs did not have a major impact on the evaluation results, as the properties of the 27 high-agreement topics are dominant in the full set. However, replacing the high-agreement set with the low-agreement set resulted in a statistically signicantly dierent system ranking, and substantially lower discriminative power. Hence, from a statistical point of view, our results suggest that a high-agreement topic set is more useful for nding concrete research conclusions than a low-agreement one.
I thank the PLY team (Peng Xiao, Lingtao Li, Yimeng Fan) of my laboratory for developing the PLY relevance assessment tool and collecting the assessments. I also thank the NTCIR-13 WWW task organisers and participants for making this study possible.