We extend the square benchmark for statistical consensus methods to include additional evaluation on two datasets from the MediaEval 2013 Crowdsourcing in Multimedia shared task. In addition to reporting shared task results, we also analyze qualitatively and quantitatively performance of consensus algorithms under varying supervision.
ALGORITHMS
We extend square1 [
Majority Voting (MV) uses random tie-breaking to avoid
bias in absence of an informative prior for tie-breaking.
ZenCrowd (ZC) [
DATA AND EXPERIMENTAL SETUP
Data. Consensus algorithms are evaluated on the MMSys
2013 and the test fraction (20%) of Fashion 10000. Both
datasets elicit binary judgments on the same set of tasks
from Amazon Mechanical Turk (AMT) workers. Task 1 asks
workers to identify an image as being fashion-related or not,
and Task 2 asks workers to indicate whether or not a desired
fashion object is present. For further details of tasks, see [
MMSys 2013 Task 2 Accuracy % of Questions “Gold” data is defined for these tasks by majority voting of trusted annotators over examples with a clear majority label.
Experiment. Following the same benchmarking
procedure as in SQUARE [
In the unsupervised setting, uninformed, task-independent hyper-parameters and class priors are unlikely to be optimal. While one might optimize these parameters by maximizing likelihood over random restarts or grid search, we do not attempt to do so. Instead, with light-supervision, we assume no examples are labeled, but informative priors are provided (matching the training set empirical distribution). Full-supervision assumes gold-labeled examples are provided. To evaluate ZC, RY, DS and GLAD methods under full-supervision, labels are predicted for all examples (without supervision) but replaced by gold on training data. 3.
RESULTS AND DISCUSSION
MMSys 2013. Average performance over Tasks 1 and 2 are reported in Table 1. We highlight best scoring methods but note these are not necessarily statistically significant.
Results show close to constant performance (roughly 91MV ZC NB DS RY
Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 Acc F1 92%) across methods, metrics, and varying supervision. Average accuracy of 91.17% across methods, with minimal variance, is perhaps not surprising given the abundance of high quality workers for these tasks (see Figure 1 for details). We hypothesize three key factors. 1. the response redundancy is limited at most three, with many instances recording only two responses (since workers could select ‘not sure’ rather than provide a binary judgment). 2. a high percentage of the work was completed by relatively small percentage of the workers (Figure 1). 3. Consistently high worker accuracies limit the value of weighted worker voting vs. simple MV.
We further observe task 1 and 2 to be highly correlated. Table 2 shows the confusion matrix of gold labels across the two tasks. Evaluation on Task 2 using Task 1 gold labels for supervision achieved 82.89% average accuracy and 84.98% average F1 across methods. This presents a possibility of joint estimation we do not investigate.
Fashion 10000. Table 3 reports F1 scores on the test fraction of Fashion 10000; since gold data was not available for the blind shared task, we only present results estimated with no supervision. MV is seen to score best on Task 1, and CUBAM on Task 2. This contrasts findings on MMSys 2013, though we still observe only a 1% threshold across methods on Task 1. However, Task 2 shows markedly superior performance of CUBAM, which remains to be investigated. Acknowledgments. This work is supported by National Science Foundation grant IIS-1253413 and by a Temple Fellowship. Opinions expressed in this work are those of the authors and do not reflect views of the sponsors.
M V
ZC GLAD
DS
RY CU BAM
Task 1
Task 2