The 2013 MediaEval Crowdsourcing task looked at the problem of working with noisy crowdsourced annotations of image data. The aim of the task was to investigate possible techniques for estimating the true labels of an image by using the set of noisy crowdsourced labels, and possibly any content and metadata from the image itself. For the runs in this paper, we've applied a shotgun approach and tried a number of existing techniques, which include generative probabilistic models and further crowdsourcing.
Crowdsourcing is increasingly becoming a popular way of extracting information. One problem with crowdsourcing is that the workers can have a number of traits that a ect the quality of the work they are performing. One standard way of dealing with the problem of noisy data is to ask multiple workers to perform the same task and then combine the labels of the workers in order to obtain a nal estimate.
Perhaps the most intuitive way of combining labels of
multiple workers is through majority voting, however other
possibilities exist. The aim of the 2013 MediaEval
Crowdsourcing task [
As described previously, the overall methodology for our run submissions was to take a shotgun approach and try three fundamentally di erent approaches (generative probabilistic model of workers; extra crowdsourcing; and joint modelling) to the problem. The techniques and data we used for each run are summarised in Table 1. Speci c details on each run are given below. l W N 2.1
Run 1
The rst run was required to only make use of the provided
crowdsourced labels. For this run, we applied the generative
model developed by Paul Mineiro [
Run 2
For the second run, we gathered additional data in two ways. Firstly, we randomly selected 1000 images from the test set and had them annotated by two reliable experts. The two experts rstly annotated the data independently and came to agreement on 671 of these (across both questions). For the images they didn't agree on for either question, they collaboratively came to a decision about the true label for 1http://code.google.com/p/nincompoop/downloads/ both questions. The relatively low-level of initial agreement between the experts is an indication of the subjectiveness of the labelling task being performed (especially with respect to question 1 \is this a fashion image"). Secondly, for the images in the test set that had at least two \NotSure" answers, we gathered more responses through additional crowdsourcing using the CrowdFlower2 platform. In total we gathered additional an 824 responses over 421 images from this extra crowdsourcing. In order to produce the estimates we performed majority voting.
Run 3
In the third run, we applied the model used in run 1 to the data in run 2. The original worker labels and additional crowdsourced labels were combined and used as the primary input. The expert labels were used to clamp the model at the respective images in order to obtain a better t. 2.4
Run 4
In the fourth run, we chose to explore the use of another
generative model developed by Paul Mineiro [
Mineiro developed an online procedure to learn the model parameters that jointly learns a logistic regressor to learn how to create classi cations (estimations of the true label) from the features. A nice feature of this approach is that in each iteration of learning/ tting, the worker model informs the classi er and the classi er informs the worker model.
For this run, the features used were bag-of-words features extracted from the tags, titles, descriptions, contexts and notes metadata of each image. 2.5
Run 5
Finally, for the fth run, we applied the same technique as used in run 4, but also incorporated a Pyramid Histogram of Words (PHOW) feature extracted from the images themselves on top of the metadata features. The PHOW features were created from dense SIFT features quantised into 300 visual words and aggregated into a pyramid with 2 2 and 4 4 blocks.
The results of the ve runs are shown in Table 2. Whilst we can't currently make global comments as to how well these runs performed compared to nave majority voting, we can note a few points. Firstly, looking at runs 2 and 3 which used the same data, we can see that the generative Probabilistic
Joint 0.7636 0.7621 0.7710 0.7528 0.6026 model used in run 3 had a minor improvement for the second label, but it had a big negative e ect for the rst label. It's also clear that the more advanced model (runs 4 & 5), that took into account features, also performed less well with this data than hoped. Interestingly, when we applied both generative models to the smaller MMSys dataset we had a slight improvement. One possible reason for the relatively low performance of the generative models on the rst label could well be due to the subjectiveness of the question being asked, which would lead to errors when tting the models. This would also help indicate why additional crowdsourcing seems to improve results. 4.
The described work was funded by the Engineering and Physical Sciences Research Council under the SOCIAM platform grant, and European Union Seventh Framework Programme (FP7/2007-2013) under grant agreements 270239 (ARCOMEM) and 287863 (TrendMiner).