This paper presents our approach for the Crowd Sourcing Task of the MediaEval 2014 Benchmark. The proposed solution is based on the assumption that the number of Human Intelligence Tasks (HITs) completed by a worker is representative of his diligence, making workers who completing high volumes of work more reliable than low-performing workers. Our approach gives a baseline evaluation indicating the usefulness of looking at the number of task completed by a worker.
Crowdsourcing creates a lot of opportunities and is gaining momentum as an area of interest within the multimedia community. Moreover, current web-based services like Amazon Mechanical Turk, Mircrowoker, and Crowd ower have simpli ed the task of leveraging the power of human computation.
The biggest problem in crowdsourcing is still the reliability of the workers. The information we receive using crowdsourcing is unreliable because of workers who try to trick the system, spam or simply don't understand the task properly. The law of large numbers (LLN) describes how noise is averaged and its e ects are removed with a large number of experiments, but increasing the number of experiments directly a ects costs. This is why the crowdsourcing exercise for the Crowdsorting Timed Comments Task this year focuses on computing correct labels based on noisy crowdsourced, metadata or content information.
Related work in this area can for example be found in [4, 3]. These approaches try to calculate correctness of the workers or use the features of the media les like the global image feature, for a classi cation.
In contrast, the proposed solution presented here is based on the assumption that workers who complete a high number of tasks are high performers, either because they enjoy the task or that they understand the task well enough to do it e ciently. We believe that both of these circumstances lead to reliable results with respect to HITs. As a secondary approach, we also used labels collected from additional crowdsourcing workers which means that we asked new workers for HITs where the original workers could not come to an agreement. 2.
This section describes our two approaches. As mentioned, our main approach is to nd the most diligent workers, while the second approach is based on the idea of collecting additional crowdsourcing votes. Quality control is a prerequisite of a well-designed crowdsourcing HIT and to increase the quality of votes for this work, the task organizers included a quali cation HIT to make sure that workers understood the task at hand. As the main task was to classify drops in music tracks, the workers had to prove that they could classify a drop correctly. Only the workers who passed the quali cation HIT were allowed to continue. Because of that pre-quality control, we did not perform any additional quality. 2.1
The idea of diligent workers is based on the work presented
by Kazai et al. [2], which describes ve di erent types of
workers: (
For the HITs without a clear result through majority vote between the three provided workers or by weighted subset of the best performing workers, we used additional crowdsourcing. We developed an HTML and SQL-based platform that gave us the opportunity to perform the tests in our lab. The requirement for this additional test was that the participants had to try their best to nd the right answer for the HIT. 3.
The provided dataset contains 591 songs, metadata, and
labels generated by human computation, but because some
of the songs are duplicates only 537 of them we used in the
evaluation. The task's main goal was to classify a drop in
music within a limited timespan. A drop can be seen as an
event that builds up to a change of the beat or melody in
the song, i.e., a characteristic also found in electronic dance
music, and is more than just a simple change. Workers could
give three di erent labels to each song segment: (
We assessed four di erent methods executed in four runs. The results are shown in Table 1 where a summarized overview and short descriptions of each method is provided. The rst method (R1) considers the majority vote (MJV) between the three votes provided by the original dataset and additional for not clear answers. While in the second run (R2) we only consider the votes provided by our diligent workers subset. Our third method (R3) takes into account the majority votes, but adds a higher weight to votes provided by diligent workers. The fourth and last method (R4) used the results provided by R3, but with additional crowdsourcing for ambiguous answers (where MJV could not clearly lead to a label). 4.
Table 2 describes our benchmark results, while Table 3 describes the results for the most frequent class baseline, in which case all labels get the most frequent class label in the dataset assigned. The performance is measured by the weighted harmonic mean of precision and recall (WF1score). This is done to avoid unreliable results based on the imbalance of the classes.
We see from Table 2 that every method evaluated outperforms the most frequent class baseline by at least 30%. The best performing method is R1 with a WF1-score of 0:7207. Compared to R1, the three other methods have a performance drop of around 3%. These methods are not distinguishable with respect to the results they produces, which might be because each of the methods rely on the votes provided by the subset of diligent workers. We nd it interesting that R3 and R4, which complements diligent workers with MJV and additional crowdsourcing, do not signi cantly increase performance compared to R2.
Moreover, the performance di erence between R1 and R2 is low, which strongly indicates that the assumption of workers who complete the majority of crowdsourcing tasks also perform better is valid. This is a promising insight that can cut costs and yield more accurate crowdsourcing results. For example, by identifying diligent workers early in task execution one can annotate their votes and only consider them as in R2, or weight their votes di erently as in R3. That said, we also want to point out that there is a chance our results are dataset speci c and further investigations on multiple and larger datasets are needed.
At last, we want to highlight that additional crowdsourcing does not increase the accuracy when considering diligent workers. This indicates that the quality of work and worker motivation is more important than the number of workers used or votes gathered. 5.
This paper presents two approaches for classifying drops in electronic dance music segments by utilizing human computation and crowdsourcing. The results and insights gained by evaluating four di erent methods indicate that the proposed approach, which assumes that diligent workers also provide better work quality, is promising. Our investigation also indicates that additional crowdsourcing does not improve results when considering diligent workers.
For assurance and increased certainty, we recognize the need for extending the work to include multiple and larger datasets. Additional future work includes pairing crowdsourcing results with computer generated content analysis and further classi cation of diligent workers. 6.
This work has been funded by the NFR-funded FRINATEK project "E cient Execution of Large Workloads on Elastic Heterogeneous Resources" (EONS) (project number 231687) and the iAD center for Research-based Innovation (project number 174867) funded by the Norwegian Research Council.