In this paper we present results of our initial research on aggregating noisy crowdsourced labels, by using a modi ed version of the EM algorithm introduced in [1]. We propose di erent methods of estimating the worker con dence, a measure that indicates how well the worker is performing the task, and of integrating it in the computation of the aggregated label. Furthermore, we introduce a novel method of computing the worker con dence by using the soft aggregated labels. In order to assess the e ectiveness of our proposed methods, we experiment on the MediaEval 2013 Crowdsourcing for Social Multimedia Task dataset.
In this paper we detail the methods proposed for the
MediaEval 2013 Crowdsourcing for Social Multimedia Task [
A similar approach was used for building probablistic
models [
In our methods the error-rate is replaced by the worker con dence, used as the weight of a worker contribution in the aggregated label computation. We attempt to improve the standard EM method by using di erent ways to boost the worker con dence, as well as proposing a novel method for computing it. We introduce the soft evaluation of the worker con dence, where the soft aggregated crowd decision is taken into account instead of the hard aggregated label.
In this section we detail the computation of the aggregated i decision of a crowd for the label of an instance i, Lcrowd (i.e. Yes or No), and of the worker con dence. We distinguish between two types of worker con dence depending on whether we make a discrimination between the quality of the positive and negative answers or not. In the case of such a discrimination each worker is characterized by a positive con dence Cu+ and a negative con dence Cu , otherwise we use a single value for the worker con dence, Cu. Majority voting means i Cu = 1. Lcrowd is computed by aggregating the individual worker labels Liu 2 fY es; N og, ignoring Not Sure labels.
In the E step we compute the aggregated crowd labels using Eq. 2 when discriminating between positive and negative and Eq. 1 otherwise, and in the M step we update the worker con dences as de ned in Eq. 3 or Eq. 4. 2.1
In case we do not discriminate between positive and negative answer quality, the probability of an instance being labeled as positive is: pi+ = Pu Cu I(Liu = Y es) + Pu Cu I(Liu = N o)
In case we di erentiate between the positive and negative answer quality this becomes: pi+ =
Pu Cu+ I(Liu = Y es)
Pu Cu+ I(Liu = Y es) + Pu Cu I(Liu = N o)
The probability of an instance being labeled as negative is obviously pi = 1 pi+. We will refer to the pi+ and pi as computed by using either method as aggregated soft labels. Moreover, the nal aggregated hard label assigned by the crowd is given by comparing the di erence between the positive probability and the negative one:
i Lcrowd =
Y es; pi+ N o; pi+ pi 0 pi < 0 2.2
The undiscriminative con dence in a worker is de ned as: Cu =
tpu + tnu tpu + tnu + f pu + f nu In case we discriminate between the quality of positive and negative answers we use two types of con dence: Cu+ =
tpu tpu + f pu ; Cu =
tnu tnu + f nu
We distinguish between two types of evaluation of the worker con dence: hard evaluation, where we use only the nal, aggregated hard labels, and a soft evaluation, where we use the aggregated soft labels.
In case of a hard evaluation of the performance of a user we use the following de nitions: tpu =
X I(Liu = Y es) I(Licrowd = Y es) i (1) (2) (3) (4) tnu = X I(Liu = N o) I(Licrowd = N o)
i f pu = X I(Liu = Y es) I(Licrowd = N o)
i f nu = X I(Liu = N o) I(Licrowd = Y es)
i
In the case of a soft evaluation of the worker con dence we use the following de nitions: tpu = X I(Liu = Y es) pi+; tnu = X I(Liu = N o) pi i i f pu = X I(Liu = Y es) pi ; f nu = X I(Liu = N o) pi+ i i
Furthermore we can apply the following corrections to the con dence when aggregating the multiple votes: boosting the con dence (C^ = boost(Cu)) or involving the worker selfreported familiarity with the category for which Label 2 is assigned to the image (f amiu) in the computation of the condence (Cu = Cu norm(f amiu)). Based on an observation of a correlation of the familiarity and the type of answers and their accuracy, we can also use a familiarity correction strategy
C^u = 80:6 f amiu < 3; Iu = Y es > ><0:9 f amiu < 3; Iu = N o >0:8 f amiu > 3; Iu = Y es >:0:8 f amiu > 3; Iu = N o The boosting function boost(x) can be ex or xp; p 2 R.
The transformation of familiarity from an integer within 1 and 7 or missing to a real subunitary positive number, is done by the norm(x) function. norm(x) = (x 1)=6 if x 2 N and 0:5 if missing.
The computation of the labels in the EM algorithm as well as of the nal decisions after the iterations are nished depend on the following settings: the use of positive/negative answer discrimination the evaluation of worker con dences using soft labels the boosting type the use of familiarity in the computation the use of the familiarity correction
For picking candidates for the submitted runs, and nding the best setting, we evaluated the performance of our methods on the MMSys 2013 Dataset. The selected settings that are used for the submitted runs are detailed in Table1.
The rst two runs use the discrimination between the positive and negative worker con dence. Run1 is using the EM algorithm with hard iterations for both labels. Run2 represents the EM algorithm using the soft iterations for both labels without any special boosting strategy or involving the familiarity.
The performance of each submission in terms of the F1measure is presented in Table 2. As already mentioned in 0.895 0.909 0.894 0.911 0.900 0.913 0.898 0.913 0.894 0.913 experiments carried out with the MMSys 2013 dataset, we notice a better performance in the case of the second label. We can see that for the rst label the best performance is achieved in Run5, and for the second label by Run3. We notice that in the case of Label 1, discriminating between positive and negative label quality provides a performance increase, while in the case of Label 2 the e ect is opposite.
This work was partially funded by the European Commission FP7 under grant agreements No. 287704 for the CUbRIK project.