In crowdsourcing, a range task is a type of creation task where only free answers belonging to the numeric domain are accepted/possible. In this paper, we present the median-onagreement (ma) techniques based on statistical and consensusbased mechanisms for determining the result of range tasks. The ma techniques are characterized by i) the distinction between the group of workers that agree (i.e., workers in the consensus) on the task result from the group that disagree, and ii) the calculation of the nal task answer through a median-based mechanism where only answers of workers in the consensus are considered.
In the recent years, crowdsourcing systems have gained
growing popularity as powerful solutions for addressing the
execution of complex, time-consuming activities where the
contribution of human workers can be decisive and the use
of automatic procedures is not completely e ective, such as
for example collaborative ltering and web-resource tagging.
Usually, in this kind of systems, crowd workers are involved
in decision tasks where they are called to select the most
appropriate answer among a set of prede ned alternatives
(e.g., [
In this paper, we focus on range tasks, namely a type
of creation task where only free answers belonging to the
numeric domain are accepted/possible [
The paper is organized as follows. In Section 2, we illustrate motivations and related work. The ma techniques are presented in Section 3. In Section 4, the application of ma to Argo is discussed. In Section 5, experimental results on a real crowdsourcing case-study are presented. Concluding remarks are provided in Section 6. 2.
Consider the scenario described in [
This is an example of a range task, in that a worker
receiving a task to execute can only provide a free numeric
answer, namely integer or decimal value, based on her/his
personal point-of-view, knowledge, perception, and
expertise. This means that no prede ned options/suggestions are
available and workers are called to independently and
autonomously provide her/his own task answer. Moreover,
the real amount of calories in a dish (i.e., in a task) is not
available/known and only a collective answer is possible [
An intuitive and popular solution for range task resolution
is to employ a mean-based approach in which multiple
workers are involved in the execution of each task and the
arithmetic mean of the whole set of worker answers is provided
as nal result [
In other words, the numeric answer of a single worker that diverges (i.e., it is very di erent) from the other more-orless equivalent worker answers has a strong in uence on the nal task result. This means that a single worker can autodetermine her/his impact on the task result independently from her/his trustworthiness. This is especially true when the group of workers involved in a task execution is small (i.e., 5-10 workers per group) and malicious or inaccurate workers can be involved as usually occurs in real systems.
Further work on resolution of range tasks are presented
in [
In the following, we propose consensus-based techniques for managing range task resolution based on two main contributions. First, use of the median value (instead of the arithmetic mean) to determine the task result which is representative of the multiple answers collected from the involved workers. Second, use of consensus as a mechanism for distinguishing workers that agree on the task result from workers that disagree and represent a sort of outlier position.
Consider a range task T assigned to a group of workers G = fw1; : : : ; wng providing a set of answers A = fa1; : : : ; ang where ak 2 A is the numeric answer provided by the worker wk 2 G. Range task resolution according to the ma techniques is articulated in two main steps: identi cation of the support group and de nition of the nal task result described in the following.
Identi cation of the support group. We call GCA1 G the support group of G, namely the group of workers that agree on the task result. Two workers agree on the task result when they provide a similar numeric answer, meaning that the values provided in the task answer are near in comparison with the overall range of answers A provided by all the workers in G. We call ACA1 A the set of task answers provided by the workers in GCA1 . Consider the median value mA of all the provided worker answers A. The group GCA1 is progressively built by including workers that provided an answer close to mA, namely: 1. Compute the median mA over the whole set of worker answers A and de ne GCA1 = ;, ACA1 = ;. 2. Select the worker answer ak 2 A which is nearest to mA. Insert ak in ACA1 and insert the worker wk in the support group GCA1 . 3. The coe cient of variation cv is exploited to decide whether an answer ak 2 A is near enough to mA for being included in GCA1 . To this end, cv is calculated over the set of answers in ACA1 : cv(ACA1 ) = r
1 jACA1 j
PjiA=C1A1 j ai
ACA1
2 ACA1 where jACA1 j is the number of answers in ACA1 , ai represents the ith worker answer in ACA1 , and ACA1 represents the arithmetic mean of the answers in ACA1 . 4. The insertion of workers in GCA1 is repeated until the coe cient of variation over the answers ACA1 is lower than a threshold thcv (i.e., go back to step 2 if cv(ACA1 ) < thcv). Otherwise, remove the lastinserted item from GCA1 and ACA1 and continue with the next step. 5. Create the set GCA2 = G n GCA1 containing the workers that are not in the support group. Analogously, the set ACA2 = A n ACA1 is created as well.
De nition of the nal task result. The nal task result A is de ned as the median value calculated over the set of worker answers ACA1 , namely A = mACA1 .
Example. Consider a task T1 where workers are asked to guess the distance between the two Italian cities Caserta and Siena in kilometers (the real distance is 352 Km). Consider the following set of worker answers: A = f300, 300, 301, 301, 350, 351, 351, 351, 351, 400, 408, 408, 450, 500, 600, 600, 600, 650, 700, 1500g. The median value over the whole set of worker answers mA = 404. According to ma, we consider a threshold for the coe cient of variation thcv = 0:15 and we identify the support group GCA1 shown in Figure 1. With this support group, the median value of the answers provided by workers in the support group is returned as nal task result: A = mACA1 = 351.
The ma techniques have been implemented in the Argo
crowdsourcing platform (http://island.ricerca.di.unimi.it/projects/
argo/ (Italian language)). In Argo, range task resolution is
enforced through consensus-based evaluation techniques and
trustworthiness-based worker management by relying on our
experience and research results in this eld [
Consensus-based evaluation of range tasks. For
consensus evaluation, Argo employs a weighted-voting
mechanism called supermajority where the answer of a worker
wk has a weight corresponding to her/his trustworthiness.
Supermajority is based on the veri cation of two di
erent constraints called quorum-constraint (q) and
balance-ofpower constraint (bop). The q-constraint veri es that the
task result A is supported by a group of workers GCA1 with
enough weight (i.e., trustworthiness) for satisfying a given
quorum q 2 [0:51; 1]. The bop-constraint veri es that a
single worker cannot shift the majority from one answer to
another one just by changing her own task answer [
Trustworthiness-based worker management. The Argo system aims at keeping into account not only the mere e ort workers spent in executing tasks, but also the quality of the e ort provided. A worker W is characterized by a worker score W , and a worker trustworthiness W .
The worker score W represents the worker revenue composed by i) a salary, the payment the worker receives each time she/he executes a task, regardless of the consensus veri cation, and ii) an award, a bonus the worker receives each time she/he contributes to commit a task.
The worker trustworthiness W 2 [0; 1] is de ned to capture the worker ability to foster the task commitment and it is based on the worker history in executing tasks. At the beginning of the crowdsourcing activities (time t = 0), the worker trustworthiness W is set to an initial value W0 = 0. Each time a task T is committed (time t + 1), the trustworthiness of a worker W 2 G is updated. In particular, the worker trustworthiness increases (i.e., Wt+1 > Wt ) when the worker belongs to the support group (i.e., W 2 GCA1 ), thus con rming her/his ability to foster task commitment in the last-executed task T . On the opposite, the worker trustworthiness decreases when the worker is not in the support group (i.e., W 2= GCA1 ).
For evaluation of the proposed ma techniques, we consider the geo-dis case-study for crowdsourcing the geographic distance between pairs of Italian cities.
The experiment has been executed by relying on the Argo prototype. We collected a dataset of 120 Italian cities with their geographic coordinates extracted from the FreeBase (http://www.freebase.com) open repository. We built a set of 634 tasks each one asking for the distance between a pair of di erent cities. The experimentation on geo-dis was conducted with a crowd of 585 workers selected in a class of master-degree students (average worker age is 21 years old). For task resolution, we asked the workers to rely on their personal knowledge and we set the allowed time to perform a task to a maximum of 15 minutes. In the experimentation, the Argo prototype has been con gured as follows: i) initial worker trustworthiness 0 = 0:5; ii) group size sG=20; iii) quorum value q = 0:51; iv) the worker salary is s = 0:1 and the worker award is a = 1.
Evaluation is based on two di erent experiments over the geo-dis case-study. The former experiment presents a comparison of the ma techniques implemented in the Argo system (maArgo) against other state-of-the-art techniques for range task resolution. The latter experiment is performed to evaluate the crowdsourcing cost of the ma techniques by measuring the number of committed/uncommitted tasks. Comparison against state-of-the-art techniques. We compare maArgo against the following competitor techniques:
Overall arithmetic mean O. This method refers to the
classical approach proposed in [
Outlier-cleaned arithmetic mean - Standard Deviation 2SD.
This method consists in applying a classical outlier removal
technique based on the standard deviation (2SD) [
Outlier-cleaned arithmetic mean - Median Rule MR. This
method consists in applying a more recent outlier removal
technique based on the median rule [
Overall median mO. This method consists in computing the result of a task T as the median value of all the provided answers. As far as we know, state-of-the-art techniques based on the median value are not provided. However, we compare maArgo against mO since this is the natural baseline for our ma techniques.
In the evaluation, we consider maArgo under three congurations characterized by di erent thresholds for the coe cient of variation thcv. Results are evaluated through average-error and average-error with outlier-removal mechanisms. In the average error mechanism, for each task T , the evaluation considers the error between the distance estimation in the crowdsourcing result A and the real distance between the two cities contained in T . The average error A is calculated as:
A =
Pii==1n jAi
Rij
jT j
where n = jT j is the overall number of tasks, Ai is the
crowdsourcing result of the task Ti, Ri is the real distance
between the pair of cities in the task Ti calculated through the
geodesic distance. In the average-error with outlier-removal,
the error evaluation follows the same approach of A
calculation, but outliers are removed according to the conventional
criterion based on standard-deviation (2SD) [
The results of this experiment are presented in Table 1. The rst consideration that has to be done is about the re
O 2SD MR mO maArgo (thcv = 0:25) maArgo (thcv = 0:15) maArgo (thcv = 0:05) sult of the O technique. The fact that the obtained average error is so high is mainly due to the presence of malicious workers in a very high number of groups. These malicious workers gave completely wrong answers (e.g., 10 millions kilometers as distance between Rome and Milan) that have a very serious impact on the task result when the arithmetic mean is considered and outlier removal is not performed. We note that the median-based techniques (i.e., mO and maArgo) provide better results than the techniques based on the standard deviation. We argue that this is due to the assumption of symmetric distribution used in O, 2SD, and MR, which is is usually false (e.g., see for example the task presented in Figure 1). As a general remark, we observe that the median-based solutions provide better results than mean-based techniques even without the outlierremoval phase. By considering the maArgo results with the di erent thresholds on the coe cient of variation, we note that the lower is the threshold thcv, the lower is the average error . This means that a more restrictive mechanism for determining the support group GCA1 increases the accuracy of obtained results.
Analysis on the task commitment We observed that a low value of thcv produces a low average error . However, on the opposite, a low value of thcv also produces a high number of uncommitted tasks, and thus high expenses for crowdsourcing execution. For this reason, in this experiment, we analyze the number of committed tasks when di erent thresholds on the coe cient of variation are considered. To this end, we de ne the commitment ratio as follows: c =
Nc
Nc + Nu where Nc is the number of committed tasks and Nu is the number of uncommitted tasks.
The commitment ratio for di erent coe cient of variation thresholds thcv are presented in Table 2. We note that the lower is the coe cient of variation threshold, the lower is the value of commitment. This behavior is motivated by the fact that the lower is the coe cient of variation, the more restrictive is the mechanism for determining the support group GCA1 . As a result, it is important to con gure the crowdsourcing execution by tuning the threshold thcv with the goal to set the desired tradeo between accuracy of results and commitment ratio. In the geo-dis case study, the threshold value thcv = 0:15 provides the best tradeo between accuracy (i.e., almost twice value on accuracy with respect to the other threshold values) and commitment ratio (i.e., c 90 %).
In this paper, we presented the ma techniques for range task resolution in crowdsourcing systems. Application to the Argo system as well as experimental results on a real case-study are provided to show the contribution of the proposed solution with respect to the state-of-the-art. Ongoing work are focused on the so-called task routing problem with the goal to specify a family of con guration patterns for dynamically choosing the most appropriate group of workers that can be selected for assignment of a given task to be executed based on worker expertise and knowledge. 7.