In this paper, we present an initial implementation with experimental results of online pro le learning in Argo+, a framework for crowdsourcing task routing characterized by i) feature-based representation of both tasks and workers, and ii) learning techniques inspired to Rocchio relevance feedback for prediction of the most appropriate task to execute by a given worker.
In the recent years, the crowdsourcing philosophy has gained a lot of
attention and many crowdsourcing systems/platforms appeared on the web scene for
satisfying the growing need of marketplaces where the o er of requesters
providing jobs to execute can meet the work-force provided by the crowd. Humans
have di erent knowledge and abilities, thus a crowd worker can be trustworthy
on a certain task campaign that is coherent with her/his attitudes, as well as
she/he can be inaccurate on another campaign with di erent topics and skill
requirements not compliant with her/his attitudes. As a result, the capability
to e ectively discover and represent the pro le of engaged crowd workers is
becoming a strategic asset of emerging crowdsourcing marketplaces. The goal is to
selectively choose a quali ed and motivated crowd to recruit/involve in a given
campaign according to the required knowledge/abilities based on the features of
the tasks to execute. In this direction, the use of machine-learning techniques in
crowdsourcing applications is being appearing in the recent literature for
mining emerging worker skills from the analysis of executed tasks [
In this paper, we present an initial implementation of online pro le learning
in Argo+, a framework for crowdsourcing task routing characterized by i)
featurebased representation of both tasks and workers, and ii) learning techniques
inspired to Rocchio relevance feedback for prediction of the most appropriate task
to execute by a given worker. A detailed description of the Argo+ framework is
provided in [
The paper il organized as follows. Section 2 presents the basic elements of the proposed Argo+ implementation. Experimental results are then discussed in Section 3. Concluding remarks are nally provided in Section 4. 2
A crowdsourcing campaign is characterized by a crowd of workers W = fw1; : : : ; wkg involved in the execution of a set of tasks T = ft1; : : : ; tng. A task t 2 T is dened as t = hidt; at; mt; dt; Fti, where idt is the unique task identi er, at is the task action, mt is the task modality, dt is the task description, and Ft is the set of task-features. A task action at denotes the task target, namely the goal that needs to be satis ed through crowd execution (e.g., picture labeling, movie recognition, sentiment evaluation). A modality mt represents the kind of worker answer required in task execution (e.g., creation, decision). A description dt represents the task request given to each worker for illustrating what is demanded to her/him in the task execution. A set of task-features Ft manually associated with the task for providing a description of task requirements, namely a speci cation of the capabilities expected from a worker for being involved in the execution of the task t. For each feature f 2 Ft, a task-feature weight !(f ) is associated to denote the relevance of f within the task-features Ft. A worker w 2 W is de ned as w = hidw; Fwi, where idw is the unique worker identi er and Fw is the worker pro le expressed as a set of worker-features. A worker-feature f 2 Fw denotes a worker capability, either knowledge expertise or skill, and it is associated with a worker-feature weight !(f ) denoting the \degree" of expertise/ability associated with the worker. 2.1
Assigning tasks to workers
For enforcing task routing, Argo+ relies on a task classi cation procedure for
aggregating the tasks T to execute into K classes, so that tasks with similar
features Ft are associated with a same class. In the proposed implementation
of Argo+, probabilistic topic modeling are exploited for task classi cation. The
choice is motivated by the need to enforce a soft aggregation mechanism, where
a task with a plurality of features can have multiple associated classes and it can
be exploited by workers with di erent expertise, each one focused on a di erent
class. In particular, the proposed solution is characterized by the use of
Latent Dirichlet Allocation (LDA) [
Consider a worker w and associated worker pro le Fw. When w asks for a task
t to execute, the probability distributions ( ; ) created by task classi cation are
exploited. Through , Argo+ calculates the maximum a posteriori estimation w
given the worker features Fw. This is done by using collapsed Gibbs sampling [
P !(f )k f2Fw
P P !(f )j f2Fw j2K ; (1) (3) where !(f )i denotes the weight of features of type f that have been assigned to class i. Then, from the distribution w, we select the class z and task t such that: z = arg max wz: z2K (2) t = arg max tz: t2T We stress that a task t is available for assignment until the number of task executions expected by the system is reached, then it is marked as nished and it is excluded from the assignment mechanism.
Example 1. Consider to enforce Argo+ on a system with task classi cation based on K = 10 and a set of thematic tags used as features both for tasks and workers. A worker w asks for a task to execute and the pro le Fw is de ned by the following features:
Fw = h (web search, 0.85), (classi cation, 0.85), (smartphone, 0.51), (text, 0.34), ... i Starting from Fw, we exploit Equation 1 in order to classify the worker w with respect to the classes K. The resulting distribution w is:
From w, we exploit Equation 2 to select the most relevant class for the worker pro le, that is k = 2. The top-3 features associated with k = 2 in 2 are: classi cation, tweets, and web search, which motivates the relevance of the class with respect to the worker pro le Fw. Given the class, it is now possible to exploit Equation 3 in order to select a task t for worker execution. The features Ft of the task selected for assignment to w are Fw = h (web search, 1.0), (classi cation, 1.0), (smartphone, 1.0)i 2.2
Learning worker pro les
Given a task t executed by a worker w, we need to assess the quality of the
provided worker answer for deciding how to update the worker pro le, and thus
how to enforce learning. We call (t) the nal task result determined by the
crowdsourcing system. We note that di erent solutions can be employed for
determining (t). Popular solutions are based on majority voting mechanisms
where the nal task result corresponds to the answer that obtained the majority
of preferences by the involved workers. Alternative solutions are also possible,
such as for example statistics-based techniques [
For updating a worker pro le, Argo+ relies on learning techniques inspired
to the Rocchio relevance feedback [
Fw = h (web search, 0.78), (classi cation, 0.78), (smartphone, 0.74), (text, 0.03), ... i
We note that the three features of t a ects the worker pro le by changing the relative feature weights. Features web search and classi cation remain the most relevant, but the weight of smartphone that is a feature of t is increased. On the opposite, the feature text of Fw becomes remarkably less relevant in the new worker pro le, due to the fact that it is not part of the task feature set Ft. After the pro le update, Argo+ will exploit the new worker pro le for the subsequent task assignments to w. 3
For evaluation, we present some preliminary experimental results based on the comparison of the proposed Argo+ implementation against a basic task routing mechanism called from now on baseline. 3.1
Experiment setup Our experiment relies on a crowdsourcing campaign (named paintings) run through the Argo crowdsourcing system (i.e., the system version implementing baseline routing) between November and December 2018. The experiment involved 367 students from the Faculty of Arts and Literature at the University of Milan, who have been asked to examine a dataset of paintings in order to choose for each painting the correct author among a choice of six possible painters. The paintings dataset is composed by 948 paintings from 56 di erent authors spanning from the 13th century to the 20th century. Each task has been executed by more than one worker, for a total of 8,573 executions. The fact that the paintings dataset is featured by a correct answer for each task (i.e., the correct name of the painting author) makes it possible to easily evaluate the e ectiveness of the work done by each worker (i.e., successful contribution) in terms of number of correct answers given to the tasks question.
To setup the experiment for evaluating Argo+, we compare the success rate of the baseline execution of paintings against the success rate obtained through two di erent executions of paintings in Argo+: i) one execution with a at worker pro le (called Argo+nopro le) where !(f ) = 0 is initially de ned for each feature (i.e., worker-feature), and ii) one execution with a custom worker pro le (called Argo+pro le) where !(f ) = 1 for each feature on which the worker has declared a competence. Competences declared by workers have been collected through a self evaluation questionnaire about knowledge of painters and di erent periods in the art history. Task and worker features have been taken from Wikidata (https://www.wikidata.org) and they include the name of the author, the
WORKER OPTIONS
Raffaello Sanzio Gustav Klimt Piero della Francesca Francisco Goya Giotto Michelangelo Buonarroti
TASK FEATURES Raffaello Sanzio 1516 High Renaissance Portrait paintings of cardinals
EXAMPLE OF WORKER ANSWER { "gold_answer" : "Q5597", "argo_answer" : "Raffaello Sanzio", "worker_answer_id" : "Q5432", "worker_answer" : "Francisco Goya", "task_id" : 1102, "answer_timestamp" : 2017-11-13T14:42:19, "worker_id" : 527, "task_refused" : false } year, and the Wikidata thematic categories available for a painting. An example of task and worker answer is given in Figure 1.
The goal of our experimental evaluation is to assess whether Argo+ improves the success rate with respect to baseline. In order to simulate the execution of Argo+nopro le and Argo+pro le on exactly the same set of workers and answers used in baseline, our experiments are based on the idea to change the timesequence of tasks executed by each worker in baseline according to the assignment schedule determined by Argo+. We aim at verifying whether the tasks successfully executed by a worker w are assigned to w before than others. Being the task answers the same for all the experiments, the overall success rate will be the same as well. However, if Argo+ performs better than baseline, we expect to execute correct tasks before. In other terms, we aim at verifying if Argo+ reaches a success rate better than baseline by taking into account the rst r tasks assignments. In particular, we call r (request timestamp), the timestamp at which the crowdsourcing system receives the request for a task to execute by a worker, and we call (r; ) the success rate of the system execution at the request timestamp r. A system execution is a stream of task answers, each one collected from a worker at a certain timestamp. In our evaluation, baseline is the reference system execution, while Argo+nopro le and Argo+pro le represent alternative system executions of baseline where the time-sequence of task answers is changed according to the Argo+ routing mechanism. The success rate (r; ) is de ned as follows:
r (a) (r; ) = 1r iP=1 (w; t)i ; (b) R = R1R (r; )dr where (w; t)i in (a) is the worker-task result received by the system at the ith request timestamp in the execution and (b) measures the overall system performance of task routing with R representing the overall number of successfully executed tasks in a system execution (i.e., R is the sum of all the (w; t) = 1 in ). Given two di erent system executions and , the delta value r( ; ) represents how much the success rate of changes with respect to at time r. The delta value r( ; ) is de ned as r( ; ) = ((rr;; )) . 3.2
Considerations Experiments have been performed with a number of classes K = 30 for task classi cation and a dumping factor = 0:3 for worker pro le learning. The com1.0 0.8 ,) (r0.6 e trsscaeuS c0.4 0.2 0.0 parison of baseline against Argo+nopro le and Argo+pro le on success rate (r; ) and delta value ( ; ) are shown for the rst 200 tasks requests in Figures 2(a) and 2(b), respectively. We observe that both Argo+noPro le and Argo+Pro le sucBaseline Argo+ noprofile Argo+ profile
Argo+ noprofile
Argo+ profile e lin6 saeb e 5 h tttsceep4 o itr h I,)(w3 e cn m2 a frr o e p ft1 o n e Inm0 0 e rc 0 25 50 75 Task request 125 150 175 200 100 25 50 75 Task request 125 150 175 200
100 (a) (b) ceed in improving the success rate of baseline, since successfully executed tasks are assigned to workers before other tasks in most cases. For the rst 200 requests, the success rate of Argo+Pro le is around 20% better than baseline. It is also interesting to note that, at the very beginning of the system execution (r < 50), the behavior of Argo+noPro le and Argo+Pro le is very unstable since learning has insu cient information for recognizing the appropriate task class for each worker. However, Argo+ quickly learns the worker pro le (r 50) and this has a positive impact on the assignment of subsequent tasks. The performance of Argo+noPro le becomes similar to baseline after the 300th worker request. This is due to the fact that Argo+ rst selects tasks that are highly relevant for the worker pro le, while subsequent assignments are about residual tasks of the K classes for which the relevance for the worker pro le is weaker.
Finally, we compare baseline and Argo+ through R and we obtain that R = 399:59 for baseline, R = 424:66 for Argo+Pro le, and R = 399:61 for Argo+noPro le. As a result, we observe that the use of a questionnaire for initializing the worker pro le provides the best performance on the three considered system executions (see also Figure 2(b) on the increment value). However, after a small number of executions, the performance of the learning system without the initial set-up of the worker pro le becomes similar to the one of the system execution initialized with the questionnaire. This con rms the intuition behind the use of at pro les which argues that the auto-evaluation of worker skill/abilities could be misplaced with respect to the real worker expertise, and thus sometimes damaging the performance of the crowdsourcing system.
In this paper, we presented an implementation of pro le learning techniques in
Argo+ crowdsourcing framework [