Using crowdsourcing platforms such as CrowdFlower and Amazon Mechanical Turk for gathering human annotation data has become now a mainstream process. Such crowd involvement can reduce the time needed for solving an annotation task and with the large number of annotators can be a valuable source of annotation diversity. In order to harness this diversity across domains it is critical to establish a common ground for quality assessment of the results. In this paper we report our experiences for optimizing and adapting crowdsourcing microtasks across domains considering three aspects: (1) the micro-task template, (2) the quality measurements for the workers judgments and (3) the overall annotation work ow. We performed experiments in two domains, i.e. events extraction (MRP project) and medical relations extraction (Crowd-Watson project). The results con rm our main hypothesis that some aspects of the evaluation metrics can be de ned in a domainindependent way for micro-tasks that assess the parameters to harness the diversity of annotations and the useful disagreement between workers. This paper focuses speci cally on the parameters relevant for the 'event extraction' ground-truth data collection and demonstrates their reusability from the medical domain.
At the basis for machine learning and information retrieval systems is the
collection of ground truth data. Typically, creating such a gold standard dataset
requires domain expert annotations to ensure high quality of the training and
evaluation data. However, expert-annotation may result in limitedly annotated
datasets, which do not capture the evolution of human expressions and the
diversity in their interpretations. With its large pool of human workers,
crowdsourcing became a mainstream source for higher volume and continuous collection of
training and evaluation data (specically for tasks that do not require domain
expertise). Thus, the new challenge became to correctly and eciently
identifying low quality or spam contributions of the micro-workers. Research shows that
micro-workers’ behavior (e.g. either as intentional spam or low quality
contributions) can inuence the overall quality of the nal results [
Recently, however, there is evidence to support the hypothesis that
harnessing diversity and disagreement between workers can improve the ground truth
data [
We conducted experiments in two domains - starting with medical relation extraction in the context of Crowd-Watson project 4 and adapting the experiences to event extraction in the context of DARPA’s Machine Reading program (MRP)5. We used the same workow in both domains: (1) pre-processing of input data and micro-task template design , (2) data collection through automatic sequencing of micro-task batches , (3) disagreement analytics through quality metrics on workers judgments and (4) post-processing of the results for spam ltering and micro-task template adaptation . The novel contribution of this work is twofold - on the one hand demonstrating a crowd truth collection workow optimized for multiple domains; and on the other hand providing reusable disagreement-harnessing micro-task templates with the corresponding spam detection disagreement metrics.
The rest of the paper is organized as follows. Section 2 places this work in the context of crowdsourcing, evaluation metrics and event extraction. Section 3 presents the Crowd-Watson workow and shows its adaptation for the event extraction task. Section 4 presents the experimental setup and Section 5 discusses the results. Section 6 draws the conclusions and presents the future work. 2
The amount of knowledge that crowdsourcing platforms like CrowdFlower 6 or
Amazon Mechanical Turk 7 hold fostered a great advancement in human
computation [
A lot of research has been focused on indentifying crowdsourced spam.
Although a commonly used algorithm for removing spam workers is the majority
decision [
Research on events detection and extraction from medical texts [
Although there has been an extensive event extraction research using
machines, the advantages of using crowdsourcing in this domain were not fully
harnessed. Our new approach (fostering disagreement between annotators) [
This section presents the workow initially developed within the Crowd-Watson project (Figure 1) for creating ground truth data for medical relation extraction, that was further adapted for creating ground truth for newspaper events extraction. The resulting ground truth we refer to as Crowd Truth. In this paper we focus on the event extraction process for event crowd truth collection and its adaptation from the medical domain. A key point here is illustrating the reusability and optimization features of the workow across the two domains.
The framework is designed as an end-to-end process which provides feedback loops that generate analysis for each stage of the workow in order to improve future results. The Pre-Processing 3.1 component handles the adaptation of the input data for making it solving-aordable in terms of micro-tasks. 8http://nlp.stanford.edu/software/lex-parser.shtml The Data Collection 3.2 uses CrowdFlower sequences of jobs for collecting judgments, while the Disagreement Analysis 3.3 component automatically handles the contributors evaluation. The Post-Processing 3.4 component automatically lters out the workers identied as spammers. Further, the process of collecting disagreement-based judgments can continue by reiterating each mentioned step. 3.1
Pre-Processing for Event Extraction
As, typically, the initial textual data collected from large sources, e.g. Wikipedia,
newspapers rst needs to be processed into smaller chunks suitable for
microtasks (paragraphs, sentences). Further, to optimize its applicability for training,
sentences that are not useful for training, e.g. too long or too short or contain
specic words that increase the ambiguity need to be ltered out. The Input
Data Filtering component clusters the input sentences based on their syntactic
criteria, e.g. presence of semicolons, comma-separated lists, parentheses, etc.
Each of those clusters can be either ignored or used for a specic micro-tasks.
For example, sentences with putative events identied in them can be given to
the crowd to conrm whether they refer to an event or not. Majority of those
lters we directly reused from our medical relation extraction use case.
Input Data: For the experiments described in Section 4 we used articles from
The New York Times. After their content was split into sentences (50 initial
sentences), we removed the short sentences (less than 15 words). Compared with
the task of medical relation extraction where the long sentences are typically
dicult for the crowd, in the task of event extraction the longer the sentence
the higher the chance that it will contain useful context for the event and the
role llers. This left us with 37 sentences to run the experiments with.
Putative Event Extraction: The rst step in extracting events is to determine
the putative events (verbs and nominalized verbs), i.e. word phrase that could
possibly indicate an event. This component rst exploits the context-free phrase
structure grammar representation from the Stanford Parser to extract all the
verbs and the nouns. Further, it follows the typed dependencies parses (also
from the Stanford Parser) to extract word phrases that being in relation with
certain verbs might trigger events. In addition to the Stanford Parser we also
used NomLex9, a dictionary of nominalizations. Thus, we extracted 205 putative
events from the 37 sentences of the input data. For the crowdsourced experiments
we selected only 70 putative events. Table 1 presents the putative events dataset.
which is the relevant level of granularity for temporal expressions. However, when
gathering gold standard data for events we are interested in collecting all possible
temporal expression. Thus, we combined four relative classes from Allen’s time
theory [
Table 2: Event Role Fillers Taxonomies
crowdsourcing platform 14, which targets nichesourcing with medical experts [
To avoid penalizing workers for contributing on dicult or ambiguous
sentences, we lter sentences based on their clarity score [
Finally, the Visual-Analytics component provides interactive visualization of (1) the workers behavior, (2) the sentence clarity and similarity. It provides a clear way to observe the dynamics in workers disagreement, completion time and the distribution of lters for spam contributions. The same visualization is used both for the relation extraction and for the event extraction tasks (Figure 3). 4
We adapted the Crowd-Watson medical relation extraction template for event extraction by constraining the workers to follow stricter rules, so that we can compare: (1) how does the new template inuence the quality of the results; 14http://crowd-watson.nl/dr-detective-game/ In this section we analyze the entire experimental workow. We observe the eect of the template design on the accuracy of the crowdsourced data, and we measure the accuracy of the worker metrics compared to the accuracy of the worker metrics together with the explanation-based metrics.
The preliminary experiment (Exp0) for identifying events and event types
did not use a conditional micro-task template. Nine workers submitted only one
or two judgments, which did not provide evaluation relevance, and were thus
excluded from the analysis. The contributions of 66 remaining workers were
analyzed further. The worker metrics identied 15 spam contributors, while the
explanation-based lters, described in [
When adapting the taxonomies we tried to conceive dierent experimental cases that could give insights in the adaptability degree of the metrics: the list of event types, time types and participant types are similar to the number of relations provided in the medical relation extraction task; the taxonomy for location type is more diverse, with overlapping concepts; one event can have multiple participants, which can increase the number of annotation for a putative event; this is a relevant factor for evaluating the behavior the average number of annotations metric;
We performed a manual sampling-based evaluation of workers in order to determine the accuracy of the spam metrics. We examined all the workers marked as spam by the lters, as well as the ones ranked as best workers. Some workers in the gray area inbetween were also manually evaluated. Figure 4 shows the precision, recall and F-measure only for the worker metrics for each job type, i.e. event type, event location, event time and event participants. Although the worker metrics identied a high percentage of low-quality contributors, the accuracy and the precision of the metrics still need improvements. A reason for this behavior could strongly relate to event properties types distribution and similarity, which varies along the four classes of event characteristics.
For the Event Type task the average number of annotations metric was able to identify correctly a high amount of spammers. However, both the workersentence and worker-worker agreement had low values in terms of correctness. The distribution of event types among 35 sentences, i.e. putative events (see Figure 3) indicates high ambiguity of the event types. One reason for this could be our choice of event types, which might not be so appropriate for the workers. However, in the Location Type task we see a dierent picture. As expected, most workers chose multiple types for one identied location. For example, the highlighted location for one putative event was "Apple’s glass cube on Fifth Avenue" and was classied as: [COMMERCIAL], [BUILDING], [ROAD], [STREET]. Although the worker did agree to a certain extent with the other workers solving the same sentence, he was wrongly classied as spammer based on the number of annotations. Thus, the accuracy of spam identication for location type is solely based on the worker-worker and worker-sentence agreements metrics. The F-measure for event type is equal to 0.89, while for event location is equal to 0.82. Even though the percentages of identied low-quality contributors were comparable for both experiments, the lower number of correct predictions for event location stands as a reason for a lower event location type F-measure. For the Time Type task the assessment of metrics behavior was the most challenging. More than half of the sentences used in the experiment did not contain any event time reference, and most of the workers chose "[NOT APPLICABLE]" as a type. This resulted in a high worker-worker and worker-sentence agreement scores. Thus, if a worker would disagree with other workers on just one sentence, (s)he would be identied as spam. Most of the spammers, however, were captured by at least two worker metrics. The ones identied by less than two metrics justify the high recall and the low precision (Figure 4) (F=0.81).
One event could have multiple participants of dierent types. However, the highlighted participants mentions in the task were of the same type, which explains why the average number of annotations per sentence did not have an erratic behavior. The event participants type has the highest F-measure value (0.91). This value is a result of the high amount of spammers correctly identied as well as the high amount of spammers identied from the entire list of spammers. Thus, we can conclude that the participants taxonomy presented to the workers is concise and covers with high precision the possibilities of interpreting the participants of an event.
The high worker disagreement in the event type experiments gave the worker metrics an important boost of eciency, by identifying a high amount of true spammers. However, the overall agreement was above mean expectations. Thus, the workers that did not highly agree with other workers were prone to be identied as spammers. For event location, however, the average number of annotations per putative event decreased the total precision of correctly identied spammers. As seen in Figure 4, the applied worker measurements had the most accurate behavior for the event participants type task.
Fig. 4: Worker Metrics Evaluation FFiilgt.e5r:s EWvoarlkueartioMnetrics and Explanation-based By looking at Figure 5 we can see that the explanations provided by the workers regarding their answers give an important boost of eciency in identifying patterns that are associated with low-quality contributions and even spam contributions. In combination with the worker metrics, these explanation-based lters are able to increase the accuracy of detecting low-quality workers with at least 5%. This situation was possible because only a small number of workers were identied as spammer by both explanation-based lters and disagreementbased metrics (worker metrics). Thus, for each batch, not more than 2 or 3 workers were identied by both quality measurements. Hence, it seems reasonable to further use the advantages brought up by those lters. This conclusion is also underlined by the usage of all the channels, situation that is usually associated with an increased percentage of spammed results. The results presented in Figure 5 make a good case to state that the usage of both worker metrics and explanation-based lters achieves high accuracy in terms of crowdsourced data. With the results mentioned in Figure 5 we can state that we succeeded to achieve high accuracy in identifying the spam workers, but we also showed how the metrics are suitable across domains. 6
This paper presents the results of our experiments on estimating the reusability and domain-dependency of crowdsourcing workow components, such as processing of input data, micro-task templates and result assessment metrics. We demonstrated how components dened in one domain (medical relation extraction) can be easily adapted to a completely dierent domain (event extraction from newspapers). Results from the experiments showed that some of the metrics for workers and content can be applied with high precision in those domains. For understanding to what extent the domains can be similarly treated, we conducted dierent research at each step of the crowdsourced process.
By directly reusing pre-processing lters from the medical relation extraction domain we showed that the input data can be optimized using syntactic text features. Thus, we can argue that the syntactic text features are mostly domain-independent. Although the template design was adapted for stimulating diversity in worker answers, the metrics were still able to capture the low quality contributions in both domains. With the nal template design for event extraction, the workers were less prone to spam the results by mistake. We have showed that especially for domains were there is no golden data known in advance, the explanations can be successfully used to identify more spam or low-quality workers. When the explanation lters are combined with the disagreement worker metrics the accuracy of detecting those low-quality contributors reaches a value greater than 92%. To sum up, the adaptation of the disagreement analysis component from the medical relation extraction to the event extraction preserved its good outcomes, and thus, these disagreement metrics are domain-independent.
As part of this research, our future work should focus on solving ambiguityrelated aspects. First, our analysis showed that there is still space for improving the event properties types. The event-type taxonomy shows a lot of ambiguity when looking at the workers annotations distribution. Further experiments should clarify whether a dierent classication of the putative events can achieve a better performance compared to the current experiments. Also, we need to conclude how the types that are overlapping inuence the results. Furthermore, each word phrase highlighted from the sentences needs to be clustered in order to determine the most appropriate structure of the event role ller.