It is increasingly evident that the realization of the Semantic Web will require not only computation, but also human contribution. Crowdsourcing is becoming a popular method to inject this human element. Researchers have shown how crowdsourcing can contribute to managing semantic data. One particular area that requires signi cant human curation is ontology engineering. Verifying large and complex ontologies is a challenging and expensive task. Recently, we have demonstrated that online, crowdsourced workers can assist with ontology veri cation. Speci cally, in our work we sought to answer the following driving questions: (1) Is crowdsourcing ontology veri cation feasible? (2) What is the optimal formulation of the veri cation task? (3) How does this crowdsourcing method perform in an application? In this work, we summarize the experiments we developed to answer these questions and the results of each experiment. Through iterative task design, we found that workers could reach an accuracy of 88% when verifying SNOMED CT. We then discuss the practical knowledge we have gained from these experiments. This work shows the potential that crowdsourcing has to o er other ontology engineering tasks and provides a template one might follow when developing such methods.
Research communities have begun using crowdsourcing to assist with managing
the massive scale of data we have today. Indeed, certain tasks are better solved by
humans than by computers. In the life sciences, Zooniverse, a platform wherein
citizen scientists contribute to large scale studies, asks users to perform tasks
such as classifying millions of galaxies or identifying cancer cells in an image [
One area amenable to crowdsourcing is ontology engineering. Ontologies are complex, large, and traditionally require human curation, making their development an ideal candidate task for crowdsourcing. In our previous work, we developed a method for crowdsourcing ontology veri cation. Speci cally, we sought to answer the following driving questions: (1) Is crowdsourcing ontology veri cation feasible? (2) What is the optimal formulation of the veri cation task? (3) How does this crowdsourcing method perform in an application? In this work, we brie y highlight each of the experiments we developed to answer our questions, and, with their results in mind, then discuss how one might approach designing crowdsourcing tasks for the Semantic Web. In previous work, we have published papers that explore each driving question in depth. The main contribution of this work is a uni ed framework that presents all of the experiments. This framework will enable us to re ect on current work and to ask new questions for crowdsourcing ontology engineering. 2
We have begun to reduce portions of ontology engineering into microtasks that
can be solved through crowdsourcing. We devised a microtask method of
ontology veri cation based on a study by Evermann and Fang [
Is every Heart an Organ? A worker then answers the question with a binary response of \Yes" or \No."
This task is particularly useful in verifying ontologies because the class hierarchy is the main type of relationship found in many ontologies. For example, in 296 public ontologies in the BioPortal repository, 54% of these ontologies contained only SubClassOf relationships between classes. In 68% of ontologies, the SubClassOf relationships accounted for more than 80% of all relationships. Thus, verifying how well the class hierarchy corresponds to the domain will enable the veri cation of a large fraction of the relations in ontologies. 3
We developed various experiments that use the hierarchy veri cation task to answer our driving questions. Generally, each of these experiments follows the same basic procedure. First, we selected the ontology and axioms to verify. Next, we created a hierarchy-veri cation task formatted as HTML from these entities and submitted the task to Amazon Mechanical Turk. Finally, we obtained worker responses, removed spam, and compared the remaining responses to a gold standard using some analysis metric. Thus, in each experiment we used a standard set of basic components outlined in Table 1. Typically, we manipulated one of these components in each experiment. Figure 1 presents an example task as it appears to a worker on Amazon Mechanical Turk. To answer the driving questions, we performed a series of experiments using the basic protocol. We began with the most basic question about feasibility of the method. Having shown that, we tested various parameters in order to optimize the method. Finally, we used the optimal method in verifying SNOMED CT. Table 2 summarizes these experiments and their parameters. In the following, we describe the speci cs of each experiment and our conclusions for each.
In this rst driving question, we wished to understand if it were possible for Amazon Mechanical Turk workers (turkers) to perform on par with other groups also performing the hierarchy-veri cation task. 4
Methods We determined whether turkers could recapitulate results from a study
by Evermann and Fang [
Knowing the results of that experiment, we asked turkers to verify the same statements. As in the initial study, we required turkers to complete a 12 question training quali cation test. We asked for 32 turkers to answer each 28 question set and paid $0.10/set. Furthermore, we o ered a bonus for good performance. After turkers completed the tasks, we removed spam responses from workers who responded with more than 23 identical answers. Finally, we compared the performance of the students with that of the turkers using a paired t -test Results In both experiments, the average accuracy of student was 3{4% higher than the accuracy of the turkers. However, the di erence was not statistically signi cant.
Conclusion Turkers recapitulated previous hierarchy-veri cation results and performed on par with students in the hierarchy-veri cation task.
Methods Verifying a domain ontology was the second component in showing the feasibility of our veri cation method. For this veri cation task, we used CARO, a well curated biomedical ontology. We selected 14 parent-child relations from the ontology as correct relations. Like with WordNet, we paired children with parents that were not in the same hierarchy to simulate incorrect relations. We then asked workers to verify these relations following the earlier experimental setup. In this situation, we had no quali cation test. As a comparison, we asked experts on the obo-anatomy and National Center for Biomedical Ontology mailing lists to perform the same veri cation. Finally, we measured worker and expert performance, and compared the groups using a t -test.
Results With the proper task design of context and quali cations (addressed later), turkers performed 5% less accurately than experts, but there was not a statistically signi cant di erence.
Conclusions Workers performed nearly as as well as experts in verifying a domain ontology. These results are quite encouraging. In addition, the results of this experiment led us to hypothesize that worker performance signi cantly depends on the task formulation. We address this next. 4.2
With the feasibility of crowdsourcing ontology veri cation established, we focused on formulating the task in an optimal fashion. There were four main parameters that we hypothesized would a ect the method's performance: Ontology Type (i.e., the domain of the ontology being veri ed), Question Formulation (i.e., How should we ask a worker to verify a relationship?), Worker Quali cation (i.e., How does the accuracy of a worker vary based on certain quali cation?), and Context (i.e., What information should be provide to assist a worker in answering the question?).
Methods Having shown that turkers perform similarly to students and domain experts, we then analyzed how turker performance varied based on ontology selection. To do so, we compared worker performance in verifying BWW and SUMO to verifying WordNet. We created a set of WordNet statements to verify by extracting parent-child relationships in WordNet and also generating incorrect statements from incorrectly paired concepts (i.e. pairing concepts in parentchild relationships that are not actually hierarchically related). We then asked workers to verify the 28 WordNet, SUMO and BWW statements following the same setup as the rst experiment (including the training quali cation), paying workers $0.10/set, giving a bonus, and removing spam.
Results Echoing the rst experiment, workers performed only slightly better than random on BWW and SUMO, respectively. However, workers had an average accuracy of 89% verifying WordNet statements. There was a clear di erence between worker performance on upper ontologies and WordNet. Conclusion While workers struggle with verifying conceptually di cult relationships, such as those contained in upper level ontologies, they perform reasonably well in tasks related to common-sense knowledge.
Methods We repeated the task of verifying 28 WordNet statements but varied the polarity and mood of the veri cation question we ask the workers. In this case, we did not require quali cations as with the earlier experiments. Table 3 shows the 6 di erent question styles through example.
Results Worker performance varied from 77% on negatively phrased statements to 91% with the positive, indicative mood (i.e., a True/False statement asserting the relationship). In addition, workers responded faster with positively phrased questions. Conclusion Generally for crowdsourcing, one should create tasks in the most cognitively simple format as possible. In this situation, asking the veri cation as simply as possible (i.e., Dog is a kind of Mammal. True or False?)
Methods Having determined the optimal method to ask the veri cation question, we theorized that workers who could pass a domain-speci c quali cation test would perform better than a random worker on tasks related to that domain. We developed a 12 question high-school level biology quali cation test. For turkers to access our tasks, they would have to pass this test. We assume that the ability to pass this test serves as a reasonable predictor of biology domain knowledge. We asked workers to complete the CARO veri cation (Experiment 3), but required them to rst pass the quali cation task, answering at least 50% of it correctly. Results With quali cations, turkers improved their accuracy to 67% (from random without quali cations) when verifying CARO.
Conclusion When crowdsourcing, some method to select experts in the domain of the task is necessary to achieve reasonable performance. However, such low accuracy was not satisfying to the authors.
Methods With the increases in performance with proper question formulation and quali cation requirements, we next proposed that concept de nitions would assist workers in verifying a relation. In this experiment, we used CARO because the ontology has a complete set of de nitions. We repeated Experiment 3, with quali cations and simply stated veri cation questions, varying whether users and experts were shown de nitions.
Results With de nitions, workers performed with an average accuracy of 82%. Experts performed with an average accuracy of 86%. So, when providing workers and experts with de nitions, there was no statistically signi cant di erence. Conclusion In crowdsourcing, context is essential, especially for non-domain experts. While workers might not have very speci c domain knowledge, with proper context or training, they can complete the task. This experiment revealed that in some situations, a properly designed microtask can indeed provide results on par with experts. 4.3
The previous experiments were all synthetic { turkers only found errors that we introduced. With the optimal task formulation in hand, we shifted our focus to a true ontology veri cation task of verifying a portion of SNOMED CT. We selected SNOMED CT because it is a heavily studied, large and complex ontology, making it an ideal candidate for our work.
Methods In 2011, Alan Rector and colleagues identi ed entailed SubClass
axioms that were in error [
Results The aggregate worker response was 88% accurate in di erentiating correct versus incorrect SNOMED CT relations. On average, any single worker performed 17% less accurately than the aggregate response. Furthermore, there was no signi cant di erence in performance for tasks with di ering quali cation tests.
Conclusion Individually, workers did not perform well in identifying errors in SNOMED CT. However, as a group, they perform quite well. The stark di erence between average worker performance and aggregate performance reinforces the fact that the power of the crowd lies in their combined response, not any worker alone.
Each of the experiments we performed highlighted various lessons we learned in developing a method for crowdsourcing ontology veri cation. A few lessons are particularly useful for the Semantic Web community. First, many of our experiments focused on changing small components of the task. Even so, through this process we greatly improved crowd worker performance. It is clear that each task will be unique, but in most cases, extensive controlled trials will assist in identifying the best way to crowdsource a task. Following this strategy, we veri ed a complex ontology with relatively high accuracy. In addition, our current results only serve as a baseline { through additional iteration, we expect the increases in accuracy to continue.
Second, using the re ned tasks, we showed that crowd workers, in aggregate, can perform on par with experts on domain speci c tasks when provided with simple tasks and the proper context. The addition of context was the single biggest factor at improving performance. In the Semantic Web, a trove of structured data are available, all of which may provide such needed context (and maybe other elements, such as quali cation tests). For example, when using the crowd to classify instance-level data, the class hierarchy, de nitions, or other instance examples may all assist the crowd in their task.
Our results suggest that crowdsourcing might serve as method to improve
other ontology engineering tasks such as typing instances, adding de nitions,
creating ontology mappings and even ontology development itself. In fact,
Sarasua and colleagues used crowdsourcing to improve automated ontology mapping
methods [
The results thus far serve only as a baseline for crowdsourcing an ontology
engineering task. We plan to focus research on other elements in the crowdsourcing
pipeline, include entity selection (e.g., selecting the axioms for veri cation that
will most likely be in error), generating context (e.g., how can we use the crowd to
also supply context for workers downstream), and optimizing performance (e.g.,
developing aggregation strategies that maximize worker performance while
minimizing task cost). We will also consider di erent incentives models including
reputation or altruism, like the successful Zooniverse platform [
Crowdsourcing is now another tool for the Semantic Web researcher and developer. In this work, we described various experiments we performed to re ne a methodology to crowdsource ontology veri cation. In summary, we arrived at a highly accurate method through iterative, controlled development of the crowdsourcing task. In doing so, we gained valuable knowledge about method design for crowdsourcing. For example, providing task context is key to enabling accurate crowd workers. Finally, our results suggest that crowdsourcing can indeed contribute to ontology engineering.
This work is supported in part by Grant GM086587 from the National Institute of General Medical Sciences and by The National Center for Biomedical Ontology, supported by grant HG004028 from the National Human Genome Research Institute and the National Institutes of Health Common Fund. JMM is supported by National Library of Medicine Informatics Training Grant LM007033.