In this paper we illustrate how we harness the power of crowds and specialized experts through automated knowledge acquisition work ows for semantic annotation in specialized and knowledge intensive domains. We undertake the special case of the Arabic script of the Qur'an, a widely studied manuscript, and apply a hybrid methodology of traditional 'crowdsourcing' augmented with 'expertsourcing' for semantically annotating its verses. We demonstrate that our proposed hybrid method presents a promising approach for achieving reliable annotations in an e cient and scalable manner, especially in cases where a high level of accuracy is required in knowledge intense and sensitive domains.
Thematic annotation of religious texts, in particular, the classical sources of
knowledge in the Islamic domain in the Arabic language, has not received much
attention, partly owing to time and knowledge constraints from experts required
for such an annotation process. In our research, we consider the application of
specialized human computation methods such as nichesourcing ([
In this paper, we show the results of an exploratory study focussing on
two knowledge intensive tasks: the thematic disambiguation and annotation of
Qur'anic verses using its Arabic script. While this has been tackled through
a pure crowdsourcing approach in our earlier work in [
The AMT crowd performs the thematic disambiguation and thematic annotation tasks. Both tasks are based on the Arabic script of the Qur'an. For the disambiguation task, a question is presented to the crowd, which includes a verse, along with a highlighted, candidate explicit assertion for the given theme, and the crowd responds by declaring this assertion as either a positive or negative by determining if the occurrence is a true occurrence of the given theme. The annotation tasks require deeper knowledge and understanding of the Arabic text. The crowd determines whether the given verse contains any implicit reference to the given theme. If their response is positive, then they are also required to provide the portion of the verse (a meaningful phrase or a word) that implies the presence of the theme. As a form of a quality measure, the crowd is also 1 http://www.mturk.com
Harnessing Crowds and Experts for Semantic Annotation of the Qur'an required to provide a con dence level (ranging from Very High to Very low), to indicate their con dence in their response.
Decision Analytics: We collect and aggregate the responses based on statistical measures of aggregation. Weighted con dence measures and thresholds are applied. Based on this aggregation, the completed tasks are marked as either Approved or Reviewable. A high con dence and aggregation threshold is applied for the approved tasks. This decision analytics results in identifying the candidate tasks for expertsourcing. The tasks marked as reviewable, which fail to meet the agreement thresholds, are sent o for expert annotations.
Expertsourcing Stage: For this purpose we designed a custom web application to engage with experts. A RestAPI connects the crowdsourcing task management engine with the expertsourcing application. The tasks are sent to the remote application and experts are noti ed when the tasks become available. The experts also see the candidate responses collected from the crowdsourcing stage. The experts have either the option to choose from the available annotations (collected during the crowdsourcing stage) or provide their own if they do not agree with either one. An example is shown in Fig 2. We present the same task to three experts to analyze the annotation agreements. The approved and validated annotations are passed on for Ontology Population and linked with existing data sources. The experimental setup assigned each task to 5 crowd workers. For the reviewable crowd tasks that were sent to experts, 3 experts were assigned to each reviewable task. Table 1 shows the results obtained.
Crowd Tasks Expert Tasks Task Disambiguation Annotation Disambiguation Annotation Approved 1267 477 34 96 Reviewable 40 107 6 11 Total 1307 584 40 107
Table 1. Results for Disambiguation and Annotation Tasks
The results of our exploratory study provide interesting insights into the application of human computation methods to knowledge intensive tasks. Our task design involved the thematic disambiguation and annotation of the Qur'anic verses based on the original Arabic script. For the disambiguation task, 99% of the tasks were able to reach an agreement by combining contributions of crowds and experts. Only about 3% tasks needed expert contributions. For the annotation tasks, about 18% tasks needed expert contributions. There were 10% tasks that did not reach an agreement with both crowd and expert contributions. An administrative review of these cases indicate that some annotations are a matter of personal taste and judgement and closed agreement is therefore di cult. Most of these annotations cannot be classi ed as wrong, nor better than the others based on an automated agreement mechanism.
Our knowledge acquisition and review work ow to selectively elicit expert annotations only where needed indeed presents a promising method. We utilize annotation agreement and distance analytics to route the appropriate tasks that need expert contributions. Our results suggest that such a hybrid approach indeed creates for a more accurate and reliable annotation process. This method can be e ectively utilized for qualitative dataset management and semantic annotation tasks in an economic and feasible manner through crowd engagement, while reducing the need for expert contributions.