This paper proposes a design for a gami ed crowdsourcing work ow to extract annotation from medical text. Developed in the context of a general crowdsourcing platform, Dr. Detective is a game with a purpose that engages medical experts into solving annotation tasks on medical case reports, tailored to capture disagreement between annotators. It incorporates incentives such as learning features, to motivate a continuous involvement of the expert crowd. The game was designed to identify expressions valuable for training NLP tools, and interpret their relation in the context of medical diagnosing. In this way, we can resolve the main problem in gathering ground truth from experts { that the low inter-annotator agreement is typically caused by di erent interpretations of the text. We report on the results of a pilot study assessing the usefulness of this game. The results show that the quality of the annotations by the expert crowd are comparable to those of an NLP parser. Furthermore, we observed that allowing game users to access each others' answers increases agreement between annotators.
Modern cognitive systems require human annotated data for training and
evaluation, especially when adapting to a new domain. An example of such system is
Watson QA [
The diversity of interpretation of medical text can be seen at many levels; as a
simple example, consider the sentence, \Patients exhibiting acute tailbone pain
should be examined for extra bone nodules." Human experts disagree routinely
on whether \acute tailbone pain", \tailbone pain", or \pain" is the primary term
in this sentence. Proponents of \tailbone pain" argue that there is a medical
term for it (Coccydynia) making it primary, others argue that it is pain which is
located in the tailbone. Traditional methods of gathering ground truth data for
training and evaluation fail to capture such interpretation diversity, leading us
to the innovative Crowd Truth approach [
Our analysis led us to believe that the diversity of interpretation occurs at two levels, depending on whether the context is being considered. Term identi cation, as exempli ed in the example above, may be done independent of the clinical context, for example when processing a textbook for background knowledge. However, in the presence of a particular patient, the role of the location and duration modi ers (e.g. tailbone, acute, resp) may or may not be important. We also observe that context-independent tasks tend to require less expertise, allowing us to use a lay crowd more e ectively.
These two types of annotation tasks can be performed by two di erent types
of crowds in order to optimize the time, e ort and the quality of the nal result.
Given the experience [
In this paper, we propose a gami ed crowdsourcing application for engaging experts in a knowledge acquisition process that involves domain-speci c knowledge extraction in medical texts. The goal of such text annotations is to generate a gold standard for training and evaluation of IBM Watson NLP components in the medical domain. First, we position our work in the context of already existing games with a purpose, crowdsourcing and other niche-sourcing initiatives. Then we outline our approach by focusing on the gaming elements used 4 www.mturk.com 5 www.crowdflower.com as incentives for medical experts, in the context of the overall game application architecture. We show how this gaming platform could t together with a micro-task platform in a joint work ow combining e orts of both expert and non-expert crowds. Next, we describe the experimental setup to explore the feasibility and the usability of such an application. Finally, we discuss the results of the pilot run of our application, and we identify the points of improvement to bring in future versions. 2
In recent years, crowdsourcing has gained a signi cant amount of exposure as a
way for creating solutions for computationally complex problems. By carefully
targeting workers with gaming elements and incentives, various crowdsourcing
applications were able to garner a signi cant user base engaged in their tasks.
The ESP Game [
A crowdsourcing framework by [
Gami cation as applied to text annotation crowdsourcing is an emerging
eld in di erent domains. For instance, the Phrase Detective project [
The Sentiment Quiz [
However, neither the Sentiment Quiz, nor the Phrase Detective applications actively seek out to capture the ambiguity in language. Phrase Detective even tries to enforce agreement, by awarding additional points for annotators that agree with the ground truth. Neither do most applications in the domain study the e ect of using specialized crowds to perform the information extraction tasks. Our goal is to build an end-to-end gami ed crowdsourcing platform that can capture disagreement between annotators, while catering speci cally to experts in the medical eld. 3
\Crowd-Watson" Architecture: The Game Perspective
In this section, we describe the architecture for Dr. Detective 6 { an application
for engaging experts in knowledge extraction tasks for creating ground truth
annotations in medical texts. We start by framing Dr. Detective as part of the
general Crowd-Watson7 framework for crowdsourcing medical text annotation [
The Crowd-Watson framework supports the composition of crowd-truth
gathering work ows, where a sequence of micro-annotation-tasks can be executed
jointly either by the general crowd on platforms like CrowdFlower, or by
specialized crowd of domain experts on gaming platform as Dr. Detective.
CrowdWatson framework focuses on micro-tasks for knowledge extraction in
medical text. The main steps involved in the Crowd-Watson work ow are:
preprocessing of the input, data collection, disagreement analytics for the
results, and nally post-processing. These steps are realized as an automatic
end-to-end work ow, that can support a continuous collection of high quality
gold standard data with feedback loop to all steps of the process. The input
consists of medical documents, from various sources such as Wikipedia articles
or patient case reports. The output generated through this framework is
annotation for medical text, in the form of concepts and the relations between them,
together with a collection of visual analytics to explore these results. The
architecture of this application, and the way its components interact with each
other, can be seen in Figure 1. In this paper, we focus on those aspects of the
architecture that relate to the Dr. Detective gaming platform for data collection.
A full description of the Crowd-Watson architecture is available at [
Typically, the input is available in an unstructured format (e.g. simple text).
As part of the input data ltering step, additional metadata, such as the
specialization eld in which it was published or, for case reports, the diagnosis
of the patient, can be extracted from these documents. In addition, some
annotation can also be generated automatically, by mapping the text to the UMLS
vocabulary of biomedical terminology, classi cation, and coding standards [
The workers are asked to perform a series of annotation tasks on the input documents. The purpose of these tasks is creating annotation in the form of concepts and the relations between them. We de ne these tasks according to four micro-task templates: 1. Term extraction { the task of identifying all the relevant terms in a text, where a term refers to a set of words that forms a coherent medical concept; 2. Term categorization { the task of classifying a medical term into an appropriate category, such as the concepts in the UMLS thesaurus; 3. Relation extraction { the task of identifying whether or not a relation exists between two medical terms; 4. Relation categorization { the task of classifying a medical relation into an appropriate category (or set of categories), such as the relations in the UMLS thesaurus.
The workers on Crowd-Watson consist of both an expert crowd, and a general crowd. Each of these crowds interacts with the input documents on a specialized platform { for the general crowd, regular crowdsourcing micro-tasks have been constructed on CrowdFlower, whereas the expert crowd employs the Dr. Detective application for solving tasks tailored to their pro le. The tasks can be solved by both the general, and the expert crowd. The target crowd setting step entails picking the di culty level of the task according to the level of expertise of the crowd. For instance, when discussing term extraction, the general crowd could reliably nd demographic terms, as they do not require signi cant medical knowledge, whereas the expert crowd can focus on annotating more di cult terminology. 3.2
After the input data is formated and ltered appropriately through the
preprocessing components, it is sent to the data collection component to to gather
either expert annotation (through the gaming platform) or lay crowd
annotations (through the micro-task platform). Next, the annotation results are
analyzed with a set of content and behavior-based metrics, to understand how the
disagreement is represented in both cases [
To track the individual performance of a user in the crowd, the expert metrics were developed. For each sentence in the input, the performance of the worker can be measured as a set of vectors, according to the task they solved on that input. Such a vector is composed of 0 and 1 values, such that for each answer a user annotated in that sentence, there is a 1 in the corresponding position, whereas answers that were not picked by the user are set to 0. These answer vectors can also be measured at the level of the domain.
At the level of the sentence, a set of task-dependent sentence metrics were also de ned. For either term extraction or relation extraction, any sentence can be expressed as a sentence vector { the sum of all the individual user vectors on that sentence, for that task. Furthermore, an added layer of granularity can be introduced by considering the categories for the terms and relations. This representation can then be used to de ne appropriate metrics for sentence clarity, what the popular answers were, how disagreement is represented, and similarity of annotation categories and domains.
The prime role of the disagreement analytics in the gaming platform are to provide explicit measures for the quality and completeness of the nal result; to identify gaps of missing types of annotations; or to discover possible contradictions and inconsistencies. This is opposed to the micro-task disagreement analytics, which follow the same approach but apply to lters for spam identi cation.
In order to collect data from a crowd for medical experts, it is imperative to
nd the necessary motivators for engaging them into contributing. To this end,
we have performed a series of qualitative interviews with medical students and
professionals. The purpose was to identify what requirements and features would
the medical crowd be interested in seeing in a crowdsourced application, and how
this application could be built to help in their work. These interviews established
incentives for crowd labor [
After discussing with 11 people in the medical eld (2 professionals, 3 lecturers, 5 students), we were able to identify several key requirements to incorporate into the gaming platform: { at the level of the input, the interviewees expressed their interest in reading medical case reports; { learning about their eld, through targeted micro-tasks and extended feedback on their answers, was the most signi cant motivator; { the interviewees expected the tasks to challenge their problem-solving skills; { competition with peers emerged as a secondary motivator; { the tasks need to be fun to solve, making entertainment as another secondary motivator; { medical professionals have di cult schedules, and would prefer to have exibility in the time required to engage with the application;
In order to attract users to the application, a goal that is seen as useful by the players needs to be rmly established. As learning proved to be the most relevant incentive from the interviews, we focused the goal of the application on this, while also trying to incorporate the problem-solving requirement. We developed the concept of a clue- nding game, where the text annotation tasks were put in the context of searching for clues in the history of a patient. For instance, when performing the task of term extraction on a patient case report, the user can annotate any of these three clue types: 1. the term is a clue leading to the nal diagnosis of the case; 2. the term is a false clue that is irrelevant to the nal diagnosis of the case; 3. the term is a normal condition that does not in uence the nal diagnosis of the case.
The clue types can be used as an incentive, involving users with the task they are solving by redesigning it as a medical puzzle, but it can also be used to generate additional annotation. The annotations retrieved from the general crowdsourcing approach are dependent on the context of the sentence where they were identi ed, so by asking the expert crowd to nd meta-relations at the level of the document, we can generate knowledge that is valid generally for the domain. This kind of task cannot be solved simply with the use of contextual information, and requires background knowledge of the eld, therefore making it suitable for an application targeted at experts.
The qualitative interviews helped us identify the extrinsic motivators for engaging the medical crowd. After the goal of the application was established, the nal step was translating the user incentives into concrete features for building the Dr. Detective gaming platform. 4.1
In order to support the user learning experience and introduce exibility in task
solving, we de ne the concept of di culty. This refers to the combination of
skill and time required for reading the document, and then performing the
annotation task. While it is di cult to hypothesize on the comparative di culty
of performing annotations, the di culty of the document can expressed as
syntactic and semantic di culty. The syntactic di culty expresses the e ort need
for reading the document in three components: the number of sentences in the
document (N oS), the number of words (N oW ), and the average sentence length
(ASL). The semantic di culty expresses the e ort needed for understanding the
text in two components: the number of UMLS concepts present in the document
(N oU M LS), and the readability of the document (SM OG). The SMOG [
In order to develop the competition incentive, a scoring system was devised, to reward players for their work. Through viewing a high score board, they are also encouraged to compete against each other.
We want to reward users when they perform in a way that is bene cial to us.
We want to collect the correct answers to the task, therefore, selecting a
highconsensus solution should yield more points. This strategy could, however, make
users rely entirely on the answers of others. Therefore, in order to encourage
a wider answer set and capture semantic ambiguity, we need to give points
for newly discovered answers. Users should also be penalized for giving wrong
answers. We also want to encourage users to return to the application, and keep
playing. Finally, in order for users to solve tasks in increasing di culty, scoring
needs to be proportional to the di culty for solving the task [
Based on this analysis, we developed the following scoring formula: score(U; D; T ) = dif f iculty(D) (popular(U; D; T ) + consecutive(U ) + discovered(U; D; T )
wrong(U; D; T )):
In order to develop the entertainment incentive, the crowdsourcing application
needs to provide immersion inside the task-solving experience. Immersion is
based on the concept of game ow [
Immersion is considered when choosing the next document that the user will be asked to solve as part of the game. When a user solves a task on Di, the document they will be asked to solve next needs to have a higher di culty in order to avoid boredom, but the increase needs to be low enough to avoid anxiety. Therefore, we de ne the set of possible documents that occur after Di as: next(Di) = fDj jdif f iculty(Dj ) = min(dif f iculty(Di) dif f iculty(Dt); 8t 6= i where dif f iculty(Dt) dif f iculty(Di))g Finally, in order to satisfy the constraint for exibility, game levels were implemented to quantify the skill required for solving the tasks. As skill is proportional with di culty, we de ned the game levels by quantifying the di culty metric previously described into three intervals: 1. easy: fD j dif f iculty(D) 2 [0; 2]g, 2. normal: fD j dif f iculty(D) 2 [3; 4]g, 3. hard: fD j dif f iculty(D) 2 [5; 6]g.
These levels should enable users to plan which task they want to solve in
accordance to the time they have at their disposal, while also providing a
goalbased incentive of progressing in their skill [
In order to test the feasability of the Dr. Detective setup, we implemented a version of the work ow described in Section 3, and set up a pilot run involving a crowd of medical professionals. As part of our pilot run, we performed an initial evaluation of both the quality of the answers, and the user enjoyment as part of this gami ed crowdsourcing platform. The goal of this experiment can be described as three questions, which will be discussed as part of our results: 1. How do the answers annotated by the crowd compare to those found by the
UMLS parser? 2. Does having access to the answers of other users stimulate diversity of opinion? 3. Did users experience immersion in the gaming experience?
In order to answer these questions, we set up two versions of the game, one in which users had the ability to see the answers of others, and one in which they did not. In addition, some of the gaming elements that would ensure the users keep in the state of game ow (high scores board, next document selection mechanism, levels) were only limited to the full version of the game. We constructed an experiment where the users would play both versions of the game, then answer a questionnaire on their experiences. The details of this experimental setup are described in this section. 5.1
Based on a suggestion in the qualitative interviews, the input was selected from clinical cases published in the New England Journal of Medicine8. 10 documents were picked out of four of the most popular specialties (Hematology/Oncology, Nephrology, Primary Care/Hospitalist/Clinical Practice, Viral Infections). The diagnosis was extracted from each document, based on a string matching procedure performed on the text marked in \diagnosis" section headings (e.g. clinical diagnosis, pathological diagnosis etc.). The documents were split into paragraphs, to increase the ease of reading, and the di culty metrics (described in
Section 4.1) were then applied to each paragraph. Finally, we selected a set of 20 paragraphs, with the values in the di culty vector uniformly distributed to represent a broad range of text types, to use for the game, as we wanted to ensure that all of the text would be annotated in the limited time frame of the experiment run. 5.2
The micro-task templates (described in Section 3.1) selected for this pilot were (1) term extraction, and (2) term categorization. Based on how relevant they are at describing patient case reports, 3 meta-types, each with a set of term types taken from UMLS, were selected and implemented in the interface for the categorization task. These term types are based on factor categories given to domain experts during the expert annotation phase for Watson. The type selection menu can be seen in Figure 4. In total, 13 term types were available for the users to annotate. As most interviewers expressed their interest in a problem-solving application, we decided to set the clue type user seek as part of the application (described in Section 4) to (1) the term is a clue leading to the nal diagnosis of the case. Finally, in order to encourage the diversity of opinion, and therefore capture ambiguity, we allowed users to look at the answers of others for the task they are solving. This feature was made available through a button, which the users could choose to press in order to toggle the other answers. The scoring formula (described in Section 4.2) ensures that users are motivated to nd new answers even in this circumstances, through the use of discovery bonus points. The users could access the details of how their score was computed through a hover noti cation in the menu. An example of how this task was presented to the users as part of the Dr. Detective interface can be seen in Figure 3. The pilot run of the Dr. Detective game had 11 participants in total, with 10 players engaging with the full game version, and 7 engaging with the simple version. In total, 155 annotation sets were collected, with each paragraph solved as part of 2 to 7 di erent game rounds. In addition, 6 players completed the feedback questionnaire. 6
In keeping with the research questions de ned in the previous section, we rst analyzed how the answers from the crowd compare to the results of the UMLS parser. We selected the top three paragraphs that were played the most, and compared the answers to the term list generated by the UMLS MetaMap parser 9 for the same paragraphs. Fig. 5 shows the crowd was able to identify the majority of the words annotated with UMLS. Additionally, Fig. 6 shows that around one third of the terms in UMLS had a full match with terms annotated by the crowd. Factoring in the partial term matches, the crowd was able to identify most of the
UMLS terms. This shows the e ciency of the crowd answers is quite high, enough for the crowd to be considered as a viable alternative to automated named-entity recognition, provided that enough users give their input for a paragraph.
Next, we look at how diversity of opinion was expressed by the game users. Speci cally, we are interested in nding out whether being able to see the results of other people will stimulate disagreement, or rather make users select each other's answers. In other to achieve this, we look at how the answers per paragraph varied according to the version of the game that the user played.
Fig. 7 shows how the number of new words per paragraph increases after each round of the game, for the top three paragraphs. Each version of the game seems to follow the same progression in the rate of new words identi ed, with the rst users nding most of the words, and then only slight increases as the paragraph is played by other people. However, the simple version of the game seems to constantly feature a higher total word count, as opposed to the full game version. The same trend was observed both for the number of new types, and the number of distinct terms. This seems to indicate that the full game version was less encouraging for collecting a wide array of terms.
In order to rule out an issue related to some other feature in the full game version, we looked at how the behavior of pressing the button to view other answers a ected the output. Out of 67 game rounds played in the full version, this button was only pressed in 18 of the rounds, so it appears this was not a popular feature to begin with. Fig. 8 shows that, actually, users tended to annotate more words in total when they pressed. However, as evidenced in Fig. 9, the ratio of new words to total words in this case was much lower than when the button was not pressed. Additionally, it appears there is not much di erence Fig. 8. Ratio of total words per round, grouped by the use of the button to view the answers of others
Fig. 9. Ratio of new to total words, grouped by the use of the button to view the answers of others between the simple version of the game, and the full version, but where the users chose not to look at the answers of others. Therefore we can infer that having access to all the answers makes the crowd act more conservative, selecting less new words, but rather choosing to validate the answers of others.
When looking at the answers in the questionnaire related to the usefulness of seeing other people's annotations, we found that most people (67%) were ambivalent to having the option of checking their answers. Some users reported using this feature as a tool for better understanding the task, while others claimed it validated the answers they had already chosen. Overall, it seems that having access to all the other answers makes users less likely to nd and annotate new words, which could mean a loss in the ambiguity of the annotation. It also provides an unfair advantage to the rst users to annotate a paragraph, as their score would likely keep increasing as other people keep selecting their answers.
Finally, we analyzed whether immersion in the game occurred for the users involved, and how each individual game feature was rated. The ow of the game was reported to be good, with 83% of the users saying they were neither too bored, or overwhelmed. Most users found the levels to be a useful addition, with 50% being satis ed with the level progression, and 33% being ambivalent to it. However, some users pointed out that they expected more challenge from the advanced level. As the di culty is currently computed only based on textual metrics, the game could potentially get boring for users. For this reason, domain di culty should be incorporated in future versions of the game. The scoring part of the game was less well received, with 83% of the users declaring they found the way their score is computed only somewhat clear. Therefore, in future game versions, a more detailed scoring breakdown should be implemented, with users being able to access the history of the cases they solved. Finally, most users reported to have enjoyed the game, and expressed an interest in returning to play, provided they can solve more di cult cases and get more feedback. The full game version was almost universally preferred by the users.
This paper proposes a design for Dr. Detective { a gami ed crowdsourcing platform to extract annotation from medical text. Dr. Detective was developed in the context of Crowd-Watson, a general crowdsourcing framework for extracting text annotation by engaging both a general crowd, and a domain expert crowd. The gaming platform was designed taking into account the requirements of the expert crowd, and illustrating their implementation in a clue nding game. Speci c gami cation elements were incorporated, such as di culty, scoring, immersion, and levels. A rst version of Dr. Detective was implemented and tested. The pilot run showed that the quality of the results of the crowd are comparable to those of an NLP parser. Allowing users to see the answers of others resulted in increased agreement, and thus decreased the desired diversity in answers. The overall user feedback for the application was positive. However, it was clear that users desire more complex challenges in order to keep them engaged.
An important next step is to de ne and test disagreement metrics that are speci c to the gaming environment. As we have seen in previous research, a promising starting point are the disagreement metrics developed for the data collected through the micro-task platform. We also plan to further test how each of the gaming features performs individually, in order to ne-tune the application to understand better their in uence on the quality and volume of the end result, as well as to adapt best to the needs of the users. Finally, we will explore how to further integrate the gaming and the micro-task crowdsourcing work ows, by using the output from one work ow to enhance the input for the other (e.g. ask one crowd to perform the term extraction, and the other crowd the relation extraction), or by asking one crowd to validate the output of the other crowd.
The authors would like to thank Kathrin Dentler and Dr. Petra Wol s for their help with nding participants for both the interviews and the application pilot run, as well as the students and medical professionals who were involved in these activities, and who provided their feedback.