In this work we leverage crowdsourcing in connection with machine learning techniques to validate candidate ICD10 Code to UMLS concept relationships that we generate. Our immediate use is in natural language understanding and machine learning approaches to automatically code electronic health record documents with ICD codes. Beyond auto-coding, the relationships will aid a wide variety of future medical applications, such as terminology-driven search in support of smart medical assistants.
Initially, EHRs manually annotated with gold standard ICD-10 CM Codes are
also annotated for UMLS concepts, and the code-concept pairs are processed
for pointwise mutual information (PMI) across a corpus [
Once we have trained our probabilistic model with crowdsourcing results, we use the results to predict whether the candidate pairs that have not yet been crowd-sourced are valid or not (using PMI and other measures as features, and results of the crowd as labeled data). In the initial phase, we leverage subject matter experts' knowledge to validate the crowdsourced judgments. Thus the crowdsourcing data is used for two purposes: training the probabilistic model and for nal judgments. As similarity measures are developed further within the project, they can be used as additional features to identify candidates to crowd source or serve as relationships. Table 1 shows three pairs examples. The overall work ow is illustrated in Figure 1.
In order to determine how best to formulate natural language questions to ask as to the direct relationship between a code (i.e., clinical situation) and a concept, we consider the UMLS semantic type, based primarily on the following: Disorders, Body Parts, Procedures, and Findings. For example: if the concept is a disorder, the question options pertain to the taxonomic relation \isa"; if the concept is a body part, the question pertain to whether it is the nding site of the disorder; if the concept is a diagnostic procedure, the questions pertain to whether the procedure is used to diagnose the disorder. Note then, that the relationships are both taxonomic, compositional, and other direct relations, therefore supporting structured knowledge source applications. Example questions posed to Mechanical Turk workers are given in Figure 2. 4
For evaluating our results, we consider majority for con rming a speci c answer,
and consensus for con rming negation (i.e., none of the above answer) is
accurate. We are in the process of adjusting parameters (pay, questions per task,
quali cation questions, assignment duration, auto-approval). Our next steps
include performing analyses for evaluation techniques for workers, answers, and
question quality. This is useful since disagreement is oftimes signal and not noise
[
The improved ICD-10 to UMLS relationships generated by our crowd-sourcing
approach will result in less noisy and more robust structured knowledge. It will
enable the use of these relationships are rules as well as evidence for auto-coding
ICD 10 CM. Also, such knowledge will support future medical applications aimed
at aiding practitioners and patients alike. Use of knowledge representation for
building medical expert systems for diagnosis has been well explored (for a
review of techniques, see[