This paper presents a system for the normalization of concept mentions in clinical narratives. We evaluate and compare it against a popular, open-source solution that is frequently used for natural language processing of clinical text. The evaluation is based on a manually annotated dataset of 72 discharge summaries taken from the i2b2-corpus. Besides the demonstration and evaluation of our system we provide an in-depth corpus analysis that guided the development of the system. Our focus lies on the task of concept disambiguation, for which we combine two unsupervised approaches that are easy to implement and computationally inexpensive. We show that some ambiguities can only be resolved by adapting to annotation guidelines and preferences which we solve via the introduction of heuristics. Finally, we present an online-demo that gives insights into the individual parts of the normalization pipeline.
Recognizing and disambiguating clinical concepts
plays a central role in many information extraction
tasks within the clinical domain. It requires the
identification of concept mentions in clinical
narratives and the disambiguation of their respective
surface strings (normalization). In recent years,
many tasks have focused on the normalization
of clinical concepts, such as the i2b2 challenge
Traditionally, disambiguation systems rely on
supervised
In this work we develop a light-weight solution to the problem of clinical concept normalization, that is easy to implement and does not require expensive computations and is therefore particularly suited for industrial application. The approach is mainly unsupervised and does not require large amounts of training data. In particular, the disambiguation is based on a densest-subgraph algorithm to ensure contextual compatibility among the normalized concepts and the string similarity between the surface string and the preferred labels of a respective concept. We achieve very good performance with this setup on a manually annotated dataset. An web-application was developed for demonstration purposes and to debug the normalization pipeline1. 2
The concept normalization task requires a well defined target vocabulary. A useful resource is the Unified Medical Language System (UMLS), which defines biomedical concepts with various names, spellings and abbreviations. Concepts within UMLS are defined by so called concept unique identifiers (CUI) that represent concepts across different biomedical vocabularies, such as NCI, NDF-RT or RxNorm. However, natural language is highly variable and surface strings can have different meanings depending on the context.
Concept-Type (Source) Symptoms (NCI) Disease (NCI) Medication (RxNorm) Diagnostic Procedure (NCI) Therapeutic procedure (NCI) Anatomy (NCI) Laboratory Tests (NCI)
The task of normalizing surface strings to unique concepts of a given vocabulary such as UMLS can be subdivided into three partial tasks: Mention Recognition, Candidate Search and Disambiguation. Given an input text, the mention recognition subtask identifies text-spans that are potential mentions of a medical concept. Subsequently, the candidate search is responsible for finding candidate concepts for the surface strings of each mention. Finally, the disambiguation step selects the candidate that fits best into the mentions context, i.e., it resolves the ambiguity among its candidates. The work focuses on the disambiguation task. 2.1
In our experiments we used a part of the
i2b22corpus
3Note, the re-annotation took place within an industrial use case and was not carried out by one of the authors. The data and the dictionaries we used were already given. ment and the second half for testing.
We also analyzed the ambiguities within the corpus based on our candidate search (x3.2). Table 2 lists different ambiguity classes and their fraction in the dataset. It shows that ambiguity arises only in 18% of mentions. Candidate search fails in about a third of all cases for which the correct candidate is not found. For most of those cases no candidate is found at all. This shows that the currently employed dictionary lookup has to be refined. However, this work addresses the problem of disambiguation. Thus, only 18% of all cases are non-trivial and are useful for evaluating disambiguation. 3 3.1
Because of the focus on disambiguation our demo system employs a simple approach to mention recognition. Given a tokenized input document all word n-grams up to a predefined n are extracted. This guarantees high recall. In the subsequent candidate search step we eliminate all extracted mentions for which no candidates are found. 3.2
We find concept candidates for each recognized mention via a string lookup to a given dictionary. The dictionary maps surface strings to concepts. Those were extracted from a predefined subset of vocabularies in the UMLS, namely RxNorm for medications and NCI for anatomical concepts, diseases, therapeutic procedures, diagnostic procedures, laboratory tests and symptoms. The surface strings of the dictionary were expanded by including additional lexical variations. 3.3
The most crucial part of the concept normalization pipeline is the concept disambiguation. Given a set of candidates for each recognized mention it selects the concept which fits best to the mention of interest. The disambiguation is guided by two algorithms, that are explained in the following. String-Edit-Distance Each concept in UMLS may include a set of synonyms containing a range of variations and spellings. Not all of those string variations are likely to represent a concept in free text. However, a small subset of strings are indicated as preferred labels for a concept. In a corpus analysis, we found that many ambiguities can be resolved by selecting the candidate concept whose preferred labels contains a close match with the mention string. We further found that preferred labels of distinct UMLS concepts are usually mutual exclusive. Thus, we employ a string-edit-distance (ED) algorithm, namely Levenshtein-distance, between the preferred labels Lc of all candidates cim and the mention string xm. We use the minimum of those distances to define the ED-score of a candidate concept.
sed(cim) = max l2Lcim distance(xm; l) + 1 1 Densest-Subgraph We employ a densestsubgraph algorithm similar to Moro et al. (2014) or Weissenborn et al. (2015) to account for the context of a mention. First we construct a graph that consists of all candidates cim for all mentions m of a document. These are the vertices of the graph. We connect candidate concepts from different mentions with each other, whenever they co-occurred at least once together in MEDLINE, a repository of abstracts from biomedical publications. This information is annually summarized by the National Institutes of Health (NIH)4. Given the concept graph G = (V; E) of a document, we iteratively select a mention with the most remaining candidates and remove its least connected candidate until each mention has at most a predefined number of candidates left5. Given the pruned graph G = (V ; E ) we score each remaining candidate by the product of its number of connections to other mention candidates and other mentions, i.e., number of mentions that have at least one connected candidate concept. sdus(cim) = fcjm0 j(cim; cjm0 ) 2 E g fm0j9j : (cim; cjm0 ) 2 E g sdus(cim) sds(cim) = Pj sdus(cjm)
We tried different combinations of both scores and found the disambiguation via sds with a fallback to sed to work best. I.e., we select always the candidate for each mention with the highest sds and apply sed in case there are more than one candidate with the same score.
5We use 5 in our system, which performs slightly better or equal to other configurations. 3.4
A problem of unsupervised disambiguation is the inability of learning corpus-specific patterns which depend on annotation guide-lines and the personal perspective of the annotators themselves. Based on our observations the following set of simple rules are defined and used to support both disambiguation techniques: Active Substance: If the given mention is a tradename (e.g., Tylenol), in most of the cases its active substance (e.g., Acetaminophen) is annotated. Therefore we map all concepts that refer to a tradename to its active substance: This information is taken from the UMLS Metathesaurus relation has-tradename.
Structure of: If a mention ‘M’ (e.g. ‘left foot’) includes two candidate concepts, one containing the preferred label ‘structure of M’ and the other one ‘entire M’, the second concept is removed from the list of candidates.
Abbreviation validation: Abbreviations tend to
be highly ambiguous
The web interface of the online demo6 is based on the BRAT NLP-tool7 to visualize the implemented candidate search and disambiguation. Figure 1 presents the output of our Demo after processing a clinical narrative. The upper part ‘Candidate Search’ displays the text including mentions with their respective concept candidates. Different colors indicate different types of concepts. In the given example, red refers to anatomy, green to
System Pre-processing ED Gold-standard DS Gold-standard DS+SE Gold-standard ED cTAKES DS cTAKES DS+ED cTAKES cTAKES cTAKES symptom, pink to disease and turquoise to laboratory test. Moving the mouse courser over a candidate mention, the GUI shows the vocabulary origin and its concept unique identifier. 5
We evaluated our system on the test part of the
dataset with different configurations. More
specifically, we compare the performance of the
individual disambiguation algorithms, namely
stringedit-distance (ED) and densest-subgraph (DS),
and their combination, as well as a widely used
reference system called cTAKES8
Table 3 shows the results on the entire testset. We achieve a high precision of over 85% which we at
tribute to the performance of disambiguation. Our system performs also better than cTAKES9 with the same pre-processing (mention recognition and candidate search). The main problem in general lies in the low recall, which is mainly due to failing candidate search. This is also a major concern in future work.
As mentioned in x2.1, only a fraction of mentions can be considered non-trivial with respect to the disambiguation. Table 4 shows the performance of our system and cTAKES for all nontrivial mentions. The observations are similar to the previous results. We can see that the precision of our system is quite robust and much better than the performance of cTAKES. 6
We presented a light-weight disambiguation system for the normalization of clinical concept mentions. The system is mainly unsupervised and utilizes string similarity metrics as well as information from concept co-occurrences. We demonstrate its robustness with respect to disambiguation and compared it to cTAKES, a popular opensource system for clinical NLP. In addition, we give examples where our unsupervised approach fails because of annotation guidelines and preferences. This problem is solved by the introduction 9standard configuration for YTEX disambiguation of simple heuristics. Finally, our system can be accessed via a web-application.
This research was partially supported by SAP, the German Federal Ministry of Economics and Energy (BMWi) through the project MACSS (01MD16011F), and by the German Federal Ministry of Education and Research (BMBF) through the project BBDC (01IS14013E).