We used MetaMap and YTEX as a basis for the construction of two separate systems to participate in the 2013 ShARe/CLEF eHealth Task 1[9], the recognition of clinical concepts. No modi cations were directly made to these systems, but output concepts were ltered using stop concepts, stop concept text and UMLS semantic type. Concept boundaries were also adjusted using a small collection of rules to increase precision on the strict task. Overall MetaMap had better performance than YTEX on the strict task, primarily due to a 20% performance improvement in precision. In the relaxed task YTEX had better performance in both precision and recall giving it an overall F-Score 4.6% higher than MetaMap on the test data. Our results also indicated a 1.3% higher accuracy for YTEX in UMLS CUI mapping.
The mapping of natural language text to a de ned dictionary of concepts is one
of the signature tasks of information extraction and a variety of software has
been written to address this problem. We informally assessed a small set of open
source and freely available software for this task, before selecting MetaMap [
The CORAL.1 system utilizes the 2012 version of MetaMap for concept
recognition whereas the CORAL.2 system utilizes YTEX 0.8. MetaMap was called using
the MetaMap UIMA annotator to allow for integration into our NLP framework
which is also UIMA [
Filtered hits generated by our system often failed to identify the full span of the annotation in some cases if the identi ed text started with acronyms and adjectives. For example, for concepts such as `LA enlargement' and `lower abdominal tenderness' present in the training data, our systems were able to capture `enlargement' and `abdominal tenderness' but left words `LA' and `abdominal tenderness' uncaptured. Thus, an additional post-processing step was done to improve results for the strict task. During the postprocessing step, we searched the concepts annotated with our models if they were preceded by any of the words `LV', `MCA',`LA',`abd',`PEA',`LE', `LGI', `ICA',`C2', `B12', `RCA', `RUQ', `GI', `VF', `lower', `chronic', but are not captured by our models. If so, the concepts were expanded to include these words within the concepts boundaries. These missing abbreviations were identi ed as the largest and most readily correctable error classes for both systems based on training data performance
It should be emphasized that the CORAL system framework built around both MetaMap and YTEX does not do any concept prediction itself, it simply renes predicted concept boundaries and removes concepts predicted by MetaMap and YTEX it determines are incorrect. 2.1
Default settings were used for YTEX, including a concept window of length 10 and the default INTRINSIC setting as a semantic similarity metric. MetaMap was run with the included word sense disambiguation server (-y option) and restricting allowable concepts to the SNOMED CT R and RXNORM vocabularies. This is implicit in the default YTEX con guration which only indexes these two vocabularies by default. The 2012AB distribution of UMLS was used by both programs. 2.2
Prior to running data for the ShareClef task we evaluated both MetaMap and
YTEX on PTL notes from the University of Alabama at Birmingham Health
System. PTL documents consist of a summary of the patient's condition, with
the majority of text in point form format along with some full sentences; a
precise breakdown was not calculated. The document was irregularly formatted but
highly structured, so in contrast to the ShareClef analysis, only disease mentions
in the appropriate "problem" sections of the PTL document were analyzed.
Document segmentation was done by a complex manually derived regular expression
to identify the start and stop of the problem section. The same such expression
was used for both YTEX and MetaMap. In total, there are 68 such annotated
PTL documents with 603 annotated entities of which 223 are problems.
Annotation was performed by two annotators (including one physician) and observed
agreement was 91.9% (uncorrected Cohen's kappa) [
Results for concept boundary detection (Task 1a) are shown in Table 1 and Table 2 for training and test data respectively.
Boundary detection was di cult for both systems. This was in part because neither MetaMap nor YTEX have the ability to annotate discontinuous concept boundaries limiting the e ectiveness of both systems and e ectively capping the maximum performance. Additionally MetaMap tended to include additional text (mostly prepositions and modi ers) that the ShareClef annotators did not. YTEX precision was signi cantly reduced by the inclusion of simple nouns when a compound noun was expected by the annotators. As a result YTEX performed poorly relative to MetaMap on the strict task.
Results were harder to compare for Task 1b, the mapping of text to CUIs. The PTL results (shown in Table 3) do not include accuracy data, since false negatives or false negative frames are not annotated in the PTL data set as they are in the ShareClef data set (Table 4). Instead results for precision, recall and F Score are shown for the PTL document. Strict boundary was di cult for both systems, a varied set of error classes were generated. The biggest problem for strict boundary detection however was the exclusion of adjacent relevant text (modi ers), a de ciency that partly overcome by our boundary extension rules. Also abbreviations were particularly and consistently di cult, both systems labelled text such as 'BACTERIA OCC' as Osteochondritis dissecans. In general, the MetaMap based CORAL.1 performed better than the YTex based CORAL.2 in the strict task.
The relaxed task showed signi cantly higher scores for both systems, but both systems still failed to identify problems - particularly phrases containing common polysemous words such as "inability", as in "inability to walk'. Overall YTEX performed slightly better at this relaxed task than MetaMap, due to the inclusion of partially mapped annotations as fully scoring (YTEX had many such annotations) and the superior ability of YTEX to correctly identify polysemous text.
In Task 1b a formatting error prevented our results from being processed but we show results for the training data and the PTL documents here. Results di ered only slightly for YTEX and MetaMap. Common sources of error in both systems were stemmed from unrecognized abbreviations and low frequency concepts that neither the semantic distributional approaches used by YTEX or the heuristics and word sense disambiguation server employed by MetaMap could overcome. For example YTEX identi es the physician abbreviation "Dr." as diabetic retinopathy and MetaMap identi ed the word call in "Call or return immediately" as "c-ALL", a precursor B-cell lymphoblastic leukemia. Another small class of errors may be due to problems in the ShareClef annotation. For example the ShareClef annotation identi es fever as CUI-less instead of pyrexia, a synonym for fever.
Results for both systems were signi cantly worse on the PTL data set (Table 3) than on the ShareClef data set (Table 4). The di erence can be explained in large part to the annotation of the PTL data set, where the annotation guidelines specify that only the most precise concept possible should be annotated and thus penalizes YTEX which generates a larger number of more general (false positive) concepts . As described in the methodology section the PTL document set is an order of magnitude smaller than the ShareClef data set and thus less reliable. Nonetheless it underlies the fact that small di erences in annotation guidelines can have a large impact on clinical information extraction evaluation.
One system that was also given serious consideration for a more formal evaluation was the NCBO annotator. However due to the di culty of the VM setup (including the loading of source vocabularies) and concerns about being banned for sending thousands of queries to the NCBO web service we declined to investigate this option further.
Finally, a higher performance for YTEX could have likely been gained by
parameter tuning. An earlier evaluation of YTEX sense disambiguation [
In conclusion, given a choice between YTEX and MetaMap our results suggest that YTEX would be a better system for "o the shelf" concept mapping. Other factors such as active development and ability to scale favor YTEX. However MetaMap may be a better choice for precisely identifying concept boundaries. This project was supported by the UAB Center for Clinical and Translational Science - grant number UL1 RR025777 from the NIH National Center for Research Resources, and the UAB O ce of the Vice President for Information Technology.