We addressed the task to automatically recognize and normalize entities in a French medical corpus. To increase the coverage of our initial French terminology, English terms were translated into French by two di erent automatic translators. Indexing with a terminology that contained the intersection of the translated terms in combination with several post-processing steps to reduce the number of false-positive detections, gave the best performance results.
The CLEF eHealth 2015 task 1b focuses on concept recognition in French
medical text [
We addressed all three subtasks. Central in our approach to entity recognition and normalization are French terminologies based on the UMLS and postprocessing steps to reduce the number of false-positive detections. The UMLS already contains a number of French vocabularies, but their coverage is rather limited. We therefore explored the possibility to expand the coverage by automatic translation of English UMLS terms into French. For this purpose, we utilized two automatic translators.
We utilized two corpora in our experiments: the Quaero medical corpus, a French
annotated resource for medical entity recognition and normalization [
The annotations in the Quaero corpus are based on a subset of the Uni ed
Medical Language System (UMLS) [
Mantra Corpus. The Mantra corpus was developed as part of the Mantra
project [
The UMLS version 2014AB contains 178,860 unique French terms from 88,986
concepts, mainly stemming from MedDRA and MeSH, and only covering a few
percent of the more than 5 million English terms and 2.6 million concepts in
the UMLS. To expand the number of French terms, we used the web services
application programming interface from Google Translate (GT) [
In our experiments on the Quaero corpus we used ve French terminologies: { Baseline: all French terms in UMLS version 2014AB. Only terms belonging to concepts in the ten SGs listed above were considered. { GT: all terms from Google Translate and the baseline terminology. { MB: all terms from Microsoft Bing and the baseline terminology. { Union: all terms from Google Translate, Microsoft Bing, and the baseline terminology. { Intersection: all terms that had the same translation by Google Translate and Microsoft Bing, supplemented with the baseline terminology.
The English terminology for indexing the Mantra corpus consisted of all English terms in UMLS version 2014 AB, ltered for the ten relevant SGs.
Both on the English terminology and the French baseline terminology we
applied a set of term rewrite and suppression rules [
The processing for the entity recognition and the entity normalization included
an indexing and a post-processing step, which are described below.
Indexing. The corpora were indexed with Peregrine, a dictionary-based concept
recognition system [
Post-processing. To reduce the number of false-positive detections that resulted from the indexing, we applied several post-processing steps. First, we removed terms that were part of an exclusion list. The list was manually created by indexing the French Mantra corpus with the largest available French terminology (union), ordering the detected terms by their frequency in the corpus, and selecting the incorrect terms from the 2,500 top-ranked terms.
Second, for any term-SG-CUI combination and SG-CUI combination that was found by Peregrine and had also been annotated in the training data, we computed precision scores: true positives / (true positives + false negatives ). For a given term, only term-SG-CUI combinations with a precision above a certain threshold value were kept. If multiple combinations quali ed, only the two with the highest precision scores were selected. If for a given term none of the found term-SG-CUI combinations had been annotated in the training data, but precision scores were available for the SG-CUI combinations, a term-SG-CUI combination was still kept if the precision of the SG-CUI combination was higher than the threshold. If multiple combinations quali ed, the two with the highest precision were kept if they had the same SG; otherwise, only the combination with the highest precision was kept. If none of the SG-CUI combinations had been annotated, a single term-SG-CUI combination was selected, taking into account whether the term was the preferred term for a CUI, and the CUI number (lowest rst). 2.5
For entity normalization given the gold-standard terms and SGs, we developed the following processing pipeline. First, we computed precision scores for all term-SG-CUI combinations in the training set. If a given term-SG combination in the test set was also present in the training set, we selected the CUI of the term-SG-CUI combination with the highest precision score. If the second largest precision score was larger than 0.3, the CUI of the corresponding term-SG-CUI combination was also selected.
Second, if a term-SG combination in the test set had not been seen in the training set, we searched the terminology for terms that had a Levenshtein edit distance of maximum one. If one such term was found, the corresponding CUI was selected. If multiple terms were found, for each term the corresponding SGCUI combination was sought in the training data. If present, precision scores were computed and the CUI of the SG-CUI combination with the largest precision was selected. If the SG-CUI combination did not exist in the training data, it was checked if the term was the preferred term for any of the CUIs. If this was the case for just one CUI, it was selected. Otherwise, a single CUI was selected, taking into account whether the CUI had been annotated in the training set, and the CUI number (lowest rst).
We used the Quaero training data to optimize the performance of the indexing and post-processing steps for entity recognition and normalization. Table 1 shows the results for the ve French terminologies that we generated: Baseline (UMLS French), GT, MB, Union, and Intersection. The results have been generated with the task 1b evaluation script, using exact matching for both entity recognition and normalization.
The terminologies based on automatic term translations (GT and MB) substantially increase recall as compared to the UMLS baseline terminology, but at the expense of a large decrease in precision. GT performs slightly better than MB in terms of F-score. The union of both terminologies results in a small further increase of the recall. The intersection improves precision considerably at the expense of some loss of recall. The performance of the terminologies with translated terms is better on the EMEA documents than on the Medline titles, primarily because the recall is higher. Interestingly, the reverse is true for the baseline terminology, which performs slightly better on the Medline titles. As expected, the performance for entity normalization is lower than for entity recognition, mainly because of a lower precision. This is largely caused by the ambiguity of many terms. At this stage, our indexing system did not try to disambiguate when multiple CUIs for the same term were found, and thus many of the CUIs were scored as false positives.
In our further experiments we decided to focus on the Union and Intersection terminologies. First, we tested the e ect of expanding our terminologies with terms from concepts in the training data that were missed by our indexing system (false negatives). In order not to optimistically bias our performance results, we split the Quaero training data in an equally-sized training set and test set. Table 2 shows the performance results on the test set.
Addition of the false negatives results in a clear improvement of the recall, with only a small decrease in precision.
Based on the expanded terminologies, we tested the e ect of our post-processing steps, aimed at removing incorrectly indexed terms (false positives). An important parameter in this process is the precision threshold (see post-processing description above). Using half of the Quaero training data, we varied this threshold between 0.1 and 0.5 with steps of 0.1, and tested on the other half of the training data. The best F-score was obtained for a threshold of 0.3. Table 3 shows the results of the post-processing steps using this threshold.
The post-processing steps reduce recall but strongly increase precision, as well as the F-scores. 3.2
We submitted two runs for both the entity recognition and normalization tasks, one run using the Union terminology, the other using the Intersection terminology. Both terminologies were expanded with all false negatives of the Quaero training set. Table 4 shows our performance results on the nal test set for exact match. (Note: we swapped the test run precision and recall values that the task organizers provided to us, since we could deduce from the FP and FN counts that they had been reversed.)
Our results on the test set were better than on the training set, mainly because of higher precision values. Overall, the system using the Intersection terminology performed best. These results are well above the average and median of the scores from all participant runs, as provided by the task organizers.
We also submitted two runs for the normalization using the gold-standard entity recognition results. The di erence between the two runs was that the rst run did not include the nal disambiguation step (selection of CUIs if they had been annotated in the training set and based on CUI number). Table 5 gives the performance results.
As was to be expected, use of the gold-standard entity recognition improved the normalization results. In particular precision was boosted, with a remarkable precision of 1 for the EMEA corpus. The nal disambiguation hardly a ected the performance results. 4
Our results show that expanding the coverage of the French UMLS baseline terminology with the use of an automated term translator is a viable way to improve the recall for entity recognition and normalization, but also reduces precision considerably. Taking the intersection of the term translations increases precision again, while only slightly reducing recall. The various post-processing steps further improve precision. The union of the term translations did hardly further improve the recall, indicating that the annotated corpus contained few terms that were uniquely provided by one of the translators. Although the precision of the Union terminology on the Quaero training set was substantially less than the precision of the Intersection, the di erence on the test set was much smaller.