We participated in task 2 of the CLEF eHealth 2016 challenge. Two subtasks were addressed: entity recognition and normalization in a corpus of French drug labels and Medline titles, and ICD-10 coding of French death certi cates. For both subtasks we used a dictionary-based approach. For entity recognition and normalization, we used Peregrine, our open-source indexing engine, with a dictionary based on French terms in the Uni ed Medical Language System (UMLS) supplemented with English UMLS terms that were translated into French with automatic translators. For ICD-10 coding, we used the Solr text tagger, together with one of two ICD-10 terminologies derived from the task training material. To reduce the number of false-positive detections, we implemented several post-processing steps. On the challenge test set, our best system obtained F-scores of 0.702 and 0.651 for entity recognition in the drug labels and in the Medline titles, respectively. For entity normalization, F-scores were 0.529 and 0.474. On the test set for ICD-10 coding, our system achieved an F-score of 0.848 (precision 0.886, recall 0.813). These scores were substantially higher than the average score of the systems that participated in the challenge.
We addressed these tasks using dictionary-based indexing approaches. For
entity recognition and normalization we used the system that we developed
for the same task in the CLEF eHealth 2015 challenge [
In the following, we describe the corpora, terminologies, indexing, and postprocessing steps separately for each subtask.. 2.1
on the Quaero medical corpus, a French annotated resource for medical entity
recognition and normalization [
The annotations in the Quaero corpus are based on a subset of the UMLS. An entity in the Quaero corpus was only annotated if the concept belonged to one of the following ten semantic groups (SGs) in the UMLS: Anatomy, Chemicals and drugs, Devices, Disorders, Geographic areas, Living beings, Objects, Phenomena, Physiology, and Procedures. Nested or overlapping entities were all annotated, as were ambiguous entities (i.e., if an entity could refer to more than one concept, all concepts were annotated). Also discontinuous spans of text that refer to a single entity could be annotated.
Coding. The data set for the coding of death certi cates is called the CepiDC corpus. Each certi cate consists of one or more lines of text and some metadata, including age and gender of the deceased, and location of death. The training set contained 65,843 certi cates from the period 2006 to 2012, and the test set contained 27,850 certi cates from 2013.
The annotations in the CepiDC corpus consist of codes from the ICD-10 and were assigned per text line. For each code that was assigned by the human coder, a term that supports the selection of the code was provided. This term was an excerpt of the text line or an entry of a coding dictionary (see below). Furthermore, the human coder provided for each code the duration that the deceased had been su ering from the coded cause, and a code rank with respect to the cause of death. 2.2
formed best in the CLEF eHealth 2015 concept recognition task [
Coding. For coding, we constructed two ICD-10 terminologies. A baseline
terminology was made by compiling the terms corresponding with each annotated
ICD-10 code in the training corpus. The number of times that each code had
been assigned in the training corpus, was also determined. For ambiguous terms,
i.e., terms that corresponded with more than one ICD-10 code, the term was
removed for those codes that occurred less than half as often as the most frequent
code with that term. A second, expanded terminology was based on the
baseline terminology, but also incorporated codes and terms from four versions of a
manually curated ICD-10 dictionary. These dictionary versions have been
developed at the Centre d'epidemiologie sur les causes medicales de deces (CepiDC)
[
with Peregrine, our dictionary-based concept recognition system [
Peregrine can nd partially overlapping concepts, but it cannot detect nested concepts (it only returns the concept corresponding with the longest term). We therefore implemented an additional indexing step. For each term found by Peregrine and consisting of n words (n > 1), all subsets of 1 to n{1 words were generated, under the condition that for subsets consisting of more than one word, the words had to be adjacent in the original term. All word subsets were then also indexed by Peregrine. We did not try to nd discontinuous terms since there frequency was very low.
Coding. For the coding task, we employed the open-source Solr text tagger
[
positive detections that resulted from the indexing, we applied several
postprocessing steps. First, we removed terms that were part of an exclusion list.
The list was manually created by indexing the French part of the Mantra
corpus, a large multilingual biomedical corpus developed as part of the Mantra
project [
Second, for any term-SG-CUI combination and SG-CUI combination that was found by Peregrine in the training data, we computed precision scores: true positives / (true positives + false positives ). For a given term, only term-SGCUI combinations with a precision above a certain threshold value were kept. If multiple combinations quali ed, only the two with the highest precision scores were selected. If for a given term none of the found term-SG-CUI combinations had been annotated in the training data, but precision scores were available for the SG-CUI combinations, a term-SG-CUI combination was still kept if the precision of the SG-CUI combination was higher than the threshold. If multiple combinations quali ed, the two with the highest precision were kept if they had the same SG; otherwise, only the combination with the highest precision was kept. If none of the SG-CUI combinations had been annotated, a single termSG-CUI combination was selected, taking into account whether the term was the preferred term for a CUI, and the CUI number (lowest rst). Coding. To reduce the number of false-positive codes that were generated during the indexing step, we computed a precision score for each term-code combination that was recognized by the Solr tagger in the training data: true positives / (true positives + false positives ). All codes that resulted from term-code combinations with precision values below a given threshold value, were removed. 3 3.1
We indexed the Quaero training data, and added the false-negative terms to our
terminology. We then ran the system on the Quaero test data, and submitted
two runs for both the entity recognition and normalization tasks: one run using
the system with a precision threshold of 0.3 (run1, this threshold was also used
in our last-year's submission for the same task [
As expected, run1 (the system with the lower precision threshold) has lower precision and higher recall than run2, but the di erences are small and the Fscores are nearly identical. The results are well above the average and median of the scores from all runs of the challenge participants. To determine the optimal precision threshold for the coding task, we split the CepiDC training data in two equally-sized sets. Precision scores were generated for all term-code combinations that were found using the expanded ICD-10 terminology in one half of the training data and were then used to lter the recognized term-code combinations in the other half of the training data. Table 2 shows the performance for di erent threshold values on the second half.
Without precision ltering (threshold 0.0), an F-score of 0.773 (precision 0.732, recall 0.818) was obtained. The highest F-score (0.827) was achieved for a threshold of 0.4, mainly because precision greatly improved (0.863), while recall only slightly deteriorated (0.795). The same optimal threshold value was obtained when we used the baseline ICD-10 terminology.
We submitted two runs on the CepiDC test set, one using the expanded ICD-10 terminology (run1), the other using the baseline terminology (run2). For both runs, precision scores were derived from all the training data and the threshold for precision ltering was set at 0.4. Table 3 shows the performance of our system, together with the average and median performance scores of the runs of all task participants.
Our results indicate that the baseline terminology (run2) performed slightly better than the expanded terminology (run1) in terms of F-score. Remarkably, the baseline terminology had higher recall than the expanded terminology. Overall, our performance results, in particular recall, are considerably better than the average and median score of all submitted runs.
We retrained our system for entity recognition and normalization that we
developed for the same task in the CLEF eHealth 2015 challenge, and newly developed
a dictionary-based system for ICD-10 coding of death certi cates. For both
systems, the performance on the test sets proved to be substantially better than
the averaged results of all systems that participated in the challenge [
Our system for entity recognition and normalization performed better on the
EMEA subcorpus than on the Medline subcorpus, primarily because of a higher
recall. This is in line with the results for this task in the CLEF eHealth 2015
challenge [
The baseline and expanded ICD-10 terminologies that we developed for our coding system, produced almost similar F-scores. Remarkably, although the expanded terminology contained much more codes and terms than the baseline terminology, recall for the baseline terminology was slightly higher. A probable explanation is that the term disambiguation that we performed when expanding the baseline terminology with the ICD-10 dictionaries supplied by the task organizers, e ectively prevented some term-code combinations seen in the training data from being included in the expanded terminology. Moreover, since the codes and terms in the baseline terminology were derived from a very large training set, there may have been few new codes and terms in the test set.
Our coding system achieved a very high precision of 0.886, with a recall of 0.813. Reasons for the lower recall include disambiguation errors, spelling mistakes and typos in the death certi cates, and missing terms in the terminology. Also, we noticed that some gold-standard code annotations erroneously corresponded with a line that preceded or followed the line that contained the term to be coded, which resulted in false-negative detections (and possibly false-positive detections that were actually correct). Further improvement of our system may be possible by using better curated terminologies and applying spelling correction techniques.