The SIFR BioPortal is an open platform to host French biomedical ontologies and terminologies based on the technology developed by the US National Center for Biomedical Ontology (NCBO). The portal facilitates the use and fostering of terminologies and ontologies by o ering a set of services including semantic annotation. The SIFR Annotator (http://bioportal.lirmm.fr/annotator) is a publicly accessible, easily usable ontology-based annotation tool to process French text data and facilitate semantic indexing. The web service relies on the ontology content (preferred labels and synonyms) as well as on the semantics of the ontologies (is-a hierarchies) and their mappings. The SIFR BioPortal also o ers the possibility of querying the original NCBO Annotator for English text via a dedicated proxy that extends the original functionality. In this paper, we present a preliminary performance evaluation of the generic annotation web service (i.e., not speci cally customized) for coding death certi cates i.e., annotating with ICD-10 codes. This evaluation is performed against the CepiDC/CDC CLEF eHealth 2017 task 1 manually annotated corpus. For this purpose, we have built custom SKOS vocabularies from the CePIDC/CDC dictionaries as well as training and development corpora, for all three tasks using a most frequent code heuristic to assign ambiguous labels. We then submitted the vocabularies to the NCBO and SIFR BioPortal and ran the annotation services on the task 1 datasets. We obtained, for our best runs on each corpus the following results: English raw corpus (69.08% P, 51.37% R, 58,92% F1); French raw corpus (54.11% P, 48.00% R, 50,87% F1); French aligned corpus (50.63% P, 52.97% R, 51.77% F1).
Biomedical data integration and semantic interoperability are necessary to enable
translational research. The biomedical community has turned to ontologies and
terminologies to describe their data and turn them into structured and formalized
knowledge [
The community has turned toward ontologies to design semantic indexes of
data that leverage the medical knowledge for better information mining and
retrieval. Despite a large adoption of English in science, a signi cant quantity
of biomedical data uses the French language. Besides the existence of various
English tools, there are considerably less terminologies and ontologies available
in French [
In the context of the Semantic Indexing of French Biomedical Data Resources
(SIFR) project, we have developed the SIFR BioPortal (http://bioportal.
lirmm.fr) [
The SIFR BioPortal includes the SIFR Annotator1 a publicly accessible and
easily usable ontology-based annotation tool to process text data in French. This
service is originally based on the NCBO Annotator [
The SIFR BioPortal also o ers the possibility of querying the original NCBO Annotator for English text via a dedicated proxy that extends the original
functionality. Thus, in this case, the +600 ontologies of the NCBO BioPortal2
may be used. This service, called the NCBO Annotator+3, is querying the original
NCBO Annotator while o ering new functionalities by pre processing of the
input text and/or post processing of the original results. For instance, we have
implemented the scoring of the results for both the SIFR and NCBO Annotator+
thanks to that proxy architecture [
In this paper, we present our participation to the task 1 [
The SIFR Annotator work ow is composed of several steps: dictionary creation from ontologies, text pre-processing, concept recognition, semantic expansion
(with mappings and hierarchy), annotation post-processing. For instance, in the
nal step, annotations are scored with relation to the context from which they
have been generated, which is a requirement when they are used to index the
original data. The SIFR Annotator can also recognize negation, experiencer and
temporality based on a customized French implementation of the NegEx/Context
technique [
Depending on the type of biomedical text, the annotator allows users to
annotate with only a subset of the ontologies available. The annotator is based
on the Mgrep [
An important aspect for the SIFR Annotator is to be available as a web service. The service results may be described in multiple syntaxes (XML or JSON-LD) and format (e.g., RDF/XML described with the Annotation Ontology or BRAT). A speci c CLEF eHealth output format has also been implemented to evaluate the service against previous campaigns (Quaero corpus). Akthough there is a web interface (Figure 1), the service is meant to be used through the REST application programming interface (API). We have created a Docker (www.docker.com) packaging that allows for an easy local installation to allow for the processing of sensitive in-house data, a common requirement when manipulating clinical data. All the code is open source and available on GitHub (https://github. com/sifrproject).
Within the SIFR project, we also developed an enhanced version of the
NCBO Annotator to annotate English biomedical text data, without having to
serve English ontologies locally. The NCBO Annotator+ uses a proxy service
architecture that enhances the capabilities of the original annotation service by
encapsulating around the original application programming interface. All the
extension implemented for the French annotator are thus automatically also
available for English e.g., the scoring of annotations [
For CLEF eHealth 2017 task 1, we have used the SIFR and NCBO Annotators (the software implementations) \as it is" without any speci c customization for the task, alough we used speci cally tailored dictionaries. For all the runs, the longest match only parameter was enabled, and we used no semantic expansion of the annotations, scoring or contextualization. 2.2
The objective of CLEF eHealth 2017 task 1 is to annotate death certi cates with ICD-10 codes both in French and in American English. For English, a corpus of death certi cates from the CDC was provided, split in a training and a test corpus. The training corpus contains 13,329 death certi cates, for a total of 32,714 lines. The test corpus contains 6,665 certi cates containing a total of 14,834 lines. For French, a corpus of death certi cates from CepiDC was provided: a training corpus of 65,844 documents and 195,204 lines, a development corpus of 27,851 document and 80,900 lines and a test corpus of 31,683 documents and 91,954 lines. The corpora are digitized versions of actual death certi cates lled in by clinicians. Although the punctuation is not always correct or present, the corpus is already segmented in lines (as per the standard international death certi cate model) which for the most part only contain single sentences.
The French corpus was provided in both an aligned and a raw format, while the English corpus was only provided in the raw format. The raw format provides two les, a CausesBrutes le and an Ident le. The former contains semicolon separated values for the Document identi er (DocID), the year the certi cated was coded (YearCoded), the line identi er (LineID), the raw text as it appears in the certi cate (RawText), an interval type during which the condition occurred (IntType - seconds, minutes, hours, weeks, years) and an interval value (IntValue). The Ident le contains a document identi er, the year the certi cate was coded, the gender of the person, the code for the primary cause of death, the age and the location of death. The aligned format is a reconciliation between the CausesBrutes and Ident les, where the elds have been aligned at the document and line number level. Thus, the aligned le contains the same unique elds that the original ones to which an extra eld is added in the gold standard dataset providing a standardized text that represents the manually annotated code.
For CLEF eHealth 2017 task 1, we have used only the \RawText" information of both the aligned and raw datasets. We did not use any other information/features such as age or gender contained in the les. 2.3
SIFR BioPortal already contained the French ICD-104 (CIM-10) reference
terminology. This OWL version was originally produced by the CISMeF team from
an automatic export from the HeTOP ontology/terminology server [
For the French tasks, a set of dictionaries was provided by CepiDC that give a standardized description text of each of the codes that appear in the corpora. Additionally, the data from the aligned corpus (French only) could also be used to enrich the lexical terms of ICD-10. A similar dictionary was provided for the English task. In order to use these dictionaries within the SIFR and NCBO Annotator, we had to encode them using a format accepted as input within the portal, which includes RDFS, OWL, SKOS, OBO or RRF (UMLS format). In this case, the ideal choice in terms of standardization, potential reusability and simplicity was to use SKOS (Simple Knowledge Organization System) a W3C Recommendation specialized for vocabularies and thesaurus. For CLEF eHealth 2017 task 1, we produced two groups of SKOS dictionaries: CIM-10DC* for French, based on the French dictionaries and aligned corpus; ICD-10-CDC* for English based on the CDC corpus dictionary alone.
We set out in this construction process by rst de ning the appropriate
schema to represent the SKOS dictionaries. We chose to use the same URIs as
concepts identi ers for skos:Concept that the owl:Class in the available
CIM10/ICD-10 which allows our dictionaries to be fully aligned ontologically speaking
with the original terminologies they enrich. Each of the codes was represented by
a skos:Concept. The URIs are composed of a base URI and a class identi er
that represents the ICD codes, in the following format: "[A-Z][
Construction algorithm We built the French SKOS dictionary from the aligned corpus and all the CepiDC dictionaries. We built the English one only from the raw corpus and the CDC dictionary. We rst built a code index, that to each code associated the list of labels retrieved from: the DiagnosisText eld in the dictionary, associated to codes through the ICD1 and ICD2 elds 7; the RawText and StandardText (only for French) elds from the corpus associated to codes through the ICD-10 eld in the corpus 8.
For each code concept the CepiDC and CDC dictionaries contained multiple labels. In order to follow SKOS speci cation, we had to select a preferred name automatically (skos:prefLabel) and assign the other labels as alternative labels (skos:altLabel). Note that this selection would note in uence the annotation process as both preferred name and synonyms are included in the concept recognizer dictionaries. The selection heuristic took the shortest label that does not contain three or more consecutive capital letter (likely and acronym).
SKOS dictionaries was to assign ambiguous labels (i.e., identical labels which correspond to di erent codes). Indeed, those labels create ambiguity in the annotations and leads to better recall at the price of a low precision. For example, the label "choc septique" was present as preferred label or synonyms for 58 di erent codes. Therefore, we had to implement a selection heuristic to determine the most suitable code to which the label should be bound. 7 For French, we used a concatenation of all the dictionaries and for English we used the one dictionary le provided. 8 AlignedCauses 2013Full for French, CausesCalculees EN training for English.
When using both the standard text and the raw text elds from the corpus, if standard text label is ambiguous, a simple heuristic is to not add it to any code but just use the raw text instead. Given that the raw text is unique, the ambiguity related to the inclusion of the test corpus is removed. We called this rst strategy "Adaptive dictionary generation" and created a CIM-10DCA French SKOS dictionary to evaluate it. Given that this strategy relied on the availability of a standardized text, it was con ned to the French aligned task, as the raw English corpus contained no standard labels.
The drawback with the previous heuristic is that we lose some labels that would otherwise have potentially increased recall. Thus, we searched a way of assigning ambiguous labels to one code only. Taking inspiration from the idea of the most frequent sense baseline often used in Word Sense Disambiguation tasks, we adopted a heuristic that assigns ambiguous labels to the most frequent code only. We use the training corpus to estimate the frequencies of use of the codes (gold standard annotations) so that when a label can belong to several codes, we can sort the codes by frequency and chose either the most frequent code (MFC) or the top k most frequent codes (kMFC).
In practice the "Adaptive" strategy led to a much lower recall without particularly improving precision. Final F1 scores were worse than with the MFC strategy which led to a precision and recall that were balanced. This is the strategy we have nally used in the reported results.
with the best ambiguous label strategy have been uploaded respectively on the SIFR and NCBO BioPortal, and are accessible in private mode, only for the replication track of the task. CIM-10DC-ALL contains 6817 concepts for a total of 295,385 labels. CIM-10DC-ALLMFC contains the same number of concepts but only 249,524 labels.ICD-10CDC contains 3738 concepts with 166,500 labels.
We have also created a resource solely from the CepiDC dictionaries (without using the corpus) called CIM-10DCD, that does not contain any sentences originally present in death certi cates. We are currently discussing with the CepiDC, for a potential public release of the SKOS dictionaries as well as the Work ow to update them on a regular basis. Indeed, we believe it is also part of the SIFR project mission to facilitate open access to resources (and adopting standard ways of describing them e.g., semantic web standards), when licensing permits.
Given that we used the SIFR and NCBO Annotators, besides manually curating the created SKOS dictionaries, the nal step to obtaining a working system for the task was to write a complete work ow to:9 1. Read the corpus in the raw or aligned formats; 9 For this purpose, we used the Java language. 2. Send the text to the Annotators REST API with the right ontologies and annotation parameters and retrieve the annotations produced by the Annotators; 3. Optionally prune some annotations (post-annotation heuristic); 4. Produce the output in the right raw or aligned format.
We implemented two post-annotation heuristics. Most Frequent Code, where if a particular line was annotated with several codes, only keep the most frequent code based on the code distribution of the training corpus. Code Frequency Cuto , we calculate a normalized probability distribution of the codes that annotate a particular line and only keep the codes below a cumulative probability threshold.
The parameters of the entire system are the combination of the parameters of NCBO or SIFR annotator (list of ontologies, longest match (T/F), expand mappings (T/F)) with any post-annotation heuristic parameter. 2.5
Using the SIFR annotator is as simple as sending a request through the HTTP REST API, for example to annotate the sentence: \Absence de tumeur maligne" with the French version of WHO-ART and MedDRA, it can be done with:10 http://services.bioportal.lirmm.fr/annotator/?text=Absence%20de% 20tumeur%20maligne&negation=true&ontologies=WHO-ARTFRE,MDRFRE
Because of the sensitive nature of the CepiDC data, we have used a local version of the NCBO and NCBO Annotators in order to avoid sending the data out on the network. However, to reproduce our results one would require to have access to the private SKOS dictionaries on the NCBO and SIFR BioPortal and in that case, use the user interface or the REST web service API. The reproduction instructions are available here: https://twktheainur.github.io/ bpannotatoreval/LIRMMCLEF2017Task1Instructions.html 3
Using the previously described work ow, we have performed six runs: French Run 1 (Aligned and Raw): Annotation with longest only parameter on and running on a local instance of the SIFR Annotator with CIM-10 and CIM-10DC-ALLMFC as target resources.
French Run 2 (Aligned and Raw): A fallback strategy starting from the result le of Run 1, and, for each line without any annotations, takes the annotations from a second run, which used CIM-10 and CIM-10DC-ALL as target resources. This is, in essence a late fusion technique, that aims at increasing the recall, without sacri cing precision. 10 The REST API requires an APIkey to be used (obtained by creating an account on the portal). In that example, one can use the demo API key and add &apikey=c34b76530639-4946-81af-8ac76fe809dd at the end of the call.
English Run 1 (Raw): Annotation with longest only parameter on and running on a local instance of NCBO Annotator with ICD-10 and ICD-10CDC as target resources.
English Run 2 (Raw): Same as Run 1 but with ICD-10CM as additional target (also available in the NCBO BioPortal): the Clinical Modi cation of ICD-10 made in the USA for the classi cation of morbidity causes. 3.1
11 teams participated for English raw. Table 1 presents the results obtained by our two runs against the average and median results of the runs submitted to this task. The NCBO Annotator obtained results that are exactly the median value of all the results submitted (all causes). We can measure a slight decrease in precision and increase in recall with the introduction of ICD-10-CM in Run 2. Regarding the external causes, the NCBO Annotator obtains a better precision and f-measure than the average and median results submitted to the challenge. 13 runs have been submitted by 9 teams to the French raw evaluation. 7 runs have been submitted by 5 teams to the French aligned evaluation. Tables 2 and 3 present the results obtained by our two runs against the average and median results of the runs submitted to this task. As expected, the SIFR Annotator did perform similarly on the raw and aligned datasets (as they were processed exactly with the same work ow). The results are exactly the median value of all the results with the raw dataset, but slightly under the median value for the aligned datasets (all causes). Indeed, teams that have used other information from the aligned dataset probably performed better than the SIFR Annotator here. Regarding the external causes, we obtain better precision and F1 than the average and median results submitted to the challenge.
Run1
Run2
The results obtained are in line with what could be expected from our approach,
which really is an evaluation of how the SIFR and NCBO Annotators concept
recognition component works. The simple string matching approach adopted by
Mgrep generally o ers a good precision (around 80%) if the ontology is properly
lexicalized and concretely captures the terms used in the text to annotate. For
CLEF eHealth 2017 task 1, the loss in precision is explained by the nature of
the dictionaries used as a source to produce our terminology: a same label can
correspond to several di erent classes (here ICD10 codes). This practice is usually
strongly avoided when designing ontologies as it inevitably creates ambiguities.
Concerning recall, our performance is also limited since the concept recognizer
does not include any natural language processing techniques that would increase
the amount of matches, handle morphological variants (as simple as plural forms)
or any other alternative concepts. All phenomena that are common in reality
but not captured as synonyms by the source ontologies will not be recognized
properly. Previous evaluation of the NCBO Annotator [
NCBO and SIFR Bioportal already include the standard versions if ICD10 and CIM-10, but as mentioned in Section 2.3, they are not meant to be used for clinical annotation but to serve as a reference for clinicians independently from actual clinical text, just using these for the tasks yields unsatisfactory results (FR Raw: P=27.8, R=03.8, F=06.7; FR Aligned. P=27.2, R=04.3, F=07.4; EN Raw P=31.1, R=08.7, F=13.6), which explains the necessity of using the dictionaries and corpora provided with the task.
The purpose of the SIFR Annotator, and originally of NCBO Annotator, was not to beat task-speci c state-of-the-art systems. The concrete advantages of the services, both connected to their respective portals come from: (i) the size and variety of their dictionaries coming from ontologies, (ii) their availability as a web service that can be easily included in any semantic indexing work ow, and nally (iii) their adoption of a semantic web vision that strongly encourages using dereferenceable URIs that can then reused to facilitate data integration and semantic interoperability. One should also note that the semantic expansion step (which uses the mappings between ontologies and the is a hierarchies to generate additional annotations) as well as the post-processing of the annotations (which scores and contextualizes the annotations) are interesting exclusive features that not evaluated within CLEF eHealth 2017 task 1.
Despite of their limitations, the NCBO and SIFR Annotators obtained median results when compared to the performance of all the participating systems. Therefore, considering the other discussed advantages, we believe they are two services that can help in a wide class of text mining or annotation problems, but of course not for all. It is important to note the systems were not tailored for this task and their performance will highly vary depending of the data to annotate and the ontologies targeted.
Participation in CLEF eHealth 2017 task 1 is a good way of improving our SIFR Annotator and potentially the NCBO Annotator also. Such improvements shall be either generic (changes to the overall work ow, independent of task) or tailored for improving the results to the CLEF eHealth series (Quaero or CepiDC corpora). In order to better understand the shortcoming of the system, we sampled 100 false positives and false negatives from the best runs of the SIFR Annotator on the French aligned development dataset and proceeded to manually determine the case of the error. Some class of errors are as follows: { Errors because of missing synonyms. A few good illustrative examples include: The code R09.2 \arr^et respiratoire" was not identi ed within the text \arr^et cardio respiratoire" or \detresse cardiorespiratoire." The code J96.0 \insu sance respiratoire aigue" was not identi ed within the text \detresse respiratoire." Although some of these false negative could be avoided with a richer dictionary (1st case) or simple synonym generation (2nd case), we found some that strongly relies on some medical expertise that can hardly be captured by a dictionary based approach (maybe by a machine learning one, assuming there is enough data to train the tool). { Single match returned whereas a multiple match was expected. Indeed, the MFC strategy resulted in assigning the synonyms to only one code in CIM10-DC therefore, when for the same text, several annotations where expected, we found only one. For instance, the code G40.9 "epilepsie, sans precision" was found with the text "epilepsie avec etat de mal" but not the code G41.9 "etat de mal epileptique, sans precision." Note that those errors are only present with the MFC strategy. { Morphosyntactic or lexical variation (e.g., accent, dash, comma, spelling).
For instance, the code J18.9 \emphyseme, sans precision" was not identi ed within the text \emphyseme pumonaire secodaire tabagisme actif" because of the spelling of "pulmonaire." { Annotations were made with a more general (i.e., parent in ICD-10 hierarchy), often because of a partial match within an expression. { Errors cased by implicit semantic information that requires medical knowledge to identify. E.g. I10 \hypertension essentielle (primitive)" was not found from the text \TC suite a une chute avec epilepsie sequellaire et tr cognitifs" as it was annotated within the corpus. Or the code R68.8 \autres sympt^omes et signes generaux precises" was not identi ed within the text \atteinte polyviscerale di use."
From this review of the pitfalls of the SIFR Annotator on the CepiDC corpus, and from other in-house experiments, we clearly identi ed the need to improve the dictionary generation process when extracting the labels from the source ontologies. The terms can be enriched by adding other alternative synonyms or morphosyntactic or lexical variations, such that we increase recall without decreasing precision.
Another source of improvement for the SIFR Annotator comes from generating and curating alignments between ontologies. For instance, on the CepiDC corpus, when used with mappings to other ontologies, the SIFR Annotator was able to identify unambiguous concepts. The system uses the mappings between ontologies to expand the original direct annotations made from the text, bu only if the exist and are uploaded to SIFR BioPortal. In the case of ICD10, there exists multiple sources of published mappings that we plan to upload in the future so as to improve recall.
The limitations on one particular application do not generalize to others. We
are currently evaluating the performance of the SIFR Annotator on the Quaero
corpus [
We would also like to point to some recent improvements that we have made
to the SIFR Annotator that are currently under evaluation. When annotating
clinical notes with medical conditions, it is important to lter out negated conditions
and distinguish present conditions from antecedents or conditions experienced
by someone other than the patient. For this reason, several methods have been
proposed to detect the context of already identi ed clinical conditions, especially
for English medical text. The English-language system, NegEx/ConText, is one
of the best and fastest algorithms for the determination of the context of medical
conditions [
In this paper, we presented our participation to the task 1 of the CLEF eHealth 2017 challenge using the NCBO and SIFR Annotators. Our results are encouraging: around 50-60% of F1 score means that more than half of the task of coding death certi cates with ICD-10 codes can be automatized. Especially considering that we have not implemented anything speci c to process these data. But of course, we will have to improve these results to be among the best performing systems. Some improvements perspectives have been discussed.
We have also argued in this paper that according to us the technical performance (F1) shall not be the only argument in evaluating a semantic annotation tool. The SIFR project has invested a signi cant amount of e ort in order to o er a generic, open and quite robust platform that could easily be used \at the click of the mouse" to annotate biomedical text data and access French biomedical ontologies and terminologies. Someone can upload a new resource to the SIFR BioPortal and get a dedicated annotation service, interconnected to other existing ontologies, in a couple of hours.
Indeed, the SIFR Annotator is di erent from other related tool in French biomedical text mining as: (i) it is a dynamic web service with JSON-LD outputs which can be integrated in current programmatic work ows; (ii) it uses public ontologies both to create annotations and to expand them; (iii) it has access to one of the largest available sets of publicly available biomedical ontologies in French. We believe the SIFR Annotator can therefore be used in a large span of biomedical applications including annotating clinical text data. We are currently using the service in the context of the French PractiKPharma project (http://practikpharma.loria.fr) which aims to validate pharmacogenomics state-of-the-art knowledge on the basis of practice-based evidences, i.e., knowledge extracted from electronic health records.
This work is supported by the French National Research Agency within the PractiKPharma (grant ANR-15-CE23-0028) and SIFR (grant ANR-12-JS0201001) projects as well as by the European H2020 Marie Curie actions (grant 701771), the University of Montpellier and the CNRS. We also thanks the US National Center for Biomedical Ontology for their assistance with the NCBO Annotator and the CLEF eHealth 2017 organizers for their help.