In this paper we present two tools for facing task 2 in CLEF eHealth 2016. The rst one is a semantic tagger aiming to detect relevant entities in French medical documents, tagging them with their appropriate semantic class and normalizing them with the Semantic Groups codes de ned in the UMLS. It is based on a distant learning approach that uses several SVM classi ers that are combined to give a single result. The second tool is based on a symbolic procedure to obtain the English translation of each medical term and looks for normalization information in public accessible resources.
We developed a semantic tagger for the medical domain [
Semantic Tagging is the task of assigning to some linguistic units of a text a unique tag from a semantic tagset. It can be divided in two subtasks: detection and tagging. The rst one is similar to term detection and Named Entity Recognition, while the latter is closely related to Named Entity Classi cation.
The key elements of Semantic Tagging task are: (i) the document, or document genre, to be processed, (ii) the linguistic units to be tagged and (iii) the 4 http://en.wikipedia.org 5 https://sites.google.com/site/clefehealth2016/ 6 https://sites.google.com/site/clefehealth2016/task-2 tagset. All these elements play a crucial role for the success of the task. In this concrete task our constraints are the following: (i) documents of medical domain, mainly scienti c articles indexed in MEDLINE and some drug monographs published by the European Medicines Agency (EMEA), (ii) the linguistic units to be tagged are the terminological strings found in the source documents, (iii) the tagset will be a subset of the top UMLS categories. Such resources will be used also for the normalization on the medical entities as de ned in the phase II.
Our approach consists of learning a binary classi er for each of the categories, whose results are combined using a simple voting schema. The cases to be classi ed are the mentions in the document corresponding to TCs, to refer to any of the concepts in the tagset. No co-reference resolution is attempted and, so, co-referring mentions may be tagged di erently. For the normalization of the entities found we used the resources available through BioPortal7.
After this introduction, the organization of the article is as follows: In section 2 we sketch the state of the art of Semantic Tagging approaches. Section 3 presents the methodology followed in the current task. The experimental framework is described in section 4. Results are shown and discussed in section 5. Finally section 6 presents our conclusions and further work proposals. 2
English is, by far, the most supported language for biomedical resources and tools. The National Library of Medicine8 (NLM R ) maintains the Uni ed Medical Language System9 (UMLS R ) that groups an important set of resources to facilitate the development of computer systems to \understand" the meaning of the language of biomedicine and health. It is worth noting that only a small fraction of such resources exist for other languages.
A relevant aspect of information extraction is the recognition and identi cation of biomedical entities (like disease, genes, proteins . . . ). Several Named Entity Recognition techniques have been proposed to recognize such entities based on their morphology and context. NER can be used to recognize previously known names and also new names, but cannot be directly used to relate these names to speci c biomedical entities found in external databases. For this identi cation task, a dictionary approach is necessary. A problem is that existing dictionaries are often incomplete and di erent variations may be found in the literature; therefore it is necessary to minimize this issue as much as possible.
2015 edition of CLEF eHealth contest contained two tasks focusing on
information extraction and information retrieval. The topic of one of them was
Clinical Named Entity Recognition in medical texts written in French (Task 1b)
[
The core idea of the system is that for a term t known to belong to the semantic category c (one of the 10 UMLS categories we deal with) not only the occurrences of t in the training material can be considered positive examples for learning but also the occurrences of t in its associated WP page if existing. This hypothesis is important because for some semantic categories the training material contains not enough terms for accurate learning.
Following [
We created a suite of binary contextual classi ers, one for each semantic
class. The classi ers are learned using, as in [
For every le of the training corpus, each tagged term is considered as a positive example for the tagged class and negative example for the rest of the classes. Features are the words occurring in the local context of mentions. The context size and POS of the context words are parametrizable.
Examples for learning correspond to the mentions of the seed terms in the corresponding WP pages. Let t1; t2; : : : ; tn the seed terms for the semantic class c, i.e. ti 2 ST c. For each ti we obtain its WP page and we extract all the mentions of seed terms occurring in the page. Positive examples correspond to mentions of seed terms corresponding to semantic class c while negative examples correspond to seed terms from other semantic classes. Frequently, a positive example occurs within the text of the page but often many other positive and negative examples occur as well. Features are simply words occurring in the local context of mentions.
For French we have used for processing documents, in learning and test
phases, the Freeling toolbox10 [
Term candidates, TC, are selected according to morpho-syntactic criteria. We have used for ltering the following regular expression: NA*(PNA*)+. Additionally, in order to take into account the peculiarities of the term selection of CLEF organizers we also decompose each complex term in its components (see section 4.1 for more details and examples).
The learning process has been performed using for each semantic category the most likely relevant documents including EMEA and MEDLINE training documents and WP pages obtained as described above. From the WP pages, besides those with purity less than 1, short pages and pages consisting mainly of itemized material or non-textual fragments were removed too.
For each example, the feature vector captures a context window of n words to its left and right11 without surpassing sentence limits. 3.2
A careful analysis of our results on CLEF e-Health 2015 participation revealed that some apparently easy to detect terms were not detected or were classi ed incorrectly. For instance French terms occurring in French DBPedia12 or translated English terms occurring in English (Princeton) WN or in English DBPedia were not detected. We decided, thus, combining, in our run2, the results of run1 10 http://nlp.lsi.upc.edu/freeling/ 11 In the experiments reported here n was set to 3. 12 http://wiki.dbpedia.org/ with two other systems, one based on the performance of a state-of-the-art term extractor, YATE, tuned to work in the medical domain, and the other based on an external knowledge source, the DBPedia. Although domain independent, DBPedia has a nice coverage of medical classi ed terminology and o ers good interlingual capabilities.
basically performs using the taxonomic structure of the nominal part of WN. Given a domain d, the medical domain here, YATE obtains the called Domain Borders, synsets that are likely to belong, both them and their descendants to d. These Domain Borders are used later for extracting from a document the set of mentions corresponding to terms belonging to d, i.e. those TC whose synsets are placed below a Domain Border. Right part of Figure 2 shows this process.
YATE uses English WN, therefore for applying this tool a translation of French TC s is needed. As the terms we are interested on are those represented in the French WP, we used the interwiki links between French and English WP (using DBPedia for getting these links). This process results on the extraction (tagging) of the medical terms occurring in the test documents. YATE is a term extractor, not a semantic tagger, so it is not able to classify the extracted terms, but, indirectly, these terms can be used as seed terms for learning the classi ers.
wikiYATE [
DBPedia-based approach Some useful information in WP is represented in infoboxes and, thus, has been automatically mapped into the corresponding DBPedia rdf triples. We take pro t of several interesting properties of DBPedia: { There exist DBPedia datasets for English (http://dbpedia.org/sparql) and
French (http://fr.dbpedia.org/sparql). { Entities (resources) in the two datasets are frequently linked through sameAs properties. { Entities in the datasets are frequently mapped to one or more linguistic referents, words and phrases, through label properties, sometimes in several languages. So an entity in the English DBPedia can be labelled with French words or phrases.
As shown in Figure 3, iterating over French and English terms and resources in French and English DBPedia datasets through the label and sameAs properties, we are able to collect from an initial French TC, t, the set of English resources likely corresponding to translations of t.
In order to be able to classify t into one of the 10 semantic categories we proceed on the following way:
During training, for each semantic category c we collect all the French terms t occurring in the training dataset and tagged with c. We lter out from these sets the terms not occurring in French WP. We then collect, as described above, the set of English DBPedia resources associated to them. For each of such resources r we obtain the set of classes to which r belongs (we reduce our search to DBPedia and YAGO classes) using the type property. From each class we recursively collect the set of super-classes using the subClassOf property. Some of the super-classes belong unambiguously to one semantic category. For instance, http://dbpedia.org/class/yago/AliphaticCompound114601294 occurs as super-class of 6 terms all classi ed as CHEM. Others, as, http://dbpedia.org/ontology/Eukaryote, are ambiguous. This super-class occurs 4 times as PHYS, 8 times as LIVB, and 2 times as DISO.
For each semantic category we collect the set of unambiguous super-classes. For ambiguous cases we proceed as follows: If the total number of terms covered by the super-class is higher than a threshold THR1 and the ratio between the higher option and the second is higher than a threshold THR2 we assign the super-class to the set corresponding to the rst option. THR1 and THR2 have been manually set to 30 and 4. For instance in http://dbpedia.org/class/yago/Location100027167, occurring 10 times as ANAT, 13 times as CHEM, and 76 times as GEOG, the two conditions hold (76+13+10 >30 and 76/13 >4, so the super-class is included into the set of super-classes corresponding to the semantic category GEOG.
In this way a number of super-classes have been associated to each semantic category. See in Table 1 the size of each set and an example of their members.
Once collected these sets (during the training phase) the process at test phase is quite straightforward. See left part of Figure 2. For each TC in the test dataset, following the approach described above, we obtain the set of English DBPedia resources. The DBPedia and YAGO classes of these resources are then obtained and from them the set of super-classes. This set is intersected with the sets of super-classes associated to each semantic category. The sizes of all the intersections are computed and the category associated to the higher size is returned as category of the term (ties are solved according to the most frequent semantic category in the training dataset). Finally the results of run1 (a semantic category), DBP-based (a semantic category), and YATE-based (a Boolean) are combined for getting the nal result. 3.3
For normalization we have used the BioPortal SPARQL endpoint. The a priori
obvious way of accessing UMLS and obtaining the CUI of a term t consisted on
getting a UMLS entity labelled with t and then getting the CUI of this entity.
This simple procedure does not work because UMLS is not directly labelled
with terms. We have used instead an indirect access to UMLS through other
ontologies. We have employed for such process Snomed-CT, Mesh, and RCD.
The process consists on translating French into English, using the approach
described above and then accessing to any of the intermediate ontologies and
from them to UMLS. In Figure 4 one of the SPARQL templates used is presented.
This template is instantiated into a real query just by replacing the placeholders
'**ont**' by the name of one of our three ontologies and '**term**' by the
name of the English TC. As can be seen in the Figure, this template uses the
prefLabel (preferred label) link. We have also built templates using the altLabel
(alternate label) link, and others using approximate string matching, for covering
decreasingly con dent matchings.
Participants of CLEF2016 are requested to perform named entity recognition and
normalization on a dataset of scienti c article titles and full-text drug inserts. For
performing such tasks we designed the working frameworks that are described
in following subsections.
Our basic working framework for entity recognition was the same than in CLEF2015.
But taking into account the results obtained in such contest (see [
As mentioned in section 3.1 the learning phase was made using the distant learning paradigm. For each mention of a TC the vector of features is built and the nine13 learned binary classi ers are applied to it. For building such classi ers all the documents of the training corpus were linguistically processed 13 In our run 1, the process of extracting GEOG entities was performed by a Geographic
NER, and, so only nine classi ers were learnt.
using the Freeling suite (see [
The Quaero corpus takes into account nested terms as di erent terms. Given this fact, when the TC is poly-lexical, all the possible combinations of components are taken into account. Table 2 shows some examples.
We also decided to include a second run that, starting with run 1 results, improves them by doing some symbolic processes as shown in Figure 2 and described in the following paragraphs.
{ Term extraction and analysis. For preparing run 2 we create a single
document that includes all documents of the test corpus. We analyse such material
with wikiYATE, a term extraction tool that uses WP for obtaining the TCs
of a given text (see description in [
Finally, the results of both analysis has been combined in a single result that was used to improve the result run 1. 4.2
The process of Entity normalization was performed independently from the process of entity recognition. This is why we submitted runs for the task of plain 14 http://adimen.si.ehu.es/web/MCR/ entity recognition and not to the task of normalized entity recognition. See 3.3 for details on our approach. 5
Table 3 depicts the global results as reported by the organization of CLEF2016
(phase I task entity recognition). The material o cially delivered included two
runs. Unfortunately, for the run 2 we incorrectly submitted the same material as
in run 1. After detecting such issue the organisation kindly accepted to evaluate
our actual run as an uno cial result. For this reason we tagged with an \*"
run2 results showed in Table 3. Additionally we performed an after challenge
improvement of our system (noted as "/3*" in the table) introducing a simple
voting mechanism for unifying the tags corresponding to multiple mentions of
the same TC in the case enough evidence for one of the choices exists.
{ entities exact match entities inexact match
docs TP FP FN Prec. Recall F1 TP FP FN Prec. Recall F1
EMEA 517 558 558 0,4809 0,4809 0,4809 517 558 558 0,4809 0,4809 0,4809
MEDLINE 673 745 748 0,4746 0,4736 0,4741 673 745 748 0,4746 0,4736 0,4741
The results obtained for the Phase I (entity recognition) are poor (although
better than those proposed in CLEF2015, see [
Indications, radiotherapie, tumeurs digestives, tumeurs, digestives
As already shown in [
Another minor issue is that text seems to include some kind of extra segmentation (see for example: l' enfant or d ' activation plaquettaire induite par l ' heparine among many others). The words by themselves are not important but such segmentation may cause errors in the POS tagging stage and this fact may be a real problem for TC delimitation (\l" and \d" will become a noun instead of a determiner and preposition respectively). Also, the generation of the nal stand-o annotation becomes a bit more complicated.
Table 6 shows a detailed analysis of run 1 results. From one side, there are two classes (PHEN and GEOG) that does not produce any correct result and another class that only detects one valid term (OBJC). In these cases, the corresponding classi ers have a extremely low accuracy, probably due to the lack of training examples. So, acquiring additional examples for these cases should result on some improvement. From other side, there are some classes (DISO, PROC and ANAT) where the number of examples is much higher and therefore show a better result .
Table 7 shows the same analysis for run 2. There is an improvement in the performance for all the classes showing that: (i) the symbolic analysis partially solves the inaccuracies of the machine learning system and (ii) the combination of methods improves the global e ciency. 6
The organizers of CLEF eHealth 2016 divided the task 2 in two phases: entity recognition and entity normalization on French medical text ofthe Quaero corpus. Our approach results in two di erent systems for solving each task.
For the rst task, we have presented a system that automatically detects and
tags medical terms in medical documents using a tagset derived from UMLS
taxonomy. The results of the system for entity recognition, as discussed in previous
section are poor, far from the obtained in our previous system (performing on
medical English wp pages and con rmed for other languages, including French,
[
It is interesting to observe that in all cases the improvement is higher in the inexact match than in the exact match. This fact may reveal some issues in the TC delimitation but also in the o set calculation. The latter issue is magni ed by the tokenization of the training corpus that di culties the linguistic analysis and the o set calculation.
The second task was solved using a totally di erent system. It is based in obtaining the normalization information from public resources after obtaining the English translation of each medical term. The results were a bit below of the other participants . Again, tokenization is an issue that a ects the performance of the system.
Several research lines will be followed in the next future: { The integration of both entity recognition and normalization in a single task may bring mutual bene ts. { To enlarge the use of BioPortal for looking in the ontologies for the recognition and classi cation task seems to be a promising direction. { A combination and/or the specialization of the resources for learning more accurate classi ers. The application of the DBPedia based approach, to all the semantic classes merits a deeper investigation. { A careful combination of learning from the training dataset and from additional material, as WP should be experimented. { The features currently used for learning the classi ers are rather crude and need some revision. We foresee to do some experimentation weighting the features, separating the features according its position in relation to the TC and adding new features as: start/end characters, typed features, etc. { Moving from semantic tagging of medical entities to semantic tagging of relations between such entities is a highly exciting objective, in the line of recent challenges in the medical domain (and beyond). { Improving the selection of medical entities by using POS pattern learning, adapting our term extractor to the tagging policy of medical entities in Quaero corpus and improving adaptation of Freeling to French medical texts. 7
This work was partially supported by the TUNER project (Spanish Ministerio de Econom a y Competitividad, TIN2015-65308-C5-5-R).