In this paper we present a semantic tagger aiming to detect relevant entities in French medical documents and tagging them with their appropriate semantic class. These experiments has been carried out in the framework of CLEF2015 eHealth contest that proposes a tagset of ten classes from UMLS taxonomy. The system presented uses a set of binary classi ers, and a combination mechanisms for combining the results of the classi ers. Learning the classi ers is performed using two widely used knowledge source, one domain restricted and the other is a domain independent resource.
Recently, we [
Semantic Tagging (ST ) can be de ned as the task of assigning to some linguistic units of a text a unique tag from a semantic tagset. It can be divided in two subtasks: detection and tagging. The rst one is similar to term detection and Named Entity Recognition (NER), while the latter is closely related to Named Entity Classi cation (NEC ).
Other Natural Language Processing (NLP ) tasks related to Semantic Tagging are Word Sense Disambiguation (WSD ), aiming to tag each word in a document with its correct sense from a senses repository, and Entity Linking (EL), aiming to map mentions in a document to entries in a Knowledge Base.
The key elements of Semantic Tagging task are:
i) the document, or document genre, to be processed. In this paper we focus
on the medical domain and the genre of documents are those included in
CLEF2015 contest, namely EMEA and Medline documents in French (see
[
Our approach consists of learning a binary classi er for each of the categories7, whose results are combined using a simple voting schema. The cases to be classi ed are, according the contest instructions, the mentions in the document corresponding to term candidates, to refer to any of the concepts in the tagset. No co-reference resolution is attempted and, so, co-referring mentions could be tagged di erently.
Most of the approaches to Semantic Tagging for small-sized tagsets, as our, use supervised Machine Learning (ML) techniques. The main problem found when applying these techniques is the lack of enough annotated corpora for
NERC is used.
learning. In our system we overcome this problem following a distant learning
approach. Distant learning is a paradigm for relation extraction, initially proposed
by [
After this introduction, the organization of the article is as follows: In section 2 we sketch the state of the art of Semantic Tagging approaches. Section 3 presents the methodology followed in our previous work while Section 4 discusses the modi cations performed for dealing with the current task. The experimental framework is described in section 5. Results are shown and discussed in section 6. Finally section 7 presents our conclusions and further work proposals. 2
English is, by far, the most supported language for biomedical resources and tools. The National Library of Medicine9 (NLM R ) maintains the Uni ed Medical Language System10 (UMLS R ) that groups an important set of resources to facilitate the development of computer systems to \understand" the meaning of the language of biomedicine and health. It is worthnoting that only a small fraction of such resources exist for other languages.
A relevant aspect of information extraction is the recognition and identi cation of biomedical entities (like disease, genes, proteins . . . ). Several Named Entity Recognition techniques have been proposed to recognize such entities based on their morphology and context. NER can be used to recognize previously known names and also new names, but cannot be directly used to relate these names to speci c biomedical entities found in external databases. For this identi cation task, a dictionary approach is necessary. A problem is that existing dictionaries often are incomplete and di erent variations may be found in the literature; therefore it is necessary to minimize this issue as much as possible.
There is a number of tools that take pro t of the UMLS resources. Some the
more relevant are:
{ Metamap [
connecting to a publicly available speci c databases (e.g. UniProtKb/SwissProt, gene ontology, DrugBank. . . .). { Semantrix 11 is a private company that has developed the Ontotext Semantic Biomedical Tagger. It is an information extraction system designed to process biomedical texts using a number of biomedical databases.
Keeping on the medical domain, an important source of information are the
proceedings of the 2010 i2b2/VA challenge on concepts, assertions, and
relations in clinical text 12 [
A more recent and also interesting source of information is the DDI
Extraction 2013 (task 9 of Semeval-2013) [
Outline
As we mentioned above, the system presented here is heavily based on [
We obtain an instance-based classi er (upper section in Figure 1) for each semantic class using seed words extracted from three widely used knowledge sources (section 3.2). The only form of human supervision is, as described below, the assignment of about two hundred Wikipedia categories to their appropriate SNOMED CT semantic class. Later (lower section in Figure 1) such models are used to classify new instances. 3.2
Features extraction
To obtain the seed terms needed for learning the classi ers, we proceed in three
ways, using two di erent general purpose knowledge sources, Wikipedia and
DBPEDIA, and one, SNOMED CT, speci c for the medical domain (see [
Wikipedia, although being a general purpose resource, densely covers the
medical domain; it contains terminological units from multiple medical thesauri
and ontologies, such as Classi cation of Diseases and Related Health Problems
(ICD-9, ICD-10), Medical Subject Headings (MeSH), and Gray's Anatomy, etc.
We describe here the main characteristics of the method followed to obtain the
seed terms from Wikipedia, for the other sources [
First we got the set of the most reliable Wikipedia categories14. This resulted
on a set of 237 Wikipedia categories. We manually assigned to such categories
a unique SNOMED CT class from the set of 6 most frequent ones. For each of
these categories we obtained the full set of associated pages. For each page, we
14 See [14] for details about the way of obtaining such categories from Wikipedia
resources
calculate a purity factor, i.e. a score (ranging in [
The seed terms have been obtained with low human supervision. As can be noticed by the way of collecting the seed terms, above, terms have associated Wikipedia pages. The results, so, are sets of Wikipedia pages to be used for learning the classi ers.
Following [15], we generate training instances by automatically labelling each instance of a seed term with its designated semantic class. When we create feature vectors for the classi er, the seeds themselves are hidden and only contextual features are used to represent each training instance. Proceeding in this way the classi er is forced to generalize with limited over tting. 3.3
ML machinery We created a suite of binary contextual classi ers, one for each semantic class. The classi ers are learned using, as in [15], Support Vector Machine (SVM) models using Weka toolkit [16]. Each classi er makes a weighted decision as to whether a term belongs or not to its semantic class.
Examples for learning correspond to the mentions of the seed terms in the corresponding Wikipedia pages. Let x1; x2; : : : ; xn the seed terms for the semantic class t and knowledge source k, i.e. xi 2 Rk. Note that in this work only t the source k = wp is used. For each xi we obtain its Wikipedia page and we extract all the mentions of seed terms occurring in the page. Positive examples correspond to mentions of seed terms corresponding to semantic class t while negative examples correspond to seed terms from other semantic classes. Frequently, a positive example occurs within the text of the page but often many other positive and negative examples occur as well. Features are simply words occurring in the local context of mentions.
The above mentioned procedure applies for regular Wikipedia pages but our mechanism foresee also to use training corpus provided by the organizers. In this case the occurrence of a given tagged term is a positive example for the class that has been tagged but negative for the remaining classes.
For processing the full corpus we use an in-house general purpose sentence segmenter and POS tagger to identify non empty words in each sentence and create feature vectors that represent each constituent in the sentence. For each example, the feature vector captures a context window of n words to its left and right16 without surpassing sentence limits.
For evaluation we used Wikipedia categories - SNOMED CT classes mappings as gold standard. We considered for each semantic class t a gold standard 15 A purity 1 means that all the Wikipedia categories attached to the page are mapped (directly or indirectly) into the same SNOMED CT class, lower values of the purity may mean that the assignment of Wikipedia categories to SNOMED CT classes is not unique or not exists. 16 In the experiments reported here n was set to 3. set including all the Wikipedia pages with purity 1, i.e. those pages unambiguously mapped to t. The accuracy of the corresponding classi er is measured against this gold standard set. 4
Although our aim is applying the previous approach to the current setting there signi cant di erences that have to be faced: 1) The tagset is greater and comes from a di erent source (UMLS instead of
SNOMED CT ). 2) The language is French instead of English. 3) The genre of documents (EMEA and Medline) is very di erent from Wikipedia pages.
So, we performed the following changes over our previous system:
First, the way of collecting seedterms described in section 3.2 is modi ed as follows: We manually mapped the UMLS tagset into the set of SNOMED CT top categories (to the full 19 categories set, not to the 6 most frequent categories as in the previous system) and, further to English Wikipedia categories. We ltered out the English Wikipedia categories lacking French counterpart. For some UMLS categories as LIVB the mapping was not biunivocal, for other cases second level SN categories needed to be considered.
After ltering out categories not containing French interwiki links, for some of the UMLS classes a rather small set of Wikipedia classes remained, so we decided to extend the set by considering the French entity mentions occurring in the training set, we collected in this way 73 additional categories. We selected for each UMLS class the set of mentions tagged with the corresponding tag in the training collection existing as page or category in the French Wikipedia. In the case of pages we obtained the corresponding Wikipedia categories. Once collected a set of candidate French Wikipedia categories, we discarded those not having English counterpart and we manually revised the resulting set for accepting or rejecting each candidate and for assigning it to the correct UMLS tag. Then, for each UMLS tag we iterated over all their English Wikipedia categories for collecting all the pages having a purity 1 and having a French counterpart. In this way we obtained the initial sets of French seed words for each UMLS class.
Further processing, described in section 3.3, is basically the same. The only di erence is that for French we have used for processing documents, in learning and test phases, the Freeling toolbox17 has been used. 5
First, we proceed to collect the seed terms for each semantic class t 18 and each knowledge source k. In our experiments we focussed on the Wikipedia-based ap17 http://nlp.lsi.upc.edu/freeling/ 18 As said previously only 9 proach. The results for obtaining French terms starting from English Wikipedia categories are shown in Table 1 and Table 2.
Table 2 shows the number of French Wikipedia pages (i.e. terms) according their UMLS class with independence of their purity gure. For obtaining such pages we started using English data as shown in Table 1. This Table shows the global gures of the extraction process from the very beginning to the gure which represents the total number of French Wikipedia pages available for training.
Table 1 shows the global gures of the extraction process. In all the cases we found for the method based Wikipedia the number of terms (row 2), the French extension from the training data (row 3), the length of the initial categories set (row 4). Rows 5 and 6 show the number of pages from the English and French Wikipedia. The reason to discard some Wikipedia articles are: i) only pages with a length greater that 100 words are accepted, ii) some pages has been discarded due to di culties in extracting useful plain text (pages consisting mainly of itemized lists, formulas, links, and so) and iii) Only Wikipedia pages with a purity 1 have been selected. In Table 2 we show the number of accepted terms splitted according the semantic class to which they belong. As mentioned above the learning phase has been followed using Wikipedia categories / UMLS classes mappings as golden standard and Wikipedia pages as input documents. For each seed term we obtained its corresponding Wikipedia page and, after cleaning, POS tagging, and sentence segmenting, we extracted all the mentions. For carrying out this linguistic process we used the Freeling suite (see [17] for details). For each mention the vector of features is built and the 9 learned binary classi ers are applied to it19. If none of the classi ers clearly classify the instance as belonging to the corresponding semantic class no answer is returned. If only one of the classi ers classi es positively the instance, the 19 As quoted above the GEOG tag have been extracted using a conventional NERC based on using French DBPEDIA as a gazetteer corresponding UMLS tag is returned. Otherwise a combination step has to be carried out.
For combining the results of the binary classi ers two methods have been implemented: { Best result. As results of binary classi ers are scored, this method simply returns the class of the best scored individual result. It takes into account two threshold values: i) minimum class score and ii) minimum delta to the next better class score. { Meta-classi er. A SVM multiclass classi er is trained using as features the results of the basic binary classi ers together the context data already used in the basic classi ers. The resulting class is returned.
For the experiments presented in this paper, only the rst combination method has been tested. Several number of tests has been done by i) changing the number of WP articles including in training and ii) changing the threshold values mentioned above.
Table 3 depicts the global results as reported by the organization of CLEF2015. Unfortunately the material o cially delivered included some severe issues regarding o set calculation. This is the main reason of the poor results reported. After detecting such issues the organisation of the contest proposed to x the issue and resubmit a run. So we plan, once xed the bug, to incorporate to the paper in the nal release a new table showing our nal results.
The results shown in Table 3 are really poor and far from the results obtained from our previous version performing on English Wikipedia pages, where we obtained accuracies of 87.4 for Wikipedia-based and Snomed-based approaches and 94.3 for DBpedia-based one. They require some explanation.
They can be justi ed from one side with some issues in our program to produce the results in a stand-o format as required by the organization. Table 5 shows very clearly that the terminological density of our Wikipedia corpus is several times lower that the training corpora provided by the organization.
From the other side, such density is obtained by a tagging that embed several terms in a single sentence. An example of this situation is shown in le 4176905.txt. The sentence and terms tagged are shown in Table 6. There is no doubt that the tagging is correct but it is not clear that such concrete sentence contains 9 terms instead of 3 as most term extractors will do. This fact partially explain the low number of strings tagged by our system (see in Table 4 columns TP versus Tagged)
Obviously, the fact that in the current experiments learning is done from Wikipedia and test is performed over very di erent genres of documents, EMEA and Medline, while in our previous system the genre of training and test documents was the same is a drawback. The di erent coverage of French and English Wikipedia and the lower accuracy of Freeling when performing on French texts are important factors, too.
Another minor issue is that text seems to include some kind of segmentation (see for example: l' enfant or d ' activation plaquettaire induite par l ' heparine among many others). The words by themselves are not important but such segmentation may cause errors in the POS tagging stage and this fact may be a real problem.
Nevertheless, Table 4 20 shows the results using only the strings as comparison element (that is, without taking into consideration the o set values). The column "Right selected" refers only to the number of strings correctly selected while the column "Semantically right" refers to the UMLS tag assigned to such strings. Obviously a string may be correctly selected but the class assigned to it may be wrong. The analysis of this table con rms that the results are poor.
Leaving aside GEOG, that is detected using a speci c mechanism, the best performing classes are ANAT and LIVB with a precision greater 50% probably due to the fact that they are the two classes more frequent in our training corpus.
In order to improve the results, we perform some tests using both WP and CLEF in the training stage. The results obtained are shown in Table 7. It shows an improvement in the performance but also shows a problem in the string selection. Examining the results in more details it reveals that if we consider only right selected strings the precision is about 50%. Table 8 gives a more detailed view about the results; it shows for a given true class it indicates which has been the estimated classes. The best result has been obtained for the class DISO that reaches a precision higher than 70%..
We have presented a system that automatically detects and tags medical terms in general medical documents. The tagset used is derived from UMLS taxonomy. The results of the system, as discussed in previous section are poor and far from the obtained in our previous system, performing on English Wikipedia pages. 20 This table shows two results for MEDLINE documents. The rst is the result actually delivered for the contest. In this result a number of documents has been lost. The issue has been corrected and its result is indicated with an '*'. Please note that the gures correspond with those resulted for EMEA documents. 43 15 15 76 2 9 0 3 0 4 4 0 0 1 1 0 1 2 0 0 1 1 0 3 1 0 1 0 5 0 0 0 0 0 0 0 0 0 0 0 Right class DISO PHEN PROC PHYS ANAT LIVB CHEM DEVI OBJC GEOG 238 2 34 14 12 15 15 2 1 0
DISO PHEN PROC PHYS ANAT LIVB CHEM DEVI OBJC GEOG Precision 71.47 An initial error analysis has detected a program issue in the way of computing o -sets of the detected mentions, also the extremely high di erence in the density of mentions in the corpus used for learning (French Wikipedia pages) and for testing (French EMEA and Medline documents) seems to point to a high disagreement between training and test. A third issue is related to limitations in the performance of Freeling, specially in the basic tokenisation task.
The framework developed allows to perform additional experimenting changing several design parameters like the number of terms used for training, context width, features de nition, etc. Some tests will be performed to optimize such parameters.
Several lines of research and a pending work will be followed in the next future (beyond xing the issues reported above).
{ As our results are based on one of the three knowledge sources used in our previous work, an obvious way of possible improvement is the use of the other two resources (SNOMED CT and DBPEDIA) { A combination and/or the specialization of the resources for learning more accurate classi ers. The application of the DBPEDIA based approach, clearly the most productive one, to all the classes merits a deeper investigation. { A careful combination of learning from the learning dataset and from additional material should be experimented { Table 8 shows that three of the classes produced no results at all and another one only detected one term. In these cases the corresponding classi ers have a extremely low accuracy, probably due to few training examples. So acquiring additional examples for these cases could result on improvements. { Moving from semantic tagging of medical entities to semantic tagging of relations between such entities is a highly exciting objective, in the line of recent challenges in the medical domain (and beyond). { Improving the selection of medical entities by using POS pattern learning, adapting our term extractor to the tagging policy of medical entities in Quaero corpus and improving adaptation of Freeling to French medical texts. 8
This work was partially supported by the SKATER project (Spanish Ministerio de Econom a y Competitividad, TIN2012-38584-C06-01 and TIN2012-38584C06-05). 13. Yeganova, L., Kim, W., Comeau, D., Wilbur, W.J.: Finding biomedical categories in medline R . In: Journal of Biomedical Semantics. (2012) 14. Vivaldi, J., Rodr guez, H.: Using Wikipedia for term extraction in the biomedical domain: rst experience. In: Procesamiento del Lenguaje Natural. Volume 45. (2010) 251{254 15. Huang, R., Rilo ., E.: Inducing domain-speci c semantic class taggers from(almost) nothing. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden (2010) 275{285 16. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.: The
WEKA data mining software: An update. In: SIGKDD Explorations. (2009) 17. Padro, L., Stanilovsky, E.: Freeling 3.0: Towards wider multilinguality. In Calzolari, N., Choukri, K., Declerck, T., Dogan, M.U., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S., eds.: Proceedings of the 8th international conference on Language Resources and Evaluation, European Language Resources Association (ELRA) (2012)