This paper describes the participation of master's students (LITL programme, university of Toulouse) and their teachers to the CLEF eHealth 2016 campaign. Two runs were submitted for task 2 (multilingual information extraction) which consisted in the recognition and categorization of medical entities in French biomedical documents. The system used consists of a CRF classier based on a number of dierent features (POS tagging, generic word lists and syntactic parsing). In addition, several patterns were used on the CRF's output in order to extract more complex entities. The best run achieved high precision (0.640.78) but lower recall (0.320.40), with an overall F1-measure of 0.430.53.
This article describes the participation of the students of the LITL master to
the CLEF eHealth 2016 Lab [
LITL (stands for Linguistique, Informatique, Technologies du Langage, i.e.
Linguistics, IT, Language technologies ) is a new master’s program at the
University of Toulouse, France. Mainly aimed at linguistics and humanities students, it
comprises, for a major part, courses in natural language processing (NLP),
computational linguistics and practical aspects of corpus analysis through
programming and using various computer tools. An important part of this curriculum is
project-oriented, and the rst year students have to build a fully operational
processing system for a precise NLP task. This year’s project was the participation
to the CLEF eHealth challenge, more precisely task 2: multilingual Information
Extraction [
The teachers in charge of this project (members of the CLLE-ERSS laboratory) deemed that this task was ideal for pedagogical purposes: information extraction (and more precisely named entity recognition NER) is a well-known, well-dened and central task in modern NLP; state of the art information extraction systems are based on machine learning techniques but can also make use of symbolic handcrafted rule-based approaches; supervised learning systems such as CRF classiers can take advantage of dierent kind of linguistic resources, many of these are available for the biomedical domain; a collaborative task is an excellent exercise for students, as it motivates them and gives them a clear feedback on their work; the target language of CLEF eHEalth 2016 is French, the students’ working language; the task’s schedule was perfectly suited to the master’s calendar.
Working as a team along the entire semester, the students were thus able (with help from their teachers) to submit two runs for the selected task, and got very satisfactory results for a rst attempt.
This paper is organized as follows. Section 2 describes the tasks and get a closer look at the data. Section 3 gives a precise description of the dierent components of the system designed for the task, while the results are given and discussed in Section 4. 2 2.1
The LITL team participation only concerns the CLEF 2016 eHealth task 2 phase 1 addressing biomedical NER in French texts. The aim is to automatically detect and classify biomedical entities. Detection requires the system to provide start and end positions of each relevant entities. The classication step consists in associating each previously detected entity with one of the ten target categories corresponding to UMLS Semantic Group:
Chemical and Drugs ( CHEM), Devices (DEVI), Disorders (DISO), Geographic Areas ( GEOG), Living Beings (LIVB), Objects (OBJC), Phenomena (PHEN), Physiology (PHYS), Procedures (PROC).
Two text types are concerned with this task: drug inserts and biomedical research papers’ titles and (sub)headings. Biomedical entities cover a wide range of linguistic expressions such as drug or city names, very common nouns (e.g. "inrmiŁre " (nurse ), "Øtudes " (studies )), technical specialized terms (e.g. " thyroglobuline ", "Ølectronystagmographie ") or complex phrases (e.g. " infectØs par le virus de l’immunodØcience humaine " (infected with human immunodeciency virus ), "enfant gØ de plus de trois mois " (children over 3 months )). 2.2 The training data set corresponds to the QUAERO French Medical Corpus [11] previously used in CLEF 2015 eHealth task 1b [12] and made of two sub-corpora: EMEA: 6 drug inserts written by the European Medicines Agency 1 for the general public ; MEDLINE: 1665 titles and (sub)headings of biomedical academic papers indexed in the MEDLINE database 2.
Drug inserts are long texts (around 5,000 words/text on average) written in a less specialized language than MEDLINE’s very short text segments (6 words/item on average), as illustrated in examples 1 from EMEA and 2 from MEDLINE, see below. MEDLINE texts are written with an uncommon syntax, considering that titles and (sub)headings usually correspond to noun phrases or non-nite sentences without nal punctuation.
Example 1. Il est utilisØ en association avec d ’ autres mØdicaments antiviraux dans le traitement des adultes et des enfants infectØs par le virus de l ’ immunodØcience humaine ( VIH ), le virus qui provoque le syndrome d ’ immunodØcience acquise ( SIDA ). [EMEA/118_1] Example 2. HypersensibilitØ retardØe dans les aections thyrodiennes ØtudiØe par le test de migration des leucocytes en prØsence de thyroglobuline humaine . [MEDLINE/4573749] Both sub-corpora have been manually annotated following similar guidelines [11]. Each annotation is encoded in the BRAT stando annotation format 3 and includes for each entity an id, the relevant UMLS category, the oset position (start and end) and the textual content of the annotation. Example 3 gives the manual annotations associated with example 2 and Figure 1 its visualization via the BRAT annotation tool.
T2 DISO 0 16 HypersensibilitØ T3 DISO 35 59 affections thyrodiennes T4 DISO 35 45 affections T5 ANAT 46 59 thyrodiennes T6 PROC 75 107 test de migration des leucocytes T7 PHYS 83 107 migration des leucocytes T8 PHYS 83 92 migration T9 ANAT 97 107 leucocytes T10 CHEM 123 137 thyroglobuline
T11 LIVB 138 145 humaine
As for the detection subtask, annotated entities can be single words (77%) or compounds (23%). Compound entities may be autonomous (e.g. " prise en charge"); nested such as in (3) where " leucocytes " is nested in "migration des leucocytes " which is in turn nested in " test de migration des leucocytes "; or discontinuous such as " hØpatites C " and "virus des hØpatites C " in (4).
Because discontinuous entities are very rare, we decided to not detect such entities. Concerning nested entities which are more frequent, we implemented post-processing as described in Section 3.
As for classication subtask, entities are heterogeneously distributed among ten types. Half of the annotated entities are categorized as DISO (26%) or CHEM (25%). The rest spread mainly among PROC (16%), ANAT (11%) and LIVB (11%) while types PHYS, OBJC, DEVI, GEOG and PHEN are fairly rare. When compound, entities seem to be usually categorized as the rst (head) component (e.g. " virus des hØpatites C " is labeled LIVB as for "virus ").
Preliminary linguistic observations of data reveal three main characteristics that will constitute features for our system. First, the length of single-word annotated entities (in characters) seems longer than non-annotated words with an average of 9.5 characters against 5.5. Secondly, single-word entities show some morphological complexity with a use of recurrent productive axes such as cyte, thØrapie , endocrino , trachØo . Finally, annotated entities may be mainly characterized as technical vocabulary in the biomedical domain especially for main types (DISO and CHEM). 3
In this section we present in details the system we designed for this task. Its
cornerstone is a Conditional Random Field (CRF) classier [
As is mandatory with machine learning supervised techniques, we rst had to build a model based on the training data. This was done according to the scheme presented in gure 2. As can be seen, this sequence starts with the raw text les from the training dataset, corrects their tokenization, performs POS tagging and syntactic parsing, then adds an additional set of features and nally provides, along with the annotated les, input for the CRF training phase. The result is a CRF model that can be used in the extraction phase.
The overall process used for extracting and categorizing entities from raw text les can be seen in gure 3. We can see that it applies roughly the same modules as for the training phase, but adds an additional post-processing step after the CRF classier.
The stages are described in details in the following sections. 3.2
The rst set of modules concerns the processing of text les. The main role of these modules is to provide a generic annotation layer for the tokens: POS tagging, lemmatization and dependency parsing, as these additional information are to be used by the CRF classier.
For both phases, we used the Talismane toolkit [13] for POS tagging and dependency parsing, with the provided models for French and using the default
parameters. In addition to being a reliable tagger and dependency parser, Talismane provides the oset for each token, which is vital for dealing with the BRAT stando annotation format.
The tokenization correction module has been developed in order to prevent many tagging errors due to the non-standard blank spaces that appear in both the training and test text les. The main problems were due to additional spaces before and/or after commas, hyphens, apostrophes, periods, digits, etc. For example, an extraneous space between " l " and the following apostrophe prevented the tagger and parser from identifying a determiner. This correction module is a simple Perl script using regular expressions to convert texts back to their supposedly original punctuation, while keeping the additional spaces in order to preserve the osets for the annotated and extracted entities. For example, extract in (5) becomes (6) after correction: Example 5. Par consØquent , lorsque vous serez traitØ par TYSABRI , vous ne constaterez peut-Œtre pas d ’ amØlioration mais le traitement par TYSABRI pourra empŒcher l ’ aggravation de votre maladie . [EMEA/425_6] Example 6. Par consØquent, lorsque vous serez traitØ par TYSABRI, vous ne constaterez peut-Œtre pas d’ amØlioration mais le traitement par TYSABRI pourra empŒcher l’ aggravation de votre maladie.
The output of the parser follows a column-based format, in which each token is described on a separate line, as in the example (7) below (related to (5)), where columns indicate, in order: word form, POS tag, lemma, syntactic dependency, oset.
As can be seen, unknown tokens (i.e. absent from the reference lexicon used by Talismane) have no corresponding lemma. We added correction procedure that reproduces the word form instead of the "_" placeholder. The POS tags indicate the token’s nature (e.g. DETerminer, Verb, ADVerb, etc.) while the syntactic tag indicate its function in the sentence, more precisely the relation it has with its syntactic governor, such as object, modier, etc. Details on both tagsets can be found in the Talismane documentation. 3.3
In addition to the lemma, POS and syntactic relations, we added several features we deemed relevant with the task (see 2). These features are the following: does the token appear in a specic word lists? what is the token’s length in number of characters? does the token begin with a recognizable prex? does the token end with a recognizable sux?
These features and the way they were calculated are described in the following paragraphs
We used several word lists in order to extend the lexical coverage beyond what is present in the training datasets. We used two main sources: SNOMED and a list of commercial drug names.
SNOMED [14] is a well-known and used resource for biomedical NLP, as it contains extensive word lists. We selected the 4 most relevant top categories, and extracted all the single-word terms in each one.
As noted above (Section 2), manual observation of the training data
(especially the EMEA subset) revealed that a large number of commercial names
of drug were present and systematically tagged as CHEM entities. Given that
SNOMED only contains scientic names of chemicals, we wanted to use an
additional resource to tag these specic tokens, as performed by several named
entities recognition systems [
We used the Vidal website 5, which is the reference compendium for pharmaceutical drugs in France. After downloading the list of raw drug names it provides, we developed an adhoc Perl script that identies the main name through removal of extraneous information such as posology indications (i.e. transforming " ABSTRAL 100 g cp subling" into "Abstral ").
Table 2 gives the number of terms used and a few examples for each of the selected 5 categories.
In the end, each token is then associated with a tag (column) containing the SNOMED/VIDAL category ( chem, morph, proc or phys) if its lemma appears in the corresponding word list, or none in other cases.
Token length
As noted above from an observation of the training data, word length can be a good indicator for biomedical terms. Given that the CRF classier cannot deal with numerical feature, we used a qualitative measure of the length using the following scale: short < 6 medium < 10 long .
Axes
Two additional features based on axes were designed in order to help technical specialized vocabulary detection. It is well known that biomedical technical
terms are extensively coined through the use of standard suxes and prexes.
Identifying these can be a useful addition to the necessarily limited reference
word lists for entity recognition [
Lexique des racines, prØxes et suxes des termes scientiques et mØdicaux by D. Pol, found at http://www.didier-pol.net/ Lexique des axes (prØxes et suxes) by A. Abbara, found at http:// www.aly-abbara.com/litterature/medicale/affixes/a.html Terminologie mØdicale : prØxes et suxes by P. Cauwel, found at https:// sites.google.com/site/cauwelphilippe/Home/terminologie-medicale-prefixes-et-suffixes
From BRAT to BIO: selection of entities from the training set Viewing the task of entity recognition and categorization as a tagging task requires the data (both training and testing) to be transformed into a compatible scheme. If the BRAT stando annotation format is perfect for the visualization and declaration of these entities, it is not appropriate for an automated tagging process.
The BIO format is traditionally used for expressing token sequences such as
chunks or named entities (cf. [
Thus dened, the task for the CRF classier is to tag each token with one of the 21 possible value (there are 10 target entity categories, so 10 X_B and 10 X_I, but only one O). This task is thus formally similar to other NLP tagging tasks such as POS tagging, where a target value has to be decided according to the descriptive features of the token (and, in the case of CRF, to the features and values of the neighboring tokens).
An alternative scheme is known as BILOU and adds the possibility for a
token to be the Last token or to be the Unique one for this entity (cf. [
However, none of these schemes is able to faithfully represent nested, overlapping or discontinuous entities. As noted above, in section 2 (example 3), "HypersensibilitØ retardØe " contains two entities: " HypersensibilitØ " and "HypersensibilitØ retardØe ". Both are of the DISO category, and both begins with the word "HypersensibilitØ ". However, only one of them can be represented using the BIO format. The same goes for more complex situations like the sequence "test de migration des leucocytes ", still in example 3.
A decision had to be made when translating the annotated entities into the BIO format: do we keep the longest entities or the shortest? More precisely, do we consider "leucocytes " to be an ANAT entity by itself (thus receiving an ANAT_B tag) or part of the PHYS entity " migration des leucocytes " (with an PHYS_I tag)? This decision impacts the whole process, as what is decided for the training phase of the CRF will of course directly inuence what is produced in the tagging phase.
This dilemma could be summarized as a choice between fewer but more complex entities versus more small ones. We decided to choose the latter, as we deemed it easier to "rebuild" complex entities from smaller ones than the other way around.
At this stage, each text is represented by a set of features available for each token, along with a BIO tag for the training data, as is the case in example (8) derived from (7):
As noted above, we chose to use a CRF classier (CRF++) in order to predict the BIO tag for each token given the features described in the previous sections. Training the CRF requires to dene a template, i.e. a selection of which pieces of information are given as an input to the model. The specicity of CRF systems is that they can take into account both the features associated to the target token and the ones of its neighbors. Thus, a template is used to dene which feature(s) of which neighbor(s) are used.
In summary, after a number of trial runs using the training data, we opted for the following: word form of target token and of the tokens appearing in a size-2 window around it; POS tag of target token and of the tokens appearing in a size-2 window around it; length, prex, sux, and presence in word lists of the target token; BIO tag attributed to the previous token.
Moreover, the second run submitted uses the syntactic dependency tag of the target token, in addition to the previous features. This decision followed a rst experiment to evaluate the contribution of each dierent features. The system evaluation was conducted by using the original train/devel split provided for the training data. Table 3 gives some results of a lesion studies, i.e. indicating for each feature how their removal from the template aects the task system’s performances. F-scores were calculated by using the BRATeval.jar script provided by the organizers (exact matching option on true). As table 3 shows, syntactic feature contribution depends on the syntactic characteristics of texts: when texts have uncommon syntax as for MEDLINE items (see 2), using the syntactic information may worsen results. All other features have a positive contribution to the model for both target subsets, and thus has been kept in the system.
We considered separating the training subsets (EMEA and MEDLINE) and creating dierent models, but experiments in this direction were inconclusive, so we decided to use all the training data as a whole and to use the same model for both test subsets.
From BIO to BRAT: identifying complex named entity NE The nal stage of the tagging phase is the translation from the BIO format back to the target BRAT stando annotation format. Beyond simple format conversion, the main problem here is the reconstruction of entities, assuming the choices made in the training phase and addressing the limitations of the BIO format.
This stage is performed in two steps. The rst one is a simple identication of entities, blindly following the BIO format. It means that, for each token tagged as XXX_B we extract one (and only one) entity, comprising the initial tokens along with all following tokens tagged as YYY_I. For example, we extract " hyperplasie focale modulaire " as a single DISO entity from the following output:
focale (ADJ) DISO_I modulaire (ADJ) DISO_I
In case of inconsistency between the categories (i.e. if XXX is dierent from YYY), we attribute the initial ( XXX) category to the NEs.
As explained above, the choices made initially imply that at this stage we only get non-nested entities, although most of the time they are parts of larger entities. For example, given the following sequence in the CRF output: Example 10. tuberculose (NC) DISO_B
mØdiastinale (ADJ) ANAT_B the rst step extracts two separate entities: " tuberculose " as a DISO and "mØdiastinale " as a ANAT. In order to extract some of the more complex entities, we designed a few extraction patterns, based on the POS tags. The rst pattern looks for a noun tagged as B, followed by an adjective tagged as B, i.e. the exact situation found in the example 10. Its output is the concatenation of both tokens, tagged with the category of the head noun. This means that this patterns produces "tuberculose mØdiastinale " as a DISO.
Similar patterns have been designed for longer sequences of adjectives: NC (B) ADJ (B) ADJ (B or I) , and for nominal compounds such as NC (B) de/du/des/au/aux NC (B) and NC (B) P DET NC (B) .
All extracted entities are collected in the BRAT stando annotation format as noted above. The osets are immediately available thanks to Talismane output format and the content of the entity is a simple concatenation of their tokens’ word forms. 4
In this last section we present and discuss the results obtained by our system on the task’s test data. As indicated in Table 4, our F1 scores are above the average (of submitted runs) for entity recognition in EMEA and slightly under it for MEDLINE with, for all subsets, a high precision and a low recall. The non-exact matching scoring procedure means that the osets of the entities can partially overlap instead of being perfectly aligned. Run1 (without syntactic feature) and run2 (with syntactic feature) scores show little dierences as expected, although run 1 gets higher scores for both subsets with exact matching. The order is partly dierent without exact matching where run2 gest slightly higher results for EMEA only. It should be noted that matching option only concerns entity delimitation and not disagreement on entity classication.
A closer look at the scores per category shows that we get the highest performance on EMEA texts for LIVB entities recognition with a 0.86 F1 (0.93 P and 0.8 R, 268 entities), followed by PROC (0.7 F1, 269 entities) and CHEM (0.67 F1, 885 entities). The detection of DISO entities, which cover the largest part of biomedical entities (988 entities in MEDLINE and 342 in EMEA), is better in MEDLINE texts (0.66 F1) than in EMEA texts (0.57 F1).
To better understand our system weaknesses, we propose a quick observation of the test dataset, then an overview of the false negatives (FN) and false positives (FP). First of all, an overview of test data reveals a dierence from the training data that may explain some of our FN. Indeed, as indicated in table 5, while the proportion of compounds is approximatively the same than in training data set, the proportion of discontinuous entities which were not taken into account by our system is double.
If we take a closer look at FP and FN, we may distinguish several cases (all subsequent examples come from the EMEA subset). First, FPs may be due to a disagreement on entity category such as the CHEM polydextrose or diacØtate de glycØrol which our system both classied as DISO. A comparison between run1 and the gold standard shows 187 cases of such miscategorization (60 in EMEA and 127 in MEDLINE), with quite a lot of confusions about subtle distinctions such as between OBJC physical objects and DEVI devices or between PROC procedures and CHEM chemical and drugs. As for example, in (11), " injection " was automatically categorized PROC instead of CHEM and "seringue " (syringe ) was automatically categorized OBJC instead of DEVI.
Example 11. L ’ injection sous-cutanØe est rØalisØe de la mŒme faon qu ’ avec une seringue classique .[EMEA/334_3] The subcutaneous injection must be conducted in the same way as with a classical syringe Classication errors may also be linked with polysemy as for example with the word "bouton " which means either a concrete object ( button ) or an anatomic/physiological entity (pimple ).
A second group of FPs may be linked to the well-known problem of distinguishing between specialized vs. common usages. Such cases occur when a common word or compound was incorrectly recognized as biomedical entities (e.g. "anomalies " recognized as DISO or "MØlange de couleur bleu " as CHEM).
A last possible explanation for FPs is related to compounds boundaries as for example when a CHEM is syntactically linked to a LIVB but not manually annotated as a compound (e.g. " olanzapine chez les enfants " or "olanzapine sur les protØines "). Same mistakes may be observed due to prepositional phrase attachment ambiguity as for example in (12) Example 12. Traitement de l’ ankylostomiase par le tØtrachlorØthylŁne chez l’ adulte et le grand enfant . [MEDLINE/13515790] where our system recognized " ankylostomiase par le tØtrachlorØthylŁne " as a DISO because of the pattern NC (B) P DET NC (B) whereas th prepositional phrase "par le tØtrachlorØthylŁne " is syntactically attached to " Traitement " and not to "ankylostomiase ".
Moreover, when exact matching, other FPs may be observed such as " troubles hØmatologiques pØriphØriques " detected as a single entity instead of two entities "troubles hØmatologiques " and "pØriphØriques ".
This compound boundaries problem has very strong consequences on recall and may explain the large amount of FNs (more than 1,300 in EMEA and 2,100 in MEDLINE, see Table 4). Indeed, when " troubles hØmatologiques pØriphØriques " is detected instead of two entities " troubles hØmatologiques " and "pØriphØriques ", it causes one FP and two FNs. A simple way for dealing with this problem will be to systematically split detected compounds into as many single entities as single words.
The converse is also observed i.e. when our system extracts only one term from a compound, as for example " Olanzapine Teva " manually annotated as CHEM while our system recognized only " Olanzapine " and not "Teva ". Such FNs were essentially due to the limited coverage of our external resources. 11. NØvØol, A., Grouin, C., Leixa, J., Rosset, S., Zweigenbaum, P.: The Quaero French medical corpus: A ressource for medical entity recognition and normalization. In: Proc BioTextM, Reykjavik (2014) 12. NØvØol, A., Grouin, C., Tannier, X., Hamon, T., Kelly, L., Goeuriot, L., Zweigenbaum, P.: Clef ehealth evaluation lab 2015 task 1b: clinical named entity recognition. In: Proceedings of CLEF (2015) 13. Urieli, A., Tanguy, L.: L’apport du faisceau dans l’analyse syntaxique en dØpendances par transitions : Øtudes de cas avec l’analyseur Talismane. In: Actes de la 20e confØrence sur le Traitement Automatique des Langues Naturelles (TALN’2013). pp. 188201. Les Sables d’Olonne, France (2013) 14. Wang, A.Y., Sable, J.H., Spackman, K.A.: The SNOMED clinical terms development process: renement and analysis of content. In: Proceedings of the AMIA Symposium. p. 845. American Medical Informatics Association (2002)