This paper describes our participation in DIANN 2018 Task: DIsability ANNotation in English and Spanish documents. Our proposal detects disabilities as well as recognizes negated disabilities. To that end, our entity typing system is applied without tuning and it does not require any external knowledge. It consists of a Random Forest machine learning classi er whose feature set includes local entity information and pro les, generated unsupervisedly. Two experiments are presented in order to investigate performance of two types of pro les. Both proposals are able to reach promising and reasonable results, obtaining a partial-matching precision greater than 87% for disabilities and negated disabilities regardless of the language. Thus demonstrating the portability and adequateness of our approach regardless of type of pro le.
Disability is de ned as \any condition of the body or mind (impairment) that
makes it more di cult for the person with the condition to do certain activities
(activity limitation) and interact with the world around them (participation
restrictions)" [
There are some e orts to annotate medical concepts for languages such as
English [
The request was not only to annotate disabilities (dis) but also to annotate the negation (neg) modi ers a ecting at least one disability as well as its scope (scp), as illustrated by Example 1. 2
were h e l d f o r 41 r e s i d e n t s o f our c e n t e r ' s N u r s i n g Home who were s e l e c t e d a s t h e y p r e s e n t e d <scp><neg>no</neg> p o t e n t i a l l y f a t a l d i s e a s e o r <d i s >c o g n i t i v e impairment </d i s ></scp >.
For the rst edition of the DIANN task we were particularly interested in
evaluating CARMEN [12{14], our general purpose Named Entity Typing (NET)
system. Such NET system decides whether a possible chunk in a text corresponds
or not to a disability. It does not use any external resource (e.g. UMLS
Metathesaurus [
The rest of the paper is structured as follows. Next, our approach is de ned in Section 2. The experiments are presented in Section 3. Results are discussed in Section 4. Last, conclusions and future work are outlined in Section 5. 2
Our approach, as can be seen in Figure 1, consists in ve main steps: (i) preprocessing: this stage takes as input an annotated corpus to perform a linguistic analysis (see Section 2.1); (ii) disability annotation: this process implies the extraction of possible candidates in order to decide which ones should be typed as disabilities (see Section 2.2); (iii) negation annotation: this task nds all negation triggers in a given text (see Section 2.3); (iv) scope annotation: this phase determines which negation triggers, previously detected, a ect a disability (i.e. must be kept) and its span (see Section 2.4); and last, (v) post-processing: this stage converts the resulting corpus to the competition format (see Section 2.5). In the following sections this work- ow is explained in detail.
Input Corpus 1. Pre-processing
Linguistically
Analyzed Corpus
Candidate extraction Output corpus
5. Post processing 4. Scope annotation 2. Disability annotation
Candidate
Typing 3.Negation annotation
The system takes as input an annotated document following DIANN format. The
DIANN format is plain text that can include XML tags to determine the anchor
of the three elements that must be annotated (disabilities, negation and scope),
as can be seen in Example 1. Such format is converted to a well-formed XML that
can be processed by Gate [
Disability annotation consists of two modules:
The rst one, candidate extraction in Figure 1, identi es possible candidates from text. This module extracts noun phrases detected in shallow parsing to be considered as the set of candidates for being a disability.
The second component, candidate typing in Figure 1, establishes which of
the previous noun phrases should be nally typed as disabilities (i.e. binary
classi cation). For this purpose we applied CARMEN, our general purpose NET
system [12{14]. CARMEN is a machine learning based system which employs
Random Forest (RF) algorithm [
For each candidate, CARMEN generates a feature vector that includes context and local information of the entity.Table 1 contains all the features generated for the sentence in Example 1, which are further explained next.
Context of the entity is built through pro les [
In brief, pro les are generated as follows: For each identi ed candidate, rst
we extract lemmas of nouns, verbs, adjectives and adverbs, in a window of size W
( W2 = 5 words after and before the candidate) and their frequency as the number
of occurrences. Second, for each entity type, the training corpus is divided in
positive instances (e.g. disability) and negative instances (e.g. no disability). Third,
each lemma found only accompanying positive instances computes a relevance
index based on the term frequency, disjoint corpora frequency [
As previously stated, the feature vector of CARMEN also contains local
information of the entity, such features are inspired by state-of-the-art NET
modules. These comprise words of the entity, length of the entity, su xes and
pre xes [
Feature Description Example
Relevance of lemmas of nouns, verbs, relevance(disease)= txpro le adjectives and adverbs that appear in RC(disease)=1.43, te a window anchored in the candidate. relevance(potentially)= on Relevance is obtain according to TFDCF(potentially)=5.1,
itaonNElen ESuntitxyesleanngdthprweitxheosutwsitthopa-wleonrgdtsh of 2c, co, cog, cogn, froma xes 1a,nd2,l3asatnwdo4rdcsharacters from the rst ment, ent, nt, t
L co, og, gn, ni, it, ti, iv, ve, e , i, im, mp, pa, ai, ir, rm, me, en, nt, t , cog, ...
Last, it should be noted that adapting CARMEN to this new scenario was straightforward, since there was no need to change the NET architecture or its parameters. It only required: (i) a linguistic analyzer that is able to deal with the two languages tackled in DIANN task (i.e. English and Spanish) in order to perform sentence detection, tokenization, lemmatization, PoS-tagging and shallow parsing; and (ii) the DIANN training corpus, which was previously annotated with the target entity. 2.3
Negation is tackled using a dictionary-based approach, thus having two phases:
{ An o ine step whose main goal is to build a lexicon of negation triggers. For
each language, a lexicon is created directly from DIANN training corpus.
{ A real-time processing step whose purpose is to annotate all negation triggers
of a text. Each lexicon is used to instance a Hash Gazetteer, included within
ANNIE plugin from GATE [
2.4
This stage is carried out as a set of heuristics at sentence-level. In order to determine which negation triggers a ect disabilities, the applied rule is: for each sentence, all negation triggers that do not co-occur with at least one disability are removed. To be able to de ne the scope of negated disabilities, scope is established as the anchor of negation trigger and disabilities. For instance, in Example 1, the scope starts at the negation position (\no") and ends with the disability (\cognitive impairment"). 2.5
At this point, the results are stored in XML GATE stando format. As a result,
they need to be converted to DIANN format again. To that end, all documents
are transformed to an inline XML format without a root tag using GATE
Embedded [
As mentioned before, our interest is focused on evaluating CARMEN, our entity typing system. Its feature set includes pro les of each entity type among other features. Pro les are composed of pairs lemma-relevance, but relevance is computed according to two di erent indexes (TFDCF and RC), as explained in Section 2.2. Therefore, two experiments were submitted aiming at studying the di erences of using both relevance indexes or only one, namely: R1 uses the full pro le (both TFDCF and RC relevance indexes) in the feature vector of CARMEN.
R2 uses a reduced pro le that only takes into account TFDCF relevance index in the feature vector of CARMEN. 4
Initially, the organization provided an annotated training set and an unannotated test for both languages. Next subsections present results for entity typing alone during training phase (Section 4.1) and o cial test results (Section 4.2) for our complete approach. Finally, the o cial results are analyzed (Section 4.3). 4.1
Since no development set was provided, CARMEN was evaluated using 2 fold strati ed cross-validation over the annotated training set. The purpose is to assess the entity typing task alone, assuming our candidate extraction module is \perfect" and all possible disabilities are included in the set of candidates. Hence, Table 2 summarizes results for being a disability reported by Weka. 6
For each language and each run, values of Area Under the ROC Curve (AUC),
Precision, Recall and F-score are reported. AUC is commonly used in biomedical
informatics research to measure the performance of a classi er, thus allowing the
comparison of several models under the same test [
The two rst rows show the results of the two experiments for English. Similarly, the last two rows show the results of both experiments for Spanish
According to AUC, in the case of Spanish, it's better to use the full pro le
(R1 - 98.7) whereas there is no such di erence for English (i.e. AUC is the
same for the two experiments). Several scales for interpreting these AUC values
exists, but there is a consensus that values greater than .96 indicate an excellent
discriminatory ability [
The o cial results reported by DIANN task organizers over the test set can be found in Table 3. For each language and each run, values of Precision, Recall and F-score are reported for di erent types of disabilities. Two types of matching are used for the evaluation: partial and exact. First, performance for all disabilities, regardless being negated or not, are shown (type DIS). The rst four rows show the results of the two experiments (R1 and R2) for English and Spanish, respectively. Similarly, the next four rows refer to negated disabilities (type NEGDIS). Finally, the last four rows concern non-negated disabilities as well as negated disabilities (type DIS + NEGDIS).
From Table 3, we can see that partial matching always bene ts our results regardless the type of disability (DIS, NEGDIS or DIS+NEGDIS). Besides, English always gets the highest results. Concerning all disabilities (DIS), the best Precision, Recall and F-score is achieved for partial matching. As in the training phase, it should be noted that our Spanish system is more accurate using the reduced pro le (see precision in Table 3), whereas English requires the complete pro le. However, di erences between training (see recall and F1 in Table 2) and test results suggest a problem in determining the boundaries of disabilities.
Regarding negated disabilities (NEGDIS), there is an striking di erence between English and Spanish performance and English is the highest by far. For both, the use of the complete pro le (R1) seems more appropriate.
Last, concerning non-negated and negated disabilities (DIS+NEGDIS), our rst experiment (R1) achieves the best results for English. Although Spanish obtains the best Precision with the complete pro le, the highest Recall and F-score is accomplished for the second experiment (R2).
In general terms, our proposal performed reasonably well, particularly given that CARMEN system was applied without ne tuning parameters or features. Besides, no additional external knowledge has been used to nd disabilities in this narrow domain. 4.3
As previously stated, GPLSI team obtains high precision values, specially identifying disabilities regardless being negated or not, but recall values are a bit lower. These results are reasonably good, especially considering that (i) CARMEN system was applied o -the-shelf; and (ii) precise system are desired in a medical environment. In order to nd reasons for the low recall, once the annotated test set was released, a 5% of the test set was examined carefully to nd possible improvements to be implemented as future work.
Analyzing the results, regardless of the language, we found that most errors are related to the boundary of disabilities and negated disabilities. This is because the extraction module often includes extra tokens or misses some of them. Another problem found are disabilities represented by acronyms. Although the acronym de nition usually appears in text, CARMEN is not able to classify it as 8
Isabel Moreno, M.T. Roma-Ferri, and Paloma Moreda a disability. Another source of errors is concerned with detecting certain tokens as a disability in a document but not detecting them in another, so e ects on recall are evident. This might be explained by two reasons. On the one hand, more local information and context may be needed in order to build a more robust representation of a disability. Examples of features to characterize local information of an entity mention could be its lemma, as well as its POS and shallow parsing tags. Additionally, such features could be also incorporated for our pro le (i.e. lemmas in a window) jointly with word-embedding or brown clusters to enhance context. On the other hand, no additional knowledge has been use to determine disabilities, but it could avoid losing disabilities and has a direct e ect on recall. Hence, as future work, we plan to implement new features to capture both.
The last source of errors concerns the negation and scope detection modules. On the one hand, our heuristics applied at sentence-level are too optimistic. Although a negation trigger appears in a sentence, it does not necessary a ects all tokens in a sentence. This could be improved considering heuristics at a lower granularity, e.g. clause-level. On the other hand, the negation lexicon does not contain all possible triggers, thus some disabilities are not considered negated in testing. This could be improved gathering negation triggers from other corpora. Both issues are particularly problematic for Spanish due to its high exibility and variance in comparison with English.
Finally, there were a few annotation issues which, in our opinion, could a ect participant systems: { Wrong tokenization in disability annotation: For example, \<dis>severe mental illness</dis>es" instead of \<dis>severe mental illnesses</dis>"; { Inconsistent disability annotation: Texts are the same for both languages, but the same disabilities are not present for both versions. For example, \nonagenarians with recent onset of <dis>functional impairment</dis> also bene t from rehabilitation in a medium-stay geriatric unit;[...]" but \los pacientes nonagenarios con incapacidad reciente tambien se bene cian del tratamiento en una UME, [...]". 5
In this paper our proposal to detect both disabilities and negated disabilities is presented. On the one hand, negation and its scope is tacked with a set of simple heuristics and dictionaries. On the other hand, disabilities are extracted using our entity typing system, CARMEN, for this new task. It employs Random Forest, a supervised machine learning algorithm. Its feature set is based on pro les (context of the surrounding words) and information gathered from the NE itself. In this manner, the actual performance of CARMEN is studied when applied to a new genre and a new entity. Two experiments are presented in order to investigate performance of two types of pro les.
Our training phase results for the entity typing task alone (AUC > .96 and F1 almost 84%) show all runs have an excellent discriminatory ability regardless of GPLSIUA Team at the DIAAN 2018 task 9 the language and pro le type. Regarding o cial testing results, our recall values are a bit lower but partial-matching precision is greater than 87% for disabilities and negated disabilities for all languages and pro le types. These results show our approach performs reasonably well when dealing with disabilities, specially considering the lack of external resources or parameter tuning.
Although the results are encouraging, there is still room for improvement. To that end, an analysis of the obtained output has been carried out to explain our results. Concerning the recall of the testing phase, it was found that (i) boundary detection needs to be more accurate and (ii) representation of entities may be enhanced with either more features or external knowledge. Thus, our participation in the DIANN task has given us an excellent opportunity to study which aspects should be considered to achieve a more versatile CARMEN.
This research is partially funded by the Spanish Government under the projects RESCATA (reference number TIN2015-65100-R) and REDES (reference number TIN2015-65136-C02-2-R).