Introduction

This paper presents a simple approach to the Disability detection shared task DIANN proposed in the framework of IBEREVAL 2018 [2].

The task consists of identifying disabilities in biomedical research articles. The documents are abstracts or short descriptions, typically a few hundred words long, and use standard grammar and orthographic conventions. The goal is to detect where a disability is described or attributed to a patient. Thus, disability mentions that are negated or discarded in the text should be marked as "negated". The task requires the participation on Spanish, and English is optional.

We approach the task in two sequential stages: Disability recognition and negation detection. The former is addressed with a classical NER approach: A CRF [3] performing IOB annotation. The later is solved using an out-of-the-box rule-based system: NegEx [1], which has been adapted for Spanish.

Disability recognition approach

The training data format is short files with XML tags indicating disabilities and negation expressions. Our approach was to transform the input into a list of words and add to them the part of speech (PoS) and IOB information. The result elements were tuples of (word, P OS, IOB − tag). Only the disabilities were considered when building the IOB information. The negation expressions were not used because they were predicted by another module which does not require IOB annotations.

The PoS tagging was done using NLTK [4]. The IOB-tagging was performed manually using the entities inside the <dis> ... </dis> XML tags.

Basic Model

The basic model is a Conditional Random Field (CRF) applied to the IOBtagged dataset. The implementation was built using NLTK's basic CRFTagger which is a module used for POS tagging that uses CRFSuite2 .

The model uses a predefined entities list extracted from the training set which contains an entity per line. It also uses an acronyms list which is built filtering all the single-word entities with all letters in uppercase.

Features

The features used are grouped by similarity in order to ease the evaluation of their utility. The following list describes them:

1. word, pos, lemma, all-caps, strange-cap, contains-dash, contains-dot current word, its lemma, POS, whether all the word's letter are uppercase, whether the word contains uppercase letters while the first is lowercase, and if it contains a dash or dot.

inside-entities, is-acronym

boolean indicating whether the word is found in the predefined entities list, and in the acronyms list.

position-X, total-position-X

position-X is true if the word is found at the position X inside an entity of the predefined entities list; total-position-X is the number of entities in the list in which the word appears. 4. prev-X-word, prev-X-pos, prev-X-lemma the word appearing X positions to the left of the current word, its POS, and its lemma. 5. next-X-word, next-X-pos, next-X-inside-ente the word appearing X positions to the right of the current word, its POS, and if it is inside the entities list. 6. next1-word, next1-pos concatenation of the current and next words, and their part of speech. 7. prev1-word, prev1-pos, prev1-lemma concatenation of the previous and current words, their part of speech, and their lemma. 8. next2-word, next2-pos concatenation of the two next words, and their part of speech.

9. prev2-word, prev2-pos concatenation of the two previous words, and the concatenation of their part of speech.

The number of words used in the features next-X-and prev-X-are tunable parameters. In the final execution, the number of preceding words considered was three and, for the next words, the number was two.

Training

The final features were chosen using 10-fold cross-validation. For each fold, the model builds the entities and acronyms list (using only the nine training chunks) and trains the model to predict the remaining validation chunk. After averaging all the folds F1-score results, the features corresponding to the best average (for the two languages) were used to train a model using all the training dataset.

For each group of features described in the previous section, the whole group was deactivated to check if they affected the precision.

Initially, the groups 4-9 contained all the features of groups 1-3 applied to their elements (i.e. the features applied to the prev-X-word, or concatenating them in the case of next1-feature. Experiments were performed deactivating one group at a time and checking the impact on performance. This allowed us to remove non-useful feature groups, leaving only those groups with actual contribution to the task. Once the useful feature groups were chosen, a more fine-grained inspection was carried to remove useless features inside each group, resulting in the final feature groups reported above.

Negation detection approach

The negations were predicted using an out-of-the-box NegEx implementation3 . After tagging the entities, each sentence and the entity it contains are passed to NegEx which marks if the entity is negated and which is the set of words negating it (if no entity is present the sentence is not fed to NegEx). We detected that almost all the correct negations were close to the entity so the negation expressions that were more than three words away of the entity were discarded.

Experiments and Results

The experiments performed were to predict the whole training dataset using ten-fold cross-validation. The best model was then used to annotate the test set. Table 1 shows the results of some experiments varying the used feature set. Using all the features gives the best results. All results are computed using the evaluation tool provided by DIANN organizers 4 . Table 2 reports the final scores obtained on the official test set. The fields evaluated here are: disability, refers to all disabilities annotation both included or not in a negation; negated disability, considers all the negation-related annotations (disability, negation trigger, and scope); and non-negated disability + negated disability which evaluates jointly the annotation of disabilities and negation (negated disability are considered correct if both negation and disability are correct). For all categories, both partial and exact results are provided. for most metrics. We consider that the presented approach has improvement margin, since the used features are a basic set, could be extended with more advanced semantic information such as word embeddings.