Introduction

The International Classification of Functioning, Disability and Health (ICF) is in charge of providing standards for describing health and health-related states. The ICF defines disability as a term comprising itself several terms such as impairments, activity limitations and participation restrictions. So disabilities although related to diseases are not synonyms of the latter ones. In ICF words disability and functioning are viewed as outcomes of interactions between health conditions (diseases, disorders and injuries) and contextual factors.

To our knowledge, DIANN-IberEval 2018 [2] represents the first attempt towards automatic tagging of disabilities in Spanish and English texts. Turning to the literature, there has been some work devoted to Medical Entity Recognition (MER) in both Spanish and English. In Spanish, [11] pursued a MER task where the goal consisted in automatically identifying disorders and drug mentions on a corpus conformed of clinical reports. The authors employed supervised machine learning algorithms (SVM, Perceptron and CRF) in conjunctions with multiple features (POS, embedding clusters, Brown clusters among others). The results showed that the combination of supervised algorithms and no-supervised features significantly improved standard supervised techniques.

[9] present a hybrid system based on both the use of gazetteers (more precisely SnomedCT) and embeddings learned in an unsupervised manner. Basically, for each paragraph (P) in a clinical report they modeled through a function they called DNER(t,P), the degree in which the term t is named in the phrase P, where the term t corresponds every noun-phrase obtained from a SnomedCT description. DNER(t,P) uses both n-gram distance and similarity between the embeddings of the terms from the document and the distance of the "word vector" with the SnomedCT description term. They did not report any results what so ever.

It is also worth mentioning two other well known task related to the actual DIANN-IberEval task, namely SemEval-2014 task 7 [12] and SemEVal-2015 Task 14 [13]. In both competitions, there were two subtasks. One consisted in performing the identification of diseases and certain other medical terms that the human taggers considered to be relevant for some reason. The other was a normalization task where each identified term had to get a unique UMLS/SNOMED-CT ontology CUI (Concept Unique Identifier) assigned. In SemEval tasks, disorders could be either continuous (lower extremity DVT ) or discontinuous spans (tumor ... ovary). Both DIANN-IberEVal and SemEval task evaluations are similar. In both partial and exact recognition are measured in order to compare the performance of the different systems.

UTH-CCB team was the winner of the identification task in SemEval-2014 with a 0,81 and a 0,90 F-score on exact and partial evaluations respectively. They used two machine learning algorithms based on CRF and SVM in addition to MetaMap. They employed several features such as Bag Of Words, POS (from Stanford tagger), type of notes, EHR section information, word representations (Brown clustering), random indexing and semantic categories (UMLS lookup, MetaMap and cTAKES). They also applied three types of ensemble on these basic algorithms: a machine learning ensemble, a majority vote and a direct merging of all.

In SemEval-2015 the winner was the ezDi team with a 0.75 and 0.78 F-score on exact and partial evaluations respectively. SemEval-2015 evaluation is not directly comparable with either SemEval-2014 nor DIANN-IberEval because it did not only consist in correctly identifying the term. Correct CUI assignment was included in the evaluations as well, and therefore results are lower than those of the previous year. Exact identification comprised perfect CUI assignment and whole entity identification. Partial match comprised perfect CUI assignment and partial entity identification. EzDi team used both CRF for simple entity recognition with the following features: BOW (window +/− 2),word stemmer, prefix-suffix length 1-5, orthographic features (word contains digit, slash, special char, word shape-kind of reg. exp.), grammatical features ( POS, chunk, head of NP/VP), dictionary look-up matches (window +/− 2), section header, document type and sentence cluster id. And an SVM for discontinuous entities with syntactic features like POS, chunk labels between candidates, rules based on the Charniak parser to find relations, position of prepositions, conjunctions, main verb in context of first candidate, a binary feature indicating which candidates contain NP head and finally lexical features like Bag of Words.

Lately, Jagannatha et al. [4,5] used different Recurrent Neural Networks to pursue MER. They obtained their embeddings applying Word2Vec's skipgram algorithm (SkipG henceforth) over PubMed articles, English Wikipedia and 100,000 EHRs. The results show that all RNN models significantly outperform the baseline while their best system improved the recall, precision and F-score of the baseline by 19%, 2% and 11% respectively.

There are also several well known ruled-based or mapping tools for Named Entity Recognition. MedLEE: The Medical Language Extraction and Encoding System (MedLEE) is a rule-based tool. MetaMap: MetaMap maps any texts to the UMLS Metathesaurus. Recognizing terms contained in UMLS and certain variations of those terms. To finish, cTAKES is an information extraction system from clinical narratives. It is an open source project to perform clinical information extraction task, mapping terms to UMLS concepts, and accomplish syntactic and semantic parsing.

System Description

We have divided this section in two parts. In 2.1 we are going to explain the external data we used, while in 2.2 we will focus on the pipeline that combines neural networks and rules.

External Data

We made use of external data with the intention of completing the information the system extracts from the corpus provided by organization. We employed Brown clusters [1] and word-embeddings [10]. For English, we extracted Brown clusters and word-embeddings from MIMIC-III corpus [6]. For Spanish, we calculated the word-embeddings from Electronic Health Records and we did not include any Brown cluster.

Hybrid Disability Detection Pipeline

In order to identify disabilities, negated disabilities and the negation scope we designed an hybrid system that combines neural networks and rules. First of all we present our module to identify disabilities, negated or not, using neural network based architecture. Then, we focus on the rule-based module to identify negation triggers. On the top of that, we also included a rule-based module that helps identifying those disabilities the neural network can not identify. Finally we explain how to detect negation scopes with a neural network.

Neural Network-Based Disability Identification We employed neural network based architecture, more precisely an specific Bi-LSTM (a RNN subclass, [3]) with a CRF on top of it [7,8] using as input raw text, word-embeddings and Brown clusters. This kind of neural network is widely used to pursue sequence to sequence tagging [8,5]. One of the advantages of using Bi-LSTM in contrast to other machine learning techniques such as SVM, Perceptron or CRFs is that the size of the context is automatically learned by the LSTM and there is no need to perform any complicated text preprocessing to obtain features to feed the tool.

Rule-Based Negation Trigger Identification

In order to identify negated disabilities we created a list of negation triggers for each language and this module labels them as negation if they are close (maximum one word distance) to disabilities identified by the neural network. The list of negation triggers for English is lack of, without, with no, no, not, no signs and no evidence of, and for Spanish is sin, ausencia, no, falta de, no hay evidencia and sin evidencia.

Rule-Based Disability Identification

This module is responsible of detecting the acronyms of disabilities that are close to the disabilities (maximum one word distance) identified by the neural network. Once the acronyms are detected the module labels them as disabilities in the entire text.

Neural Network-Based Negation Scope Identification The main objective of this last module is to identify the negation scope. Although the used neural network is the same we use to identify disabilities, this time we use the output of the previous modules as features instead of using Brown clusters.

Results and Discussion

Three tracks have been evaluated in the present shared task: Disabilities included or not in negation, non-negated disabilities and negated disabilities, and disabilities, negation triggers and the scope of negations. In addition, two types of matching have been used for the evaluation: partial and exact. We show the results obtained by all the systems for English and Spanish in table 1.

If we analyze the results, the good performance of the system in almost all the tracks is notorious both in exact match evaluation as in partial match evaluation. These results have helped us to beat the results of the rest of the systems in the shared task in almost all the tracks. However, there has been a track where the systems results have fallen short, specifically in Negated Disability track for English (exact match). For the evaluation of this track, it is considered as negated disability the set of annotations disability, negation trigger and scope of the negation. With the intention of clarifying the reasons for these low results we have carried out an error analysis and we have concluded that there are three main reasons for these results: -The tendency of the system to include the previous verb to the negation trigger in the negation scope:

• showed no cognitive impairment.

• had no cognitive deterioration. -The tendency of the system to finish the negation scope with the last annotated disability:

• No cognitive deterioration was found.

• no cognitive impairment according to Reisberg's global deterioration scale (GDS). -The difficulty of catching all the disabilities and negation triggers. In that way, without correctly annotated disabilities or negation triggers is really difficult for the neural network to catch negation scopes and is not possible to perform well in exact match evaluation.

Taking that into account we realize there are some aspects of the system that can be improved. Nevertheless, we consider the performance of the system proposed by us has been really good and the suitability of it for this task is undeniable.

Conclusions

This paper presents a hybrid pipeline combining neural networks and rules for disability identification in clinical texts that provide the best results among the systems presented in the shared task.

A key aspect in order to achieve this performance is the complementarity of neural network-based modules and rule-based modules. To conclude, although it is clear the most important modules are neural network-based ones, those that make the difference are the rule-based modules.