In this paper we describe our participation to the DIANN 2018 shared task at IberEval 2018 for the annotation of disabilities in the medical domain. We use IXA pipes to model the annotation of disabilities as a sequence labelling task. Our system consists of a combination of clustering features implemented on top of a simple set of shallow local features. We show how leveraging distributional features obtained from large in-domain unlabelled data helps to easily develop a robust system for the medical domain. The system and models generated in this work are available for public use and to facilitate reproducibility of results.
(1) In the patients <scp><neg>without<neg> <dis>dementia</dis></scp>, significant differences were obtained in terms of functional and cognitive status (Barthel index of 52.3438 and Pfeiffer test with an average score of 1.48 3.2 (P<.001)). In Example (1) it can be seen that, apart from disabilities such as ‘dementia’, both negations and their scope is also annotated.
Our submission is placed in the context of the TUNER project1 on “Multi-faceted Domain Adaptation for Advanced Textual Semantic Processing”. In Natural Language Processing (NLP) technology is crucial to extract accurate, complete, relevant, interoperable and timely structured knowledge from large amounts of unstructured multilingual text to make informed decisions. The aim of TUNER is to address these needs through the research and development of domain adaptation techniques to apply them to the NLP technology developed within the project. Summarizing, the project aims to develop domain-oriented cross-lingual systems that will provide deep semantic capabilities to process multilingual data.
In this setting, we take an existing sequence labelling system developed for Named
Entity Recognition (NER) on newswire text [
The DIANN task is modelled as a sequence labelling task. In order to do so, we convert
an annotated sentence such as the one in Example (1) into the BIO scheme for learning
sequence labelling models [
As it can be seen in Example (2), we do not treat the scope of negation, but in a post-processing step, we proceed as follows: if at least a negation and a disability are annotated in a sentence, then the negation is considered to have a wide scope. In other words, if more than one disability appears in the same sentence as a negation, then it is interpreted that the negation affects to every disability present in that sentence.
For our experiments we trained on the training data provide for each language and
choose the best settings via 5-fold cross validation. The chosen models were then used
to annotated the test data. The datasets were tokenized using the IXA pipes tokenizer
[
Our sequence labeling system is ixa-pipe-nerc, which aims to establish a simple and shallow feature set, avoiding any linguistic motivated features, with the objective of removing any reliance on costly extra gold annotations (POS tags, lemmas, syntax, semantics) and/or cascading errors if automatic language processors are used. The underlying motivation was to obtain robust models to facilitate the development of NER systems for other languages and datasets/domains while obtaining state of the art results.
The system consists of: (i) Local, shallow features based mostly on orthographic,
word shape and n-gram features plus their context; (ii) three types of simple clustering
features, based on unigram matching; (iii) publicly available gazetteers. ixa-pipe-nerc
learns supervised models via the Perceptron algorithm as described by [
The ixa-pipe-nerc tagger includes a simple method to combine and stack various
types of clustering features induced over different data sources or corpora, with state of
the art results in newswire Named Entity Recognition [
We train ixa-pipe-nerc with the default parameters and features as described in Agerri
and Rigau [
The shared task provides partial and exact matching results. For example, for a disability such as “severe cognitive impairment”, exact match would be “severe cognitive impairment” whereas a partial matching may be matching one of the terms included in the sequence, such as “cognitive impairment”. However, as the released evaluator4 only provides exact results, we will limit our discussion to the exact evaluation. The metrics used to evaluate the systems will be precision, recall and their harmonic mean F-measure. For each system, the organizers provided three types of results: 1. DIS: Identifying the sequences in the text denoting disabilities. 2. ALL: The annotation of negated disabilities, namely, the set of annotations fdisability, negation trigger and scope of the negationg. 3. JOINT: The joint annotation of disabilities and negation. This evaluation is correct when both negation and disability are correct.
Best results for every setting are obtained using the BILOU encoding scheme5. As we expected, using clustering features substantially improves the results, especially in
4 https://github.com/diannibereval2018/evaluation 5 The BIO scheme suggests to learn models that identify the Beginning, the Inside and the Outside of sequences. The BILOU scheme proposes to learn models Beginning, the Inside and the Last tokens of multi-token chunks as well as Unit-length chunks. DIS
DIS
terms of recall. However, unlike previous works using the same system [
Together with the official results, the task organizers also released the Gold standard test set, which allowed us to perform an extra experiment using exactly the same models trained for the official runs and showed in Tables 1 and 2. We tokenized and segmented the test set and fed it to ixa-pipe-nerc for testing. Given that ixa-pipe-nerc works at token level, we suspected that our official runs were hindered by the fact that the Raw test set was not tokenized and we had to annotate the Raw test set without proper segmentation and tokenization.
DIS
ALL
JOINT Features (run 2) Local (run 1) Local + CPUB300
Precision Recall F1 Precision Recall F1 Precision Recall F1 82.0 63.7 71.7 68.7 47.8 56.4 79.2 59.6 68.1 85.5 65.8 74.4 68.7 47.8 56.4 82.9 61.7 70.7
IxaMed 78.6 86.0 82.1 47.6 43.5 45.5 74.6 81.1 77.7 Table 3. DIANN 2018 Exact English Results on tokenized test data. CPUB300 (run 1): Clark PubMed 300 classes. Local (run 2).
Tables 3 and 4 show the results of applying the models used for the official runs to the tokenized and segmented test set.
As it can be seen, the improvements are substantial, to the point that our system now obtains the second best results across languages and evaluations, although still lower DIS
ALL
JOINT
Features
(run 2) Local
(run 1) Local + BBE2000
(run 3) Local + BBEC2000
than the results obtained on most evaluations by IxaMed [
In this paper we have presented some experiments showing how to quickly and easily adapt a general purpose sequence labeller to the medical domain. The tagger uses shallow local features enriched with semi-supervised features based on distributional semantics. The result is a robust, language independent tagger suitable to be easily applied across languages and domains. The official results obtained, while fairly competitive themselves, are substantially improved by providing the sequence labeller with a tokenized and segmented input. If we compare our results with IxaMed, the best system in most of the evaluation settings, we obtain better results in terms of precision but recall needs to be further improved. We believe that this could be achieved by training clusters using more closely related data sources to the DIANN datasets, instead of using readily available data, as we have done for these experiments. The models and tools are freely available under Apache License 2.07.
This work has been supported by Spanish Ministry of Economy and Competitiveness (MINECO/FEDER, UE), under the projects TUNER (TIN2015-65308-C5-1-R) and CROSSTEXT (TIN2015-72646-EXP).