In this work, we describe a Deep Learning architecture for Named Entity Recognition (NER) in biomedical texts. The architecture has two bidirectional Long Short-Term Memory (LSTM) layers and a last layer based on Conditional Random Field (CRF). Our system obtained the rst place in the subtask A (identi cation) of TASS-2018-Task 3 eHealth Knowledge Discovery, with an F1 of 87.2%.
Currently, the number of biomedical literature is growing at an exponential rate. The substantial number of research works makes it extremely di cult for researchers to keep up with the new development in their research areas. Therefore, the e ective management of a large amount of information and the accuracy of knowledge is a vital task.
Named Entities Recognition (NER) is one of the fundamental tasks of biomedical text mining, with the aim of identifying pieces of text that refer to speci c entities of interest.
There are di erent scopes to address the
NER problem. Among them, we can nd
methods based on dictionaries, which are
limited by the size of the dictionary, spelling
errors, the use of synonyms and the constant
growth of vocabulary. Rule-based methods
and Machine Learning methods usually
require both syntactic and semantic features
as well as characteristics of the language of
the speci c domain. One of the most e
ective method is Conditional Random Fields
(CRF)
In this paper, we propose a hybrid
model combining two bidirectional Long
Short Memory (Bi-LSTM) layers with a CRF
layer. To do this, we adapt the NeuroNER
model proposed in
Copyright © 2018 by the paper's authors. Copying permitted for private and academic purposes.
of Wikipedia and ii) a sense-disambiguation
embedding model
The rest of the paper is organized as follows. In Section 2, we describe the architecture of our system. Section 3 presents the results. In Section 4, we provide the conclusions. 2 2.1
All texts were preprocessed in four steps.
First, sentences were split by using Spacy
(Space.io, 2018), an open source library for
advanced natural language processing with
support for 26 languages. Second, sentences
and their annotated entities were
transformed to the BRAT format1, a stando
format similar to BioNLP Shared Task
stando format. Then, sentences were tokenized.
Finally, each token in a sentence was
annotated using the BMEWO-V extended tag
encoding, to capture information about the
sequence of tokens in a given sentence. The
BIOES label scheme introduced in the work
of
In our work, we propose as input of our network two di erent embeddings: word embeddings and sense-disambiguation embeddings. Below we describe them in more detail.
Word embedding is an approach to
represent words as vectors of real numbers. There
are di erent methods to obtain these
vectors such as probabilistic models and neural
1http://brat.nlplab.org/stando .html
networks. In last years, neural networks for
training word embedding models have gained
a lot of popularity among NLP community
because they are able to capture syntactic
and semantic information among words. The
most popular methods are word2vec
In this work, we used the Spanish
Billion Words
Algorithm: Skip-gram Bag of Words
We also used the sense2vec
Reddit Vector is a pre-trained model of
sense-disambiguation representation vectors
presented by
Corpus size: approximately 2 billions words Vocab size: 1 million Array size: 128 Algorithm: Sense2Vec
Although the word embeddings are able to
capture syntactic and semantic information,
other linguistic information such as
morphological information, orthographic
transcription or PoS tags are not exploited.
According to
The output of the rst layer is concatenated with the word embeddings and with the sense-disambiguation embeddings of the tokens in a given input sentence. This concatenation of features is the input for the second Bi-LSTM layer. The goal of this layer is to obtain a sequence of probabilities corresponding to each label of the BMEWO-V encoding format. In this way, for each input token, this layer returns six probabilities (one for each tag in BMEWO-V). The nal tag should be that with highest probability.
The parameters of the sets and the hyper parameters of the models are the following: Words Embedding Dimension: 300 Characters Embedding Dimension: 25 Hidden Layers Dimension: 100 (for each LSTM: for the forward and backward layers) Learning method: SGD, learning ratio: 0.005 Dropout: 0.5
Epochs: 100
To improve the accuracy of predictions, we also used a CRF model trained, which takes as input the output of the previous layer and obtains the most probable sequence of predicted labels. 2.4
Once tokens have been annotated with their corresponding labels in the BMEWO-V encoding format, the entity mentions must be transformed to the BRAT format. V tags, which identify nested or overlapping entities, are generated as new annotations within the scope of other mentions. 3 3.1
The evaluation of the proposed model was carried out using the annotated corpus proposed in the TASS-2018Task 3 eHealth Knowledge Discovery (https://github.com/tass18-task3/data).
The training set is made up of 5 documents with 3276 entities annotations. The development set consists of 1 text document with 1958 entities annotations. The test set consists of 1 text document (see Table 2). There are two types of of entities: concepts and actions. For this reason, tokens can be annotated with di erent labels (see Table 1) following the BMEWO-V encoding format.
Concept B/M/E/W/V-Concept Action B/M/E/W/V-Action Others O
In our experiments, we used precision, recall and F1 score to evaluate the performance of our system. The TASS-2018-Task 3 considers two di erent criteria: the partial matching (a tagged entity name is correct only if there is some overlap between it and a gold entity name) and exact matching (a tagged entity name is correct only if its boundary exactly match with a gold entity name). A detailed description of evaluation is in the web (http://www.sepln.org/workshops/ tass/2018/task-3/evaluation.html). Moreover, we used evaluation script (https://github.com/TASS18-Task3/ data/blob/master/score_training.py) provided by the shared task organizers to evaluate our system. 3.2
As it was described above, our system is based on network with two Bi-LSTM layers and a last layer for CRF. In the rst BiLSTM layer, we consider the character embeddings. In the second layer, we concatenate the output of the rst layer with word embeddings and sense-disambiguate embeddings. Finally, the last layer uses a CRF to obtain the most suitable labels for each token.
Table 3 compares the results obtained using the NeuroNER system with our extended version using pre-trained embeddings models and the BMEWO-V encoding format. Our extended version of NeuroNER achieves a signi cant improvement of the results (more than 7.2% in F1).
NeuroNER 0.824 ext. Neu- 0.862 roNER
In the substask A (identi cation of key phrases), our system obtained the top micro F1 (0.872) (see Table4). It signi cantly outperform the rest of participating systems. We will wait to review the proposed systems in greater depth in order to establish comparisons and possible improvements to our implementation.
plubeda upf-upc VSP Marcelo P R F 0.862 0.882 0.872 0.77 0.86 0.31 0.11 0.81 0.75 0.32 0.32 0.79 0.80 0.32 0.17 4
Named Entity Recognition (NER) is a crucial tool in text mining tasks. In this work, we propose a hybrid Bi-LSTM and CRF model adding sense-disambiguation embedding and an extended tag encoding format to detect discontinuous entities, as well as overlapping or nested entities. Our system is able to achieve satisfactory performance without requiring speci cally domain knowledge or hand-crafted features. It is also important to highlight the language independence, which is key to multi-language tasks. Our results demonstrated that the extended BMEWO-V encoding improves the result of the predictions. Moreover, the pre-trained models help to reduce training time and increase the accuracy of labeling, achieving the highest F1 for the subtask A.
We plan to try with other embeddings models such as the FastText model, which contains morphological information. Moreover, we will extend the encoding format to capture distinct types of overlapping or nested entities.
This work was supported by the Research Program of the Ministry of Economy and Competitiveness - Government of Spain (project DeepEMR: Clinical information extraction using deep learning and big data techniques-TIN2017-87548-C2-1-R)