This paper describes E2ENC, the system that we have developed to participate in CANTEMIST (CANcer TExt Mining Shared Task - tumor named entity recognition). E2ENC is a data-driven and end-to-end neural network-based system. It does not rely on external resources such as part-of-speech tagger. It proposes to solve two problems jointly; the first problem is to automatically extract the tumor morphology mentions that can be found in medical documents written in Spanish. The second task is to find the corresponding eCIE-O-3.1 codes for each extracted entity. E2ENC shows promising results, comparing to the baseline system. The reported results show that the proposed system achieve a micro-F1 of 84.9% and 77.7% on the test set for the first and second sub-tasks, respectively, and a MAP of 73.7% on the third sub-task.
Recently, cancer has become one of the first causes of death in the Globe, surpassing cardiovascular diseases. To perform research and improve standards of healthcare and to evaluate cancer treatment outcomes easy—and ideally, in an automated way —access to a variety of data sources is required. The knowledge embedded in unstructured textual documents (e.g., pathology reports, clinical notes), is crucial to achieve all these goals. Hence, there is an imperative need to take advantage of natural language processing (NLP) and text mining technologies to develop Information Extraction (IE) systems that can automatically process unstructured medical resources, i.e., pathology reports and clinical notes, and extract critical information that leads to better clinical decision-making.
There are multiple approaches to building such IE systems. In general, such systems leverage NLP tools, such as tokenizers, part-of-speech taggers and parsers to pre-process the documents. After that, modules, that may be rule-based or machine learning-based, are designed, to solve IE related tasks.
Starting by a seed collection of entities, the idea of rule-based systems is to manually engineer
some rules based on regular expressions, syntactic, or dependency structures to expand the
collection iteratively [
The feature-engineering-based systems aim to train a sequence tagger with rich, hand-crafted
features based on linguistic or syntactic information from annotated corpus to predict a label
(e.g., , − < > or − < > ) on each token in a sentence [
Rule-based and feature-engineering-based approaches are labor-intensive for constructing
rules or features using linguistic and syntactic information. Despite some promising results,
there are two main issues with these approaches. First, the engineering of rules and features
is a time-consuming task. Moreover, rules always need to be updated. Second, the systems of
these two categories are dependent on some external requirements like a parser analyzing the
syntactic and dependency structure of sentences. Therefore, the performances of these systems
rely on the quality of the parsing results [
CANTEMIST (CANcer TExt Mining Shared Task – tumor named entity recognition) [
We participated in the CANTEMIST challenge by developing E2ENC, an end-to-end neural coder system for the three sub-tasks. The proposed system provides an end-to-end solution and does not require any parsers or other linguistic resources. Specifically, the proposed system is a multilayer neural network, where the first seven layers aim to learn high representation for a sequence of tokens, then we pass, jointly, the output of these layers to four Conditional Random Field (CRF) models that are learned jointly. One is for extracting the tumor morphology mentions, while the others is for identifying their ICD-O-3 codes.
The rest of the paper structured as follows: Section 2 presents the Methodology; Section 3 explains the experimental settings and discusses the results; finally, Section 4 concludes this paper.
In this section, we introduce our system, i.e., E2ENC, and its implementation steps in detail. We ifrst describe the problem and the corresponding sub-tasks. Given a clinical document written in Spanish, our goal is to build a system that can i) automatically extract tumor morphology mentions. The output of this step is the start and end positions of the text spans that represent the character corresponding ofsets of the tumor morphology mentions. ii) Normalising the extracted entities by returning all tumor morphology entity mentions together with their corresponding eCIE-O-3.1 codes. iii) Finally, given the list of eCIE-0-3.1 codes extracted from the document, find a ranked list of ICD-O-3 codes (Spanish version: eCIE-O-3.1). Docum ent
Sentence
Sentences
Tokenizer
Tokens
Or der ed Uniqe Codes
Coder
ICD-O-3 Codes
Nor m alizer
Entities
NER Module tags in BIOES tagging schema. That is ∈ {
− , is one of the set of labels in the task .
We propose a deep learning-based system to solve the tasks defined above. Figure 1 depicts the overall architecture of the system. First, we use a sentence splitter to find all the sentences in the document. After that, we tokenize each sentence and find the list of tokens. As soon as we get the tokens, we pass them into the Name-Entity-Recognizer (NER) model to extract the tumor morphology entity mentions. Then, we use a Normalizer (NORM) model to find the code of each extracted tumor morphology. Finally, we take the set of the codes, i.e., removing the duplicates, and use that set as an output for the the third task.
Our system is based on a deep learning-based model that solves the first two tasks, i.e., NER and NORM, jointly. The idea is to regard the problem as a multi-label multi-tags token classification. Given a sequence of tokens token , 1 ≤ ≤ a vector of labels = { 1 , 2 , ..., = { 1, 2, ..., }, the model must produce for each } . Where refers to the sequence length, i.e., the number of tokens and is the number of labelling tasks. The symbol is one of a set of − , − , − ,
In our case, the number of labelling tasks, i.e., , is four. The first task is the NER task and has only one label that is "MORFOLOGIA_NEOPLASIA". The rest three tasks are related to the NORM task. Considering that the format of eCIE-O-3.1 codes which is \d{4,}/\d{1,}[/H] and by splitting the code by the "/" symbol, we decompose the NORM task into three tagging tasks. Each is corresponding to produce a part of the the code.
We designed a deep learning-based model, shown in Figure 2 that is composed by eight consecutive layers to jointly solve the four tagging tasks. The following subsections explain in details the model and the training procedure that we followed to train the model.
First, we start by explaining the Embedding layer in Subsection 2.1. After that, in Subsection 2.2, we describe the structure of the encoder layer. Subsection 2.3 explains the output layer. Finally, in Subsection 2.4, we define the objective function and detail the training routine. En coder
W i t h 6 Layer s Norm#2 Norm#1
Ner LSTM ....
+ LSTM LSTM LSTM LSTM W1
Norm#3 Norm#2 Norm#1
Ner LSTM ....
+ LSTM LSTM LSTM LSTM .... .... .... .... .... .... .... .... Norm#2 Norm#1
Ner LSTM .... + LSTM LSTM LSTM
LSTM
W2 ....
Wn
The goal of the embedding layer is to represent each word ∈ by a low-dimensional
vector space ∈ ℝ . Here, , which is 400d in our case, is the size of the embedding layer. As
shown in Figure 3, we use two levels of embedding: word-level and character-level. For the
word-level embedding, we replace with its pre-trained word embedding vector [
characters of the word pooling to represent the word at character-level as the following. Suppose that is made up of a sequence of characters [ 1, 2, ..., ], where is the length of . First, we pass the sequence of to a randomly initialized character embedding layer to get the matrix obtain a feature map ∈ ℝ − +1. Specifically, the -th element of is given by: ∈ ℝ × - that is the character-level representation of . Here, the −th column corresponds to the character embedding for . After that, we apply a narrow convolution between and a filter (or kernel) ∈ ℝ × of width , after which we add a bias and apply a nonlinearity to
[ ] = ℎ (⟨ [∗, ∶ + − 1], ⟩ + ) (1) inner product. Finally, we take the max-over-time where [∗, ∶ + − 1] is the -to-( +
− 1)-th column of and ⟨, ⟩ is the frobenius = [ ] = [ ; ] the character-level vector. as the feature corresponding to the filter (when applied to word ). A filter is basically picking out a character -gram, where the size of the -gram corresponds to the filter width.
The final representation of the word is given by concatenating the word-level vector and (2) (3) a vector that is regarded as the representation of that word.
The embedding layer is the first layer in xx model. It takes as an input a word and produces
The main objective of the encoder is to transform the sequence of vectors 1, 2, ..., produced
by the embedding layer into a sequence of recurrent states ℎ1, ℎ2, ..., ℎ
representation. The encoder of our system is inspired by the encoder in the translation system
in a high level of
GNMT system from google [
It is composed by six Long Short-Term Memory (LSTM) [
The output layer receives as an input the sequence of states ℎ1, ℎ2, ..., ℎ obtained from the
encoder and send pass them in parallel to four fully-connected layers each is responsible to
solve a single tagging problem as mentioned above. It has been shown that in such tasks, i.e.,
sequence labelling, it is beneficial to consider the correlation between labels in neighbourhoods,
specifically when there are strong dependencies across the output labels. For example, the
tag B is more likely to be followed by the tag I. Thus, instead of modeling tagging decisions
independently, we model them jointly using a Conditional Random Field (CRF) [
Formally, let by the encoder layer, and = {ℎ1, ℎ2, ..., ℎ } be the sequence of vectors to be labeled, which is produced = { 1, 2, ..., } is the corresponding tag sequence for the task . model can be represented as an undirected graph
In this work we employed a linear-chain CRF, where = ( , ) with cliques .
Both
and L Each element of is one of the , − or {1, 2, ..., }, = {(, + 1)}). It has two diferent cliques (i.e. representing the input-output connection, and a pairwise clique ( the score of the ℎ tag of the ℎ word in a sentence and it is computed as follows: output connection. We consider to be the matrix of output scores, where , corresponds to is a simple chain or line:
= ( = = { , }
): a unary clique ( ) ) representing the adjacent size of the hidden state.
In this equation, the parameters are , ∈ ℝ2∗ ℎ and ∈ ℝ1, where ℎ is the dimensionality sequence of predictions as follows:
The clique
is considered to be the matrix of transition scores such that , represents the score of a transition from the tag to the tag . Given that, we define the score function of the to the set of possible tags. sequence as follows:
In this expression 0
and +1 denote the start and the end tags of the sentence, that we add A softmax over all possible tag sequences ( ∗) on a sequence yields a probability for the ( , ) = ∑ , + ∑ , +1 In this expression | | means the number of tasks.
= 1 | | ∈ ∑ During training, we minimize the negative log-probability of the correct tag sequence: = − ( ( | )) = − ( , ) +
∑ ( ̃ ∈ ∗ ( , ̃ ) ) given by:
During inference, we search for the output sequence ̂ that obtains the highest probability ̂ = arg max ( ̃ | )
̃ ∈ ∗
In this model, Eq. 7 and Eq. 8 can be solved eficiently using dynamic programming.
by Eq. 7, we define the joint objective function as the following: Lets referes to one of the tagging tasks and the objective function of that task computed
The derivative of the objective function , Eq. 9, is taken through back-propagation with
respect to the whole set of parameters of the model, which are the transition matrix , the
parameters of the BiGRU model and the parameters of the matrix defined in Eq. 4. The
parameters are optimized using Adam optimizer with a learning rate of 0.0001. To reduce the
efects of gradient exploding, we set the clipping threshold of the gradient to 5. We apply a
dropout [
In this section, we discuss the dataset used and diferent experimental settings devised to evaluate our system.
We trained and fine-tuned our system respectively on the training and the development sets provided by the organizers of the CANTEMIST challenge. The statistical description of the datasets is shown in Table 1. After that, we submitted the predicted labels of the test set that are produced by our system to evaluate its performance. The organizers omitted the golden labels of the test for the tree tasks. We used the train split to train the model, the Dev1 split to ifne-tune the system and find the best parameters and Dev2 for testing the system performance locally.
The oficial evaluation metrics used to validate the performance of the system are: Precision, Recall and F1-score for Cantemist-NORM and Cantemist-NER. For the Catemist-CODING Mean Average Precision (mAP) was used.
Table 2 shows the results of our submitted system on the Test and the Dev2 sets. It also shows the results of the baseline system on the Test set, which is simply a dictionary lookup based on Levenshtein distance. It looks for train and development annotations in the test set. From the reported results, we can note that in general, our system outperforms the baseline system and gives comparable performance on both the Dev2 and the Test sets. The maximum diference in the performance of the system on the Dev2 test and the Test set is recorded for the Recall metric (with 1% in case of the NER task and 0.3% in case of the NORM task). In terms of the CODING task, the diference is 1.3%. Another remarkable observation is that our system gives a similar performance in all the evaluation metrics, which shows its consistency. It is also clearly shown that the performance of the system in the NORM task afects the outcome in the CODING task (as we only take the distinct codes as the final ranking result for the CODING task). To overcome this issue, we plan to develop a ranking model that takes as an input the extracted entities and the set of coding as a candidate set of codes and produce a ranked set of codes. We leave this extension for the future work.
In this paper, we describe in detail the implementation of End2End Neural Coder (E2ENC)
system, including all the techniques that are crucial to its accuracy, and robustness. The system
is used to solve the tasks defined in CANTEMIST (CANcer TExt Mining Shared Task – tumor
named entity recognition) shared tasks. E2ENC contains four sub-models that were trained
jointly. The first one aims to automatically extract tumor morphology mentions in health reports
written in Spanish. The three sub-models are used to perform the normalization task and return
all tumor morphology entity mentions with their corresponding eCIE-O-3.1 codes. E2ENC
provides an end-to-end solution and does not require any external tools or other linguistic
resources. The efectiveness of the proposed system has been evaluated by participating in the
CANTEMIST shared task. The reported results show that the proposed system is stable and
consistent. In our future work, we plan to perform extensive error analysis and inspect the
performance of the system and improve it. For example, we plan to use a transformer-based
interpretable model like BERT [