In Healthcare domain, several documents are provided in a narrative way, following textual unstructured formats. This is the case of the discharge summaries, which are clinical texts where physicians describe the conditions of the patients with natural language, making the automated processing of such texts hard and challenging. The objective of the tasks of the 2020 CLEF eHealth for Multilingual Information Extraction is to develop solutions to automatically annotate Spanish clinical texts with codes from the International Classification of Diseases, 10th version (ICD-10). In the present paper, we show our approach which is based on Named Entity Recognition (NER) to detect the diagnoses and procedures, and semantic linking against a Knowledge Graph to extract the ICD10 codes. Besides, we exploit text augmentation techniques to generate synthetic input samples and we use BERT pre-trained models and architecture to train the NERs.
Automated Clinical Encoding covers multiple computer-assisted techniques to extract
valuable knowledge within clinical documents written in natural language and
transform such knowledge to structured information. Clinical documents usually include
medical entities that correspond to diagnoses, procedures, symptoms, drugs, etc., but
the use of narrative and informal language is challenging for the automatic processing
of this information. Among Automated Clinical Encoding tasks, a popular one is the
assisted assignment of codes to the clinical documents from standard medical
classifications, such as the International Classification of Diseases (ICD) [
The CLEF eHealth challenge on Multilingual Information Extraction of this year
is focused on this kind of techniques [
In previous years, the challenges worked in the same line of research. For CLEF
eHealth 2017 challenge, participants provided solutions to extract ICD codes (10th
version , ICD-10) in death certificates for English and French [
In these past challenges, the best approaches were mainly based on neural network
architectures. In the best approach of 2017, the authors provided sequence-to-sequence
deep learning models based on Recurrent Neural Networks (RNNs) [
In the literature, a wide range of approaches have been published, since
semanticbased or rule-based solutions to machine learning proposals. Several examples of
semantic approaches are works such as Pakhomov et al. [
To address the CLEF eHealth 2020 challenge, our FLE team has developed a solution focused on Named Entity Recognition (NER) and semantic-based approaches exploiting Knowledge Graphs. Our Knowledge Graph has been enriched with the annotations coming from training and validation sets provided by the organizers of the challenge. Besides, we extended the input datasets for the NER models by creating synthetic samples with text augmentation techniques over the train/validation sets and we also used BERT pre-trained models and architecture for the NER training. 2 2.1
Current CLEF eHealth 2020 task deals with multilingual Information Extraction (IE).
Concretely, this year, the challenge is focused on automatic coding of Spanish clinical
textual documents to the International Classification of Diseases [
The challenge is divided in three sub-tasks: – Task 1: Automatic code assignment of Spanish texts to CIE10-CM, i.e. to specific diagnoses of the standard. – Task 2: Automatic code assignment of Spanish texts to CIE10-PCS, i.e. to specific procedures of the standard. – Task 3: Addition of explainability references for the two aforementioned tasks. It requires a joint automatic code assignment of diagnoses and procedures, including the positions of the key entities that justify such code assignment.
The output has to be a list of ICD-10 codes for each text document. In the first two sub-tasks, this list must be arranged in descending order, based on the relevance of the code to the corresponding document. In the last sub-task, order of relevance is not required but a joint list of codes has to be presented for diagnoses and procedures, specifying the position of related entities in the text documents.
Our team has participated in the three sub-tasks through a multi-task approach. In our proposal, the core and main techniques have been shared and reused to address the three sub-tasks in a unified way. 2.2
For all the sub-tasks, a synthetic corpus of 1000 clinical case studies has been published. The dataset was manually annotated by clinical professionals. In the official source of the challenge it is specified that the dataset comprises 16,504 sentences and 396,988 words, with an average of 396.2 words per clinical case. This corpus is freely accessible3. There are separate directories for train, dev and test datasets. The train set has 500 clinical cases, the dev set has 250, and the test set with gold standard annotations has 250 clinical cases. In addition, organizers shared a background set without annotations of 2,751 clinical cases. Besides the texts of the clinical cases, in the corpus of train and dev set, tab-separated files for each sub-task are included. These files include the annotations associated to each clinical case. In figure 1, an excerpt sample of these files is shown for task 1 (CIE10-CM) and task 3 (Explainability).
The training dataset of the task 3 (Explainability) has 9,211 annotated codes, of
which, 2,392 are unique. Taking into account that CIE10-CM reports a number of
71,486 diagnoses and CIE10-PCS has a number of 87,170 procedures, we have a total
of 158,656 codes [
Aside from these core datasets of the challenge, we also tried the additional Spanish
abstracts provided by the organizers4. These abstracts were a total of 176,294 texts and
they were annotated automatically [
External sources such as PubMed [
Other linguistic resource that we used is the NegEx-MES tool6 to detect entities negated in Spanish texts. This resource was exploited in several of our final versions for post-processing steps.
And, finally, we used the lists of valid codes for Spanish ICD-10 [
The main workflow and steps of our system are depicted in figure 3. We followed an approach based on named entity detection from clinical texts and later Knowledge Graph (KG) entity linking to CIE-10 codes. We also tested an approach based on text classification through simple Convolutional Neural Networks (CNNs). However, performance decreased and the system was less scalable because of imbalance nature and low coverage of datasets, as we explained in previous section. Because of these reasons, we discarded a full Machine Learning approach in our final system, instead, we developed a combination of Machine Learning for Named Entity Recognition (NER) and semantic-based approach for CIE-10 entity linking through Knowledge Graph (KG) construction.
The first steps include the activities of KG Creation and Data Pre-process. For the KG Creation, we take the lists of valid codes in CIE-10 and we implement a Knowledge Graph with that information. This data contains nodes with the CIE-10 code, the description in Spanish, the description in English and hierarchical relations between the CIE-10 nodes. In this implementation we use the framework of Neo4j7. In Data
Pre-process, we perform Terms Population to the KG. These terms come from the samples annotated in the training and validation sets provided by the organizers. Taking the datasets annotated in the task 3, we get the CIE-10 codes and its related term mentions and we create a new array attribute for that CIE-10 node in the KG, adding all the different ways to name a diagnosis or procedure (term mentions). We also perform Dataset Preparation, cleaning the clinical texts of special characters and encoding issues, and solving several errors in the annotated samples regarding wrong labels (e.g. a diagnosis with the type label of procedure or vice versa), and codes that belonged to other standards. In this step, we also adapt the format of input annotated samples and clinical texts to the BIOES format or IOBES8 in order to be used later for training the NER.
In the intermediate step, we proceed to train a Named Entity Recognition (NER) model. We use the pre-trained language model of Multilingual BERT to initialize the neural network. We have performed different trainings depending on variations in the input annotated samples. Below, we point out the different settings (all samples follow the BIOES format): – Baseline: Input annotated samples from train and dev sets provided by organizers. – Baseline + Abstracts: Previous samples + annotated samples from the additional
Spanish abstracts resource. – Baseline + MIMIC-III: Baseline samples + annotated samples from MIMIC-III dataset. – Baseline + Text Augmentation: Baseline samples + annotated samples augmented with Fujitsu’s proprietary technology.
For the final versions, we carried out the setting of ’Baseline + Text Augmentation’
because we achieved the best results as shown in the evaluation section. We trained
two different NER models for each type of label; one for diagnoses recognition, and
the other for procedures recognition. In the neural network, we follow the BERT
architecture. Bidirectional Encoder Representations from Transformers (BERT) [
Next, in the CIE-10 Linking process, the previous models were run over the test
sets to extract all named entities within the clinical texts. In this way, we are able to
obtain the named entity, its label (diagnosis or procedure) and its position in the text.
Once we have detected the named entities, we perform the linking algorithm to extract
the correspondent CIE-10 codes. For this activity, we use Levenshtein string similarity
distance [
In the final step, we develop Data Post-process methods to create the definitive results. For the task 1 and the task 2 we apply frequency-based techniques to sort the results. We assume the most frequent named entities in a text are the most relevant ones. The CIE-10 codes of such named entities would be in the first positions of the list of a clinical text. However, in those cases when the frequency is the same, we apply reordering based on the text position of named entities. We give more relevance 8 https://donovanong.github.io/ner/tagging-scheme-for-ner.html to named entities nearer to the end of the text, where conclusions and the diagnostics usually take place. We assign lower relevance to the entities located at the beginning of the text, where the antecedents are usually exposed. For all these tasks, we created a version of results where we removed the entities negated with NegEx-MES tool. We also created other versions where we analyzed overlapping of named entities in different text positions to normalize the CIE-10 code of the longest named entity. 3
In this section we first describe the environment and the tools that we used. Then, we describe the experiments carried out to select the tools and the methods used. And finally, we present and discuss the CLEF eHealth performance evaluation of our results. 3.1
All the NER models described here are obtained by fine-tuning a BERT-Based
Multilingual cased architecture with 110 million trainable parameters. The weights are available
on GitHub9. It supports 104 languages (Spanish included). We chose this architecture
empirically from the experience we have on training biomedical domain models. We
used the script run ner.py, which is available on GitHub10 to fine-tune our NER
models. This script was also previously used to train BioBERT[
For text augmentation of the NER input datasets, we trained a text generation model
with Fujitsu’s proprietary technology based on decentralized learning. We used a
subset of MIMIC-III [
After the prediction of the NER and the linking process to the CIE-10 codes with the help of the Knowledge Graph, we performed 4 different methods of post-processing of the results. We presented 4 versions of our results, one for each type of post-processing method applied. Table 1 presents the identifiers of each version, and then we describe the methods of post-processing followed.
– No position overlap: After locating the position of each entity found in the text, we remove entities with overlapping positions and we only keep the longest entity. For instance, if we detect the entities “hipertensio´n ocular” and “hipertensio´n”, the second entity is a substring of the first, that is, it is in the same position of the word “hipertensio´n” from the first entity. In this case, we only keep the first entity, which is the longest one.
The performance of each experiment in the building of the NER model is shown in the Table 2.
Data-2 + Augmentation extended Val Data-2 diagnostico Data-2 + Augmentation extended Val Data-2 procedimiento F1-Score 56.82 24.05 57.10 – Baseline: Firstly, we obtain a baseline performance by fine tuning BERT for the entity “diagnostico”. This baseline is trained with the first version of the dataset provided by the organizers (Data-1) and it is tested with the validation set (Val Data1). We achieve an F1-Score of 56.82 with this experiment. Based on this result, we are going to proceed to adjusting training parameters. – Abstracts: Besides training and validation sets, organizers provided extra text annotations from literature abstracts. So, in this experiment, we trained an NER with the initial training set and the data from the abstracts. The performance decreased, so we discarded the abstract data for the following training iterations. We believe the data imbalance and noise introduced to the training by the abstracts are responsible of this low performance. – MIMIC-III: Similar to the previous experiment, we trained an NER with the initial training set plus data from MIMIC-III dataset, taking care not to outnumber the data points of CLEF eHealth data. This experiment resulted in a better performance over the baseline. Therefore, we thought that data augmentation would be a useful tool to achieve higher performance. – Fujitsu Augmentation: We used the Text Generation model trained with Fujitsu’s proprietary Decentralized Learning technology. This Text Generation model was trained with MIMIC-III and PubMed data. This model receives a seed text as input and it generates text similar to the medical domain text. In our experiments, the seed text is the word or the set of words that conform to an entity. By using our Text Generation model, we duplicate the number of samples in the initial train set. And by the randomness of the generated text, we add robustness to the final NER model. In this experiment we achieve an F1-Score of 58.03. – Fujitsu Augmentation fine-tune: In this experiment we fine-tune the Text Generation model with data from CLEF eHealth challenge. This means the Text Generation model learns from MIMIC-III, PubMed and CLEF eHealth data. Similar to the previous experiment, we duplicate the training samples and we obtain an NER with a performance of 58.40. This is the final methodology used to train the NER model in our system.
Using the method of experiment “Fujitsu Augmentation fine-tune” we trained two NER models with the latest released datasets (Data-2) from the task 3 for type “diagnostico” (Final T1) and for type “procedimiento” (Final T2). With this method and the updated dataset we go from 56.82 to 72.45 F1-Score for the entity “diagnostico”. And these NER models are the ones used alongside the linking algorithm with our Knowledge Graph to obtain our results over the test set.
After post-processing the data with the 4 methods described in Section 3.1, we submitted the results for evaluation. Table 3 shows the performance obtained after the evaluation carried out by the organizers of the CLEF eHealth challenge.
From the evaluation tables we see that for task 1, V 1 achieves the highest Mean Average Precision (MAP) and F1-Score. V 3 and V 4 achieved the highest MAP and F1-Score for task 2 of the challenge and V 1 was the highest performance approach for task 3. We associate the lower results in task 2 to issues with bigger specificity regarding body parts and concrete parameters in CIE10-PCS. There are cases where our approach is missing the specifications of the procedure. In those cases, we are retrieving wrong code annotations, what decreases the precision and recall.
We can highlight the versions of post-processing that did not remove negated entities as the highest achieving approaches. We conclude, after analysis over the groundtruth, that negated entities are contemplated as part of the expected results. In the case of task 1, out of the 2841 negated entities that we removed from our results, 2059 appear in the ground-truth. And in case of task 2, the ground-truth contains 61 appearances from the 58 negated entities removed from our results. This is the reason why versions of our results that do not remove negated entities (V 1 and V 3) outperform the versions that remove negated entities (V 2 and V 4) across all the tasks. And due to the small number of negated entities we found in task 2, the performance differences between versions is negligible. 4
In this paper we present the methods used in the CLEF eHealth 2020 Challenge for Automated Clinical Encoding. This challenge consisted of 3 tasks for CIE-10 code assignment of texts written in Spanish for diagnoses (task 1) and procedures (task 2), and identifying the positioning of those entities in the texts (task 3).
We followed an approach composed of a Knowledge Graph created with the CIE-10 standard and the training data annotations. Then we fine-tuned a multilingual BERTbased network for Named Entity Recognition to predict entities in the clinical texts. We used data augmentation through Fujitsu proprietary Text Generation Model, trained with Decentralized Learning from the datasets MIMIC-III and PubMed, to create synthetic samples for the NER training. With the output of our NER models and the Knowledge Graph, we developed a linking algorithm to assign a CIE-10 code to predicted entities in the texts. Finally, we post-processed the output of our linking algorithm to remove negated entities using the NegEx-MES tool. We also analyzed the output to provide 4 different versions of our results taking into account the overlapping words and positions of predicted entities.
Our approach achieves F1-Scores of 0.67, 0.51 and 0.61 for tasks 1, 2 and 3 respectively. The versions of our results that achieve higher performance are the ones that do not remove negated entities, named CodiEspD v1, CodiEspP v3 and CodiEspX v1.