This working notes paper presents our contribution to the CLEF eHealth 2020 Task 1. Our team has participated in the CodiEsp-D subtask, the rst shared task consisted in the automatic clinical coding of medical cases in Spanish, annotated with ICD-10-CM codes. We tackled the task as a multi-label classi cation problem using BERT model [4]. With the aim of leveraging all the language modeling capacities of the deep bidirectional encoder architecture of BERT, we developed a tailored approach to annotate short fragments of text extracted from the long clinical cases present in the CodiEsp corpus and use them as input to the model. Two publicly available Spanish versions of BERT, namely BETO [3] and BERT-SciELO [1], were ne-tuned on the CodiEsp-D corpus extended by a set of abstracts annotated with ICD-10 codes, following our fragment-based classi cation approach. BERT-SciELO, a BERT-Base model pre-trained from scratch on an unlabeled corpus of biomedical articles in Spanish, achieved the best results among our three submitted systems, obtaining a nal Mean Average Precision (MAP) metric score of 0.482 on the evaluation set.
The increasingly adoption of electronic health records (EHRs) as a key
component in many hospital information systems across the globe has posed a series
of questions to the scienti c community that remain partially unresolved. One
of the main issues is how to e ectively leverage the information stored in the
system to improve patient care. EHRs store heterogeneous data in a wide variety
of formats, including free-text documents like clinical notes [
Automatic clinical coding is the task of transforming unstructured clinical
text into a structured format, following standard coding terminologies and using
computational methods [
Historically, natural language processing (NLP) techniques have been applied
to the problem of clinical coding [
This year, CLEF eHealth 2020 Task 1 corresponds to the CodiEsp track [
In this work, we present our contribution to the CLEF eHealth 2020 [
The rest of the paper is organised as follows. In Section 2, a short description of the CodiEsp corpus is given, as well as the details of our proposed strategy to tackle the CodiEsp-D subtask are described. We analyze the obtained results in Section 3, whereas Section 4 presents some conclusions and perspectives for future work. 2 2.1
Corpora According to the task organisers1, the CodiEsp corpus comprises 1K clinical case studies covering a broad diversity of medical subjects, such as pneumology, urology, cardiology or oncology. The entire corpus was randomly split into three di erent subsets, the training (500 documents), development (250 documents) and test (250 documents) sets. Also, jointly with the test set, a background set of 2751 clinical cases was released. The latter had the intention of guaranteeing that teams participating in the CodiEsp track could not do manual corrections, since submissions to the task had to include predictions for the 3001 medical documents present in both the background and test sets, but they were only evaluated on the test set.
For the CodiEsp-D subtask, annotation tables containing the assignment of ICD-10-CM codes to the medical documents in the corpus were also available. Furthermore, along with the CodiEsp corpus, a set of additional documents was provided2. The supplementary documents correspond to Spanish abstracts obtained from LILACS3 and IBECS4 biomedical literature databases, which were 1 https://temu.bsc.es/codiesp/index.php/category/data/ 2 https://zenodo.org/record/3606662#.XvxBT59fg8o 3 https://lilacs.bvsalud.org/es/ 4 http://ibecs.isciii.es/ annotated with ICD-10 codes information. From the whole set of abstracts, we only selected those texts annotated with ICD-10-CM codes present in the list of valid codes for the CodiEsp track supplied by the organisers5.
Table 1 contains a basic description of the CodiEsp-D corpus (training,
development and test subsets) as well as the additional abstracts annotated with
diagnosis codes information. As it is shown in the table, CodiEsp-D is a
considerably challenging task, given the limited number of clinical cases (1000) and the
large number of unique codes (2557) present in the corpus, including 363 codes
that are only present in the test subset. Additionally, the number of codes
annotations is also scarce, resulting in a highly imbalanced multi-label classi cation
problem, where for each code, the number of negative samples clearly surpasses
the number of positive cases, i.e. number of documents annotated with a certain
code. For these reasons, we decided to expand the CodiEsp-D corpus using the
additional set of abstracts. Considering that most of the diagnosis codes (2153
out of 2984) included in the abstracts corpus are not present in the CodiEsp-D
annotations, we experimented with two distinct ways of expanding the
training and development CodiEsp-D corpora: either using all available abstracts, or,
alternatively, solely using the abstracts annotated with ICD-10-CM codes
contained in the training and development sets, leading to a reduced version of the
abstracts corpus comprising 115457 documents and 160652 codes annotations,
from which 733 are unique ICD-10 codes.
We have tackled the CodiEsp-D challenge using BERT model [
used in a downstream task, by ne-tuning the whole architecture in a supervised way, then adapting all its pretrained weights to solve a speci c task.
BERT has gained a lot of attention in the NLP community, including in
biomedical and clinical domains, where BERT-based approaches have obtained
state-of-the-art results in a wide range of tasks [
In contrast to sequential models such as recurrent neural networks (RNNs),
for computational feasibility reasons, Transformer architectures cannot deal with
long input sequences of variable length, since for the self-attention layers the
complexity is quadratic on the length of the input sequence [
Over the last year, a few works have already explored di erent strategies to
overcome this limitation. The most straightforward approach is to use a text
truncation method, like the one adopted by the CLEF eHealth 2019 Task 1 best
performing team [
Another recent work explored a di erent approach that does not make use of
any truncation method to adapt BERT to solve a text classi cation task. In this
way, authors in [
8 Since BERT always adds two special tokens ([CLS] and [SEP]) at the rst and last positions, respectively, of an input sequence. clinical case with the same ICD-10-CM codes present in the complete document. This constitutes a problematic situation, as many fragments would be annotated with diagnosis codes which are not represented within the fragment, but appear in other segments from the same clinical document.
With the aim of adopting a strategy to adjust BERT to the distinctive
features of the CodiEsp-D subtask, we have developed a three-phases custom
approach that transforms the multi-label long-text classi cation task into a
multilabel short-fragment classi cation problem. Unlike [
Splitting each clinical case into fragments. As indicated before, BERT supports input sequence lengths up to a maximum value of N (N = 512 in the original implementation). For this reason, for every clinical case in the CodiEsp-D corpus, after performing WordPiece tokenization, we split the resulting sub-tokens sequence w = (w1; w2; :::; wk), of length k, into a sequence of m = dk=(N 2)e contiguous sub-token fragments f = (f1; f2; :::; fm) = ((w1; :::; wN 2); (w(N 2)+1; :::; w2 (N 2)); :::; (w(m 1) (N 2)+1; :::; wk)), in which the m 1 rst fragments f1; ::; fm 1 have a length of N 2 sub-tokens while the last fragment fm contains the remaining nal sub-tokens of the document (considering that the tokens [CLS] and [SEP] have to be later added at the beginning and the end of each fragment, respectively, to get sequences of size N sub-tokens that will constitute the input to BERT).
Annotating text fragments with ICD-10-CM codes. Annotations available for the CodiEsp-D subtask contain the assignment of a set of diagnosis codes to each clinical document (see Fig. 1A). On the other hand, in CodiEspX subtask|which uses the same CodiEsp corpus as the CodiEsp-D subtask|, codes annotations include an extra eld indicating the reference in the text that explains the coding assignment (see Fig. 1B). In this way, using the information provided for the CodiEsp-X subtask, we managed to annotate each of the fragments|resulting from the splitting procedure|exclusively with those ICD-10-CM codes annotations whose text references were contained within the fragment. Thus, given a certain fragment fl and the set of its associated diagnosis codes Cfl , as well as a CodiEsp-X annotation ai consisted in a code ci and a text reference spanning a group of continuous or discontinuous sub-tokens9 ti = (wi1; wi2; :::; wi ), we annotated fl with ci when any of the sub-tokens of ti was contained inside the limits of fl, i.e. ci 2 Cfl () 9wij 2 ti j wijs fls ^ wije fle , where wijs and wije stand for the starting and ending character positions of the sub-token wij, respectively, whereas fls and fle represent the 9 Considering that annotations text references were also tokenized using WordPiece. starting and ending character positions of the fragment fl. To illustrate the annotation process, in Fig. 2, we show the annotations generated for the fragments extracted from the S1139-76322012000400011-1 CodiEsp document using the information available for the same document in the CodiEsp-X subtask (see Fig. 1B).
Obtaining codes probabilities at document level. Using the sub-tokens fragments from the clinical cases in the CodiEsp corpus annotated with ICD10-CM codes, we trained the BERT model on a fragment-level classi cation problem. Since many distinct codes may be assigned to the same medical document, we treated the task as a multi-label classi cation problem. To ne-tune the whole architecture of BERT on a supervised learning task at text-level, the representation produced by the model for the initial [CLS] token was fed into a nal output layer for classi cation. As we were dealing with a multi-label task|which is equivalent to multiple independent binary classi cation tasks|, we used the sigmoid activation function and D output units, with D representing the number of unique diagnosis codes occurring in the texts used to train the model. Hence, given a text fragment as input to the model, the produced output vector could be interpreted as the probability of each code to occur within the input fragment. However, CodiEsp-D task was a document classi cation problem, and the evaluation of the participating systems was performed at document level. Accordingly, using a maximum probability criterion, we post-processed the model predicted probabilities to obtain codes probabilities at document level. Thus, given a sequence f containing all fragments generated from a single document d as input, the model produces the output probability matrix M 2 Rjfj D. Then, selecting the maximum probability value across every column of M , a nal vector p 2 RD of codes probabilities at document level is generated, which contains the probability of each code to appear in d. Using this method, we were able to produce, for each clinical case, a list of D distinct codes sorted in descending order according to their probability values predicted by the model, which was nally used by the organisers to evaluate the performance of the classi cation system.
It should be noted that, in the case of the additional abstracts corpus (see Section 2.1), the available codes annotations did not contain any extra eld indicating the text reference that supports the coding assignment. Therefore, our fragment-based custom approach could not be applied to the abstracts corpus, but only to the CodiEsp-D corpus enriched with the information available for the CodiEsp-X subtask. For this reason, exclusively the abstracts that, after performing WordPiece tokenization, contain a maximum number of N 2 subtokens were used to expand the CodiEsp-D corpus. 2.3
Experiments We experimented with two Spanish versions of the BERT-Base model, namely BETO and BERT-SciELO. Since the vocabulary of the BERT-SciELO model R30.0
R31.0
R31.9
R50.9
B65.0
B65.9 Presentamos el caso de un varón de 11 años original de Gambia que consulta por hematuria macroscópica de predominio al final de la micción y disuria de un año de evolución; sin antecedentes de fiebre. Al realizar la anamnesis refieren un viaje reciente a su país de origen y en el transcurso del mismo varios baños en lagos de la región. La exploración física es anodina.
Ante la sospecha clínica de bilharzhiasis, se contacta con el Servicio de Microbiología del hospital de referencia, donde indican recogida de orina de tres días consecutivos, preferentemente del mediodía y del final de la micción (momento en que es máxima la excreción de huevos) y se solicita estudio mediante ecografía renovesical. El estudio microbiológico demostró huevos de Schistosoma haematobium.
La ecografía vesical practicada puso de manifiesto un engrosamiento parietal que llegaba a alcanzar un grosor máximo de 9 mm en un radio de 20 mm, lo que sugiere esquistosomiasis, por lo que se prescribió tratamiento con prazicuantel.
B Presentamos el caso de un varón de 11 años original de Gambia que
R31.9
R31.0 consulta por hematuria macroscópica de predominio al final de la
R50.9 R30.0 B65.0
B65.9 micción y disuria de un año de evolución; sin antecedentes de fiebre. Al realizar la anamnesis refieren un viaje reciente a su país de origen y en el transcurso del mismo varios baños en lagos de la región. La exploración física es anodina.
B65.9 Ante la sospecha clínica de bilharzhiasis, se contacta con el Servicio de Microbiología del hospital de referencia, donde indican recogida de orina de tres días consecutivos, preferentemente del mediodía y del final de la micción (momento en que es máxima la excreción de huevos) y se solicita estudio mediante ecografía renovesical. El estudio microbiológico demostró huevos de Schistosoma haematobium.
La ecografía vesical practicada puso de manifiesto un engrosamiento parietal que llegaba a alcanzar un grosor máximo de 9 mm en un radio de 20 mm, lo que sugiere esquistosomiasis, por lo que se prescribió tratamiento con prazicuantel.
Fig. 2. Illustration of the text fragments ICD-10-CM codes annotations
obtained after applying the rst two phases of our developed approach to the
S1139-76322012000400011-1 clinical case from the CodiEsp training corpus. To
generate the sub-tokens fragments, we used the WorPiece tokenizer of the BETO model,
setting a maximum fragment length of N 2 = 78 sub-tokens.
did not include any punctuation character, we used a pre-processed version of the
expanded CodiEsp-D corpus (together with the additional abstracts) in which
punctuation marks were substituted by spaces. In the case of BETO, the raw
text from the documents was employed, as punctuation marks were contained
in the vocabulary of the model. Regarding the hardware resources employed,
all experiments were executed on a single GeForce GTX 1080 Ti 11 GB GPU.
Given the limitations imposed by the hardware, we used a maximum input
sequence length of N = 230 for the BERT-SciELO and a value of N = 275 for
the BETO model, with both models having 184M trainable weights. Finally,
with respect to the hyperparameters of the models, for ne-tuning, we used
RAdam [
In this section, we present the results obtained by our team, ICB-UMA, at the CodiEsp-D subtask. Task organisers allowed each participating team to submit up to 5 runs of their classi cation systems. We submitted three distinct runs. The rst two submissions (ICB-UMA-run1 and ICB-UMA-run2) corresponded to the BERT-SciELO model ne-tuned on the CodiEsp-D training and development corpora expanded using additional abstracts annotated with ICD-10-CM codes present in the training and development sets. Thus, the extended corpus contained 2194 distinct diagnosis codes|the number of unique ICD-10-CM codes present in the training and development corpora (see Table 1)|, having a nal BERT-SciELO model with an output classi cation layer of D = 2194 units. Since exclusively the abstracts containing a maximum number of 230 2 WordPiece sub-tokens were considered for the BERT-SciELO model (see Section 2.2), a nal set of 66620 abstracts with 92239 codes annotations were employed to expand the training and development corpora. Submission ICB-UMA-run2 used a dropout probability of 0:1 on the output layer of BERT-SciELO model, whereas in submission ICB-UMA-run1 no dropout was used on the output layer. For its part, in submission ICB-UMA-run3 the BETO model was ne-tuned on the CodiEsp-D training and development sets extended using all available abstracts with a maximum number of 275 2 WordPiece sub-tokens, which comprised a total of 87871 documents and 208076 diagnosis codes annotation, from which 2204 were unique ICD-10-CM codes, i.e. codes that were not present in the training and development corpora. Finally, for submission ICB-UMA-run3 the output layer of the BETO model had D = 4398 units, with no dropout on the nal layer. Apart from these three runs, we also experimented with the BERTSciELO model ne-tuned on the CodiEsp-D corpus expanded using all available abstracts, as well as with the BETO model ne-tuned on the CodiEsp-D corpus extended using additional abstracts annotated with training and development diagnosis codes. However, these two approaches obtained worse results on the development set, and we submitted the three strategies that best performed on the development corpus.
Table 2 and Table 3 show the predictive performance of our three di erent
submitted runs on the CodiEsp test corpus, as a result of the evaluation
performed by the CodiEsp track organisers. Namely, Table 2 presents the results
obtained according to the o cial evaluation metric of the CodiEsp-D subtask,
i.e. the Mean Average Precision (MAP). The second column of the table
contains the MAP values calculated considering all codes present in the CodiEsp
test subset, whereas the results shown in the third column (MAP codes ) were
computed taking into account only the predictions for the test codes that were
also present in the training and development subsets. Finally, the fourth
column (MAP30 ) measures MAP-at-k metric (MAP@k), with k equals 30, while
the last column (MAP30 codes) measures MAP@30 considering only the test
codes that were present in the training and development corpora (as in the third
column of the table). According to the results observed in Table 2, the
BERTSciELO-based model outperformed the BETO-based classi er on the CodiEsp-D
subtask, since both the ICB-UMA-run1 and ICB-UMA-run2 systems achieved
higher values for all analyzed metrics than the ICB-UMA-run3 submitted
system. The best performance is obtained by the BERT-SciELO model when no
dropout is used on the output layer, as the ICB-UMA-run1 results slightly
surpassed the performance of the ICB-UMA-run2 system across the four examined
evaluation metrics. To summarize, we can say that, according to our obtained
results, the BERT-SciELO model outperformed the BETO on the CodiEsp-D
predictive task. Therefore, pre-training the BERT-Base architecture from scratch
on an unlabeled corpus of Spanish biomedical articles, rather than using a general
domain Spanish corpus, leads to a better performance of the model on a clinical
coding task in the context of Spanish medical narrative. The results obtained in
this work for the CodiEsp-D subtask support the hypothesis already explored
in previous works [
On the other hand, to perform a larger analysis of the results, the task
organisers evaluated the performance of the systems according to a set of additional
metrics, other than the o cial ones. In this way, in Table 3, the second, third
and fourth columns show the calculated values using precision (P ), recall (R)
and the F1 score (F1 ) metrics, respectively, considering all codes present in the
CodiEsp-D test set, while the fth (P codes), sixth (R codes) and seventh (F1
codes) columns contain the results computed using the same three metrics but
taking into consideration only the codes present in the training and development
subsets. Finally, in the last three columns (P cat, R cat and F1 cat ), the
previous metrics are used to evaluate the submitted predictions at the hierarchical
category level of the ICD-10-CM codes contained in the test set. As it can be
observed from Table 3, for the precision and consequently for the F1 score, our
three submitted systems obtained extremely poor values, though for the recall
the computed values are abnormally high. The reason for this is that, with the
aim of maximizing the score obtained for the o cial evaluation metric, i.e. MAP,
for each test document we submitted all codes considered by the model|2194
codes in the case of ICB-UMA-run1 and ICB-UMA-run2 and 4398 codes for
the ICB-UMA-run3 submission|ordered by their predicted probability of
occurrence (see Section 2.2). If we had optimized precision, recall and F1 score
metrics instead of MAP, in place of submitting all codes, a decision threshold
would have been de ned to select solely a subset of the codes according to their
predicted probabilities.
In this paper, we present our contribution to the CodiEsp-D subtask from the
CodiEsp track [
We have tackled the challenge as a multi-label text classi cation task using
BERT model [
In future works, we will try to enhance the developed fragment-based classi cation strategy to further improve the obtained results on the CodiEsp-D subtask. For instance, when splitting each clinical case into fragments, we could perform the text segmentation at the sentence level, producing fragments comprising a sequence of sentences with a complete semantic meaning. On the other hand, given the superior performance observed from the BERT-SciELO model, it is worth investigating whether alternative clinical-speci c versions of BERT pre-trained on Spanish medical corpora more similar to the CodiEsp corpus could increase the results further. Additionally, due to the widespread adoption of BERT in multi-lingual setups across domains, we could also explore the pre-training of the model on a multi-lingual medical corpus. This would permit the creation of enormous medical corpora comprising clinical documents written in many di erent languages. Because of the sub-word vocabulary employed by BERT and the common etymology of numerous medical terms across distinct languages, multi-lingual clinical corpora could serve as a valuable source of data to pre-train BERT models. The resulting BERT's deep bidirectional architecture could leverage its language modeling capabilities to produce e ective contextual representations that could be used in applications within a vast number of medical NLP information-extraction problems.
Acknowledgments. This work was partially supported by the project TIN201788728-C2-1-R, MINECO, Plan Nacional de I+D+I, and I Plan Propio de Investigacion y Transferencia of the Universidad de Malaga.