Clinical coding consists in the transformation (or classi cation) of patient record information into a structured or coded format using internationally recognized class codes. Coding accuracy is an ongoing challenge which has led to the organization of challenges and shared tasks to evaluate AI-enhanced, computer-assisted coding systems. In this paper we present our contribution at CodiEsp: Clinical Case Coding Task (CLEF eHealth 2020) on the automatic assignment of clinical coding (diagnosis and procedures) to clinical cases in Spanish. We approach the task as multi-label classi cation problem and leverage the powerful language model: Multilingual BERT (M-BERT) to represent the clinical cases and design various deep learning architectures based on a Convolutional Neural Network and a Long Short-Term Memory Network (CNNLSTM) classi er. To handle the class-imbalance problem, we present other models based on data augmentation techniques (i.e. word-level transformations and text generation methods) for synthesizing labeleddata. Models based on data augmentation pipelines obtain the best results, measured by the F1-score, in comparison to the other proposed models for both tasks. The pipeline based on the word-level transformations obtains the best F1-score (0.143) for the CodiEsp-D task, while the data augmentation technique using the text generation method achieves the best F1-score (0.216) for the CodiEsp-P task.
The International Classi cation of Diseases (ICD) is a health care classi cation
system which provides standardized codes for reporting diseases and health
conditions1. ICD codes are widely used in Electronic Medical Records (EMR) to
describe a patient's diagnosis or treatment. In current practice medical coders
review a physician's clinical diagnosis (almost always recorded as free text) then
manually assign ICD codes according to coding guidelines. While the process
of standardizing EMR is important for making clinical and nancial decisions
manual ICD coding is expensive, time-consuming and prone to error [
This paper presents our contribution at the CLEF eHealth CodiEsp 2020
CodiEsp-D and CodiEsp-P sub-tasks. In total ve models were submitted
during the o cial evaluation, all based on a Convolutional Neural Network and Long
Short-Term Memory Network (CNN-LSTM) classi er. Multilingual BERT
(MBERT) achieved the best performances in the CLEF eHealth 2019 Multilingual
Information Extraction task [
The Codiesp corpus4 consists of a set of 1000 clinical cases manually annotated by clinical coding professionals 5. Documents were coded with clinical diagnosis and procedure codes from the Spanish o cial version of ICD10-Clinical Modi cation and ICD10-Procedure. The released corpus has around 16,504 sentences and 396,988 words, with an average of 396.2 words per clinical case. The corpus has been randomly sampled into three subsets: the training set (500 clinical cases), the development set and the test sets (250 clinical cases each). Each subset provides clinical cases in plain text format stored as single les (each lename corresponds to an unique clinical case identi er) and a tab-separated le with either ICD10-Diagnostico (equivalent to ICD10-CM) or ICD10-Procedimiento (equivalent to ICD10-PCS) code assignments according to the target task. Table 1 summarises the top-5 most frequent ICD10-Diagnostico and ICD10-Procedimiento codes from the training and development datasets for both tasks.
As we can observe in Table 1, the datasets provided are high imbalanced
(i.e. there is high disparity between classes), with 15.73% and 9.87%
respectively as the highest frequency rates for the Codiesp-D and Codiesp-P tasks.
In total, 10,711 codes were assigned for both tasks, of which, 1819 are unique
in the Codiesp-D datasets and 608 in the Codiesp-P datasets. The proportion
of rare classes (i.e. classes with only one observation) is also high, 1022 classes
(i.e. 56.18%) in the codiesp-D datasets and 393 (i.e. 64.64%) in the
CodiespP datasets. These ndings led us to investigate data augmentation techniques
which have shown promise in scarce labeled data situations [
Moreover, to expand the training and development corpora, the organizers have also released several additional data resources6 including medical literature abstracts (i.e. abstracts from Lilacs and Ibecs with ICD10 codes), linguistic 4 Codiesp corpus available online: https://zenodo.org/record/3837305# .XvsEN5bTVhF Date of access: 30th June 2020. 5 Information about annotation guidelines: https://zenodo.org/record/3632523# .Xvw2N5bTU5m Date of access: 1st July 2020. 6 https://temu.bsc.es/codiesp/index.php/2019/09/19/resources/ Date of access: 30th June 2020. resources, gazetteers and a machine-translated version from English of Codiesp corpus clinical cases. 3
Empirical studies conducted on the development sets for each task7 found best
performance using a CNN-LSTM classi er. Figure 1 details the architecture
used and the shared parameters for both tasks.The model takes as input a
timeordered sequence of tokens (words) of arbitrary length (truncated to 396 words
which corresponds here the averaged number of words per document, and then
padded with zero vectors) and outputs a document-level prediction. After the
embedding layer, the layer corresponding to the CNN classi er (one-headed) is
introduced using a con guration of 100 parallel feature maps and a kernel size
of 3. Immediately afterwards, a LSTM layer is added (set to 100 internal units).
Then a dense layer of 64 nodes with ReLu is inserted. Finally an output layer is
used with one node containing softmax function. The models have been trained
using the Adam optimizer, with a learning rate of 0:001 and a batch size xed
to 32 for both tasks.
All proposed methods were trained and tested using the Spanish version of the
released corpora. Concerning the preprocessing steps, clinical cases were converted
to lowercase and stop-words were removed. After the tokenization process, all
tokens based only on non-alphanumeric characters and all short tokens (with
< 3 characters) were also deleted. In total ve models were submitted to the
o cial evaluation, we provide below a detailed description of each of them:
- CNN-LSTM: this default approach (used here as a baseline) is based on
the architecture presented in section 3.
- M-BERT: this approach is based on the BERT language model [
(PTM) have been built from this text encoding model which has previously
been successfully applied to various biomedical NLP tasks [
- original: Paciente de 50 an~os con antecedente de litiasis renal de
repeticion que consulto por hematuria recidivante y sensacion de
malestar. El estudio citologico seriado de orinas demostro la
presencia de celulas at picas sospechosas de malignidad.
- edited: Paciente de 50 an~os con antecedente de litiasis nefr tico de
repeticion que consulto por hematuria recidivante y percepcion de
malestar. El estudio citologico seriado de orinas demostro la
apariencia de celulas at picas sospechosas de malignidad.
- WordNet M-BERT: based on the two previous approaches: WordNet
and M-BERT, we explore the combination of a word-level data
augmentation technique and the M-BERT pre-trained language model representation.
Also implemented as a preprocessing step of the data generator pipeline,
synonym replacement is based on the same setup as in the original approach
i.e. 10% of documents' words are substituted. Then, as introduced in the
M-BERT approach, models are trained as a part of a STL scenario.
- TEXT GEN: in this approach we propose to explore a novel data
augmentation technique based on a text generation method. This strategy was recently
8 BioBERT trained from the original BERT pre-trained model and medical resources
in English can't be applied to clinical cases in Spanish
9 Synonym replacement is performed using the python library NLPAug10
introduced for synthesizing labeled data to improve text classi cation tasks.
Approaches leveraging text generation have shown promise, outperforming
state-of-the-art techniques for data augmentation, speci cally for handling
scarce data situations [
- original: Analytical analysis showed hydroxyvitamin lion ponesium sodium. - edited: Analytical analysis showed sequence made transoperative urine outpatient image microbiological markers ap immunohistochemical intravenous dorsolumbar remained level signs 1788 partially establishing transplantation. 5
Models were trained on a workstation with a 36-core CPU and an AMD FirePro W2100 GPU. Systems were evaluated according to the following metrics: Mean Average Precision (MAP), MAP@30, precision, recall and F1-score. For experimental purposes two versions of the F1-score metric are computed: the F1-score measure which considers the full code for both tasks and the F1-score CAT which considers only the rst three digits of ICD10-Clinical Modi cation codes (e.g. codes R20.1 and R20.9 are mapped to R20) and the rst four digits of ICD10-Procedure codes (e.g. the code bw40zzz is mapped to bw40). Table 2 summarizes the results obtained for both the CodiEsp-D task and the CodiEspP task on the test sets. For readability purposes only the MAP, the F1-score and the F1-score CAT are reported. Due to lack of time, not all models for each task were proposed at the o cial evaluation. However we performed the missing evaluations using the evaluation library released by the organizers11, results are presented in italic font.
As we can observe the results obtained depend on both the tasks and the models used. For the CodiEsp-D task the Wordnet model (WN) achieves the best MAP, followed closely by the model based on the pre-trained language model M-BERT. For the F1-score, the WN model also obtains the best performance against the other proposed approaches (+0:029 from the baseline). For the F1score CAT, the baseline is rst-ranked, slightly outperforming the WN model ( 0:001 from the baseline). For the CodiEsp-P task the best MAP is obtained using the combination of M-BERT and the data augmentation technique based on synonym replacement (WN M-BERT) while the TEXT GEN model achieves the best performance for both F1-scores (+0:031 for the F1-score and +0:096 for the F1-score CAT, measured relative to the baseline). Concerning the uno cial results (in italic font), the model WN is rst-ranked on the CodiEsp-P task while the TEXT GEN model is the less e cient for the CodiEsp-D task.
In the overall evaluation, the use of a STL-based architecture combined with a pre-trained model has shown its e ciency, outperforming the baseline on both tasks on the majority of evaluation metrics. Concerning data augmentation techniques, despite missing evaluations, the proposed techniques produced strong results in comparison with the other models, outperforming both the baseline and the models based on M-BERT on both tasks on the majority of evaluation metrics. 6
In this paper we presented our contribution to the CLEF eHealth CodiEsp 2020 CodiEsp-D and CodiEsp-P sub-tasks. In total we proposed ve models during the o cial evaluation in which we explored both the powerful language model: Multilingual BERT (M-BERT) and two data augmentation techniques, wordlevel transformation and text generation methods, for synthesizing labeled-data. Models based on data augmentation pipelines achieved the best performances in comparison to the other proposed models for both tasks on the majority of evaluation metrics. 16. Shi, H., Xie, P., Hu, Z., Zhang, M., Xing, E.P.: Towards automated icd coding using deep learning. CoRR arXiv:1711.04075 (2017) 17. Wei, J., Zou, K.: Eda: Easy data augmentation techniques for boosting performance on text classi cation tasks. CoRR arXiv:1901.11196 (2019)