Electronic Clinical Narratives (ECN) are the standard for storing relevant information to describe and evaluate a patient's clinical episode or evolution. The ClinAIS task aims to tackle the problem of automatic identification of sections in unstructured Spanish ECNs. In this work, we tackle this challenge by first ifne-tuning a language model with the ClinAIS dataset, for later applying a second fine-tuning stage for section identification. The performance of several models was studied using this approach, and a Longformer based model obtained the best results in the validation set. Using this model, we achieved the third position in the ClinAIS challenge with a weighted B2 score of 0.7036 in the test set.
The amount of digitised data available from healthcare systems is increasing exponentially [
A fundamental task for the advancement of higher-level applications in healthcare is the accurate identification of medical sections within patient narratives documented in ECNs. This task involves the division of the text into semantic segments and assigning them specific predefined labels. Through section identification, valuable insights can be gleaned regarding diferent entities, which may vary significantly depending on the section in which they are found. For example, a pathology mentioned in the patient’s medical history section could be utilized to predict future conditions and assess the risk of illness. Similarly, the presence of specific symptoms in the Evolution section of the narrative might indicate adverse reactions to a particular treatment.
The ClinAIS task presented at IberLEF 2023 [
The ClinAIS dataset’s corpus was obtained from the CodiEsp dataset, which was presented in
the eHealth CLEF 2020 task [
For annotation, a set of guidelines were initially created to identify patterns and categorize each section in unstructured clinical notes into 7 categories: Present Illness, Derived from/to, Past Medical History, Family history, Exploration, Treatment and Evolution. From those guidelines, a group of experts went through several rounds of annotating a small set of notes and updating the guidelines accordingly. When the annotation process became more mature, two doctors, trained in clinical report annotation for diferent tasks, performed a double annotation on the notes. The annotation task was iterative and the evaluation metric was employed to measure the inter-tagger agreement, reaching 75%.
Once the dataset was annotated, the 1038 notes were split into three groups: training (75%, 781 notes), validation (12.5%, 127 notes), and test (12.5%, 130 notes) sets. The training and validation sets were publicly released with their annotations, whereas the annotation of the test set was kept private. The evaluation on the test set was conducted by the judges of the competition.
The approach followed here for section identification is based on the transfer learning method
proposed in the ULMFIT paper [
For our work, we have studied several pre-trained language models available at HuggingFace.
In particular, we considered two versions of the multi-lingual XLM-RoBERTa model (base and
large versions) [
All the models were trained using the functionality provided by the HuggingFace libraries [12]
and using a GPU Nvidia GeForce 3090. The hyperparameters used for training the models can
be checked in the code available on the project webpage. The validation set was used to evaluate
the performance of the models and to select the model that was finally used to perform the
predictions in the test set. The metric employed to evaluate the models is an adaption of the
boundary distance developed by C. Fournier [13] called weighted B2 metric — details about
this metric can be found in [
Given a sentence, the final models classify each token into B-Class, E-Class or Class — where Class corresponds with one of the 7 predefined medical sections of the ClinAIS dataset, B-Class indicates the beginning of a section, E-Class indicates the end of a section, and Class means that the token is inside a section. Since such an output might produce inconsistencies (for instance a token classified as Class2 inside a sentence that started with B-Class1), the output of the model is post-process to deal with those inconsistencies.
We start by evaluating the trained models, but we skip the first stage of fine-tuning with the ClinAIS dataset. Instead, we solely train the models for the task of section identification. Please refer to Table 2. The best model using this approach, with a weighted B2 score of 0.7341, was a Longformer-based model combined with the post-processing step. If we analyze the rest of the models, the 3 best models were initially pre-trained with a biomedical Spanish dataset; whereas, the other models, which were trained with generic Spanish or multi-lingual datasets, obtained considerably worse results. It is also worth mentioning the importance of the post-processing stage, since the plain output of all models was inferior to their post-processed counterparts.
In our second set of experiments, we evaluated the models trained using the two-stage approach presented previously, see Table 3. In general, all the models, except for the Longformerbased model, obtained better results than their counterparts trained only for section identification. We can draw similar conclusions to those noticed in the first set of experiments: the models initially pre-trained with biomedical Spanish datasets obtained better results, and the post-processing stage improved the performance of the models. However, the best performing plain output post-process plain output post-process plain output post-process model, a RoBERTa-based model with a weighted B2 score of 0.7311, achieved worse results than those obtained by the Longformer-based model trained only on section identification.
Finally, we applied the two-stage procedure to an artificially augmented version of the ClinAIS dataset. In particular, we hide some of the words of the notes, and used a RoBERTabased language model to predict those missing words. This allowed us to obtain a training dataset of 2850 notes. The results obtained for this augmented dataset are presented in Table 4. Unfortunately, this approach did not lead to any improvement.
Following this study, we utilized the one-stage fine-tuned Longformer-based model, determined as the best model based on our evaluation, to generate predictions for the test set of the ClinAIS challenge. This model achieved a weighted B2 score of 0.7036.
In this paper, we have used the two-stage training procedure presented in the ULMFIT work for training several models for medical section identification in unstructured clinical notes. In general, the two stage procedure (first building a specialized language model, and then ifne-tuning the model for section identification) provides better results than only training the models for section identification. However, the best model was a Longformer-based model that was only trained for section identification and achieved a final score of 0.7036 in the test set.
This work was partially supported by Ministerio de Ciencia e Innovación [PID2020-115225RB-I00 / AEI / 10.13039/501100011033]. [9] L. Tunstall, The large spanish corpus, 2022. https://huggingface.co/datasets/large_spanish_ corpus. [10] C. P. Carrino, J. Armengol-Estapé, A. Gutiérrez-Fandiño, J. Llop-Palao, M. Pàmies, A. Gonzalez-Agirre, M. Villegas, Biomedical and clinical language models for spanish: On the benefits of domain-specific pretraining in a mid-resource scenario, 2021. arXiv:2109.03570. [11] I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer, arXiv:2004.05150 (2020). [12] L. Tunstall, L. Von Werra, T. Wolf, Natural language processing with transformers, "
O’Reilly Media, Inc.", 2022. [13] C. Fournier, Evaluating text segmentation using boundary edit distance, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2013, pp. 1702–1712.