This paper presents the participation of the SINAI team in the ClinAIS shared task at IberLEF 2023, focusing on the task of section identification in clinical reports. The approach involves a multiclass token classification framework for section boundary detection, utilizing two system variants tuned for detecting one-token and three-token long boundaries. The system is built upon a RoBERTa architecture model pre-trained on biomedical and clinical corpora, and fine-tuned for the token classification task through hyperparameter optimization trials.The results show that fine-tuning for longer boundaries improved performance (0.6766 vs 0.6986 weighted B2 score). Error analysis revealed challenges in detecting "DERIVED_FROM/TO" and "EVOLUTION" sections due to class imbalance and semantic confusion.
entities and map them into codes from a controlled vocabulary like SNOMED-CT [
The main challenges of clinical NLP involve dealing with the overall linguistic complexity of
an EHR: negated expressions, co-references, misspellings, abbreviations, anaphoric relations
and so on [
The realization of SI was proven to be beneficial for the performance in clinical information
extraction tasks such as entity recognition in Chinese clinical reports [
However, the capability to analyze clinical text in languages other than English holds
significant potential for accessing crucial medical data pertaining to patient cohorts treated in
countries where English is not the oficial language[
ClinAIS shared task at the Iberian Languages Evaluation Forum (IberLEF) 2023 aims to impulse
the research in the automatic SI applied to the Spanish language by providing a dataset of 1,038
clinical reports annotated with seven predefined medical sections: Present Illness, Derived
from/to, Past Medical History, Family history, Exploration, Treatment and Evolution [
The main objective of this paper is to describe the system presented by the SINAI team at
the ClinAIS shared task. Our approach follows the line of the best performed system described
in Goenaga et al. [
The remainder of the paper is organized as follows: Section 2 provides a detailed description of the datasets that were made available by the organizers of the competition, Section 3 is dedicated to the system we developed to tackle the task of automatic SI, Section 3.1 outlines the specific procedures and configurations used during the experiments, Section 4 discloses the results obtained by the presented system during the oficial evaluation that are subsequently analyzed in Section 5 which critically examines the overall system performance and investigates potential sources of errors. Finally, Section 6 summarizes the main aspects of our contribution and ofers insights to enhance future iterations of the automatic section identification system.
ClinAIS corpus [
The data was divided into three subsets in a stratified manner so that the section distribution is similar in all three sets: training, development and test. The proportion of notes in each is 0.75, 0.125 and 0.125 respectively. It is worth mentioning that the evaluation set was released alongside 2,751 unannotated documents in order to prevent the participants from performing manual corrections and/or annotations.
Each corpus entry was annotated with borders of sections from the following list: Present Illness, Derived from/to, Past Medical History, Family history, Exploration, Treatment and Evolution. Figure 1 shows the class label distribution across training and development datasets. The sections Derived from/to and Family are scarcely represented compared to other labels.
As it can be inferred from the Table 1 which summarizes some relevant statistics of the data subsets, the average report length (measured in tokens produced by RoBERTa tokenizer) doesn’t exceed 512, the maximum input length of the majority of pre-trained transformer models available for Spanish, in all of the subsets. We can also point out that the data is quite homogeneous across subsets in terms of the average of the annotated sections.
Notably, section borders in many cases do not match with sentence starts which makes it unviable to rely on a sentence classification approach. Number of reports Avg. report length (STD) Avg. sections (STD) Avg. unique sections (STD)
Fine-tuning large pre-trained transformer models was proven to be an efective approximation to
many NLP tasks like text classification [
In order to convert the corpus into a format that would be coherent with our problem formulation, we performed pre-tokenization by splitting the corpus by white space and treating the punctuation marks as separate units. As for the token labeling, we compared the approach of marking only the boundary word as an entity to the method of selecting the first three words of each section and labeling these according to the BIO (Beginning, Inside and Outside) scheme.
We opted for building our system with a RoBERTa architecture model [
For each system, we performed 5 trials on a single NVIDIA Ampere A100 GPU. The hyperparameter search space was defined as follows: • Learning rate: a float value between 3 − 5 and 5 − 5 • Per device train batch size: either 8 or 16 • Weight decay: a float value between 1 − 12 and 1 − 1 • Adam epsilon: a float value between 1 − 10 and 1 − 6 • Warmup steps: an integer value between 0 and 1000
The number of training epochs was selected by making use of the Early Stopping strategy that interrupts fine-tuning when the reference metric doesn’t improve during 3 epochs. Table 2 presents the hyperparameters selected for each of the experiments.
Single word boundary
Three word boundary Learning rate Training epochs Batch size Weight decay AdamW epsilon Warmup steps
Our team submitted a total of two runs: one per each of the described pre-processing
methodologies. Identification of sections within unstructured clinical notes poses numerous challenges,
thereby impeding the comprehensive assessment of this task. Prominent among these challenges
is the inherent interconnection between the end of one section and the start of another. Another
issue is related to the fact that sections are not delimited by paragraphs, lines or phrases. To
address this the organizers designed the B2 evaluation metric, which is an adaptation of the
boundary distance [
This section attempts to provide insights regarding the overall system performance and investigate the possible reasons for its errors. As shown in Table 3, the variant that involved recognition of the first three tokens of each section was proven to be the best option scoring 0.6986 weighed B2 during the oficial evaluation on the test set. This performance improvement can be explained by the fact that the majority of the words at the beginning of each section are adpositions (e.g. con (with), en (in)), determiners (e.g. el/la (the)) and pronouns (e.g. su (his/her)). This type of words lack of semantic information almost completely which has a negative impact on the amount of information that can be captured by an embedding of a contextual model such as RoBERTa clinical. Therefore, the classifier layer cannot learn and generalize correctly distinctive features of such tokens which leads to classification errors. Figure 2 displays the frequency of apparition of diferent parts-of-speech at the beginning of the sections.
Moreover, we carried out an error analysis using the predictions made with the best performing variant of our system on the development subset and evaluated with the oficial evaluation script scoring 0.7477 weighted B2. We distinguish between four types of errors: additions (when the system identifies a section that is not present in the report), deletions (when the system is not capable to detect a section), transpositions (the section was identified, but the extracted span is transposed) and substitutions (the identified section is assigned a wrong label). Figure 3 summarizes the error type and rate per each section type.
The highest number of matches with respect to a total number of reference annotations (169 out of 218) is shown for the label “PRESENT_ILLNESS” that is the most commonly seen across the training dataset. Nevertheless, there is no clear direct relation between the most represented labels in the dataset and the quality of predictions: the categories with the worst matches/reference annotations ratio are “DERIVED_FROM/TO” - 10 matches out of 24 reference annotations and “EVOLUTION” - 81 matches out of 198 references. Notably, the “DERIVED_FROM/TO” class is the only one where the number of additions (11) is higher than the matches. As for the identification of the “EVOLUTION” sections, the frequency of deletions (70) is worth mentioning.
As for the substitutions, 14 errors of this type were detected when evaluating our best performing system on the development dataset. The most confounded label resulted to be “EVOLUTION” which was substituted with “EXPLORATION” 4 times and once with “TREATMENT”. Moreover, it substituted the label “TREATMENT” twice and “PRESENT_ILLNESS” once. The “EVOLUTION” was defined in the annotation guidelines as the “ Evolution of the patient’s health status. It may include diferential diagnoses ”, which means it can include references to the explorations performed and its results, as well as the response to treatments and updates on the present illness, so semantically this section might be close to many other sections depending on the report and section position within it.
This paper covers the participation of the SINAI team at the ClinAIS shared task held on IberLEF 2023. We describe our approach to the task of section identification in clinical reports. Our problem formulation follows a multiclass token classification approach for section boundary detection and we compare two variants of systems: one tuned to detect one-token boundary and the other tuned to recognize three-token-long boundaries. We based our system on a RoBERTa architecture model pre-trained on a combination of biomedical and clinical corpora that was fine-tuned for the token classification task with hyperparameters selected during a 5 trial optimization.
The approach of fine-tuning a model to detect longer section boundaries performed better scoring 0.6986 weighted B2 score. This improvement is probably due to the fact that the majority of sections start with a function word like a preposition or a determiner.
In order to shed light on the errors made by our system we conducted an error analysis of the predictions of our best performing system on the development set. The two sections that resulted to be the hardest to detect have proven to be “DERIVED_FROM/TO”, one of the less represented classes in the training set, and “EVOLUTION” which might be semantically close to other sections like “TREATMENT”, “PRESENT_ILLNESS” or “EXPLORATION” and is thus frequently confounded or omitted.
With the purpose of dealing with these issues, an implementation of a weighted loss function can be proposed as a way of mitigating the impact of class imbalance. With the same aim data augmentation techniques like back translation or generative language models can be used. A more in-depth dataset analysis can provide more detailed information regarding the linguistic characteristics of each section which can be employed as an extension to the contextual embedding during the classification process.
This work has been partially supported by WeLee project (1380939, FEDER Andalucía 2014-2020) funded by the Andalusian Regional Government, and projects CONSENSO (PID2021-122263OBC21), MODERATES (TED2021-130145B-I00), SocialTOX (PDC2022-133146-C21) funded by Plan Nacional I+D+i from the Spanish Government, and project PRECOM (SUBV-00016) funded by the Ministry of Consumer Afairs of the Spanish Government.