In this paper, we explore possible improvements of transformer models in a low-resource setting. In particular, we present our approaches to tackle the rst two of three subtasks of the MEDDOPROF competition, i.e., the extraction and classi cation of job expressions in Spanish clinical texts. As neither language nor domain experts, we experiment with the multilingual XLM-R transformer model and tackle these low-resource information extraction tasks as sequence-labeling problems. We explore domain- and language-adaptive pretraining, transfer learning and strategic datasplits to boost the transformer model. Our results show strong improvements using these methods by up to 5.3 F1 points compared to a ne-tuned XLM-R model. Our best models achieve 83.2 and 79.3 F1 for the rst two tasks, respectively.
Information extraction in non-standard domains is a challenging problem due to
the large number of complex terms and unusual document structures [
In this paper, we explore possible enhancements of transformer models to
overcome this domain and language gap in the context of the MEDDOPROF shared
task [
We approach this challenge as Neither Language Nor Domain Experts
(NLNDE) and model them as sequence labeling tasks. Our solution for these
tasks is a neural sequence tagger based on multilingual transformer models.
In particular, we experiment with continuing the masked language modeling
pretraining of the multilingual XLM-R model [
Our results highlight the importance of domain- and language-adapted transformer models, as well as the advantages of combining several models trained on challenging datasplits with ensembling techniques. Using these methods, our best models achieve F1-scores of 83.2 and 79.3 for the two tasks and outperform a ne-tuned XLM-R model by 4.2 and 5.3 F1 points, respectively. 2
The MEDODPROF challenge follows a series of shared tasks on Spanish clinical
information extraction, including the MEDDOCAN shared task on medical
document anonymization [
As the inclusion of domain knowledge via domain-speci c embeddings in
these special settings is often bene cial [
In addition, we analyze the e ects of model transfer between the rst tasks
of the challenge, as model transfer between related tasks in similar domains can
result in signi cant performance gains [
Further, there is a line of work now questioning traditional train-dev splits
[
Document
Train ?
Dev ?
Strategies: S1,2 Everything for Training S3 Strategic Datasplits
PROFESION (T1)
SANITARIO
(T2) Paciente mujer de 38 años remitida a consulta desde su médico de cabecera para estudio de eccemaen dorso de manos que viene presentando desde hace 3 años.
De profesión manipuladora de rosquil etas.
...
XLM-R Subword
Tokenization _Pacient e _mujer _de _38 _años _remit ida _a _consulta _desde _su _médico _de _cabe cera _para _estudio _de _ecce ma _en _dor so _de _manos _que _viene _presenta ndo _desde _hace _3 _años.
_De _profesión _manipula dora _de _ros qui l e tas . ...
O O O O O O O O O O O O O O O T1: O O B-PROF I-PROF I-PROF I-PROF I-PROF I-PROF E-PROF O
O O ... O B-PROF I-PROF I-PROF E-PROF O O ...
PROFESION
(T1) PACIENTE (T2)
CRF XLM-R
S2-3 Multilingual MLM S2-3 + Spanish General MLM S1-3 + Spanish Clinical MLM
S4 + Trained on other Task <DOCSTART> __Pcaacbieencteera...__psaura_m_eésdtuodcoio_..d.e _De _profesión ... _Pacient e _mujer ... _De p_rdoefe_srióons _qmuialneiptausla.dora _La _paciente ... _De profesión ... _La _paciente _refiere _emp e or ar _No _puedo ...
_tras _el _contacto ...
Left context Target sentence Right context
(100 tokens max) (300 tokens max) (100 tokens max)
Step 1: Datasplit Creation Step 2: Document Preprocessing
Step 3: Model Training & Prediction
Tokenization can be challenging in non-standard domains, including the clinical
domain [
Domain- and Language-speci c Masked Language Modeling
We use XLM-R [
Thus, we explore further pretraining of this model and tune it towards Spanish
documents by pretraining either on (1) a medium-size Spanish corpus with general
domain documents [
We use masked language modeling for pretraining and trained for three epochs
over the corpora, which roughly corresponds to 30k steps for the smaller clinical
corpus and 685k steps for the general-domain corpus using a batch-size of 4.
Finally, we have three XLM-R variants that we compare in this paper:
1. Standard XLM-R pretrained on 100 languages by [
Spanish documents from the general domain. 3. Spanish Clinical XLM-R based on standard XLM-R with further pretraining using Spanish documents from the clinical domain. 3.3
Sequence Tagger
For the sequence tagger, we use one of the XLM-R models, either the standard
XLM-R or one of our adapted models, and apply a CRF layer on top [
We split all sentences to a maximum length of 300 subtokens and add the context of up to 100 subtokens to the left/right to get cross-sentence information. The labels are in BIOSE encoding, which is an extended BIO encoding with additional labels for the last token of an annotation (E-) and single-token annotations (S-)
Our model architecture is basically the same across all runs. We only exchange the transformer model. The models are trained using an AdamW optimizer for a maximum of 20 epochs. Our hyperparameters are given in Table 2. 3.4
Strategic Datasplits We test two options to train the sequence taggers:
(1) Using all of the available training data and stop training according to the training loss. This method provides the model with the most input instances. However, the stopping criterion is not as meaningful as using the task's metric on a held-out validation set.
(2) Thus, as our second method, we split the data into train and validation
sets. Then, we train the model using only the train-fraction of all the data and
use the held-out validation data to determine the best model, which is then used
to annotate the test data. As an alternative to random splits, we follow [
For this, each document is represented as the average vector of the XLM-R embeddings for each token. This document representation is reduced to ve dimensions using PCA. Finally, the documents are clustered into ve equallysized splits using k-Means clustering. We train ve models for each task and embedding with each having a di erent validation split. Our splits are visualized in Figure 2. We see that clusters 1, 2 and 4 are densely populated with highly similar documents, while clusters 3 and 5 contain more distinct documents.
To better understand the strategic datasplits, we analyzed whether the di erent medical topics included in the corpus correlate with the splits. However, we found that the strategic clusters incorporate more diverse information than just topic similarity as there is no substantial overlap between topics and datasplits. 2C 10 P 40 30 20 0 10 20
Fig. 2. Our strategic datasplits in the two-dimensional space with PCA. In order to capture the di erent advantages of multiple models, we combine them using ensembling. This is particularly helpful when models carry di erent types of information. For example, the models trained using our strategic datasplits all have seen a slightly di erent training set, and, thus, combining them using ensembling should further improve performance. We apply ensembling by majority voting. For this, we use hard voting that counts the labels by each model and does not consider the CRF probabilities. We convert the BIOSE labels to BIO labels for the ensembling process as the simpler BIO encoding leads to fewer con icts. Further, we apply postprocessing of the label sequence to correct inconsistencies in the resulting label sequence and enforce for 0.23% of predictions for the test set that the rst token of each annotations begins with B-. As the rst two tasks of the challenge are related and can possibly bene t from each other, we explore the potential of model transfer between them. For example, having basic knowledge of what and which occupations (task 1) are mentioned in a text can be useful to determine whether occupations are related to the patient or to someone else (task 2). For this, we rst train models on the auxiliary task and then transfer the resulting model to the targeted main task. In our case, the auxiliary task is either task 1 or 2 and the other task is the main task. The following ve runs are the NLNDE submissions to the MEDDOPROF shared task. We use the same model architectures for both tasks. Note that all submissions, except for S1 are ensembles of multiple models based on the three di erent embeddings. In Section 4, we compare these submissions with further model variations.
S1 : The Spanish clinical XLM-R model trained on the complete training data. S2 : All three XLM-R language models combined in one ensemble (3 models). S3 : Ensemble of models trained using strategic datasplits (15 models). S4 : Ensemble of models based on transfer learning from the auxiliary task to the main task (3 models).
S5 : The combination of all above models into one ensemble (21 models). 4
Our o cial results for the rst two tasks of the MEDDOPROF shared task are given in Table 3. In addition, we include several other models to analyze the performance of each embedding, because most of our submissions are ensembles combining all three XLM-R embeddings. The o cial evaluation metric is the F1-score and our best models are highlighted.
We nd that the Spanish XLM-R trained on general-domain Spanish data is often the best transformer compared to the standard XLM-R and the clinical one, probably because it was trained on the largest amount of data. In addition, the extraction of occupations is not unique to the clinical domain and general-domain Spanish knowledge seems to be bene cial for this as well.
We nd that model transfer (S4 ) is only useful when transferring models from task 1 (the detection of occupations) to task 2 (the classi cation of occupations). Reusing models that already learned the detection of profession expressions as an auxiliary task improves the main task, e.g., by up to 2.9 F1 points for XLM-R.
Training models on strategic datasplits (S3 ) provides the best results overall, and is even better than the ensemble of all models (S5 ). The strategic datasplits improved the ensemble model S2 by 0.4 and 2.3 F1 points for task 1 and 2. Note that this submission unintentionally contained the Spanish general-domain XLMR models with transfer learning in the ensemble. These were trained without our strategic datasplits. The corrected results are marked with "*".
The overall best model for task 1 is the ensemble of general-domain XLM-R trained using the strategic datasplits with an F1-score of 83.2. This model was not submitted as a run to the shared task, but shows the importance of the language-adaptive pretraining and the usefulness of strategic datasplits. ensemble of all models (S5)
In this paper, we described our submissions for the rst two tasks of the MEDDOPROF competition. By utilizing domain- and language-adaptive pretraining, strategic datasplits and ensembling methods, we were able to improve already high-performing transformer-based models by up to 5.3 F1 points and achieved competitive results in the competition as neither language nor domain experts. Future work will include the exploration of di erent clinical corpora with our newly trained Spanish XLM-R models.
The authors would like to thank the anonymous reviewer for the helpful comments and the text mining group at the barcelona supercomputing center for the smooth organization of MEDDOPROF.