This document describes the system submitted by TALP team for IberLEF 2021's MEDDOPROF Shared Task. The joint occupation mention identi cation and family relation classi cation model is composed of a pre-trained DistilBERT architecture followed by a Bidirectional LSTM layer. Occupation normalisation uses Sentence-BERT pre-trained for Semantic Text Similarity (STS) to map the ESCO and SNOMED-CT categories as well as the mentions of occupations from the documents to a vectorial space. K-nearest neighbours is then used to nd the most likely category assignments.
We present the work carried out by the TALP Team in the context of IberLEF
2021's MEDDOPROF Shared Task [
Due to the provided training dataset being relatively small for some
occupation classes such as activities and also fairly unbalanced compared to professions,
we also use a simple data augmentation algorithm. Similarly to the rare word
substitution approach from Fadaee et Al. [
As already mentioned in Section 1, our system is comprised of two independent components. In this section, we give an in-depth look at them and describe the data augmentation approach that we followed.
We reserved a 10% of the documents from the shared tasks' training set for validation and the remaining 90% for training, using the same split for all three sub-tasks. The batch-wise evaluation for Tracks 1 and 2 was performed at the level of tokens rather than entities, since it is simpler and more e cient to compute in GPU. A full entity-wise evaluation was performed after the training process had ended. 2.1
Occupation Identi cation and Family Relation Classi cation Even though the Shared Task splits the occupation mention identi cation and family relation classi cation into two di erent tracks, we approached these two tasks as a single joint task. We considered two alternatives: either using a single output whose set of labels is the result from the cross-concatenation of the occupation classes and the family relation classes or using two independent outputs for occupations and family relations. We used binary cross-entropy as the loss function for all outputs. In the two output case, the combined loss was computed as an average of the two independent output losses. After some preliminary evaluations, we discarded the single output model, since the increased number of classes led to a degradation of around 10% in token-wise F1 score. Output Encoding As for output encoding, we considered In-Out (IO) and Begin-In-Out (BIO) encodings. We didn't see any noticeable improvements in the two-output model when using BIO, so we opted for using IO encoding. Similarly, and due to the fact that BERT uses word-piece encodings, that is, a sub-word representation of the tokens; very few input sequences were only one word-piece in length, so we also discarded other widely-used output encodings such as Begin-In-Out-Unitary (BIOU).
Model Structure Our NERC model structure is shown in Figure 1. Input text
is encoded using word-piece embeddings and then fed to the input transformer
of a DistilBERT model [
We initialise the weights of the DistilBERT layers from a pre-trained
generalpurpose multi-lingual model from Huggingface (distilbert-base-multilingual-cased
pre-trained model, a distilled version of bert-base-multilingual-cased with 6
layers, 768 dimensions per layer, 12 attention heads and 134M parameters.). We
opted for DistilBERT instead of the original BERT [
Training and Fine-tuning As for the training process, we explored several strategies that are worth mentioning.
First of all, we added con guration options to tweak the balance between positive, that is, sequences containing at least one valid entity; and negative, sequences with no entities. Note that even though all documents contain entities, due to computational limitations, we split the documents into overlapped sequences of up to 128 tokens. We saw that no matter the learning rate, training solely with positive examples led to very low precision while training with the raw documents led to low recall especially for under-represented classes. Our nal training strategy was to interleave epochs limited to positive examples with full epochs while decreasing the learning rate each epoch.
As it was mentioned in Section 2.1, we initialise the weights from a pretrained DistilBERT model. We also explored several alternatives for ne-tuning these weights to our task. More in particular, we froze a number of these layers during all or some training epochs. In our nal submission, we included two options: either no ne-tuning or full ne-tuning of all DistilBERT weights. Data Augmentation The labels in the training corpus for Track 1 of the MEDDOPROF Shared Task are fairly unbalanced: 2528 entities for PROFESION, 1011 for SITUACION LABORAL status and just 122 for ACTIVIDAD. In order to improve F1 score for the under-represented class, we decided to double that amount by taking examples in the training set and replacing their entities by other activities that are less common in the training set. In order not to add grammatically or semantically incorrect examples such as replacing "Aunque acud a con frecuencia al gimnasio, y practicaba fundamentalmente pesas." by "Aunque tocar la guitarra, y practicaba fundamentalmente pesas" (real example from the dataset), we used a general purpose BERT Language Model to compute a likelihood score so as to rank the synthetic examples (The default bert-base-multilingual-cased pre-trained model.). 2.2
Occupation Normalization Our occupation normalization system is based on Semantic Text Similarity (STS) between sentences. The general idea is to put every occupation in a generic context and embed the whole sentence using a language model pre-trained for STS. These sentence embeddings are then associated with their respective ESCO or SNOMED-CT identi er. The inference is then performed by mapping the candidate occupation to the aforementioned vectorial space and then assigning the closest pre-mapped point's identi er. A visual representation of the inference process is shown in Figure 2.
We used both the provided meddoprof valid codes.tsv le and examples from the gold-standard training corpus as pre-mapped points in the vectorial space. We also added additional points computed as the centroid of subsets of occupation descriptions that were mapped to the same identi er. For example, "operador de centrifugadora", "operaria de centrifugado" and "responsable de centrifugacion" are all associated to the identi er 8160.14 so in addition to all three points, their geometric center is also added.
We de ned two generic contexts in which the occupation descriptions are introduced: "Trabaja de OCUPACION." y "Se dedica a OCUPACION.", where OCUPACION is replaced by the occupation's description.
Among all the publicly available pre-trained STS models, we opted for
DistilUSE (We used the pre-trained distiluse-base-multilingual-cased-v2
multilingual model) [
We submitted three models for evaluation at the MEDDOPROF Shared Task: default, with full ne-tuning of DistilBERT's weights but no data augmentation; extended, with the same parameters as the former but adding examples from data augmentation; and no ne tune, with data augmentation but no ne-tuning. We used the same occupation normalization system for all three submissions.
In our tests with the training and validation split described in Section 2, the best performing model of the three was extended, which achieved 0:72 in F1 score for Track 1, compared to default 's 0:705. This advantage funnels to Tracks 2 and 3. The results in test corpus show a di erent picture though: default achieves 0:698, outperforming extended by 0:027 (0:671).
If we look at precision and recall, we can see that extended balances precision and recall (0:671 and 0:671 respectively) whereas default favours precision over performance (0:761 and 0:645). This was to be expected, as adding synthetic examples often leads to the inclusion of noisy examples.
Sadly, at the time of writing this system description paper, the competition's nal results have not yet been made public and no competitive comparisons can be made. We will then explore those results at a later date. 4
This paper describes the participation of the TALP team in IberLEF 2021's MEDDOPROF Shared Task. Our system makes extensive use of BERT-like language models for all three tracks of the challenge. We present e ective strategies to deal with the dataset's data imbalance either with training scheduling or data augmentation, although this might not be re ected in the nal score as the Shared Task's evaluation framework does not provide independent evaluation of the classes. This data augmentation strategy not only bene ts under-represented occupation classes but also makes our system more balanced with regards to precision and recall.
It has not been possible to do a qualitative analysis of our system compared to the rest of the participants, nor an in-depth error analysis of our system due to the tight schedule. However, we strongly believe that the simple approach presented in this document has clear potential and could be improved and extended upon for the next iterations of this Shared Task. 5
This contribution has been funded by the Spanish Ministry of Science and Innovation (MICINN) (PID2019-106942RB-C33).