In this paper, we present our approach for performing relation extraction on clinical texts in the context of the CLinkaRT task at EVALITA 2023. Our system ranked first in this task with an F1-score of 62.99, outperforming most other submissions by a significant margin, with an increase of 6.5% over the second best score of 59.16, while also improving over the mBERT baseline of 62.83. We pursue a simple yet unexplored method to determine sentence level relations in text by relying on Named Entity Recognition models to identify the components of a relation. We apply this method to link laboratory results to their appropriate events in medical reports.
biomedical domain from digital repositories like PubMed
Central has led to the development of highly specialized
resources and language models [
The typical approach for downstream tasks in these
languages is to resort to multilingual models, such as
mBERT [
In the context of creating better systems for Italian, the
CLinkaRT shared task [
After determining all targets in a sentence, we trans- predict just target entities, while the second NER model form the training examples to incorporate target loca- is trained solely with labels for source entities. The contions directly in text by adding a special marker token sequence is that our models have a lower number of [T] before each target token, which should help the sec- possible labels, determined by fewer IOB2 tags, therefore ond NER model find relevant source entities. This is a improving prediction performance. Each model has 3 viable strategy to denote one-to-one, one-to-many and tags: beggining, inside, outside. While the NER model many-to-one relations between sources and targets, thus used to predict targets only denotes the target head, we efectively eliminating the need of a relation classifier still need “inside” tags due to sub-word splitting required model. The target indicates just the syntactic head, so we by transformer models. Contrast this with a traditional do not add an end marker because it might hinder the sec- pipeline that would have a NER model with 5 tags (2 ond NER model’s ability to properly learn representing beginning tags, 2 inside tags, 1 outside tag) followed by adequate targets. a relation classifier model.
All relation types (one-to-one, one-to-many,
many-toone) are handled in a uniform manner. For every target in 2.3. Data augmentation
a sentence, we generate one sample with a single target
marker [T]. In this regard, there is no diference between The CLinkaRT training dataset consists of 83 Italian
doca source with multiple targets and several one-to-one uments with 658 annotated relations. Due to the
limrelations. For one target with many sources, only a single ited number of examples, we decide to augment the
iniexample is created. This way, we augment our training tial dataset with contextual word embeddings using the
data for the second NER model. nlpaug library [
As an example with two one-to-one relations, the sen- words with other similar words in the embedding space, tence “La creatinina oscillava tra 1,5–2 mg/dL con protein- except for labeled tokens, since there is a risk of injecting uria sempre < 1 g die” has two targets, “creatinina” and noisy labels. “proteinuria”. The samples for our second NER model We preserve the annotated entities and the target will be: marker [T] in the augmented examples, ignoring sentences with 9 words or less. Samples that are not identical (1) “La [T] creatinina oscillava tra 1,5–2 mg/dL con in terms of word count are discarded because the original proteinuria sempre < 1 g die” with only “1,5–2 labels would be misplaced.
mg/dL” labeled as source Given that one of our main concern is finding labora(2) “La creatinina oscillava tra 1,5–2 mg/dL con [T] tory tests and measurements, many of these entities are proteinuria sempre < 1 g die” with only “< 1 g numeric values. To further our data augmentation, we die” labeled as source introduce tiny changes of ±2 for decimal values (age, year or quantities with a higher tolerance) and ±0.1 for real
In the following example, there is one source linked values (tests or percentages). In most cases, this process to three targets: “Gli esami colturali (germi comuni, BK) should not significantly alter the labeling. risultavano negativi.”. The source is “negativi”, while the The training set has a much greater number of negatargets are “esami”, “germi” and “BK”. Three sentences tive samples (examples without relations) than positive will be added, all with a single source to be predicted samples. We augment each example one or more times, (“negativi”): with positive instances denoted by in_multiplier and (3) “Gli [T] esami colturali (germi comuni, BK) risul- negative instances by out_multiplier. We use the multavano negativi.” tipliers shown in Table 1, where the NER-tgt model predicts targets and the NER-src model predicts sources. The (4) “Gli esami colturali ([T] germi comuni, BK) risul- second model requires fewer auxiliary examples because tavano negativi.” the preprocessing step created additional samples for Multiplier type in_multiplier out_multiplier
NER-tgt
NER-src sentences with more target entities.
The bottleneck of this augmentation process is the li- for selecting appropriate values for some of the
parambrary call that executes the transformation. Considering eters regarding training and augmentation. Although
that the operation runs on GPU, it should be natural to the models are trained at the sentence level, this split
attempt to speed up this step by augmenting several ex- is by document id so we do not overestimate the model
amples in parallel. While the nlpaug library has an API performance on unseen examples.
that allows augmenting multiple sentences at once and The main results for relation extraction in the
at first it appeared to work on a few samples in the train CLinkaRT task are displayed in Table 2. Our team obtains
set, a significant number of augmented examples con- the first place across all teams, with an F1-score of 62.99,
structed by nlpaug turned out to be empty sentences due an improvement of 6.5% in F1-score over the second best
to limitations or issues of this library. The batch imple- competing team of 59.16 and a slight increase over the
mentation required a bit of efort due to the need to apply mBERT baseline with a score of 62.83. We also achieve
diferent multipliers. Since this attempted optimization the highest recall of 60.62 among other participants, with
did not succeed, we resumed augmenting examples one the second best score of 50.65, while the mBERT baseline
by one. has a recall of 64.37. We improve the baseline precision
of 61.37 with a 6.8% increase, reaching a score of 65.55.
2.4. Model training and inference We report the performance of our system on the
validation set averaged across 10 folds together with the
We implement our NER models as standard token clas- oficial results. We used cross-validation to carry out
sifiers with the help of HuggingFace Transformers li- parameter and model selection. Besides these 10%
rebrary [
All of our training experiments are carried out by phenomenon is the small size of the model validation set, mostly preserving default parameters: AdamW optimizer with too few samples to properly adjust the parameters with 5 −5 learning rate with linear decay and no warmup when training. Another related explanation is given by steps, 1 −2 weight decay, 8 samples per batch trained for the high variation between some of the folds, with half 4 epochs, with 10% examples held out for validation. Both of the folds obtaining F1-scores over 82%, while the other models are trained independently using gold labels, with folds consistently scored lower, between 73% and 77% the second NER model (NER-src) receiving the target F1-score. Even so, it might simply be the case that the marker tokens [T] from these gold annotations. test set is intentionally constructed with novel situations
In inference mode, the models are asked to output to determine the performance on unseen data more
acsource and target ofsets with respect to the original raw curately, which would justify the gap between test and
text. For each sentence converted into an example, we validation.
store the ofset of the first token. Since HuggingFace Outside the evaluation window, we repeated the
inDatasets library employs a separate tokenization, we ference process a second time on the test set keeping
align the transformed concatenated sentences with the the same parameters and achieved 64.09 F1-score,
showinitial full texts by using spaCy [
We conduct our experiments by creating a test set from
the training set with a 90/10 split in order to simulate
the final test set, switching to 10-fold cross-validation
points. The results of the multilingual model are
averaged only over the first 5 folds (out of 10 folds) because
the training process takes much longer. We believe the
increased training time is not justified given that a much
smaller model can achieve better results. For this reason,
we do not conduct additional experiments with
multilingual transformers. Concerning the BioBIT model, it
ofers slightly worse results than the MedBIT-R3-plus
counterpart, as noted likewise in the original paper [
The results are summarized in Table 3, using data augmentation for all variants except where noted otherwise.
Due to time constraints, we did not run additional experiments with these models.
In addition to typical false positives and false negatives, we observe types of errors that show the system is on the right track, but fails to output the exact ofsets in the reference file.
There are a couple of incomplete entity spans: MedBIT-R3-plus MedBIT-R3-plus (no aug) Italian BERT DistilBERT BioBIT
the design choices of our implementation and conduct an error analysis.
The data augmentation process has three main parameters: the minimum number of words that should be replaced in a sentence, min_aug, and the two multipli- (6) The true source is “pari 0 inferiori a 1.5 mg/dl” ers for positive and negative examples defined earlier as and the predicted source is “inferiori a 1.5 mg/dl”. in_multiplier and out_multiplier. We pick values Other similar examples: true source is “fino a 12.8 between 3 and 6 for min_aug, based on the number of mg/dL”, predicted source is “12.8 mg/dL”; true failed replacements, fixing the value at 4 words. The source is “punte di circa 1200 pg/mL”, predicted reasoning behind this decision is that the augmentation source is “circa 1200 pg/mL”. library is sometimes unable to adequately generate valid examples due to misplaced or missing words, so the gold (7) The true target is “antitrombina”, while the prelabels cannot be applied, in turn leading to fewer exam- dicted target is “anti” ples in the train set. The first situation appears due to modifying compara
The multipliers are selected by cross-validation, stop- tives not being present or being scarcely existent in the ping early in case of unsatisfactory results on the first training data, which one could solve with additional exfolds. For out_multiplier, we vary this parameter amples or through careful augmentation. The second between 0 and 2 for both NER models, while for in_- issue seems to be a defect on our side that can be handled multiplier we use values in the range 1–5 for NER-tgt in post-processing by inspecting the initial tokenization. and 1–4 for NER-src. Our experiments confirm that aug- Another common mistake is the prediction of one rementation is also needed for negative samples. This step lation instead of two (or vice versa) in the case of interhas a significant impact in our system, boosting the score vals, which we explain by ambiguities in the training set. on the validation set with over 20 percentage points in For example, our system outputs “1.9 – 2.5 mg/dl” linked F1-score. As it would be expected, adding too many ex- with “creatininemia”, but there are two expected relations: amples by using larger multipliers eventually induces “1.9” linked with “creatininemia” and “2.5 mg/dl” linked overfit. with “creatininemia”. Conversely, our system detects two
The main drawback of augmentation is the slow data relations, “sostanzialmente” linked with “obiettività” and generation. As we mentioned earlier, nlpaug runs se- “nei limiti di norma” linked with “obiettività”, while there quentially, so we had to make educated guesses of what is only one true relation, “sostanzialmente nei limiti di combinations of parameters to include in our experi- norma” linked with “obiettività”. ments. Lastly, a challenging facet of this task is the presence
For most of our experiments, we rely on the model of reference values for some tests, which are picked up
called MedBIT-R3-plus [
CLinkaRT task [
We present a straightforward strategy to extract sentence-level relations based on two plain NER models, illustrating the learning capabilities of transformer networks to solve challenging tasks with the help of special tokens. We intend to further explore this direction since NER models are well established and usually require fewer resources than alternative relation extraction (RE) models. The presented method is not limited to the clinical domain and it can be easily applied in other contexts, with the added benefit of shorter development cycles. In certain domains and applications, the overhead of a generic RE model may be unjustified if the relations in question are simple enough.
Data augmentation is a valuable, but underused technique in natural language processing contexts. We look forward to enhancing the augmentation procedure to account for in-domain information. Another area we believe to be worth pursuing is the handling of numeric values and ranges, either by finding a way to inject fuzzy intervals or by masking these values altogether, therefore simplifying the initial problem.
Our system implementation is available at https://gitl ab.com/marius.micluta-campeanu/testlink-clinkart-2023 to encourage an open environment for future work.