In this paper, we introduce an intuitive solution to extract relations using solely Named Entity Recognition (NER) models. Our approach achieves the best scores in the TESTLINK task among all other participants by a large margin, with 68.38% F1-score for Spanish and 72.65% F1-score for Basque. General purpose relation extraction methods typically require a classifier to identify the relationship type or leverage generative models. When we are only interested in extracting a single relation type, such approaches might be overly complex or costly. We show that simple methods are capable of performing relation extraction by using NER models, data augmentation and basic text processing, without an additional relation classifier.
Medical and clinical texts have been widely studied for information extraction purposes on
English corpora [
The shared TESTLINK task [
The TESTLINK task is based on a subset of the E3C Corpus [
In this paper, we present our team’s contribution to the TESTLINK task, introducing a straightforward methodology for extracting basic relations. First, we transform the input into an adequate internal representation and apply data augmentation. Then, we train one NER model to predict targets, followed by training a second NER model to predict sources with the aid of a special token for targets. Finally, we determine the final relations and apply minor corrections in a post-processing step. This pipeline is summarized in Figure 1.
The idea of using special tokens in discriminative transformer models to aid learning certain
patterns dates to at least the original BERT paper [
We use two NER models to predict sources and targets at the sentence level, then link the predictions via post-processing. The first NER model called NER-target predicts all targets in a sentence. Sentences are determined using the provided tokenizations. For each target identified by the first model, we create a new example containing a single labeled target using a special marker token [T].
Example: albúmina sérica de 2,5 g/dl y proteinuria de 6 g/24 h. NER-target predicts the targets albúmina and proteinuria.
We generate two examples for our next NER model: • [T] albúmina sérica de 2,5 g/dl y proteinuria de 6 g/24 h • albúmina sérica de 2,5 g/dl y [T] proteinuria de 6 g/24 h
The second model named NER-source is trained on examples containing target markers, with labels only for source tokens. Since the added marker is a simple pattern from the model’s perspective, the model is capable of inferring whether a sentence contains relations based on the presence of a target marker.
After sources are predicted for each sentence, we align the inferred sources and targets with the initial raw text in order to output appropriate text spans. Predictions are ordered by the start mention of sources since we train and evaluate at the sentence level.
This simple approach turns out to be very efective for determining relations of a single type. In the following subsections, we provide additional details regarding our system.
We parse the provided annotations into an internal representation, then we serialize the results as JSON for easier dataset loading by the training script. Given that most examples have a lot more tokens than the limit of transformer models, we consider each sentence as an example. To ease post-processing, we serialize the start ofset of every sentence for all examples, along with the example id and the sentence number. For the NER-source model, we also store the target span if that example contains a relation.
First, we create IOB2 tags (inside, outside, beginning) for target entities in order to train a NER model to predict targets from raw text. We choose to learn predicting only the targets at this stage because they consist of a single labeled token. While typical relation extraction methods use just one NER model to identify both sources and targets as an initial step, our approach does not require labeling sources because they would be discarded anyway in a later stage. Moreover, having a lower number of possible labels helps the model achieve better results.
Since targets are single tokens in the training set, NER-target only needs to predict two labels: “B-T” (begin target) and “O” (other). An example for NER-target has the following labeling: (1) [O]Resto [O]de [B-T]parámetros [O]dentro [O]de [O]la [O]normalidad [O].
The relation extraction part of the system is divided into two pipelines: one for training and one for inference. This model receives target spans for each sentence from the train set or from the previous model. When training, source spans are also available. All relations are represented as source-target tuples of spans (ranges). We convert relations to IOB2 tags containing both sources and targets. Before each target, we insert a special marker token [T]. Example 1 is relabeled for the NER-source model in a similar fashion: (2) [O]Resto [O]de [O][T] [O]parámetros [B-S]dentro [I-S]de [I-S]la [I-S]normalidad [O].
In order to distinguish between multiple relations in the same sentence, source and target tokens are labeled with their respective targets, ignoring tags from other relations. If multiple sources reference one target, modeling a many-to-one relation, we do not want to add the same target several times to our set of tags, because it would be redundant. In this case, we only add source tokens. However, if we have a one-to many relation between sources and targets, the same source token will be added once for each corresponding target (see example 3).
This strategy allows us to unambiguously determine all separate relationships. It can be seen as a sliding window approach, asking the NER-source model to focus only on one target at a time. Moreover, creating additional examples for every target in sentences with multiple targets acts as a form of data augmentation.
(3) The example “estudios negativos para bacilo” has a one-to-many relation with the source “negativos” and the targets “estudios” and “bacilo”.
The NER-source model will receive two samples: a. [O][T] [O]estudios [B-S]negativos [O]para [O]bacilo b. [O]estudios [B-S]negativos [O]para [O][T] [O]bacilo
For the second NER model, we keep IOB2 tags only for source tokens and convert target IOB2 tags to O (outside) tags. We choose this strategy because we already have the target embedded in every example using the special marker token [T]. Similar to the previous NER model, having fewer labels increases model performance.
The E3C subset corpus annotated for this task contains 597 relations for Spanish and 1291
relations for Basque, excluding examples without relationships. The small number of labeled
sentences appears to be a limiting factor for our system. With the help of nlpaug library [
We perform data augmentation using contextual word embeddings by replacing random words and aligning the result with the original sentence in order to assign proper labels. While it is possible to perform word insertion or deletion in this scenario, label assignment would be more problematic due to larger ofsets. The contextual word embeddings for augmentation are provided by pretrained models used for training.
Sources, targets and the target marker are copied from the initial sentence without changes because we cannot aford noisy labels in this task. For numeric values that are not part of a relation, we perform small perturbations of ±2 for decimal numbers (usually age or large quantities) and ±0.1 for real numbers (typically measurements). These numerical perturbations are applied together with the other changes. We chose not to create separate examples for word substitutions and numeric adjustments because we risk to add too many similar samples which would lead to overfitting.
In theory, these alterations could also be applied on labeled tokens since they should not change the relation or sentence meaning. They were left out in the final implementation due to corner cases involving intervals and sign changes. We intend to explore this area in future work because we believe it might help models generalize better in extracting this kind of information.
The special marker token is not included in this process and is added back afterwards. If a sentence contains less than 10 words, it is skipped. All augmented examples with a diferent word count than the original sentence are dropped since we cannot reliably apply any previous labels.
We experiment with values between 3 and 6 for the minimum word replacements. We choose 6 substitutions for Spanish and 4 for Basque to minimize the number of discarded samples. We repeat the augmentation process with diferent multipliers for positive examples containing relations (in_multiplier) and negative examples without any labels (out_multiplier). Based on our experiments, we pick the values in Table 1 for these multipliers. We need to augment with fewer source tokens used in relation extraction due to the fact that there are already duplicate examples for sentences with several targets.
Our first named entity recognition model extracts target tokens from raw text. We use the
popular HuggingFace Transformers library [
All models are trained for 4 epochs with default parameters: AdamW optimizer with a learning rate of 5 −5 using linear decay and no warmup, weight decay of 1 −2, batch size of 8 samples and 10% examples reserved for validation.
The inference step demands more post-processing work since transformer-based models provided by HuggingFace employ a diferent tokenization strategy than the one available in the TESTLINK dataset. We use spaCy [13] to align the examples prepared by HuggingFace Datasets with the initial texts. For each predicted target, we create one relation with a dummy source in order to apply the logic from our preprocessing step. The alignments are needed to output proper ofsets for the target token.
The previous model is tasked to determine targets. Given a specific target in a sentence, the relation extraction model learns to predict the relevant source tokens. We process the input in a manner that allows this step to also become a named entity recognition problem instead of a relation classification problem.
When training, we take advantage of gold labels for both sources and targets, with the latter being converted to the special marker [T] as described earlier in section 2.1. Due to the similarities between the two models, we apply the same training procedure presented in the previous section with the same pretrained models.
At inference, this model expects sentences to have an embedded target marker [T] for examples that contain a relation, with the possibility to supply this marker through either gold labels or inference of the first model. Like before, we align the processed text required by this model with the original text. If the predicted source appears after its corresponding target, we also have to account for the additional [T] marker and subtract its length from the predicted text span. Finally, we perform minor corrections for of-by-one errors due to alignment mismatches and then sort relations by the start mention of the first source token in each relation.
The results of our system along with provided baselines are shown in Table 2 and Table 3. In the Spanish subtask, we obtain an F1-score of 68.38, with a 13% improvement over the mBERT baseline with a score of 60.57. For the Basque subtask, our method achieves an F1-score of 72.65, while the mBERT baseline has a score of 78.56. Our results are significantly better than the other competing systems. For completeness, we also include the results obtained locally on a development set extracted from the provided training set.
Our Spanish submission has an underlying model pretrained on medical and clinical texts, while the Basque submission is pretrained on generic texts such as news articles. One explanation for the better results obtained on Basque is that we have more than twice as many annotated relations for the downstream task, which compensates the lack of a dedicated pretrained model.
We perform parameter selection for the best performing model for each language using 10-fold cross-validation. The models training employ a validation of their own, meaning that they are trained on 81% of the initial train set: 10% examples are held out for cross-validation and 10% of the remaining 90% are reserved for validation during training.
Throughout our experiments, we noticed a significant gain in performance after adding more train samples. Given the small number of examples in the training set, even after data augmentation, we try to use as many samples as possible. Therefore, we train the final system on the entire train set, leaving only 10% for validation used in training. This approach allows us to fully utilize the available examples, although it might also lead to some overfit since we do not perform more splits.
Regarding the types of errors in our system, we notice other situations besides obvious false positives (usually acronyms, quantities or numbers) and false negatives. The training data contains a few ambiguous instances where consecutive tokens form either one relationship or more. Our model seems to prefer predicting more relations in these scenarios. Example of a relation split:
(4) There is one true relation: 29,5 cm x 27,5 cm x 16 cm masa
The system predicts two relations: 29,5 cm x 27,5 cm masa and 16 cm masa The reverse situation is also present, as seen in the following example: (5) There are two true relations: > 2 adenopatías and < 5cm adenopatías
There is only one predicted relation: > 2 y <5 cm adenopatías
Sometimes, the text span stops in the middle of a word, which can be fixed relatively easy in post-processing by leveraging the existing tokenization. Examples of incomplete predictions are shown below: (6) True relation: 4340 g/l IgM, predicted relation: 4340 g IgM
True target: alfafetoproteína, predicted target: alfafe
In this paper, we presented an alternative method for relation extraction relying solely on NER models. We have successfully applied this method in the TESTLINK shared task at IberLEF 2023 for linking laboratory results to their respective tests and measurements.
Our team achieved the first place across all competing teams in both languages. On the Spanish dataset, we surpass the mBERT baseline score by 8 percentage points regarding F1-score, while we come close to the mBERT baseline on Basque with less than 6 percentage points below in F1-score, proving the efectiveness of our approach. One of the main diferences between our solution and the provided baseline is that we used a monolingual model. We have further applied our method in the CLinkaRT twin task [14] where we also reached the first place and performed a few additional experiments [15].
We provide the source code of our system to promote the design and use of simpler, lightweight models (available at https://gitlab.com/marius.micluta-campeanu/testlink-clinkart-2023). For future work, we intend to systematically analyze the system errors more thoroughly and explore this method in a multilingual environment or with the help of machine translation. Another unexplored path is finding the minimum amount of context necessary for better results, since our model relied on sentence-level predictions. Finally, we are interested in further development of medical applications based on extracted relations. your text representation models some love: the case for Basque, in: Proceedings of the Twelfth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 4781–4788. URL: https://aclanthology.org/2020. lrec-1.588. [13] M. Honnibal, I. Montani, S. Van Landeghem, A. Boyd, spaCy: Industrial-strength Natural
Language Processing in Python (2020). doi:10.5281/zenodo.7715077. [14] B. Altuna, G. Karunakaran, A. Lavelli, B. Magnini, M. Speranza, R. Zanoli, CLinkaRT task overview: building links between clinical tests and their results (2023). [15] M. Miclut,a-Câmpeanu, L. P. Dinu, Simple Ideas@CLinkaRT: LeaNER and MeaNER Relation Extraction (2023). (to appear).