This paper presents the proposed solution by our team, Gut-Instincts, for the GutBrainIE task, which introduces a named-entity recognition (NER) subtask and three relation extraction (RE) subtasks on biomedical articles related to the gut-brain axis. To address the domain-specific terminology involved in the tasks, we rely on biomedical pretrained transformer-based models. For NER, we extend these with three diferent classification heads: (1) a dense layer, (2) a dense layer followed by a conditional random field (CRF), or (3) a bidirectional long short-term memory layer followed by a CRF. For RE, we introduce negative samples and experiment with diferent ratios between positive and negative samples. For all subtasks, we use model ensembling to reduce variability and improve robustness. Furthermore, since the provided dataset is of diferent quality levels, we use weighted training that enables the models to utilize all available data, while ensuring that high-quality data has a stronger influence during optimization. Our experimental results suggest that a large ratio of negative to positive samples, model ensembling, and weighted training improve performance in the NER and RE subtasks. In the GutBrainIE task, we placed second in the NER subtask (6.1) with an F1micro score of 0.8382, and first place in all three RE subtasks 6.2.1, 6.2.2, and 6.2.3 with F1micro scores of 0.6864, 0.6866, and 0.4635, respectively.
In recent years, there has been a surge of research on the gut-brain axis, which refers to the intricate
relationship between the gut and the brain. Once thought to afect only digestion, the gut microbiome
is now recognized as a key component in neurological and mental health conditions [
As research in this domain intensifies, the volume of biomedical literature on the gut-brain axis has
grown rapidly, presenting new challenges for researchers to keep up with the latest developments [
To support research on the gut-brain axis, the GutBrainIE task [
CEUR
ceur-ws.org • Subtask 6.1 - Named-entity recognition (NER): Classify specific text spans, referred to as entities, according to 13 predefined labels. • Subtask 6.2 - Relation extraction (RE): Classify relations between entities through three distinct subtasks: – Subtask 6.2.1 - Binary tag-based RE (BT-RE): Identify which entity labels are related. – Subtask 6.2.2 - Ternary tag-based RE (TT-RE): Identify which entity labels are related and classify each relation according to 17 predefined labels. – Subtask 6.2.3 - Ternary mention-based RE (TM-RE): Identify which of the entities are involved in relations and classify each relation according to the 17 predefined labels.
When developing information extraction systems, a major cost is annotating the training dataset.
According to Lawson et al. [
Due to the high costs of annotating data and the expertise required to do so in the biomedical domain, it is often necessary to balance annotation quality with scalability. As a result, datasets of varying annotation quality are commonly created to support development. In this task, four datasets of diferent quality levels are provided: • Platinum dataset: 111 articles with expert-curated annotations reviewed by external biomedical specialists. • Gold dataset: 208 articles with expert-curated annotations. • Silver dataset: 499 articles with annotations generated by trained students under expert supervision.
• Bronze dataset: 749 articles with automatically generated annotations, using fine-tuned models. Additionally, the task also includes held-out development and test datasets, each consisting of 40 articles of the gold and platinum qualities.
As a result, the GutBrainIE task presents a wide range of obstacles. Firstly, biomedical articles
are rich in specialized vocabulary, laden with ambiguous abbreviations, and often exhibit irregular
grammar [
This paper presents the proposed solution by our team, Gut-Instincts, for addressing the tasks of the
GutBrainIE task. Our approach is an information extraction pipeline that uses ensembles of specialized
models, built by extending well-known language models that have been pretrained on biomedical
corpora [
Our experiments show that the proposed solution performs well across all four tasks. This is further supported by the oficial results in the GutBrainIE task, where our best NER ensemble resulted in second place for the NER subtask (6.1) with an F1micro score of 0.8382 and our best RE ensembles achieved ifrst place in the RE subtasks 6.2.1, 6.2.2, and 6.2.3 with F1micro scores of 0.6864, 0.6866, and 0.4635, respectively.
This paper is structured as follows: In Section 2, we introduce related work on NER and RE in the biomedical domain. In Section 3, we explore and analyze the training datasets to uncover insights that are key in the development of our approach. In Section 4, we describe our approach to ensembling specialized models as well as the approach to training with a weighting scheme. In Section 5, we present 2https://github.com/P10-Natural-Language-Processing/Gut-Instincts the experimental evaluation and key results for the final selection of the configurations of the models. Finally, in Section 6, we conclude the paper, presenting takeaways and open challenges.
Since the introduction of BERT [
After the success of BioBERT, the pretraining of transformer-based models on biomedical corpora
has been further explored [
NER and RE are core tasks of biomedical information extraction [
In state-of-the-art language models, NER is treated as a token-level classification task, typically
accomplished by adding a classification layer on top of the output of the language model to map each
token to an entity type or non-entity class [
Regarding the RE task, a simple approach is replacing the entities in a relation with special tokens
representing the entity label and inputting the concatenated embeddings of these tokens to a classifier
layer [
Beyond architectural improvements, there are also other techniques for improving model performance.
Bölücü et al. [
Another technique is to ensemble models to improve performance [
This section presents an exploratory data analysis that expands upon the initial analysis conducted
by the organizers of the GutBrainIE task [
We analyze the article lengths across all dataset quality levels by tokenizing the combined title and
abstract of each article using WordPiece [
200 400 600 800 200 400 600
800
this threshold, with a few reaching over 400 relations. Such outliers can introduce noisy patterns that may negatively impact the ability of models to learn how entities relate to one another, potentially reducing overall performance.
The bronze dataset difers from the other datasets since the annotations are generated by a model rather than human annotators. As a result, it contains a higher level of noise. For example, the entity “,” is labeled as Biomedical Technique, and the entity “, and progressive” is labeled as DDF. These examples suggest that the model struggles with correctly annotating the articles. Furthermore, some articles from the bronze dataset contain very few entities and relations, with 21 articles having no relations at all. Including these in the training could introduce noise and negatively impact model performance. Despite these issues, we want to exploit the bronze dataset, as it makes up a significant amount of the total amount of data.
The silver dataset also includes some annotation issues besides the relation outliers previously mentioned. It includes instances where entity text spans are incorrectly annotated in singular form when the correct form should be plural. This does not adhere to the annotation guidelines and introduces inconsistencies.
We also observe that in all four datasets, there are HTML tags in many of the articles, including in the annotated text spans. We consider the HTML tags problematic because they can lead to an increased amount of noise in the training data since they are not used consistently and do not provide relevant context. During tokenization, the HTML tags will result in the generation of spurious tokens that do not represent meaningful content, leading to the model learning irrelevant patterns, which ultimately leads to a degradation in performance.
This section presents our proposed solution to the GutBrainIE challenge, detailing the information extraction pipeline we developed. An overview of the pipeline is illustrated in Figure 3. It takes as input the titles and abstracts of biomedical articles about the gut-brain interplay. To address the NER subtask (6.1), the pipeline processes these articles using an NER ensemble that combines the predictions of multiple NER systems and outputs the final set of predicted entities. For the three RE subtasks (6.2.1, 6.2.2, and 6.2.3), the predicted entities along with the titles and abstracts of the original articles are passed to an RE ensemble, which similarly combines the predictions of multiple RE systems to output the final set of predicted relations. The information extraction pipeline outputs the predicted relations in a format corresponding to the specific RE task being addressed.
NER system 1 NER system 2 NER system n NER ensemble
RE system 1 RE system 2 RE system n
RE ensemble Entities
For both the NER and RE tasks, we use pretrained models. The benefits of using pretrained models are the reduced computational cost from not training from scratch and the improved performance through transfer learning. We considered two selection criteria. Firstly, we considered models that are pretrained on biomedical corpora. This ensures that the models are better equipped to handle specific terminology and patterns encountered in the GutBrainIE task. Secondly, all of the selected models are uncased models, since the entities to be predicted in the NER task do not have a consistent casing, and therefore, including casing in the input representation would not provide a significant benefit and could introduce unnecessary variability.
Based on the criteria outlined above, we selected seven transformer-based language models:
• BioLinkBERT-base [
PubMed and full-text articles from PubMedCentral5.
• BiomedBERT-base-uncased-abstract [
We also considered other pretrained models: BioM-ALBERT-xxlarge [
The first task in the information extraction pipeline is NER, which we approach as a token classification
problem using the beginning-inside-outside (BIO) tagging scheme [
In the NER ensemble shown in the information extraction pipeline overview in Figure 3, there are multiple NER systems. Each NER system consists of multiple components, which is illustrated in the NER system architecture in Figure 4. An NER system takes the titles and abstracts of biomedical articles as input, which are first preprocessed to ensure compatibility with the classification model. The classification model consists of a pretrained model extended with a classification head. The outputs of the classification model undergo a post-processing step, which converts them to entities, constituting the final output of the NER system. In the following, we detail the individual components of the NER system architecture, and then we discuss how we combine NER systems to create an NER ensemble.
Preprocessing
Pretrained model
Dense + CRF
The first component in the NER system architecture is the preprocessing of the input text. The input
text is tokenized using WordPiece [
The tokenization is performed with a maximum sequence length of 512 tokens. The resulting sequence of tokens is either padded or truncated to the maximum length to maintain compatibility with the architecture of the pretrained model. Given that only a small fraction of the dataset (77 out of 1567 samples) exceeds this limit, as shown in Section 3.1, this truncation minimally impacts the overall performance while ensuring compatibility with the architecture.
The second component in the NER system architecture is the classification model, which consists of a pretrained model (Section 4.1) extended with one of three classification heads: • Dense: A single dense layer with size equal to the number of labels. • Dense + CRF: A dense layer with size equal to the hidden size of the given pretrained model with a GELU activation function. This is connected to another dense layer whose size matches the number of labels, which feeds into a CRF. • BiLSTM + CRF: A BiLSTM layer with hidden size set to half the hidden size of the given pretrained model with a GELU activation function. This is connected to a dense layer whose size matches the number of labels, which feeds into a CRF.
The output of the classification model consists of token-level predictions, each represented by a BIO label. The BIO labels are composed of the entity labels each prefixed with B- or I-. Tokens that do not belong to any entity are labeled as O. 4.2.3. Post-processing The last component in the NER system architecture is the post-processing step, which transforms sequences of token-level predictions into entities. This step merges tokens into entities based on their predicted labels and their adjacency in the original input text. Two subsequent tokens 1 and 2, with start and end indices ( 1, 1) and ( 2, 2) in the original input text, are considered adjacent if 1 = 2 or 1 = 2 − 1. The token-level predictions are processed sequentially and merged into entities based on three rules: 1. A new span is initiated if the current token has a B- or I- prefix and there is no prior adjacent token with a B- or I- prefix. 2. Once a span is initiated, subsequent tokens can be appended to it if they are adjacent and share the same label, regardless of their prefix. 3. If there is no adjacent subsequent token or if it has a diferent label, then the current span is merged into an entity, and a new span is initiated.
Furthermore, we employ a lookahead mechanism to resolve rare cases, where the first and last tokens of a sequence of adjacent tokens share the same label, but the labels of the intervening tokens are diferent. Every time a span is initiated with a token with a B- prefix, the mechanism considers multiple subsequent tokens. If these are adjacent, have I- prefixes, and the last token has the same label as the initial token, then the labels of the intervening tokens are changed to the label of the initial token, allowing all considered tokens to be merged into a single entity.
When the token-level predictions are merged into entities, the casing of these entities is adjusted to match the casing of the original input text.
We consider two diferent NER ensemble methods to leverage the capabilities of multiple NER systems, to enhance the reliability of the model predictions. We name them entity-level ensemble and token-level ensemble.
Entity-level Ensemble The first method is an entity-level ensemble, which combines the outputs of individual NER systems at the entity-level, meaning this method is applied after the post-processing. The method involves an exact matching of the location of entities. Only entities for which the majority of participating NER systems agree are retained. Subsequently, the final label for each retained entity is determined based on a majority vote across the entity-level predictions from the participating NER systems.
Token-level Ensemble The second method is an token-level ensemble, which combines the outputs of individual classification models at the token-level prior to the post-processing. A prerequisite for this approach is to have consistent tokenization across all classification model outputs, ensuring that token-level predictions correspond to the same underlying input tokens across models.
This method involves a two-step process. In the first step, it is decided which token-level predictions are retained from the participating classification models. At each token position in the input sequence where at least one model predicts a label other than O, the predictions from all participating models are examined. The method adopts a token retainment strategy to determine whether the predictions at that position should be retained for the subsequent step. Three diferent token retainment strategies are proposed: • Union: All token-level predictions from the participating models are retained regardless of agreement. • Majority: Token-level predictions are retained only if the majority of the participating models have predicted a label other than O at the given position. • Intersection Token-level predictions are retained only if all models assign a label other than O at the given position.
In the second step, the method determines the final label for each token position. Two diferent strategies are proposed for this purpose: • Softmax sum: The softmax outputs of all the participating models are summed at the given token position. Then, the label corresponding to the highest score is selected as the final label. • Majority: The label predicted by the majority of the models at a given token position is selected as the final label for the token at that position. In cases where labels receive an equal number of votes, the label is decided with the softmax sum strategy.
We approach the RE task as a sequence classification problem. The approach is inspired by the method
proposed by Baldini Soares et al. [
Each RE system in the RE ensemble shown in the information extraction pipeline overview in Figure 3 consists of multiple components. These are illustrated in the RE system architecture in Figure 5. An RE system takes as input the titles and abstracts of the original articles together with the predicted entities from the NER ensemble.
The structure of the RE system architecture is similar to that of the NER system architecture. The input is preprocessed to ensure compatibility with the model. The classification model consists of a pretrained model (Section 4.1) followed by a classification head. In contrast to the NER systems, which classifies each token in a sequence, the RE systems assign a single label to an entire sequence. Therefore, we exclusively use a dense classification head for the RE systems, as architectures with a CRF layer are more suitable for structured prediction tasks like NER. The classification model outputs relation predictions that are post-processed to produce the set of relations, which are the final output. In the following, we first describe the preprocessing and the post-processing components, and then we detail the RE ensemble method.
Preprocessing
Pretrained model
Dense head
Classification model The first component in the RE system architecture is the preprocessing of the input text. For each pair of entities, a separate sample is generated by inserting the special entity markers around the subject and object within the full input text. This ensures that each sample highlights exactly one pair of entities. Each sample is tokenized using the tokenizer of the given pretrained model, with the vocabulary extended with the four special tokens. The resulting sequence of tokens is either padded or truncated to the maximum length of 512 tokens to maintain compatibility with the architecture of the pretrained model.
Entities
classification model, three post-processing steps are applied. The first step is to filter out any predictions where the classification model indicates that there is no relation between the subject and object, to retain only the meaningful relationships. The second step involves transforming each prediction into the format of the specific RE task. The third step identifies and removes duplicate predictions, ensuring that each relation is represented only once in the final output.
We propose a relation-level ensemble method to improve the robustness of the RE predictions. This method combines the outputs of multiple RE systems after post-processing and relies on exact matching of relations. For BT-RE (6.2.1), this requires the labels of both the subject and object to match. For TT-RE (6.2.2), the relation label is also included in the matching. For TM-RE (6.2.3), in addition to the relation label, the method further requires the exact text spans of the subject and object to match. Similarly to the entity-level ensemble strategy (Section 4.2.4), this method retains only those relations for which a majority of the participating models produce identical predictions.
To enhance performance, we construct a more realistic training dataset distribution using a negative sample multiplier for RE and incorporate weighted training to address diferences in annotation quality across datasets.
For the RE tasks, to create a more realistic training dataset, the dataset is not limited to only containing the positive relations, which are the annotated relations from the datasets.
We extend it with a configurable number of negative samples, which are instances of relations with random entities that are not related. We denote this configurable number as the negative sample multiplier. This parameter determines how many negative samples are added relative to the number of positive samples. We hypothesize that the negative sample multiplier influences model performance, and we study this efect in the experiments.
We train the classification models for NER ( Section 4.2) and RE (Section 4.3) using diferent combinations of the datasets of diferent qualities. We introduce a dataset weight vector, which is used as a hyperparameter, to account for the varying annotation quality across datasets: ⃗= ⎡ ⎤ ⎢ ⎥⎥ , ⎢
⎥ ⎢ ⎣ ⎦ ℒ = 1
=1 ∑ () ⋅ . where , , , denote the weights for the platinum, gold, silver, and bronze datasets, respectively. We associate each training sample to a weight based on its dataset quality. The weights are used during training to scale the loss contributions from individual samples according to the dataset quality of their source. then computed as:
Let denote the loss for training sample , and let () be the mapping from training sample to its corresponding dataset weight. The weighted average training loss ℒ over a batch of samples is This approach enables the model to utilize all available data, including noisier datasets, while ensuring that higher-quality annotations have a stronger influence during optimization.
All models use the cross-entropy loss function, which is defined for training sample as: =1 = − ∑ , log( ,̂ ), where is the number of classes, , is the true probability of class c, represented as 1 for the correct class and 0 for all other classes, and ,̂ is the predicted probability of class .
In this section, the setup, experiments, and results are presented. Section 5.1, Section 5.2, and Section 5.3 detail the setup, dataset preparation, and training configurations, respectively. In Section 5.4, we analyze the experiments on the development dataset for the NER and RE subtasks. This includes exploring dataset weights, model architectures, and ensemble methods. In Section 5.5, the classification models of the best-performing systems are retrained. Then, the systems with the retrained models are evaluated on the test dataset and the results are shown. Finally, the final GutBrainIE leaderboard results are presented. 5.1. Setup All training jobs are conducted on a high-performance computing cluster at Aalborg University, AILAB6. It uses the SLURM workload manager7 to schedule and manage jobs, and leverages Singularity8 to allow for creating and running containers. Each training job is allocated access to an NVIDIA L4 GPU with 24 GB VRAM, a 32-core AMD EPYC 7543 CPU, and 24 GB of RAM.
We use all dataset qualities for training in the NER and RE subtasks. As the silver and bronze datasets contain some incorrect annotations, as described in Section 3.3, these are either corrected or removed.
Additionally, for all subtasks, we concatenate the title and abstract of each paper before tokenization to maintain a consistent input length and to accommodate RE, as relations can span from an entity in the title to an entity in the abstract.
In the bronze dataset, there are 21 articles with no relations at all, as detailed in Section 3.3. We exclude these articles from the training dataset for the RE subtasks.
As detailed in Section 3.2, the vast majority of articles contain fewer than 100 annotated relations, with only a small number of outliers in the silver dataset exhibiting more than 100 relations. We hypothesize that excluding these outliers enhances model performance. Therefore, we conduct experiments with and without the inclusion of these outliers.
The training is performed using the AdamW optimizer [38] and a custom learning rate scheduler.
For the NER subtask, the models are trained for 20 epochs with a batch size of 16. The learning rate scheduler applies a linear warmup from 2 ⋅ 10−5 at epoch 1 to 8 ⋅ 10−5 at epoch 4, as using a warmup is proven to improve training stability and model performance [39]. From epoch 4 until epoch 13, the learning rate is kept constant at 8 ⋅ 10−5. Then it decays with a factor of 0.8 every second epoch until ifnished.
For the RE subtasks, the models are trained for 10 epochs with a batch size of 16 using a similar scheduling strategy. A linear warmup is applied from 10−6 at epoch 1 to 4 ⋅ 10−6 at epoch 3. From epoch 3 until epoch 6, the learning rate is kept constant at 4 ⋅ 10−6. Then it decays with a factor of 0.5 every second epoch until finished.
The development dataset is used to validate the performance of each model after every epoch. The model state corresponding to the highest F1micro score is selected for each model. This metric is used because it is the oficial evaluation metric of the GutBrainIE task.
This section presents the experimental results for the NER and RE subtasks on the development dataset.
Impact of Dataset Quality Weighting To assess the impact of dataset quality on NER performance, the classification models are trained using various dataset weight combinations for the platinum, gold, silver, and bronze datasets. To ensure a fair comparison, the BioLinkBERT-base model with a dense classification head is used exclusively for this experiment. Table 1 presents the 18 dataset weight combinations considered, along with their corresponding F1micro scores on the development dataset. The best performance training with dataset weights achieves an F1micro score of 0.8314, whereas training without weighting achieves an F1micro score of 0.8189. The two dataset weight combinations yielding the highest F1micro scores are ⃗= [1.5 1.5 1 0.75]⊤ and ⃗= [1.25 1.25 1 0.75]⊤. NER Classification Model Performance To investigate the performance across the three classification heads described in Section 4.2 and the seven pretrained models described in Section 4.1, we study three head variants of each of the seven pretrained models. This results in 21 classification models, which we train using the two best dataset weight combinations identified in Table 1. Table 2 shows the resulting F1micro scores.
The diference in F1micro scores across the three classification head categories is minimal. The classification models with dense heads perform slightly worse on average. However, although there are slight diferences, the best-performing models are distributed across all three classification heads and diferent pretrained models. Hence, we do not observe a pattern where one classification head consistently outperforms the other two.
Moreover, the results do not reveal a pattern across dataset weight combinations or architectures, and therefore, no single architecture or classification head can be conclusively identified as the best. NER Ensemble Performance Finally, we investigate how the NER ensemble methods described in Section 4.2.4 perform. Token-level ensemble aggregates predictions for each token individually, whereas entity-level ensemble retains complete entities based on majority voting across models. Both strategies improve the performance, however, the entity-level ensemble approach consistently outperforms the token-level approach. Consequently, only entity-level ensembles are considered in the following experiment and for the submission.
To determine which number of NER systems yields the best performance for the entity-level ensemble approach, we conduct experiments with diferent sizes, ranging from 3 to 17 NER systems. For each -ensemble size, we select the top- NER systems based on the results of the previous experiments, ranked by their individual F1micro scores on the development dataset. As a result, each larger ensemble includes all NER systems from the smaller ensembles. For example, the top-5 ensemble contains all NER systems from the top-3 ensemble, including the next two highest-scoring NER systems. Table 3 shows the performance of each ensemble size on the development dataset.
The results indicate that ensemble strategies lead to a notable performance improvement compared to individual models. The highest F1micro score is observed with an ensemble size of 9. However, as performance remains consistently high across all ensemble sizes in Table 3, we consider all the ensemble sizes for the submission of the NER subtask (6.1).
We conduct all RE experiments presented in this section using BT-RE classification models. Due to time constraints, the configurations yielding the best performance in the BT-RE subtask (6.2.1) are also used in the TT-RE (6.2.2) and TM-RE (6.2.3) subtasks.
Impact of Dataset Hyperparameters To determine the best configurations, we conduct an experiment to assess the impact of diferent combinations of dataset hyperparameters. These hyperparameters encompass the dataset weights, negative sample multiplier, and exclusion of articles with relation outliers. All RE classification models in this experiment are trained using BioLinkBERT-base as the pretrained model, and we evaluate them on the development dataset. Table 4 presents the results.
Lower negative sample multiplier values produce high recall but low precision, indicating a tendency to generate an excessive number of false positive relations. In contrast, higher negative sample multiplier values provide a better balance between precision and recall, resulting in the best overall performance.
We also observe that removing articles with relation outliers often improves performance. Finally, from the experiments, there does not emerge clear performance trends across diferent datasets and dataset weight combinations.
Hence, for the subsequent experiments and the final classification models, a negative sample multiplier of 10 is used, the articles with relation outliers are excluded, and all dataset and weight combinations are still considered.
RE Classification Model Performance We evaluate the performance of the seven pretrained models that are described in Section 4.3 with dense classification heads. We train the classification models on the platinum and gold datasets, with outlier articles excluded and a negative sample multiplier of 10. Table 5 shows that the results are similar across the diferent pretrained models. BioLinkBERT-base yields the best performance among the base models, while BiomedBERT-large-uncased-abstract and BiomedElectra-large-uncased-abstract yield the highest scores among the large models. RE Ensemble Performance Similar to the NER ensemble experiment, the top- RE systems from the previous RE experiments are used in a relation-level ensemble (Section 4.3.3). However, due to time constraints, we only consider ensemble sizes of 3 and 5. Table 6 shows the performance of relation-level ensembles of sizes 3 and 5 for each RE subtask. The level of performance is very similar, and it appears that the two extra RE systems in the ensemble of size 5 do not lead to an advantage.
When preparing for the submission to GutBrainIE, we retrained the classification models of the bestperforming systems from the previous experiments on a training dataset that included the development wp dataset to make use of all the available annotated data. When all available data is used for training, no separate dataset remains for validation during training. Therefore, we train the classification models both with and without the development dataset to account for this limitation, in case those trained on the development dataset perform unexpectedly. When the development dataset is included in the training, it is assigned the same dataset weight as the platinum and gold datasets.
For the NER subtask, we use the configurations presented in Table 2. We select the top 17 NER systems based on the results from all previous experiments. This selection is conducted separately for NER systems with the classification models trained with and without the development dataset included in the training data. Based on these selections, we create entity-level ensembles using the top- NER systems, where ∈ {3, 5, 7, 9, 11, 13, 15, 17} . This results in two sets of eight entity-level ensembles.
The test results for the NER subtask are shown in Table 7. As expected, including the development dataset in the training dataset increases the performance. The best result is achieved with an entity-level ensemble of the five NER systems with classification models trained with the development dataset included.
Table 3 shows that the entity-level ensemble of size 9 achieves the best performance on the NER subtask without including the development dataset in the training dataset. Since this ensemble yields the highest score prior to evaluation on the test dataset, we select it as the best configuration. By extension, we also select the entity-level ensemble size of 9 trained with the development dataset, assuming it may yield the best performance. Hence, these two ensembles are used to generate the entities for the subsequent RE subtasks.
Based on the results shown in Table 5, the three pretrained models resulting in the best performance are BioLinkBERT-base, BiomedBERT-large-uncased-abstract, and BiomedElectra-largeuncased-abstract. Hence, the RE systems in all three RE subtasks use these models. Due to time constraints, the classification models with large pretrained models are trained only using the platinum and gold datasets, while those with the base model are trained on all four datasets and dataset weight combinations shown in Table 4.
For each RE subtask, we use the relation-level ensembles of size 3. This is due to time constraints, as well as what we observed in Table 6, where there is no clear diference between the relation-level ensembles of 3 or 5 RE systems.
Since all models for the submission are trained with and without the development dataset included, there are two entity-level ensembles and for each RE subtask there are two relation-level ensembles. Combined this results in four relation-level ensembles for each RE subtask.
Table 8 shows the test results for the RE subtasks. For each subtask, the best result is achieved by using the entities from the entity-level ensemble that is trained with the development dataset and by using the relation-level ensemble that is trained without the development dataset.
BT-RE (Subtask 6.2.1) TT-RE (Subtask 6.2.2) TM-RE (Subtask 6.2.3)
Table 9 illustrates the final leaderboards for each of the subtasks of the top three teams in the GutBrainIE task. Our proposed information extraction pipeline achieved second place in the NER subtask (6.1), and ifrst place in all three RE subtasks (6.2.1, 6.2.2, and 6.2.3).
In the NER subtask (6.1), the F1micro score diference between first place and our result is 0.0026, corresponding to a diference of 0.0031%. In the RE tasks, our approach outperformed second place by 0.0291, 0.0408, and 0.0906 for subtasks 6.2.1, 6.2.2, and 6.2.3, respectively. This corresponds to a diference in performance of 4.43%, 6.32%, and 24.3%, respectively.
In this paper, we proposed a biomedical information extraction pipeline, leveraging transformer-based models pretrained on biomedical corpora for NER and RE, as part of the GutBrainIE task.
To address the domain-specific terminology involved in the tasks, we relied on biomedical pretrained models, combined in ensembles. Specifically, for NER we selected and trained seven diferent pretrained models combined with three diferent classification heads. Based on individual model performance, we formed token-level and entity-level ensembles, ranging in size from 3 to 17. We discovered that both methods and all ensemble sizes led to significant improvements in performance and that entity-level ensembling consistently outperformed token-level ensembling. Performance across ensemble sizes was comparable, with only minor variations. For RE, we applied the same set of pretrained models as in NER and evaluated ensembles of sizes 3 and 5. Both ensemble sizes improved performance, with only minor diferences observed between them.
Moreover, to address the availability of datasets of varying quality, we introduced a weighted training method, which improved performance for both NER and RE. Finally, for RE, we further improved the performance by introducing negative samples. Our experiments suggested that a large amount of negative samples (10 negative samples for each positive one) achieved the best performance.
The organizers of the GutBrainIE task conducted an external assessment of the performance of our information extraction pipeline. Our pipeline achieved the second-best performance in the NER subtask (6.1) and the best performance in the RE subtasks (6.2.1, 6.2.2, and 6.2.3).
For future work, we aim to improve the ability of the information extraction pipeline to distinguish between Chemical and Gene entities, as it currently struggles with it. This could be addressed by incorporating domain-specific gazetteers. For RE, we want to investigate the implementation of typed entity markers, as they may provide more contextual cues to improve relation classification. Furthermore, we want to improve data preparation by addressing the presence of HTML tags in the input texts. These could not be removed during the challenge, as the annotations of the test data were hidden. Removing HTML tags prevents the introduction of spurious tokens during tokenization, reducing the risk of the model learning irrelevant patterns and improving performance.
We gratefully acknowledge the support of CLAAUDIA through AI-Lab, which provided the computational infrastructure essential for conducting the experiments in this study.
This work is partially supported by the HEREDITARY Project, as a part of the European Union’s Horizon Europe research and innovation programme under grant agreement No GA 101137074.
During the preparation of this work, the authors used Grammarly and Writefull in order to: Grammar and spelling check, and Paraphrase and reword. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content. Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2021, pp. 10367–10378. [37] D. T. Nguyen, C. K. Mummadi, T. P. N. Ngo, T. H. P. Nguyen, L. Beggel, T. Brox, Self: Learning to iflter noisy labels with self-ensembling, arXiv preprint arXiv:1910.01842 (2019). [38] I. Loshchilov, F. Hutter, Decoupled Weight Decay Regularization, in: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019. [39] D. S. Kalra, M. Barkeshli, Why Warmup the Learning Rate? Underlying Mechanisms and Improvements, in: The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.