Biomedical relation extraction plays a crucial role in extracting crucial biomedical information from extensive literature, thereby supporting disease treatment and the construction of biomedical knowledge bases. However, biomedical texts contain highly technical language and domain-specific terminology, which makes it dificult for models to fully understand their semantics. Furthermore, imbalances in the distribution of samples across diferent categories in biomedical datasets result in reduced classification accuracy for categories with limited training samples. In this study, we propose a biomedical relation extraction model based on domain knowledge and prompt learning. The prompt template guides the model to focus on key features and information, so that more knowledge can be obtained from limited data. Utilizing domain knowledge to acquire refined entity representations, thereby mitigating the challenges posed by technical language and domain-specific terminology. The model is evaluated on the DDI Extraction 2013 dataset and the ChemProt dataset, and the experimental results demonstrate that our model can achieve state-of-the-art performance.
With the rapid development of the biomedical field, the
amount of biomedical literature has exploded, which
contains a wealth of biomedical information [
The highly technical language and domain-specific
terminology used in biomedical texts complicates this task, and
traditional approaches often struggle to achieve high
performance [
The application of pre-trained language models in
biomedical texts has received widespread attention and
exploration [
We propose a biomedical relation extraction model based on domain knowledge and prompt learning. Domain knowledge can provide entities with richer feature representations, which can better reflect the essence of entities and improve the efect of entity representation. Prompt learning is a method that can efectively bridge the gap between pretraining and fine-tuning on downstream tasks. The core idea of this method is to transform the traditional classification task into a cloze problem. By designing a prompt template, replace a word or a continuous short sentence (usually represented by [MASK]) in the input text with the corresponding label words, and ask the model to predict the label words. This approach makes the model need to consider more contextual information when predicting, so as to better understand the semantics of the input text. Overall, the contributions of this paper are as follows: 1) We propose a biomedical relation extraction model based on prompt learning, which can guide the model to focus on key features and information by constructing multiple task-related prompt. By introducing prompt learning, more knowledge can be obtained from limited data, which efectively alleviates the problem of insuficient knowledge that the model can learn when the amount of data is small. 2) The model obtains detailed information of biomedical entities through domain knowledge and obtains enhanced entity representation. In addition, special tokens are embedded around entities, enabling entities to better integrate domain knowledge, thereby reducing the impact of high-tech language and domain-specific terminology in biomedical texts on model performance. 3) The model is experimented on the ChemProt dataset and the DDI Extraction 2013 dataset. Experimental results demonstrate that the proposed model outperforms existing methods and achieves state-of-the-art performance in biomedical relation extraction.
Recently, various neural network-based approaches have
demonstrated commendable outcomes in diverse relation
extraction tasks and have been extensively employed in
biomedical research. Liu et al. [
Sahu et al. [
Peng et al. [
Peng et al. [
Given a sentence sequence = {1,2,. . . ,−1 ,}, where is a word in sentence and n is the length of the sentence. The subject entity 1 ={,. . . , } and the object entity 1 ={,. . . ,} are located in the same sentence. Biomedical relation extraction aims to identify the relation between 1 and 2, where is either selected from a predefined relation set or .
Fig.2 shows the architecture of the biomedical relation extraction model based on domain knowledge and prompt learning. The model consists of four modules: input module, encoding module, knowledge enhancement module, and prompt learning module. We have designed three prompt templates, namely the prompt for biomedical entity 1, the prompt for biomedical entity relations, and the prompt for biomedical entity 2. Firstly, input biomedical text and prompt templates into the model for encoding. Then, the enhanced entity representation is obtained through knowledge enhancement. Finally, through the prompt module, the model can predict the label words at the [MASK] position and select their corresponding labels for classification.
For the biomedical relation extraction task, it is represented as = , , where represents the input text and represents the category label. The sentence in the biomedical dataset is represented as = {1,. . . ,1,. . . 2,. . . ,}, where 1, 2 represents two biomedical entities, respectively. A key part of prompt learning is to construct an appropriate template P and label word V. M: Y → V is a mapping that connects the task label with the label word V.
The model’s input comprises two components, specifically the input text denoted as and the prompt template denoted as (). The sentence is subjected to tokenization, and each token is encoded using a vector of d dimensions. Moreover, an embedded "CLS" token is added at the beginning of each sentence sequence. To denote the boundaries of each biomedical entity, special symbols are introduced. The first entity is enclosed by "$" symbols on both sides, while the second entity is enclosed by "#" symbols on both sides.
In addition to retaining the original input in , multiple [MASKs] need to be fed into the model. Three prompts are designed in the input prompt template, respectively, the prompt 1() corresponding to the biomedical entity 1, the prompt () corresponding to the biomedical entity relation and the prompt 2() corresponding to the biomedical entity 2. Denote the prompt template () corresponding to the input text as: () = {1(), (), 2()}
The prompt 1() corresponding to biomedical entity 1 and the prompt 2() corresponding to biomedical entity 2 can be formalized as follows: (1) (2) (3) (4) (5) 1() = {, ℎ[ ]1} 2() = {, ℎ[ ]2}
Then, the prompt () for the relation between biomedical entities is designed. For example, in the biomedical example sentence above, the relation type is CPR: 4, which means that the relation between entity 1 and entity 2 is "inhibition". The prompt template for the relation type is "1 [MASK] 2", and the prompt label word is "has curved the". Prompt () for the relation corresponding to the input text can be expressed as:
() = {, 1[ ]2}
The complete input composition can be formalized as follows:
= {, ()}
BioBERT is a pre-trained model based on BERT, which is suitable for natural language processing tasks of biomedical texts. The BioBERT model is trained using a large corpus in the biomedical field, which can improve the text understanding and classification performance in the biomedical field, making BioBERT a model widely used in the biomedical natural language processing field.
The model’s input consists of biomedical text and a prompt template, wherein [MASK] denotes the portion that requires completion by the model. Within the input sequence, [MASK] is substituted with a special token, signifying its prediction requirement. To ensure the model comprehends the word’s position within the sentence, each word embedding vector is added to its corresponding position vector in the sequence. The Transformer architecture is employed to encode the sequence of embedding vectors and position vectors. This architecture comprises multiple layers, each containing a multi-head attention mechanism and a feed-forward neural network. Each layer encodes an input vector sequence to extract its representation. This encoding approach efectively captures both the semantic and syntactic information present in the input sequence, thereby enhancing the model’s ability to predict the content to fill the [MASK].
of the sentence.
Biomedical entities are sourced from Wikipedia and DrugBank using crawler technology to obtain interpretation information in the biomedical domain. This interpretation information is denoted as = { 1,2,3,..., }, where represents the − ℎ word and represents the length
The vector 1 for a biomedical entity is computed as the average of the hidden layer vectors from to in the model. Similarly, the vector 2 for another biomedical entity is obtained as the average of the hidden layer vectors from to in the model. The calculation formulas for these vectors are as follows: [︃ [︃ 1′ = 1 ℎ 2′ = 2 ℎ ︃( ︃( − + 1 1 1 − + 1 ∑︁ =
)︃]︃ ∑︁ = )︃]︃ + 1
(6) + 2 (7) 1, 2 denote bias vectors. where 1 ∈ × , 2 ∈ × denote weight matrices.
The semantic feature representation of domain knowledge is acquired by the model using BioBERT. This vector is then combined with entity interpretation information and the corresponding entity vector to generate an improved vector representation of biomedical entities. When a sentence containing biomedical knowledge is successfully matched with entity 1, the final hidden layer vector 1 of "CLS" can be obtained from BioBERT. The acquired enhanced representation is integrated into the model, with the calculation formulas being as follows: 1 = 4 [︀ (︀ 1′, 3 (ℎ (1)) + 3)︀] + 4 2 = 6 [︀ (︀ 2′, 5 (ℎ (2)) + 5)︀] + 6 where 3, 4, 5, 6 denote weight matrices. 3, 4, 5, 6 denote bias vectors.
In the prompt module, multiple prompts are combined directly to form a complete prompt for a specific task. The complete prompt template is as follows: () = ℎ [ ]1 1
[ ]2 ℎ [ ]3 2 where [ ]1 is mask of entity, [ ]2 is mask of entity relation, and [ ]3 is mask of entity. The corresponding label words are as follows: [1] = {ℎ, , } (8) (9) (10) (11) (12) (13) (|) = ∏︀=1 ([ ] = () | ()) (14) where is the number of mask positions in (), and () is the label word set [ ] that maps class to the − ℎ mask position [ ] .
During the training process, the model will predict the [MASK] part of the input sequence through the masked language model (MLM) according to the information in the context, which makes the goal of the model consistent with the task goal of the MLM, thus efectively reducing the pretraining and downstream task gap.
In our model, label words are critical to accurately classify the relation between biomedical entities. We design a set of label words for each relation type, and further verify their efectiveness by using them for model training and testing. Entity label words refer to words that describe biomedical entity types, such as Chemical or Gene. Label words can help the model better understand entity types and thus correctly predict the relation between entities.
Relational label words are key short sentences describing the relation types of biomedical entities, which are very important for the classification results of biomedical entities. During the learning process of the model, fill in the prediction result of [MASK] and the closest set of label words in the label word set, and the relation label words can make the model better understand the relation between biomedical entities. Table 1 and Table 2 show the details of biomedical entity label words and relation label words in CPI dataset and DDI dataset respectively.
The performance of the model is evaluated by the DDI
Extraction 2013 dataset [
DDI Extraction 2013 Dataset
The DDI Extraction 2013 dataset is a dataset for extracting drug-drug interaction relations. This dataset contains medical texts from multiple sources such as DrugBank and MedLine. DrugBank provides drug names, chemical formulas, and pharmacological information, while MedLine provides abstracts and full-text articles containing DDI information. All drug pairs in the text are annotated as having or not having interactions, with a total of four types of interactions, namely Advice, Efect, Mechanism, and Int. The quantity statistics of the dataset are shown in Table 3.
ChemProt Dataset
ChemProt dataset is a benchmark dataset used for extracting chemical-protein interactions (CPIs) from biomedical literature. The dataset consists of documents from PubMed False
Advice Mechanism
Efect Int
DRUG DRUG DRUG DRUG DRUG has nothing to need advice with generate mechanisms with make efect with will interact with
DRUG DRUG
DRUG DRUG
DRUG and PubMed Central, which are annotated with diferent types of CPIs, such as inhibition, activating. The dataset was originally created for the BioCreative IV challenge in 2013, and has since become a widely used benchmark dataset in the field of biomedical natural language processing. Detailed statistics are shown in Table 4.
Evaluation Metrics
To assess the eficacy of the proposed model, its performance is measured using precision, recall, micro-F1 and macro-F1 metrics. In particular, the micro-averaged metrics are employed to derive an average metric by amalgamating the contributions of all classes. The macro-F1 score is more efective in accurately reflecting the superior performance of the model in classes with fewer samples.
Implement the model proposed in this article through the Python programming language and PyTorch development framework. The Python language has good compatibility with existing deep learning frameworks. Set the batch size to 8. During the training process, an Adam optimizer was used to optimize the parameters that afect model training and output. Set the maximum sentence length to 512 and the learning rate of the model to 2e-5. The experimental parameter settings are detailed in Table 5.
Comparison with Other Models The CPI dataset and the DDI dataset were employed to evaluate the performance of the model. Table 6 presents the experimental results of the model and other approaches on the DDI dataset. Precision, recall, Micro-F1 and Macro-F1 scores were used to assess the model’s performance. The Micro-F1 and Macro-F1 scores provide a comprehensive evaluation of the model’s performance, with higher values indicating better performance. The model achieved P, R, Micro-F1 and Macro-F1 scores of 84.2%, 83.4%, 83.8% and 76.8%, respectively, better than achieved baselines on the DDI dataset .
Furthermore, the model achieved F1-scores of 84.3%, 78.4%, 86.3%, and 58.2% in the Advice, Mechanism, Efect, and Int categories, respectively. Notably, the F1-scores in the Int type, which has limited data, surpassed those of other methods. Comparison results with alternative models suggest that the model proposed in this study efectively enhances biomedical relation extraction performance.
Table 7 exhibits the comparison results between this model and other approaches on the CPI dataset. The model achieved P, R, Micro-F1 and Macro-F1 scores of 80.0%, 81.1%, 80.5% and 77.1%, respectively, representing a 4% improvement in Micro-F1 score compared to the BioBERT model. Moreover, the model obtained F1-scores of 74.3%, 81.4%, 77.7%, 82.3%, and 69.4% in the CPR:3, CPR:4, CPR:5, CPR:6, and CPR:9 types, respectively. The comparison results with other models demonstrate that the model proposed in this paper efectively enhances the classification performance of types with limited data.
Ablation Study The ablation studies were conducted to assess the individual contributions of each module in the model towards the overall performance. The outcomes of these studies are presented in Table 8. After eliminating domain knowledge from the model, the Micro-F1 score decreases by 1.1% and 1.6% in the DDI and CPI datasets, respectively. The experimental findings indicate that domain knowledge plays a moderating role in mitigating the impact of domain-specific terminology on model performance.
When prompt learning is removed from the model, the Micro-F1 score experiences a decline of 1.9% and 2.7% in the DDI dataset and the CPI dataset, respectively. We hypothesize that prompt learning can narrow the gap between pre-training and downstream tasks, enabling the model to acquire more knowledge from limited data and thereby enhancing the efectiveness of biomedical relation extraction. Upon removing both domain knowledge and prompt learning from the proposed model, the Micro-F1 score exhibits a decrease of 2.5% and 3.1% in the DDI dataset and CPI dataset, respectively. The experimental results demonstrate that domain knowledge and prompt learning are crucial components of the model, contributing significantly to the improvement of biomedical relation extraction performance.
The dataset in the relation extraction task usually requires manual annotation of a large amount of high-quality data, which requires the participation of domain experts. However, the cost of collecting these data is high, especially in the biomedical field. Therefore, in the context of resource scarcity, how to make the model fully utilize existing data to achieve better performance has become a highly concerned issue.
The relation extraction performance of the model is evaluated by simulating low-resource relation extraction when biomedical data is scarce. The K-shot support set is constructed using the training set of the biomedical dataset, where each entity type contains K samples. To simulate low-resource biomedical relation extraction, 8, 16, and 32 samples are sampled for each entity type, and each relation type is sampled at least once. Table 9 shows the comparison of biomedical relation extraction performance of our model and other pre-trained models under low-resources.
According to the comparative findings presented in Table 9, it is evident that our model exhibits commendable performance in scenarios characterized by limited resources. In such instances, our model surpasses other pre-trained models in terms of eficacy. Notably, even when working with a relatively modest data volume at K=8, our model manages to attain desirable outcomes. Even upon increasing K to 16, the F1 score of our model remains superior to that of other models. As K is further elevated to 32, the discrepancy between our model and other pre-trained models gradually diminishes alongside the expansion of the sample size. Nevertheless, our model’s performance continues to outshine that of other models. Empirical evidence substantiates the notion that our model efectively enhances the accuracy of biomedical relation extraction when confronted with limited resources.
As shown in Figure 3, we selected some examples from the biomedical dataset for detailed analysis. We compare the prediction results of BioBERT with our model. According to Case 1, the result of the BioBERT model is Negative, indicating that the prediction is incorrect, while our model is CPR: 9, indicating that the prediction is correct. The sentence contains multiple biomedical entities, which makes it dificult for the model to fully learn the Semantic information of biomedical text. Upon integrating the biomedical entities with the expertise found in the knowledge base, the model is fortified to represent the said entities, facilitating a better understanding of the textual information. The prediction results show that our model can obtain enhanced text representation after integrating domain knowledge, and improve the classification efect in sentences containing complex biomedical entities.
According to Case 2, there are multiple biomedical entities in the sentence, which makes it dificult for the model to fully learn the Semantic information of biomedical text, and BioBERT model makes wrong predictions. Our model fused domain knowledge and made correct predictions. According to Case 3, the BioBERT model incorrectly predicts Int type text as Mechanism. The small number of Int type training samples makes it dificult for the BioBERT model to fully learn its class characteristics. Our model can obtain more knowledge from limited data by introducing prompt learning, efectively alleviating the problem of insuficient learning knowledge when the data volume is small. Therefore, our model made the correct prediction.
In this study, we propose a biomedical relation extraction model based on domain knowledge and prompt learning. The model can enhance entity representation by integrating domain knowledge, thus reducing the impact of highly technical languages and domain specific terms in biomedical texts on model performance. By introducing prompt learning, more knowledge can be obtained from limited data, efectively alleviating the problem of insuficient knowledge that models can learn when the data volume is small, thereby improving the classification efect of biomedical relation. The experimental results show that the model can efectively improve the accuracy of biomedical relation extraction by introducing domain knowledge and prompt learning.
In the future, we will continue to explore the potential of prompt learning, try diferent prompt methods, and apply our model to document-level relation extraction.
This work is supported by grant from the Natural Science Foundation of China (No. 62072070 and 62106034)