Biomedical entity normalization, which links entity mentions in biomedical texts to their corresponding standard concepts in a knowledge base(KB) or an ontology is an important task in biomedical text mining. A prevalent solution is to generate the most similar concepts, and then rank those concepts with semantic models. Herein, to improve the performance of candidate concepts ranking for entity normalization, we rank the candidates by ne-tuning the domain-speci c pre-trained BioBERT model and enhancing the representation information with entity mentions and candidates. We have achieved signi cant improvement over the state-of-the-art method on the Bacteria Biotope data of BioNLP-OST191 .
Mapping entity mentions in texts to a certain standard knowledge base (KB) or an ontology is a fundamental task, which can link the unstructured text to a structured dataset. Ambiguity and variation are the main challenges of this task. Unlike in the general domain, variation is much more common than ambiguity in the biomedical domain. Therefore, a variety of methods have been proposed to deal with this challenge, including rule based methods [1], machine learning and deep learning based methods [2, 3].
Recently, pre-trained models have been applied to many NLP tasks in the biomedical domain such as named entity recognition and relation classi cation tasks, resulting in signi cant improvements. BioBERT [4], which is based on the BERT model and pre-trained on large-scale biomedical articles, is applied to many biomedical NLP tasks to improve state-of-the-art performance. But few researchers have used the models for entity normalization. In this paper, we propose a biomedical entity normalization approach by ne-tuning the BioBERT model. We enhance the representation information by using the embedding of the special rst token as well as the embeddings of the entity mention and its candidate. The performance of our approach achieves signi cant improvement compared to the best scores in the BB-norm shared task of BioNLP-OST19.
Our approach includes two principal steps: (
We rst pre-process all entity mentions and concept names in KB with abbreviation and tokenization resolution. The Ab3p tool2 is utilized to identify abbreviations in documents and replace entity mentions in abbreviations with their corresponding full names. The Snowball toolkit 3 is used to tokenize all the entity mentions and concept names. Then, we implement two types of methods to generate the candidate concepts: Similarity based method: We calculate the cosine similarity between vector representations of each concept name and the mention, and also the Jaccard similarity between them [5]. Then we choose the top n1 concept names as a set C1, which have cosine similarity greater than or equal to threshold t1, top n2 concept names as a set C2, which have Jaccard similarity greater than or equal to threshold t2. The nal candidate set is composed of C1 S C2. In our experiments on training data, we set t1 = 0:7, t2 = 0:1, n1 = 3, n2 = 7.
Information Retrieval based method: It is ine cient to calculate the similarity between a given mention and each concept name in the KG when the number of concepts is very large. In this case, IR is a more e cient method with which to obtain similar concept names. We implement the IR system using Lucene4. First, we index all mentions in training data and concept names with their identities. Second, we retrieve the top-20 concept names for each mention, then the nal candidate concept names are obtained from the retrieved results using the similarity based method described above. 2.2
We rank the candidate concepts by ne-tuning the pre-trained BioBERT model. Inspired by the work [6], for each entity mention m and one of its candidates c, we feed a sequence [CLS] m [SEP] c to BioBERT for the ne-tuning procedure, where [CLS] is the beginning of each sequence, and [SEP] is a special token used to separate m and c. V is supposed to be the nal hidden state generated from BioBERT model, and d is the dimension of the hidden state of the model. V0 2 Rd represents the output of the rst token [CLS], Vm and Vc are the nal hidden vectors for m and c respectively, Vm = [Vi; :::; Vj] and Vc = [Vl; :::; Vn].
Then we get the representation of m (V m0) and the representation of c (Vc0 ) using the following equations:
V m0 = W [tanh(
1 j i + 1
j X Vt)] + b Vc0 = W [tanh( t=i
1
n l + 1
n
X Vt)] + b (
Where we use an average operation to the vectors, we then add an activation layer and a fully connected layer, there are j i + 1 words in m and n l + 1 words in c. For V0, an activation layer and a fully connected layer are added:
V00 = W0[tanh(V0)] + b0 We then concatenate V 0 , V m0, Vc0 and add a fully connected layer to generate the 0 nal representation for a mention and one of its candidates:
r = Wcon[concat(V00 ; V m0; Vc0 )] + bcon
In Equations (
It is supposed that there are K candidates for each entity mention. We use rk
to represent the nal vector output by our model for the kth candidate name. To
rank the K candidate names, we rst de ne R = [r1; r2; :::; rK ], and then compute
the probability of the kth candidate to be the normalized one as follows:
(
Dataset: We use the Bacteria Biotope (BB) data to evaluate our approach. Three types of entities are involved: microorganism, habitat and phenotype. Microorganisms are normalized to taxa from the NCBI taxonomy5, which contains 903,191 taxa plus synonyms. While habitat and phenotype entities are normalized to concepts from the OntoBiotope ontology6 which includes 3,601 concepts plus synonyms. Table 1 shows the number of mentions, unique mentions and concepts for each entity type. In the candidate generation step, we use the IR based method to generate candidates for microorganisms, and the similarity based method to generate candidates for habitat and phenotype entities. Metrics: Since the entity mentions in the data are given and every mention is normalized to a concept, we evaluate the performance of our biomedical concept normalization algorithm with precision, following the BB-norm task. The o cial on-line testing platform7 is used to calculate scores on the test data.
Habitat Phenotype Microorganism Entity mentions 3,506 1,102 2,487 Unique entity mentions 1,774 498 950
Concepts 440 141 491 Parameters Settings: For ne-tuning, the parameters are the same as those in the pre-trained BioBERT model. We set the learning rate to 5e-5, the batch size to 16, and the number of training epochs to 16. Early stopping is employed according to the precision of the validation set.
Experimental Results The performance of the method is displayed in table 2. Compared to the methods from o cial teams who participated in the BB-norm shared task, our method achieves signi cant improvement of, respectively +4, +6, and +4 points compared to the best scores for habitat, phenotype, and microorganism normalization, and +10 points compared to the best score for all types. We also discard the hidden vector output of the entity mention and candidate, and only use the hidden vector output of the special rst token for ranking, the results show that combining the output of entity and candidate vectors further enriches the information and improves the accuracy. In this paper, we develop an approach for biomedical entity normalization by ne-tuning the pre-trained model, we also leverage the embeddings of entity mentions and their candidates to enrich the information and improve the performance. We conduct experiments on the BB dataset provided by the challenge of BioNLP-OST and our results signi cantly outperform the state of-the-art methods. In the future, we will try to add the context information of a mention to improve the performance of this problem.