In this paper, we present our system for the BioNNE English track, which aims to extract 8 types of biomedical nested named entities from biomedical text. We use a large language model (Mixtral 8x7B instruct) and ScispaCy NER model to identify entities in an article and build custom heuristics based on unified medical language system (UMLS) semantic types to categorize the entities. We discuss the results and limitations of our system and propose future improvements. Our system achieved an F1 score of 0.39 on the BioNNE validation set and 0.348 on the test set.
The state-of-the-art nested NER models are the machine reading comprehension (MRC) model [
Recently, large language models (LLM) have shown great potential in solving NLP tasks. Since they are
pre-trained on a large corpus, they can solve various problems in diferent domains including biomedical
question answering and information extraction. Prompt engineering [
Chen et al. [
Mixtral 8x7B [
We build a system for the BioNNE English track that uses a general-purpose LLM and a biomedical domain language model to identify entities in an article and then uses custom heuristics based on unified medical language system (UMLS) semantic types to determine the entity types. The system design is shown in Figure 1. Figure 2 shows an example of how DISO and LABPROC entities are extracted from an article by LLM and categorized by UMLS heuristics.
Given an article abstract, we use Mixtral 8x7B instruct LLM model to find the entities for each category. We construct a prompt for each category using two of the examples in the training dataset and an instruction that describes the entity type. Then the LLM returns a list of entities separated by semicolon in the response. We then parse the returned entities and store them as candidates for the category. We perform post-processing on the response list to remove duplicates as well as entities that are not present in the original text. If the entities in the returned string are not separated by semicolons, we resample the LLM output. For example, Table 1 shows the prompt and response to query ANATOMY entity for record 26271422_en.
Instruction: Return phrases or entities that comprise organs, body part, cells and cell components, body substances in TEXT, in the ANATOMY concatenated by ";" [TEXT]: Impact of bosentan therapy on stress-induced pulmonary hypertension in patients with systemic sclerosis. AIM To describe hemodynamic and clinical changes in patients with elevated mean pulmonary artery pressure (MPAP) ... [ANATOMY]: pulmonary; artery; pulmonary artery; lung;heart; left heart; atrial; right atrial; cardiac; arterial; vascular; pulmonary arterial; pulmonary vascular ### [TEXT]: The authors present the material of their study of the morphological and molecular biological features of damage to the stem cell ... [ANATOMY]: lung biopsies; respiratory acinus; lung tissue; mesenchymal cell; myofibroblast; mesenchymal stem cell; SCN; stem cell; cell; lung;pulmonary; acinus; stem cell niches; tissue;mesenchymal; SCN areas; respiratory acini; biopsies; sections; acini; cells ### [TEXT]: Endoprosthetic replacement with lifting of abdominal wall in treatment of umbilical and postoperative ventral hernias. The results of ...
[ANATOMY]: Response: abdominal wall defect; abdomen ptosis; polypropylene prosthesis; main flap; roundish edges; additional flap; super lightweight; hypogastric area; musculoaponeurotic tissues lifting; physical health component; psychic health component; standard; original
In addition to LLM, we also use the ScispaCy biomedical NER model en_ner_bc5cdr_md [12] to find the named entities. The ScispaCy model is trained on the BC5CDR corpus, which detects DISEASE and CHEMICAL entities in the document. We store the entities found by the ScispaCy model as candidates for the DISO and CHEM categories.
Finally, we query UMLS [13] to find the semantic types for the candidate entities. The UMLS release version used in this study is 2023AB. Specifically, we use UMLS [ 14] "/search/current" API and set the search term as our entity to retrieve the top 5 Concept Unique Identifiers (CUI) and their names associated with the entity. For each CUI, we use the "/content/current/CUI/cui" API to retrieve the semantic type name for the CUI and then use the semantic_uri to find the treeid of the semantic type. Finally, the treeid is mapped to the BioNNE category to finalize the category of an entity.
An entity is finalized as a specific category only when both the language model (either LLM or
NER model) and the UMLS heuristics agree on the category. If UMLS heuristics show that the entity
ifts multiple categories in the top 5 CUI results, we will use the category type that comes first as the
ifnal category. The mapping between BioNNE category and UMLS semantic type is shown in Table 2.
The mapping is based on the NEREL-BIO paper [
We use the ScispaCy abbreviation detector [12] to detect acronyms in the abstract. If the long form of the acronym is identified by the language model and UMLS heuristics as a named entity, the acronym will be assigned the same category as the long form entity.
Our model achieved 0.348 F1 score on the leaderboard for the BioNNE English track test set. The F1 score we achieved on the validation set is 0.39, which is close to the test F1 score. Since the golden answers for the test set have been not released, we will discuss the validation set results here.
The English validation set contains 50 records. The F1 scores for each category sorted in descending order are shown in Table 3. Our model performs well on DISO, CHEM, and ANATOMY entities, with F1 scores above 0.5, but the F1 scores for FINDING and PHYS are below 0.3.
We examine the model predictions for FINDING and PHYS entities and have the following observations.
1. The LLM model generates a lot of false positives for the FINDING entities. The UMLS heuristics can filter out some of the false positives. However, since our UMLS concept search does not search for exact terms, some of the LLM-recognized phrases that are part of some FINDING entity word are not filtered out, even though those LLM-recognized phrases (when looked at independently) do not belong to the FINDING type. For example, for the sentence (in record 27029443_en), "The authors suggest the algorithm for choosing the order of priority of surgical interventions on coronary and brachiocephalic arteries", the LLM recognizes "suggest" as a FINDING entity, but the UMLS heuristics do not filter it out because it corresponds to "Abnormal/suggest Ca", which is a FINDING concept. 2. PHYS entity recognition has a low recall. On one hand, PHYS entities sometimes contain generic one-word terms such as "healthy", "lifetime", "size", "shape" and "adults", which are not recognized by LLM or UMLS as PHYS entities. On the other hand, some long phrases (which are outer nested entities) recognized by LLM as PHYS entities are excluded by UMLS heuristics because the UMLS search cannot find such terms. For example, "spirometric indicators" and "peripheral blood oxygen saturation" in record 27030325_en are excluded by UMLS heuristics. The general takeaways from the results are: 1. Context is important in NER task. The same phrases have diferent entity types in diferent documents. For example, the words “inflammatory” and “albuminuria” can be PHYS or DISO in diferent contexts. Our UMLS heuristics only check the semantic meaning of phrases without looking at the context, which limits the performance of our model. 2. Our model is often unable to detect the outer nested named entities. For example, it recognizes “cardiac contractility” as PHYS, but fails to recognize “low cardiac contractility” as FINDING. Sometimes those outer nested named entities are identified by LLM but rejected by UMLS heuristics because UMLS search cannot find matching terms in the UMLS system.
To understand the impact of UMLS heuristics on the system performance, we remove the UMLS heuristics from the system pipeline. When the LLM model identifies multiple entity types for a phrase, we assign the final entity type according to the INJURY_POISONING, ANATOMY, PHYS, DISO, CHEM, LABPROC, DEVICE and FINDING order. The macro-F1 score on the validation set without UMLS heuristics is 0.2151, which is significantly lower than the F1 score (0.348) when UMLS heuristics are used. The F1 scores for every category are shown in Table 4. We observe that without UMLS heuristics, the precision score for each category becomes lower. This is because the LLM model generates many false positives. In terms of recall, only the INJURY_POISONING and ANATOMY types have better recall scores when UMLS heuristics are not used. This is because we prioritize assigning the final entity type as INJURY_POISONING and ANATOMY when the LLM detects multiple matching entity types. This result shows that UMLS heuristics play an important role in eliminating the false positives of LLM predictions.
The Mixtral 8x7B model oficially supports five languages, which do not include Russian. Therefore, we only attempt the English track. It is possible that other LLM models that support Russian can be used to accomplish the Russian/bilingual track using the same modeling pipeline. The method we use in this study is few-shot prompt engineering and we only use two examples from the training dataset to construct the LLM prompt. We did not include more examples in the prompt, as the input token size of LLM is limited and the cost of processing a long prompt is high. We also do not expect adding more examples in the prompt would improve the performance significantly, as the prompt is mostly used to guide the model to generate outputs in the desired format. The entities recognized by the model are mostly based on the original knowledge of the model. However, we believe the remaining training examples will be useful if we want to finetune the LLM model on the BioNNE task in the future.
There are two directions for future improvement.
First, we can improve the heuristics for determining the category of an entity. Our current algorithm is weak at detecting the outer nested entities. Even though the LLM model recognizes them, we fail to assign the correct category for those entities using the existing UMLS heuristics. We can add a new set of heuristics for detecting the outer nested entities. Specifically, if a long phrase contains an inner entity that is a named entity, we can assign the outer entity with the same category as the inner entity or assign the outer entity with a category detected by the LLM.
Second, we can improve the performance of the LLM model. Currently LLM model generates many false positives and it may assign several entity types to a single phrase. For example, it recognizes "neuronal dysfunction" as DISO, ANATOMY, PHYS, FINDING and CHEM in the sentence "the severity of coronary artery lesions and low cardiac contractility afect the degree of cerebral ischemia and neuronal dysfunction detected by spectral EEG power" (in record 25726786_en). This is probably because the general-purpose LLM model (Mixtral 8x7B instruct) trained on the general text does not have the domain knowledge of the biomedical field. Therefore it cannot distinguish the subtle diferences among the BioNNE entity types. We can fine-tune the LLM model on the BioNNE training data as well as other biomedical NER datasets to improve the performance of the model in this task.
Although the Mixtral 8x7B model does not oficially support the Russian language, a recent study by Fenogenova et al. [15] shows that Mistral 7B (a smaller version of Mixtral 8x7B) performs well on Russian language benchmarks compared to other LLMs. Future work can explore using Mixtral or other LLM models to solve the BioNNE bilingual NER problems using prompt engineering techniques in the paper or fine-tuning the model with bilingual datasets.
We build a system that uses a general-purpose LLM, a biomedical domain NER model and UMLS-based
heuristics to extract nested named entities from biomedical text. Our model achieved an F1 score of
0.39 and 0.348 on the BioNNE English validation and test sets. Although the results are not comparable
to those of the state-of-the-art models [
We want to thank the Data Science @ Georgia Tech (DS@GT) CLEF team for their support and acknowledge the use of Grammarly [16] to proofread this paper. Llama open and eficient foundation language models, arXiv preprint arXiv:2302.13971 (2023).
URL: https://arxiv.org/abs/2302.13971. [12] M. Neumann, D. King, I. Beltagy, W. Ammar, ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing (2019) 319–327. URL: https://www.aclweb.org/anthology/W19-5034. doi:10.18653/v1/W19-5034. arXiv:arXiv:1902.07669. [13] O. Bodenreider, The unified medical language system (umls): integrating biomedical terminology, Nucleic acids research 32 (2004) D267–D270. URL: https://doi.org/10.1093/nar/gkh061. doi:10. 1093/nar/gkh061. [14] Umls terminology services (2023). URL: https://uts.nlm.nih.gov/uts/umls/home. [15] A. Fenogenova, A. Chervyakov, N. Martynov, A. Kozlova, M. Tikhonova, A. Akhmetgareeva, A. Emelyanov, D. Shevelev, P. Lebedev, L. Sinev, U. Isaeva, K. Kolomeytseva, D. Moskovskiy, E. Goncharova, N. Savushkin, P. Mikhailova, D. Dimitrov, A. Panchenko, S. Markov, Mera: A comprehensive llm evaluation in russian, 2024. URL: https://arxiv.org/abs/2401.04531. arXiv:2401.04531. [16] Grammarly, Grammarly handbook (2024). URL: https://www.grammarly.com/handbook/.