In this paper, an approach for concept extraction from documents using pre-trained large language models (LLMs) is presented. Compared with conventional methods that extract keyphrases summarizing the important information discussed in a document, our approach tackles a more challenging task of extracting all present concepts related to the specific domain, not just the important ones. Through comprehensive evaluations of two widely used benchmark datasets, we demonstrate that our method improves the 1 score compared to state-of-the-art techniques. Additionally, we explore the potential of using prompts within these models for unsupervised concept extraction. The extracted concepts are intended to support domain coverage evaluation of ontologies and facilitate ontology learning, highlighting the efectiveness of LLMs in concept extraction tasks. Our source code and datasets are publicly available at https://github.com/ISE-FIZKarlsruhe/concept_extraction.
Concept/Keyword extraction is recognized as a fundamental task in Natural Language Processing
(NLP), crucial for identifying and extracting noun phrases that summarize and represent the main
topics discussed in a document [
However, it is not only helpful in those fields of research but also in ontology evaluation methods.
Usually, competency questions (CQs) are created to specify the ontology requirements and are used later
for the evaluation of the ontology [
The task of concept extraction can be formally defined as follows: Given a document
represented as a sequence of words = [1, 2, 3, ..., ], the goal is to extract a set =
(1, 1), (2, 2), ..., (, ) where consists of one or multiple words that best represent the topics
of the document . The additional score ∈ [
Concept extraction is typically categorized into two types: (i) extracting present keyphrases
(extractive), which are directly found within the input document such that ∈ , and (ii) extracting
2nd International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs (NSLP 2025), co-located
with ESWC 2025, June 01–02, 2025, Portorož, Slovenia
* Corresponding author.
$ ebrahim.norouzi@fiz-karlsruhe.de (E. Norouzi); sven.hertling@fiz-karlsruhe.de (S. Hertling);
harald.sack@fiz-karlsruhe.de (H. Sack)
0000-0002-2691-6995 (E. Norouzi); 0000-0003-0333-5888 (S. Hertling); 0000-0001-7069-9804 (H. Sack)
© 2025 Copyright © 2025 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
absent keyphrases (abstractive), which are generated even though they do not appear explicitly in the
document [
In this paper, we concentrate on extracting present concepts from documents using large language
models (LLMs) because this approach best fits the task of ontology evaluation. Ontology evaluation [
Our contributions include: • Utilizing large language models for efective concept extraction from scientific documents. • Providing comprehensive evaluations on benchmark datasets. • Demonstrating improved performance over state-of-the-art techniques in extracting present concepts.
The paper is structured as follows: Section 1 introduces the problem and outlines the objectives of this study. Section 2 reviews the existing literature on concept extraction and large language models, highlighting the gap this study aims to fill. Section 3 details the methodology, the experimental setup, and the various prompts tested. Section 4 presents the evaluation metrics, the datasets used, and the results of the experiments, with a detailed analysis of the performance of diferent models and prompts. Section 5 concludes the paper, summarizing the key findings and outlining directions for future work.
Several comprehensive surveys provide a thorough overview of concept extraction techniques [
Recent advancements in unsupervised concept extraction have led to the development of several
innovative approaches that do not rely on annotated data. Statistical methods, such as TF-IDF [
Graph-based methods have also been widely explored. TextRank [10] represents text as a graph, where words serve as nodes and their co-occurrences act as edges. These methods employ node ranking algorithms, such as PageRank [11], to rank words and extract the top-k words as keyphrase candidates. Bougouin et al. [12] introduced TopicRank, which clusters candidate phrases into topics in the initial phase and subsequently ranks these topics based on their significance within the document. Positionrank [13] further enhances this approach by incorporating the positional information of words in the text to improve ranking accuracy.
Embedding-based models have emerged as another efective approach for keyphrase extraction. EmbedRank [14], for instance, utilizes part-of-speech tagging to identify potential keyphrases from a document. These keyphrases are then represented as low-dimensional vectors using a pretrained embedding model and ranked according to their Cosine similarity to the document’s overall embedding vector. This technique leverages the semantic representations provided by embedding models to improve keyphrase extraction. MultPAX [15] employs semantic similarity between candidate keyphrases and an input document using the pretrained embedding of BERT model. AutoKeyGen [16] constructs a
D = [w1, w2, ..., wn] System Prompt: You are a helpful, respectful and honest assistant for extracting keyphrases from the provided document.
User Prompt: I have the following document: {{D}} Please give me the keyphrases that are present in this document and separate them with commas: Assistant Prompt: C = (c1, s1), (c2, s2), ..., (cm, sm)
C’={c’1, c’2, ..., c’n}
phrase bank by combining keyphrases from all documents into a corpus, considering both lexical and semantic similarities for selecting top candidate keyphrases for each input document. PromptRank [17] is an unsupervised approach based on a pre-trained language model (PLM) with an encoderdecoder architecture. PromptRank feeds the document into the encoder and calculates the probability of generating the candidate with a designed prompt by the decoder to rank the candidates. A limitation of PromptRank is that its performance heavily depends on the design of prompts and may not always guarantee optimal results.
Recent eforts on pre-trained large language models, such as ChatGPT and ChatGLM, have
demonstrated promising performance using zero-shot prompts, inspiring the exploration of prompt-based
methods for keyphrase extraction [18]. However, these eforts are limited to ChatGPT and ChatGLM
in zero-shot prompts. While these models perform well in various NLP tasks without parameter
tuning, experimental results indicate that ChatGPT has significant room for improvement in keyphrase
extraction compared to state-of-the-art models. This gap is attributed to limited resource settings and
basic experimental configurations. More sophisticated methods, including complex prompt designs and
contextual sample construction, are necessary to optimize its performance. A comprehensive study
reports that models like GPT-3, InstructGPT, and ChatGPT show modest improvements in keyphrase
predictions [
The concept extraction approach uses various Large Language Models (LLMs) to identify and extract keywords from input documents. Figure 1 shows the workflow. Initially, an input document is provided containing the text from which concepts need to be extracted. This document and a specific prompt are used for the extraction process. The prompts include a system prompt, a user prompt, and an assistant prompt. The system prompt sets the context for the extraction task. The user prompt directly asks the model to perform the extraction, and the assistant prompt is where the LLM outputs the extracted keywords. Before the evaluation, the extracted concepts are further filtered because only concepts present in the input document should be returned. Thus, it is easily possible to check if each extracted concept is actually contained in the document and filter those that are not.
The LLM will generate natural language text as an answer. Even though it is asked to only return the concepts separated with commas, various conversational statements can appear at the beginning of the generated text, like "Sure, I’d be happy to help! Based on the information provided in the document ...". Another issue is that the output format does not always need to fit the requested format which should separate the concepts with commas.
To overcome all these problems, the extraction of the concepts, given the produced text, works as follows: The text is split by tokens ",", ";", "*", and "\n"(newline). The reason for the first two is that the LLM might separate the concepts not only by a comma but also by a semicolon. The star is used as a separator because some models might also return a markdown formatted unordered list. Finally, the newline is also included because conversational sentences or phrases are sufixed with a newline to nicely format the output (most probably due to fine-tuning to a chat-style). The keywords that only appear in the document are selected. Thus also those conversational sentences will be removed.
The task of concept extraction involves not only to extract the concepts themselves but also an attached confidence score to finally rank the extracted concepts. An LLM usually does not provide a confidence score in the generated output even though it is asked for it because, on the one hand, it is dificult to generate numbers that reflect the confidence of the tokens previously generated, and on the other hand, the training data usually prevent the model from generating such numbers (the model often outputs sentences such as "As an LLM, I’m not able to provide confidence scores"). Therefore we implemented another approach to extract the confidence scores of the model. When generating a new token, the model computes a probability distribution over their entire vocabulary, where each token is assigned a likelihood based on the model’s internal scoring function. To extract a confidence score for a generated concept, we retrieve the probability assigned to each token in the phrase during generation and compute the geometric mean of these probabilities, providing a single aggregated confidence value. There are several generation strategies for how the next token is then generated. Greedy search, for example, always selects the token with the highest probability such that the results are always the same (given the same input tokens). Other approaches like multinominal sampling or contrastive search [19] are non-deterministic because they sample the next token based on the computed distribution and thus could generate diferent results. For reproducibility reasons, we selected the greedy search and finally computed a confidence score for each extracted concept by multiplying the probabilities of all tokens that form this concept.
For the prompt design, we chose to make a systematic analysis. To design the prompt configurations, various search terms, and prompt setups were explored for efective concept extraction. We first started with a simple prompt (see ZS Keywords in Table 1) but modified the search term to identify the most suitable terminology. The search term is the word that is used to describe the task, e.g., "give me the keywords/concepts/entities in this document," where keywords, concepts, and entities are the search terms. Overall, five of those search terms are tried out, and the best-performing word is selected.
All other prompts are based on the previously selected prompt but extended in various ways. This includes prompts with more precise domain information, situational contexts, and detailed task descriptions. For example, the ZS + Domain prompt includes more precise domain information, guiding the model to extract keyphrases related to specific fields such as Computer Science, Control, and Information Technology (which are the topics of the selected datasets). The ZS + Extracting Context prompt asks the model to act as a helpful assistant specifically for extracting keyphrases. The ZS + Expert Context prompt requires the model to take on the role of an ontology expert to extract keyphrases. Lastly, the ZS + Task Context prompt provides a detailed description of the extraction task, outlining the description of keyphrases, and asking the model to identify these keyphrases within the document. All previously mentioned prompts do not use any training example and are thus also called zero-shot (ZS) prompts.
Few-shot (FS) prompts, on the other hand, utilize a specific number of examples from the training data to guide the extraction process. This study uses prompts with one, three, and five examples. These examples can be fixed, randomly selected, or chosen based on the closest embeddings to the training data. The n-Fixed prompts use n training examples sampled from the training data and those are ifxed during all test examples whereas n-Random prompts use n randomly selected examples from the training data for each document in the test set. The n-Closest prompts select the top n examples in the training set based on the topical similarity of the document that needs to be predicted. Computing the similarity measure is done by embedding the training corpus using Sentence-BERT [20] and retrieving the top-n closest documents given the embedded prediction document using cosine similarity. The model ’all-mpnet-base-v2’ is used because of its superior quality across 14 diverse tasks from diferent domains1. These categorically diferent prompts allow for a flexible and comprehensive approach to concept extraction, accommodating varying levels of context and specificity required for diferent tasks.
Since each LLM model is trained on datasets specifically formatted as a chat history with user and assistant roles, each model needs a diferent chat template e.g. a prompt for Llama2 model needs to start with a special token "<s>" and each user prompt is encapsulated in instruction tokens "[INST]" and "[/INST]". Thus, only the system, user, and assistant phrases are defined, and the final prompt is generated by applying the correct chat template for the model currently being executed. This is done by executing the apply_chat_template function of the Huggingface tokenizer2.
4.1. Dataset In this section, the ConExion approach is evaluated on two common datasets. All experiments are executed on a server running RedHat with 256 GB of RAM, 2x64 CPU cores (2.6 GHz), and 2 Nvidia A100 (40GB) graphics cards (depending on the model size, only 1 GPU is used).
This study utilizes two publicly available datasets: Inspec [21], and SemEval2017 [22]. For each dataset, documents without ground truth keyphrases were filtered out, and the statistics presented in Table 2 are based on the filtered documents used in our approach. The Inspec 3 dataset comprises abstracts from 2,000 English scientific papers in the domains of Computers, Control, and Information Technology, published between 1998 and 2002. The SemEval-20174 dataset contains abstracts from 500 English scientific papers from ScienceDirect open access publications. The papers topics span computer/material science and physics. Concepts were annotated by student volunteers and subsequently validated by expert annotators. The training and testing split was included to ensure a fair comparison with other traditional models. Additionally, examples from the training data were utilized in the prompts for few-shot learning purposes, where the model generates outputs based on contextual examples provided during prompting. 4.2. Evaluation Metrics This section outlines the metrics used for evaluating concept extraction systems and presents the results on commonly-used datasets. The performance of our concept extraction approach is assessed using standard metrics: Precision, Recall, and 1-score [16]. Precision is calculated as the proportion of correctly predicted concepts out of all predicted concepts. Recall measures the proportion of correctly predicted concepts relative to the total number of concepts in the ground truth. 1-score is the harmonic mean of Precision and Recall.
Many other works only report precision, recall, 1-score at whereas is usually set to 5, 10,
and 15 [
Precision, recall, 1-score at are computed as follows given a list of predicted concepts = (1, 2, . . . , ) and target concepts ′ = (′1, ′2, . . . , ′):
The Precision at top- ( @) is defined as: @ = | : ∩ ′|
| : | @ = | : ∩ ′|
|′ : | Recall (@) measures the proportion of correctly matched concepts among all ground truth concepts: whereas ( : ) represents the top concepts according to the confidence score.
Especially for the use case of ontology evaluation, not the top- concepts are of interest but all concepts that are relevant to the document. Thus, we cannot rely on methods that only do the ranking of potentially many concepts but leave the decision of a good threshold to the user. As a result, only precision, recall, and 1 are important and meaningful measures in this context. 4.3. Selected Large Language Models In these experiments, we evaluated a diverse set of large language models to ensure coverage across both open and closed models, varying in scale and architectural design. Specifically, we selected: Llama-27b-chat-hf6, Llama-2-13b-chat-hf7, Llama-2-70b-chat-hf8, Llama-3-8B-Instruct9, Llama-3-70B-Instruct10, Mistral-7B-Instruct-v0.311, Mixtral-8x7B-Instruct-v0.112, as well as GPT-3.5 Turbo13. 4.4. Evaluation Results The evaluation of our approach was conducted using various types of prompts and diferent large language models (LLMs) to extract concepts from the Inspec dataset. The performance metrics considered are Precision, Recall, 1-score, 1@5, 1@10. The average number of extracted concepts ( ) is included as well. This metric provides insight into how well the model aligns with the expected number of concepts typically found in the dataset. By comparing the average number of extracted concepts to the average number of present concepts in the dataset, we can better understand the model’s extraction behavior and its tendency to over-extract or under-extract concepts. In the Inspec test dataset, the average number of concepts is 5.55. Therefore, reporting helps in evaluating whether the model’s output is consistent with this benchmark, ensuring a meaningful comparison of extraction performance across diferent models and prompts.
Table 3 presents the results for diferent zero-shot (ZS) prompts on the Inspec dataset. In these experiments, we tested several variations of the search terms (e.g., keywords, keyphrases, concepts, entities, topics) to see how diferent prompt wordings afected performance. These were selected based on their frequent appearance in prior literature on keyphrase and concept extraction. The results indicate that the Llama3 70B model, when used with the search term keyphrases, achieved the highest 6https://huggingface.co/meta-llama/Llama-2-7b-chat-hf 7https://huggingface.co/meta-llama/Llama-2-13b-chat-hf 8https://huggingface.co/meta-llama/Llama-2-70b-chat-hf 9https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct 10https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct 11https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 12https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1 13https://platform.openai.com/docs/models/gpt-3-5-turbo Precision, 1, 1@5, and 1@10 scores. This suggests that this particular combination is highly efective for concept extraction in this context. On the other hand, the GPT-3.5 Turbo model with the search term keywords showed the highest Recall. This result may be due to the GPT-3.5 Turbo model being trained on datasets similar to Inspec, which could impact the generalizability and fairness of the evaluation when compared to other open-source models.
Based on the results of Table 3, the Llama3 70B model was fixed, and term keyphrases is selected. Table 4 evaluates the efect of diferent zero-shot and few-shot prompt designs. For the ZS + Domain prompt (adding information to only extract keyphrases related to a domain), the Precision, Recall, and 1 scores decreased. This shows that the model focuses on keyphrases specific to the given domain, leading to the exclusion of some relevant keyphrases present in the ground truth but not directly related
to the specified domain. For example, when using simple keyphrase extraction, keyphrases such as "graphical user interface" and "scanning data" were extracted. However, when the domain-specific prompt was used, these keyphrases were not extracted. However, this approach ensures that the extracted keyphrases are highly relevant to the specified domain, which can be advantageous depending on the application requirements.
Table 4 illustrates the impact of diferent context prompts on the evaluation metrics. The incorporation of context into the zero-shot prompts improved the evaluation results. The zero-shot prompt with the task description showed the highest 1, 1@5, and 1@10 scores. This suggests that providing the model with a clear task description enhances its ability to extract relevant keyphrases more accurately.
Additionally, Table 4 presents the evaluation results for diferent few-shot prompts. The use of random examples improved 1, 1@5, and 1@10 scores, suggesting that variability in the examples enhances the model’s generalization and keyphrase extraction performance. The efect of increasing the number of examples was also analyzed. While incorporating one example generally improved performance, adding more examples beyond three did not result in significant improvements. Furthermore, the impact of using the closest documents based on embeddings was evaluated. The few-shot 5-closest prompt achieved the highest Precision score, indicating that carefully selected examples can lead to better precision in keyphrase extraction. Overall, the diferences between the diferent few-shot prompts are not huge. This shows that it is important to provide an example, but the main advantage is that the model can train on how the output should look and which words are of interest.
Table 5 presents the overall evaluation results, showing the Precision, Recall, 1, 1@5, 1@10, and
an average number of extracted keyphrases ( ) for each dataset. The evaluation covers a broad
spectrum of models, including EmbedRank [14], MultPAX [15], Pyate [24], RAKE [
For the baseline models, various models including MultPAX [15], PyATE [24], RAKE [30] were employed, alongside several models using the PKE library [25], including FirstPhrases, KPMiner, Kea, MultipartiteRank, PositionRank, SingleRank, TextRank, TfIdf, TopicRank, and YAKE. The use of these libraries ensured that all model versions were fixed, facilitating precise replication of our results.
In this paper, an approach for concept extraction from abstract documents using pre-trained large language models (LLMs) was presented. Our method addresses the challenging task of extracting all present concepts related to a specific domain, rather than just the keyphrases summarizing the important information discussed in a document. Comprehensive evaluations of two widely used benchmark datasets demonstrate that our approach performs better than state-of-the-art models. Our emphasis on reproducibility has ensured that the findings have been reliably replicated across various models. One limitation of this work is its reliance on exact lexical matching to filter concepts from the generated output. While this approach ensures that only terms present in the input document are retained, it fails to account for situations where an LLM generates semantically accurate concepts that do not have an exact match in the text. As a result, relevant concepts that capture the meaning are going to be discarded, negatively afecting recall and the overall evaluation.
In the future, we plan to create datasets annotated by domain experts in the field of Materials Science and Engineering (MSE). This will allow us to further test and refine our ConExion models, ensuring they are robust and efective across a broader range of scientific domains. We also want to explore more methods on how to restrict the LLM models to only produce specific tokens (those that appear in the 14https://github.com/ISE-FIZKarlsruhe/concept_extraction document). Additionally, we are interested in investigating the efects of continued pre-training and task-specific instruction tuning of LLMs on concept extraction performance. This could provide further insights into how domain adaptation influence model behavior on this task.
The authors thank the German Federal Ministry of Education and Research (BMBF) for financial support of the project Innovation-Platform MaterialDigital through project funding FKZ no: 13XP5094F (FIZ). MatWerk funding acknowledgement This publication was written by the NFDI consortium NFDIMatWerk in the context of the work of the association German National Research Data Infrastructure (NFDI) e.V.. NFDI is financed by the Federal Republic of Germany and the 16 federal states and funded by the Federal Ministry of Education and Research (BMBF) – funding code M532701 / the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) - project number 460247524.
During the preparation of this work, the author(s) used Chat-GPT-4 and Grammarly in order to: Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. extraction using prompt, 2023. URL: https://arxiv.org/abs/2305.04490. arXiv:2305.04490. [24] K. Lu, Pyate: Python implementation of term extraction algorithms, https://github.com/ kevinlu1248/pyate/tree/master, 2020. Accessed: 2024-07-03. [25] F. Boudin, Pke: an open source python-based keyphrase extraction toolkit, https://github.com/ boudinfl/pke, 2016. Accessed: 2024-07-03. [26] S. R. El-Beltagy, A. Rafea, Kp-miner: Participation in semeval-2, in: Proceedings of the 5th international workshop on semantic evaluation, 2010, pp. 190–193. [27] I. H. Witten, G. W. Paynter, E. Frank, C. Gutwin, C. G. Nevill-Manning, Kea: Practical automatic keyphrase extraction, in: Proceedings of the fourth ACM conference on Digital libraries, 1999, pp. 254–255. [28] F. Boudin, Unsupervised keyphrase extraction with multipartite graphs, arXiv preprint arXiv:1803.08721 (2018). [29] X. Wan, J. Xiao, Collabrank: towards a collaborative approach to single-document keyphrase extraction, in: Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), 2008, pp. 969–976. [30] C. Surfer, Rake: Rapid automatic keyword extraction algorithm, https://github.com/csurfer/ rake-nltk/tree/a80f633098dba19c409cb1778206189d8573696a, 2018. Accessed: 2024-07-03.