This project explores methods to enhance the accuracy and reliability of Large Language Models (LLMs), which are prone to factual inaccuracies and hallucinations. The focus lies on improving accuracy by integrating Case-Based Reasoning and to address operational eficiency with pre- and post-processing techniques. Additionally new techniques for retrieving, adapting and retaining information will be researched. PCWrEooUrckResehdoinpgs ISSNc1e6u1r-3w-0s0.o7r3g
Language serves as a cornerstone for human communication and self-expression, and it also plays a
crucial role in how humans interact with machines. The increasing need for machines to efectively handle
complex language tasks, such as translation, summarization, information retrieval, and conversational
interactions, has driven the demand for generalized language models. Recent advances in language
modeling, primarily driven by the development of transformer architectures, increased computational
power, and the availability of massive training datasets, have led to significant breakthroughs. These
advancements have revolutionized the field by enabling the creation of Large Language Models (LLMs)
that can achieve near-human performance on a wide range of tasks. LLMs have emerged as
state-ofthe-art AI systems capable of processing and generating human-like text, demonstrating impressive
abilities in coherent communication and generalization across various tasks [
This PhD research focuses on enhancing the performance and eficiency of LLMs through improved
information retrieval and processing techniques. While LLMs have shown remarkable capabilities in
various language tasks, they often struggle with knowledge-intensive tasks and can generate incorrect
information (hallucinations) [
A basic RAG approach involves: 1) Indexing: extracting, segmenting, embedding, and storing data
chunks in a vector database ("Retrieve-Read" [
This naive approach faces challenges like imprecise retrieval and the potential for hallucination,
irrelevant, toxic, or biased outputs. Efectively integrating retrieved information is also dificult [
This work is grounded in the principles of Case-Based Reasoning, a powerful problem-solving paradigm
that leverages past experiences to address new challenges. Case-based reasoning has been formalized
as a four-step process [
Knowledge Graphs (KGs) represent knowledge as nodes and edges [
This flowchart-like notation empowers stakeholders to design, manage, and realize business processes eficiently. Its independence from specific implementation environments ensures flexibility and adaptability.
Previous work with BPMN-models is found in [
In [
The first approaches in this thesis can be assigned to the process-oriented CBR research. Here the idea is to work with Business Process Models and improve RAG systems with eficient pre-processing.
This research primarily focuses on improving the performance of Retrieval Augmented Generation (RAG) systems, particularly in the context of Business Process Models. A core objective is to improve the retrieval component of RAG by minimizing redundant information, accurately predicting necessary information, and optimizing data structures for eficient retrieval. Additionally, this work aims to advance the automated generation and evaluation of natural language descriptions from BPMN models, including developing robust quality assessment mechanisms and error correction techniques. Furthermore, this research explores several related objectives to improve the overall eficiency and reliability of language models and knowledge integration. This includes investigating methods for automatically extracting and structuring knowledge from text into knowledge graphs, and evaluating the efectiveness of integrating these knowledge graphs into the RAG retrieval process to mitigate hallucinations. Finally, it examines the feasibility of fully automated evaluation metrics for LLM outputs and how CBR principles can be leveraged to enhance RAG performance by learning from past retrieval and generation experiences.
Latest Work: A paper on using CBR generated explanations for business process models by using clever pre-processing to improve the accuracy while increasing the scalability of the input size, which has been accepted to ICCBR 2025.
LLMs Used: Diferent iterations of Llama 3 and Mistral were used in experiments leading to the paper mentioned above. The focus lied on smaller models, which can be run locally on a conventional computer to ensure accessibility and easy reproducibility of any result generated.
Prompt Engineering: Prompt phrasing was refined through pre-testing, with the best-performing versions used in the final experiment. These prompts are likely to be reused or adapted in future studies, with domain-specific modifications.
Construction of a Case-Base: For the experiments in the paper, the case-base was written by hand using parts of business process models and writing a corresponding descriptive text.
Retrieval of cases: In pre-testing, diferent retrieval strategies were used. While embedding-based similarities did not satisfy the needs for our experiments, because cases were too similar and only a small selection of keywords decided the usefulness of a case, another metric was used in the paper. A taxonomy-based metric considering keywords to ensure retrieving a relevant case for the RAG system.
ICCBR 2025: In my first and only submission, I presented a novel approach to addressing the challenges associated with the description of large business process models. Recognizing the inherent complexity and scale of these models, which often hinder eficient analysis, I developed a methodology that helps RAG techniques. A core component of this methodology is the implementation of eficient preprocessing strategies. These strategies are crucial for transforming the raw business process model data into a format that is both useful for efective retrieval and the generation of coherent and contextually relevant descriptions.
Building upon the foundational work presented in my submitted paper, my research trajectory focuses on a significant refinement and enhancement of the proposed methodology. Specifically, a key objective is to elevate the precision and eficacy of the pre-processing stage. To achieve this, I intend to integrate more sophisticated algorithmic approaches, moving beyond the initial strategies. This includes working with a wider range of business process models, that are not well-modeled.
Furthermore, a significant advancement will be the implementation of an automated evaluation framework for the generated textual descriptions. This framework will move beyond subjective human assessments and introduce quantifiable metrics to rigorously assess the quality of the generated outputs. I plan to utilize Knowledge Graphs (KGs) as an information repository to develop novel retrieval methods. This approach aims to significantly enhance the relevance of retrieved information. Furthermore, I will explore the integration of the retain phase, dynamically expanding the KG with new knowledge, to continuously improve retrieval accuracy.
Possible research questions regarding my future work are:
1. How can we optimize the workload distribution between pre-processing algorithms and LLMs
for maximum eficiency and accuracy?
2. How can the principles of Case-based Reasoning be efectively leveraged to improve RAG
performance?
3. Can fully automated evaluation metrics reliably assess the quality of LLM outputs without the
need for human intervention?
4. What are the most efective methods for automatically extracting knowledge from textual
documents and representing it in a structured knowledge graph format?
5. Can the integration of knowledge graphs or textual graphs into the retrieval process of LLMs
efectively mitigate the risk of hallucinations?
Question 1 and 2 were already partly explored in [
Question 4 focuses on the challenges of information extraction, including named entity recognition, relation extraction, and event extraction, which are crucial for identifying and classifying key components and relationships within the text. Efective knowledge representation plays an important role in ensuring the accuracy, completeness, and consistency of the constructed knowledge graph. Question 5 investigates the potential of integrating knowledge graphs or textual graphs into the retrieval process of Large Language Models (LLMs) to efectively mitigate the risk of hallucinations. Hallucinations, the generation of factually incorrect or nonsensical information by LLMs, pose a significant challenge to their reliability and trustworthiness.
By incorporating structured knowledge from knowledge graphs or the contextual relationships captured in textual graphs, the retrieval process can be enhanced. This can involve using graph-based algorithms to retrieve relevant information or improve context understanding.
During the preparation of this work, the author(s) used Google Gemini, Grammarly for the purpose of: Grammar, spelling check, minor Paraphrase and translations. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.