We present Martialis1, a framework designed to enhance the performance of Large Language Models (LLMs) in domains-specific tasks (e.g. medical, legal) by implementing an ontology-based data representation approach. We designed the framework to be almost transparent to the end user: once set up with domain-relevant documents and an ontology, it enables both complex reasoning and domain-specific text generation. That is made possible by our novel information extraction pipeline that improves existing Retrieval Augmented Generation (RAG) techniques with a Domain Specific Knowlegde Graph - inferred from the documents - and a sanity check on the output - inferred from the ontology. This dual-layered approach ensures accuracy and relevance, addressing common limitations in existing solutions. The Martialis framework has been rigorously evaluated through collaboration with domain experts, comparing its performance against similar state-of-the-art systems. Results indicate improvements across key metrics, and speed-up in the eficiency of executing user tasks.
The development of Natural Language Processing (NLP) systems has rapidly advanced in recent years
with the rise of Large Language Models (LLMs), specifically generative ones, which have demonstrated
great capabilities in understanding and generating human-like text. These advancements have opened
new possibilities for addressing a variety of linguistic tasks. However, despite their potential,
generative LLMs face significant challenges, including aligning outputs with user intent and mitigating
hallucinations [
The introduction of ontology-based knowledge representation could pave the way for the development of
Intelligent Information Systems (IIS) [
In this context, although several preliminary studies [
To evaluate our framework, we collaborated with a team of medical experts from a hospital in Rome to benchmark the performance in the healthcare domain. Specifically, we assessed its ability to answer medical questions and generate various domain-specific documents derived from Electronic Clinical Records (ECRs).
The paper is structured as follows: in Section 2, we review the existing literature on the automatic or semi-automatic construction of KGs from unstructured documents, as well as frameworks implementing RAG pipelines and ontology-based compliance processes. In Section 3, we present the architecture of martialis, along with a case study in the healthcare domain. Section 4 details the dataset composition and the evaluation process used to assess our framework. Lastly, in Section 5, we discuss the limitations and challenges of our framework and propose future directions for improvement.
In this section, we review existing works that directly relate to the core features and theoretical
foundation of Martialis: (i) Text2KG, for extracting structured knowledge from unstructured text to
create KGs in an automatic or semiautomatic way; (ii) KG RAG, namely methods that combine LLMs
with KGs for context-aware text generation; (iii) and Ontology-Augmented Text Generation approaches
that enhance generation by integrating ontologies for validation and consistency. Our objective is to
analyze existing frameworks that claim practical, implementable outcomes, emphasizing contributions
with concrete applications or open-source solutions. These works have been assessed by evaluating six
key criteria: code availability (open source), innovative methodology, adaptability (to multiple domains),
correctness, repeatability, and support the generation (of complex and detailed responses from KG).
Text to KG. The task of transforming unstructured text into a Knowledge Graph has always been
done by experts in the domain due to the high level of abstraction required to comprehend a text and
structure it into nodes and relationships. To automate this task several attempts have been made by
using various machine learning techniques [
RAG with Knowledge Graphs. Hallucination is the most common and impactful limitation of
LLMs [
Ontology Augmented Text Generation. Ontologies have always been the pillar for structuring
data and enabling reasoning. The advent of LLM has drawn increasing interest in combining language
generation capabilities with the structured knowledge represented in ontologies [
martialis is a next-generation, open-source IIS designed to perform complex linguistic tasks in a specific domain of knowledge, addressing a significant gap in the current literature: directly supports advanced text generation martialis (i) answers complex questions regarding the selected domain even if they require reasoning steps and (ii) generates structured text for that domain.
The complete architecture of the framework is presented in Figure 1. The design is modular,
comprising the following key components, each of which can be individually replaced with customized
implementations if necessary: Automatic KG-Extractor, Advanced Retrieval-Augmented-Generation,
and Ontology-Based Validator. The implementation leverages the Python libraries Llamaindex3 and
Langchain4, with GPT-4o [
In the remainder of this section, we will detail the required inputs, elucidate the operational logic of martialis using the healthcare domain as an illustrative example (also employed for validation), and provide an in-depth analysis of martialis’ modules.
After identifying the target knowledge domain, it is necessary to indicate to martialis, through a
configuration file, the source folder of the available documents. If the user wants to generate some
advanced domain-specific artifact, they have also to provide an ontology that describes that artifact.
Domain Documents The documents inherent to the specified domain are the main pillar for
martialis. We have no constraints on the data format, as long as it contains meaningful text. In the
healthcare domain, we used several plain text and .pdf files that describe patient hospitalizations. The
documents are parsed and processed into a Vector Store [
As illustrated in Figure 2, when the user adds domain-specific documents or an ontology to the
martialis folder—either for the first time or incrementally—the framework automatically initiates
a preliminary processing step consisting of the following actions: (i) the documents are parsed and
stored in a Vector Database; (ii) a domain-specific Knowledge Graph is automatically generated from
the documents using the Automatic KG-Extractor module. This KG can then be queried to retrieve
precise and relevant information required for task completion; (iii) the ontology, which serves both
as a foundation for extracting the entities necessary for text generation tasks and as a reference for
correctness validation in the final output, is parsed and prepared for use within martialis.
KG-Automatic Extractor Module The Domain KG must be an instance-level Knowledge Graph.
Consequently, unlike ontologies or schema-level knowledge, it is typically impractical to construct it
manually, even with the assistance of domain experts. Given that the objective of this work is to make
martialis’ real-world application feasible, we deemed it essential to address the need for automatically
building its own Domain KG. Since the majority of knowledge and information about organizations
remains stored in unstructured text documents, martialis automatically constructs the Domain KG
from the unstructured documents provided. To implement the domain KG extraction based on [
martialis can understand if the user is asking a question about the domain or wants to generate some advanced text. It routes the user prompt to the Advanced-Retrieval-Augmented-Generation module or the Ontology-Based-Validator module that we will present below in detail.
question answering. It is capable of producing the answer relies on a chain of thoughts [
The Domain KG enables the LLM to follow a more strict path in generating the final output, efectively simulating an improvement in the LLM’s reasoning capabilities. Its function is carried out in the augmented generation phase of the RAG pipeline and is iterative. In the standard RAG procedure, once the enriched prompt is created, it is given to the LLM as input for generating a response. However, the accuracy or relevance of the response is not verified using external tools. For complex language tasks, producing the desired response may not be straightforward, which is why the Domain KG is used to add a reasoning and abstraction layer. During the augmented generation phase, the specialized LLM generates a preliminary output based on the enriched prompt. This output is bounded by the logic of the Domain KG. An example from the healthcare domain would be:
Q: "Which medications were prescribed to the patient Patient_008? Provide a comprehensive list" A: Sinemet 100/25 (1 tablet 3 times daily: 7:30 AM, 12:30 PM, 5:30 PM), Zoloft 50 mg (1 tablet in the evening).
Ontology-Based-Validator Module. This module is capable of generating text artifacts that require deep comprehension of a specific knowledge domain. The final text output is validated against the provided ontology. When this kind of request is detected, martialis uses prompt engineering to retrieve from the ontology the relevant class and properties for the specific task (e.g. medical_condition, treatment_outcome, patient_information). Those extracted objects are integrated into a chain of prompts to generate the requested text. The text generated in this way is then passed to the validation step: it is converted into a Resource Description Framework KG using the KG-Automatic Extractor module and checked against the ontology. A score from 0 to 1 based on compliance with the ontology is computed by checking the percentage of constraints on the class and the attributes that are present or missing. In this way, martialis can show the user a numeric value for confidence in the generated output.
= ∑︀=1 () + ∑︀=1 () + (1) Where: is the generated text, () is 1 if property is present in , and 0 otherwise. () is 1 if class is present in , and 0 otherwise. is the total number of properties. is the total number of classes. An example we generated with score 0.63 is:
Class Extracted: ment_outcome [...] Q: "I need a detailed summary of the clinical history for the patient. patient_008" patient_information, medical_condition, therapy, treatA: The patient, patient_008, female, born on May 26, 1952, is being treated at the Neurology Unit of Hospital. She initially presented with tremors and was prescribed Artane, which she later reduced due to internal tingling and nausea. Subsequently, she was switched to Sinemet, which led to a significant improvement in her tremors [...].
To validate our framework, we employed two types of evaluations, automatic and human, tailored to the structure and objectives of our approach. For tasks focused on answering specific questions, we utilized a single automatic evaluation approach. This involved assessing both the reliability of the extracted context and the consistency of the generated responses about the posed questions. For the text generation tasks, automatic validation was conducted by comparing the knowledge graph derived from the final answer against the ontology provided as the initial context for the task. Additionally, for the text generation task, we performed a human evaluation. This involved developing and administering detailed questionnaires to medical experts to assess the quality and domain-specific relevance of the generated outputs.
Dataset. The dataset used for our case study and testing consists of 32 clinical records5, which were collected and selected by an expert medical team from a hospital in Rome. This selection was carried out following approval from the hospital’s ethical committee to process the data strictly for research purposes. The clinical records have been fully anonymized through PII masking with Named Entity Recognition (NER) implemented through Llama Index. To maintain the integrity of the information, the clinical records were left unprocessed in their original language, Italian. As a result, any information produced by the model is also in Italian.
Automatic Evaluation for QA task. We designed a test involving eight questions of increasing complexity. These questions required the extraction and re-elaboration of various types of medical information to produce accurate answers. Each of the eight questions was applied to all test documents available in our dataset. In parallel, we collaborated with the team of domain experts to extract the correct answers for each of the eight questions from the clinical records. These expert-verified answers 5Datset for QA, clinical record, and all evaluation results are open and accessible at the following link: Healthcare_Dataset_Martialis served as the ground truth, allowing us to directly compare the outputs of our framework with the expected responses. Finally, we selected and calculated four automatic metrics using the DeepEval framework6. Selected metrics are the following ones: (i) Contextual Precision. It evaluates the quality of a ranked list of retrieved nodes by considering both their relevance and their position in the ranking. The metric is computed using the total number of relevant nodes in the retrieval context and the relevance of each node; (ii) Contextual Recall. This metric assesses the quality of the RAG pipeline’s retriever by measuring the extent to which the retrieved context aligns with the expected output; (iii) Faithfulness. It evaluates whether the model generates factually correct information by comparing the actual output of the LLM to the provided context; (iv) Answer Relevancy. This metric measures the quality of the RAG pipeline’s generator by evaluating how relevant the actual output of the LLM is compared to the provided input.
The results of this automatic evaluation were aggregated using two complementary approaches: (i) Overall Performance Analysis; (ii) Performance by Question Complexity Analysis. The overall performance analysis shows strong results across the metrics. The Contextual Precision (80.36%) and Contextual Recall (83.07%) demonstrate the model’s ability to efectively understand and retrieve relevant contextual information. The Faithfulness score (80.64%) indicates that the model’s responses are fairly faithful to the original content. The Answer Relevancy score (88.84%) suggests that the answers are highly relevant to the questions asked.
As for question complexity analysis, Contextual Precision shows the highest degree of variability with relevant low values for question 2 and question 5. Notably, on most complex questions (e.g. questions 6, 7, and 8) it seems to be very precise in exacting context reaching values above 90%. Contextual Recall shows a similar situation: variability is high, we can note that as for Contextual Precision, we have the lowest values for questions 2 and 5. Again, greater values are distributed in the last three questions exceeding the threshold of 90%. Furthermore, Faithfulness shows a diferent distribution: variability is lower, and the smallest values are concentrated in questions 1 and 7. Question 8 shows still a great result. Finally, Answer Relevancy shows the best results and the lowest variability as it is the only metric having all values above the threshold of 75%. In particular, for question 1 we reached 100% of answer relevancy.
Automatic Evaluation for Text Generation. We evaluated the model’s outputs in two distinct scenarios: (i) generating the patient’s clinical history, and (ii) drafting the patient’s discharge letter. In both cases, the initial stage of response generation involves extracting key entities and relationships from the ontology preliminarily extracted. Once the text-based response is generated, a knowledge graph is constructed from the output using the KG-Automatic Extractor Module. This module identifies the entities and relationships present in the generated text and maps them into a structured knowledge graph format. The resulting graph is then compared against the existing ontology to assess its alignment with the ground truth.
The evaluation metric used was the score of matching elements (both entities and relationships)
between the extracted knowledge graph and the reference ontology presented in Section 3. The results,
reflecting the model’s ability to preserve the semantic integrity of the data during text generation,
shows low values but this outcome align with expectations. This evaluation is not absolute; rather, it
serves as a trigger to initiate the response revision process
Human Evaluation for Text Generation. To carry out and report the human evaluation of the
text generation by martialis, we followed the guidelines in [
Generation Approaches Used. We compared the quality of draft documents written by a domain expert
with those generated by an LLM using diferent approaches. In particular, we used the martialis’
generation approach, a prompt guided by a domain ontology, and standard zero/few-shot prompts.
For each approach, the LLM used is GPT-4o [
As far as the automatic evaluation, results are solid; the insights suggest that LLMs, if correctly guided, might be truly useful in drafting semi-structured documents like those in our case study. This assumption is reinforced considering the human evaluation, where there is a clear gap in documents quality between the ones generated without guidance (zero-shot prompt) and the ones generated with guidance (fewshot prompt and martialis). Furthermore, since the performances of martialis and few-shot are complementary, it seems interesting to explore martialis’ RAG on KGs approach in synergy with classical few-shot prompting techniques.
We argue that martialis has the potential to stand out for completing complex linguistic tasks. We intend to continue working in this direction by (i) releasing evolutionary updates working on each module and (ii) investigating, and eventually fine-tuning, open LLMs to replace GPT-4o, ensuring accessibility and repeatability.
The work of Mattia Macrì has been supported by the PhD fellowship Pubblica Amministrazione DM118 - CUP B83C22003460006. The work of Marco Calamo and Filippo Bianchini has been supported by the Next-Generation EU (Italian PNRR - M4 C2, Invest 1.3 - D.D. 1551.11-10-2022), named PE4 - MICS (Made in Italy - Circular and Sustainable).
During the preparation of this work, the authors used GPT-4o-2024-08-06 in order to: Grammar and spelling check.