Generative AI applications must integrate users' personal information into the response generation process to ofer an advanced user experience. One of the most efective methods for obtaining accurate and current user information is by capturing this data from AI interactions. This paper examines conversational knowledge capture using ontology and knowledge-graph approaches. We propose enhancing the large language model's (LLM) ability to capture precise and relevant information by training it with a subset of the KNOW ontology, which models personal knowledge. Our paper details the ontology-guided training process and evaluates the success of knowledge capture using a specially constructed dataset. Additionally, we emphasize the importance of privacy in handling personal information and investigate the implementation of knowledge capture with on-device language models. Our findings highlight the potential of on-device solutions to efectively capture personal knowledge while preserving user privacy.
CEUR ceur-ws.org
Expectations for the quality and sophistication of human-AI interactions are steadily increasing. Generative AI applications are now expected to recognize users, understand their characteristics and preferences, and augment this information to enhance interactions. A fundamental challenge in providing this level of user experience is capturing up-to-date knowledge about the user through conversations. This process of identifying and recording personal knowledge and preferences from user interactions is defined as conversational knowledge capture (CKC).
CKC presents several critical challenges. Key issues include determining which knowledge
from conversations should be captured, how the captured knowledge should be represented,
whether the captured knowledge requires updating previous records, and whether the
knowledge is duplicate. Fortunately, the emergence of neurosymbolic approaches, which combine
large language models (LLMs) and symbolic AI, has provided researchers with new perspectives
to address these challenges [
Another significant challenge related to CKC is ensuring the privacy of captured sensitive
knowledge. Personal data, which is entirely owned by the user, should be considered vulnerable
CEUR
Workshop
Proceedings
if it is sent to the cloud. On-device AI solutions, which do not require any data to leave the user’s
device, provide the most appropriate response to privacy needs [
In this paper, we explore the feasibility of generating personal knowledge graphs on-device through conversational interaction. Our approach focuses on ontology-guided knowledge extraction from prompts in the form of subject-predicate-object triples1. We have investigated various methods to enable the underlying language model to comprehend a predefined ontology, ensuring efective personal knowledge-graph generation. Subsequently, we selected the most suitable method based on the requirements of on-device execution. Utilizing a specially designed dataset, we evaluate the efectiveness of this method, emphasizing its strengths and identifying potential areas for improvement.
The structure of this paper is as follows: Section 2 discusses various approaches, including in-context learning and fine-tuning for ontology-guided knowledge capture, and focuses on the fine-tuning approach due to its suitability for on-device execution. Section 3 describes the experimental setup, presenting the development framework, language model selection, and the ontology and dataset creation process. Section 4 outlines our performance evaluation framework and the test results. Finally, Section 5 concludes the paper and suggests future directions.
In the literature, language models have demonstrated their capability to transform unstructured
text into knowledge graphs [
An alternative to in-context learning involves training a language model with a predefined ontology so that the model internalizes it. There are two strategies to consider: pretraining the LLM on the ontology or fine-tuning it. This paper does not explore pretraining due to its extensive data, computational resource, energy, and time requirements. Additionally, pretraining 1https://www.w3.org/TR/rdf12-concepts/ does not ofer a flexible response to ongoing changes or expansions in the ontology. Therefore, this paper focuses on fine-tuning as a method to train language models on personal ontologies, highlighting its advantages in feasibility and maintainability.
Fine-tuning is a process whereby a pretrained language model is further trained on a specific dataset to tailor its capabilities to a particular task. In our study, the language model is expected to understand the ontology classes and their properties, and use them to populate a knowledge graph from user prompts. The first step involves preparing a fine-tuning dataset, which includes user prompts, system prompts, and expected model responses for each concept in the ontology. This dataset is used to fine-tune the language model, which is then evaluated by testing it with new prompts to assess the efectiveness of the CKC process.
The following points highlight the key aspects of ontology fine-tuning: • The training dataset’s coverage and diversity are vital for successful fine-tuning. These characteristics greatly influence the LLM’s ability to capture knowledge efectively. Details about the dataset and how it is constructed are discussed in Section 3.4. • The training dataset must include a variety of examples for the predefined ontology.
Research related to the structure of the examples prepared for ontology concepts is detailed in Section 4. • If the language model encounters a user prompt that is not relevant to the predefined ontology concepts, it should not attempt to capture knowledge. Therefore, the dataset should also contain suficient out-of-context samples to enable the language model to distinguish between relevant and irrelevant information for capture.
The methods suggested in this paper have been implemented using the Apple MLX framework
[
Ontology fine-tuning has been conducted using the parameter-eficient QLoRA adapters
[
Due to the constraint of on-device execution, our study does not use state-of-the-art
largeparameter cloud-based language models. Instead, we opted for a relatively low-parameter model
with proven efectiveness across diverse domains. Based on its performance in the Hugging
Face Open LLM Leaderboard [
Our study is inspired by KNOW[
Because of the requirement that each element in the ontology be associated with a diverse set of prompt and response samples within the training dataset, our research focuses on a specific subset of the KNOW ontology. This subset concentrates on core family relationships with four ontology classes, eleven object properties, and one data property. A visual depiction of this subset is presented in Figure 1.
spouse child parent partner knows sister mother sibling brother father {<partner{}} {<knows{}} {<knows{}} {<knows{}} {<sibling{}} {<parent{}} {<knows{}} {<sibling{}} {<parent{}}
Person Thing name:string
sex
Sex =Female{} or Male{}
mother =sex{} value Female{}
father =sex{} value Male{}
sister =sex{} value Female{}
brother =sex{} value Male{}
Female
Male
For a language model to efectively learn a predefined ontology and use it to perform knowledge extraction and capture, a robust and diverse training dataset is essential. Our paper focuses on a subset of the KNOW ontology that includes the concepts of ‘person’, ‘name’, ‘sex’, ‘child’, ‘father’, ‘mother’, ‘sibling’, ‘sister’, ‘brother’, ‘spouse’, ‘partner’, and ‘knows’. We created 145 manually crafted user prompts along with their respective ontology responses for training. Additionally, to manage inputs that fall outside these ontology concepts, we included 32 generic user prompts in the dataset. The composition of this training dataset, which consists of 177 user prompts, is illustrated in Figure 2. Concepts not associated with the ontology are labeled as the ‘none’ in the figure. Since each sample prompt usually includes multiple concepts, the chart shows more concept occurrences than prompts.
For the test set to be used in the evaluation, we generated 100 manual test prompts and their expected responses about family relations based on the television series ‘The Waltons’. The Waltons is a classic American television series that aired from 1972 to 1981, depicting the life and challenges of a close-knit family in rural Virginia during the Great Depression and World War II. The show focuses on the daily experiences, values, and growth of the Walton family, emphasizing themes of love, perseverance, and community. The composition of this test dataset, which consists of 100 user prompts, is illustrated in Figure 2.
Occurrences of Ontology Concepts in the Training and Test Datasets
The Turtle format was chosen for serializing the ontology population in our research because of its straightforward structure, readability, and prevalent use in existing pretraining datasets for LLMs.
Person
Name
Sex
Child Brother
Sister Sibling Mother Father Spouse Partner Knows
None 0 100
200
Training Test 300
Our research focuses on fine-tuning an on-device language model with predefined ontology concepts and capturing knowledge from user prompts that fit the ontology. This section will detail the fine-tuning approaches and the relevant performance evaluations.
One of our main objectives is to comprehend the training dynamics of ontology-guided knowledge capture. Consequently, we will not emphasize the optimization of fine-tuning hyperparameters. For a fair performance evaluation, all tests in this section were conducted using the default QLoRA hyperparameters specified in the MLX framework. To ensure consistency in the test results, each training session was configured to run for 18 epochs.
Firstly, we investigated whether training data associated with each ontology concept is necessary for the successful fine-tuning process. For instance, we evaluated the ability of a language model, trained with only one of two semantically related concepts (e.g., ‘brother’), to capture knowledge related to the concept that was not included in the training data (e.g., ‘sister’). During the evaluations, the generated prompt responses were processed triple by triple and compared against the ground truth established for the test set. The findings are presented in Figure 3.
Precision
As illustrated in Figure 3, a CKC QLoRA adapter trained exclusively on the ‘brother’ concept shows a significant decline in performance when tested with the ‘sister’ relation prompts. In contrast, an adapter trained on both the ‘brother’ and ‘sister’ concepts demonstrates excellent performance on the same test set. Although the language model’s pretraining is suficient to distinguish between the ‘brother’ and ‘sister’ concepts, our tests reveal that this is inadequate for efective knowledge capture.
The second issue we examined was whether it is necessary for each concept to be used in conjunction with other concepts in the fine-tuning dataset. To provide a concrete example, we assessed the performance of fine-tuning using samples that included only the ‘brother’ or only the ‘sister’ relationship when prompted with contexts where both the ‘brother’ and ‘sister’ concepts co-occur. We compared the performance of a knowledge capture adapter fine-tuned with samples containing only the ‘brother’ and ‘sister’ concepts against another adapter trained with the same data set, with extra samples where both ‘brother’ and ‘sister’ concepts appear together. The results of this comparison are presented below in Figure 4.
Performance of 'Brother-Sister' Relation vs. Two CKC Adapters
Precision Recall F1-Score 1.00 0.75 0.50 0.25 0.00 0.75 0.50 0.25
Performance 'Father-Son' & 'Mother-Daughter' Relation vs. Two Knowledge
Capture Adapters
Precision Recall F1-Score CKC adapter trained with separate 'father-daughter' and 'mother-son' relation prompts
CKC adapter trained with all 'fatherdaughter', 'father-son', 'motherdaughter', and 'mother-son' 0.75 0.50 0.25
Similarly, in Figure 5, we presented the knowledge capture performance of two diferent adapters, one trained with only ‘father-daughter’ and ‘mother-son’ prompts and the other trained with all ‘father-daughter’, ‘father-son’, ‘mother-daughter’, and ‘mother-son’ prompts. Both Figure 4 and Figure 5 demonstrate that having cross relations between ontology concepts in the training set increases knowledge capture performance. In the future, we aim to repeat these tests with various ontologies and language models to obtain more general results regarding this approach.
Performance of 'Brother', 'Sister', and 'Brother-Sister' Relations vs.
Two CKC Adapters
Precision Recall F1-Score CKC adapter trained solely with 'fatherdaughter', 'father-son', 'motherdaughter', and 'mother-son'
CKC adapter trained with the whole core-family ontology prompts Figure 7: CKC performance (precision, recall, and f1-score) of ‘father-daughter’, ‘father-son’, ‘motherdaughter’, and ‘mother-son’ relation prompts for two diferently fine-tuned QLoRA adapters CKC adapter trained solely with 'brother', 'sister', and 'brother-sister' relation prompts
CKC adapter trained with the whole core-family ontology prompts
Another aspect we researched in our study was the impact of the ontology size used in training on knowledge capture performance. In our previous tests, we trained the language model on specific ontology concepts and then examined the knowledge capture performance of the resulting adapter. At this stage, we compared the performance of a knowledge capture adapter trained solely on the ontology concepts present in the test set with an adapter trained on the entire core-family ontology. Our objective was to investigate whether adapters experience any degradation in performance as they are trained on more ontology concepts. In other words, we observed the scalability of ontology learning for the on-device language model we used. The results for the two diferent groups are presented in Figures 6 and 7. As seen from the ifgures, as the number of concepts in ontology training increases, there is a slight decrease in the performance of the knowledge capture adapter. However, the observed degradation within the scope of the tests does not appear to lead to a major scalability problem.
In the subsequent phase, we explored the optimal number of training epochs required to achieve maximum performance for the training set. For this analysis, we continued using the default MLX QLoRA hyperparameters, but trained the QLoRA adapter over various epoch lengths. We then conducted evaluations on the test set using each trained adapter, and the ifndings are presented in Figure 8.
Precision
Recall
F1-Score 1.00 0.75 0.50 0.25 0.00 9 18
27 Training epochs 36
As depicted in Figure 8, the success rate of the ontology population increases with longer training. However, considering the resource usage and energy consumption, we observe that 18 epochs is suficient for fine-tuning.
In this paper, we first explored on-device ontology-guided conversational knowledge capture and its importance in the generative AI domain. We then discussed the ontology approach and how to train an on-device LLM with ontology concepts. The language model was fine-tuned using a custom dataset focused on core family relationships, and we evaluated the model’s ability to learn personal ontology concepts.
Our findings indicate that fine-tuning is particularly efective for training an on-device language model with ontology concepts for conversational knowledge capture. In our future work, we aim to integrate the generated knowledge graph with the language model for knowledge utilization, combining the strengths of the neural and symbolic AI approaches.