Ontology-Guided On-Device Conversational Knowledge Capture with Large Language Models Tolga Çöplü, Arto Bendiken, Andrii Skomorokhov, Eduard Bateiko and Stephen Cobb Haltia, Inc. Abstract Generative AI applications must integrate users’ personal information into the response generation process to offer an advanced user experience. One of the most effective methods for obtaining accurate and current user information is by capturing this data from AI interactions. This paper examines conver- sational knowledge capture using ontology and knowledge-graph approaches. We propose enhancing the large language model’s (LLM) ability to capture precise and relevant information by training it with a subset of the KNOW ontology, which models personal knowledge. Our paper details the ontology-guided training process and evaluates the success of knowledge capture using a specially constructed dataset. Additionally, we emphasize the importance of privacy in handling personal information and investigate the implementation of knowledge capture with on-device language models. Our findings highlight the potential of on-device solutions to effectively capture personal knowledge while preserving user privacy. 1. Introduction Expectations for the quality and sophistication of human-AI interactions are steadily increasing. Generative AI applications are now expected to recognize users, understand their characteris- tics and preferences, and augment this information to enhance interactions. A fundamental challenge in providing this level of user experience is capturing up-to-date knowledge about the user through conversations. This process of identifying and recording personal knowledge and preferences from user interactions is defined as conversational knowledge capture (CKC). CKC presents several critical challenges. Key issues include determining which knowledge from conversations should be captured, how the captured knowledge should be represented, whether the captured knowledge requires updating previous records, and whether the knowl- edge is duplicate. Fortunately, the emergence of neurosymbolic approaches, which combine large language models (LLMs) and symbolic AI, has provided researchers with new perspectives to address these challenges [1, 2, 3, 4]. LLMs’ capabilities in natural language processing can be integrated with the knowledge representation and factual reasoning abilities of knowledge graphs, enhanced by the structure, rules, and inference mechanisms offered by an ontology. Another significant challenge related to CKC is ensuring the privacy of captured sensitive knowledge. Personal data, which is entirely owned by the user, should be considered vulnerable KBC-LM’24: Knowledge Base Construction from Pre-trained Language Models workshop at ISWC 2024 Envelope-Open tolga@haltia.ai (T. Çöplü); arto@haltia.ai (A. Bendiken); andriy@haltia.ai (A. Skomorokhov); eduard@haltia.ai (E. Bateiko); steve@haltia.ai (S. Cobb) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings if it is sent to the cloud. On-device AI solutions, which do not require any data to leave the user’s device, provide the most appropriate response to privacy needs [5]. Capturing knowledge with the support of a local language model running on the device, securely storing this information in a local knowledge base, and utilizing it on the device when needed, provides a suitable environment for maintaining the privacy of personal knowledge. However, on-device language models come with their own limitations. The size, capabilities, and power consumption of language models running on personal devices such as smartphones, tablets, and computers must be carefully considered [6]. Fortunately, remarkable developments are emerging every day. Thanks to R&D efforts, LLMs with fewer parameters are now offering faster responses and improved performance compared with older large models. In this paper, we explore the feasibility of generating personal knowledge graphs on-device through conversational interaction. Our approach focuses on ontology-guided knowledge extraction from prompts in the form of subject-predicate-object triples1 . We have investigated various methods to enable the underlying language model to comprehend a predefined ontology, ensuring effective personal knowledge-graph generation. Subsequently, we selected the most suitable method based on the requirements of on-device execution. Utilizing a specially designed dataset, we evaluate the effectiveness of this method, emphasizing its strengths and identifying potential areas for improvement. The structure of this paper is as follows: Section 2 discusses various approaches, including in-context learning and fine-tuning for ontology-guided knowledge capture, and focuses on the fine-tuning approach due to its suitability for on-device execution. Section 3 describes the experimental setup, presenting the development framework, language model selection, and the ontology and dataset creation process. Section 4 outlines our performance evaluation framework and the test results. Finally, Section 5 concludes the paper and suggests future directions. 2. Ontology-Guided Symbolic Knowledge Capture In the literature, language models have demonstrated their capability to transform unstructured text into knowledge graphs [7, 8, 9, 10, 11]. However, the process of populating a knowledge graph from user prompts in alignment with a predefined ontology has been explored only marginally [12, 13, 14, 15, 16, 17]. Except for [17], these studies have enjoyed unconstrained processing and memory capacity. Large models with large context windows have enabled in- context learning methods relying on prompt engineering. However, on-device conversational knowledge capture is not similarly unconstrained. Given current context-window capacities, embedding an entire personal ontology into the system prompt would be unrealistic. Addition- ally, considering the inference speed of language models running on personal devices, the high token overhead introduced by this would present a barrier to efficient system operation. An alternative to in-context learning involves training a language model with a predefined ontology so that the model internalizes it. There are two strategies to consider: pretraining the LLM on the ontology or fine-tuning it. This paper does not explore pretraining due to its extensive data, computational resource, energy, and time requirements. Additionally, pretraining 1 https://www.w3.org/TR/rdf12-concepts/ does not offer a flexible response to ongoing changes or expansions in the ontology. Therefore, this paper focuses on fine-tuning as a method to train language models on personal ontologies, highlighting its advantages in feasibility and maintainability. Fine-tuning is a process whereby a pretrained language model is further trained on a specific dataset to tailor its capabilities to a particular task. In our study, the language model is expected to understand the ontology classes and their properties, and use them to populate a knowledge graph from user prompts. The first step involves preparing a fine-tuning dataset, which includes user prompts, system prompts, and expected model responses for each concept in the ontology. This dataset is used to fine-tune the language model, which is then evaluated by testing it with new prompts to assess the effectiveness of the CKC process. The following points highlight the key aspects of ontology fine-tuning: • The training dataset’s coverage and diversity are vital for successful fine-tuning. These characteristics greatly influence the LLM’s ability to capture knowledge effectively. Details about the dataset and how it is constructed are discussed in Section 3.4. • The training dataset must include a variety of examples for the predefined ontology. Research related to the structure of the examples prepared for ontology concepts is detailed in Section 4. • If the language model encounters a user prompt that is not relevant to the predefined ontology concepts, it should not attempt to capture knowledge. Therefore, the dataset should also contain sufficient out-of-context samples to enable the language model to distinguish between relevant and irrelevant information for capture. 3. Experimental Setup 3.1. Development Framework The methods suggested in this paper have been implemented using the Apple MLX framework [18]. MLX is a specialized array framework designed for machine learning applications, akin to NumPy, PyTorch, or JAX, with the distinction of being exclusive to Apple silicon. Ontology fine-tuning has been conducted using the parameter-efficient QLoRA adapters [19] on our custom dataset, comprising randomly selected, non-overlapping sets of training, validation, and test samples. 3.2. Language Model Due to the constraint of on-device execution, our study does not use state-of-the-art large- parameter cloud-based language models. Instead, we opted for a relatively low-parameter model with proven effectiveness across diverse domains. Based on its performance in the Hugging Face Open LLM Leaderboard [20] and its robust ecosystem, we selected Mistral-7B-Instruct-v0.2 [21], which is based on the Llama 2 [22] architecture. The MLX 4-bit quantized version, with a disk size of 4.26 GB, stands out as a suitable model for many personal computers, tablets, and even new-generation smartphones. 3.3. Applied Ontology Our study is inspired by KNOW[23]—the Knowledge Navigator Ontology for the World—and utilizes it for representing personal information. KNOW was introduced as a pioneering framework designed to capture everyday knowledge to enhance language models in real-world generative AI applications such as personal AI assistants. The ontology focuses on human life, encompassing everyday concerns and significant milestones, and limits its initial scope to established human universals, including spacetime (places, events) and social dimensions (people, groups, organizations). This pragmatic approach emphasizes universality and utility, contrasting with previous works like Schema.org[24] and Cyc[25] by building on language models’ inherent encoding of salient commonsense knowledge. Because of the requirement that each element in the ontology be associated with a diverse set of prompt and response samples within the training dataset, our research focuses on a specific subset of the KNOW ontology. This subset concentrates on core family relationships with four ontology classes, eleven object properties, and one data property. A visual depiction of this subset is presented in Figure 1. spouse child parent partner knows sister mother sibling brother father {