A knowledge integration framework is presented for eficient field service engineering in the context of powertrains. The framework enables semantic integration of otherwise siloed data from diferent operations and maintenance system databases. Starting with an expert-curated and database schema-aligned ontology, and using LLMs, a knowledge graph is created from unstructured data sources. A comparison of two leading LLM-based approaches is provided for unstructured data integration in knowledge graphs, namely LangChain LLM Graph Transformer and Microsoft GraphRAG. We also suggest customizations for fine-tuning of the generation process. Besides eficient powertrain fault resolutions, potential applications include Root Cause Analysis (RCA), Failure Mode and Efects Analysis (FMEA), as well as prescriptive maintenance.
for diferent FSE applications; Section 3 presents a comparison of leading LLM-based approaches for unstructured data together with a comparative evaluation, findings and related customizations; Section 4 concludes the paper with remarks and a future outlook.
The overall powertrain knowledge integration architecture is organized in layers as shown in Figure 1. Diferent information such as service instructions, installed base data, service history, customer requests, maintenance history and operational data are available via the various specialized databases from multiple vendors in ‘Powertrain Operations and Maintenance Systems’. Due to the siloed information sources and heterogeneous data models, relations across information entities can easily be missed. Field service engineers looking for information about a particular asset on site must refer to various databases and manually find relevant interconnections, which tends to be a time-consuming and error-prone process. As a result, services such as fault resolution and root cause analyses requiring integrated information from diferent operations and maintenance systems are dificult to perform.
The ‘Powertrain Knowledge Graph Layer’ consists of an ontology with TBox statements and a
knowledge graph with instances corresponding to ABox statements [
Troubleshooting, resolution and maintenance are among the primary tasks of FSEs, see e.g., [
Field service engineers typically rely on service manuals to perform maintenance tasks. These manuals
are in the form of unstructured data and are usually complex containing domain specific concepts and
terminology. Capturing the meanings in such unstructured text and understanding the connections
between diferent entities is not a trivial task. Large Language Models (LLMs) have recently shown
significant capabilities in several natural language processing research areas. Among these areas is
converting unstructured data into entities and relationships to construct knowledge graphs for large
amounts of unstructured text, e.g., as investigated in [
In this section, we explore diferent LLM techniques and frameworks to convert unstructured text into knowledge graphs. We highlight our findings and present customizations to overcome some challenges.
We investigate three techniques to explore LLM capabilities in converting large amounts of unstructured text to entities and relationships.
A schema-less technique is an unstructured approach that allows LLMs to build knowledge graphs without relying on any pre-defined graph structure. Here, the input to the LLM is only the unstructured text. The LLM subsequently generates a schema representing the underlying entities and relationships and populates instances accordingly.
In a schema-based technique, a structured representation of the TBox statements in the form of an ontology is provided to the LLM to act as a guide for capturing entities and relationships. The objective of the LLM is to find triples in the text conforming with the given ontology schema. Here, the expertcurated ontology aligned to database schemas in a second step, as shown in Figure 1, is provided. Example nodes include fault codes, which are identifiers representing specific faults generated by a drive or a motor as well as likely causes and instruction explaining how to resolve such faults. Examples of relationships include linking a drive product family to a fault code, by specifying that a drive generates a fault code and a fault code occurs due to a likely cause.
Providing additional prompt instructions is a widely used approach to provide clear guidance to the LLM
[
We investigate two widely used LLM frameworks to generate knowledge graphs: LangChain LLM
Graph Transformer and Microsoft GraphRAG. To drive a fair comparison, we use GPT-4 model [
LLM Graph Transformer [
Microsoft’s GraphRAG [21] extracts structured entities and relationships from natural language text using LLMs. A graph index is built and an entity knowledge graph is derived. GraphRAG also leverages global information in the graph by using community detection to cluster closely related communities together. This is then used for summarization and RAG retrieval applications. This framework supports providing an ontology to guide the structuring of the knowledge graph. Several prompts are provided for extracting the graph, clustering communities, and performing local and global searches on the knowledge graph.
Below we present our findings for the three LLM techniques used to construct the knowledge graph. We also compare the two LLM frameworks using several aspects.
We evaluate the knowledge graph generated from unstructured text in manuals. The manuals contain a description of faults encountered by drives, their likely causes, and resolution steps. Figure 2 shows the schemas generated from LLM models when using each of the three LLM techniques. First, for the schema-less technique, although the LLM has more freedom and flexibility to construct the knowledge graph, we observe that it fails in capturing the underlying information in the unstructured text. This leads to misrepresenting entities and relationships in the knowledge graph as well as ignoring relevant concepts in the text, such as likely causes of drive faults. Second, in the schema-based technique, we observe a more informative structure in the knowledge graph. The entities and relationships accurately represent the domain information present in the unstructured text. However, we still notice the LLM missing essential semantic connections present in the text. Finally, when providing detailed prompt instructions in addition to the ontology, we observe an improvement in the quality of the generated schema and instances. The LLM is able to accurately capture the required domain knowledge in the text and form meaningful semantic connections. For example, fault code entity representation accurately follows the provided prompt examples. This is crucial for service engineers to be able to accurately recognize faults generated by drives. Also, when providing instructions to the LLM, full instructions from the manuals are included in the instance graphs. This is important for efective diagnoses and repair of faults.
We compare both frameworks against several aspects in Table 1. The first aspect is stable runs. Both frameworks use LLMs, as a consequence the graph generation process is non-deterministic which may lead to variant output among diferent runs. Producing diferent output is problematic in real-world scenarios where a consistent data generation workflow is required. To assess the magnitude of the variation for both frameworks, we perform 10 runs using the same input and prompts. We examine the consistency of the output in terms of the following: the total number of generated entities/relationships and the number of generated entities per each type in the ontology schema. When using GraphRAG, we notice consistent output across multiple runs. However, when using LLM Graph Transformer, a substantial variation of results is observed. The second aspect of the comparison is related to the schema-based technique. LLM Graph Transfomer ofers more control over specifying the ontology structure. A detailed specification of relationship types and properties can be provided to guide the LLM. Next, we notice both frameworks fail to semantically link and understand domain specific terminology. Also, both frameworks are prone to hallucinations. LLM Graph Transformer exhibits a hallucination rate of ∼ 15%, while for GraphRAG, the hallucination rate is ∼ 12%. In Section 3.4 we further discuss the concept of hallucinations. Hence, summarizing, our preference based on scalability-relevant criteria such as ‘Stable runs’ and ‘Triple hallucination avoidance’ is to use GraphRAG as it performs better compared to the LLM Graph Transformer approach. Additionally, the ‘Provide summary descriptions’ and ‘Generate communities’ features can aid in the future for processing of the constructed knowledge graph.
LLM approaches are prone to hallucinations. We observe several examples of such behavior. First, hallucinations exist in the form of new relationship types that do not exist in the ontology schema. This leads to incorrect semantic linking between entities. Second, hallucinations exist in the form of triple hallucinations. Although the type of relationship as well as the types of generated entities exist in the ontology schema, the LLM fabricates and mixes false associations between entity types that are not consistent with the domain knowledge. Next, we observe cases where the triple types and associations follow the ontology schema, however, the generated instances are not present in the input data. Finally, in some cases, entities are generated with no assigned types. In this case, the LLM fails to understand the meaning of such entities. These examples are flagged in the graph for further domain expert review.
Diferent from hallucinations, LLMs may generate triples that are slightly deviated from the ontology structure. For example, a relationship can be reversed between two entities. We adjust such inconsistencies to ensure that the instances are aligned with the defined ontology concepts. The verification step involves the experts towards the final ‘Expert-Reviewed Knowledge Graph’ of Figure 1.
We provide the capability to store the output of the LLM, in the form of entities and relationships, in diferent RDF-native and labeled property graph (LPG) databases. To achieve this, a mapping layer is needed to transform the entities and relationships to diferent formats via the RDF representation, supported widely by diferent LPG databases as mentioned in Section 2. Our mapping layer generates queries using diferent graph query-programming languages to store and manipulate data in various graph databases.
FSEs stand to benefit from integrated knowledge in a variety of ways for providing services in an eficient manner based on up-to-date information. The described LLM-based Knowledge Graph generation based on unstructured data sources together with related customizations is promising, particularly for scaling knowledge graphs for powertrain knowledge integration. In the future, LLMs could also be utilized for automatically suggesting ontology extensions for TBox alignments with database schemas, for an initial reviewing step where LLMs act as a judge [22], as well as for enabling a chat-based user interface with translation of natural language questions to graph queries for knowledge retrieval and response. Beyond powertrain manufacturing, the proposed architecture and evaluation methodology could also be applied to further verticals involving field service engineering such as productive robotics, oil and gas, aerospace and healthcare.
The author have not employed any Generative AI tools in creating the paper.