Data quality is often defined as "fitness for use" [1] and is a multidimensional concept, which is determined by the stakeholders and factors involved in the creation of the data [2]. The quality of the data will influence the usefulness of data for use cases of consumers [1]. Currently, quality assessment within the linked data domain is commonly performed on the published data and is the responsibility of data consumers rather than the producers [3]. a complex, time-consuming task, which is frequently error prone [2]. Often quality issues within these mappings are not detected until the dataset has been published [1,2]. In addition, creating high quality mappings requires a high level of relevant background knowledge [4]. Background knowledge in this context is described as information which informs design decisions involved in declarative mappings, such as knowledge on ontologies, RDF data querying and validation.
The research described in this paper makes a connection between LLMs [7] and semantically represented knowledge graphs, which are generated by declarative uplift mappings. The recent increasing uptake of ChatGPT [8] has demonstrated its ability to provide semantically correct natural language, however, can it provide semantically correct RDF concepts? In addition, can it syntactically correct RDF related code, such as Terse RDF Triple Language (Turtle) [9] and SPARQL Protocol and RDF Query Language (SPARQL) [10]?. Previous research [11] has applied the ChatGPT LLM to enrich data contained in resulting linked data and states that "One of the aspects which have not yet taken over the spotlight is the combined application of these models with semantic technologies to enable reasoning and inference.". Thus, additional exploration is warranted to discover the possible benefits of the application of LLMs in the publication of linked data. The intuition is that applying LLMs early in the publication process of linked data could result in improved quality, by informing agents while completing important uplift mapping design decisions. In order to explore the use case, we propose the R2[RML]-ChatGPT Framework, which was designed to provide insights from ChatGPT 3.5 turbo2 on aspects of uplift mappings defined in R2RML and RML.
This paper is structured as follows: Section 2 presents the design and implementation of the framework. Section 3 presents the experiments completed on the framework. Section 4 describes related work. Section 5 outlines future work and concludes the paper.
This section presents the design and implementation of the R2[RML]-ChatGPT Framework3 .
Figure 1 presents an overview of the components and activities involved in the framework. The initiation of the framework involves an input of an uplift mapping. The concepts in the mapping are used to retrieve information from ChatGPT by inserting them into predefined prompt templates. The processing of user input and output from ChatGPT shown in the diagram is described in the following subsections. Pre-processing involves retrieving necessary information from the input uplift mapping and generating prompts for ChatGPT. First, an uplift mapping is input into the framework using the GUI. Thereafter, a SPARQL [10] query4 is executed on the mapping to retrieve distinct concepts (classes and properties). Each of the concepts are fed into prompt templates, which are designed to provide initial useful insights related to each concept. Initially, the framework executed a new request to ChatGPT [8] for each concept in order to retrieve the initial response. However, the observed processing time was longer than expected as multiple requests were being executed at once. In particular, those mappings with large numbers of concepts were observed to be performance intensive. Threading 5was experimented with to improve the performance, however, the results varied depending on the current load of the ChatGPT model. Thus, it was decided to include a relational database to store previously retrieved prompts, in order to improve the performance of the framework (acting as a cache). The database is queried to find if the initial generated prompt has been previously requested. Thereafter, the cached response retrieved from the database is output to users. Otherwise, ChatGPT is queried and the response cached for later reuse. It was decided not to use a triple store for storing responses as these are represented in natural language rather than semantic data. Values which could be later used for additional prompts are stored in specific chat threads (<chat_id>). For instance, the value retrieved for the range (rdfs:range) of a property (<property>), which was requested can be extracted from the response using a regular expression ("'rdfs:range' is '(.*)' "). Thereafter, the value is stored in a dictionary default mapping (<chat_id>:{'rdfs:range':<regex_result>}). The dictionary can be queried when additional prompt buttons available on the framework are pressed, such as the "SHACL Shape #1" button, which creates a constraint in the Shapes Constraint Language (SHACL) [12]. SHACL is a W3C recommendation designed to allow the definition of constraints in RDF format using the provided ontology terms.
Post-Processing. Figure 3 presents an overview of the activities involved in postprocessing of the framework. Post-processing involves retrieving necessary information from responses to feed into other prompts or extraction and validation of sample data. The response from ChatGPT is either processed to extract relevant values, which can be used in additional queries or to extract requested code. A regular expression ("(.*?)") is used to extract concepts from responses, which can be inserted into additional prompt templates. For instance, the range of a property which was requested can be inserted into another prompt in order to create a SHACL [12] constraint to validate the resulting dataset. However, it was observed that some of the data returned was missing required prefix definitions. Regular expressions ('Undefined prefix "(.*)"', 'Prefix "(.*)"') are used to extract information from the output of the syntax parser if the code was unsuccessfully parsed. The extracted information can be used to improve the quality of the code by inserting prefixes stored in a CSV file 6 . The resulting data can be exported into a file using the framework, with a visual indication of syntactic correctness provided to users.
The framework was implemented using several Python libraries 7 . Flask was used to develop the web framework. SPARQLWrapper is used to execute SPARQL queries. RDFLib is used to parse RDF data. The Open-AI library is used to communicate with ChatGPT. Figure 4 presents a screenshot of the implementation information that is displayed by the framework related to prov:generatedAtTime 8 .
The framework allows users to interact with chats related to the various concepts using the side bar shown. The initial prompt response shown is created using the following prompt templates: "Can you provide me with key information related to the '<concept_name> used in RDF/OWL Technology?", which inserts concepts retrieved from the input mapping. It is hoped this initial information can provide useful insights on each concept in the uploaded mapping. The blue buttons above the input text area are designed to input further prompts based on the current concept. For instance, the "Range" button, inserts the respective concept into the following prompt: "Can you tell me the 'rdfs:range' value for the RDF concept named '<concept_name>?". Thereafter, the term returned for the range can be compared with the mapping to ensure consistent use of the respective ontology. Hover text is provided on the framework for each prompt button to further clarify their intended usage. Certain buttons are designed to output SPARQL ("SPARQL Query #1"), Turtle ("Sample Graph #1") and SHACL ("SHACL Code #1") code.
Figure 5 presents the functionality used to export and validate RDF related code from the framework. First, the code can be extracted ("Extract Code") from raw the response received from ChatGPT. The processed code can be export ("Export Code") into a file for reuse later. A green export button (as shown) indicates the code was successfully parsed. A red button indicates to users that the syntax parsing was unsuccessful, and the framework could not repair the code. The sample shown includes a SPARQL query for checking if a resource is defined as a type of rdfs:Class [13]. As can be seen, the initial response from ChatGPT was missing prefix definitions, which were added by the framework.
A system experimentation 9 was conducted in order to validate two key aspects related to the application of ChatGPT for supporting the mapping quality improvement use case. These aspects related to the semantic and syntactic correctness of RDF concepts and code output by the developed R2[RML]-ChatGPT framework. "Code" in this experiment, refers to Turtle data and SPARQL queries. Two research questions (RQ) were posed in order to explore these aspects:
RQ2: To what extent will ChatGPT produce syntactically correct RDF data and SPARQL queries?
RQ1 was tested by retrieving ontology terms for concepts using ChatGPT and then comparing it with the term definition from the respective ontology. Matching terms would indicate that ChatGPT has output a semantically correct concept. For instance, the range (rdfs:domain) of the foaf:based_near property in the FOAF ontology [14] is defined as geo:SpatialThing10 , which would be compared to the domain that is output by ChatGPT when provided with a respective prompt. The comparison takes into account subclasses of the concept output. The foaf:Person concept is a valid domain for this case as it is a subclass of geo:SpatialThing. RQ2 was tested by validating the syntax of the RDF and SPARQL code output by ChatGPT. The framework attempted to resolve issues in incorrect syntax using the post-processing described in Section 2.1. Thereafter, the updated syntax was assessed similarly. Figure 6 presents an overview of the activities involved in the experimentation. 11 were input into the framework. 10 concepts (5 classes and 5 properties) were retrieved from the following five well-known ontologies: RDF [15], RDFS [13], FOAF [14], SKOS [16] and PROV-O [17]. It was decided to use these ontologies as they are designed to represent diverse information such as provenance (PROV-O), social networks (FOAF) and taxonomies (SKOS). Only concepts with the respective related properties were chosen to ensure that a value for comparison existed. For instance, only properties with an associated range were tested for range correctness. Ontology terms and RDF related code related to these concepts were retrieved from ChatGPT using prompt templates, where each respective concept name was inserted. Thereafter, relevant values were extracted from the response and stored for comparison. The comparison was completed by inserting respective values into an ASK SPARQL [10] query. Thereafter, the query was executed on the respective ontology which was stored in the local Apache Jena Fueski Triple Store 12 . The queries when executed resulted in boolean (True or False) values, which represented the result of comparison. In addition, 150 files containing SPARQL, SHACL and Sample instances were generated using the framework, by asking ChatGPT to produce sample data related to each ontology term in the sample mappings. Relevant information for testing RQ1 was the concept name returned for each requested ontology term. Relevant information for testing RQ2 was associated RDF graphs and SPARQL queries.
Testing of semantic correctness (RQ1) involved retrieving the type (rdf:type), range (rdfs:range), domain (rdfs:domain) and human readable label (rdfs:label) associated with the 50 concepts. The results from ChatGPT were compared to the corresponding term defined in the ontology using an ASK SPARQL query template. For instance, the type of prov:atLocation was tested. ChatGPT was provided with the following prompt: "Can you provide the 'rdf:type' value for the 'prov:atLocation' concept defined in an ontology used in RDF/OWL Technology?". The type returned was extracted from the response and inserted into a SPARQL query template 13 , which queries the namespace ontology. The results of the comparison were manually validated to ensure consistent results. Table 1 presents the results of the comparison of the output for each of the ontology terms. The ASK query returning True ("True Count") indicate the ontology term returned by ChatGPT was the same as the ontology. The query returning False ("False Count") indicates the concept output does not match the ontology. In addition, no respective value ("No Response") could be returned from ChatGPT. The label and type term related to all 50 concepts. However, the domain and range only related to properties as classes do not have these restrictions. The results indicate that ChatGPT produces correct terms for 118 (78.6%) of the tested cases, with the correctness influenced by the ontology and requested term. It scored best for retrieving the correct domain (rdfs:domain) of properties with 88% semantically correct terms retrieved. Slightly worse scores were identified for the range (rdfs:range) of properties with 84% correct. While ChatGPT was capable of providing semantically correct types (rdf:type) of concepts with a slightly worse degree of accuracy (76%). Most correct cases related to the generalized RDF class (rdfs:Class) or property (rdf:Property), which all RDF concepts are types of [13]. In addition, a high proportion of domain and ranges returned related to the generalized RDF resource (rdfs:Resource) which all RDF resources are instances [13]. Most incorrect cases related to ChatGPT returning a type of the name of the concept itself, such as prov:Agent, which returned a type of prov:Agent. Labels (rdfs:label) scored worst (74%), which could be a result of the natural language representation of them. As LLMs [7] are trained using natural language it could be harder for them to distinguish these values from other text when compared to RDF concepts. Interestingly, it was observed that ChatGPT made inferences about certain labels. For instance, the label returned from the rdf:HTML class was "HTML/XML Syntax for RDF", however, the value defined in the ontology is "HTML". It could have made inferences about the usage for RDF due to the context of the request. Similarly, the label returned for the rdf:rest property was "rest of list", whereas the correct value is "rest". As ChatGPT knows that the property is related to lists, it could infer the label based on the background information. In addition, a limitation for labels is that ChatGPT may not understand the common naming convention (property name in camel case). Table 2 presents an overview of the results categorized by respective ontology. The results indicate that the ontology where the term was defined influenced the semantic correctness of values returned. In addition, the high standard deviation (3.4) indicates that the scores for each ontology were spread around the mean. As can be seen, RDF [15], RDFS [13] and FOAF [14] scored best, while SKOS [16] and PROV-O [17] scored worse. Thus, these results indicate that the ontology where the tested concept originates influences the semantic correctness of ChatGPT. However, all ontologies scored between 63% (SKOS) and 93% (RDFS) correctness for each of the 30 tests completed on them. The worse scores could be as a result of the amount and quality of documentation published by the ontology, which was used to the train the LLM [7]. Table 3 presents a sample of results from this experiment. The tested ("Concept") is shown, along with the term returned from ChatGPT ("ChatGPT Output") and the semantically correct corresponding term ("Ontology Term") from the respective ontology. The overall results show that ChatGPT can provide semantically correct concepts with 118 (78.6%) correct for this use case. Overall, it can be concluded from RQ1 that the framework could be beneficial for agents involved in quality assessment of the publication process of linked data. The information related to ontology reuse could be used by agents to inform crucial design decisions which will impact overall quality.
Testing of RQ2 involved retrieving a sample instance Turtle [9] graph, sample SHACL [12] shape and sample SPARQL [10] queries related to the 50 ontology concepts. In total, 150 files (3 for each concept) of code were generated. Table 4 presents the results of inputting these 150 files into the RDFLib parser 14 in order to validate syntactic correctness. Each tested category consisted of a total of 50 files. Files which were tested and contained initially correct code syntax required no further actions. Incorrect code was post-processed by the framework as described in Section 2.1, resulting in the final syntax of the code. The results show that ChatGPT can produce RDF related code with a high degree of accuracy. Initially, all categories scored better than 42 (84%) syntactically correct. The postprocessing of the incorrect files resulted in an improvement to a mean of 48 (96%) for all categories. The results show SHACL constraints scored slightly better than the other categories, which could be due to less prefixes being needed in most cases. SPARQL queries and Sample instance graphs scored similar. Majority (14 out of 18) of the syntax problems were due to missing prefixes in the initial response from ChatGPT. Only 1 out of the 11 initially incorrect Turtle graphs could not be repaired using the post processing, which indicates that ChatGPT has a good understanding of the overall structure of these graphs. SPARQL queries accounted for the most (3) files where syntax could not be repaired. These results provide an indication that ChatGPT has the least understanding of them.
In recent years, research into frameworks to support the generation of high-quality mappings have been conducted. The Mapping Quality Improvement (MQI) Framework [18] to improve and maintain the quality of R2[RML] [5,6] uplift mappings has been proposed and evaluated. The framework consists of two core components. The mapping quality assessment and refinement component is designed to detect quality issues and suggest semi-automatic refinements to resolve them. The change detection component detects changes in respective source data and provides suggestions on how to maintain alignment between them. An approach [1] exists which was designed to assess and refine the quality of R2[RML] mappings using rule-based reasoning, which involves executing various test cases on them. Some of the related approaches [2,3] extend existing linked data quality assessment frameworks. These approaches are designed to target mappings represented in RDF format, such as R2RML [5] and RML [6]. EvaMap is an approach [19] which was designed with the requirements in mind and uses information contained in respective ontologies to assess the quality of YARRRML15 mappings. The assessment involves quality metrics in 7 dimensions, which are used to calculate a weighted mean score. However, these approaches are limited to information contained in ontologies used by respective mappings, which are queried in order to assess quality. Thus, mapping engineers who use these approaches are required to search other forms of data on the web to resolve issues out of scope of the used ontologies, such as alternative ontologies to reuse.
Future work includes expanding the test cases applied during the experimentation outlined in this paper. It is hoped expanded test cases will provide further indications of the accuracy of the relevant knowledge supplied. In addition, conducting a usability experiment on the framework will help to identify limitations for respective end users. A standardized questionnaire, such as the Post-Study System Usability Questionnaire (PSSUQ) [20], which was designed by IBM could be used. The PSSUQ measures user satisfaction from a software system, which involves rating various key aspects using a Likert scale. Furthermore, the semantic correctness of generated SPARQL queries and SHACL shapes could be tested, by executing them on respective input and comparing the output with expected results. Moreover, knowledge from the generation of SHACL [21] shapes using Ontology Design Patterns [22] could be integrated into the framework to generate ontology specific constraints. Finally, the framework could be extended to support other mappings represented in RDF format, such as the Database to RDF Mapping Language (D2RQ) [23].
The R2[RML]-ChatGPT framework proposed in this paper provides possible direction for future applications of LLMs in the publication process of linked data. It is hoped the approach can be used to provide accurate quality insights on various aspects of associated artefacts to alleviate the high requirement of background knowledge from domain experts. In addition, it is hoped that the availability of diverse prompt templates will result in a more straightforward knowledge discovery process using the framework. The results of the experiments demonstrated that ChatGPT is capable of providing syntactically and semantically correct data. 118 (78.6%) of the 150 tested ontology terms were semantically correct and a mean of 48 (96%) of the 50 code files for the tested categories (after postprocessing) were syntactically correct, which indicates a high level of accuracy. These results indicate that the information could be beneficial to mapping engineers when making crucial design decisions within the linked data publication process. It is hoped the easily accessible information covering various knowledge domains could be used to support domain experts when retrieving required knowledge during the publication process. In addition, it is hoped the automation of syntactically correct RDF related code will reduce workload for involved agents.
This research was conducted with the financial support of the ADAPT SFI Research Centre (Grant No. 13/RC/2106_P2) at Trinity College Dublin.