Data quality is a multidimensional problem encompassing heterogeneous and multiple quality dimensions, including but not limited to accessibility, interlinking, performance, syntactic validity, and completeness [1]. The significance of each dimension depends on the domain or particular use cases. The result returned by assessing these quality dimensions can be combined to generate a score ranging from 0 to 1, allowing datasets to be ranked accordingly, where a higher score indicates a higher quality.
Once datasets are ranked based on their quality score, the key challenges are how to effectively represent the quality of the datasets and how to explain the produced results. Analogous to Machine Learning, explainability is often replaced with the notion of interpretability [2], which are considered interchangeable terms within the broader Artificial Intelligence (AI) community and particularly among scholars specializing in automated learning and reasoning. Conversely, the software engineering community prefers the term understandability [3]. In general terms, interpretability is often defined as the ability to convey or extract the meaning of an abstract concept, whereas understandability refers to the capability of making it comprehensible to end-users [2]. This work focuses on the interpretability and explicability of quality scores, treating those terms as synonyms.
Knowledge Graphs (KGs) are often preferred in scenarios where explainability and interpretability are crucial [4], as they explicitly represent relationships between entities and provide a structured knowledge representation. Hence, a crucial step in the direction of interpreting quality scores is modeling them as a KG. The process of KG construction requires a significant amount of manual effort and expert knowledge in identifying and labeling sentences or patterns, performing named entity recognition, relation extraction, and semantic parsing [5]. Often, this process encompasses at least the involvement of experts in the modeled field and computer scientists or experts in the targeted ontology. As a matter of fact, in the complex realm of machine learning and natural language processing (NLP), data annotation stands out as a critical yet challenging step, transcending simple label attachment to encompass a rich array of auxiliary predictive information. Data annotation poses significant challenges for current machine learning models due to the complexity, subjectivity, and diversity of data, requiring domain expertise and the resource-intensive nature of manually labeling large datasets.
Cutting-edge Large Language Models (LLMs) such as GPT-3.51 , Gemini2 and Llama-23 offer a promising opportunity to revolutionize data annotation. LLMs play a pivotal role in enhancing the accuracy and efficiency of data annotation processes. Their ability to automate annotation tasks, ensure consistency across large volumes of data, and adapt through fine-tuning or prompting for specific domains, mitigates challenges inherent in traditional methods, setting a new standard for NLP [6]. By providing a well-devised input sequence (e.g. a textual prompt), LLMs can adeptly undertake knowledge extraction tasks. However, the question is how to formulate an effective input prompt, and this is where prompt engineering assumes significance.
In our investigation, we consider the free GPT-3.5 a state-of-the-art model for prompt engineering to explore prompt engineering's role in knowledge extraction and representation of tabular datasets quality. We adapt to different situations that generate knowledge in the context of knowledge extraction [6]. Specifically, modeling quality dimensions and metrics score as a CSV file, we aim to extract all potential triples from the text, without supplying any predetermined labels nor imposing constraints within the prompt as in Closed Information Extraction settings. This work provides the adaptation of state-of-the-art prompt engineering methods in the context of knowledge extraction from the quality of the datasets. It introduces a first step through the interpretability of the final score, verifying the feasibility of an automatic process of knowledge extraction leveraging LLMs.
This paper is organized as follows: Section 2 provides the background on quality and discusses the related work on knowledge extraction and representation. Section 3 documents the performed approach reported in such detail to enable reproducibility. Section 4 introduces the preliminary results of our approach, which are discussed in Section 5. We report conclusive thoughts in Section 6 along with future directions.
Data quality assessment involves the measurement of quality dimensions relevant to the consumer and considering dataset characteristics, which can be grouped in dimension clusters. Inspired by Zaveri et al. [1], we consider the following dimension clusters:
• Accessibility dimensions which involves aspects related to the access, authenticity, and retrieval of data to obtain either the entire or some portion of the data (or from another linked dataset) for a particular use case. It includes availability, licensing, security, and performance. • Intrinsic dimensions are those that are independent of the user's context. It includes semantic accuracy, consistency, and conciseness. These dimensions focus on whether the information (syntactically and semantically) correctly and compactly represents the real world and whether the information is logically consistent in itself. • Contextual dimensions are those that highly depend on the context of the task at hand, assessing the amount of published data, their relevancy, trustworthiness, understandability, and timeliness. This dimension can be further refined by considering the following aspects as separate dimension clusters:
-Trust dimensions focusing on trustworthiness in terms of verifiability, reputation, and believability; -Dataset dynamicity focusing on the currency and timeliness.
• Representational dimensions capture aspects related to the design of the data, such as representational conciseness, interoperability, interpretability, and versatility.
While quality dimensions are rather abstract, they can be measured via quality assessment metrics which rely on quality indicators. An assessment score is computed from these indicators using a scoring function.
Extracting knowledge from tabular data like databases, Web tables and CSV files is a common way for KG construction. If users are aware of tables semantics, they can define and use heuristic rules to transform their data into KG facts. However, usually end-users lack a deep understanding of tables meta information, such as table name and column header. In recent years, transformer-based LMs have been investigated for processing tables and representing learning, such as TURL [7], RPT [8], Starmie [9]. There have been several attempts that use LLMs for these tasks, such as Doduo [10] which focuses on the prediction of column types and identification of inter-column relationships, and Korini et al. [11] who prompt ChatGPT to annotate semantic column types. Some attention has been given also to utilizing LLMs for tabular data processing and KG construction, such as the work authored by Kommineni et al. [12]. However, there is still room for investigation, mainly in representing non-textual tabular data, like numbers [13]. Moreover, LLMs are mostly applied to process and understand tables but rarely applied to the final step of knowledge extraction [13]. As some examples in this direction, OntoGPT [14] and Trajanoska et al. [15] extract instances from texts to populate an ontology, but there are no counterparts for tables. Our contribution target this direction, exploring how to populate an ontology using ChatGPT starting from a tabular representation of quality scores. It goes in the direction of leveraging LLMs to represent and interpret KG quality results integrating LLMs in the KG quality measurement pipeline as a way to enhance the interpretation of quality assessment report. As a result, using LLMs to automatically convert CSV data quality assessment to RDF triples, make assessment results machine-readable and potentially useful for an automatic elaboration.
This section describes the performed process to move from CSV quality assessment returned by KGHeartBeat 4 to RDF triples leveraging ChatGPT. KGHeartBeat is a fully-automatic community-shared open source quality assessment tool to periodically perform quality analysis on all the freely available KGs that can be automatically retrieved by widely used data and knowledge aggregation platforms, such as LOD Cloud 5 and DataHub 6 . The KGHeartBeat web-application can be configured to query a list of KGs and implements a large set of KG quality metrics proposed by Zaveri et al. [1] belonging to different quality dimensions, focusing on those that can be automatically and objectively computed without requiring a gold standard. The implementation details of all the supported quality dimensions and the related metrics are freely accessible online 7 . Once selected the quality dimensions of interest, quality results can be downloaded as CSV files. The CSV stores a KG quality assessment per line, listing all the metrics' and dimensions' scores as columns. As a convention, all the metrics related to the same dimensions share as prefix the name of the dimension. Per each dimension, there is both a weighted and a normalized score. Such as an example, the CSV file focused on the availability dimension results concerning a single KG is structured as follows reporting the header in bold and the value attached to it:
The quality assessment CSV files can be used along with an ontology to model KG quality scores as input to LLM, such as ChatGPT, as summarized in the process visible in Figure 1. We used the CSV returned by KGHeartBeat and the Data Quality Vocabulary8 (DQV) in TTL format to perform the experiments reported in this article. DQV defines quality measures as specific instances of dqv:QualityMeasurements and it relies on quality dimensions (dqv:Dimension) e.g. the availability of a dataset, and quality metrics (dqv:Metric) e.g. whether or not a SPARQL endpoint is accessible.
We consider three scenarios of incremental complexity, defined as follows:
• Single dimension, multiple metrics. The end-user focused on a single dimension. Hence, KGHeartBeat returns a CSV containing the detail for each metric implemented for the user-selected dimension and an overall score at dimension level. As an example, we focus on scores concerning availability of ten different randomly selected KGs. • Single dimension category, multiple dimensions and multiple metrics. The end-user focused on a single dimension category, which include multiple dimensions, each computed in terms of multiple metrics. Hence, KGHeartBeat returns a CSV containing the detail for each metric implemented for the user-selected dimension and an overall score at dimension level. As an example, we focus on scores concerning the trust category of ten different KGs randomly selected. • Different dimensions categories. According to the task or scenario of interest, end-users can select different dimensions and the related metrics. As a result, we consider a CSV of ten different KGs randomly selected with a set of quality dimensions related to different categories.
This section reports the performed experiments describing the used prompts and discussing ChatGPT replies, mainly focusing on positive aspects and what is missing. Files used as input, the transcript of conversations performed on ChatGPT 3.5, and the resulting RDF skeleton are reported in the KGHeartBeat GitHub 9 , this is to promote the reproducibility of the experiment performed.
Single Dimension, Multiple Metrics: the Availability Case. The simplest interaction with ChatGPT is uploading both the CSV and the DQV.ttl ontology without any explanation, and inspect the resulting RDF triples. Consequently, we report the used prompts in the following:
1 Consider the following csv entitled "availability_scores.csv": <PASTE CSV FILE CONTENT>.
2 Consider the following ontology in ttl format entitled "dqv.ttl": <PASTE ONTOLOGY IN TTL FORMAT> 3 Can you model the "availability_scores.csv" file content according to the "dqv.ttl" ontology and return the resulting triples in rdf format?
As a result, ChatGPT correctly models each KGs as a dqv:QualityMeasurementDataset, correctly attaches each value to a modeled metric, and correctly creates the correspondence between metric scores and datasets. However, it considers all the columns in the CSV as metrics, without distinguishing metrics' scores to the overall score at the dimension level. Of course, we need to clarify how the CSV file must be interpreted. Moreover, ChatGPT only returns a skeleton of the RDF file, suggesting to repeat similar patterns for other KGs and quality metrics. By reformulating the third prompt as follows, we obtained a complete modeling of RDF triples of a specific KG modeled in the CSV, e.g., DBpedia in French.
1 Considering that all the columns listed in the "availability_score.csv" file having the name starting with "availability_" are metrics of the availability dimension which is one of the accessibility dimensions, the column entitle "availabilityScoreValue" is the value corresponding to the availability dimension, and you can ignore the other columns ending with score, can you model the data contained in "availability_scores.csv" file and related to the KG entitled "DBpedia in French" according to the "dqv.ttl" ontology and return the complete and detailed set of resulting triples in rdf format both reporting the score of the availability dimension and detailing all the metrics' measurements?
The explanation of the CSV file is needed to correctly distinguishing quality dimensions and metrics in the ontology. The suggestion to focus on a single KG is due to the trend of ChatGPT of proposing the skeleton of the RDF triples for a small set of metrics and KGs and suggesting to complete it following similar patterns, as observed before. Moreover, if end-users do not specify the desired level of details in terms of metrics' measurements, ChatGPT only models the quality dimension score.
According to the interaction described before, the used prompts specify the CSV, the ontology and describe how to interpret the CSV header as follows:
3 Let's consider that the CSV file contains all dimensions concerning the trust category and for each dimension, the file details its metrics. To distinguish metrics and dimension, consider that all the file column names follow the pattern of DIMENSION_METRIC. With these premises, can you model the data contained in csv file according to the "dqv.ttl" ontology and return the complete and detailed set of resulting triples in rdf format?
As a result, ChatGPT correctly recognizes category, dimensions and metrics and models them as an hierarchy. However, it stops to a single metric and to a single KG. Even if we suggest to focus on a single KG, such as DBpedia in French, it returns an incomplete formulation, just modeling a single metric and suggesting to add similar triples for other metrics and dimensions. Moreover, even when we ask to focus on a single dimension, e.g., the believability dimension, and a single KG, it returns a incomplete formulation, completed by explicitly asking to for it with three iterative interactions. The used prompt is as follows:
1 Let's focus on the Believability dimension. Can you provide with a complete rdf concerning the KG entitled "DBpedia in French" modeling it according to the "dqv.ttl" ontology and return the complete and detailed set of resulting triples in RDF format both reporting the score of all the dimensions and detailing all their metrics' measurements? Different dimensions categories cases. Similarly to the trust case, once provided ChatGPT with the quality scores CSV, the DQV ontology and the explanation of the CSV header to distinguish metrics and dimensions, ChatGPT returns a skeleton correctly modeling the dimension category, dimensions and metrics hierarchy, but sends the user the role of instantiating the names of the metrics and dimensions and completing the triples by replicating the identified patterns.
This section summarizes potentialities and limitations of a LLM-driven KG generation to convert CSV into RDF according to a data quality ontology.
Automatic RDF skeleton to be manually completed. ChatGPT returns a useful skeleton for modeling the CSV content according to the data quality vocabulary requiring a minimum explanation of the CSV header to distinguish metrics' and dimensions' scores and automatically inferring the ontology structure, without requiring any clarification. However, the more data available in terms of metrics and KGs, the more the skeleton only reports the structure that must be manually replicated for each metric and KG observed. Tricks used to complete RDF files seems to work only when end-users narrow down to a focused scenario, e.g., a single dimension and a single KG. As a result, ChatGPT successfully saved human effort in proposing the initial draft of KG, but requires human expertise in assessing and refining, if needed, its outcome, as confirmed by similar work in this field [12,14].
Light-versions of files used as input might be required. With respect to other available and freely usable LLM, ChatGPT accepts prompts long enough to paste the entire Data Quality Vocabulary at once. However, any LLM has its own prompt limit [16]. In fact, we encountered some limitations with CSV starting from the second case due to the presence of several columns populated with a list of values, such as the list of used vocabularies.
This article explores the opportunity to automatically generate RDF triples via LLM. We initialized ChatGPT 3.5 with a (light version of) CSV modeling KGs' quality scores automatically generated by KGHeartBeat and the data quality vocabulary and we explored the returned RDF triples. ChatGPT automatically returns a skeleton of the RDF triples inferring patterns to model metrics and KG assessment that must be manually completed by end-users. According to the level of details in describing metrics, ChatGPT correctly model data quality categories, dimensions and metrics. While manual effort is mitigated, experts in the field are still required to assess the quality of the automatically extracted triples and complete them if needed.
Future directions. This article proposes a feasibility study to generate RDF triples from a CSV file modeling KG quality assessment. As a future direction we are interested to consider the effect of prompting ChatGPT with just a portion of the DQV and compare the results in terms of quality of the returned output. Moreover, we are interested in investigating the perceived utility of a (semi-)automatic KG generation as the one described in this article by inquiring experts in the field. Finally, besides qualitative insights as the ones reported in this article, it is crucial to quantify the quality of the returned output and compare the quality of the KGs resulting by ChatGPT with the one obtained with a manual traditional approach.