This paper describes a new intelligent, data-driven dashboard for linked data quality assessment. The development goal was to assist data quality engineers to interpret data quality problems found when evaluating a dataset using a metrics-based data quality assessment. This required construction of a graph linking the problematic things identified in the data, the assessment metrics and the source data. This context and supporting user interfaces help the user to understand data quality problems. An analysis widget also helped the user identify the root cause multiple problems. This supported the user in identification and prioritization of the problems that need to be fixed and to improve data quality. The dashboard was shown to be useful for users to clean data. A user evaluation was performed with both expert and novice data quality engineers.
Data quality is often defined as "fitness for use", which characterizes the ability of data
to meet users’ requirements [
Data Quality is a key challenge in Linked Data as the data is often transformed from
multiple heterogeneous sources, including semi-structured and unstructured data,
which are of varying quality [
Data quality assessment is a process for evaluating if a data meets the user's specific
needs [
To improve data quality, data corrections are required. It is crucial that the user performing the data correction should have a clear understanding of which quality problems (flaws) are present in the data, how to fix them, where they occur in the dataset, which metrics are impacted by each flaw and how much improvement each fix would bring. Data quality is typically an expensive process and thus a prioritization of fixes is important as there is a trade-off between cost and quality.
In this paper, the research question is to determine “to what extent will an intelligent dashboard based on knowledge graphs and root cause analysts assist the user understand quality problems in Linked Data and thus enable the user to identify appropriate repairs and prioritize them“.
The following technical approach was followed: i) an existing Data Quality Assessment Framework was selected to assess the data (i.e. generate metrics) and generate data quality problem reports; ii) design and build a new high usability user interface to fetch and view data quality metadata and problem reports; iii) develop supporting services to integrate the data and problem reports, and then identify the related problems for each problem through root cause analysis; iv) display flaws in the dataset in the context of all the related problems.
The remainder of the paper is structured as follows: Section 2 discusses the use cases for the dashboard, section 3 presents background and related work on Linked Data quality assessment and Root Cause Analysis; section 4 describes the design of the dashboard tool; section 5 describes the evaluation of the dashboard tool; section 6 describes the conclusion along with the scope of future work. 2
The users of this dashboard will either be data quality engineers or managers in organisations that manage data. The following use cases were developed in the context of the Ordnance Survey Ireland data pipeline to identify dashboard features. UC1: Assessment of Data. Users should have the means to assess data. In the case of Linked Data, the data can be either in any Linked Data supported serialization formats or published in some database which can be accessed using a Uniform Resource Locator (URL). The dashboard should also allow the stakeholders to configure the data details that need to be assessed along with what needs to be assessed in the data. UC2: Display Data Quality. The quality of an assessed dataset needs to be displayed to the users to help them decide whether the data can be published or whether the data can be used in their application. Display the overall data quality, dimension level data quality and also the metric level data quality.
UC3: View Data Quality Problems. Users need a means to view the data quality problems that are present in the problem reports of an assessed dataset, to assist the user’s in understanding it. Any relevant information’s related to each of the data quality problem, such as what exactly does the problem means, what caused this problem to occur, where else does this problem occur and are there any related problems, can help the stakeholder to get a clear understanding of the problem.
UC4: Enable Root Cause Analysis. Stakeholders should be able to decide on the exact cause of a problem in a problematic thing, so as to decide on how it needs to be fixed. This can be performed by the stakeholder’s once all the relevant information’s are available and attached to each problematic things. Backend service which prepares this relevant information’s to the user and corresponding widgets to display them in a userfriendly manner so as to conclude on the exact root cause of a problem. 3
The foundations discussed are Linked Data, Data Quality and Root Cause Analysis.
Root Cause Analysis (RCA) is an event analysis technique [
Three generic standard tools for Root Cause Analysis are: Cause-and-effect diagram
(CED), Interrelationship Diagram (ID), Current Reality Tree (CRT). These three tools
have been evaluated in [
In 2007 Berners-Lee outlined Linked Data as a set of ‘rules’ for publishing data on
the Web so all data can be connected as a single global data space [
Linked Data uses the W3C Resource Description Framework (RDF) [
The SPARQL Query Language is the widely used query language to retrieve and manipulate RDF/Linked Data. It provides the means to query required and optional graph pattern along with their conjunctions and disjunctions. 3.3
Data Quality issues in Linked Data have some unique aspects but most are common
to the general discipline of Data Quality. The survey paper published by Zaveri et al.
[
For the purposes of satisfying our use case (section 2), a comparison between the LinkQA, Sieve, RDF Unit, Triple Check-Mate, and Luzzu Linked Data quality assessment frameworks is given here and summarized in Table 2 below. The focus is on the support for interoperability (standard metrics observation reports), generation of problem reports in RDF format, ability to suggest which problems to be corrected and root cause analysis capability.
LinkQA [
Sieve [
RDF Unit [
Triple Check-Mate [
Luzzu [
Of these tools, the Luzzu framework’s feature to generate the detailed problem reports in RDF makes it highly suitable for our technical approach. Its web services make it highly interoperable and with a few additional features (described later), this makes Luzzu the best tool to assess the quality of Linked Data. Table 2 provides a comparison between the old versions of the Luzzu GUI with the proposed one (fig. 3). Luzzu ✓ ✓ ✓ Feature \ Tool Interdependence between Framework and user Interface Overall Data Quality Summary Statistics Dimension Summary Statistics Metric Summary Statistics Detailed Problem Report Support Root Cause Analysis and Fix Prioritization High ✓ ✓
This section will give you an overview of the new system design which includes the Luzzu Framework for quality assessment, the Intelligent Dashboard for data analysis and the supporting tools and services which makes it possible.
The system with the new analytics dashboard consist of 4 major components (as shown in Fig 1): Luzzu Framework, Triplestore, Analytics Dashboard, and Service Wrapper.
The Luzzu Framework is extended to include direct connection to a triplestore for storage of the metadata and problem report generated during each assessment of a Linked Data dataset. This makes the system more flexible as the Dashboard can directly query the outputs using SPARQL in the triplestore. The metrics and its corresponding dimensions listed in table 3 are assessed by Luzzu for the experiments listed later. Dimension Interpretability Understandability Conciseness Licensing Consistency Consistency Consistency Consistency Syntactic Validity Apache Jena Triplestore: Apache Jena Fuseki is used as the SPARQL triplestore. It consists of Trivial Database (TDB) component for RDF storage and query. File Upload service exposed by Fuseki Server is utilized to load the Metadata and Problem Report generated in Luzzu. The metadata is stored as a separate graph in the triplestore where the metadata graph name can be found in the default graph. Each problem report generated is stored as an independent graph in the triplestore with a unique link to the metadata graph. SPARQL queries are used for remote access of data from the Triplestore via HTTP.
Service Wrapper: This component consists of several API services to assist the Analytics Dashboard in performing back-end knowledge base operations as well as state management of the dataset being assessed. The state management services maintain the configuration state of each dataset that is added in the dashboard for assessment in persistent file storage. The configuration details include the dataset name, dataset SPARQL end-point or dataset dump filename, unique identifier and the metrics that need to be assessed along with the required acceptance threshold per metric. Any changes made to the dataset configuration details in the dashboard is also updated in the persistent file storage with the help of these services. The knowledge base operation services generate the unified knowledge base which consists of all the triples with a problematic thing along with all the related exceptions to support root cause analysis. Exceptions are the issues identified for a resource, triple or part of the triple in Luzzu. The knowledge base generation pipeline is shown in Figure 2.
Table 4 lists how knowledge base generation service identifies the correct part of the triple which is a problematic thing and hence map it to identify the related problems based on to which part of the triple does the problematic things belongs to. Metric
Based on the metric assessed, all the problems identified for that metric is fetched from the triplestore. The triplestore internally fetches the complete triple from the dataset to form the Subject, Predicate and the Object. This builds the complete problematic triple. Each problematic triple constructed is then added to the knowledge base along with the failed metric and the corresponding exception. Once all the problematic triple is added to the knowledge base, it is then enhanced by identifying all the related exceptions raised for the problematic part of the triple. The identified related exceptions are considered as the causal factors of the problem in root cause analysis. This knowledge base in the added to the cache which is maintained by the service and cleared after a certain configured time period.
Analytics Dashboard: The new high usability Analytics Dashboard (Fig. 3) provides a GUI where the user can configure the details of the dataset that need to be assessed along with the metrics. It also consists of a new Problem Report Widget and Analytics Widget. The Problem Report Widget lets the user view all the problems related to the assessed dataset on a per metric basis. The Analytics Widget provides a summary of each problem in the assessed dataset along with the casual factors identified via RCA for each independent problem. This helps the user to identify the root cause of the problem. The summarized view also helps the user to pick out those issues that need to prioritize in fixing. The dashboard intelligently sorts the problem list based on the problems which have the most impact on the dataset. The Resource View and Problematic Thing View let the user view the problems in two additional ways. The Resource View helps the user understand all the problems identified per resource. It also identifies any resource replicas. The Problematic Thing View helps the user understand each problem type detected. It provides information such as impacted resources, related exceptions raised for the same problematic thing and linked problems. It is intended to provide inferential tests to assist in this process in further work. This section describes the user experiment performed to evaluate the new dashboard. Overview: The purpose of this experiment was to evaluate the extent to which the new dashboard with data-driven and analytics widgets was helpful for the users in understanding specific data quality problems discovered by the Luzzu data quality assessment tool in Linked Data. It also evaluates the extent to which the widgets were able to assist the user in identifying the root cause of a problematic thing (data issue) and thus identify the priority issues to resolve in order to improve the data quality score.
H1: The Problem Report Widget and Analytics Widget provided both expert and novice users with a better understanding of the data quality problems identified by Luzzu.
H2: The Analytics Widget enhanced the user’s ability to identify and prioritize which problems to be fixed than the Problem Report Widget.
Dataset: The Ordnance Survey Ireland’s buildings data1 is used for the experimental evaluation. This consists of geospatial data about various buildings in Ireland as Linked Data which is published on the Web. This is a general data which adheres to the Linked
Data best standards in publishing and usage. The active collaboration on ADAPT Center with OSi, the interest of OSi data engineers to this dashboard and prior understanding of data quality problems related to this data are the main factor for using this data for the experiment. Access to this data can be requested by researchers by requesting to OSi. The quality of this data is assessed by Luzzu for those metrics that are listed in table 3.
Users: The users consisted of two groups: i) 4 novice users and ii) 4 expert users. The novice users consisted of computer science master’s students in DCU and the expert users were Linked Data researchers from the ADAPT center. The novice users consist of those who have basic knowledge of web technologies and user interfaces. The expert users consist of those who have basic knowledge of web technologies, user interfaces and also Linked Data quality concepts.
Procedure: The experiment consisted of four tasks which the users had to perform either using the problem report widget or the analysis widget. The users were given the following set of tasks: i) List all the problems identified for a particular resource. ii) Identify which part of a triple is a problematic thing. iii) List the problem identified for problematic thing for a particular metric. iv) Identify the root cause problem for each problematic thing and prioritize the problematic things that need to be fixed.
The tasks i, ii and iii are created to determine to what extent the user has understood about the data quality problem, identified by Luzzu, with the help of Problem Report Widget and Analytics Widget in the new dashboard. Task iv is created to determine which widget was better at assisting the user to identify and prioritize the fix required to improve the data quality. These tasks were designed to ensure that the user uses multiple features of the dashboard to conclude on the answers which help the user understand a data quality problem. The users were asked to perform these tasks independently using the dashboard, which was hosted online and then selects appropriate answers from the questionnaire. The qualitative measures for each of these tasks are recorded by the user in the questionnaire along with other metrics such as effort, confidence, and usefulness.
Data Collected: Table 5 contains the raw data collected for each task. Analysis Method: The values collected during the experiment is used to gather evidence that the hypothesis H1 and H2 are true. The response score was calculated from the questionnaire response for each of the tasks. This score is based on the correct and incorrect responses logged by the user for each task. The response score for task i, ii and iii is used to gather evidence for hypothesis H1 and the response score for task iv is used to gather evidence for hypothesis H2. The values for the effort, confidence, and usefulness metrics are used to compare the usability and effectiveness of both the widgets. The effort score determines how organized and concise the widget was to help the user to point to the required information. The lesser the effort, the better. The confidence score shows how effective the widget was in helping the user to find the problem or required information regarding a problem. The higher the confidence score, the better the widget performed. The usefulness score indicates how much useful the user felt the information’s provided by the widget was in helping them to understand the problem and then answer the relevant queries. 4 8 6 8 8 8 4 7 8 Analysis of Data: The average scores for tasks i, ii and iii (problem understanding) for novice users and expert users was 25 and 30 respectively out of a maximum of 30. This is the average sum of the score for all the tasks performed by the user. This shows that the expert users were able to understand the problem better than novice users. This shows that the hypothesis H1 is not true.
The average score of task iv (fix identification and prioritization) for the Problem Report Widget and the Analysis Widget were 5 and 7.1 respectively. The average score shows that users found the Analytics Widget to be better than the Problem Report Widget in identifying and prioritizing the problems to be fixed. This suggests that the hypothesis H2 is true.
The average SUS usability and effectiveness scores for the Problem Report Widget and the Analysis Widget were 72.25 and 78 respectively. The average score for novice users was 74 and 79.25 for expert users. This shows that the users found Analytics Widget to be more effective and usable than the Problem Report Widget in general. Also, the expert users found the dashboard to be useful than novice users. 6
We have studied the extent to which an intelligent data-driven dashboard can assist the user to understand the data quality problems in Linked data and thus enable them to identify the fixes was investigated. User experiments were performed to investigate i) if the users were able to understand the identified problems better using the widgets in the new dashboard, ii) which widget was better in assisting the user to identify and prioritize the problems that need to be fixed. The overall result of the experiment was to find that the dashboard proved to be useful to the users but that expert users were better supported that non-experts.
Future work will improve the dashboard with additional reporting and machine learning algorithms to further automate the root cause analysis of the problems. Acknowledgement. This research has received funding from the ADAPT Centre for Digital Content Technology, funded under the SFI Research Centres Programme (Grant 13/RC/2106) and co-funded by the European Regional Development Fund.