Data quality is commonly defined as fitness for use. The problem of identifying the quality of data is faced by many data consumers. To make the task of finding good quality datasets more efficient, we introduce the Dataset Quality Ontology (daQ). The daQ is a lightweight, extensible vocabulary for attaching the results of quality benchmarking of a linked open dataset to that dataset. We discuss the design considerations, give examples for extending daQ by custom quality metrics, and present use cases such as browsing datasets by quality. We also discuss how tools can use the daQ to enable consumers find the right dataset for use.
The Linked (Open) Data principles of using HTTP URIs to represent things have facilitated the publication of interlinked data on the Web, and their sharing between different sources. The commonly used Resource Description Framework (RDF) provides both publishers and consumers of Linked Data with a standardised way of representing data. A substantial amount of facts has already been published as RDF Linked Open Data.1 These facts have been extracted from heterogeneous sources, which also include semistructured data, unstructured data, documents in markup languages such as XML, and relational databases. The use of such a variety of sources could lead to problems such as inconsistencies and incomplete information. A common open data user’s perception2 is that the five star scheme for open data3 would automatically approve a
2This is following a discussion with some Open Data enthusiasts
at the University of Malta.
3http://5stardata.info
dataset’s quality. This statement is incorrect as the five star scheme
serves as a guide in order to lead data to reach increasing levels of
interlinkage, openness and standardisation. Therefore, although it
is favourable to have a five star linked open dataset, dataset quality
issues might still be unclear to data publishers. Various works
promote quality measurements on linked open data [
To put the reader into the context of this work, we introduce a use case:
Bob is a medical doctor and a computer enthusiast. During his free time he is currently working on a mobile application that would help colleagues to find out possible medicines to treat patients. Currently he is experimenting with a popular data management platform, hoping to find a suitable medical dataset for reuse. Fascinated by the views, especially the faceted filtering techniques available on this platform, Bob is particularly interested in reputable medical datasets. As he downloaded some datasets and viewed them in a visualisation tool, he found out that most of the data is either irrelevant for his work or contains many incorrect and inconsistent facts.
Data quality is commonly defined as fitness for use [14]. The problem of identifying the quality of data is faced by many data consumers. A simple approach would be to rate the “fitness” of a dataset under consideration by computing a set of defined quality metrics. On big datasets, this computation is time consuming, even more so when multiple datasets are to be filtered or compared to each other by quality. Apart from this, the identification of the quality of a dataset cannot be reused; other data consumers would have to do this process all over again.
To make the task of finding good quality datasets more efficient, we introduce the Dataset Quality Ontology (daQ). The daQ is a light-weight ontology that allows datasets to be “stamped” with its quality measures. In contrast to related vocabularies that represent quality requirements (cf. Section 5), our ontology allows for expressing concrete, tangible values that represent the quality of the data. Having this metadata available in the datasets enables data publishers and consumers to automatically perform tedious tasks such as filtering and comparing dataset quality. With the Dataset Quality Ontology we aim to add another star to the LOD five star scheme, for data that is not just linked and open, but of a high quality. 1.1
To prepare the reader for the discussions carried out within the following sections, we define some terminology, paraphrasing definitions by Zaveri et al. [15]:
A Quality Dimension is a characteristic of a dataset relevant to the consumer (e.g. Availability of a dataset).
A Quality Metric is a procedure for measuring a data quality dimension, which is abstract, by observing a concrete quality indicator. This assessment procedure returns a score, which we also call the value of the metric. There are usually multiple metrics per dimension; e.g., availability can be indicated by the accessibility of a SPARQL endpoint, or of an RDF dump. The value of a metric can be numeric (e.g., for the metric “human-readable labelling of classes, properties and entities”, the percentage of entities having an rdfs:label or rdfs:comment) or boolean (e.g. whether or not a SPARQL endpoint is accessible).
A Quality Category is a group of quality dimensions in which a common type of information is used as quality indicator (e.g. Accessibility, which comprises not only availability but also dimensions such as security or performance). Grouping the dimensions into categories helps to arrange a clearer breakdown of all quality aspects, given their large number. Zaveri et al. have identified 23 quality dimensions (with almost 100 metrics) and grouped them into 6 categories [15].
Whenever it is necessary to subsume all of these three concepts, we will use the term Quality Protocols. 1.2
The remainder of this paper is structured as follows: in Section 2 we discuss use cases for the daQ vocabulary. Then, in Section 3 and 4 we discuss the vocabulary design and give examples of how this vocabulary can be extended and used. Finally, in Section 5 we give an overview of similar ontology approaches before giving our final remarks in Section 6.
Linked Open Data quality has different stakeholders in a myriad of domains, however the stakeholders can be cast under either publishers or consumers.
Publishers are mainly interested in publishing data that others can reuse. The five star scheme, which we propose to extend by a sixth star for quality, defines a set of widely accepted criteria that serve as a baseline for assessing data reusability. The reusability criteria defined by the five star scheme and by quality metrics are largely measurable in an objective way. Thanks to such objective criteria, one can assess the reusability of any given dataset without the major effort of, for example, running a custom survey to find out whether its intended target audience finds it reusable. (Such a survey may, of course, still help to get an even better understanding of quality issues.)
Without an objective rating that is easy to determine, data
consumers – both machine and human – may find it challenging to
assess the quality of a dataset, i.e. its fitness for use. Machine agents,
e.g. for discovering, cataloguing and archiving datasets, may lack
the computational power required to assess some of their quality
dimensions, e.g. logical consistency. Tools for human end users, such
as semantic web search engines [12] or Web of Data browsers [
Software such as CKAN4, which is best known for driving the datahub.io platform5, makes datasets accessible to consumers by providing a variety of publishing and management tools and search facilities. A data publisher should be able to upload to such platforms, whilst on the other hand the platform should be able to automatically compute metadata regarding the dataset’s quality. With the knowledge from this metadata, the publisher can improve the quality of the dataset. On the other hand, having quality metadata available for candidate datasets, consumers would be given the opportunity to discover certain quality aspects of a potential dataset. 2.2
Tools for data consumers, such as CKAN, usually provide features such as faceted browsing and sorting, in order to allow prospective dataset users (such as Bob, introduced in the previous section) to search within the large dataset archive. Using faceted browsing, datasets could be filtered according to tags or values of metadata properties. The datasets could also be ranked or sorted according to values of properties such as relevance, size or the date of last modification. Figure 1 shows a mockup of a modified datahub.io user interface to illustrate how quality attributes and metrics could be used in a faceted search with ranking.
With many datasets available, filtering or ranking by quality can become a challenge. Talking about “quality” as a whole might not make sense, as different aspects of quality matter for different applications. It does, however, make sense to restrict qualitybased filtering or ranking to those quality categories and/or dimensions that are relevant in the given situation, or to assign custom weights to different dimensions, and compute the overall quality as a weighted sum. The daQ vocabulary provides flexible filtering and ranking possibilities in that it facilitates access to dataset quality metrics in these different dimensions and thus facilitates the (re)computation of custom aggregated metrics derived from base metrics. To keep quality metrics information easily accessible, we strongly recommend that each dataset contains the relevant daQ metadata graph in the dataset itself.
Alexander et al. [
The Dataset Quality Ontology (daQ) is a vocabulary for attaching the results of quality benchmarking of a linked open dataset to that dataset. The idea behind daQ is to provide a core vocabulary, which can be easily extended with additional metrics for measuring the quality of a dataset. The benefit of having an extensible schema is that quality metrics can be added to the vocabulary without major changes, as the representation of new metrics would follow those previously defined.
daQ uses the namespace prefix daq, which expands to http: //purl.org/eis/vocab/daq.
The basic and most fundamental concept of daQ is the Quality Graph (Figure 2 – Box A), which is a subclass of rdfg:Graph. daQ
Classes rdfg:Graph
Concept in an existing ontology Dataset Category
Concept in proposed ontology Abstract concept in proposed ontology.
Not for direct use.
Undefined object: Concept or Literal requires hasDimension
Properties
Subclass of
Abstract Property
(Not for direct use)
Property in proposed ontology
instances are stored as RDF Named Graph [
The daQ ontology distinguishes between three layers of abstraction, based on the survey work by Zaveri et al. [15]. As shown in Figure 2 Box B, a quality graph comprises of a number of different Categories (C), which in turn possess a number of quality Dimensions (D). A quality dimension groups one or more computed quality Metrics (M ). To formalise this, let G represent the named Quality Graph (daq:QualityGraph), C = fc1; c2; : : : ; cxg is the set of all possible quality categories (daq:Category), D = fd1; d2; : : : ; dyg is the set of all possible quality dimensions (daq:Dimension) and M = fm1; m2; : : : ; mzg is the set of all possible quality metrics (daq:Metric); where x; z; y 2 N, then: DEFINITION 1.
G C D
C; D;
M ;
Figure 3 shows this formalisation in a pictorial manner using Venn diagrams.
Quality metrics can, in principle, be calculated on a collection of statements - datasets or graphs. This vocabulary allows a data publisher to create multiple graphs of quality metrics for different data. For example, if one dataset consists of a number of graphs, quality metrics can be defined for each graph separately. The property daq:computedOn with domain daq:QualityGraph allows a data publisher to define a quality graph for different rdfs:Resources. The resource should be the URI of a dataset (including instances of void:Dataset6) or an RDF named graph. 3.1
This ontology framework (Figure 2) has three abstract classes/concepts (daq:Category, daq:Dimension, daq:Metric) and four abstract properties (daq:hasDimension, daq:hasMetric, daq:hasValue, daq:requires) which should not be used directly in a quality instance. Instead these should be inherited as parent classes and properties for more specific quality protocols. The abstract concepts (and their related properties) are described as follows, assuming the definitions given in Section 1.1: daq:Category represents the highest level of quality assessment.
A category groups a number of dimensions. daq:Dimension – In each dimension there is a number of metrics. daq:Metric – The smallest unit of measuring a quality dimension is a metric. Each metric has a value, representing a score for the assessment of a quality attribute. Since this value is multi-typed (for example one metric might return true/false whilst another might require a floating point number), the 6http://www.w3.org/TR/void/#dataset value’s daq:hasValue range is inherited by the actual metric’s attribute. A metric might also require additional information (e.g. a gold standard dataset to compare with). Therefore, a concrete metric representation can also define such properties using subproperties of the daq:requires abstract property. Each metric can record the date when it was actually computed using the daq:dateComputed. 4.
We start this section by first showing how the daQ vocabulary can be extended, and then proceed by giving general recommendations on how to publish daQ metadata records with datasets. We then continue by showing how a typical daQ instance is represented and we give some SPARQL examples to demonstrate how a data consumer (including tools such as the filtering UI presented in Section 2.2) can query the daQ vocabulary and a graph instance. We conclude this section by describing an application which will use the proposed vocabulary to filter and rank datasets. 4.1
The classes of the core daQ vocabulary can be extended by more specific and custom quality metrics. In order to use the daQ one should define the quality metrics which characterise the ”fitness for use“ [14] in a particular domain. However, we are currently in the process in defining the quality dimensions and metrics described in [15], as the standard set of quality protocols for Linked Open Data in daQ.7 Extending the daQ vocabulary means adding new quality protocols that inheriting the abstract concepts (CategoryDimension-Metric). In Figure 4 we show an illustrative example of extending the daQ ontology (TBox) with a more specific quality attribute, i.e. the RDF Availability Metric as defined in [15], and an illustrative instance (ABox) of how it would be represented in a dataset.
The Accessibility concept is defined as an rdfs:subClassOf the abstract daq:Category. This category has five dimensions, one of which is the Availability dimension. This is defined as a rdfs:subClassOf daq:Dimension. Similarly, RDFAvailabilityMetric is defined as a rdfs:subClassOf daq:Metric. The daq:hasValue property is also extended with a sub-property called daq:hasDoubleValue which types its range as xsd:double. The specific properties hasAvailabilityDimension and hasRDFAccessibilityMetric (sub-properties of daq:hasDimension and daq:hasMetric respectively) are also defined (Figure 4). The advantage of extending the abstract ontology concepts in Figure 2 Box B is that the domain and range of daq:hasDimension and daq:hasMetric are restricted to the appropriate quality protocols.
Extensions by custom quality metrics do not need to be made in the daQ namespace itself; in fact, in accordance with LOD best practices, we recommend extenders to make them in their own namespaces. Extending the daQ vocabulary with additional metrics assumes that their exact semantics (such as how they are to be computed) is understood by some software implementation, because daQ is intended to remain light-weight and thus not capable of expressing such semantics by its own means. Therefore, a user extending the daQ would not normally need to specify the technical requirements of the quality metric, although pointers to such requirements descriptions can be given via specialisations of the daq:requires abstract property. 4.2
7The metrics defined so far can be found under the namespace URI given above, or in the source files at https://github.com/ diachron/quality/blob/master/vocab/.
C1
D1 M1
M2
M3 D2
Cx
Dx
Mx
Dataset publishers should offer a daQ description as an RDF Named Graph in their published dataset. Since such a daQ metadata record requires metrics to be computed, it is understandable that it is not easy to author manually. Therefore, as suggested in Section 2, publishing platforms should offer such on-demand computation to dataset publishers. Another possible tool would be a pluggable platform which calculates quality metrics on datasets. Since the daQ vocabulary can be easily extended by custom quality metrics, as shown in the previous section, such a pluggable environment would allow users to import such custom metrics. One must keep in mind that the computation of metrics on large datasets might be computationally expensive; thus, platforms computing dataset quality must be scalable.
Listing 1 shows an instance of the daq:QualityGraph in a dataset. ex:qualitygraph1 is a named daq:QualityGraph. The triples show that quality metrics were computed on the whole dataset. Consumers’ queryies of the dataset for daq:QualityGraph instances will resolve to the named graph. In this named graph, instances for the daq:Accessibility, daq:Availability, daq:EndPointAvailabilityMetric and daq:RDFAvailabilityMetric are shown. A metric instance specifies the metric value and the date when it was last computed. # ... prefixes # ... dataset triples ex:qualityGraph1 a daq:QualityGraph ;
daq:computedOn <> . ex:qualityGraph1 { # ... quality triples ex:accessibilityCategory a daq:Accessibility ; daq:hasAvailabilityDimension ex:availabilityDimension . ex:availabilityDimension a daq:Availability ; daq:hasEndPointAvailabilityMetric ex:endPointMetric ; daq:hasRDFAvailabilityMetric ex:rdfAvailMetric . ex:endPointMetric a daq:EndPointAvailabilityMetric ; daq:dateComputed "2014-01-23T14:53:00"^^xsd:dateTime ; daq:doubleValue "1.0"^^xsd:double . ex:rdfAvailMetric a daq:RDFAvailabilityMetric ; daq:dateComputed "2014-01-23T14:53:01"^^xsd:dateTime ; daq:doubleValue "1.0"^^xsd:double . } # ... more quality triples
Listings 2 and 3 show typical SPARQL queries, which could be performed by data consumers. The first query retrieves all category and dimension instances from the quality graph. This query could be useful, for example, for those consumers who require to visualise all categories and dimensions available in a faceted manner. The second query retrieves all metric instances whose value (in this case a double-precision floating point number) is less than 0:5. This might be useful for identifying those metrics w.r.t. which the dataset needs serious improvement. Listing 4 shows a SPARQL
The DIACHRON project (“Managing the Evolution and
Preservation of the Data Web”8) combines several of the use cases
mentioned so far. DIACHRON’S central cataloguing and archiving hub
is intended to host datasets throughout several stages of their
lifecycle [
The daQ vocabulary is the core ontology underlying these services. It will help these services to do their jobs, i.e. adding quality metadata to datasets, which in turn is displayed on the web frontend. 8http://diachron-fp7.eu }
} }
} }
} select ?metricInst where { ?qualGraph a daq:QualityGraph . graph <http://purl.org/eis/vocab/daq> {
?metric rdfs:subClassOf daq:Metric . } graph ?qualGraph { ?metricInst a ?metric ;
daq:doubleValue ?val . filter(?val < 0.5)
To the best of our knowledge, the Data Quality Management
(DQM) vocabulary [
Our design approach is inspired by the digital.me Context
Ontology (DCON9) [
The voID10 and dcat11 ontologies recommended by the W3C
provide metadata vocabulary for describing datasets. The
“Vocabulary of Interlinked Datasets” (voID) ontology allows the high-level
description of a dataset and its links [
In this paper we presented the Dataset Quality Ontology (daQ), an extensible vocabulary to provide quality benchmarking metadata of a linked open dataset to the dataset itself. In Section 2 we presented a number of use cases that motivated our idea, including cataloguing, archiving and filtering datasets, and that helped in developing the daQ ontology (Section 3). The ontology is still in its initial phases, thus further modelling will be required in the coming months to make sure that the core vocabulary covers all concepts required for the intended use cases. This will be possible by (i) exchanging ideas with interested LOD quality researchers, and (ii) making sure that the vocabulary meets the standards required to be easily adapted by both data producers and consumers.
We are currently in the process of giving more precise definitions of the quality dimensions and metrics collected in [15]. A number 9http://www.semanticdesktop.org/ontologies/ dcon/ 10http://www.w3.org/TR/void/ 11http://www.w3.org/TR/2014/ REC-vocab-dcat-20140116/ of quality metrics are also being implemented, with the aim of providing information about the quality of big LOD datasets. This would allow us to create meaningful daQ Named Graph instances at a large scale, i.e. creating quality metadata on real datasets. The DIACHRON platform will support the daQ by ranking and filtering datasets according to the quality metadata, like we sketched in the mockup explained in Section 2.2. Having tools and platforms supporting the daQ will finally allow us to test and evaluate the vocabulary thoroughly, to see whether the daQ itself is of a high quality, i.e. fit for use. 7.
This work is supported by the European Commission under the Seventh Framework Program FP7 grant 601043 (http:// diachron-fp7.eu). [10] A. Harth. Visinav: A system for visual search and navigation on web data. Web Semantics: Science, Services and Agents on the World Wide Web, 8(4):348–354, 2010. Semantic Web Challenge 2009 User Interaction in Semantic Web research. [11] P. Heim, J. Ziegler, and S. Lohmann. gFacet: A browser for the web of data. In Proceedings of the International Workshop on Interacting with Multimedia Content in the Social Semantic Web (IMC-SSW 2008), volume 417 of CEUR Workshop Proceedings, pages 49–58, Aachen, 2008. [12] A. Hogan, A. Harth, J. Umbrich, S. Kinsella, A. Polleres, and S. Decker. Searching and browsing linked data with SWSE: The semantic web search engine. Web Semantics: Science, Services and Agents on the World Wide Web, 9(4):365 – 401, 2011. {JWS} special issue on Semantic Search. [13] A. Hogan, J. Umbrich, A. Harth, R. Cyganiak, A. Polleres, and S. Decker. An empirical survey of linked data conformance. J. Web Sem., 14:14–44, 2012. [14] J. M. Juran. Juran’s Quality Control Handbook.
Mcgraw-Hill (Tx), 4th edition, 1974. [15] A. Zaveri, A. Rula, A. Maurino, R. Pietrobon, J. Lehmann, and S. Auer. Quality assessment methodologies for linked open data (under review). Semantic Web Journal, 2012. This article is still under review.