Over the past years, a vast number of datasets have been published based on Semantic Web standards, which provides an opportunity for creating novel industrial applications. However, industrial requirements on data quality are high while the time to market as well as the required costs for data preparation have to be kept low. Unfortunately, many Linked Data sources are error-prone which prevents their direct use in productive systems. Hence, (semi-)automatic quality assurance processes are needed as manual ontology repair procedures by domain experts are expensive and time consuming. In this article, we present CROCUS { a pipeline for cluster-based ontology data cleansing. Our system provides a semi-automatic approach for instance-level error detection in ontologies which is agnostic of the underlying Linked Data knowledge base and works at very low costs. CROCUS was evaluated on two datasets. The experiments show that we are able to detect errors with high recall.
The Semantic Web movement including the Linked Open Data (LOD) cloud1
represents a combustion point for commercial and free-to-use applications. The
Linked Open Data cloud hosts over 300 publicly available knowledge bases with
an extensive range of topics and DBpedia [
Industrial environments need to provide high quality data in a short amount of time. A solution might be a signi cant number of domain experts that are checking a given dataset and de ning constraints, ensuring the demanded data quality. However, depending on the size of the given dataset the manual evaluation process by domain experts will be time consuming and expensive. Commonly, a dataset is integrated in iteration cycles repeatedly which leads to a
generally good data quality. However, new or updated instances might be errorprone. Hence, the data quality of the dataset might be contaminated after a re-import.
From this scenario, we derive the requirements for our data quality evaluation process. (1) Our aim is to nd singular faults, i.e., unique instance errors, con icting with large business relevant areas of a knowledge base. (2) The data evaluation process has to be e cient. Due to the size of LOD datasets, reasoning is infeasible due to performance constraints, but graph-based statistics and clustering methods can work e ciently. (3) This process has to be agnostic of the underlying knowledge base, i.e., it should be independent of the evaluated dataset.
Often, mature ontologies, grown over years, edited by a large amount of processes and people, created by a third party provide the basis for industrial applications (e.g., DBpedia). Aiming at short time-to-market, industry needs scalable algorithms to detect errors. Furthermore, the lack of costly domain experts requires non-experts or even layman to validate the data before in uencing a productive system. Resulting knowledge bases may still contain errors, however, they o er a fair trade-o in an iterative production cycle.
In this article, we present CROCUS, a cluster-based ontology data cleansing framework. CROCUS can be con gured to nd several types of errors in a semiautomatic way, which are afterwards validated by non-expert users called quality raters. By applying CROCUS' methodology iteratively, resulting ontology data can be safely used in industrial environments.
Our contributions are as follows: we present (1) a pipeline for semi-automatic instance-level error detection that is (2) capable of evaluating large datasets. Moreover, it is (3) an approach agnostic to the analysed class of the instance as well as the Linked Data knowledge base. Finally, (4) we provide an evaluation on a synthetic and a real-world dataset. 2
The research eld of ontology data cleansing, especially instance data can be regarded threefold: (1) development of statistical metrics to discover anomalies, (2) manual, semi-automatic and full-automatic evaluation of data quality and (3) rule- or logic-based approaches to prevent outliers in application data.
In 2013, Zaveri et al. [
schema extension for preventing already identi ed errors. Those methods measured an error rate of 11.93% in DBpedia which will be a starting point for our evaluation.
A rule-based framework is presented by Furber et al. [
In 2013, Kontokostas et al. [
A rst classi cation of quality dimensions is presented by Wang et al. [
To the best of our knowledge, our tool is the rst tool tackling error accuracy (intrinsic data quality), completeness (contextual data quality) and consistency (data modelling) at once in a semi-automatic manner reaching high f1-measure on real-world data. 3
First, we need a standardized extraction of target data to be agnostic of the
underlying knowledge base. SPARQL [
Second, CROCUS needs to calculate a numeric representation of an instance to facilitate further clustering steps. Metrics are split into three categories: (1) The simplest metric counts each property (count ). For example, this metric can be used if a person is expected to have only one telephone number.
(2) For each instance, the range of the resource at a certain property is counted (range count ). In general, an undergraduate student should take undergraduate courses. If there is an undergraduate student taking courses with another type (e.g., graduate courses), this metric is able to detect it.
(3) The most general metric transforms each instance into a numeric vector and normalizes it (numeric). Since instances created by the SCDB consist of properties with multiple ranges, CROCUS de nes the following metrics: (a) numeric properties are taken as is, (b) properties based on strings are converted to a metric by using string length although more sophisticated measures could be used (e.g., n-gram similarities) and (c) object properties are discarded for this metric.
As a third step, we apply the density-based spatial clustering of applications
with noise (DBSCAN) algorithm [
Finally, identi ed outliers are extracted and given to human quality judges. Based on the revised set of outliers, the algorithm can be adjusted and constraints can be added to the Linked Data knowledge base to prevent repeating discovered errors. 4
LUBM benchmark. First, we used the LUBM benchmark [
The LUBM benchmark generates random but error free data. Thus, we add di erent errors and error types manually for evaluation purposes:
{ completeness of properties (count) has been tested with CROCUS by adding a second phone number to 20 of 1874 graduate students in the dataset. The edited instances are denoted as Icount. { semantic correctness of properties (range count) has been evaluated by adding for non-graduate students (Course) to 20 graduate students (Irangecount). { numeric correctness of properties (numeric) was injected by de ning that a graduate student has to be younger than a certain age. To test this, 20 graduate students (Inumeric) age was replaced with a value bigger than the arbitrary maximum age of any other graduate.
For each set of instances holds: jIcountj = jIrangecountj = jInumericj = 20 and additionally jIcount \ Irangecount \ Inumericj = 3. The second equation overcomes a biased evaluation and introduces some realistic noise into the dataset. One of those 3 instances is shown in the listing below: 1 @prefix rdf : <http : / /www. w3 . org /1999/02/22 rdf syntax ns#> . 2 @prefix r d f s : <http : / /www. w3 . org /2000/01/ rdf schema#> . 3 @prefix ns2 : <http : / / example . org/#> . 4 @prefix ns3 : <http : / /www. Department6 . U n i v e r s i t y 0 . edu/> . 5 6 ns3 : GraduateStudent75 a ns2 : GraduateStudent ; 7 ns2 : name " GraduateStudent75 " ; 8 ns2 : undergraduateDegreeFrom <http : / /www. U n i v e r s i t y 4 6 7 . edu> ; 9 ns2 : emailAddress " GraduateStudent75@Department6 . U n i v e r s i t y 0 . edu " ; 10 ns2 : t e l e p h o n e "yyyy-yyyy-yyyy" , "xxx xxx xxxx " ; 11 ns2 : memberOf <http : / /www. Department6 . U n i v e r s i t y 0 . edu> ; 12 ns2 : age "63" ; 13 ns2 : t a k e s C o u r s e ns3 : GraduateCourse21 , ns3:Course39 , ns3 :
GraduateCourse26 ; 14 ns2 : a d v i s o r ns3 : A s s o c i a t e P r o f e s s o r 8 .
Listing 1.1: Example of an instance with manually added errors (in red ).
DBpedia - German universities benchmark. Second, we used a subset of the English DBpedia 3.8 to extract all German universities. The following SPARQL query (Listing 1.2) presents already the di culty to nd a complete list of universities using DBpedia. 1 SELECT DISTINCT ? i n s t a n c e 2 WHERE f 43 f 5 6 7 8 9 10 11 12 13 14 gg ? i n s t a n c e a dbo : U n i v e r s i t y . ? i n s t a n c e dbo : country dbpedia : Germany .
? i n s t a n c e f o a f : homepage ?h . g UNION f ? i n s t a n c e a dbo : U n i v e r s i t y . ? i n s t a n c e dbp : : country dbpedia : Germany .
? i n s t a n c e f o a f : homepage ?h . g UNION f ? i n s t a n c e a dbo : U n i v e r s i t y . ? i n s t a n c e dbp : : country "Germany"@en .
? i n s t a n c e f o a f : homepage ?h .
Listing 1.2: SPARQL query to extract all German universities.
After applying CROCUS to the 208 universities and validating detected instances manually, we found 39 incorrect instances. This list of incorrect instances, i.e., CBD of URIs, as well as the overall dataset can be found on our project homepage. For our evaluation, we used only properties existing in at least 50% of the instances to reduce the exponential parameter space. Apart from an increased performance of CROCUS we did not nd any e ective drawbacks on our results.
Results. To evaluate the performance of CROCUS, we used each error type individually on the adjusted LUBM benchmark datasets as well as a combination of all error types on LUBM5 and the real-world DBpedia subset. count
range count numeric 5 The datasets can also be found on our project homepage.
Table 2 presents the results for the combined error types as well as for the German universities DBpedia subset. Combining di erent error types yielding a more realistic scenario in uences the recall which results in a lower f1-measure than on each individual error type. Finding the optimal M inP ts can e ciently be done by iterating between [2; : : : ; jIj]. However, CROCUS achieves a high recall on the real-world data from DBpedia. Reaching a f1-measure of 0:84 for LUBM and 0:91 for DBpedia highlights CROCUS detection abilities. M inP ts F1
dbp:staff, Values are typed as dbp:estab- xsd:string although lished, they contain numeric dbp:internat- types like integer or ionalStudents double. dbo:country, dbp:country dbp:country "Germany"@en collides with dbo:Germany
In general, CROCUS generated many candidates which were then manually validated by human quality raters, who discovered a variety of errors. Table 3 lists the identi ed reasons of errors from the German universities DBpedia subset detected as outlier. As mentioned before, some universities do not have a property dbo:country. However, we found a new type of error. Some literals are of type xsd:string although they represent a numeric value. Lists of wrong instances can also be found on our project homepage.
Overall, CROCUS has been shown to be able to detect outliers in synthetic and real-world data and is able to work with di erent knowledge bases. 5
We presented CROCUS, a novel architecture for cluster-based, iterative ontology data cleansing, agnostic of the underlying knowledge base. With this approach we aim at the iterative integration of data into a productive environment which is a typical task of industrial software life cycles.
The experiments showed the applicability of our approach on a synthetic and, more importantly, a real-world Linked Data set. Finally, CROCUS has already been successfully used on a travel domain-speci c productive environment comprising more than 630.000 instances (the dataset cannot be published due to its license).
In the future, we aim at a more extensive evaluation on domain speci c knowledge bases. Furthermore, CROCUS will be extended towards a pipeline comprising a change management, an open API and semantic versioning of the underlying data. Additionally, a guided constraint derivation for laymen will be added.
Acknowledgments This work has been partly supported by the ESF and the Free State of Saxony and by grants from the European Union's 7th Framework Programme provided for the project GeoKnow (GA no. 318159). Sincere thanks to Christiane Lemke.