The 3cixty platform relies on a continuous integration workflow for the generation and maintenance of evolving knowledge bases in the domain of culture and tourism. This approach is inspired by common practices in the software engineering industry in which continuous integration is widely-used for quality assurance purposes. The objective of this paper is to present a similar approach for knowledge base population and publishing. The proposed approach consists of two main steps: (i) exploratory testing, and (ii) fine-grained analysis. In the exploratory testing step, the knowledge base is tested for patterns that may reveal erroneous data or outliers that could indicate inaccuracies. This phase is knowledge-base agnostic and provides inputs for the second phase. In the finegrained analysis step, specific tests are developed for a particular knowledge base according to the data model and pre-defined constraints that shape the data. More precisely, a set of predefined queries are executed and their results are compared to the expected answers (similar to unit testing in software engineering) in order to automatically validate that the knowledge base fulfills a set of requirements. The main objective of this approach is to detect and to flag potential defects as early as possible in the data publishing process and to eliminate or minimize the undesirable outcomes in the applications that depend on the knowledge base, typically, user interfaces that enable to explore the data but rely on a particular shape of the data. This two-fold approach proves to be critical when the knowledge base is continuously evolving, not necessarily in a monotonic way, and when real-world applications highly depend on it such as the 3cixty multi-device application.
Mihindukulasooriya et al. argue that the KB generation process should learn from the experiences in the software engineering field and adapt some of these approaches for assuring KB quality.
In modern days of complex software development, it is hard to envision the
complete set of user requirements upfront and to immediately have all functionalities and
data models fully-defined. Both functionalities and data models of the software should
evolve based on user feedback, changes in business and technical environments, and
other organizational factors [
3cixty is a semantic web platform that enables to build real-world and
comprehensive knowledge bases in the domain of culture and tourism for cities. The entire
approach has been tested first for the occasion of the Expo Milano 2015 [
Data quality is a complex multi-dimensional concept commonly defined as “fitness
for use” [
Once the quality assessment is done, the assessment results will indicate the presence of quality issues. The ultimate goal is to improve the overall KB quality based on the quality issues identified by eliminating them and making sure that they will not be introduced in the future. In this paper, we present the data quality approach we used for developing the 3cixty KB which uses a continuous integration approach inspired by the software engineering industry.
The remainder of this paper is structured as follows. We discuss some related work in Section 2. Then, we present a two-fold approach for quality assurance of dynamic KBs in Section 3. We detail how this approach has been implemented in the case of the 3cixty KB, using the LOUPE and SPARQL interceptor tools in Section 4. We propose some lessons learned in Section 5 and we finally outline some future work in Section 6. 2
Related work on the topic of quality assessment of knowledge bases can be categorized
into two main groups: a) research works that define techniques and metrics for
measuring quality, and b) tools and applications that assess quality. Zaveri et al. [
Quality assessment tools and applications can be categorized into two main groups.
First, tools such as ProLOD [
Other quality assessment tools enable to perform a more fine-grained analysis of
knowledge bases. WIQA [
The Linked Data quality assurance methodologies generally consist of (i)
identification of quality requirements, (ii) execution of quality assessment, and (iii) data repair
and quality improvements [
This section presents a methodology suitable for quality assessment of continuously evolving knowledge bases that are consumed by real-world applications. The main objective of the methodology is to ensure that the periodically generated datasets (e.g., on a daily basis) meet the quality requirements of the use cases. The methodology consists of two main phases: (i) exploratory testing of the KB using dataset statistics and common patterns, and (ii) fine-grained analysis of the KB aligned with the quality requirements. 3.1
Exploratory testing (also known as ad-hoc testing) is a branch of blackbox software
testing where the testers simultaneously learn, design and execute tests in an exploratory
manner [
information gained while testing. This same idea, which is quite successful in the
software industry [
Testers can explore the statistics of the knowledge base and identify potential outliers. For example, all instances of a given class can be analyzed for checking what are the properties that are used with those instances and their cardinality. Concretely, a tester can analyze what is the cardinality of the geo location properties associated with the instances of the dul:Place class that belong to the YelpBusiness category. Whilst exploring the data instances, a tester might identify that there are certain individuals that have multiple geo location properties or some individuals without a geo location property even though the most common pattern is to have exactly one geo location property. This could indicate a quality problem in the knowledge base and the tester may report this to the developers to find the root cause of this inconsistency so that the developers correct the errors in the knowledge base generation process. Further, this indicates the need for including fine-grained analysis tests for this property so that if the same errors are repeated, they will be detected again. Thus, exploratory testing is complementary to fine-grained testing.
Furthermore, it is possible to analyze the subjects and objects of all the triples containing a given property, which enables to count the number of IRIs and blank nodes related by this property for all subjects, and to analyze the type of each subject. This information allows the exploratory tester to identify cases where the data does not fit a priori the schema (e.g., violation of domain and range constraints) or other concrete modelling decisions (e.g., no blank nodes used) that were made during the use case requirements phase.
Exploratory testing of the KB allows to identify incorrect values such as outliers. For example, if more than 99% of the values of the schema:interactionCount property are positive integers but one finds two negative values for this property, it is recommended to look into those two particular values. As discussed above, the goal of the first phase of the approach is to adapt exploratory testing for maintaining quality of the knowledge base generation process. 3.2
The goal of the fine-grained analysis of the KB is to perform a more systematic analysis using a set of test queries that are KB-specific. These test queries are run automatically every time a new version of the KB is generated and if any defects are found, they are reported to the developers. Fine-grained testing follows a similar approach to unit tests in software engineering in which tests are incorporated to the knowledge base through a continuous integration process.
Similar to the software engineering process, these tests are planned and designed in advance based on the constraints and validation rules of the KB and on the requirements of different use cases of the application that are built on top of the KB. These finegrained test scripts can check various quality dimensions such as syntactic validity, consistency, completeness, or correctness. For instance, for syntactic validity, certain values such as the location URI are checked for well-formed URI strings or the values of date fields are checked for correct date formats with a data-type declaration.
Regarding consistency, the 3cixty data model requires mandatory properties for some entities. For example, all entities of type dul:Place or lode:Event must have geo-coordinates (using respectively the geo:location or lode:inSpace property). Thus, tests are included to verify that all instances of dul:Place or lode:Event have such a property. Similarly, other cardinality restrictions and functional properties can be converted into tests in the fine-grained analysis. For instance, all entities of type dul:Place or lode:Event must also have exactly one lode:poster property while all entities of type lode:Event must have at most one dc:title per language.
For completeness, the knowledge base can be checked to see whether it contains all the entities of a given type. For instance, the number of bus stops (or metro stations) in Milan is known. Therefore, a test can be included to check if the knowledge base contains information about all bus (metro) stops in Milan. However, it is important to update regularly such tests whenever those values change so the tests are synchronized with the context of the knowledge base.
Different facts in the knowledge base can be checked for correctness using different techniques. For example, the latitude (geo:lat) and longitude (geo:long) property values of places in a given city must belong to a particular range (greater than a lower bound and lesser than an upper bound) corresponding to the city bounding box area. Hence, for each city described in the knowledge base, tests can be included to identify the entities that are a priori located outside of the city grid. 4
The two-fold methodology described in the previous section has been implemented using two tools: Loupe and SPARQL Interceptor. 4.1
Loupe5 is a tool that supports the inspection of datasets to understand which vocabularies (classes and properties) are used, including statistics and frequent triple patterns. Starting from high-level statistics, Loupe allows a tester to dig into details down to the
corresponding triples and analyze potential quality problems. As illustrated in Figure 1, Loupe consists of two main components: (a) Loupe Core and (b) Loupe UI (User Interface). In order to make the knowledge-base inspection seamless and more efficient, Loupe performs an initial indexing of the statistics and the indexed statistics and patterns are stored in an Elasticsearch server. Loupe makes also use of a docker image of a Virtuoso server6 to ease the indexing process, given an RDF dump, and to avoid overloading the public SPARQL endpoint of the knowledge base.
Once the knowledge base is indexed, the Loupe UI provides an intuitive web interface for exploratory testers to inspect the knowledge base. Loupe UI consumes the index generated by Loupe Core and generates several UI components including a summary, a class explorer, a property explorer, a triple pattern explorer, and an ontology explorer. Each of these UI components provides several levels of information starting from highlevel overviews and then allowing users to navigate into further details. Loupe UI allows testers to identify the different defective patterns that were described in Section 3.1.
Fig. 1. Loupe: The Linked Data Inspector 4.2
SPARQL Interceptor7 integrates a set of user-defined SPARQL queries (namely unitary tests for an RDF dataset) inside the Jenkins continuous integration system. SPARQL queries are defined by the dataset developers, with the objective of checking whether some specific characteristics hold in the released dataset. For example, one may want to check that a property is systematically valued for a particular entity: in 3cixty, all entities of type dul:Place must have geo-coordinates. One may also want to check
7 A deployed instance is available at http://alzir.dia.fi.upm.es:8080 that all entities are encompassed in a particular geographical coverage: in 3cixty, a city is delimited by a rectangular bounding box area and all entities of type dul:Place must be geolocalized within this bounding box. These queries are evaluated regularly, something that is done automatically by Jenkins, and summary reports are provided on the compliance of the outputs of these query evaluations with the expected results as described in the definition of the queries. Queries can then check both schema inconsistencies and data instances. The Figure 2 shows a snapshot of the SPARQL Interceptor in a real setting. Error logging follows the conventional presentation layout provided in
Jenkins with a general status of the build and a breakdown per test. For each of the failed tests, the system provides the reason for the failure, plus a description of the query and a link to execute it against the SPARQL endpoint, which is useful for the expert while checking the obtained results. 5
We use both Loupe and SPARQL Interceptor over six months to monitor the continuous deployment of the evolving 3cixty knowledge base which was at the same time used by a real application available to the million potential visitors of EXPO 2015. In this section, we report on some lessons learned, either frequent errors that we observe over iterative deployments of the knowledge base, or additional functionalities that Loupe and SPARQL Interceptor could provide.
One of the challenges that the testers have, when using Loupe for exploratory
testing, is to find the initial set of uncommon patterns so that they can explore more to
find potential defects. This happens specially when the knowledge base under test is
significantly large (both in the amount of facts as well as the number of different
vocabularies used) and when the tester does not have sufficient domain knowledge about the
knowledge base. We consider two different approaches for addressing this challenge.
On the one hand, Loupe can provide reports that highlight the uncommon patterns and
potential outliers. These reports can use different outlier detection techniques such as
population distribution based approaches or Local Outlier Factor (LOF) [
One challenge in designing fine-grained tests is to keep them synchronized with the external data sources being used to build the knowledge base. Some tests, such as the ones related to cardinality of the data model, are sometimes precise and include exact values, while others are values greater or lesser than a certain threshold. For instance, if we consider the number of bikes available in a given place, such a value changes over time, therefore setting up a priori threshold may not be effective or it would require a certain degree of abstraction that a highly dynamic context would not support. On the expert side, a plethora of error messages is something that we avoid, thus constraining the fine-grained test query types to checking to: presence of unexpected properties or property values Evaluate against a Boolean value.
In 3cixty, we are interested in points of interest that have an added value for the users and we delete in particular all places that are categorized as residential places that we have harvested during the scraping process. Therefore, we check using a unit test that no more ”home places” are present in the KB once deployed; presence of properties Evaluate against a Boolean value. In 3cixty, both Place-type and Event-type instances have required properties so we check whether there are instances with missing values for those required properties or not; presence of duplicated properties Evaluate against a Boolean value. In 3cixty, some properties cannot have more than one value so we check whether there are duplicates in the property set or not; syntax Evaluate against a Boolean value. These tests check the syntax of the data value; presence of unsolicited instances Evaluate against a Boolean value. These tests check whether an instance is present despite it should not, for example, Place-type instances that do not belong to any cell contained in the grid; number of graphs Evaluate against a cardinal number. In 3cixty, data is always associated to a publisher graph which enables to check that proper attribution has been made; number of instances/properties Evaluate against a cardinal number. These tests enable to check that the right number of properties is being used in a particular 3cixty deployment.
This classification offers a general picture of the types of query that are been tested in the 3cixty platform. Those benchmark queries are classified into: i) generic test queries and ii) city-related test queries (such as the ones evaluating the number of instances/properties that depend on the final numbers we can observe in the given city). SPARQL Interceptor visualizes such a classification in the UI, grouping the queries according to the domain and the city (namely Milan, London, Nice in the case of 3cixty). Despite that this was necessary in analysing the KB and the different components, this has increased the complexity for the KB developers in generalizing as much as possible the queries and, thus, evaluating them. 6
This paper presents a two-fold approach for ensuring the quality of dynamic knowledge bases that evolve continuously. This approach is inspired by several popular techniques in software engineering such as exploratory testing and continuous integration. Based on our experiences in the 3cixty use case, we conclude that these practices of software engineering can be adapted to knowledge-base generation. They help to maintain the quality of a KB while allowing its rapid evolution. We advocate a two-phase approach enabling to provide an overview of the evolution of a KB and a fine-grained analysis. We have described Loupe and SPARQL Interceptor, two tools that provide the necessary features for performing this two-phase approach.
Future work for Loupe includes improvements in (a) generation of quality reports, (b) utilization of machine learning techniques for (semi)-automatic detection of outliers and potentially defective patterns, (c) improvements to the visualizations. About the fine-grained analysis, future research activities will be carried out to explore the automatic generation of KB-specific test queries. This will save time in the generation ad-hoc queries, and this will attract the attention of the expert or developers in updating the queries and correctly checking the number of instances or properties being recently deployed. We will also consider expressing the SPARQL Interceptor test queries using the new Shapes Constraint Language (SHACL8) being standardized by W3C.
This work was partially supported by the FPI grant (BES-2014-068449), the innovation activity 3cixty (14523) of EIT Digital, and the 4V project (TIN2013-46238-C4-2-R).
Mihindukulasooriya et al.