One of the most di cult aspects of developing matching systems { whether for matching ontologies or for other types of mismatched data { is evaluation. The accuracy of matchers are usually evaluated by measuring the results produced by the systems against reference sets, but gold-standard reference sets are expensive and di cult to create. In this paper we introduce crptr, which generates multiple variations of di erent sorts of dataset, where the degree of variation is controlled, in order that they can be used to evaluate matchers in di erent context.
One of the central problems of data matching is the issue of evaluation: when a system returns a set of matches, how are we to determine whether they are correct or not? How exactly do we de ne what a correct match is, and how do we determine whether the proposed matches fall into that category? If we have a range of di erent options, how do we determine which is the `best' match?
In this paper we describe the use of the crptr system to create evaluation datasets for matching. crptr was developed to simulate data quality issues for test datasets used for record linkage evaluation. It can create multiple similar datasets with varying amounts of variation controlled by input settings, and provides a clear mapping back to the original dataset. This creates training and evaluation sets for matchers to run against. We have extended the crptr system to deal with structure in a context where we want to corrupt data sources in order to evaluate the semantic rewriting of queries to unknown data sources.
In Section 2 we describe the crptr system and its original application domain. Section 3 then details how we extended crptr to address corruption of other data sets and of queries. We discuss issues around evaluation in Section 4 and touch on related work in Section 5 before concluding the paper in Section 6. 0 Copyright 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
Synthetically generated data is a common approach for evaluating and testing
data analysis and mining approaches [
crptr works by using a corruption pro le that controls which methods are used and how much. The idea is that the pro le attempts to capture the data quality characteristics of the dataset being modelled. The corruption pro le consist of many factors that de ne the way data need to be corrupted such as the total number of records that need to be corrupted and the corruption methods required to be applied on the data. By controlling the factors of the corruption pro le, the user can con gure crptr to mimic the data quality characteristics that t the purpose of the research. 3
The CHAIn system (Combining Heterogenous Agencies' Information) [
Evaluation in this context therefore means determining whether the scores returned are a reasonable re ection of the distance between the original query and the rewritten query, and hence whether the ranked list is a reasonable ordering of the likely relevance of the responses to what the decision maker actually wants to know. In this context, the matching is done between the schema of the query and the schema of the target datasource1. In order to mimic the process of rewriting a query designed for one data source to succeed on a di erent data source, we create a query based on the data in a particular data source (i.e., so that it would be able to successfully query that data source) and then introduce corruption re ecting naturally-occurring di erences. We can either keep the query xed and corrupt the data source in multiple ways, or keep the data source xed and corrupt the query. In practice, we focused on corrupting datasources and then generating corrupted queries from these corrupted datasources - rstly, because it created a more generic process that was able to corrupt both datasources and queries; secondly, because it allows us to more easily focus on the part of the query that is relevant in this context, which is the terminology referring to the target datasource.
We therefore needed to extend the functionality of crptr in two ways. (i) We need to consider the domain in which this matching is occurring to determine how terms should be corrupted; (ii) Because there is a structural element to schema, we need to consider how this could be corrupted and extend the system to perform this.
In terms of the rst requirement, some of the corruptions methods in crptr
(e.g., those focusing on spelling errors) are not relevant, whilst others such as
abbreviations, need to be adapted, as some kinds of abbreviations (e.g., of rst
names) are unlikely to occur in our data sources. We need to determine what
kinds of mismatches are likely to occur in our domain, and determine what
sources we can automatically extract them from. CHAIn is designed to be
domain independent, and when addressing the problem of matching di erent (but
similar) data sources in the general case, we need a domain-independent lexical
resource to suggest the kinds of synonyms, hyponyms, hypernyms and meronyms
that di erent creators of data sources in a similar domain may naturally use. We
therefore turned to WordNet [
In terms of the second requirement, we needed to make sure any potential structural change in the schema of a CSV le was considered. This is structurally simple, consisting of columns which are named and ordered, and thus structural changes are restricted to reorganisation (addition, deletion and reordering) of the columns. For SPARQL queries in general there are, of course, many more structural elements (e.g, the potential presence of SPARQL commands such as aggregate functions), and a complete list of potential structural mismatches would be more complicated. As we are only concerned with the terms in the query which correspond with those of the expected data source, we can ignore all of the additional SPARQL structure, stripping out the relevant terms and reinserting the new terms after matching. 1 Matching at the data level is required when queries are partially instantiated.
The quality of the crptr output depends on whether the corrupted data sources it produces are a reasonable facsimile of di erent but related data sources that would naturally be found. If this is the case then we can infer that the performance of a matching system when matching di erent data sources created by crptr is a good indication of the matchers performance in real-world settings, and that therefore crptr is a useful matching evaluation tool.
This depends on two things: (i) are the terms in the look up table a good approximation of terms that could be used interchangable or in a similar way: is it modelling genuine semantic and syntactic mismatches?; (ii) are the structural mismatches introduced through the corruption process a good approximation of how similar data sources may di er? The rst is highly domain dependent. We use WordNet, which is a very widely used lexical resource. It is also likely to be of bene t to also use domain-speci c ontologies and lexicographies for each particular domain; however, these are hard to nd and often of questionable quality, so this kind of domain-speci c corruption may be hard to perform. Matching in such domains is also more e cient for the same reasons. The second aspect is domain independent but format speci c. For each format the system is extended to, an analysis of what structural mismatches are possible is necessary in order to demonstrate that the corruptions produced are plausible and thorough. 5
To the best of our knowledge, a system to generate reference sets (records, queries, RDF data sources, ontologies, etc) in order to evaluate matching in these domains is unique.
Since reference ontologies are expensive to generate and often not available,
[
Aside from ontology matching, there is existing work on generating
synthetic datasets with structural variations for relational and RDF data for use
in benchmarking. The Linked Data Benchmark Council [
In this paper we have discussed using the crptr system for generating multiple similar datasets for evaluating matchers within di erent domains. We brie y described how crptr was developed to focus on records and then extended to deal with queries based on CSV les, and could be extended to deal with other kinds of data sources. We discussed what evaluation of these corruption systems means in di erent contexts.