Current linked open data standards have encouraged the publication of a large number of data sets on the public Web. While some data providers put a lot of energy and resources into maintaining high quality data, others do not, meaning that the quality of the data in many LOD sources is variable and unpredictable. This makes the construction of novel applications on top of the data more di cult and expensive than it otherwise would be. However, these same data standards also open up possibilities for new ways of managing information quality (IQ). In this paper, we propose one such approach, the IQ-bot, and present the results of our study of its feasibility. An IQbot is a 3rd party component that crawls the Web of data, looking for changes that have been made to data sets, and inferring from them where a correction to a data defect has been made. These corrections can then potentially be made available for application to other databases showing evidence of the presence of the same data defect. In this way, the bene ts of the curation e ort put into a small number of data sets can be propagated throughout the Web of data.
Linked Open Data standards make possible the creation of
a range of new applications, built with data sets that were
formerly hidden behind web forms or proprietary query
interfaces, or else not publicly accessible at all.
Unfortunately, the quality of many linked open data sets is extremely
variable. Poor quality data can raise the cost of building
and maintaining new applications; prohibitively so, in some
cases. Or, the problems caused by the poor quality data
Ideally, such data consumers will report the defects they nd
back to the owners of the original source. In practice, this
does not always happen (although the process is becoming
easier as more sources provide convenient routes for error
reporting). Even when errors are reported, the source
owners do not always have the resources to be able to check and
make the corrections immediately. The result is that the
same error may be found and corrected repeatedly by many
di erent data consumers, in isolation from each other's
efforts. Few resources exist (beyond promising experiments
such as the PatchR repository [
There is therefore a need for a mechanism to facilitate sharing of both identi ed defects and their corrections. Any such mechanism should preserve the freedom of data providers and consumers to structure their data as they wish, and should avoid the imposition of heavy-duty global standards or conventions. In other words, the autonomy of the participating sources should be respected, as should the scarcity of time and attention from the domain experts that are available to nd and x the problems.
In this paper, we propose one possible approach to providing such a mechanism: the IQ-bot (short for "Information Quality Bot"). An IQ-bot is a lightweight, 3rd party component that crawls data sets of interest to a community, looking not for data defects per se, but for corrections to data. The IQ-bot reports any corrections it nds to a central Data Corrections Server. This server is accessed by data set owners, who can query it to see if any of the corrections found so far might be applicable to their own data. Relevant corrections can then be applied to the local data, to raise its quality level.
The fact that our IQ-bots search for data corrections rather than data defects is important. The semi-automated discovery of data defects is a Holy Grail for the IQ community. Many tools have been developed with this goal in mind, the most successful of which are probably the data pro ling tools such as TS Discovery1 and Talend Data Pro ler2. These tools can help a data owner or consumer to understand the overall landscape of their data, and to identify outliers and other deviations from the \normal" behaviour of the data. However, the identi cation of true defects from the mass of information provided by the tools typically requires both domain knowledge and an understanding of the state of the world relative to the data at speci c points in the past. This, like any task, requiring signi cant human input, is expensive.
Data corrections, on the other hand, are concrete changes to data, identi able using entirely automatic means. They form a visible trail in the data, in some cases recorded explicitly in the form of a change (or transaction) log, in other cases encoded implicitly in the di erence between versions or snapshots of the data. Some data changes arise from the addition of new data (addressing incompleteness of the data set), while others make changes to existing data (addressing inaccuracies, inconsistencies or a lack of currency in the data, for example). In other words, the changes can be viewed as actions taken by the data owner in response to a perception of poor data quality or the presence of a data defect. The corrections tell us both something about the defect that was identi ed and the action that was decided on as being an appropriate response.
By searching for and storing these corrections, we can gain extra value from the work done by curators and domain experts to locate data defects and remove them. The e ort put into curating one resource can be picked up by an IQ-bot, and made available for easy application by the owners of other sources. This acts as a counterpoint to the (currently more common) process of data pollution, whereby data from one source is copied into another, propagating defects from the rst source to the second, and onwards to any other dependent sources. IQ-bots give us a mechanism for spreading not defects, but corrections to defects throughout the Web of data. Thus, e ort put into raising the quality levels of core data sets can have bene ts for a much wider range of data sources, including sources unknown to the curators doing the original defect correction work.
In the remainder of this paper, we present our preliminary 1www.trilliumsoftware.com/home/products/TSDiscovery .aspx 2www.talend.com/resource/data-profiler.html study into the feasibility of the IQ-bot approach for nding and applying data corrections on typical linked data sources. We begin by surveying the state of the art in information quality management for linked open data (Section 2). Next, we present an architecture for the IQ-bot approach, showing the core components that are needed and the way they work together (Section 3). Subsequently, we present the results of our two feasibility studies, the rst of which explores the extent to which current linked open data sets are versioned or are amenable to snapshot di erencing techniques (Section 4) and the second of which considers the changes made to one speci c versioned data set, and whether they can be applied to other data sets to improve data quality (Section 5). Finally, we conclude and outline some of the near-term future directions for the work (Section 6).
Linked Open Data (LOD) is a set of principles and
technologies that enable owners of data sets to share their data across
the Web. With the constant growth in the number and size
of LOD data sources, LOD has given rise to a wealth of freely
available, structured information on the Web, which
organizations as well as citizens have been taking advantage of by
browsing and analysing the open Web of data. LOD is
usually represented in RDF (the Resource Description
Framework), having the form of an entity-relationship graph in
which facts consisting of triples of subjects, predicates and
objects are speci ed. Data sets are typically connected to
one another through links between entities, showing where
they represent the same or related real world entities.
The growing popularity of LOD has led the information
quality community to investigate its quality under a
variety of use cases. For example, Kontokostas et al. proposed
a methodology for quality assessment of linked data, which
employs SPARQL query templates that can be
automatically instantiated into quality test queries [
Despite the amount of work addressing the use of data
dependencies and integrity constraints for error detection and
data repairing in recent years (e.g., [
Perhaps the most comprehensive contribution to LOD
quality management to date is the work of Abedjan et al., in
which a Web-based tool for a variety of LOD data
management tasks is proposed (ProLOD++) [
All this work represents progress, but many important aspects of the data quality problem in a LOD context are still unaddressed. To the best of our knowledge, no tool or resource has yet been proposed for identifying corrections in curated LOD sources, to enable the bene ts of the work done to maintain high quality in those sources to be propagated further through the Web of data.
Three di erent types of component are needed to support crawling the Web of data for data corrections and storing them in a reusable form. Figure 1 illustrates these component types in one typical con guration. In the gure, boxes with a grey outline are pre-existing data sources, while boxes outlined in black show the new components needed to carry out the crawling and correction application processes. On the leftmost side of the gure, we see a number of preexisting data sources. These are the versioned, curated data sources that a particular user community wishes to monitor for data corrections3. A number of IQ-bot components are connected to these sources. The IQ-bots perform regular monitoring of the sources for data updates. For example, an IQ-bot monitoring a database of gene information might discover from an update that the gene formerly known as \FTHFD" is now known as \ALDH1L1"4. Exactly what this monitoring entails depends on the sources themselves and the facilities they o er for notifying external agents of changes (or the lack of them). We envisage that multiple IQ-bot components might be active simultaneously in the general case, providing load balancing and also handling different kinds of sources with di erent change management facilities.
The IQ-bots report any changes they nd to a central Defect Corrections Server. This server stores the corrections in its persistent store, as well as potentially classifying and ordering the corrections in order to make them easier to nd and apply. In the gure, we show just a single server, although it is possible to imagine scenarios in which multiple servers might exist (where, for example, the di erent servers store corrections from data in di erent domains).
Meanwhile, a bioinformatician working with her own collection of gene data (created by combining data generated in her lab with data from species-speci c public databases such as SGD5) wishes to carry out an important analysis. Before running the lengthy analysis, she rst runs her local Correction Engine, to make any improvements to the quality of the data that might have been discovered. The Correction Engine connects to the user's preferred Defect Correction Server, and checks if any new corrections have been discovered since the last update to this source. The Correction Engine then examines the local data sources, looking for opportunities to apply the new corrections to the data. If there are matches, then these are presented to the bioinformatician, who can choose to accept them (apply them to the 3In this con guration, the users prede ne the list of sources to be monitored. In others, the sources might be identi ed by some other means, such as by following links in data sets. 4www.uniprot.org/uniprot/Q8R0Y6 5yeastgenome.org data) or reject them as being inappropriate.
The two studies described in the rest of this paper aim to assess the feasibility of providing this kind of facility, in the context of the current linked open data landscape.
In order to determine whether the IQ-bot approach just described is feasible, we undertook a study of a sample of linked data sets. Our aim was to answer the following questions.
The IQ-bot approach depends crucially on the ability to e ciently identify changes that have been made to data sources. This is made much easier if sources are explicitly versioned by their providers, using software platforms that give details of the changes between versions. We therefore wanted to know: what software support is available for versioning of linked open data sets? and what proportion of linked open data sets currently use this software support? If a data set of interest is not accessible through a version management framework, the IQ-bot approach can still deliver results if previous versions of the data set are made available. That is, instead of providing just the most recent version of the data, with all updates applied, some linked open data providers maintain RDF dumps of previous major versions of their data. In this case, we can identify the changes that have been made in a version by comparing its contents with its most recent predecessor. So, if a source creates new versions relatively frequently, and makes them publicly available for some time afterwards, the IQ-bot approach can be considered feasible if su cient time is available to perform the snapshot di erencing. We therefore wanted to know: what proportion of LOD sources make their previous versions available, after the data has been updated? and what frequency of updates are typical? If versions are maintained for only a very small number of LOD sources, then the IQ-bot approach may not be feasible. In this case, we have another option. We can take regular snapshots of the data set to be monitored, and store the most recent locally for comparison when the next snapshot is taken. By comparing successive pairs of snapshots, we can discover the changes that have been made recently. This option requires only that we can take an RDF dump of the data set (or can produce one through a query or set of queries at a SPARQL endpoint), and that we have su cient local disk storage to be able to keep copies of the RDF dumps of all sources we are monitoring, between snapshots. This requirement doesn't seem onerous, given current disk storage costs. However, for completeness, we have also attempted to gain answers to the following questions: what proportion of linked open data sets are available as an RDF dump, or offer a SPARQL endpoint? and what sizes are typical for linked open data sets? The answers that emerged from our feasibility study are described below.
Versioning Support for LOD. We are beginning to see the emergence of platforms for linked open data that provide explicit support for versioning. The prime example of this at present is Apache Marmotta6, which o ers a versioning module on top of the KiWi data store. When versioning is enabled, a Marmotta application will automatically track changes to RDF data, at the transaction level, and allow speci c versions of the data set to be queried. It also supports the creation of snapshots of the data, which can be referred to and queried directly, too.
This kind of facility is exactly what we need to implement
IQ-bot-style correction crawling e ciently. Unfortunately,
we couldn't nd any publicly accessible versioned data sources
that used Marmotta. The main RDF triple stores (we looked
at Jena's TDB7, Virtuoso8 and Sesame9) o er support for
transactions but not versions per se. Some proposals for
handling versioning of RDF data can be found in the
literature (e.g. the work of Van de Sompel et al. [
Snapshot Comparisons for LOD. Many more data sets become eligible for IQ-bot-style crawling if we can record snapshots of the source contents at regular intervals, and work out the di erences between them. We again used a script to query the Datahub.io index of data sets. The tags indicate that just 27 out of the full set of 902 data sets o ered an RDF dump of their data. This seems very low, calling the accuracy of this tag into question. We did a manual search of 5 of the sources, and found that 4 out of 5 o ered an RDF dump, but on their own Website and not through Datahub.io. Data sizes of these dumps range from 17,000 triples to greater than 1 billion triples. Clearly, the storage of many RDF dumps from the larger sources would have to be justi ed by signi cant data improvement resulting from the data corrections that could be detected and reused. The tags on Datahub.io suggest that 510 of the data sets o er a SPARQL endpoint through datahub, which looks promising as a last ditch route to obtaining snapshots. However, the (anecdotal) experience of our colleagues working with linked open data is that the proportion of data sets with live, usable SPARQL endpoints is likely to be much lower than this.
Our general feasibility study showed that there are su cient access routes to change information for linked open data sets to make IQ-bots a realistic option in certain domains, even if some work has to be put in (for example, to allow snapshot comparisons). We further wanted to assess whether we could identify actual data corrections (and the defects that lie behind them) from a real data source, and whether the corrections we found could be applied to other data sources. To achieve this, we needed a versioned, wellcurated, widely-used data source. We chose the well known UniProt database10 for this second feasibility study. UniProt is a comprehensive, heavily curated database of information on protein sequence, structure and function. The database aims to document the set of proteins that have been discovered so far, plus our current understanding of their form and function within living organisms, and the evidence (experimental or otherwise) that supports this interpretation. It is a major global resource for scientists working in molecular biology and cognate disciplines.
Information in UniProt comes from two main sources. A team of expert curators works to extract high-quality information from the scienti c literature, and to associate this with the relevant protein records (called \entries" in UniProt terminology). Though well resourced, this team can cover only a small fraction of the proteins that are stored in the database, with new proteins being discovered frequently every year. Therefore, a second source of information is 10uniprot.org utilised: automated analyses of the literature and the data in UniProt make predictions about the features of un-curated proteins. This data is also added to the database, but is annotated with evidence codes to indicate to users that it comes with a weaker supporting evidence base than the information entered by the expert curators.
The UniProt team produces a new release of the database every four weeks. We examined a single release, to see if we could identify defect corrections from the changes made in that release. UniProt was a good choice for this study because of the high frequency of changes made by the curators, and because it provides several ways to access the details of the changes that have been made in each release: The UniProt Web search form allows comparison of arbitrary versions of a particular entity, showing the results as a di erence between two textual representations of the entry from the selected versions.
A Java API for accessing UniProt remotely allows us to query for the entries that have changed in a particular release. (We then have to compare the updated entries for ourselves, before we can get information about the speci c changes that have been made.) Although access to these versioning facilities is not currently available for RDF/SPARQL users, the UniProt database is made available as an RDF dump, leaving open the possibility of storing the dump after each release, for snapshot comparison purposes.
Since for our feasibility study, we planned to do the analysis by hand, we chose to use the search form for extracting a text-based representation of the changes. We examined changes made in the release covering 1st March 2014 to 24th March 2014. During this time, changes were made to 5,139 protein entries in UniProt. This is typical of the scale of changes that occur in each release. Of these changed entries, we looked at 14 in detail to extract data corrections11. Most of the changed entries had been updated in several ways, so the actual number of detailed changes we examined was 134. By \change" here we mean a small conceptual change, such as adding a new feature to a protein or deleting a keyword from a set of keywords. Since we were working with the textual comparison interface, without full knowledge of the underlying database schema, it was not possible for us to work out in detail the exact number of attribute value changes, additions and deletions. Table 1 summarises the results.
Space precludes us from giving our detailed analysis of each of the entries. Instead, we will present some examples of the changes we examined, and our classi cation of them according to the underlying defect, for the di erent IQ dimensions covered.
Completeness: several of the entries were changed in order to increase the completeness of the information stored. As our collective understanding of the structure and function of proteins grows, incompleteness 11This number was chosen based on the amount of time we had available to research each changed entry, rather than having any signi cance in itself. can arise in our scienti c databases, unless work is done to repair it. This is a major task for the expert curators for UniProt, who examine the literature, and amend the data to include new results and interpretations appearing in credible research sources. An example of this can be seen in the changes to entry Q9LFN6. Here, a new publication has been added (indicating a new source of evidence about the protein), and new features have been added to the entry as a result. The old descriptions of the function and subcellular structure of this protein were annotated with the phrase \By Similarity". This means that they were added to the entry not because of strong experimental evidence for this particular protein, but because strong experimental evidence linked these terms to a protein that is structurally similar to this one. The new publication apparently provides information and evidence directly about this protein. The old features acquired \by similarity" were therefore replaced with the new information, for which stronger evidence exists. This change increases both the completeness and also the credibility of the entry.
Another source of new data for UniProt comes from the results of automated analyses. New results can be generated when better algorithms are implemented, or when existing algorithms are run over new (more complete) data sets. This kind of completeness correction can be seen in entry P31414. It seems likely that all the 67 changes made to this entry in this record arose because new information was added to a protein to which P31414 is similar. Prior to this release, this entry had a small set of annotations postulating the existence of various helical structures within the protein chain. The evidence codes indicated that these annotations had come from an automated analysis tool. In this release, these speculative annotations are replaced by a host of more detailed and precise annotations coming from another protein \by similarity". Presumably, new (strongly supported) information had become available for this similar protein, and was therefore copied over in the entry for P31414.
This particular completeness correction is a complex
one; it has elements of consistency, currency and
precision. (We have noted elsewhere the fact that real
data defects often combine elements of several IQ
dimensions, rather than being cleanly classi able within
the scope of just one [
Currency: both entries P62258 and P31947 have a change to an annotation placed on a link to another database. In this case, the link is to a record in BioGrid12, a database of protein and gene interactions. The value that annotates the link is a number which seems to correspond to the number of interactions currently present in BioGrid for the protein that the entry represents. If a recent new release of BioGrid had more interactions for this protein, then the count of the interactions stored in UniProt would need to increase, to maintain consistency with BioGrid, and currency of the number of interactions recorded. This appears to be what happened to both these entries in this release. Accuracy: the entry with accession number P322169 was involved in just one change: an update to correct the end page number on one of the publications associated with the entry. The page numbers for this paper were changed from \1454{1474" to \1454{1462". It is not clear whether the error being corrected here arose during data entry into UniProt, or whether erroneous data was copied from another system.
Syntactic accuracy: the changes to entries P0CI58 and D9U298 both involve the correction of a typo. Both entries contain a link to a speci c record in the InterPro database13 with accession number IPR002061. InterPro is a database that uses predictive models to assign proteins to protein families, based on their sequence and structure, and information in other databases. As well as pointing to the relevant InterPro record, entries 12thebiogrid.org 13www.ebi.ac.uk/interpro in UniProt appear also to contain a copy of the family name from InterPro.
In this case, a typo was present in the family name. Originally, the family name was given as the string \Scorpion toxinL/defesin", but this is not correct. The last word of the family name should be \defensin". Presumably, this error was originally present in InterPro, but has since been corrected there, and this change is the propagation of that correction to UniProt. (We cannot easily check this, as InterPro does not appear to provide access to its recent versions.) Precision: An example of a correction where information was made more precise can be found in entry P98057, where two terms describing the function of the protein are replaced by the term that is their common child in the ontology (in this case, the GO ontology of structural and functional terms for genes and proteins14). The version of this entry prior to this data release was annotated with the following two terms: { ion transmembrane transport { proton transport In the version of the entry that is part of this March 2014 release, however, these two terms were removed and replaced with:
{ hydrogen ion transmembrane transport which is a child of both the deleted terms in the GO ontology. This change, then, appears to be a re nement of the interpretation of the function of this protein, to use a more speci c (and therefore more precise) term. Our experience in looking at these changes was encouraging. With some basic domain knowledge, and some checks of related information in other databases, we were able to form a set of hypotheses about the kinds of defect being corrected in the case of all but one of the changes. This hard-to-classify change was an update to an evidence code assigned to an annotation. Unlike the other changes to evidence codes we observed, this involved replacing a (slightly) stronger evidence code (whose source was given as the UniProtKB itself) with a weaker evidence code (with its source labelled as being the Ensembl database). We could not form any good hypotheses as to why this stronger evidence code was being replaced by a weaker one, and therefore had di culty in categorising this correction under one of the defect types. However, it is entirely possible that this failure is due to gaps in our knowledge of this domain, and of the speci c circumstances of this change.
We next set about looking for other databases (not just copies of UniProt) where some of these corrections could be applied, to improve the data quality. Since we were performing this process manually, we were limited to shallow Web searches using Google, and (where available) individual deep Web searches using keyword based search forms on the resources themselves. Nonetheless, we were able to identify the same defect present in other databases for several of the changes. Here, we list a few representative examples: 14geneontology.org The replacement of the two GO terms with their common child (from entry P98057). We found a couple of databases which had the two deleted GO terms from this change, but not the child term. One example is from ZFin, the database of gene information about the Zebra Fish species15. The two deleted terms appear in the set of \biological process" annotations for the ATP6V1AA gene. Of course, there may be good reasons why the child term is not used in the case of this gene; an expert decision would need to be made before the correction could be safely applied. But, the IQ-bot can ag the existence of the possibility to the expert, and assist in making and documenting the change if the expert decides to apply it. We also found the pattern of terms relevant to this correction in BioCyc16, on the record for the gene mrpE (accession number: BSU31640), and in SGD17, on the record for the gene YGR020C.
The correction for the \Scorpion toxinL/defesin" typo (from entries P0CI58 and D9U298). We were able to locate a database containing this typo: the LAMP database of anti-microbial peptides, managed by Fudan University in China. The relevant record could be found at the following link, at the time of writing: biotechlab.fudan.edu.cn/database/ lamp/detail.php?id=L01A000663 The addition of a new publication (to entry Q9LFN6). We were not able to locate a data source that has exactly the same set of publications as the UniProt version of this entry before the March release. However, we did locate databases with records for this protein where this publication was missing. This was the case for both the Gramene database18, which recorded no publications at all for the protein (despite having schema elements for the recording of publications), and the NCBI database19 (which records the full set of publications for the gene, but only one publication for the corresponding protein).
From the above, we can see that we were able to view almost all the changes in the examined entries as defect corrections, and that we were able to infer something about the form and kind of the defect that is being corrected by the change. With a little e ort, we were also able to locate other datasets where the corrections might have been able to point out problems to the data owners. The next step is to undertake a more systematic study, using a prototype IQ-bot, to discover whether more places where the corrections could be applied can be discovered by chasing links in the linked data sets.
From our rst feasibility study, we showed that some access to change information can be gathered from a significant proportion of linked open data sets, even if speci c software support for working with versions is not at present 15zfin.org 16biocyc.org 17yeastgenome.org 18ensembl.gramene.org 19www.ncbi.nlm.nih.gov/protein/238481238 widely adopted. The usefulness of the (somewhat simple) version management facilities provided by UniProt (behind the scenes, when viewed from a linked data perspective) in carrying out our second feasibility study illustrates the value that could accrue if more use is made of tools such as Apache Marmotta's versioning module.
A second feasibility study, focussing on UniProt, also showed that di erences between versions of well curated sources can be a rich source of information on the data defects discovered in those sources by the curation processes (both manual and automatic), as well as providing details of how to deal with those defects. Defects from all the basic IQ dimensions were discovered in our small study of a tiny part of just one release, which bodes well for the usefulness of the technique as a whole, when applied to entire releases over a period of time. In addition, we were able to nd related databases that also contained some examples of the defects these corrections were intended to address.
We are in the process of implementing our rst IQ-bot, plus a Corrections Server to record the corrections/defects it nds and a simple Correction Engine to allow the recorded corrections to be applied to a locally stored data set. Our rst IQ-bot monitors the UniProt database for changes, using its embedded versioning support that allows us to request a list of changed entries between speci c versions programmatically.
A number of important research questions remain to be answered before a full IQ-bot system can be deployed for actual use, and before the value of the approach proposed here for crawling for data corrections can be assessed. How far is snapshot comparison a realistic approach to obtaining update information from real scale data sets? Can all changes to data sets be seen as data corrections (that is, as changes that improve the data quality in some way)? If not, how can we distinguish changes which do improve data quality from those that don't? What information must be stored alongside each data correction, to enable it to be reused in a wide range of contexts, and also to prevent corrections from continually being suggested in contexts in which they are not suitable? What information must be stored to allow the data owner to make good decisions about whether to apply a correction to a data set (or record) or not? Can we nd places where the corrections might be applied by following the links in the data, or are more general deep Web searches required? In our current and future work, we are exploring these and other questions relating to the concept of IQ-bots.