-
Data sharing, reuse and integration are increasingly important for industry, research and
government, as more and more data are being produced and collected, e.g. due to the
influence of Internet of Things technologies. The continuous integration of data from
a multitude of sources serves as foundation and enabler of significant productivity
improvements, digital business models, and novel applications, e.g. in artificial intelligence
(AI) [
6
]. As data-driven control and AI systems take increasingly autonomous decisions,
the assurance of reliability, accountability and trust in the underlying evolving data base
gains significant importance. As such, one of the emerging and fundamental problems
of the Internet is the management and preservation of evolving data on the Web and in
knowledge graphs.
In the following, we provide a concrete breakdown of the associated open challenges
as identified from extensive analysis and literature review, as well as giving a concise
overview of existing approaches addressing these challenges and references to in-depth
surveys on the respective issues, where possible. We then propose the idea of a logical
preservation and persistence layer, integrable into the classical OSI reference model to
address these challenges in a uniform way.
Challenges for Data Management and Preservation
According to Moore [
29
], digital preservation fundamentally revolves around the
principal assurance of infrastructure-independent data persistence, verifiability of authenticity
and integrity of the preserved data, as well as that of the preservation environment and
resource metadata. Based on this foundation and the analysis of relevant related work,
we further decompose data management and preservation to describe the following eight
principal corresponding challenges:
C1 Archiving – How to archive a particular version of a resource immutably?
Since resources on the Web typically change over time and frequently become entirely
unavailable [
23
], saving corresponding archive copies is essential for their
preservation. Following Moore’s principle of infrastructure independence [
29
], an archiving
mechanism should be externally transparent. As such, it should at least logically allow
for later access to independent copies [
10
] of preserved states, even if more efficient
storage representations, such as change-based or timestamp-based are used internally.
The compression and indexing of such archives (cf. e.g., [
10,31
]), as well as structured
query mechanisms for the temporal data in such archives (cf. e.g., [
2,8,37,38
], including
continuous query evaluation, as e.g. in [
39
]) are further relevant subchallenges of
archiving. A recent survey of current Web archiving initiatives can be found in [
9
], as well as a
survey of RDF Archiving approaches in [
31
].
C2 Citation – How to reference a particular resource variant persistently?
As resources may change or disappear over time, being able to persistently reference a
certain version or variant is essential to enable reliable data reuse, especially for data
not controlled by the consumer. To address this challenge, it is common practice to
employ persistent identifiers (PIDs). Hilse and Kothe [
21
] provide a good overview of
the available systems and their respective strengths and weaknesses. As postulated by
Kunze [
24
], persistence is however purely a matter of service of the resource provider,
not a property of any specific naming syntax, and lastly depends on the commitment
of any given repository to provide it. Whenever a PID exists, the identified resource
should ideally also specify, i.e., wear this PID as its data identifier. Since data on the
Web is however typically only identified using a non-persistent URL, relating resources
and PIDs is another subchallenge of citation, as discussed in [
35
]. To avoid this issue,
Gleim and Decker [
13
] recently proposed to reuse generic timestamped URLs as PIDs
in combination with the time-based HTTP Memento retrieval mechanism [
34
].
C3 Retrieval – How to retrieve a particular resource variant from an archive?
Given a PID, it should be possible to resolve and retrieve the underlying resource version
or variant using a standardized and open access mechanism to enable practical data
inspection and reuse. While a wide variety of data retrieval and access mechanisms
exist in theory, resolution of resources on the Web effectively converged towards HTTP
as a common, best practice resolution and retrieval protocol. Nevertheless, to date, no
explicit HTTP PID retrieval mechanism has emerged as a standard. Instead, a variety of
approaches, typically based on the semantic of Cool URIs [
33
], are employed. Details
on advanced HTTP resource resolution and retrieval can be found in [
24,35,40,42
].
C4 Discovery – How to discover archives, data and available variants?
Data discovery is a multi-dimensional challenge, incorporating (i) the discovery of
resources available on the web (i.e., in need of archiving) and within a given archive,
addressable e.g., using the Linked Data Platform [
36
], Web of Things Description [
27
] or
classical XML Sitemaps [
16
], as well as (ii) the discovery of digital archives preserving
certain data, (cf. e.g., [
41
]) and (iii) the discovery of different available variants and
versions of individual resources, e.g. over time, both of which are addressable e.g.
through features of the HTTP Memento protocol, as described in [
42
].
C5 Synchronization – How to monitor resource changes and synchronize state?
Especially for applications with continuous data exchange and backup, the ability to
rapidly and efficiently detect resource changes, all well as subsequent state
synchronization, is of critical importance. While depending on the type of data, efficient delta
formats (such as Diff [
4
] for RDF or JSON Patch [
7
] for JSON data) are available, a
simple and generic state synchronization mechanism for any type of data is needed. Due
to the already mentioned dominance of the HTTP transfer protocol in the Web,
corresponding synchronization mechanisms should similarly build upon HTTP if possible.
Employing content-negotiation and other HTTP facilities, such as already done by the
HTTP Memento protocol [
34
], would further enable the efficient discovery of historic
resource versions though the usage of TimeMaps.
C6 Provenance – How to track data origins and influences?
Data provenance is, simply put, metadata information about data origins, influences and
revisions. As data evolves, e.g., through modification, exchange and reuse, provenance
information enables the tracking and tracing of responsible entities for specific changes,
original authors and relevant modifications, as well as the assessment of data
believability [
32
] and data quality [
19
]. The storage of data’s provenance all the way down
to the atomic level (insertions and deletions) can be very useful to backtrack the data
transformation process and spotlight possible errors on different levels during the data
life-cycle [
30
], as well as in the data itself. A summary of requirements for provenance
on the Web is provided by Groth et al. [
17
], while the W3C PROV standard [
18
] serves
a solid basis for its practical implementation and usage. Nevertheless, data provenance
information is rarely collected to date. Therefore, a tighter integration of its collection
with core business processes [
14
] and its semi-automatic collection [
15
] have been
explored recently. Finally, a standardized connection between data and its provenance is
frequently missing, e.g. for typical binary files shared on the Web.
C7 Verifiability – How to assert data authenticity and integrity?
To enable trust in the data management and preservation process, the verifiability of
authenticity and integrity of the preserved data, as well as its provenance and other
metadata, is another open challenge. While proposals to ensure authenticity and integrity
do exist, e.g., through HTTP content signing [
3,43
], no technical mechanism for it is
well established to date. Instead, current best practices typically rely on manual and
labor-intensive data repository audits [
1
], a process not sufficiently well integrated into
the digital infrastructure of the Web to enable automatic verification.
C8 Sustainability – How to enable long-term data preservation at scale?
Finally, all presented challenges are subject to the additional challenge of sustainability,
to facilitate long-term data preservation at scale. Notably, this incorporates subchallenges
such as achieving data redundancy through efficient data duplication, discovery,
synchronization, archiving, etc. with minimal management overhead and cost. In general terms,
the data management and preservation mechanism should maximize data persistence
while introducing the minimal amount of associated overhead and cost possible. Recent
work addressing this issue includes a large variety of approaches, ranging from
decentralized, usage-based data caching [
11
], via abstract interoperability models [
14
], to full
architecture proposals for sustainable publishing and querying of distributed Linked
Data archives on the Web [
42
]. Relating back to Moore’s theory of digital
preservation [
29
], the principal assurance of infrastructure-independence finally mandates the
establishment of an open ecosystem, capable of adapting to change, e.g., of the data
models, metadata structure and employed protocols.
Even though we focus on these eight principal challenges for data management
and preservation, there are a number of additional related issues worth mentioning,
pertinent to the usage of evolving data: Appraisal (How to assess the quality of a dataset?
Cf. [
44,45
]), Curation (How to integrate data, repair imperfections and ensure data
quality? Cf. [
12
]), Reasoning and Prediction (How to extract and predict knowledge
from evolving knowledge graphs? Cf. [
26
]), and Visualization (How to visualize data
and knowledge evolution? Cf. [
5
]). In the following, we propose to uniformly address
these challenges using an interoperability layer for data preservation.
3
The Need for Interoperable Data Preservation
While a wide range of data management and preservation systems have been proposed
in the past, their practical adoption and impact have been limited. It is our understanding,
that even though viable solutions for several of the mentioned challenges are readily
available in principle, such as PID systems or the PROV standard [
18
] for data provenance,
the associated effort and overhead of adoption currently render active implementation
of infrastructure for data management and preservation infeasible for all but the few
stakeholders, for which it is core to their business or mission. However, we believe that
the success of version control and metadata management systems such as GIT [
25
] in the
software development domain have shown, that the availability of easy to use and readily
available tooling and standards leads to the adoption of corresponding systems, even
by casual users. Notably, GIT serves as an underlying, interoperable data management
and persistence layer, independent of the tooling employed for software development.
It is our firm conviction, that a similar interoperable and easily adoptable data
management and interoperability layer is needed for data on the Web, to simplify and drive the
adoption of corresponding solutions.
Inspired by an early layered model for interoperability on the Web by Melnik and
Decker [
28
], the layered architecture of Linked Data Applications by Heitmann et
al. [
20
] and the FactDAG interoperability model [
14
], we propose the introduction of
the Preservation and Persistence Layer, as illustrated in Figure 1. We position it in
between Transport Layer and Application Layer of the TCP/IP stack, respectively in
between Transport Layer and Session Layer of the classical OSI reference model [
22
],
which characterizes and standardizes the communication functions of computing systems
using abstract layers. As such, this new layer would enable uniform data persistence
and preservation support for data on the Web, easily incorporable into the existing Web
infrastructure and providing promising opportunities for the automation of many aspects
of the preservation challenges, such as the archiving of novel resource variants, the
discovery and retrieval of persistently identified archived data revisions, the collection
of basic provenance information and the verification of a resource’s authenticity and
integrity, lastly also contributing to Web sustainability, by establishing a more uniform
data management and preservation ecosystem.
4
Conclusion and Outlook
In this work, we have analyzed open challenges for the management and preservation
of evolving data on the Web, identifying eight fundamental challenges. We have given
an overview of important corresponding research addressing each of these challenges,
as well as potential solution candidates. Based on these findings, we have identified
the need for interoperability and deeper integration of corresponding approaches and
proposed the development of a uniform Preservation and Persistence Layer, providing
data management and preservation facilities for all types of data on the Web.
Future work should devise and evaluate means for the practical implementation and
adoption of such a layer, possibly based on existing technologies such as the HTTP
Memento protocol, the PROV standard for data provenance or HTTP content signing
mechanisms. Implementing such a layer has the potential to uniformly and interoperably
address all principal challenges of data management and preservation, furthering open
data sharing, reuse and integration for applications in industry, research and government.