Addressing the long term preservation issues associated with scientific data is a complex challenge compounded by: the scale and multidisciplinary nature of the problem; the wide range of formats and data types involved; and the often proprietary and transitory nature of the hardware and software used to generate the data. In this paper we present the PANIC system - an integrated, extensible architecture based on preservation metadata, automatic notification services, software and format registries and semantic grid services - that we believe, offers a sustainable, dynamic approach to the long term preservation of large collections of heterogeneous scientific data.
Addressing the preservation and long -term access issues associated with digital scientific data is one of the key challenges facing research, government and scientific organizations today. High performance grid computing, space and earth observation sciences and large scientific experiments produce enormous quantities of data that require effective and efficient management. Digital data files and information objects require constant and expensive maintenance because they depend on hardware, software, data, models and standards which are upgraded or replaced every few years. Accelerating rates of data collection and content creation and the increasing complexity of digital resources means that many organizations can no longer keep pace with the preservation needs of all of the data entrusted to them.
In the field of science the problem is compounded due to heterogeneous nature of the large quantities of data being generated from experiments and observation. For example, astronomers are producing Flexible Image Transport System (FITS) files and VOTable compliant XML files [1]; geoscientists are producing Arc/Info SHP files and GeoTIFF images; bioscientists are generating huge genomic databases and associated EST (expressed sequence tags), GSS (genome survey sequence) and HTGS (high throughput genomic sequence) files; medical scientists are generating vast sets of DICOM images; weather researchers are generating HDF5 datasets.
The task of ensuring long term access to scientific data collections is so overwhelming that scientists spend much of their time managing the data through special purpose handcrafted solutions, rather than using their time effectively for scientific investigation and discovery. This has been partly due to the lack of available tools and services but largely due to the heterogeneous nature of scientific data, not only between different science domains but also within science domains. Organizations such as CODATA [2] have been active in promoting improved management of digital scientific data and the importance of preserving contextual information with the archived scientific datasets. In particular CODATA recommend the use of the OAIS model [3], a high-level conceptual model developed by a consortium of space agencies, to facilitate scientific data management. However to date there has been limited availability of practical, available preservation tools and services . This is beginning to change primarily as a result of activities in the digital library domain. For example, Cornell’s Virtual Remote Control (VRC) project [4] and OCLC’s INFORM [5] project are developing risk measurement and notification services. The Global Digital Format Registry (GDFR) [6] initiative, the UK National Archive’s PRONOM project [7] and VersionTracker [8] are developing format and software registries that can be used to determine required preservation actions. The UK Digital Curation Centre is developing a Representation Information Repository using ebXML [9]. Projects such as the Typed Object Model (TOM) [10] and IBM’s UVC Emulation project [11] are generating migration and emulation services. Many scientific communities are developing their own sets of migration services (e.g., converter programs ). Currently each of these components is being developed independently but altogether they can be leveraged to help build a complete preservation solution.
Moreover, it is generally recognized that there is no single best solution to digital preservation. Differences in the needs and practices of various scientific disciplines, make it difficult if not impossible to define a ‘one size fits all’ approach to selecting, appraising and retaining scientific data. [12] The most appropriate strategy depends on the particular requirements of the custodial organization, the producers and users of its collection and the nature of the objects in the collection. Hence within the PANIC project [13] we combine the efforts of the different domain-specific preservation initiatives by integrating the range of tools and services being developed into a single encompassing Grid framework. More specifically PANIC uses a flexible, dynamic, semiautomated approach which provides access to a range of metadata tools and risk assessment, notification, emulation and migration services through a Semantic Web/Grid services architecture.
The remainder of the paper is structured as follows. The next section describes the system objectives through a motivational example. Section 3 describes the overall system architecture. Section 4 concludes with an evaluation of the results and a discussion of problem issues and future work. 2
Russel Coight is an astronomer at the Australian Telescope National Facility (ATNF). He has a massive collection of legacy FITS (Flexible Image Transport System) files. The International Virtual Observatory maintains online registries of the latest recommended formats, format versions and software tools and services, (for authoring, rendering, viewing, and converting files) for the astronomy community. The PANIC system periodically compares the metadata associated with files and information objects ni the ATNF collection with the IVO’s registries. PANIC determines that IVO now recommends that FITS format files should be replaced with VOTable 1.1 format because the latest version of the Xanadu editing and analysis software will no longer support FITS f iles. The system sends an email to Russel Coight notifying him that certain datasets that he owns are in danger of becoming obsolete. The email message includes a list of the endangered FITS files. The message also recommends that the FITS format be replaced by VOTable 1.1 which has now become the defacto astronomical data standard, recommended by the IVO. Russel Coight must now find a FITS-to- VOTable conversion service that meets all his service quality parameters. He uses the PANIC system to specify the parameters he requires in the conversion service. For example, he specifies that he requires a free service that converts from FITS to VOTable 1.1. He prefers a distributed converter that will process multiple files concurrently using parallel processors on the Grid. He also specifies that the distributed converter must be highly reliable, high-speed and not result in any loss in data quality. His request is handled by a Discovery Agent which searches a Grid Service registry for the appropriate service description. The Discovery Agent cannot find any exact matches to Russel’s request but it can find one near match and two conversion services which can be chained to approximate the required service: 1. A service which converts from FITS to VOTable 1.1 but which is lossy - developed by NASA’a HEASARC 2.
One service which converts from FITS to VOTable 1.0 and another which converts from VOTable 1.0 to VOTable 1.1– both are lossless, reliable and high speed and can be distributed across GRID processors.
The Discovery Agent pres ents these alternatives to
Russel, ranked according to how well they match his
request. Russel is able to choose his preferred option
and the selected Provider Agent then executes this
service, (ideally processing multiple files in parallel
across Grid procesors e.g., [
The next two sections of this paper describe the architecture, components and implementation details of the PANIC system that we have developed in order to turn the s cenario outlined above into reality.
Notification Service Registry(s) Requester Agent
Discovery
Agent
Preservation
Service
Registry Retrieve and Invoke Appropriate Service(s)
Preservation
Service Provider Agent Preservation Web Services
TOM XENA UVC
There is strong agreement that preservation
metadata is crucial to the long term preservation of
digital objects. CODATA [12] recommend that
scientific organizations adopt the OAIS model [3].
METS [
Although METS is the more widely used
preservation metadata schema, MPEG-21 [
Based on these metadata schemas, we developed
both a stand-alone Java application and a JSP (Java
Server Pages) version of the PREservation Metadata
INput Tool (PREMINT) metadata input tools. The
application consists of a set of metadata input forms,
constrained by the underlying XML Schema.
PREMINT collects metadata by dynamically
presenting the user with a series of forms that collect:
Descriptive Metadata, Technical Metadata,
Instrumentation Metadata. In order to streamline the
preservation metadata capture process we plan to
integrate services such as JHOVE, the
JSTOR/Harvard Object Validation Environment [
The Obsolescence Detection module periodically compares the preservation (formatting) metadata for each object in the collection with information stored in the following three registries: •
Software Version Registry – this contains information about the latest versions of authoring, rendering, viewing, editing or analysis software required to access and use objects in the collections.
For each softwa re tool, the registry sto res: Title, Description, Creator, CurrentVersion, ReleaseDate, DeveloperPage, License, Requirements, DownloadSite, DownloadSize, Rating, EaseOfUse, FormatsSupported, Features, Stability, Price.
Format Registry - this contains detailed
information about digital formats including:
Identifier, Description, Version, Author, Owner,
RelationshipsOtherFormats,
ApplicationsUsingThisFormat;
FormatSpecification; ProvenanceEvents etc [
For the purposes of demonstrating the PANIC prototype, we have developed three MySQL databases containing sample data, to represent the registries. However there are existing initiatives focusing on developing and maintaining the corresponding real-world registries. For example, Global Digital Format Registry (GDFR) and PRONOM are both developing digital format registries for the purposes of long term preservation. These mainly focus on digital library applications and would have to be extended to include information relevant t o scientific data formats . VersionTracker [8] also maintains a website with a human searchable registry of software versions that enables users to determine whether they should update, upgrade or patch their existing applications. Again, this mainly focuses on commercial software. We envisage that each specific scientific community (e.g., astronomers) would have an authoritative body (e.g., IVO) responsible for maintaining registries of recommended formats and available software versions.
When there is an incompatibility between a digital object’s current preservation/formatting metadata and the latest version recommended in the registry, then a message is sent to the owner of the data or some nominated person(s) or software agent, notifying them of a potential risk. Figure 3 is an example of a notification window. 3.3.
Our aim is to build a system which dynamically incorporates the expanding range of preservation services available and also provides decision-support tools or recommender services which can assist the scientific data manager to select the best single service or combination of services for a particular digital object or a particular set of circumstances.
The modular, distributed nature of the Semantic Grid/Web services architecture makes it perfectly suited to the dynamic, large-scale, heterogeneous nature of the digital preservation problem. A key objective of the PANIC project is to test this hypothesis by developing and evaluating a semiautomated preservation system based on the Semantic Web services architecture which provides access to a suite of independent preservation service components which can be discovered, linked, and invoked in arbitrary combinations across the Grid to fulfil the specific preservation tasks and requirements of different scientific organizations. 3.3.1
Web services are enabling networked computer
programs to process and consume information. Based
on open standards such as XML, SOAP and WSDL,
Web services provide a standardized way of enabling
Web-based application-to-application
interoperability. More recently the Semantic Web
services initiative has developed OWL-S/DAML-S,
an OWL ontology which enables Web services to be
described semantically and their descriptions to be
processed and understood by software agents. A
number of projects are using OWL-S to describe their
domain-specific services and enable software agents
to automatically discover, compose, invoke and
monitor the most appropriate Web services [
OWL-S
The purpose of OWL-S is to provide computerinterpretable descriptions of services so that they can be located, selected, employed, composed and monitored automatically over the Internet. Multiple web services can be matched and chained interoperating to perform complex tasks and transactions for users dynamically and on-demand.
The OWL-S ontology has a top-level Service ontology with three main subontologies : • ServiceProfile – provides a description of what the service does, enabling advertising and discovery; • ServiceModel – provides a detailed description of a service’s opera tion or how it works; • ServiceGrounding – provides details of how to interoperate with or access a service using messages.
The advantage of OWL-S is that it provides
generic upper-level classes that can be refined to
describe any Web service. We have extended the
OWL-S classes to create more preservation-specific
subclasses. Figure 4 illustrates how we have extended
the generic Service class by defining a
PreservationService subclass. PreservationService
has two subclasses – emulation and migration. These
new types of service are defined in the
PreservationService ontology, which extends the
Service ontology provided by the OWL-Service
Coalition [
Within the ServiceProfile ontology, the Profile class provides three types of information: • Service name, description and contact (person or organization); • •
Functional description in terms of inputs, outputs, pre-conditions and effects; An extensible set of properties used to describe features of the service e.g., service category, quality rating, etc.
We have also extended the ServiceProfile ontology to create a PreservationServiceProfile.
Semantic Matchmaker [
ExecutionStatus SystemRequirment
Creator ReleaseDate ServiceQuality
Speed Reliability Emulation
Download e.g. Windows XP e.g. John Doe e.g. 8-12-2003 e.g. Low e.g High subClassOf Normalisation
OriginalObjectFormat OriginalObjectVersion e.g. 5.12 TargetObjectFormat TargetObjectVersion e.g. TIFF e.g. JPEG 2000 e.g. 2.02 Lossiness e.g. lossless
EmulatedObject EmulationType SystemSetting e.g. MAC OS e.g. OS e.g. 256 bit palette and compared against the stored ServiceProfiles for available services. A ranked list of matching services is retrieved and displayed. Matching services can be either atomic o r chained, composite services.
For example, a user might specify that he requires
a service that converts from FITS to VOTable 1.1. He
also prefers a distributed converter that must be
highly reliable, high-speed and not result in any loss
in data quality. His request is forwarded by the
Requester Agent to a Discovery Agent which
searches a Web Service registry for a service
description matching this specification. In the past
UDDI registries [
Given the results of the search and the recommendations of the Discovery Agent, the collections manager can (through the system configuration interface) choose to allow the system to automatically invoke the best matching service or interactively select a particular preservation action and invoke it manually. The system configuration interface enables the collections manager to set certain runtime parameters prior to service execution e.g., whether to process multiple files in parallel, where to save the output files, whether to update preservation metadata, where to email the logfile etc.
After service selection, the Requester Agent sends
the inputs (e.g., FITS files) to the Provider Agent
which schedules and executes the conversion services
(ideally across multiple parallel procesors using a
Grid DataFarm such as [
In this paper we have briefly described how PANIC, a prototype preservation system which we have developed based on : preservation metadata; software and fo rmat registries ; and Semantic Grid Services , can streamline the long term preservation of scientific data.
The distributed nature of the pro posed Web/Grid services architecture offers many advantages. It leverages existing work on preservation metadata and preservation software tools (e.g., emulation and migration services) by integrating them and making them available through a single interfa ce. It enables institutions to coordinate and share their digital preservation activities whilst retaining the flexibility to meet local requirements. The proposed system is scalable and extensible. It has the potential to provide preservation services for a very wide variety of data and media formats. Because the system is based on standards including: METS, MPEG-21, XML, SOAP, WSDL, UDDI, OWL, OWL-S, interoperability between services and information is optimized. The design offers maximum flexibility as an organization’s preservation needs change, the system adapts accordingly. As new preservation services, tools, standards and recommendations evolve, they can automatically be incorporated into the system by adding their semantic descriptions to the relevant registries. As well as providing unified access to the wide range of preservation services available, the system also provides decision-support and recommender services to assist the scientific data collections manager to select the best single servic e or combination of services for a particular set of objects . The user interface allows easy customization of the system and human intervention where required – offering the best combination of human and software agents.
To conclude, we believe that the semantic web services approach, as employed within the PANIC system, provides the optimum architecture for viable long term preservation of large scale collections of scientific data. By enabling the automatic detection of potentially obsolescent data objects and the dynamic discovery and execution of the most appropriate preservation service – the system can potentially save research, government and scientific organizations vast amounts of time and effort, as well as prevent the loss of valuable data.
The work reported in this paper has been funded in part by the Co -operative Research Centre for Enterprise Distributed Systems Technology (DSTC) through the Australian Federal Government's CRC Programme (Department of Education, Science, and Training). Global Digital Format Registry (GDFR) http://hul.harvard.edu/formatregistry/ PRONOM –The file format registry http://www.nationalarchives.gov.uk/pronom/ VersionTracker http://www.versiontracker.com/ D. Giaretta, “Draft DCC Approach to Digital Curation”, Jan 2005. http://dev.dcc.rl.ac.uk/twiki/bin/view/Main/DCCApp roachToCuration [10] The Typed Object Model (TOM)
http://tom.library.upenn.edu/ [11] J.R. van der Hoeven, “Permanent Access Technology for the virtual heritage”, May 2004 http://jeffrey.famvdhoeven.nl/Researchtask%20IBM %20TU%20Delft%20%20J.R.%20van%20der%20Hoeven.pdf [13] Preservation webservice Architecture for Newmedia & Interactive Collections (PANIC), http://www.metadata.net/panic/