The work describes the collaboration between a large experimental research facility and emerging national and cross-national data infrastructures, with the purpose of sharing experimental data and making it findable in common multi-disciplinary data catalogues.
Many of the major centres of scientific research provide both the instruments for the research, and the infrastructure for storing and processing data. This is typical for large research facilities like synchrotrons, neutron sources, powerful lasers that grant timeslots to visitor scientists for their specific investigations and provide infrastructure for data collection and preservation. Generally, scientists work on the science and facility IT engineers work with the data; this leads to a requirement that these two groups collaborate. Another requirement for collaboration comes from the emerging e-infrastructures that transcend institutional and national borders and research disciplines.
Although research facilities make the data available, they do not provide a large range of access methods. The purpose of our work was to provide an industry standard protocol for accessing the data so that a large number of researchers can find the records about datasets produced by research facilities and access them easily.
New routes to existing data and metadata are
important as in the last decade the number of data sources
in Europe has increased enormously. It is no longer
viable for most researchers to track all of the data which
are relevant to their investigations, so data discovery
services provided by a cross-discipline infrastructure are
essential. Our work is an example of a productive
collaboration between a discipline-specific data centre –
ISIS neutron and muon facility [
Europe [
EUDAT has developed several services, namely: B2SHARE – a data publishing service; B2SAFE – a secure and reliable replication service; B2FIND – a data discovery service (data catalogue); B2STAGE – a data delivery service for the rapid delivery of large volumes of data towards highperformance computing; B2ACCESS – user authentication service used by some of the above services.
EUDAT services are deployed centrally by project
participation organizations with free registration and
access for researchers, or the services can be deployed by
interested parties in their own environment as all the
software in support of these services is open source. We
have focused on using the centrally deployed instance of
EUDAT B2FIND [
The data provider in our case is the ISIS neutron and
muon source [
The work of providing ISIS data in EUDAT involved the following steps: evaluation of the available technology; building the metadata harvester; mapping the domain-specific metadata to a more popular schema; mapping the data provided by the service end point to the requirements of B2FIND; provision of a service end point for publishing metadata; liaison with EUDAT B2FIND for testing the end point and harvesting the data records.
There were two main challenges to address during implementation. The first challenge was the mapping of the metadata: from ISIS to OAI, then from OAI to B2FIND. The second challenge was to avoid compromising the data policy set by ISIS.
The first challenge was technical and required careful programming as well as discussions with specialists knowledgeable of the metadata models for both the data provider and the data consumer.
The second challenge required access to the data
records so that the harvester could collect them. In order
to get this access, ISIS provides suitable credentials, and
it was decided to restrict harvesting to the data records
with persistent identifiers in DataCite [
We chose the Qualified Dublin Core (QDC) metadata
schema [
We then developed software that harvests the data
records from the ISIS data catalogue, maps them to the
QDC schema and passes them to the OAI-PMH server
that implements a popular standard for automatic data
harvesting [
The resultant technology stack is presented by Figure 1. The bottom layer is a domain-specific data catalogue supported by the research facility (ISIS); the top layer is a multidisciplinary data catalogue supported by a common data infrastructure (EUDAT); the middle layers are components that enable a transformation from a domain-specific implementation to a common data discovery service.
We have stored the software which was developed in this project in a public repository, so that others can
Investigation
->doi
Investigation
->title
Investigation
->summary
Instrument
->fullName
Investigation
->name
InvestigationP
arameter->name
(multiple)
“dx.doi.org/”
+
Investigation>doi
User->fullName
Name of the
organization
(as a literal)
Description of
a facility (as
a literal)
Investigation>releaseDate
en
Facility->name
Facility
->fullName
Facility->url
DatafileFormat
->name
DatafileFormat
->type
DatafileFormat
->version
DatafileFormat
->description
Facility title
(as a literal)
Web link (URL)
to ISIS Data
Management
Policy
Country code
(as a literal)
Investigation
->startDate
Investigation
->endDate
examine it for the details [
For the published information to be visible, it is necessary to register the jOAI server with a discovery service, such as B2FIND. The operation of the discovery service is the responsibility of a third party such as EUDAT.
The essential flow of work of the software is the following: Once per hour, the software connects to the ICAT and requests details of any new records to publish. A suitable record has a Digital Object Identifier and a Release Date since the last time the software was run; For each record identified, the software serializes the record as a QDC object and passes it to the jOAI publisher; Once per hour, the jOAI publisher checks for new objects and publishes them.
In this way, new records created by the data owner, are generally available within two hours, with no manual processing. No changes, other than configuration, are required to the ICAT server, the jOAI server or the discovery service. For the owner of the data, the additional processing required to provide this service is negligible. For the owner of the discovery service, the additional processing is negligible.
The services that we have developed in course of this
work support the following data discovery use case. In
order to find data, the researcher uses a Google-style free
string search in the B2FIND data catalogue [
Having received search results, the user selects one
of the candidates located by B2FIND. B2FIND presents
more information about the chosen candidate. In the case
of an ISIS record, this information includes the DOI
assigned to the dataset by the DataCite service [
Apart from its usage in EUDAT B2FIND, the
OAIPMH endpoint for ISIS ICAT and the appropriate
metadata mapping are being tested for the new Research
Data Discovery Service (RDDS) which is a national UK
initiative similar to EUDAT B2FIND but with a different
scope of research data records collected [
Figure 2 Data records and data DOIs flow
After a period of testing with a few harvesting
einfrastructures, the OAI-PMH stack has the potential to
become part of the ICAT software distribution [
We considered the effort to implement the OAI-PMH
endpoint and supply data records in e-infrastructures
worthwhile for the following reasons:
large research facilities such as ISIS have an
interest in sharing data; it may be a legal or
policy requirement that they publish this data,
especially data that is collected in a publicly
funded investigation; many investigators
consider that the provision of data enhances the
value of their research and consider that data
citation is as valuable as publication citation,
hence more routes to citable data are beneficial
for researchers;
sharing data in multi-disciplinary catalogues like
B2FIND and RDDS attracts new collaborators,
facilitates data reuse within a discipline, and
encourages cross-discipline research;
we are working within a community of European
facilities which are adopting common standards
for software and infrastructure [
This work provides foundation IT-components and from an organizational point of view, may serve as a model for sharing data collected by large research facilities in common cross-disciplinary data infrastructures. The work is a contribution to the emerging European research data ecosystem comprising traditional research centres, common national and transnational e-infrastructures, research teams located in smaller labs in universities and industry, as well as individual researchers willing to share data. The work aims to increase the efficacy and efficiency of using the public funds allocated for research and development, by providing new routes for data publishing and data reuse.
This work is supported in part by Horizon 2020 EUDAT and the UK JISC RDDS projects, although the views expressed are the views of the authors and not necessarily of the projects.