Many institutions grant access to their metadata repositories via the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). However, this protocol has two signi cant drawbacks: it does not make its resources accessible via dereferencable URIs, and it provides only restricted means of selective access to metadata. The OAI2LOD Server handles these shortcomings by republishing metadata originating from an OAI-PMH endpoint according to the principles of Linked Data. As the ongoing OAI-ORE speci cation process shows, these principles are gaining growing importance also in the digital libraries domain.
The Open Archives Initiative Protocol for Metadata
Harvesting (OAI-PMH) [
The design of OAI-PMH is based on the Web
Architecture [
With the OAI2LOD Server we provide a possible
solution for these shortcomings by following the Linked Data
design principles [
Client applications can use the OAI-PMH protocol to harvest metadata from Data Providers using open standards such as URI, HTTP, and XML. Institutions taking the role of data providers can easily expose their metadata via OAIPMH by implementing light-weight wrapper components on top of their existing metadata repositories. 2.1
The main conceptual entities in the OAI-PMH speci
cation are Item, Record, and MetadataFormat. An item
represents a digital or non-digital resource and is uniquely
identi ed by a URI. It can be described by an arbitrary number
of metadata records, each of which is bound to a certain
metadata format, which can freely be chosen by the data
provider. To guarantee a basic level of interoperability, all
data providers must support the unquali ed Dublin Core [
OAI-PMH is implemented on top of HTTP and de nes a set of verbs to request di erent information types: an Identify request retrieves administrative metadata (e.g., name, owner) about a repository as a whole. GetRecord is used to fetch an individual record for a certain item in a given format, whereas the request ListRecords harvests all metadata for all available items in a certain metadata format. ListIdentifiers returns the identi ers (URIs) of all available items, ListMetadataFormats the formats in which the data provider exposes metadata, and ListSets returns the available sets in an OAI-PMH repository.
Figure 1 shows a sample GetRecord request for a Dublin Core metadata record available in the Library of Congress and the corresponding response. The request URI contains the address of the repository, the verbs, and required parameters like the item URI. The response consists of a <header> section, which contains the item's URI, and a <metadata> section encapsulating the metadata record. 2.2
There exist a number of OAI Data Provider Registries12, from which we know that currently 1765 institutions worldwide maintain OAI-PMH repositories. Regarding their application domain, we can observe that the protocol has been implemented in a variety of institutions, ranging from small research facilities to national libraries that have integrated this protocol with their catalogue systems. Examples are the Institute of Biology of the Southern Seas, exposing 403 records, and the U.S. National Library of Medicine's digital archive, exposing 1,272,585 records.
In order to estimate the amount and the characteristics of metadata one can retrieve via OAI-PMH, we have carried out an analysis on the 915 registered repositories that delivered valid responses. Figure 2 illustrates the size of these repositories using a logarithmic scale on the Y-axis. 1http://www.openarchives.org/Register/BrowseSites 2http://gita.grainger.uiuc.edu/registry/ 3Further information about these standards: http://www. loc.gov/standards and http://rfc.net/rfc1807.html 4http://www.theeuropeanlibrary.org Frequency not consider them in our analysis.
Another reason why we expect the number of OAI-PMH endpoints to grow is that popular open source digital library systems, such as Fedora5, DSpace6, and EPrints7, provide an OAI-PMH endpoint by deTfoapul1t.0 TMheetsaedsaytastSemtasndcuarrrdesntly nd a widespread adoption in various small and medium instUintquutaiolifniesd (Deu.bgl.in, Cuonreiversities or museums) and will fost9e0r0 the global distrRiFbCu1t8io07n of open and Web accessible metadata even more. 110
OAI MARC
Shortcomings1o08f OAI-PMH 2.3
The OAIM-PAMRCH21pSrloimtoco9l4has been designed for transferring large amounts ofMmETeStadata from a server to a client over 69 the Web. From that perspective, it provides a reasonable solution for clienEtTsDtMhSat 5n2eed to aggregate or index metadata. However, it has two signi cant drawbacks:
UK ETD DC 45 Non-dereferencable identities : although OAI-PMH is
MPEG-21 DIDL 90b0uilt on the Web 4in1frastructure, we believe that it does 11n0ot yet make u?se39of its full potential. To retrieve in10f8ormation from a repository, a client must execute an HTTP GET reque0st on an OA3I0-0PMH spec6i00c URI (see900 96F49igure 1). This prevents Web clients that are unaware 5o2f the protocol speci cs from accessing the repository. 4R5estricted selective access to metadata : the record se4l1ection criteria in the OAI-PMH harvesting process are 3r9estricted to item identi ers, metadata formats, sets, 3a1nd record creation date intervals. However, some clients might only be interested in records matching certain criteria (e.g., \all records describing items created by X") or even just a subset of the available metadata values (e.g., \all authors of all books in a library").
One could argue that these features are out of the scope of OAI-PMH and already implemented by other digital library protocols such as Z.39.598 or SRU9. However, because of the popularity and widespread adoption of OAI-PMH in contrast to other protocols, we believe that it should be enhanced in order to solve the above mentioned drawbacks. 5http://www.fedora.info 6http://www.dspace.org 7http://www.eprints.org 8http://www.loc.gov/z3950/agency/Z39-50-2003.pdf 9http://www.loc.gov/standards/sru/specs/ Institutions, which employ the OAI-PMH, could then provide powerful metadata access functionality by implementing just a single protocol.
At a rst glance, the OAI2LOD server is a wrapper that
exposes metadata of OAI-PMH compliant data sources as
Linked Data on the Web and provides a SPARQL query
interface to these metadata. During design time we have
noticed that it also covers large parts of the OAI-PMH features
by simply following the Linked Data rules [
The rst Linked Data rule says that things should have URIs. In the context of OAI-PMH, items and sets are such things. By de nition, items already ful l that rule because, according to the OAI-PMH speci cation, each item must be identi ed by a URI (e.g., oai:lcoa1.loc.gov:loc.gdc/ gcfr.0018_0163). This not the case for sets as they are identi ed by arbitrary strings consisting of any valid URI unreserved characters (e.g. ascfrbib). However, such strings are no valid URIs.
According to the second rule, URIs that identify resources should be resolvable HTTP URIs. In OAI-PMH it is common to use non-resolvable URNs to identify items. The OAI2LOD server bridges this gap by wrapping item URNs and set identi ers with resolvable HTTP URLs. Continuing the above example, the item's URI becomes http://example .com/resources/item/oai:lcoa1.loc.gov:loc.gdc/gcfr. 0018_0163, and the the set's identi er becomes http:// example.com/resources/set/ascfrbib.
The third Linked Data rule proposes to deliver useful information whenever a URI is dereferenced. The OAI-PMH protocol delivers useful information for harvesting clients that can parse and process OAI-PMH responses. We believe that this information might also be valuable for other human and non-human Web agents. For humans we should provide the possibility to browse, display, and search metadata using an ordinary Web browser. Other (non-human) Web agents such as Web crawlers should be able to access OAI-PMH metadata without knowing the protocol details. We ful l this requirement (i) by assuring that the responses delivered to a client contain only resolvable HTTP URIs, and (ii) by exposing data in various representations.
When delivering metadata records to the client, we must assure that each eld (e.g., creator) within a record has assigned a resolvable URI. For some formats (e.g., Dublin Core) this is the case by de nition (e.g., http://purl.org/ dc/elements/1.1/creator), for others we must publish a machine-readable representation (e.g., in RDF/S or OWL) on the Web. Further, we have de ned a machine-processable vocabulary10 de ning OAI-PMH speci c concepts such as Item and Set.
XHTML and RDF serialisation formats, i.e. RDF/XML
and N3, are the data representations the OAI2LOD Server
currently supports. While Web browsers can process the
former and display the returned information to humans, the
latter can be processed by machines. The server uses content
negotiation, as explained in [
In the context of OAI-PMH, the forth Linked Data rule recommends that metadata records should contain links to other related resources. One kind of link that should be included in a record delivered to a client is a reference to its origin, i.e., the OAI-PMH endpoint and all relevant protocol parameters required to retrieve the corresponding XML representation of an item and its records. We express this information using the OAI2LOD speci c oai2lod:origin property, which is de ned as a sub-property of rdfs:seeAlso.
Searching other OAI2LOD Server instances for equivalent
or similar metadata records, is another strategy for adding
links. If we refer to the example presented in Figure 1, it is
quite likely that other institutions also have a copy of this
book. This fact can be captured by adding an owl:sameAs
property to the metadata record. Currently we do this by
regarding metadata records originating from distinct server
instances and comparing the values of a set of manually
selected attributes according to their lexical similarity using
the Levensthein string distance [
Figure 3 shows the RDF/XML representation of our example metadata record as it is returned by the OAI2LOD Server. It contains the same metadata as the record in Figure 1 but represents them according to the Linked Data principles. We can see that by following the Linked Data rules, we have bridged the problem of non-dereferencable identities and support access to metadata repositories for a variety of Web agents. The other shortcoming is solved by SPARQL endpoint which allows selective record retrieval oai2lod_response.txt fPrroimntetdh:eWdedanteasdsatyo,reFdebirnuatrhye27O,A2I020L8O2:D13:s2e5rvPeMr. <rdf:RDF ... xmlns:oai2lod="http://www.mediaspaces.info/vocab/oai-pmh.rdf#"> Printed For: Bern <rdf:Description rdf:about="http://www.mediaspaces.info:2020/resource/item/ oai:lcoa1.loc.gov:loc.gdc/gcfr.0018_0163"> <rdf:type rdf:resource= "http://www.mediaspaces.info/vocab/oai-pmh.rdf#Item"/> <oai2lod:setSpec rdf:resource= "http://www.mediaspaces.info:2020/resource/set/ascfrbib"/> <oai2lod:origin rdf:resource= "http://memory.loc.gov/cgi-bin/ oai2_0?verb=GetRecord&identifier=oai:lcoa1.loc.gov:loc.gdc/ gcfr.0018_0163&metadataPrefix=oai_dc"/> <owl:sameAs rdf:resource= "http://example.com/resource/item/oai:example.com/itemX"/> <dc:title>Don Christopher Columbus to his friend, Don Louis de Santangel, on his arrival from his first voyage.
At the Azores, Feb. 15, 1493. </dc:title> <dc:creator>Columbus, Christopher.</dc:creator> <dc:subject>America--Discovery and exploration--Spanish-Early works to 1800. </dc:subject> <dc:identifier rdf:resource= "http://hdl.loc.gov/loc.gdc/gcfr.0018_0163"/> <dc:coverage>America</dc:coverage> </rdf:Description> </rdf:RDF> 10http://www.mediaspaces.info/vocab/oai-pmh.rdf
The OAI2LOD Server, as illustrated in Figure 4, is a
stand-alone server implemented in Java and based on the
architecture of the D2RQ Server [
HTML Browser HTTP
SPARQL Clients
Linked Data Clients OAI2LOD Server HTTP
Request Handler / Dispatcher
Triple Store OAI-PMH Harvester OAI-PMH
Data Provider
Config & XSL
URI paths are used to expose di erent types of information in di erent representations. The /resource path holds the URIs of all items and sets exposed by the server. When a client requests such a URI, the OAI2LOD Server examines the Accept property and points to the URI path that delivers information in a representation suitable for the client: the /data path provides access to all machine-readable RDF descriptions for a certain resource; the /page path returns the same information in XHTML. Further, the /directory path lists what types of resources (e.g., items, sets) are available in an XHTML representation. Analogously, the /all path delivers that information in a machine readable RDF representation. Figure 5 shows example OAI2LOD Server requests and the corresponding OAI-PMH requests that return the same information.
OAI2LOD Request
OAI-PMH Request
N/A /oai?verb=ListIdentifiers& metadataPrefix=oai_dc All available resource types
All item identifiers / (in HTML) /all (in RDF) /directory/Item (in HTML)
/all/Item (in RDF) /resource/item/oai:lcoa1.loc.g ov:loc.gdc/gcfr.0018_0163
-/data/item/oai:lcoa1.loc.gov:lo c.gdc/gcfr.0018_0163
(RDF) The metadata /page/item/oai:lcoa1.loc.gov:l identifier=oai:lcoa1.loc.gov:loc.gdc/ /oai?verb=GetRecord& dceesrrtcearciinobirindtegma oc.gdc/(gXcHfrT.0M0L1)8_0163 megtacdfra.0ta0P1r8e_fi0x1=6o3a&i_dc
Currently the OAI2LOD Server exposes metadata records only in a single pre-de ned format. When setting up a server instance for a speci c OAI-PMH repository, the administrator decides in which format the metadata records are harvested. Since this approach contradicts a central idea of OAI-PMH we will further investigate how the OAI2LOD Server could serve metadata in multiple formats. One potential solution is to de ne mappings between formats.
Another important OAI-PMH feature is batch retrieval of metadata records. Using the ListRecords request, a client can iteratively retrieve a chunk of records. The OAI2LOD Server currently supports these features through SPARQL and its LIMIT and OFFSET clauses. However, we believe that alternatively we could o er that feature via a dereferencable URI.
The OAI2LOD Server's capabilities of linking items with other resources on the Web are limited and still rely on human intervention. We need to experiment with further duplicate detection algorithms and similarity metrics, in order to achieve better and scalable results. 4.
OAI-ORE
The Open Archives Initiative Object Reuse and Exchange
(OAI-ORE) [
OAI-ORE is a set of standards for the description and exchange of aggregations of Web resources. A resource can be anything that is identi ed with a URI such as Web sites, online multimedia content, or items stored in institutional digital library systems. In the ORE data model an aggregation is an instance of the conceptual entity Resource Map and is identi ed by a URI. A resource map describes the encapsulated resources as a set of machine readable RDF statements, which makes them readable for a variety of Web agents. Clients can retrieve aggregations by executing an HTTP GET request on a resource map's URI. The ATOM Syndication Format12 is speci ed as the primary serialisation format for delivering resource maps to clients. However, since the ORE data model is de ned in RDF, resources can not only be mapped to the ATOM format but also serialised in other RDF exchange formats such as RDF/XML or N3.
Regarding the OAI-ORE speci cation from the perspective of Linked Data, we can observe that the rst two Linked Data rules are fundamental building blocks of the standard: all things, i.e., resource maps and the aggregated resources, are identi ed by dereferencable URIs. Further, all terms used for describing aggregations have a well-de ned semantics, published in terms of a Web accessible vocabulary de nition. It also considers the third rule because resolving the URIs returns useful |i.e., processable and interpretable| information for both human and machines. Finally, OAIORE also follows the fourth rule by providing several possibilities to link resources: rst, an aggregation of resources is by de nition a collection of linked (ore:aggregates) resources; second, the ORE model uses the owl:sameAs property to denote that two identi ers refer to the same information object; third, it supports the concepts of nested aggregations.
OAI-PMH and OAI-ORE overlap in the fact that Resource Maps can be included as metadata records in OAIPMH responses, which allows batch retrieval and harvesting of aggregation information. We believe that there lies a great potential in a tighter integration of these two standards: if OAI-PMH metadata repositories expose their items as Web resources by assigning them HTTP-dereferencable URIs, these items could take part in OAI-ORE aggregations. One possible strategy could be to de ne a common core data model that links these two standards so that the ORE speci cation builds on top of the OAI-PMH protocol. Meanwhile, the OAI2LOD Server can serve as a bridge between these two standards.
In this paper we have presented the OAI2LOD Server, a software component that republishes metadata from OAIPMH compliant repositories according to the Linked Data principles. It ful ls two major purposes: rst it exposes the conceptual OAI-PMH entities (item, set) as dereferencable Web resources, and second, it provides selective access to metadata via a SPARQL endpoint. These features make OAI-PMH metadata accessible also for Web clients not being aware of the OAI-PMH protocol speci cs.
Since the alpha version of the OAI-ORE speci cation has been released, we can observe that also in the digital libraries domain the Linked Data principles will play an important role. Also for the already established OAI-PMH protocol, it would make sense to treat its conceptual entities (items, sets) as resources that can be dereferenced via URIs. In that way, they could take part in OAI-ORE aggregations. Meanwhile, the OAI2LOD Server can be used for bridging the conceptual gap between these standards.
Our work on the OAI2LOD Server will continue: rst we will deal with the open issues mentioned in Section 3.4. Second, we will investigate techniques for linking metadata and third, we also plan to implement OAI-ORE support for aggregating items. 6.