Digital access to our cultural heritage assets was facilitated through the rapid development of the digitization process and online publishing initiatives as Europeana or the Google books project. As Galleries, Libraries, Archiving institutions and Museums (GLAM) created digital representations of their masterpieces new concerns arise regarding the longterm accessibility of digitized and digitally born content. Repository managers of institutions need to take well documented decisions with regard to which digital object representations to use for archiving or long term access to their valuable collections. The digital preservation recommender system presented within this paper aims at reducing the complexity in the process of decision making by providing support for classification and the preservation risk analysis of digital objects. Technical information which is available as linked data in open knowledge sources facilitates the construction of the DiPRec's recommender knowledge base. This paper presents the DiPRec recommender system, a community approach on how to achieve the generation of well founded and trusted recommendations through open linked data and inferred knowledge in the domain of long-term information preservation for GLAM institutions.
H.3.7 [Information Systems Applications]: Digital Libraries; M.8 [Knowledge Management]: Knowledge Reuse Digital preservation, Recommender systems Knowledge based recommender, open recommendations, linked open data, preservation planning
Knowledge based recommender systems (KBRs) as natural followers of expert systems are nowadays used for supporting the decision making process in multiple application areas as: e-commerce, financial services, tourism, etc. One of the most important challenges of KBRs is the construction of their underlying knowledge base. This is typically composed by sets of factual knowledge, i.e. information describing the application’s domain and business rules. Both together enable the drawing of conclusions and support the decisions making process when analyzing the utility of a specific item in a given context as for example, analyzing the effectiveness of digitizing and publishing Mircea Eliade’s book ”History of Religious Ideas” within Google books.
Even though the world wide web has turned out to be the largest knowledge base, information published lacks an unified well-formed representation and mainly is intended for human readers. The Linked Open Data (LOD)1 and Open Knowledge2 initiatives address these weaknesses by describing a method on how to provide structured data in a well-defined and queriable format. By linking together and inferring properties of di↵ erent independent and publically available information sources like FreeBase3, DbPedia4 and Pronom 5 within the specific context of a digital preservation scenario we shortcut the well known challenge of KBRs, the knowledge acquisition bottleneck.
In this paper we present our work carried out in the context of the Assets6 project with the aim of preparing the ground for digital preservation within Europeana7. The Europeana portal serves as a central point for the large public to easily explore and research European cultural and scientific heritage online. It aggregates and collects data on digital resources from galleries, libraries, archives and museums accross Europe and by now manages about 19 million object descriptions collected from more than 15 hundred institutions. Within this very heterogeneous context it is easily understandable that digital objects are encoded in very heterogeneous file formats and versions throughout various di↵ erent hardware and software content repository systems. Depending on the underlying use case it is likely that mul1http://linkeddata.org/ 2http://www.okfn.org/ 3http://www.freebase.com 4http://dbpedia.org/ 5http://www.nationalarchives.gov.uk/PRONOM/ 6http://www.assets4europeana.eu/ 7http://www.europeana.eu/portal/ tiple representations of the same ’physical’ object exist at a time. For example in most cases it is useful to provide access copies on demand which are easily distribuatable via the web while the master record needs to adhere to di↵ erent requirements as for example the institution’s long-term scenario and preservation policy.
A key topic in preservation planning is the file formats used for encoding the digital information. The Pronom Unique Identifiers (PUIDs) registry provides persistent, unique and unambiguous identifiers for file formats and therefore takes a fundamental role in the process of managing electronic records. Currently it lists information on about 820 di↵ erent PUIDs. While some of the formats are properly documented, open-source and well supported, others may be outdated, redeemed by software vendors and no longer functional in modern operating systems. As always the the binary file’s dependencies on the underlying platform, its configuration (codecs, plugins, etc.) as well as the rendering software are responsible on generating a concrete user performance, it is vital to have a solid understanding on all of them. This process is costly and requires a high degree of engineering expertise. Many of the GLAM institutions already outsource IT related activities and don’t have the resources to keep track of the required level of complexity in house.
The Digital Preservation Recommender (DiPRec) system addresses the topics of ’preservation watch’ and ’preservation policy recommendation’. It proposes a solution in the domain of digital long-term preservation for making documented recommendations based on risk scores, while the underlying knowledge base is built through a linked data approach. Information from FreeBase, DbPedia and Pronom in the areas of file formats, file conversions tools, hardware and software vendors is taken into account. The main contribution of this paper consists in the integration of open (general or domain specific) data when constructing knowledge based recommendations. The ”knowledge acquisition bottleneck” and the high costs of setting up and maintaining KBRs are still an impediment for extensively adoption by the industry. Recommendations provided by DiPRec are meant to support GLAM institutions across Europe in the process of analyzing their digital assets. The technical foundation and the explanation of the DiPRec recommendations are computed on top of shared and collaboratively built data sources, trust in the area of LOD and digital preservation is a key issue which has been left out for this paper due to simplicity.
The novelty of our work consists in combining expert tools (as File, Droid or Fido) and automated object identification processes, with structured information (e.g. technical information on file formats) from open data repositories. This information is use for infering new knowledge, calculate preservation risks and finally for computing recommendations on preservation actions in the domain of digital long-term preservation. We present the rationale used for the construction of the DiPRec recommender by presenting concrete examples of a given content analysis which was provided for the Assets project. The rest of the paper is organized as follows; in Section 2 we present related work carried out on recommender systems and in the field of digital preservation. Section 3 highlights the architecture of DiPRec by comparing it against the construction of classical KBRs. The functionality provided by our system is explained in detail through a concrete example on the TIFF file format. The evaluation of our approach is presented in Section 4 by analyzing the digital collections of the Assets project. This is followed in the last Section of the paper (nr. 5) by the summarization of the concluding remarks for our work. 2.
RELATED WORK
Knowledge Based Recommender systems gained broad
popularity in e-commerce and e-tourism [
A key topic in preservation planning is the process of
evaluating objectives under the limitation of well-known
constraints. A state of the art report on technical
requirements and standards as well as available tools to support
the analysis and planning of preservation actions is given in
[
Open Domain
Knowledge Pronom
UDFR Domain Expert Knowledge
User requirements 3 documents? ). This is a significant improvement over the Plato tool where all this information needs to provided by domain experts.
The Scape13 project is one of the major current initiatives
[
The paradigms of semantic Web and linked open data [
Typically the creation of classic knowledge based recommender systems consists of three main tasks. Dealing with the collection of detailed descriptions of products o↵ ers is followed by the process of constructing a recommendation knowledge base (see section 3.2). At runtime user requirements elicitation takes place and recommendations are computed based on the underlying recommendation knowledge base and the items that match the given user requirements. DiPRec follows the same process but improves the way the knowledge base is built in order to reduce the e↵ orts spent on domain knowledge acquisition. This is especially relevant for being exploited in GLAM preservation scenarios, where the underlying knowledge base contains broader information than the domain specific KBRs. Within the DiPRec recommender the Domain Information Aggregation module is responsible for collecting file format related information (e.g. formats, vendors, applications, etc.) from the open knowledge bases Pronom, DBPedia and Freebase. Furthermore the Domain Knowledge Aggregation module combines the outcome of a risk analysis process with the knowledge manually provided by domain experts. Figure 1 compares the process used by regular KBRs and the one presented by DiPRec which enhances the process of building the underlying knowledge base. In the following sections we present extended details on how the knowledge base of DiPRrec is built by using as example the Tagged Image File Format (TIFF).
The TIFF format is still very popular among the
publishing industry, as it is a very adaptable file format although
it did not have a major update since 1992. It was originally
created by Aldus and since 2009 it is now under control of
Adobe Systems. There are a number of extensions
available (e.g. TIFF/IT, TIFF-FX) which have been based on
the TIFF 6.0 specification, but not all of them are broadly
used. A standard and broadly accepted approach in the
archiving world is the migration of TIFF encoded content
to the JPEG2000 format. In [
The DiPRec system, on the one hand applies to the approach of well-documented and trackable decision making, and at the same time it uses a semi-automatic approach on domain knowledge acquisition. This reduces the human e↵ ort invested by domain experts when providing reservation recommendations, reduces the financial e↵ orts invested in the context evaluations, and in the same time is able to o↵ er good quality recommendations.
Format Name Tagged Image File Format (P), Tagged Image File Format (D), Tagged Image File Format(F)
Mime Type /media type/image/ti↵ -fx, /media type/image/ti↵ (F), image/ti↵ (P)
Software QuickView Plus, Acrobat, AutoCAD, CorelDraw, Freemaker, GoLive, Illustrator, Photoshop,
Software Homepage http://adobe.com/photoshop(D)
Standards ISO 12639:2004 (W) Vendors Aldus, Adobe Systems, Apple Computer, now Apple Inc., Microsoft (D), Adobe Systems
Dec 1982 (F) 6068 (Jan 2007), 8660 (2009)(F), 9,117 (2010)(W) 3,579,890,000 US$ (Nov 28, 2008) (F) http://adobe.com/photoshop(F)
Di↵ erently to the e-commerce domain where KBRs import detailed item descriptions from product catalogs there is no such catalog for computer file formats. The Unified Digital Format Registry (UDFR)14 project was started in 2009 by a group of Universities and GLAM institutions with the aim of building a single, shared technical registry for file formats based on a semantic web and linked data approach. The project is based on the Pronom database which provides basic information about a large number of file formats and will be extended by data on migration pathways and available software/tools. The registry should be available from the beginning of 2012. As Pronom data is not rich enough to build a recommendation and reasoning mechanism for preservation scenarios of file formats on top, we collect additional information sources and aggregate them into a single homogeneous property representation in the recommender’s knowledge base. DiPRec uses two types of operations for aggregating domain information: • data unification: the data representation retrieved from di↵ erent knowledge bases is unified and combined under the DiPRecs property model definition. For example, the number of software tools supporting a given file format is calculated over di↵ erent data sources. The individual object’s namespace, the transformation process of values, the query on how to extract a given record, etc. are preserved and are part of the property’s model representation. • property composition: more abstract properties which require a hierarchical composition are computed by aggregating basic properties by weighted numbers. The model on property definition is meant to be kept very simple. For example ”supported by major vendors” will check if at least one of the software companies is considered to fulfill this requirement by combining the properties like ”NUMBER OF EMPLOYEES”, ”VENDOR REVENUE”). See Table 2.
When aggregating domain information we are
interrogating external knowledge sources like DbPedia and Freebase
which manage huge amounts of linked open data triples.
This allows us to extract fragmental descriptions on file
formats, software applications and vendors supporting given
file formats (see Table 1). DbPedia allows to post
sophisticated queries using SPARQL query and OWL ontology
languages [
Is an object preview available? yes no yes 6.0 6.0 yes yes/no yes/no yes/no yes/no datetime URL
DIGITAL PRESERVATION
TOOLS (Droid/Pronom)
Property Sets Domain Information
Aggregation FILE METADATA
PRESERVATION
RISK Score / Report Open Knowledge
Sources (DBPedia, Freebase,
PRONOM )
Pronom as presented before is a viable resource for anyone requiring impartial and definitive information about the file formats, software products and other related data. Extremely valuable to the DiPRec recommender is the information related to the file conversion tools based on a given PUID. Therefore we employ the Droid 15 characterization service for automatically extracting technical metadata and identifying file formats from physical media files. This metadata is then used in conjunction within the domain knowledge aggregation process presented in the Fig. 2
The risk analysis module is in charge of evaluating information previously aggregated in the DiPRec knowledge base for a given record at hand over following (exemplary) dimensions of digital preservation: • Web accessibility: Dissemination copies are published and accessible on e.g. the content provider’s web portal. There should be previews of objects (e.g. thumbnails for images, video summaries, short intro for audio files) and ’rich’ object descriptions to increase their visibility and retrieval. The chosen file representations should render in the latest browsers without plugin support and cope with modern features (e.g. pseudo streaming, progressive image display, HTML5, X3D, etc). Content is made available through di↵ erent exploitation channels. • Archiving and costs: The decision of following a specific institutional preservation policy for a given technology is heavily influenced by given hardware and budget constraints. Future exploitations on the costs for content exploitation need to be predicted and taken into account.
Other scenarios may include: • Provenance metadata • Data exchange and collaborative data enrichment • Publishing and digital rights management
The definition of preservation dimensions is not orthogonal and therefore certain properties might be involved more than once when computing di↵ erent risk score. Due to management and maintenance reasons properties are also grouped by sets and a property may belong to one or more property sets. The extent to which a property belongs to a 15http://sourceforge.net/projects/droid/ property set and consequently contributes to the risk computation over a given dimension is modeled through the introduction of specific weighting factors (see Equation 1).
The value of the overall risk score for a given collection of objects is computed as a weighted sum over all digital preservation dimensions:
Ri =
X ps2 P Si wps,i ⇤
X p2 P ROPps wp,ps ⇤ d(p, P F V (p)) (1)
Where Ri represents the preservation risk computed over the dimension i. ps represents the index of the current property set within all sets associated to the dimension i. The w(ps,i) is the weight of the contribution of the property set ps to the dimension i. Similarly, p stands for the index of current properties within the list of properties available in the given property set P ROPps. wp,ps denotes the importance of a property p for the property set ps. The distance between the current property and the defined - ’preservation conform’ - value for this property is represented through d(p, P F V (p)). 3.3
DiPRec is designed to work as a multi-purpose digital preservation support tool which can be used in various scenarios by di↵ erent types of customers. For examples the tool may support content providers in analyzing the ’preservation friendliness’ of their infrastructure, their archiving solutions or the visibility of their artifacts published in the Europeana portal. Recommendations are always to be seen in the context in which the digital objects are used. Within the scope of the Assets project there is the common interest to o↵ er public access to digital assets through the Europeana portal (i.e. web discovery), to provide advance search functionality (i.e. description richness and preservation of provenance information) as well as the topic of the data archiving dimension.
As a result of the requirements elicitation process user profiles are created. A set of multiple choice questions is used to distinguish the relevant dimensions of available preservation objectives. According to di↵ erent levels of complexity, role and required domain knowledge the system o↵ ers a subset of questions which are well understood and the best available choice for a user to express his needs. Fig. 3 presents sample workflow which could be used to determine a given user profile. For example a private user (ut = private person) with a solid level of IT knowledge (itk = expert ) will be asked about preferred encodings and compression types of the digital content, while others would define attributes about storage limitations and upload samples of a given collection. 3.4
Di↵ erently to classic KBRs where the application’s scope is very well delimited in terms of selecting the best matching items in a list of known possibilities, the DiPRec system relies on expressing an institutional preservation context in form of user requirements that are combined with the knowledge acquired about the long term accessibility threatening. We employ tools to evaluate the content of a given collection from a technical point of view and to generate fine grained preservation risk scores. When records are identified to have vulnerabilities on certain preservation dimensions a rule based engine as JBoss Drools16 is used to propose appropriate preservation actions. The set of available business rules are defined by domain experts in form of simple IFTHEN-ELSE rules. These rules are neither complete nor meant to be non-overlapping. Unified tool access for processing executable preservation plans is provided through the Assets preservation normalisation framework which is able to invoke the tools with exactly defined settings and parameter configurations.
IF ( rac > 0.5 AND ia == true AND iwa == true AND open f ormat == FALSE) THEN migrate(preservation f ormat) IF (content type == IMAGE) THEN preservation format = (JPEG/2000:1, TIFF/6:0.8) IF (f ile f ormat == TIFF/5 AND preservation f ormat == JPEG/2000)
T HEN migration tool = IM AGE MAGICK (2)
The preservation recommendations are computed using
the constraint solving problems (CSP) theory [
EVALUATION
The evaluation of the first prototype of DiPRec was conducted within the scope of the Assets project. Ten partners of the project consortium provided metadata and binary content (10 collections with a total size of 516GB contained in 368067 media files) for supporting the development and testing of services developed within the scope of the project. The first step in the evaluation process was the identification of file formats, definition of property sets and the aggregation of the domain knowledge available in open knowledge bases on these file formats.
The Table 3 lists the distribution of file formats by content type. Even the experimental data was taken from a small number of content providers, we discovered a variety of 18 formats in 38 di↵ erent versions used for encoding the digital content.
The Digital Record Object Identification tool (DROID) version 5, signature file 45 was executed through the Assets preservation normalisation tool suite and was able to successfully identify file formats in 95 percent of the cases through its binary signature method except of the 3D model objects which have not yet been collected by Pronom. Appropriate information on all of the file formats was contained in DbPedia and Freebase and the domain knowledge acquisition process was completed by successfully computing the preservation risk analysis scores.
The second part of the evaluation consisted in computing recommendations for the given content. Therefore, we created a user profile for content providers that are interested in making their content accesible through Europeana. Within this context, the content providers manifest interest for the web accessibility digital preservation dimension.
The highest diversity of file formats was found in the image collections. The recommendation to migrate these files to the JPEG 2000 format didn not get a high priority and will be performed within the next period of scheduled storage migration. The Image Magick tool was the recommended choise for performing this transformation action. The whole audio content available in Assets was provided in the mp3 format and no recommendation was made for transforming audio collections. The most restrictive constraints for web accessibility are defined for the video content. The pseudostreaming protocol is an advanced technological solution used for distributing information over the web. It allows the user to interact with the media-player and to quickly navigate within the content without the needed to download the entire media file. This protocol is supported by two file formats: flash video (FLV) and MPEG4 with H2.64 video encoding. It has native support in HTML5 and is used in HTML4 with an adequate browser plugin. A part of the Assets content is already available in FLV format and another part is available in MPEG1 or MPEG2. The DiPRec resulting recommendation is to migrate the content to FLV by using the ↵ mpeg 17 tool.
CONCLUSION
Within this paper we introduced the DiPRec recommender system, an expert support tool in the domain of digital longterm preservation for GLAMs. An important contribution of this papers is the exploitation of an open linked data approach for constructing the recommender’s knowledge base built upon open registries as DbPedia and Pronom. Since the knowledge acquisition, aggregation and unification process is fully automated it is easy to upgrade the recommender’s knowledge base.
We looked at preservation planning which is the process of specifying clearly defined and relevant trees of objectives in a defined preservation dimension and evaluating them within a given (institutional) context to generate well-documented decisions. DiPRec is able to advance the process with inferred community knowledge and reduces the degree of manual evaluation processes or require technical expertise in this process.
Am important concern related to the KBRs is the trust in the provided recommendations. This is especially relevant for the digital preservation domain where we deal with a large amount of multimedia material and the execution of the preservation actions is associated with considerable costs. Within this paper we did not examine the completeness, correctness and quality degree of the underlying data. We however argue that data from open knowledge bases like DbPedia or Freebase could protect from biases introduced by the economical interests of professional companies by its underlying community approach.
The tool has been designed by reusing our past experience
in building knowledge based and case based recommender
systems [
ACKNOWLEDGMENTS
This work was partially supported by the EU project ”ASSETS - Advanced Search Services and Enhanced Technological Solutions for the European Digital Library” (CIP-ICT PSP-2009-3, Grant Agreement n. 250527).