This paper presents specific aspects of the Repolytics platform: a data analytics platform for digital repositories. An overview of the platform is presented alongside an example of how one can employ it's services to analyze a digital repository's data and identify quality related issues. Data is captured, analyzed and used to drive all aspects of our lives in a data driven world. Recently, we have seen a rapid growth of digital repositories and open data catalogues being made available to the public. In the case of digital repositories that target the libraries-archives, scientific-research data domains and open data portals the market is dominated by open source solutions such as DSpace, Omeka, DKAN, CKAN due to their simplicity and low-no cost. Most solutions enable ingest / cataloguing of information either through automated means (SWORD, REST APIs) or through simple and untuitive forms. Quality assurance in most cases comprises of a number of mandatory metadata the user has to enter. The reality however is far more complex and this has a profound effect in the quality of the the ingested data. Repolytics [1] is a platform that aims at filling this gap through intelligent data analytics. The data loaded into the platform are analyzed and specific quality metrics are presented alongside a more thorough analysis per metadata element. The metrics include metadata completeness, accuracy and consistency. Furthermore, similar metrics are calculated for the data as well.
Metadata quality is an important issue for the Digital Library domain and has
attracted several researcher groups to deal with it. One of the main demands is the
establishment of a conceptual framework consisted of a set of well-defined quality
assessment criteria such as completeness, validity, consistency, timeliness, appropriateness
and accuracy constituents [
An important evolution of this scientific field was the introduction of the
contextdependent metadata quality approach [
The core principle of the Repolytics platform is the provisioning of an expandable set of middleware services that operates both on metadata and data level of a digital repository. The most fundamental services of the platform involve: (meta)-data integration services (meta)-data profiling services (meta)-data quality services The Repolytics platform enables the user to load data from different data sources either directly (e.g. through a file archive) or by using one of the supported data providers such as DKAN API and OAI-PMH protocol. Each data source provides one or more metadata (e.g. OAI-DC, MODS) and data (e.g. CSV, excel) representations. Once the data has been loaded, each digital object is analyzed and the main workflow seen in Figure 1 below is executed.
For each digital object, if metadata is provided, the format is identified, every metadata element is indexed and a set of core analyzers are invoked for each element. For specific elements depending on their type, a set of custom analyzers is invoked. If the data is provided (usually throuh some kind of URL) the actual data is fetched and verified followed by a profile of each bitstream. Again, a set of custom analyzers is then invoked depending on the file type.
Some of the metrics that are calculated include: metadata completeness distinct values for each metadata element accuracy for specific element types (e.g. dates, actors) where all values are classified according to their class itemset frequencies when applicable (e.g. in the case of subject terms and keywords) One of the primary challenges include the efficeint visualization of the results to the end user. For that reason a series of bar charts, radar charts, tables, gauge meters etc are employed per case. For example, as shown in Figure 2 a radar chart is employed to fingerprint an entire repository according to it's completeness.
Similary, an accuracy detector identifies and classifies all metadata element values according to their class and provides an insight on the accuracy level of each class (low, medium, high).
In this paper, Repolytics, a platform for repository data analytics is presented. The platform enables the use to easily load data from supported data sources and analyzes this data focusing primarily on completeness and accuracy whereas in specific cases, more specific metrics (such as itemset frequencies) are employed to help gain insight on highly subjective metrics such as consistency. The platform can also access and profile data as well.