In this demonstration we present LinkedPipes DCAT-AP Viewer (LP-DAV), a data catalog built to support DCAT-AP, the European standard for representation of metadata in data portals, and an application profile of the DCAT W3C Recommendation. We present its architecture and data loading process and on the example of the Czech National Open Data portal we show its main advantages compared to other data catalog solutions such as CKAN. These include the support for Named Authority Lists in EU Vocabularies (EU NALs), controlled vocabularies mandatory in DCAT-AP, and the support for bulk loading of DCAT-AP RDF dumps using LinkedPipes ETL.
Introduction ? This work was supported in part by the Czech Science Foundation (GAČR), grant number 16-09713S and in part by the project SVV 260451. 1 https://joinup.ec.europa.eu/release/dcat-ap-v11 2 https://github.com/ckan/ckanext-dcat and export of CKAN metadata to its DCAT representation. Nevertheless, the extensions still only provide a syntactical mapping of the models, which does not solve the underlying issues which demand a different design approach to the whole software stack, especially regarding the usage of the multilingual Named Authority Lists in EU Vocabularies (EU NALs)3.
Our primary goal when implementing the Czech National Open Data Catalog was to be compliant with standards, i.e. DCAT-AP. We had approximately 120 000 datasets harvested from local catalogs of institutions publishing open data in the Czech Republic, represented using DCAT-AP in an RDF dump file. When trying to load this data into CKAN or DKAN, we faced the following challenges: 1. Poor bulk load performance. The local institutions kept updating the datasets daily in an automated fashion, resulting in the need to reload most of the datasets daily, which we were unable to do using the CKAN API and the CKAN bulk load utilities. This is because CKAN is focused mainly on manual data entry. 2. Insufficient support for license information. The DCAT Recommendation attaches licensing information to distributions of datasets using its URL. CKAN has the license support hardwired for datasets (packages) and provides its own list of licenses identified only by their proprietary codes such as cc-by. 3. Poor support of datasets with many distributions. According to the DCAT-AP Implementation Guideline on Dataset series4, when users are expected to be interested in the dataset series as a whole, the individual files in the series should be represented as distributions of one dataset. In our case, we had a dataset with approx. 7 000 distributions. Since in CKAN, they are represented as resources and there is no paging implemented for them, CKAN kept crashing on these datasets. A hotfix for this had to be implemented e.g. in the CKAN based European Data Portal5, which simply limited the maximum number of resources, which is not an optimal solution. 4. Insufficient support for controlled vocabularies. The DCAT-AP standard mandates the usage of, often multilingual, EU NALs. Since in the RDF dump of DCAT there are links to those vocabularies, the catalog needs to be aware of them to display human readable labels for the vocabulary items, in the appropriate language.
These issues led us to believe that there is a need for a new data catalog software built with native Linked Data, DCAT-AP and controlled vocabularies support, focused on automated loading of larger numbers of datasets from data preparation pipelines. In this demo we present the LinkedPipes DCAT-AP Viewer, our solution to addressing these issues. 3 https://publications.europa.eu/en/web/eu-vocabularies/authority-tables 4 https://joinup.ec.europa.eu/release/dcat-ap-how-model-dataset-series 5 https://www.europeandataportal.eu/ LinkedPipes DCAT-AP Viewer
LinkedPipes DCAT-AP Viewer demonstrated features
The LinkedPipes DCAT-AP Viewer (LP-DAV) is a Node.js6 and Bootstrap7
based application (see Figure 1) using Apache Solr8 for search capabilities. The
dataset records themselves can be stored either in a SPARQL endpoint such
as the Openlink Virtuoso RDF store9, or for increased performance in Apache
CouchDB10 document store. It is open-source and it is developed on GitHub11.
Its main features which will be presented during the demonstration session on
real world deployment of LP-DAV as the Czech National Open Data Catalog12
are:
1. LP-DAV is built for DCAT-AP v1.1 with Linked Data in mind
2. Multilingual user interface and multilingual EU NALs support
3. Paging support for datasets with many distributions
4. LinkedPipes ETL (LP-ETL) [
Data preparation pipeline in LinkedPipes ETL
The loading of data from a DCAT-AP RDF dump file can be done using the
supplied LinkedPipes ETL [