Open data is now a heavily discussed topic around the world and in the European Union. In the Czech Republic, open data is a term anchored in legislation, which includes the requirement of registration of all open data in the Czech National Open Data Portal (NODC). In the journal paper [5] we describe the NODC, its architecture, dataset registration processes including the harvesting of Local Open Data Catalogs (LODCs), proprietary XML API and its obsolete dataset viewer. Next we describe the process of transformation of the NODC metadata to the DCAT-AP v1.1 RDF representation from the data model point of view and from the technical environment point of view. We describe the dataset quality measurements computed using the new data representation and its further impact on the Linked Open Data (LOD) environment including the harvesting of the metadata by the European Data Portal (EDP). Finally, we evaluate the data transformation and publishing environment from the usability, portability, availability and performance perspectives.
Open data is currently a hot topic among institutions of public administration
and data users around the world. From the political point of view, publishing
open data is important for public administration institutions to show that they
are transparent and open to citizens. From the legal point of view it is
important that the data is published using an open license, permitting users to use
the data freely. From the technical point of view, it is important that the data
is published as machine readable data in an open format, accessible on the web
with minimal effort. From the point of view of potential open data users it is
important that the data can be searched for and found. Finally, from the economic
? This work was supported by the Czech Science Foundation (GAČR), grant number
19-01641S.
point of view, open data is expected to support creation of new services and new
business models1. At the intersection of all of these points of view lay open data
portals of the data publishing institutions, which typically include open data
catalogs where datasets can be found along with the metadata describing them.
The metadata descriptions contain the necessary information about the licenses
of datasets, formats of their distributions, and the textual descriptions, all of
which can be used for dataset search. This results in a number of open data
catalogs, typically one per each institution willing to publish open data. Therefore,
a problem with discoverability of the data catalogs and the individual datasets
described in them arises, and along with it a need for aggregate views over
multiple data catalogs. To address this need, a standard for representation of dataset
metadata, the Data Catalog Vocabulary (DCAT) [
In the Czech Republic, the Ministry of the Interior (MoI) is in charge of the open data agenda. The NODC is run by the MoI, and the public administration institutions can either register their datasets directly in the NODC, or, preferably, they can run their own Local Open Data Catalog (LODC) which, after registration, gets automatically harvested to the NODC. Unfortunately, until now, the MoI did not consider DCAT-AP v1.1 as a metadata publishing format. The NODC internal data model was inspired by the DCAT-AP v1.0, however, it is XML based and accessible only via a proprietary XML API. In addition, the metadata registered in the NODC was only viewable to users via a rather unfriendly user interface.
As the NODC became known and used, new requirements on its functionality
arose, mainly the need to monitor dataset metadata quality, and the need to
be harvested by the EDP. To address these requirements, we first transform the
current NODC metadata to the DCAT-AP v1.1 RDF representation and publish
it. Then we use this new representation to compute the dataset metadata quality
measurements and to facilitate the harvesting of the NODC metadata by the
EDP. Similar requirements are identified also elsewhere, e.g. in Serbia [
economic-benefits-open-data 2 https://joinup.ec.europa.eu/release/dcat-ap-v11 3 https://www.europeandataportal.eu
In the journal paper [
The rest of this extended abstract is structured according to the original paper
[
In Section 2 we describe the NODC, its architecture, the processes of harvesting LODCs, and its proprietary XML API. In Section 3 we describe the data transformation to DCAT-AP v1.1 and the linking to related datasets and code lists. In Section 4 we describe the technical environment used for the data transformation and publication process. In Section 5 we describe data quality measures which we compute based on the transformed data. In Section 6 we show additional impact of the published dataset by presenting some of the known usages of the published data, which include harvesting by the European Data Portal. In Section 7 we evaluate the data transformation and publishing environment according to various criteria. Finally we survey related work and in Section 8 we conclude. 2
The institutions in the Czech public administration are required to register their published data in the Czech National Open Data Catalog (NODC) before they can call it open data. There are two ways of registering to the NODC. For smaller institutions such as village councils there is the possibility of registering individual datasets directly in the NODC using a form to fill in all the necessary metadata. For larger institutions such as ministries or city councils, there is the possibility of registering their own local Open Data catalog (LODC), as there is an assumption that such institutions will have their own data portals anyway. Once a LODC is registered, the metadata from it is automatically and regularly harvested by the NODC, giving the institutions more flexibility in their dataset management. The registered datasets can be viewed in a rather unfriendly web user interface, which does not provide many features known from wide-spread data catalog implementations such as CKAN4 and DKAN5. 3
Our goal is to publish the dataset metadata from the NODC according to the
DCAT-AP v1.1 specification. The proprietary NODC XML API will serve as our
data source. In this section of the original paper [
Data transformation and publishing environment
In this section of the paper [
The transformation is done using LinkedPipes ETL [
LinkedPipes ETL is an open-source ETL tool for production and
consumption of Linked Data, which is in use by multiple organizations in the Czech
public administration. It is also used in the OpenBudgets.eu platform [
In this section of the paper [
Since there is no validation of input data in the current NODC harvester of LODCs, there are metadata records violating DCAT-AP v1.1 constraints or constraints dictated by the Czech legislation. The first part of the quality measures in this section aims at detecting such anomalies. In addition, there are measures aiming at providing an overview of common practice. The results of these measures are published on the Czech Open Data Portal12. The measures are: 1. Number of distributions with unspecified license per publisher 2. Number of datasets with distributions with unspecified licenses per publisher 3. Number of datasets missing required attributes per publisher 4. List of datasets missing required attributes per publisher 5. Number of distributions with a given mime type per publisher 6. Distribution licenses per publisher 7. Number of publishers per license 8. Number of datasets with a given accrual periodicity per publisher 9. Number of datasets and distributions per publisher
These measures were selected based on the most frequently appearing errors in the metadata records. These erroneous records cause a decline in the overall metadata quality in the NODC. When the users encounter them, they tend to blame the NODC for the inconsistent looking record, therefore, it is important to us that the publisher correct their records. This is also why it is important 12 https://opendata.gov.cz/statistika:datova-kvalita (in Czech only) to consistently point out errors in the records and demand their correction. From our experience, the most effective way to achieve the correction in the (Czech) public administration is to publicly display the errors attributed to the originating publishers, along with clear instructions on how to correct the mistakes. The errors in the records also usually reveal a deeper problem with data management at the original publisher. 5.2
The measures listed in this section check whether the resources linked from the metadata records actually exist, and whether they are served correctly. There are four types of resources linked from the metadata, i.e. distributions, licenses, dataset documentation and distribution schema. For each of the four resource types we compute a summary statistic and a list of offending resources.
Here, we list the quality measures along with the description of the individual columns in the result. 1. Statistics of unavailable dataset distributions per publisher 2. List of unavailable distributions 3. Statistics of unavailable schemas of dataset distributions per publisher 4. List of unavailable distribution schemas 5. Statistics of unavailable licenses of dataset distributions per publisher 6. List of unavailable distribution licenses 7. Statistics of unavailable documentation of datasets per publisher 8. List of unavailable dataset documentation
In addition to the availability measures described above, there is one more measure dealing with inconsistency between the distribution Media Type registered in the NODC and the Media Type returned by the web server serving the distribution. 6
Additional impact of publishing NODC as Linked Open
Data using DCAT-AP v1.1
In this section of [
Theoretical advantages of LOD described in existing literature are not convincing
enough to the representatives of public administration institutions regarding
publishing their data as LOD. We are using the example of the NODC and others,
such as the Czech Social Security Administration [
A clear added value of the DCAT-AP v1.1 representation of the NODC data is the ability to be harvested by the European Data Portal (EDP), a well-known pan-European open data catalog, using a native LOD way. In fact, this was beneficial not only to the Czech publishers, as their metadata got published on the European level, but also to the developers of the European Data Portal. This is because the Czech NODC was the first European data portal to publish the metadata using DCAT-AP v1.1 in RDF natively, i.e. as an RDF data dump, dereferencable IRIs and a SPARQL endpoint. At the same time, the Czech NODC is the largest catalog in EDP. 6.3
The original NODC viewer was quite unfriendly to the users. However, the
proprietary nature of the published metadata made it hard to convince someone
to develop an alternative frontend. Thanks to the transformation of the data to
DCAT-AP v1.1 [
In our research group we are focusing not only on LOD publishing, but also LOD
consumption, where we identified a distinct lack of tools for actually using LOD
that is published [
In this section of [
system-usability-scale.html
prepare and publish the DCAT-AP v1.1 NODC RDF dataset. We evaluate the
environment using so called quality attributes as introduced in [
The environment integrates various open-source tools. We do not evaluate each individual tool but the environment as a whole.
In the paper [
During the time of writing the paper, the MoI changed their supplier of the implementation of the original NODC, making it permanently unavailable, as the new supplier was not able to take over the original implementation. This nevertheless showed another benefit of publishing open data, as our copy of the NODC used to demonstrate the transformation to DCAT-AP v1.1 and the browsing of the data in LinkedPipes DCAT-AP Viewer remained the only existing publicly available NODC instance. Currently, it is already running as the official NODC instance at https://data.gov.cz.
Finally, we conclude the paper with a summarization of the lessons learned during our work with the MoI and publishers of open data in the Czech Republic.