Linked Data principles are increasingly employed to publish high-fidelity, heterogeneous statistical datasets in a distributed way. Currently, there exists no simple way for researchers, journalists and interested people to compare statistical data retrieved from different data stores on the Web. Given that the RDF Data Cube vocabulary is used to describe statistical data, its use makes it possible to discover and identify statistical data artifacts in a uniform way. In this article, the design and implementation of an application and service is presented, which utilizes federated SPARQL queries to gather statistical data from distributed data stores. The R language for statistical computing is employed to perform statistical analyses and visualizations. The Shiny application and server bridges the front-end Web user interface with R on the server-side in order to compare statistical macrodata, and stores analyses results in RDF for future research. As a result, distributed linked statistical data can be more easily explored and analysed.
Statistical data artifacts and the analyses conducted on the data are fundamental to testing scientific theories about our societies and the universe(s) we live in. As statistics are often used to add credibility to an argument or advice, they influence the decisions we make. The decisions are, however, complex beings on their own with multiple variables based on facts, cognitive processes, social demands, and maybe even factors that are unknown to us. Regardless of uncontrollable forces, in order for the society to tract and learn from its own vast knowledge about events and things, it needs to be able to gather statistical information from heterogeneous and distributed sources. This is to uncover insights, make predictions, or build smarter systems that the society needs to progress. This brings us to the core of our research challenge; how do we reliably acquire statistical data in a uniform way and conduct well-formed analyses that is accessible to different types of data consumers and users?
This article presents an approach - Statistical Linked Data Analyses - towards
this challenge with its contributions. In a nutshell, it takes advantage of Linked
Data design principles that are widely accepted as a way to publish and
consume data without central coordination on the Web. The work herein offers a
Web based user-interface for researchers, journalists, or interested people to
compare statistical data from different sources against each other without
having any knowledge of the technology underneath or the expertise to develop
themselves. The service, which we built, proceeds with running decentralized
(federated) structured queries to retrieve data from various endpoints, runs an
analysis on the data, and provides the analysis back to the user. For future
research, analysis is stored so that it can be searched for and reused.
As pointed out in Statistical Linked Dataspaces [
The RDF Data Cube vocabulary [
Linked SDMX Data [
Performing Statistical Methods on Linked Data [
Defining and Executing Assessment Tests on Linked Data for Statistical
Analysis [
Linked Open Piracy: A story about e-Science, Linked Data, and statistics [
Towards Next Generation Health Data Exploration: A Data Cube-based
Investigation into Population Statistics for Tobacco [
Publishing Statistical Data on the Web [
Google Public Data Explorer [
Generating Possible Interpretations for Statistics from Linked Open Data [
Looking at this state of the art, we can see a common pattern which is that the analysis is conducted on central repositories. As statistical Linked Data is published by different parties independently from one another, it is only reasonable to work towards a solution that can gather, integrate and analyze the data without having to resort to centralism.
The analysis platform is focused on two goals: 1) a Web user interface for
researchers to compare macrodata observations and to view plots and analysis
results, 2) caching and storage of that analysis for future research and reuse.
Here, we describe the platform at stats.270a.info [
The requirements for functionality and performance are that Linked Data design principles are employed behind the scenes to pull in the statistical data that are needed to conduct analysis, and to make the results of the analysis available using the same methods for both, humans and machines. While achieving this workflow includes many steps, the front-end interface for humans should aim for minimum interactivity that is required to accomplish this. Finally, the performance of the system should be reasonable for a Web user interface, as it needs to display a visualization and present analysis. Additionally, essential parts of the analysis should be cached and stored for future use both, for application responsiveness and data discovery.
A web application was created to provide users with a simple interface to
conduct regression analysis and display of scatter plot(s). The interface presents
three drop-down selection areas for the user: an independent variable, a
dependent variable, and a time series. Both, the independent and dependent
variables are composed of a list of datasets with observations, and time series
are composed of reference periods of those observations. Upon selecting and
submitting datasets to compare, the interface then presents a scatter plot with
the best line of best fit from a list of linear models that is tested. The points in
the scatter plot represent locations, in this case, countries, which happen to have
a measure value for both variables, as well as the reference period that was
selected by the user. Below the scatter-plot, a table of analysis results is
presented. Figure [
The datasets are compiled by gathering qb:DataSets (an RDF Data Cube class for datasets) from each statistical dataspace at 270a.info. Similarly, the reference periods are derived from calendar intervals e.g., YYYY, YYYY-MM-DD, YYYY-QQ.
Our expectation from the data is that it is modeled using the RDF Data Cube vocabulary and is well-formed. Specifically, it needs to pass some of the integrity constraints as outlined by the vocabulary specification. For our application, some of the essential checks are that: a unique data structure definition (DSD) is used for a dataset, DSD includes measure (value of each observation), Concept dimensions have code lists, Codes from code list.
In addition to well-formed adherence, to compare variables from two datasets,
there needs to be an agreement on the concepts that are being matched for in
respective observations. Here, the primary concern is about reference areas
(locations), and making sure that the comparison made for the observations
from datasetx (independent variable) and datasety (dependent variable) are
using concepts that are interlinked (using the property skos:exactMatch).
Practically, a concept e.g. Switzerland, from at least one of the dataset's code
lists should have an arc to the other. It ensures that there is a reliable degree of
confidence that the particular concept is interchangeable. Hence, the measure
corresponding to the phenomenon being observed, is about the same location in
both datasets. Concepts in the datasets were interlinked using LInk discovery
framework for MEtric Spaces (LIMES) [
The limitations of the interlinks are that reference areas (concepts) are interlinked based on their notations and labels, excluding their temporarilty and changes in their space. The shortcoming is that the maching-readable metadata is unavailable from many sources. Feature plans to accommodate this will provide a richer concept assignment system, and incorporate provenance.
One additional requirement from the datasets is that the RDF Data Cube component properties (e.g., dimensions, measures) either use sdmx-dimension:refArea, sdmx-dimension:refPeriod, sdmx-measure:obsValue directly or are rdfs:subPropertyOfs. Given decentralized mappings of the statistical datasets (published as SDMX-ML), their commonality is expected to be the use, or a reference to SDMX-RDF properties in order to achieve generalized federated queries without having complete knowledge of the structures of the datasets, but rather only the essential bits.
In order to proceed with the analysis, we use the selections made by the user: datasetx and datasety, reference period, and then gather all observations with corresponding reference areas, and measures (observation values). Only the observations whose values for the reference areas with interlinked concepts area are retained in the final result.
Shiny [
The application assembles a SPARQL query using the input values and then sends them to stats.270a.info/sparql endpoint which dispatches federated queries to two SPARQL endpoints where the datasets are located. The SPARQL query request is handled by the SPARQL client for R. The query results are retrieved and given to R for statistical data analysis. R generates a scatter plot containing the (in)dependent variables, where each point in the chart is a reference area (e.g., country) for that particular reference period selection. Regression analysis is done where correlation, p-value, and the line of best fit is determined after testing several linear models, and shown in the user interface.
During this research, establishing a correct and reasonably performing federated query was one of the most demanding steps. This was due in part by ensuring dataset integrity, finding a balance between processing and filtering applicable observations at remote endpoints and at the originating endpoint. The challenge was compromising between what should be processed remotely and sent over the wire versus handling some of that workload by the parent endpoint. Since one of the requirements was to ensure that the concepts are interlinked at either one of the endpoints (in which case, it is optional per endpoint), each endpoint had to include each observation's reference area as well as its interlinked concept. The result from both endpoints was first joined and then filtered in order to avoid false negatives. That is, either conceptx has a skos:exactMatch relationship to concepty, or vice versa, or conceptx and concepty are the same. One quick and simple way to minimize the number of results was to filter out exact matches at each endpoint which did not contain the other dataset's domain name. Hence, minimizing the number of join operations which had to be handled by the parent endpoint.
In order to put the cost of queries briefly into perspective i.e., the conducted
tests and sample sizes of the dataspaces that were used; the total number of
triples (including observations and metadata) per endpoint are: 50 thousand
(Transparency International [
The anatomy of the query is shown in Figure [
SELECT ?refAreaY ?x ?y ?identityX ?identityY WHERE {
SERVICE <http://example.org/sparql> {
SELECT DISTINCT ?identityX ?refAreaX ?refAreaXExactMatch ?measureX WHERE { ?observationX qb:dataSet <http://example.org/dataset/X> . ?observationX ?propertyRefPeriodX exampleRefPeriod:1234 . ?propertyRefAreaX rdfs:subPropertyOf* sdmx-dimension:refArea . ?observationX ?propertyRefAreaX ?refAreaX . ?propertyMeasureX rdfs:subPropertyOf* sdmx-measure:obsValue . ?observationX ?propertyMeasureX ?x . <http://example.org/dataset/X>
qb:structure/stats:identityDimension ?propertyIdentityX . ?observationX ?propertyIdentityX ?identityX .
OPTIONAL { ?refAreaX skos:exactMatch ?refAreaXExactMatch .
FILTER (REGEX(STR(?refAreaXExactMatch), "^http://example.net/")) }
} } SERVICE <http://example.net/sparql> {
SELECT DISTINCT ?identityY ?refAreaY ?refAreaYExactMatch ?measureY WHERE { ?observationY qb:dataSet <http://example.net/dataset/Y> . ?observationY ?propertyRefPeriodY exampleRefPeriod:1234 . ?propertyRefAreaY rdfs:subPropertyOf* sdmx-dimension:refArea . ?observationY ?propertyRefAreaY ?refAreaY . ?propertyMeasureY rdfs:subPropertyOf* sdmx-measure:obsValue . ?observationY ?propertyMeasureY ?y . <http://example.net/dataset/Y>
qb:structure/stats:identityDimension ?propertyIdentityY . ?observationY ?propertyIdentityY ?identityY .
OPTIONAL { ?refAreaY skos:exactMatch ?refAreaYExactMatch .
FILTER (REGEX(STR(?refAreaYExactMatch), "^http://example.org/")) }
} } FILTER (?refAreaYExactMatch = ?refAreaX || ?refAreaXExactMatch = ?refAreaY || ?refAreaY = ?refAreaX) } ORDER BY ?identityY ?identityX ?x ?y
For the time being, the use of NAMED GRAPHs in the SPARQL queries were
excluded for a good reason. For federated queries to work with the goal of
minimal knowledge about store organization, the queries had to work without
including graph names. However, by employing Vocabulary of Interlinked
Datasets (VoID) [
As statistical datasets are multi-dimensional, slicing the datasets with only reference area and reference period are insufficient. It is likely that there would be duplicate results if we leave the column order to reference area, measurex, measurey. For this reason, there is an additional expectation from the datasets indicating one other dimension to group the observations with. This grouping is also used to display faceted scatter-plots.
Recommendations from On the Formulation of Performant SPARQL Queries
[
In order to optimize application reactivity for all users, previously user selected options for analysis are cached in the Shiny server session. That is, the service is able to provide cached results which were triggered by different users.
In addition to a cache that is closest to the user, results from the federated queries as well as the R analysis, which was previously conducted, is stored back into the RDF store with a SPARQL Update. This serves multiple purposes. In the event that the Shiny server is restarted and the cache is no longer available, previously calculated results in the store can be reused, which is still more cost efficient than making new federated queries.
Another reason for storing the results back in the RDF store is to offer them over the stats.270a.info SPARQL endpoint for additional discovery and reuse of analysis for researchers. Interesting use cases from this approach emerge immediately. For instance, a researcher or journalist can investigate analysis that meets their criteria. Some examples are as follows:
analysis which is statistically significant, and has to do with Gross Domestic Product (GDP) and health subjects a list of indicator pairs with strong correlations using the line of best fit of a regressional analysis to predict or forecast possible outcomes
countries which have less mortality rate than average with high corruption
The design pattern for the analysis URIs which refer to the data and the analysis is aimed to keep the length as minimal as possible, while leaving a trace to encourage self exploration and reuse. The general URI pattern with base http://stats.270a.info/analysis/ is as follows:
As URIs for both independent and dependent variable are based on datasets, and the reference period is codified, their prefixed names are used instead in the analysis URI to keep them short and friendly:
For example, the URI http://stats.270a.info/analysis/ worldbank:SP.DYN.IMRT.IN/transparency:CPI2009/year:2009 refers to an analysis which entails the infant mortality rate from the World Bank dataset as the independent variable, 2009 corruption perceptions index from the Transparency International dataset as the dependent variable, for the reference interval year:2009. The variable values are prefixed names, which correspond to their respective datasets, i.e., worldbank:SP.DYN.IMRT.IN becomes http://worldbank.270a.info/dataset/SP.DYN.IMRT.IN, and transparency:CPI2009 becomes http://transparency.270a.info/dataset/CPI2009 when processed.
Besides the common vocabularies: RDF, RDFS, XSD, OWL, the RDF Data Cube vocabulary is used to describe multi-dimensional statistical data, and SDMX-RDF for the statistical information model. PROV-O is used for provenance coverage.
A statistical vocabulary (http://stats.270a.info/vocab)[
Future plans for this vocabulary is to reflect back on the experience, and to
consider alignment with Semanticscience Integrated Ontology (SIO) [
Some research and application areas that are planned as future work:
Making the query optimization file from Jena TDB available in RDF and at
SPARQL endpoints can help to devise better performing federated queries, or
placed in VoID along with LODStats [
With the availability of more interlinks across datasets, we can investigate analysis that is not dependent on reference areas. For instance, interlinking currencies, health matters, policies, or concepts on comparability can contribute towards various analyses.
Enriching the datasets with information on comparability can lead to
achieving more coherent results. This is particularly important given that the
European Statistics Code of Practice [
The availability of the analysis in JSON serialization, and the cached scatter plot in SVG format, makes it possible for webpage widget to use them. For instance, they can be dynamically used in articles or wiki pages with all references intact. As the Linked Data approach allows one to explore resources from one item to another, consumers of the article can follow the trace all the way back to the source. This is arguably an ideal scenario to show provenance and references for fact-checking in online or journal articles. Moreover, since the analysis is stored, and the queried data can also be exported in different formats, it can be reused to reproduce the results.
This brings us to an outlook for Linked Statistical Data Analyses. The reuse of Linked analyses artifacts as well as the approach to collect data from different sources can help us build smarter systems. It can be employed in fact-checking scenarios as well as uncovering decision-making processes, where knowledge from different sources is put to their potential use when combined.
Many thanks to colleagues whom helped one way or another during the course of this work (not implying any endorsement); in no particular order: Deborah Hardoon (Transparency International), Axel-Cyrille Ngonga Ngomo (Universität Leipzig, AKSW), Alberto Rascón (Berner Fachhochshule [BFS]), Michael Mosimann (BFS), Joe Cheng (RStudio, Inc.), Government Linked Data Working Group, Publishing Statistical Data group, Apache Jena, Andy Seaborne (Epimorphics Ltd), Richard Cyganiak (Digital Enterprise Research Institute [DERI]). And, DERI for graciously offering to host this work on their servers.