The rise of Open Data initiatives has led to the publication of many datasets from di erent organizations and governments. These datasets cover a wide range of knowledge domains, from budget to education to health care. However, not all datasets have the quality, granularity or type of information that is relevant to each user. Moreover, in many cases the description or metadata does not specify clearly the content of a dataset, di culting the exploration of datasets by stakeholders. In this paper we propose the use of dashboards and visualizations as a way to preview the content of datasets for easier exploration. The use of visualizations can provide a rapid way to select or discard datasets based on their content, reducing the potential datasets that a user may need to look in order to get what she needs.
Over one million datasets [
For example, it is not clear what speci c data is contained in a dataset, what structure is used or the scope of this dataset. As mentioned early,i n the case of the US dataset about child obesity, it is labeled as \federal", however the data describes only information about New York State; it is likely that other manually curated tags and descriptions may not be precise in terms of the content or scope of the datasets published. Thus, the question in this and many other cases is how can stakeholders know in advance what's in a dataset before downloading it? We propose the use of dashboards and visualizations to describe and preview the content of datasets; this visual representations will help stakeholder to decided whether a dataset is useful for them or not.
This paper is structured as follows: Section 2 describes related work found in the literature and state of the art technology. Section 3 discusses di erent pieces of information that can be used to create visual overviews from some of the more common le formats used to publish Open Government Data. In Section 4 we show a prototype developed as an example of what can be done to create visual overviews of datasets using the information discussed previously. Section 5 presents the future challenges on our research and we discuss our conclusions in Section 6.
The problem of good data visualizations has been studied
many years [
One of the seminal works in dataset preview was made by
Doan et al. in 1999. They studied the e ects of visual
previews of queries for NASA's EODIS datasets [
A generalization of query previews is presented in the work
of Tanin et al. [
Nowadays, the principles behind above works seems to be
suitable for open data publication, as it has been reported
to be for web searching by the work of Dork et al.[
From the perspecting of the Open Government Data,
visualizations are valuable and useful artifacts for users [
Di erent formats provide di erent support for data, metadata, annotations and other extra information that can be helpful for users to identify datasets that area valuable for them. In order to understand what format are more often used to publish Open Government Data, we looked at Data.gov and Data.gov.uk, two of the largest government data portals. We took the most popular formats reported by these portals and we found that most datasets are published in HTML, followed by XML, ZIP, CSV, PDF and JSON, as can be seen in Figure 3. It is important to note that in many cases a dataset is published in multiple formats, so these numbers are not related to the number of datasets available.
It is reasonable then to focus our e orts on the most common formats in order to cover an important number of datasets with our study. For this work, we do not considered ZIP les as part of the list of datasets to study, due to the fact that ZIP les are actually archives containing other les, such as CSV. Hence, for this study a ZIP le can be considered only as an \extra layer" of communication, and not a le format that we should study.
We identify three di erent sources of information in a dataset
that can be used to create visual overviews: data, metadata
and annotations. We understand metadata di erent from
annotations in that the former is aimed to provide
machineprocessable data about the dataset (e.g., creation date,
author of the dataset), while the latter is more focused on
explaining to a human reader certain aspects of the data (e.g.,
what does a eld mean or information about how the data
was collected). As mentioned before, di erent data formats
provide di erent levels of support for data and metadata;
thus, extracting data, metadata and annotations from
different le formats present di erent challenges.
3.2 HTML
HTML is a markup language aimed to write \scienti c
documents, although its general design and adaptations over the
years have enabled it to be used to describe a number of
other types of documents" [
Data: Representing data in HTML can be done in
multiple ways, from HTML tables to full web
applications. In the most basic case, data can be presented
as a list or a table, structured using the <ul>, <ol>
or <table> elements. The process of extracting data
from HTML documents is know as Web Scraping and
there are many tools to do so [
Metadata: HTML provides a mechanism to store
metadata, by using the <meta> element. In the case of
well-formed HTML tables, the header of these tables
contain valuable metadata as well; the <th> element on
a table will describe the name of each column,
something that will tell a user if the dataset is useful for her
purposes or not. These metadata elements can be
extracted with web scraping techniques as well and used
to give more insight about the structure of the data as
well as more information about the provenance of it.
Annotations: HTML supports comments in the code
between <!{ and {> strings sequences. These
annotations can be used to extract information about
the document and the data described in it as well.
For example, it is possible to obtain the most
relevant words in the comments and visualize them using
a word cloud. It is important to note that in the case
of HTML, many annotations might be related to the
JavaScript code used in the document; a smart
heuristic could discard potentially confusing annotations of
this type.
3.3 XML
The Extensible Markup Language [
Data: It is possible to check for common words,
numbers or phrases that occur in the content of XML tags.
One way to do so is by using XPath [
Annotations: XML allows comments in a similar way
as in HTML (See Figure 4). XML Schema [
Data: Since a CSV le is basically a table, it is possible to extract the most common terms found in the cells and display them as a bar chart or a word cloud
or other way to present it as a visual overview of the dataset. There are tools and libraries for virtually any programming language to read and extract data from CSV les.
Metadata: Due to its simplicity, little metadata can
be found in a CSV le. However, as mentioned before,
in many cases CSV les contain headers that can be
used to identify the topics described in the dataset.
Annotations: CSV does not support annotations,
however in many cases, the direct translation from a
spreadsheet, such as Microsoft Excel, carries the
title and other comments available on it (see Figure 5
as an example). These annotations break the table
structure of the CSV le and makes it di cult to read
it by programs. Still, these annotations can provide
useful information about the content of the le. An
heuristic to obtain such annotations could be the
following: Read each line of a CSV le and consider it as
an annotation, until the header is found.
3.5 PDF
The use of PDF les to publish data is a common
practice among practically all governments and organizations,
although it is widely discouraged and criticized [
Data: As mentioned before, in the best of cases PDF les contain data tables that can be extracted semiautomatically to generate visualizations, similar to the case of CSV les.
Metadata: Although PDF supports metadata and
embeddable raw data [
Annotations: Similar to the case of XML and HTML
documents, annotations in PDF can be used to identify
relevant terms that can be later used create a visual
overview.
{
}
]
persons: [
{
}
name: "John",
lastname: "Doe",
language: {
value: "English",
iso: "EN"
}
3.6 JSON
The JavaScript Object Notation JSON, is an open standard
format that has gained popularity, especially in the Web
development community, due to the simplicity for
consumption by humans and machines alike. JSON provides a
mechanism to transmit objects that can be use to communicate
di erent types of variables. Many see JSON as a simpler,
easier-to-use alternative to XML [
Similar to XML, JSON provides a tree-like structure, but supports di erent data types, arrays and other objects as well. Thus, it is possible to extract similar information as in the case of XML to later be visualized.
Data: The values in a JSON document can be used to obtain the most signi cant words or phrases that can be used later to create a visualization.
Metadata: Collecting the words used as keys can give insights on what type of data is presented in the document. Also, the tree structure could be used to identify how the data is modeled.
Annotations: JSON does not provide a way to annotate or comment documents.
As a way to test our ideas, we developed a demo tool that
creates a visual overview of a dataset. This visual overview
consist on a sample of the data and a dashboard based
on the information extracted from a dataset. Due to
simplicity, our prototype only works with CSV les, but the
principles shown are the same for the other le formats
described in Section 3. We implemented this demo using
JavaScript and the D3.js library [
The prototype presents three di erent levels of detail of the
data contained in a dataset. First, we considered useful to
give the user a sample of the data, so she can get an idea of
what it looks like as a table. To do so, we included the rst
three rows of the dataset.
Second, in our experience most CSV les describe data
properties in terms of columns (in contrast to rows); a CSV
column usually contains values related to a speci c dimension
(e.g., age, latitude, name). Thus, one reasonable approach
is to create visualizations for each column. As a way to
provide a visual representation of the values on each column,
we used word clouds [
It is important to note that as a prototype, there are many issues with this software. For example, a more sophisticated approach would consider the type of data (i.e., generic numbers, strings, geographical coordinates, time and dates) and use di erent visual strategies that are more suitable for each case. The variety of the available values may also a ect what visual strategy could be used; for example, for the columns sex and count the use of word clouds is not necessarily the best strategy.
After a user has entered the URL of a dataset, the prototype will analyze the data in order to extract the more common terms. Although our prototype processes the data live, it is possible to imagine more sophisticated mechanisms that deal with larger datasets, such as o ine or batch processing. As mentioned early, our prototype provides several visualizations for each column of the CSV le, including data sample, a wordcloud and a histogram for each column. A screenshot of our prototype can be seen in Figure 7.
Our hypothesis is that these visualizations can facilitate the process of deciding if a dataset is useful for a person or not. Thus, we propose to perform a user study to evaluate how easy or hard is for a user to nd valuable information in the presence/absence of visual overviews. Also, the e ectiveness of visual overviews may also depend on the type of visualizations that are displayed in di erent scenarios. Further research is necessary in this regard.
From this prototype, we can also take several paths. We plan to include support for other data formats, as described in Section 3. Having a web-based service available to preview and give insights about a dataset can be a valuable tool for journalists, activists and Open Government Data researchers in general. Another option is to promote the use of tools similar to our prototype to be part of government data portals by default. Most of government organizations already provide a series of tags to help people identify and understand what each dataset is about. Adding an visual overview will help them on that e ort. Finally, a smarter set of heuristics could be included in our prototype to provide more suitable visual representations, based on the type of data available in each dataset. Also, the use of annotations in datasets could be used to highlight certain visualizations over others.
In this paper we have proposed the use of visualizations to preview and give insights about datasets that can be useful and valuable to many stakeholders. We showed that for most of the more common le formats used to publish Open Government Data, it is possible to extract valuable information that can be later used to create visual overviews. We also showed how these visual overviews can be created using a prototype developed by the authors that present a dashboard of visualizations based on the information obtained from a dataset. Finally, we discussed the di erent paths this work can take in the future.