Discovering and accessing digital data, which is predominantly spatial, is a problem often faced by researchers and policy- and decision-makers in Australia. Significant efforts are underway to provide access to discovery mechanisms to large data repositories, such as the INSPIRE geoportal in the European Union and the Research Data Australia portal developed by the Australian National Data Services (ANDS) in Australia. However, whilst such portals are significant vehicles to data discovery, they typically do not provide direct access to the data assets themselves. In this paper we review the state of play of 'data hubs' where data can be searched, discovered, downloaded and in some cases analysed and visualised. Data hubs are typically web services or data download playgrounds accessible via a portal. In recent times there has been a push for government agencies to open data access to support research and development and innovation in industry. In this paper we will focus on reviewing a number of data hubs initiatives across Australia and internationally in relation to data relevant to enabling smart cities. We will provide specific attention to the Australian Urban Research Infrastructure Network (AURIN), which is developing a portal infrastructure focused on supporting urban researchers, and policy and decision-makers. The AURIN portal aims to facilitate programmatic access to data held in many emerging data hubs across Australia. AURIN is implementing a federated data approach, providing a single access point and common interface for interrogating and visualising datasets. This paper outlines the data hub concept, describing the process and benefits of data hub integration within the AURIN e-infrastructure context, and critically examines this approach and other similar approaches being undertaken by comparable initiatives around the world.
In this paper we will introduce the concept of data hubs as a mechanism to provide better access to data for
urban researchers, policy and decision-makers in Australia. With the advent of the digital city, also referred to as
the ubiquitous or smart city, there is a growing need for data to be more accessible and to better support evidenced
based decision-making
The Australian Urban Research Infrastructure Network (AURIN, http://aurin.org.au/) is a project funded by the Australian Government’s Super Science initiative, tasked with building an e-infrastructure oriented to the needs of Australia’s urban researchers. AURIN aims to assist in improving connections between researchers and data custodians across Australia, and to “offer open access to data arising from research infrastructure provided through the AURIN” (AURIN, 2011). As stated Data hubs are considered an integral part of the AURIN einfrastructure. In this paper we will discuss how AURIN is establishing a number of data hubs in a federated architecture to provide better access to Australia’s growing digital urban data asset, and will discuss how these hubs compare to other initiatives around the world, particularly the Infrastructure for Spatial Information in the European Community (INSPIRE) project. The Melbourne Data Hub is presented as a case study as the first data hub realised through the AURIN project, and the only hub project fully completed and integrated within the AURIN e-infrastructure. The paper concludes by discussing the next steps in implementing a series of federated urban data hubs across Australia, linked via the AURIN e-infrastructure, and discusses some of the challenges and opportunities in doing so. 2
Data hubs have been created to address issues arising from the discovery, access, and format diversity of research data. The Open Knowledge Foundation (OKF) defines a data hub as ‘a community-run catalogue of useful sets of data on the Internet’, and notes that in addition to data searching, the data hub ‘may also be able to store a copy of the data or host it in a database, and provide some basic visualisation tools’ (OKF 2013B). From a technical perspective, a data hub can be deployed using a ‘Hub-and-Spoke’ architecture, where data is housed in a central system, or hub, with users accessing the data from many locations, or spokes. While these definitions are useful, they also limit the potential application of data hubs. For example, a data hub can benefit the data user community (community), but the stored and operated by a private company or government institution. It is important to note that the data hub concept does not just apply to free or open data, but to data in general, including data which requires a fee for access. As such, the data hub concept can be seen to focus more on making data accessible rather than making data free.
As such, the AURIN project has broadened the definition of a data hub to represent the scope and variety of applications of the data hub concept. At the core a data hub needs to be a single access point that: 1. Allows users to search for a variety of data; 2. Allows users to access and use this data; 3. Provides data in a discrete set of formats; 4. Creates a discussion/feedback loop between custodians and users; 5. Allows users/data custodians to contribute data to the hub; and 6. Provides information about the data (metadata).
Based on the above definition
One of the main limitations of the data hub is that any downtime of the data infrastructure results in users
losing access to data, unless redundancy is factored into the system. This is an inherent risk in the consolidated
data hub approach, one shared with data warehouses and geoportals (
The primary focus for AURIN has been to access data with a spatial component and provide urban researchers
with a system to discover and analyse this data, which aligns closely with the geoportal concept.
The data hub as a concept has many similarities to the concept of geoportals, but with the increasing focus on
big data and open data distribution, data hub has been applied to many disparate fields and many types of data.
Large technology organisations such as Microsoft have invested heavily in developing enterprise level solutions
to the data hub concept, with specific application development using SQL server to develop a ‘hub and spoke
architecture’ solution to data storage and distribution
The advent of the data.gov open data movement in 2008, discussed in detail in Section 3.2, resulted in
increasing volumes of data available and distributed to the public, and cloud computing allowed a new way to
think about this storage and distribution
The data hub concept has been realised in many locations and contexts globally. Many scientific fields have
collaborated to create research specific data hubs to store, discover and distribute research data to other
researchers. Examples of these hubs can be found in the fields of health, environmental and engineering research,
for example the “ObamaCare” online heath portal manages data through a Federal Services data hub, drawing
information from distributed federal government sources such as the IRS, Social Security Administration and
Homeland Security, and is used as a distributed, secure means to verify identity
In the urban research community, the European Union (EU) has established the Urban Audit Data Hub as a
collection of comparable city statistics across the EU
Broad application data hubs are much more prevalent than research domain specific hubs, and have been set up to create a community of general knowledge sharing. They have the advantage of not being constrained by a particular type of dataset or field of interest, but can be more difficult to search, and contain less detailed information than the field specific hubs.
Initiatives such as www.datahub.io have been launched by the OKF as a free data management and distribution
system to allow any user or organization to host and distribute data. In addition OKF has released CKAN, an open
source software solution for data publishers to use to create data hubs, making their data accessible for the public.
There are also many examples where the private sector has developed technical platforms to support the
establishment of new data hubs, including the Oracle Life Sciences Data Hub
One of the largest geoportal initiatives underway anywhere is the world is the Infrastructure for Spatial
Information in the European Community (INSPIRE) directive. This directive aims to create a standard EU Spatial
Data Infrastructure, which will improve the management of spatial data and metadata, data interoperability and
data sharing between the 27 member states of the EU, across 34 identified spatial themes
The main INSPIRE portal allows users to search for datasets from across the EU from a single interface, and allows advanced search filters to be used to narrow down searches by geography, format or spatial theme. The INSPIRE portal only displays metadata for each dataset, it does not allow users to directly access any of the datasets, either manually or programmatically. However, each metadata resource contains a link to the data source, which may be a file, service or web application.
While INSPIRE represents a significant achievement in spatial data standardisation and management, there are
still significant steps required to improve usability through effective user interface design and improved data
access
In the case of data being made available through web applications, the link provided through the INSPIRE metadata is often to a web mapping interface where a user has to go through the process of discovering the dataset again from a new search window as the link often does not go to the individual dataset identified in the metadata. As a significant portion of the datasets in INSPIRE are orthophotography tiles or georeferenced cadastral plans (~40-50%), which are generally accessed through a separate portal, this process can be time consuming and often a duplication of effort.
In addition, the number of datasets listed in this portal can be misleading. For example, an orthophotography
series over a town represents one data collection. However, technically the data is too large to hold as a single file,
and may be divided in to hundreds or thousands of tiles. In INSPIRE, each of these tiles can be listed individually
as a dataset, where in reality they should be listed as one. This also applies to series of scanned cadastral plans
which represent a single collection of data, not many individual datasets. It may be more efficient from a user
perspective in these cases for the resource to be linked as a collection, not as individual file records, which has
been the approach of the
Open data, tools and software have become increasingly popular and accessible over the past decade. With this
popularity, there has been increasing pressure on government organisations to release their data openly as well.
The principle of open access aims to support public policy developments, and provide a support mechanism for
citizens to examine information generated by their governments
As discussed in Section 3, OKF has released an open source data hub platform called CKAN to encourage
organisations to distribute datasets to the public. Many local and national government agencies across the world
have taken advantage of this open data hub technology, including Berlin, United States of America, Canada,
United Kingdom, Germany, Mexico and many more, including the Australian Commonwealth Government, and
the State Government of New South Wales, Queensland and South Australia
Data hubs established using these platforms conform to each of the data hub characteristics as defined in
Section 2, though there are some caveats around the enabling of community or public participation. While many
open data platforms contain a dedicated ‘Suggest/request a dataset’ section, which allows for the community to
request specific datasets (see https://explore.data.gov/nominate), there are many examples of open data sites
where this has not been implemented. A preliminary search of CKAN instances
Another limitation of the government open data hubs is that data can only be requested as whole datasets for
download, subsets of data cannot be extracted, queried or imported in to other applications. A review of Open
Government Data identified that this data often required “substantial human workload to clean them up for
machine processing and to make them comprehensible”
In July 2010 the Australian government released a Declaration of Open Government to promote an open
government based on ‘better access to and use of government held information, and sustained by the innovative
use of technology’
Other notable data hubs which have been established within Australia include the University of Wollongong
Smart Dashboard
Like the other data hubs from within the data.gov discussion, the open data hubs detailed above only link to
whole datasets, where subsets of large datasets cannot be requested, and programmatic integration of these
datasets becomes difficult. The Smart Dashboard and SLIP hubs overcome the dataset subset problems, and
allows for the data to be programmatically accessed, machine-to-machine
An initiative by the Australia and New Zealand Land Information Council (ANZLIC) is currently looking at addressing the broader accessibility of spatial data through its Foundation Spatial Framework. This framework has the goal of “making common foundation spatial data ubiquitous across Australia and New Zealand” (ANZLIC, 2012). This initiative may address the access and usage limitations listed above, though the framework is still in development stages. 5
AURIN has been funded to establish a dedicated urban data portal in Australia, and AURIN aims to use the
data hub concept to facilitate machine-to-machine access to datasets across the country held by key national, state
and local government agencies, and private sector organisations. AURIN leverages both the data hub hosting and
distribution features to access data at the source, and consume the data within the AURIN portal
A data hub represents an opportunity for data custodians and data users to work together to determine which data is important for release to the user community. This represents a paradigm shift from the data warehousing environments where data custodians made all the decisions regarding which data was made available to users. This benefits both groups in this equation: Data custodians can minimise resources spent releasing data by focusing on what is identified as having the highest demand/need, as well as receiving feedback on the quality and usability of datasets. Data users can then receive more of the data they require, and reduce the time taken negotiating access to datasets held within organisations.
Viewed with these goals, a data hub is as much a hub of people as one of data, though a facilitator is often required at the centre to coordinate communication between parties, prioritise data needs and negotiate for access to datasets. The dynamics of relationships are captured in the communication cycles illustrated in Figure 2. AURIN acts as this facilitator within the Australian urban research environment.
End user confidence in the quality of data will increase when data is hosted as close to the creators or custodians of data as technically possible. Where possible, data custodians should be either distributing data through their own hub, or contributing data directly to a third party hub if the custodian does not have sufficient technical resources to set up a programmatic in-house data service. Understandably, not all data custodians have sufficient resources available to enable this programmatic access, and as such AURIN is working to establish data hubs across Australia, focusing on either a specific discipline or focussing on a geography such as a state.
The hub and spoke model of data hubs illustrated in Figure 3 shows a hub having multiple end users, and as such a data hub should be established to service long term needs of an end user community. While a facilitator, such as AURIN, may act as a catalyst to establish a new data hub, the resulting networks, technology and data should be set up such a way that it can continue to operate effectively without the original facilitator. This will ensure that users can have confidence in a data hub as a long-term method for accessing relevant data.
Research from Dawes et al. (2005) highlights that user understanding of data increases with increased understanding of how a dataset is created. Dawes further notes that in depth metadata allows a user to ‘determine if it (the data) would be worth working with for their purposes’. As such, any data served through a data hub must be accompanied by sufficient information to allow a user to assess the suitability of a dataset for a given purpose, including any limitations, restrictions and caveats on any given dataset.
Supporting these principals, the following criteria are used to assess a data hub for suitability for integration in to the AURIN platform (Delaney, 2012). An AURIN data hub is one which: • provides programmatic access (machine to machine access) to a number of data services. • aligns to a federated data service model and supports data interoperability so that other portals web mapping tools can build off the hub. • aligns to data and metadata standards driven approach where possible. • has a consolidated level of technical operational support at the designed hub. • encourages a collaboration and consortium approach • realises economies of scale through multiple data feeds. • leveraging existing data services infrastructure where possible; • facilitates licensing arrangements with data custodians through relevant government department or agency.
AURIN aims to connect to many hubs, along the ‘spokes’, and make the data in these hubs available for discovery, analysis and download. In this way, the AURIN portal can be considered as a hub of hubs – a single access point allowing users to access and combine information from hubs in different geographic locations and from different specialist themes, a concept illustrated in Figure 2.
AURIN is currently exploring incorporating up to 12 data hubs across Australia. The first of these hubs implemented is the Melbourne Data Hub, explored in a case study below. Other data hub projects underway include national projects with the Australian Bureau of Statistics (ABS), and the Australian Property Monitors (APM), as well as state-based hubs looking at housing in NSW, planning and transport data in WA, transportation modelling in Brisbane, and an energy and water data hub in Townsville. Figure 3 – The AURIN Data Hubs of hubs concept, highlighting geographic locations of select AURIN data hubs. 6
To assess the differences between some of the major data hubs discussed in this paper, a table has been developed identifying some of the key characteristics and metrics of these. This table allows for a non-technical comparison to be made between each geoportal and data hub, including identifying the target user group and technical maturity of each of the stated portals. These examples have been chosen to allow comparisons between international initiatives and Australian examples. This is not an exhaustive list, and other initiatives such as the Canadian Geospatial Data Infrastructure would also provide useful comparisons, but from initial investigations would have yielded similar results. As such, these examples were chosen to provide a comparison of each distinct data hub initiative as discussed in this paper.
This table identifies some clear similarities and differences between these data initiatives. Each portal recognises the significance of metadata by ensuring all data contributed aligns to a minimum standard. However, the detail of this metadata varies within each portal. Both data.gov sites contain varied levels of metadata, from a full ISO compliant implementation to simple information with just a dataset title and abstract. INSPIRE and ANDS both have extensive metadata compliance required before data is published, with INSPIRE using the ISO19115 standard and ANDS recommending different metadata standards for different research data types. AURIN has implemented a simplified ISO 19115 standard, including attribute level metadata, and also a metadata entry for level of measurements (nominal, ordinal, interval, ratio).
Table 1 – Comparison table of major data access initiatives, INSPIRE in Europe, data.gov in the USA, and data.gov.au, ANDS RDA and AURIN data hubs from Australia
✖
✓
✓
✖
✓
Programmatic access to raw data Programmatic querying of data Standardised Metadata Data User/Custodian collaboration Open data access ✖ ✓ ✖ ✓ ✖ ✓ ✓ ✓ ✖ ✓ ✖ ✓ ✖ ✓ ✓ ✓ ✓ ✓ ✓ Research Only
Another clear difference is the number of datasets found in the portal. It has already been discussed in section 3.1 how these numbers may be misleading; in the case of INSPIRE, over 160,000 of the 300,000 metadata records are distributed from just two of the 27 countries. Data.gov clearly provides direct access to the largest set of data, however it should be noted that this is data access without associated analytical toolsets provided to assist in data understanding. The AURIN portal is the only initiative which has enabled data to be queried, analysed and visualised in addition to providing the data access. However, this portal has the fewest number of datasets, which is likely to be a result of the fact that this portal is still in beta development, and not released yet as a full production system as with each of the other initiatives.
All portals distribute data from Government agencies, but AURIN is the only portal to distribute data from the private industry as well. This is due to focused user market of the AURIN platform compared with the other portals. AURIN is designed to support academic researchers and policy/decision makers, not the general public, making the negotiation of private license agreements easier than would be the case for a public portal.
Table 1 highlights that the use of any data portal should be subject to the end-user needs. A user looking to understand data availability would be best served by the INSPIRE and ANDS RDA projects. A user wanting to access full data records should use the data.gov and data.gov.au sites with the understanding that the quality of metadata is inconsistent, and a user will need to undertake any data analysis using a third party application. Researchers in Australia can use the AURIN project to access, download, analyse, query and visualise data from a variety of urban research disciplines, with sufficient metadata to judge the suitability for purpose of each available dataset. 7
Recognising the significant challenges that population growth will have on a city’s liveability, the North-West
Melbourne Regional Management Forum (NWM – RMF) has identified the need to work collaboratively across
government and academia to develop an integrated spatial data platform to support research in the region
The North-West Melbourne Data Integration project included members from 14 Local Government Authorities
as well as AURIN and ANDS, with the aim to use web enabled technology to connect computers, exchange data
and undertake analysis. The key component of the NWM project was the ability to access and distribute spatial
datasets to various project stakeholders. This was achieved these through the creation of the Melbourne Data Hub,
which made an extensive range of health, housing, transport and planning datasets available both to contributing
agencies and to urban research across Australia via the AURIN portal
The data hub was established and maintained by the Centre for Spatial Data Infrastructure and Land
Administration (CSDILA), and consisted of two main components: a server to distribute the datasets and a tool to
harvest and enrich metadata for each dataset. The data was distributed using GeoServer Web Feature Service
(WFS), which is an open source spatial data server. CSDILA collaborated with several government agencies to
collate urban data, then clean and geocode many of the supplied datasets which were not in a suitable format for
distribution. These datasets were then imported to a Postgres/PostGIS database for distribution through the
GeoServer. A second GeoServer was also used for the project, housed at the Department of Environment and
Primary Industries (DEPI). This GeoServer was used to distribute data which was already held within the
Victorian Spatial Datamart
To bring these WFS feeds in to the AURIN portal, CSDILA also created a metadata harvesting and enrichment tool using GeoNetwork as the basis, with custom metadata requirements built in to meet the AURIN specifications. This tool was used to capture and enhance title, dataset abstract and attribute abstract information for each of the datasets before being ingested in to AURIN. This allowed the data to be consumed by AURIN with fully compliant metadata, allowing users to understand each component of the datasets.
Common metadata standards are integral to the operation of the AURIN e-infrastructure, as the portal operates using a federated architecture. This allows data to be stored and distributed in a number of different geographic locations and file formats, but understood and integrated in a consistent manner in the AURIN portal.
The data hub was also used to serve various datasets to four demonstrator projects, which were used to
demonstrate the value and utility of the Melbourne Data Hub in the urban research areas of health, housing,
walkability and employment. The data hub was fundamental in consolidating and distributing datasets for various
research application purposes. Each of the four demonstrator projects successfully used the data distributed from
the data hub. At the project completion, the Melbourne Data Hub is distributing more than 120 datasets through
the AURIN portal, from more than 10 contributing government agencies within Victoria, all with fully compliant
AURIN metadata
Setting up a data hub in this manner has also overcome the limitations encountered by the data.gov platforms, and the INSPIRE project, where subsets of data cannot be requested and access to data may be through separate data portals. The GeoServer allows for subsets of each exposed dataset to be extracted either by geography or by a tabular request, which cannot be completed in the data.gov sites. The data held in the hub itself was determined of significance by the urban research community and collaborating government agencies and local councils who had participated in the development of the hub. The project is also supported by the general AURIN “Request a Dataset” option, which allows the research community to identify future datasets for inclusion in the AURIN portal.
Unlike the INSPIRE project, the AURIN portal allows for a programmatic link directly to the data source, so no additional data searches are required. This also ensures that everything can be requested from a single source. Linked to the AURIN federated architecture, this allows data from other data hubs external to the Melbourne Data Hub to also be requested from a single portal. In addition to allowing access to data, the AURIN portal also allows user to visualise and analyse this data using ‘(i) graphs and charts, (ii) choropleth mapping, (iii) heat mapping, (iv) flow mapping, (v) brushing, (vi) space time cube (STC) representation and (vii) Decision Support Dashboards’ (Pettit et. al., 2012).
Figure 4 illustrates a typical AURIN portal session. The data cart highlights many of the contributing agencies to the Melbourne Data Hub, along with data from other data hub projects within AURIN, all of which can be accessed, analysed and visualised in the AURIN portal. The data contains detailed metadata, including links to more information for researchers to assess data quality and lineage. The figure also demonstrates a choropleth map of population, overlayed with a pedestrian catchment map from the AURIN Walkability tool.
The application of the data hub concept in Australia increases the ease of data discovery and access for urban researcher and policy/decision makers from participating Government agencies. By building upon the concept of data hubs, AURIN is also allowing researchers to collaborate on multi-disciplinary research endeavours which cut across domains such as urban health, transport, housing, water and energy supply and consumption and innovative urban design. The data hub model is also connecting researchers with Government agencies in delivering evidenced-based research outcomes that address real-world problems facing cities including liveability, housing affordability, economic prosperity and population growth. The success of the Melbourne Data Hub illustrates the benefits that can be gained from a consolidated data hub across a number of research domains. The hub also overcame the access and usability limitations experienced by some other open data initiatives where data can not be directly accessed, or where subsets of data could not be extracted and analysed. As such AURIN is further pursuing the data hub concept as the means of enabling collaborative, innovative urban research to support better urban planning and design in Australia. This will be achieved through up to 12 data hub projects during the current funding cycle of the AURIN project. These hubs will provide access to public sector, private sector and research data in the support of urban researchers across Australia. 9
Data hubs provide many opportunities for Australia, and internationally, in enabling a collaborative research environment. However, there are several challenges still to be faced in broad-scale implementation of the data hub concept, including distributing and consuming live data feeds and crowd sourced data, simplifying public and private data licensing, enabling access to a wider variety of data formats, and enabling more efficient incorporation of open government data.
Discussions have already begun between INSPIRE and AURIN, and both initiatives plan to work collaboratively to determine the most appropriate outcomes for delivering data. Further workshops detailing specific technical developments, lessons and outcomes will be required to achieve these outcomes.
The increasing availability of free data will provide more opportunities for the establishment of data hubs, and this data will need a mature framework to work within. As such, a key step for developing the data hub concept will be to develop a common framework for establishing data hub infrastructure. This will increase the interoperability of data hubs and which will support future research endeavours, particularly in multi-disciplinary collaborative research, and capturing user stories to highlight the benefits of the data hub concepts. Addressing these opportunities will maximise the use, collaboration and volume of data provided through data hubs.
Finally, an array of performance tests will need to be developed to ensure AURIN data hubs meet a consistent set of requirements, supporting stability and speed for users. These metrics will examine performance measures, and allow the assessment of strengths and weaknesses in each system. Presenting the performance framework along with the results of performance testing will be the topic of a subsequent research paper.
The authors would like to thank the AURIN groups and committees that are directly shaping these efforts. The AURIN project is funded through the Australian Education Investment Fund Super Science initiative. We gratefully acknowledge their support. 10 References