=Paper=
{{Paper
|id=Vol-3049/paper27
|storemode=property
|title=Making Sense of Open Data Policies: a Self-Evaluation Tool for Public Administrations
|pdfUrl=https://ceur-ws.org/Vol-3049/Paper27.pdf
|volume=Vol-3049
|authors=Michele Benedetti,Marco Gaeta,Claudio Russo,Luca Tangi,Irene Vanini
|dblpUrl=https://dblp.org/rec/conf/egov/BenedettiGRTV21
}}
==Making Sense of Open Data Policies: a Self-Evaluation Tool for Public Administrations==
https://ceur-ws.org/Vol-3049/Paper27.pdf
Making Sense of Open Data Policies: a Self-
Evaluation Tool for Public Administrations
Michele Benedetti*, Marco Gaeta**, Claudio Russo***, Luca
Tangi****, Irene Vanini*****
*Politecnico di Milano, Department of Management, Economics and Industrial Engineering, via Lambruschini
4b building BL26b 20156 Milan, Italy michele.benedetti@polimi.it
**EasyGov solutions s.r.l, Via Comina, 39 – 20831 Seregno (MB) Italy, marco.gaeta@easygov.it
***Politecnico di Milano, Department of Management, Economics and Industrial Engineering, via
Lambruschini 4b building BL26b 20156 Milan, Italy claudio.russo@polimi.it
****European Commission – Joint Research Centre, Via Enrico Fermi, 2749, 21027 Ispra (VA), Italy
luca.tangi@ec.europa.eu
*****Politecnico di Milano, Department of Management, Economics and Industrial Engineering, via
Lambruschini 4b building BL26b 20156 Milan, Italy irene.vanini@polimi.it
Abstract: The goal of this paper is to put forward a pragmatic approach to the debated question
of open data impact assessment, acknowledging the wide agreement on the difficulty to secure
a reliable method for objective measurement. The paper addresses the need for a viable method
to monitor open data policies through a qualitative evaluation of their impacts on administered
territories. To do so, it proposes a tool to help public administrations, especially local
governments, to perform such evaluation autonomously and in a quick and simple way, through
a questionnaire detecting the perception on the effects produced by shared information, and a
form to calculate the overall costs for data management. This self-evaluation activity aims at
encouraging internal discussion and information sharing within the administration. The tool was
developed and tested in Italy with altogether 12 interlocutors (public administrations and open
data experts) and is available to all public organisations.
Keywords: open data; open data impacts; impact evaluation; public policy evaluation
1. Research and Operational Goals
The goal of this paper is to put forward a pragmatic approach to the largely debated question of
assessing the impacts of open data, acknowledging the wide agreement on the difficulty to secure a
reliable method for objective measurement. In fact, if on the one hand the scholarly community faces
significant obstacles in establishing a scientifically robust and comprehensive evaluation procedure,
on the other hand it is consistent in advising practitioners of the public sector to pursuing policies
of openness and information sharing, seeking for a range of diverse benefits. Moreover, even if a
comprehensive dashboard of indicators could be provided, the measurement exercise would be too
extensive and time consuming for organisations of the public sector to perform it. As a result,
Copyright ©2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
282 Reflections & Viewpoints
administrations are left to autonomously strategise about opening data, without the adequate means
to monitor such policies and make better informed decisions.
The paper aims at addressing this need by proposing a tool to help public administrations, and
especially local governments, to perform an evaluation of their own open data policies. We firstly
imagined a framework for such evaluation (Figure 1). Drawing from the academic debate on impact
categorisation and from the several case studies available in literature, we developed a synthetic
indicator to calculate impacts. This measure was then to be weighted against the costs the
administrations have to bear to manage (collect, produce, publish) data in an open format
([anonymised for review]). To make this method available, and most importantly usable, for local
governments, we designed a tool made up of a questionnaire for detecting the perception of
administrations on the effects produced by shared information, and a form to calculate the overall
costs for data management. The main goal of this tool is to invite administrations to reflect on their
choices concerning data opening by providing them with a simple and rapid evaluation method.
Figure 1: The original framework
2. Literature Review
Attempts to calculate an objective measurement of open data impacts are rare and this lack of
conceptualisations and methods results in insufficient scientific evidence about the public value
behind open data (Jetzek et al., 2019). Even the Open Data Barometer report recognises that evidence
of impacts is still inadequate and mostly anecdotal (World Wide Web Foundation, 2018). Dawes et
al. (2016) reports and attempt to collect objective measurements carried out by New York City
government in 2015 that was however mostly limited to dataset management, rather focusing on
their content. Examples of indicators are: number of datasets published, number and percent of
existing datasets prioritised and scheduled for release by each agency, number of datasets removed
from the portal and reasons for removal. In the same year, the Finnish Ministry of Finance released
Reflections & Viewpoints 283
a study to invite policy makers to pursue the opening up of public information, solely arguing on
expected benefits and acknowledging “measurement difficulties and a lack of statistical or
systematically collected information on the use of public data” (Ministry of Finance, 2015). In
scientific literature too, it is a widespread practice to evaluate impacts through case studies and
theoretical analyses based on qualitative observations. However, the study of such evidence can
provide us with a categorisation of benefits, helping practitioners to carry out reflections on open
data policies.
Transparency, together with inter-institutional collaboration and civic participation, is a pillar of
the open government paradigm and open data are its main component1. The achievement of
transparency and the boost it gives to the capacity of governments to interact with their external
environment is hence considered one of the effects of opening up data (Bertot et al., 2010; Zuiderwijk
et al., 2014); but also a means to obtain further effects (Attard et al., 2015; Janssen et al., 2012a), on
governmental organisation themselves (internal impacts) or on society as a whole (external impacts).
In fact, administrations can use open data as a driver for the planning of new policies or for
improving internal processes (Bak et al., 2013). Better processes lead to higher internal efficiency,
that together with increased civic engagement enables the administration to better address citizens’
needs (Kassen, 2013). Through the release of public information, citizens have the opportunity to
take part to decision-making and policymaking processes becoming effective part of the governance
architecture rather than be involved only through electoral procedures (Attard et al., 2016). Not only
citizens, but the private sector as a whole can be stimulated (increasing revenues and employment
rate) (Abella et al., 2017): Dawes et al. (2016) reports on application developers and data analysts
that make use of data in technical formats; Loutas et al. (2012) observed how the majority of open
data based services and applications are provided by free-lancers and researchers using a single
static dataset; Borzacchiello & Craglia (2012) investigated the effects of spatial data on small and
medium enterprises.
Already in 2012 Janssen et al. (2012) proposed a categorisation of impacts, drawing from evidence
in literature, organising them in: operational and technical impacts (overlapping with what we
identified as internal impacts); political and social impacts; economic impacts. However, the
peculiarity of social impact, and the debate about its measurement (see for example Maas & Liket,
2011) suggest to treat it separately, alongside environmental impacts.
3. Methodology
The definition of the components and functions of the tool took the form of a “translation” of the
original framework, a theoretical product, into a practical, public administration-friendly device. We
chose to use an Excel spreadsheet with macro (Microsoft Excel macro-activated worksheet (.xlsm))
developed with Visual Basic Application (VBA) programming language, due to the wide use of
Office programmes on the Italian territory, where the tool was tested.
1 See https://www.opengovpartnership.org/
284 Reflections & Viewpoints
The tool was tested during 6 interviews with public administrations and open data experts and
was sent out for feedback in asynchronous mode to 6 public administrations. This process led to
changes in the tool design, the categorisation of impacts, the form of requests of information and the
intelligibility of functions and results. The process was iterated to achieve a version of the tool that
was considered useful and easy to fill in. Three versions of the tool were then produced: (i) in Italian
with valuation of costs in €, for Italian local governments (ii) in Italian with valuation of costs in
CHF, for local governments in the Canton of Ticino (CH) (iii) in English, for wider dissemination
and future research.
4. Designing the Tool
The tool is divided into four different sections, each delimited by a dedicated Excel sheet.
Instructions to fill the tool with required information are displayed on each sheet and dedicated
buttons facilitate the user.
In the first section ("Landing") the user enters the categories of datasets that the administration
publishes and the number of issued datasets for each category. The categorisation of datasets is the
one provided by the Italian national agency for digital transformation2.
The second section (“Impacts”) includes a 15 items questionnaire on perceived impacts, grouped
into four thematic areas built upon the categorisation of impacts found in literature and revised
through the tests, following the suggestions of interviewed administrations: economic, technical and
operational, social and environmental, political and cultural). For each question, respondents can
express a qualitative perceptive judgment on a Likert scale basis ranging from 0 to 5, about the extent
to which, according to them, the publication of specific categories of data generated impacts. In this
section, public administrations are also required to indicate the total number of downloads by
category of datasets, as a proxy of uses of such datasets.
Figure 2: Evaluation results: map of dataset categories on the framework
2 https://dati.gov.it/
Reflections & Viewpoints 285
The third section, “Costs”, estimates of the costs sustained for data management. To ease the
filling procedure, two different tables were included: one to list general raw costs faced by the
administration in the timeframe under analysis, another to quantify the effort in terms of hours spent
in each activity of the entire data publication process.
The fourth and final section, "Results", contains a summary of the inserted data and a visual
representation of the so completed evaluation on the framework. Three graphs are displayed: a map
of dataset categories on the framework (Figure 2), on which the dimension of bubbles varies
depending on the number of downloads; a pie chart that shows the breakdown of costs items along
all the inserted data categories; a column chart of impacts by thematic area.
To conclude, we believe the proposed tool has the following advantages:
• it has theoretical foundation: the two axes are built on the basis of the results deriving from
the review of the literature on impacts and costs.
• it is flexible: the categories of the datasets are not linked in advance to any specific impact.
Instead, the user is free to assign any impact value associated with the various questions.
Moreover, shall the respondent wanting to evaluate a single dataset instead of the whole
category, the “Landing” section provides the possibility to customise an item of evaluation.
• allows ex-post analysis: this is the main purpose of the framework, which intends to allow the
analysis of where and how the impacts have occurred and to deduct the importance of each
category of datasets. This enables administrations to reflect on the effectiveness of their own
open data policies by looking at their overall impact and costs.
• allows an ex-ante analysis: the framework is at the service of administrations for the definition
of the expected results in terms of the impacts generated by open data. Nonetheless, it can be
used also to make strategic decisions, using the “Impacts” section to specify the objectives the
administration wants to achieve.
References
Abella, A., Ortiz-De-Urbina-Criado, M., & De-Pablos-Heredero, C. (2017). A model for the analysis of data-
driven innovation and value generation in smart cities’ ecosystems. Cities, 64(April), 47–53.
https://doi.org/10.1016/j.cities.2017.01.011
Attard, J., Orlandi, F., & Auer, S. (2016). Value Creation on Open Government Data. In Proceedings of the
49th Hawaii International Conference on System Sciences (HICSS-49) (pp. 2605–2614). IEEE.
https://doi.org/10.1109/HICSS.2016.326
Attard, J., Orlandi, F., Scerri, S., & Auer, S. (2015). A systematic review of open government data initiatives.
Government Information Quarterly, 32(4), 399–418. https://doi.org/10.1016/j.giq.2015.07.006
Bak, T., Almirall, E., & Wareham, J. (2013). N2: A Smart City Initiative : the Case of Barcelona. January 2012,
135–148. https://doi.org/10.1007/s13132-012-0084-9
286 Reflections & Viewpoints
Bertot, J. C., Jaeger, P. T., & Grimes, J. M. (2010). Using ICTs to create a culture of transparency: E-
government and social media as openness and anti-corruption tools for societies. Government
Information Quarterly, 27(3), 264–271.
Borzacchiello, M. T., & Craglia, M. (2012). The impact on innovation of open access to spatial environmental
information: A research strategy. International Journal of Technology Management, 60(1–2), 114–129.
https://doi.org/10.1504/IJTM.2012.049109
Dawes, S. S., Vidiasova, L., & Parkhimovich, O. (2016). Planning and designing open government data
programs: An ecosystem approach. Government Information Quarterly, 33(1), 15–27.
https://doi.org/10.1016/j.giq.2016.01.003
Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012a). Benefits, Adoption Barriers and Myths of Open Data
and Open Government. Information Systems Management, 29(4), 258–268.
https://doi.org/10.1080/10580530.2012.716740
Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012b). N1: Benefits, Adoption Barriers and Myths of Open
Data and Open Government* Marijn Janssen, Yannis Charalabidis & Anneke Zuiderwijk. Information
Systems Management, 29(4), 258–268.
Jetzek, T., Avital, M., & Bjørn-Andersen, N. (2019). N146.The sustainable value of open government data.
Journal of the Association for Information Systems, 20(6), 702–734.
https://doi.org/10.17705/1jais.00549
Kassen, M. (2013). A promising phenomenon of open data: A case study of the Chicago open data project.
Government Information Quarterly. https://doi.org/10.1016/j.giq.2013.05.012
Loutas, N., Varitimou, A., Peristeras, V., & Galway, N. U. I. (2012). Unraveling the mystery of Open
Government Data Apps. Using Open Data: Policy Modeling, Citizen Empowerment, Data Journalism
(PMOD 2012), 1380.
Maas, K., & Liket, K. (2011). Social Impact Measurement: Classification of Methods.
https://doi.org/10.1007/978-94-007-1390-1_8
Ministry of Finance. (2015). The Impact of open data – a preliminary study.
World Wide Web Foundation. (2018). Open Data Barometer. From Promise to Progress.
Zuiderwijk, A., Janssen, M., & Davis, C. (2014). N11.Innovation with open data: Essential elements of open
data ecosystems. Information Polity, 19(1–2), 17–33. https://doi.org/10.3233/IP-140329
About the Authors
Michele Benedetti
Michele Benedetti is a research fellow and lecturer at the School of Management of the Politecnico di Milano.
Since 2001 he has carried out research on digital innovation, studying the role and impact of digital on Public
Administration organization and management and deepening new models enabled by ICT technologies for
the provision of public services. He also gained almost twenty years of experience in managing complex
projects of digital transformation in the PA in collaboration with Municipalities, Provinces, Regions and
Ministries. Since 2009 he has been director of the eGovernment Observatory of the School of Management
of the Politecnico di Milano and since 2017 also of the Digital Agenda Observatory.
Reflections and Viewpoints 287
Marco Gaeta
Marco Gaeta is a research fellow in the e-government field at the Department of Management, Economics
and Industrial Engineering of the Politecnico di Milano. Here, after his Master's degree in Management
Engineering, he started working as a research fellow, following the focus and the research interests of his
master thesis concerning the digitalization of Italian local governments and innovation in the public sector.
Claudio Russo
PhD in Management, Economics and Industrial Engineering at the Politecnico di Milano, has been working
for 15 years in researches related to organizational and management models of inter-institutional
cooperation, multi-level governance, digital transformation and support activities in the management of
organizational and technological change projects in the Public Administration. Lecturer at the School of
Industrial and Information Engineering and the MIP Business School of the Politecnico di Milano.
Luca Tangi
Luca Tangi is a project officer at the Joint Research Centre (JRC) of the European Commission. He earned
a PhD in Management, Economics and Industrial Engineering at the Politecnico di Milano. His doctoral work
focused on understanding how ICTs are affecting public service delivery and transforming the way public
organisations are structured and organised. Since June 2021 he collaborates with the JRC carrying out
research on the introduction of new, cutting-edge technologies and in particular Artificial Intelligence in
public settings.
Irene Vanini
After earning a PhD in Political Theory at the Università degli studi di Milano, and a three-year long research
and teaching experience at the Universities of Sheffield and York (UK), she joined Politecnico di Milano as
a research associate in 2018. She collaborates with the eGovernment and Digital Agenda Observatories and
carries out funded projects on innovation, digital transformation and governance in the public sector.