Making Sense of Open Data Policies: a Self- Evaluation Tool for Public Administrations Michele Benedetti*, Marco Gaeta**, Claudio Russo***, Luca Tangi****, Irene Vanini***** *Politecnico di Milano, Department of Management, Economics and Industrial Engineering, via Lambruschini 4b building BL26b 20156 Milan, Italy michele.benedetti@polimi.it **EasyGov solutions s.r.l, Via Comina, 39 – 20831 Seregno (MB) Italy, marco.gaeta@easygov.it ***Politecnico di Milano, Department of Management, Economics and Industrial Engineering, via Lambruschini 4b building BL26b 20156 Milan, Italy claudio.russo@polimi.it ****European Commission – Joint Research Centre, Via Enrico Fermi, 2749, 21027 Ispra (VA), Italy luca.tangi@ec.europa.eu *****Politecnico di Milano, Department of Management, Economics and Industrial Engineering, via Lambruschini 4b building BL26b 20156 Milan, Italy irene.vanini@polimi.it Abstract: The goal of this paper is to put forward a pragmatic approach to the debated question of open data impact assessment, acknowledging the wide agreement on the difficulty to secure a reliable method for objective measurement. The paper addresses the need for a viable method to monitor open data policies through a qualitative evaluation of their impacts on administered territories. To do so, it proposes a tool to help public administrations, especially local governments, to perform such evaluation autonomously and in a quick and simple way, through a questionnaire detecting the perception on the effects produced by shared information, and a form to calculate the overall costs for data management. This self-evaluation activity aims at encouraging internal discussion and information sharing within the administration. The tool was developed and tested in Italy with altogether 12 interlocutors (public administrations and open data experts) and is available to all public organisations. Keywords: open data; open data impacts; impact evaluation; public policy evaluation 1. Research and Operational Goals The goal of this paper is to put forward a pragmatic approach to the largely debated question of assessing the impacts of open data, acknowledging the wide agreement on the difficulty to secure a reliable method for objective measurement. In fact, if on the one hand the scholarly community faces significant obstacles in establishing a scientifically robust and comprehensive evaluation procedure, on the other hand it is consistent in advising practitioners of the public sector to pursuing policies of openness and information sharing, seeking for a range of diverse benefits. Moreover, even if a comprehensive dashboard of indicators could be provided, the measurement exercise would be too extensive and time consuming for organisations of the public sector to perform it. As a result, Copyright ©2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 282 Reflections & Viewpoints administrations are left to autonomously strategise about opening data, without the adequate means to monitor such policies and make better informed decisions. The paper aims at addressing this need by proposing a tool to help public administrations, and especially local governments, to perform an evaluation of their own open data policies. We firstly imagined a framework for such evaluation (Figure 1). Drawing from the academic debate on impact categorisation and from the several case studies available in literature, we developed a synthetic indicator to calculate impacts. This measure was then to be weighted against the costs the administrations have to bear to manage (collect, produce, publish) data in an open format ([anonymised for review]). To make this method available, and most importantly usable, for local governments, we designed a tool made up of a questionnaire for detecting the perception of administrations on the effects produced by shared information, and a form to calculate the overall costs for data management. The main goal of this tool is to invite administrations to reflect on their choices concerning data opening by providing them with a simple and rapid evaluation method. Figure 1: The original framework 2. Literature Review Attempts to calculate an objective measurement of open data impacts are rare and this lack of conceptualisations and methods results in insufficient scientific evidence about the public value behind open data (Jetzek et al., 2019). Even the Open Data Barometer report recognises that evidence of impacts is still inadequate and mostly anecdotal (World Wide Web Foundation, 2018). Dawes et al. (2016) reports and attempt to collect objective measurements carried out by New York City government in 2015 that was however mostly limited to dataset management, rather focusing on their content. Examples of indicators are: number of datasets published, number and percent of existing datasets prioritised and scheduled for release by each agency, number of datasets removed from the portal and reasons for removal. In the same year, the Finnish Ministry of Finance released Reflections & Viewpoints 283 a study to invite policy makers to pursue the opening up of public information, solely arguing on expected benefits and acknowledging “measurement difficulties and a lack of statistical or systematically collected information on the use of public data” (Ministry of Finance, 2015). In scientific literature too, it is a widespread practice to evaluate impacts through case studies and theoretical analyses based on qualitative observations. However, the study of such evidence can provide us with a categorisation of benefits, helping practitioners to carry out reflections on open data policies. Transparency, together with inter-institutional collaboration and civic participation, is a pillar of the open government paradigm and open data are its main component1. The achievement of transparency and the boost it gives to the capacity of governments to interact with their external environment is hence considered one of the effects of opening up data (Bertot et al., 2010; Zuiderwijk et al., 2014); but also a means to obtain further effects (Attard et al., 2015; Janssen et al., 2012a), on governmental organisation themselves (internal impacts) or on society as a whole (external impacts). In fact, administrations can use open data as a driver for the planning of new policies or for improving internal processes (Bak et al., 2013). Better processes lead to higher internal efficiency, that together with increased civic engagement enables the administration to better address citizens’ needs (Kassen, 2013). Through the release of public information, citizens have the opportunity to take part to decision-making and policymaking processes becoming effective part of the governance architecture rather than be involved only through electoral procedures (Attard et al., 2016). Not only citizens, but the private sector as a whole can be stimulated (increasing revenues and employment rate) (Abella et al., 2017): Dawes et al. (2016) reports on application developers and data analysts that make use of data in technical formats; Loutas et al. (2012) observed how the majority of open data based services and applications are provided by free-lancers and researchers using a single static dataset; Borzacchiello & Craglia (2012) investigated the effects of spatial data on small and medium enterprises. Already in 2012 Janssen et al. (2012) proposed a categorisation of impacts, drawing from evidence in literature, organising them in: operational and technical impacts (overlapping with what we identified as internal impacts); political and social impacts; economic impacts. However, the peculiarity of social impact, and the debate about its measurement (see for example Maas & Liket, 2011) suggest to treat it separately, alongside environmental impacts. 3. Methodology The definition of the components and functions of the tool took the form of a “translation” of the original framework, a theoretical product, into a practical, public administration-friendly device. We chose to use an Excel spreadsheet with macro (Microsoft Excel macro-activated worksheet (.xlsm)) developed with Visual Basic Application (VBA) programming language, due to the wide use of Office programmes on the Italian territory, where the tool was tested. 1 See https://www.opengovpartnership.org/ 284 Reflections & Viewpoints The tool was tested during 6 interviews with public administrations and open data experts and was sent out for feedback in asynchronous mode to 6 public administrations. This process led to changes in the tool design, the categorisation of impacts, the form of requests of information and the intelligibility of functions and results. The process was iterated to achieve a version of the tool that was considered useful and easy to fill in. Three versions of the tool were then produced: (i) in Italian with valuation of costs in €, for Italian local governments (ii) in Italian with valuation of costs in CHF, for local governments in the Canton of Ticino (CH) (iii) in English, for wider dissemination and future research. 4. Designing the Tool The tool is divided into four different sections, each delimited by a dedicated Excel sheet. Instructions to fill the tool with required information are displayed on each sheet and dedicated buttons facilitate the user. In the first section ("Landing") the user enters the categories of datasets that the administration publishes and the number of issued datasets for each category. The categorisation of datasets is the one provided by the Italian national agency for digital transformation2. The second section (“Impacts”) includes a 15 items questionnaire on perceived impacts, grouped into four thematic areas built upon the categorisation of impacts found in literature and revised through the tests, following the suggestions of interviewed administrations: economic, technical and operational, social and environmental, political and cultural). For each question, respondents can express a qualitative perceptive judgment on a Likert scale basis ranging from 0 to 5, about the extent to which, according to them, the publication of specific categories of data generated impacts. In this section, public administrations are also required to indicate the total number of downloads by category of datasets, as a proxy of uses of such datasets. Figure 2: Evaluation results: map of dataset categories on the framework 2 https://dati.gov.it/ Reflections & Viewpoints 285 The third section, “Costs”, estimates of the costs sustained for data management. To ease the filling procedure, two different tables were included: one to list general raw costs faced by the administration in the timeframe under analysis, another to quantify the effort in terms of hours spent in each activity of the entire data publication process. The fourth and final section, "Results", contains a summary of the inserted data and a visual representation of the so completed evaluation on the framework. Three graphs are displayed: a map of dataset categories on the framework (Figure 2), on which the dimension of bubbles varies depending on the number of downloads; a pie chart that shows the breakdown of costs items along all the inserted data categories; a column chart of impacts by thematic area. To conclude, we believe the proposed tool has the following advantages: • it has theoretical foundation: the two axes are built on the basis of the results deriving from the review of the literature on impacts and costs. • it is flexible: the categories of the datasets are not linked in advance to any specific impact. Instead, the user is free to assign any impact value associated with the various questions. Moreover, shall the respondent wanting to evaluate a single dataset instead of the whole category, the “Landing” section provides the possibility to customise an item of evaluation. • allows ex-post analysis: this is the main purpose of the framework, which intends to allow the analysis of where and how the impacts have occurred and to deduct the importance of each category of datasets. This enables administrations to reflect on the effectiveness of their own open data policies by looking at their overall impact and costs. • allows an ex-ante analysis: the framework is at the service of administrations for the definition of the expected results in terms of the impacts generated by open data. Nonetheless, it can be used also to make strategic decisions, using the “Impacts” section to specify the objectives the administration wants to achieve. References Abella, A., Ortiz-De-Urbina-Criado, M., & De-Pablos-Heredero, C. (2017). A model for the analysis of data- driven innovation and value generation in smart cities’ ecosystems. Cities, 64(April), 47–53. https://doi.org/10.1016/j.cities.2017.01.011 Attard, J., Orlandi, F., & Auer, S. (2016). Value Creation on Open Government Data. In Proceedings of the 49th Hawaii International Conference on System Sciences (HICSS-49) (pp. 2605–2614). IEEE. https://doi.org/10.1109/HICSS.2016.326 Attard, J., Orlandi, F., Scerri, S., & Auer, S. (2015). A systematic review of open government data initiatives. Government Information Quarterly, 32(4), 399–418. https://doi.org/10.1016/j.giq.2015.07.006 Bak, T., Almirall, E., & Wareham, J. (2013). N2: A Smart City Initiative : the Case of Barcelona. January 2012, 135–148. https://doi.org/10.1007/s13132-012-0084-9 286 Reflections & Viewpoints Bertot, J. C., Jaeger, P. T., & Grimes, J. M. (2010). Using ICTs to create a culture of transparency: E- government and social media as openness and anti-corruption tools for societies. Government Information Quarterly, 27(3), 264–271. Borzacchiello, M. T., & Craglia, M. (2012). The impact on innovation of open access to spatial environmental information: A research strategy. International Journal of Technology Management, 60(1–2), 114–129. https://doi.org/10.1504/IJTM.2012.049109 Dawes, S. S., Vidiasova, L., & Parkhimovich, O. (2016). Planning and designing open government data programs: An ecosystem approach. Government Information Quarterly, 33(1), 15–27. https://doi.org/10.1016/j.giq.2016.01.003 Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012a). Benefits, Adoption Barriers and Myths of Open Data and Open Government. Information Systems Management, 29(4), 258–268. https://doi.org/10.1080/10580530.2012.716740 Janssen, M., Charalabidis, Y., & Zuiderwijk, A. (2012b). N1: Benefits, Adoption Barriers and Myths of Open Data and Open Government* Marijn Janssen, Yannis Charalabidis & Anneke Zuiderwijk. Information Systems Management, 29(4), 258–268. Jetzek, T., Avital, M., & Bjørn-Andersen, N. (2019). N146.The sustainable value of open government data. Journal of the Association for Information Systems, 20(6), 702–734. https://doi.org/10.17705/1jais.00549 Kassen, M. (2013). A promising phenomenon of open data: A case study of the Chicago open data project. Government Information Quarterly. https://doi.org/10.1016/j.giq.2013.05.012 Loutas, N., Varitimou, A., Peristeras, V., & Galway, N. U. I. (2012). Unraveling the mystery of Open Government Data Apps. Using Open Data: Policy Modeling, Citizen Empowerment, Data Journalism (PMOD 2012), 1380. Maas, K., & Liket, K. (2011). Social Impact Measurement: Classification of Methods. https://doi.org/10.1007/978-94-007-1390-1_8 Ministry of Finance. (2015). The Impact of open data – a preliminary study. World Wide Web Foundation. (2018). Open Data Barometer. From Promise to Progress. Zuiderwijk, A., Janssen, M., & Davis, C. (2014). N11.Innovation with open data: Essential elements of open data ecosystems. Information Polity, 19(1–2), 17–33. https://doi.org/10.3233/IP-140329 About the Authors Michele Benedetti Michele Benedetti is a research fellow and lecturer at the School of Management of the Politecnico di Milano. Since 2001 he has carried out research on digital innovation, studying the role and impact of digital on Public Administration organization and management and deepening new models enabled by ICT technologies for the provision of public services. He also gained almost twenty years of experience in managing complex projects of digital transformation in the PA in collaboration with Municipalities, Provinces, Regions and Ministries. Since 2009 he has been director of the eGovernment Observatory of the School of Management of the Politecnico di Milano and since 2017 also of the Digital Agenda Observatory. Reflections and Viewpoints 287 Marco Gaeta Marco Gaeta is a research fellow in the e-government field at the Department of Management, Economics and Industrial Engineering of the Politecnico di Milano. Here, after his Master's degree in Management Engineering, he started working as a research fellow, following the focus and the research interests of his master thesis concerning the digitalization of Italian local governments and innovation in the public sector. Claudio Russo PhD in Management, Economics and Industrial Engineering at the Politecnico di Milano, has been working for 15 years in researches related to organizational and management models of inter-institutional cooperation, multi-level governance, digital transformation and support activities in the management of organizational and technological change projects in the Public Administration. Lecturer at the School of Industrial and Information Engineering and the MIP Business School of the Politecnico di Milano. Luca Tangi Luca Tangi is a project officer at the Joint Research Centre (JRC) of the European Commission. He earned a PhD in Management, Economics and Industrial Engineering at the Politecnico di Milano. His doctoral work focused on understanding how ICTs are affecting public service delivery and transforming the way public organisations are structured and organised. Since June 2021 he collaborates with the JRC carrying out research on the introduction of new, cutting-edge technologies and in particular Artificial Intelligence in public settings. Irene Vanini After earning a PhD in Political Theory at the Università degli studi di Milano, and a three-year long research and teaching experience at the Universities of Sheffield and York (UK), she joined Politecnico di Milano as a research associate in 2018. She collaborates with the eGovernment and Digital Agenda Observatories and carries out funded projects on innovation, digital transformation and governance in the public sector.