An Assessment of Open Government Data Benchmark Instruments

Abstract

Open Government Data (OGD) is heralded as a pillar for promoting openness and eGovernment. Several OGD benchmark instruments have been proposed, and so many options could confuse open data users. We intend to help practitioners and researchers decide which benchmark instrument is most appropriate to evaluate a specific purpose. We aim to investigate the different dimensions that OGD benchmarks evaluate to discover what aspects of publishing and using open data they measure. To achieve this goal, we built upon previous research on how the Open Data Charter principles are measured in OGD assessments and enriched the analysis with additional dimensions. Findings reveal that what differentiates these benchmark instruments is their scope or focus, as their creators have varying interests and objectives. All benchmark instruments measure "data openness"; however, each emphasizes different aspects of "openness." Concepts measured are deeply connected with the six Charter principles. OGD impact, use, and usefulness are addressed by half of the benchmark instruments. Measurements can be compared. Their assessments can be used to help improve data quality and the conditions for sharing and reusing OGD.

Keywords

• e-Government
• Open Data
• Open Government Data
• Benchmark
• Assessment

1. Introduction

Governments produce, collect, maintain, and disseminate significant amounts of data (Kalampokis et al., 2011) . Thus, data are the driver of the digital transformation as government transitions to eGovernment (Attard et al., 2016) . A contemporary definition of Open Government Data (OGD) encompasses not only the idea of open data but also of the e-Government and Open Government (OECD, 2019) .

Relevant literature shows that Open Government Data initiatives are often carried out to achieve objectives such as increase transparency, participation (Ubaldi, 2013; Alexopoulos et al., 2013; Lee & Kwak, 2012; Attard et al., 2016) , and collaboration (Alexopoulos et al., 2013; Attard et al., 2016) ; foster innovation; improve economic value (Johnson & Robinson, 2014; Jetzek, 2015; Klein et al., 2018) ; and create public value (Attard et al., 2016) . Researchers, practitioners, and international organizations consider the creation of public value as one of the most important benefits of OGD (Attard et al., 2016; Zuiderwijk & Janssen, 2014; Janssen et al., 2012; Lämmerhirt et al., 2017; Cecconi & Radu, 2018; Lee & Kwak, 2012) . Government data is seen as a valuable resource offering potential benefits to stakeholders if it is openly available (Hitz-Gamper et al., 2019) . However, publishing OGD does not ensure public value creation, as value is only realized when data are used (Davies & Bawa, 2012; Ubaldi, 2013) .

Over the years, academics and advocacy institutions, such as the World Wide Web (Web) Foundation and Open Knowledge Foundation (OKFn), proposed assessments to improve Open Government Data usage. The International Open Data Charter (IODC) principles (Open Data Charter, 2013) provide a consistent approach to assess the degree of data openness. The Open Data Charter Measurement Guide (Brandusescu et al., 2018) analyzed the Charter principles and how they were assessed based on five current "open government data measurement tools." Davies (2013) identified over a dozen OGD "evaluation and assessment frameworks," which were classified into three categories: open data readiness, open data implementation, and open data impact. (Susha et al., 2015) conducted a meta-analysis of five international open data benchmarks or assessments for evaluating open data progress. In a recent publication, Vancauwenberghe (2018) identified and evaluated 15 open data assessments. In the literature, the terms OGD benchmark, assessment framework, assessment, evaluation, and survey are used indistinctively. For the sake of clarity, this paper adopts the terms "benchmark" or "assessment" to refer to the measurement of the nature, quality, or ability of OGD; and "instrument" or "benchmark instrument" for the measuring artifact itself (e.g., a questionnaire).

Although all those studies compare Open Government Data assessments, the majority of the assessments are not used anymore. They were proposed, executed once, and may not reflect the actual state of OGD initiatives. We intend to help open data users, practitioners, and researchers identify which current assessment (executed after 2016) is most appropriate for evaluating a specific purpose. Thus, this study aims to investigate the different dimensions of contemporary OGD benchmark instruments and determine what aspects of publishing and using open data they currently measure. To achieve this goal, we built upon the Measurement Guide (Brandusescu et al., 2018) by adding the evaluation of a sixth OGD measurement tool to its basket and new analysis dimensions. We also expand the (Susha et al., 2015) study by applying the proposed frame of reference to compare the six current OGD benchmark instruments in terms of metadata, metamethod, and meta-theory. This paper contribution lies not in proposing new ways of evaluating OGD benchmark instruments but rather in drawing new insights into them by combining previous research (Brandusescu et al., 2018) and enriching with the addition of other dimensions such as level of analysis, geographic coverage, frequency, methodological approach and validation, and data collection (data source).

The remaining of this paper is organized as follows. Section 2 presents the research design. Section 3 briefly reviews related work and describes the six OGD benchmark instruments. Section 4 presents results and discussion. In Section 5, we draw our conclusions and suggest future work.

2. Research Design

This study aims to investigate the different dimensions of contemporary OGD benchmark instruments and determine what aspects of publishing and using open data they currently measure. We compare benchmarks instruments to determine how they are developed in terms of metadata, method, underlying theory, and the International Open Data Charter principles. Hence, this study is framed by (1) What are the conceptual differences and similarities of OGD benchmark instruments?; (2) How do OGD benchmark instruments measure the different aspects of the open data?

OGD assessments were identified in the literature (Davies, 2013; Susha et al., 2015; Brandusescu et al., 2018; Vancauwenberghe, 2018) . The criteria to include them in this study were (1) benchmark instrument evaluates and may rank countries, organizations, and projects based on the publication and/or use of OGD; (2) benchmark instrument evaluates and may rank countries, organizations, and projects internationally; (3) benchmark methodology and data are publicly available or can be accessed; (4) benchmark instrument is current, i.e., the last assessment is from 2016 or later. Any benchmark that did not focus on government data was excluded. The application of the inclusion/exclusion criteria in over 20 assessments identified in the literature resulted in the selection of six international OGD instruments: the Open Data Barometer (ODB) (W3C, 2019) produced by the World Wide Web Foundation (W3CF); the Global Open Data Index (GODI) (OKFn, 2019a) created by the Open Knowledge Foundation (OKFn); Open-Useful-Re-Usable Government Data Index (OURdata Index) (OECD, 2019) developed by the Organization for Economic Cooperation and Development (OECD); the Open Data Inventory (ODIN) (Open Data Watch, 2019) developed by the Open Data Watch (ODW); the European Open Data Maturity Assessment (EODMA) (Cecconi & Radu, 2018) developed by European Data Portal (EDP); and the Open Data Monitor (ODM) (European Commission & Consortium of Collaborators, 2019) was created by the European Commission (EC) and Consortium (ECC).

Several open data assessments are conducted as surveys, questionnaires, and field studies. Findings and results are available in the form of indexes or rankings published online and sometimes complemented by analytical reports.

To answer the first research question, we adopt the frame of reference (Susha et al., 2015) obtained from the meta-analysis of five OGD benchmark instruments. The framework is appropriate for comparing the scope and focus of the assessments, their theoretical foundations, and the methodology used in rankings. Thus, it can be adopted in this study, as it has been applied in similar assessments. The underlying concepts of the framework are frequency (how often assessments are conducted/published), source (independent, to sell research findings, marking purposes and academic), and scope, or focus (the purpose of the benchmark instrument determines focus/scope). The scope varies with benchmark instruments and with time, and scale (international, regional, municipal) (Bannister, 2007) ; the methodologies underlying individual studies, including sampling, data collection, and research design; and meta-theory (reflecting on the theoretical assumptions at the foundation of the individual studies).

The rationale and data presented in the Open Data Charter Measurement Guide (Brandusescu et al., 2018) were used to address the second research question. The Guide's reasoning was applied to analyze the Open Data Monitor against the Charter principles. The International Open Data Charter principles (Open Data Charter, 2013) were used to verify whether and how the selected assessments evaluate the several aspects of publishing and using OGD. The Charter principles are Open by Default, Timely and Comprehensive, Accessible and Usable, Comparable and Interoperable, For Improved Governance and Citizen Engagement, and For Inclusive Development and Innovation. Principles represent a global agreement of how to publish and access open data and offer a consistent approach to evaluate and compare assessments. Charter principles and their components are considered as "commitments" governments and institutions pledge to carry out. They aim to improve the quality of the data published and the conditions for sharing and reusing open data (Open Data Charter, 2013) .

3. Related Work

Open data assessment is about collecting, analyzing, and providing information on the performance of open data initiatives (Vancauwenberghe, 2018) . OGD benchmark instruments can be used to evaluate and rank countries, organizations, and projects based on how they publish and use OGD in different ways (Atz et al., 2015) . The process can help disseminate the use of OGD standards across OGD projects and consequently improve accountability. It also helps to understand and communicate what are the best ways to use OGD to solve problems. Maheshwari & Janssen (Maheshwari & Janssen, 2013) define benchmarking as the measurement of specific elements and the comparison of results to a baseline (or assessment). It provides an organization with a diagnostic of its current position and offers paths for improvement and growth. However, the authors argue that OGD benchmark instruments often lack the critical elements required to fuel further development. Although benchmarks may have limited practical meaning, they may have a significant political and potential economic impact (Bannister, 2007) . The author cautions that OGD benchmarks outcomes may influence decision-makers with wrong perceptions. Results should be treated with caution, as some of them are grossly simplified.

Davies (2013) classified over twelve OGD assessments into three categories. OGD readiness appraises weather conditions to start or successfully implement OGD initiatives exist. OGD implementation assesses whether data are available and open. OGD impact investigates what economic, social, political benefits OGD initiatives might generate.

Although all categories deal with OGD assessment, each focuses on different aspects of OGD initiatives and practices. The level of analysis also differs. Most readiness assessments operate at the country level. In contrast, implementation assessments may focus on some individual datasets, portals, individual institutions, OGD initiatives, and whole countries. Some assessments aggregate the evaluation of initiatives, portals, or institutions based on summing up numerical scores for the "openness" of the parent entity's datasets. Approaches to measurement include the survey of technical features, specific dataset checklist, domain-specific assessments, added-value features), features of the environment (describe the social, technical, legal, political, economic, and organizational contexts for OGD), and expert surveys. None of the assessments explicitly addressed the impact or the use of OGD. Most assessments are no longer used; an exception is The Open Data Census (OKFn, 2019b) .

The Measurement Guide (Brandusescu et al., 2018) analyzed how the Charter principles are being measured or not by five OGD benchmark instruments. They are the Open Data Barometer (ODB), the Global Open Data Index (GODI), Open Data Inventory (ODIN), Open Useful Reusable Government Data (OURdata), and the European Open Data Maturity Assessment (EODMA). The metrics of these five benchmark instruments are compared against each of the six Charter principles and their commitments. The evaluation was performed by the open data analysts responsible for creating the assessment.

(Susha et al., 2015) conducted a meta-analysis of five international OGD benchmark instruments for evaluating open data progress. Their theoretical assumptions were compared to four existing academic open data maturity models from the literature. They were evaluated in terms of metadata, meta-method, and meta-theory to measure open data progress. The authors reflect that the lack of assessment of the actual use of data is the OGD benchmark missing link. They explain that the use of data is complicated to measure, and therefore is only indirectly accounted for (i.e., as community activity or emerging impacts). The proposed framework reveals important aspects for practitioners and academia, thus our interest in it. Two out of five benchmark instruments provide recent assessments: the Global Open Data Index (OKFn, 2019a) and the Open Data Barometer (W3C, 2019).

Vancauwenberghe (2018) study classified 15 OGD assessments using the four dimensions defined in the Common Assessment Framework (CAF) (Caplan et al., 2014). CAF provides a standardized methodology for the analysis of the supply, use, and impact of open data. CAF dimensions are (1) Context/Environment: the context within which open data is being provided (national, or sectorial such as health, education, or transport); (2) Data: deals with the nature of data (legal, technical and social, openness, and relevance) and quality of open datasets; (3) Use: the types of users accessing data, the purposes for which the data is used and the activities being undertaken to use it; and (4) impact: the benefits obtained from using specific open datasets, or from open data initiatives in general. Benefits can be social, environmental, political, and economical. Eleven out of the 15 assessments focus on readiness and data dimensions. The dimension impact was addressed in six assessments, as was the use dimension. The combination of both dimensions was addressed only in four assessments. These findings suggest that an investigation opportunity exists in terms of OGD use and impact dimensions.

4. Results and Discussion

Data and analysis of Charter principles coverage were gathered from (Brandusescu et al., 2018) for five benchmark instruments. The exception was ODM, which evaluation was carried out by the authors using ODM's methodology document ODM. OURdata index information was complemented with information from OECD (Lafortune & Ubaldi, 2018; OECD, 2018) documents. The results are shown in the last line of Table 1.

ODM ECC

National

First 2015: EU countries

Periodicall y 34

Benchmark instrument

Developed by

Level of analysis

Geographic coverage

Frequency

Methodologi

cal approach / validation 1 EU28+ encompasses the 28 EU member states plus Iceland, Liechtenstein, Norway, and Switzerland. Hungary did not participate. 2 G8, G20, most OGP, and OECD countries.

Open data Independen Coverage Metadata

readiness, t assessment and completene implementati of OGD openness ss, open on, and publication of official license, impact based from a civic national formats on the perspective statistics and scope,

Portals,

and Impact reuse based the Charter principles on

Charter

principles and data availability

Charter

Principle coverage

Partially Partially Partially covers all covers all covers all six six six principles principles principles

Covers Partially principles 2 covers and 3 principles 2, 3 and 4

Partially covers principles 3 and 4

4.1. RQ1 - What are the conceptual differences and similarities of OGD benchmark instruments? Benchmark instruments were compared in terms of the level of analysis, geographic coverage, frequency, methodological approach and validation, data collection/data source, concepts measured, and International Open Data Chapter principles coverage.

Most assessments are conducted at the national level. The exception is ODIN, which context is subnational (administrative levels 1 and 2). All benchmarks are international. European Union members and OECD countries are accounted for in all assessments. ODIN and OBD represent a broader spectrum with over 100 countries, while OEDMA and ODM focus on the European Union. Four benchmarks are annual. OURData is biennial, ODM started in 2015. It runs periodically.

The majority of the benchmark instruments rely on surveys to collect data and validate results with the help of experts, government officials, and the community. Government officials complete the OEDMA government survey. The European Data Portal team, in cooperation with these officials, validate and analyze the data. High-level government officials from OECD countries and partners respond to the survey. The OECD Secretariat conducts the analysis, which includes secondary thirdparty indicators. The International Open Data Charter (IODC) implementation is monitored. ODB relies on expert surveys and secondary data. Assessment is based on quantitative and qualitative data that combines contextual data, technical assessments, and secondary third-party indicators. Results are peer-reviewed and pass a Quality Assurance (QA) process. GODI uses ongoing crowdsourcing with expert review to create an annual index. A checklist with qualitative justifications is used to validate outcomes. Discussions from the survey and review process are displayed publicly. Trained researchers carry out ODIN research. Input from government officials is taken into consideration. Open Data Watch staff conducts two rounds of review to validate the data. ODM periodically collects metadata from external portals and catalogs (Harvesting).

The study reveals that regarding concepts, each benchmark instrument was created to serve different purposes with varying degrees of specificity, scope, and focus. The variation in countries rankings, which is often important to decision-makers, can be explained by using different methodologies, especially the different data collection techniques. These findings are aligned with (Davies, 2013; Susha et al., 2015) . Concepts measured by each assessment are deeply connected with the six Charter principles, as most commitments were mapped to specific questions of the input surveys and other data sources. 4.2.

RQ2 - How do OGD benchmarks measure the different aspects of the open data? All benchmark instruments cover the six Charter principles. However, each study's focus differs, as each benchmark was developed by different types of organizations with diverse goals, scope, and concepts. A clear overlap of the assessment of the technical aspects of the data and "data openness" exists.

The Open Data Barometer (ODB) measures the impact of open government data on socioeconomic outcomes. OBD measures all six Charter principles. A few items, such as commitments related to updating domestic laws or availability of high spatial disaggregation (of environmental pollution levels), are not covered (Brandusescu et al., 2018).

OURdata evaluates the capacity of the government to carry out OGD initiatives. For example, it measures if governments stimulate data reuse, train civil servants, and foster businesses and civil society awareness through the promotion of events. OURdata 2017 (Lafortune & Ubaldi, 2018) reports that the Index does not cover international knowledge sharing. It is challenging to capture the engagement of countries with international organizations in an index. OURdata 2017 does not assess the engagement with subnational levels of government and educational institutions (only acts at the national level). The Index needs to define measurements to assess whistle-blower protection. Lifecycle dataset management is hard to measure, as the concept of "retaining value" is ultimately subjective. The governments' lifecycle dataset management needs to be better understood. All six Charter principles are addressed. However, principles 2, 4, 5, and 6 are partially covered.

Until 2017, EODMA's focus was on the evaluation of the two technical dimensions, while the impact of Open Data was a secondary measurement. Data quality was not yet a concern. Thus, principle commitments such as if data is released in various formats or is easily discoverab le were not assessed in 2017 (Brandusescu et al., 2018). The 2018 EODMA was updated and now covers four dimensions: Open Data Policy, Open Data Portal, Open Data Impact, and Open Data Quality (Cecconi & Radu, 2018) . Its measurements are likely to change. The 2017 EODMA partially covers all six Charter principles.

GODI focuses on the publication of data. Its methodology assumes that open data is defined according to the Open Definition (OKFn, 2005) . It does not measure other common aspects of open data assessment, such as context, use, or impact. The Index does not cover data quality, which is a significant barrier to reuse (Janssen et al., 2012) . However, it measures aspects of "practical openness" like data findability that are part of the Charter principles (OKFn, 2019a) .

ODIN addresses Charter principles 2, 3, and 4, but not entirely. Of Charter principle 4, only P4.b "Ensure that open datasets include consistent core metadata and are made available in human and machine-readable formats" is partially surveyed. ODIN does not score policies. Thus it does not assess the Open by Default principle, although it only measures open datasets (Brandusescu et al., 2018). The five elements of openness (non-proprietary format, terms of use, metadata availability, download options, and machine-readable) used in ODIN match the principles of the Open Data definition (OKFn, 2005) and the Open Data Charter (Open Data Charter, 2013)

The analysis of the metrics described in ODM methodology (European Commission & Consortium of Collaborators, 2019) suggests that Charter principles 3 and 4 are covered, but not entirely. Like Open Data Inventory (ODIN), ODM does not survey policies, so it does not assess the Open by Default principle.

ODM, GODI, and ODIN only survey specific data; they do not assess data policies or promote a culture of openness. Their measurements focus on the technical aspect of the open data, such as accessibility, not on governance or impact. What differentiates these benchmark instruments is their scope or focus. All of them measure "data openness" however, each emphasizes different aspects of "openness."

OGD measurements focus on the supply side. Principles 5, For Improved Governance and Citizen Engagement, is measured by qualitative proxy indicators. The following question is an example: "if the government offers support for civil society organizations projects that focus on identifying policy solutions to challenges faced by marginalized communities using OGD." Principle 6, For Inclusive Development and Innovation, is measured by the number and quality of open data initiatives, business products, and services developed, educational programs, and research partnerships created.

The European Union, OECD, and the other organizations that developed the surveyed benchmarking endorse the Open Data Charter. Auditing assessments against the Charter principles provides evidence that these institutions "walk the talk." They use a consistent approach, based on the principles they endorse, to assess open data practices. Their measurements can be compared, and their results can be used to help improve data quality and conditions for sharing and reusing open data. There is a clear overlap in readiness and data assessments, as all six benchmark instruments measure these dimensions. Our findings are aligned with researchers and practitioners (Carrara et al., 2017; OECD, 2018; Vancauwenberghe, 2018) . GODI, ODIN, and ODM measurements focus on the data dimension. They measure concepts linked to openness, such as data quality and findability, which are prerequisites of open data use.

Nevertheless, the use and impact dimensions are barely addressed. EODMA, OURData, and ODB measurements include the use of data and the impact of OGD across the political, social, and economic areas. However, benchmark instruments use proxies such as the number of site visitors, the number of dataset views and downloads to gauge the use of OGD. Our findings are aligned with other researchers (Susha et al., 2015; Lämmerhirt et al., 2017 ; Vancauwenberghe, 2018) as the use and impact dimensions are still the weakest links of all assessments.

5. Conclusion and Future Research

This study was limited by the number of assessments studied. It focused on recent and international benchmarks in which measurements could be compared using the Open Data Chapter principles. To choose which OGD benchmark instrument best suits their purposes, practitioners, and decisionmakers should be aware of their differences and similarities. If they are interested in data, "openness" aspects should look into the Open Data Monitor, ODIN, and GODI as they emphasized the data dimension. EODMA offers the broadest OGD assessment as the data, readiness, impact, and use dimensions are addressed. ODB and OurData stay in the middle, with a strong focus on readiness and data. However, ODB assesses the impact. OurData evaluates usefulness. All benchmarks consistently use the Charter principles to guide their measurements of open data initiatives.

The impact and use of Open Government Data are accounted for in only half of the benchmark instruments. This finding suggests a gap that can be explored in future research. We will investigate OGD prerequisites to determine how they can be used to create practical measurements and indicators of the actual use and usefulness of the data. Another venue is to investigate whether the relation between the Charter principles adoption influences the use and usefulness of OGD. -Kunigami, A., Pérez, A. R., Ubaldi, B., Iglesias, C., Ngounou, C. M., Onerhime, E., Zapata, E., Swanson, E., Vaughan, F., Vollers, H., & Crowell, J. (2018). Open Data Charter Measurement Guide (p. 41). Open Data Charter, Open Knowledge International and World Web Foundation.

Caplan, R., Davies, T., Wadud, A., Verhulst, S., Alonso, J. M., & Farhan, H. (2014). Towards common methods for assessing open data: Workshop report & draft framework. W3C Foundation and GovLab. http://opendataresearch.org/sites/default/files/posts/Common%20Assessment%20Workshop%20Re port.pdf Open Data Charter. (2013). Principles International Open Data Charter.

https://opendatacharter.net/principles/ Open Data Watch. (2019). ODIN - Open Data Inventory. https://odin.opendatawatch.com/ About the Authors Ilka Kawahita Ilka Kawashita is currently a Ph.D. student at the University of Minho. She works as a public servant at the Ministry of Economy in Brazil (on leave). She is an associate faculty at the University of Phoenix, USA. After obtaining a bachelor's degree in Mechanical Engineering from the University of Brasilia, Brazil, she obtained a master's degree in Information Systems from the University of Montreal, Canada. For more information, please access https://www.cienciavitae.pt/pt/6B10-3BFE-3BDA.

Ana Alice Baptista Ana Alice Baptista is a professor at the Information Systems Department and a researcher at ALGORITMI Center, both at the University of Minho, Portugal. She was chair of the Dublin Core Metadata Initiative (DCMI). She participated as a PI and as a regular researcher in several R&D projects, and she is currently the PI of IViSSEM and EMPOWER-SSE projects. Her main areas of interest include Metadata, Linked Data, and the Open Movement, both under their technological and social perspectives. For more information, please access https://www.cienciavitae.pt/201D-B2FC-E126.

Delfina Soares Delfina Soares is the Head of the United Nations University Operating Unit on Policy-Driven Electronic Governance (UNU-EGOV). She is also a professor at the Department of Information Systems and researcher at the ALGORITMI Center at the University of Minho (currently on leave). Over her career, Delfina has supervised multiple research projects in the area of IST in governance, with a focus on the use of IST to promote the transformation and modernization of States' governance activities. Delfina has also coordinated and collaborated in advisory, consultancy, and capacity-building projects with government entities in different countries. For more information, please access https://www.cienciavitae.pt/en/701C-3C63-E400.

References

Alexopoulos , C. , Spiliotopoulou , L. , & Charalabidis , Y. ( 2013 ). Open data movement in Greece: A case study on open government data sources . Proceedings of the 17th Panhellenic Conference on Informatics , 279 286 .

Attard , J. , Orlandi , F. , & Auer , S. ( 2016 ). Data driven governments: Creating value through open government data. In Transactions on Large-Scale Data- and Knowledge-Centered Systems XXVII (pp. 84 110 ). Springer.

Atz , U. , Heath , T. , & Fawcett , J. ( 2015 ). Benchmarking open data automatically ( Technical Report ADI-TR-2015- 000) . Open Data Institue.

Bannister , F. ( 2007 ). The curse of the benchmark: An assessment of the validity and value of e-government comparisons . International Review of Administrative Sciences , 73 ( 2 ), 171 188 . https://doi.org/10.1177/0020852307077959

Carrara , W. , Radu , C. , & Vollers , H. ( 2017 ). Open Data Maturity in Europe 2017 (No. 3; EDP Landscaping Insight Report) . European Data Portal . https://www.europeandataportal.eu/en/dashboard#2017

Cecconi , G. , & Radu , C. ( 2018 ). Open Data Maturity in Europe 2018 (No. 4). European Data Portal . https://www.europeandataportal.eu/en/dashboard#2018

Davies , T. ( 2013 ). Notes on open government data evaluation and assessment frameworks . http://www.opendataimpacts.net/ 2013 /02/506/, http://www.opendataimpacts.net/ 2013 /02/506/

Davies , T. , & Bawa , Z. A. ( 2012 ). The Promises and Perils of Open Government Data (OGD). 8 .

European Commission , & Consortium of Collaborators. ( 2019 ). OpenDataMonitor. https://opendatamonitor.eu/frontend/web/index.php?r=dashboard%2Findex

Hitz-Gamper , B. S. , Neumann , O. , & Stürmer , M. ( 2019 ). Balancing control, usability and visibility of linked open government data to create public value . International Journal of Public Sector Management , 32 ( 5 ), 451 466 . https://doi.org/10.1108/IJPSM-02-2018-0062

Janssen , M. , Charalabidis , Y. , & Zuiderwijk , A. ( 2012 ). Benefits, adoption barriers and myths of open data and open government . Information Systems Management , 29 ( 4 ), 258 268 .

Jetzek , T. ( 2015 ). The sustainable value of open government data. Uncovering the Generative Mechanisms of Open Data through a Mixed Methods Approach , Copenhagen, Copenhagen Business School.

Johnson , P. , & Robinson , P. ( 2014 ). Civic Hackathons: Innovation, Procurement, or Civic Engagement? REVIEW OF POLICY RESEARCH , 31 ( 4 ), 349 357 . https://doi.org/10.1111/ropr.12074

Kalampokis , E. , Tambouris , E. , & Tarabanis , K. ( 2011 ). Open Government Data: A Stage Model . In M. Janssen, H. J. Scholl , M. A. Wimmer , & Y. Tan (Eds.), Electronic Government (Vol. 6846 , pp. 235 246 ). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3- 642 -22878-0_ 20

Klein , R. H. , Klein , D. C. B. , & Luciano , E. M. ( 2018 ). Open Government Data: Concepts , Approaches And Dimensions Over Time . 18 ( 49 ), 21 .

Lafortune , G. , & Ubaldi , B. ( 2018 ). OECD 2017 OURdata Index: Methodology and Results (OECD Working Papers on Public Governance No . 30 ). https://doi.org/10.1787/2807d3c8-en

Lämmerhirt , D. , Montiel , O. , & Rubinstein , M. ( 2017 ). The State of Open Government Data in 2017 . 15 .

Lee , G. , & Kwak , Y. H. ( 2012 ). An Open Government Maturity Model for social media-based public engagement . Government Information Quarterly , 29 ( 4 ), 492 503 . https://doi.org/10.1016/j.giq. 2012 . 06 .001

Maheshwari , D. , & Janssen , M. ( 2013 ). Measurement and benchmarking foundations: Providing support to organizations in their development and growth using dashboards . Government Information Quarterly , 30 , S83 S93. https://doi.org/10.1016/j.giq. 2012 . 11 .002

OECD. ( 2018 ). Open Government Data Report: Enhancing Policy Maturity for Sustainable Impact . OECD. https://doi.org/10.1787/9789264305847-en

OECD. ( 2019 ). Open Government Data OECD . http://www.oecd.org/gov/digital-government/opengovernment-data.htm

OKFn. ( 2005 ). The Open Definition Open Definition Knowledge . http://opendefinition.org/

OKFn. ( 2019a ). Global Open Data Index . https://index.okfn.org/

OKFn. ( 2019b ). Request a Local Open Data Census . Google Docs. https://docs.google.com/forms/d/e/1FAIpQLScaczo-A4hdiNik44YmNXALBvi82h0X_ dAjR4djyyDCuUylA/viewform?embedded=true&usp=embed_facebook

Susha , I. , Zuiderwijk , A. , Janssen , M. , & Grönlund , Å. ( 2015 ). Benchmarks for Evaluating the Progress of Open Data Adoption: Usage, Limitations, and Lessons Learned . Social Science Computer Review, 33 ( 5 ), 613 630 . https://doi.org/10.1177/0894439314560852

Ubaldi , B. ( 2013 ). Open Government Data: Towards Empirical Analysis of Open Government Data Initiatives (OECD Working Papers on Public Governance No . 22 ). https://doi.org/10.1787/5k46bj4f03s7-en

Vancauwenberghe , G. ( 2018 ). Assessing Open Data . In B. van Loenen, G. Vancauwenberghe , & J. Crompvoets (Eds.), Open Data Exposed (pp. 149 172 ). T.M.C. Asser Press. https://doi.org/10.1007/ 978 - 94-6265-261- 3 _ 8

W3C. ( 2019 ). Open Data Barometer . https://opendatabarometer.org/?_year= 2017 &indicator=ODB

Zuiderwijk , A. , & Janssen , M. ( 2014 ). Barriers and development directions for the publication and usage of open data: A socio-technical view . Public Administration and Information Technology , 4 , 115 135 . https://doi.org/10.1007/978-1- 4614 -9563- 5 _ 8