Competency model for open data literacy in professional learning within the context of Open Government Data (OGD)

Abstract

Research on Open Government Data (OGD) use reveals that the data is not being used as expected. Many governments have opened their data but lack the development of the capacities required for OGD usage. There is a need of having frameworks of reference for open data literacy (ODL). The initial screening of the literature uncovers there is a dearth of systemic interventions to develop ODL, and there is limited research on what works. This research will focus on understanding the contexts and barriers of OGD use to study the role of technical and critical data literacy concerning the current low usage. It will map practices to develop the ODL and expert's knowledge to create an instrument that could be applied for the diagnostic baseline of ODL. Also, it will explore the applicability of such an instrument for the self-analysis and external learning recognition on ODL. This model can be used in the sphere of government, universities, and business, to assess the level of competencies in OGD usage in their employees or students, and to identify ODL competencies' gaps in a different context of professional practice.

Keywords

• 1 Open government data
• open data usage
• open data literacy
• critical data literacy
• technical data literacy
• professional learning

1. Introduction

The open data movement is an emerging political and socio-economic phenomenon that promises to promote civic engagement and drive public sector innovations in various areas of public life [ 1 ]. The Open Data Handbook (https://opendatahandbook.org/guide/es/whatis-open-data/) defines open data as data that can be freely used, reused, and redistributed by anyone, subject only, at most, to the requirement to attribute and share equally.

The open data initiative initially arises from the universal declaration on human rights of 1948, where the right to information is already mentioned in Art.19 (https://www.un.org/en/about-us/universaldeclaration-of-human-rights). Along the same lines, the Open Knowledge Foundation, established in 2004, is recognized for its mission of “a just, free, and open future, where all non-personal information is open and free for all to use”.

Open data has great potential for use, specifically, Open Government Data (OGD) for the development of public policies, democratic dialogue, entrepreneurship, among others [ 2 ].

There are many benefits expected with the opening of government data to citizens and companies, such as improving transparency, reliability in administration, promoting public participation and public-private collaboration, as well as revitalizing the economy, with the recognition that public data is assets of people. [ 3 ].

However, while many open databases are available, only a limited number of them are used [ 2 ], their active use is still limited because of issues with data quality and linkage [ 4 ]. In addition, for the use of open data, users require a framework of open data literacy skills essential for advanced use of data in each context. Raffaghelli [ 6 ] has stated that reference frameworks are needed for educators’ data literacy since after reviewing the literature corpus it was detected that data literacy connected to OGD is never considered in the adult´s data literacy educational frameworks even though it is a crucial dimension of educators’ professional competence.

The expected outcomes and significance of this Ph.D. research are to identify a set of skills and knowledge required to perform in an advanced level of usage of open government data, thus finding the dimensions of ODL. As well as the development and deployment of a measurement instrument to assess the level of ODL capacities for the quantification of progress on ODL.

2. Justification

The relevance of this research relates the need of having a set of skills of reference for data literacy overall and for open data literacy, specifically.

Data literacy, as a research topic, stems from numeracy and statistical literacy. However, the most recent developments connect data literacy with data-driven digital environments [ 6 ]. The research tries to identify the needed skills and knowledge concerning professionals and adults in relation to open data. Open data is indeed a digital resource that can both trigger learning or be a product of formal, non-formal and informal learning. In this regard, Open Data can be deemed part of technological environments and has the potential to enhance learning.

As it appears from our initial screening of the literature, there is a dearth of systemic interventions to develop data literacy, and there is limited research on what works, as initiatives face funding and organizational challenges limit scaling up training [ 7 ].

According to Khayyat and Bannister [ 8 ], OGD field experiments such as hackathons and competitions continue to be conducted, but there has been no systematic research on the factors that contribute to a vibrant and sustainable ecosystem of co-creation with civil communities.

This research is intended to contribute to creating an instrument that allows the identification and assessment of open data literacy levels of knowledge. This tool can be used in the sphere of government, business and in universities, to recognize and measure the level of competences in open data usage in their employees or students in different contexts of professional practice.

3. Research problem

In the field of open government data, it is known that an effort was made to open data in many governments, but not so much has been done for the development of the necessary capacities for the exploitation or the optimal use of the same for the taking of government decisions. The World Bank recognizes that its current support models have focused more on data production and exchange than on building capacity to use data [ 9 ]. Furthermore, within the models developed for capacity building, only a few of them have been tested at scale [ 10 ]. For example, hackathons and local training activities within international cooperation such as Open Data Day (https://opendataday.org/). In the same line, one study of the use of the public sector data analytics in The Netherlands shows that the use of public sector data analytics requires developing organizational capabilities to ensure effective use, foster collaboration, and scale-up [ 11 ]. Due to the problem of the low use of open data, this research focuses on studying the open data literacy required for the effective and more frequent use of these by interested sectors including citizens.

3.1. Theoretical antecedents and empirical

Open Government Data is characterized by being data and information produced or commissioned by public bodies [ 12 ]. Broadly speaking, the OECD (https://www.oecd.org/) defines Open Government Data as "a philosophy, and increasingly a set of policies, that promote transparency, accountability and value creation by making government data available to all."

Citizens’ participation in open government can improve their perceptions towards government as a transparent, participatory, and collaborative institution and such participation of citizens increases operational capacity and trust [ 13 ]. It promises other benefits such as greater accountability and increased public participation, but few of these initiatives have been evaluated in terms of their implementation and results.[ 14 ]. And while many open databases are available, only a limited number of them are used [ 2 ].

A decade has passed since the first International Data Conference (https://opendatacon.org/), which is designed to bring the global open data community together to learn, share, plan and collaborate on the future of open data and data for development. Although efforts have been made to open government data in many countries of the world, there has not been a similar effort to develop the necessary capacities for the use of data by citizenship.

Publishing OGD can lead to innovation since it allows external parties to access, explore and handle OGD, which in turn will help to develop and build useful services, products, and applications for the benefit of society [ 15 ]. However, Bonina & Eaton [ 16 ] state in their research on the governance of the ecosystems of Government Open Data (OGD) platforms, that after a decade of open data initiatives few economic and social benefits have been achieved due to incomplete or lowquality data, mismatches between the data that are needed and those that are published, and the existence of technical barriers to participation, besides lack of skills and training of users.

The use of open data, "is the activity that a person or organization performs to see, understand, analyze, visualize or in other ways use a set of data that a government organization has provided to the public" [ 2 ]. This definition of use can be identified as technical literacy in open data, delimited in this research as the competencies, knowledge, and skills necessary to download, clean, order, analyze and interpret open data in a specific context. Just publishing raw data, may not result in transparency, as without formatting the data may not be easy for most people to understand and use [ 17 ]. Some authors suggest helping users using visuals, “geovisualizing open data seems the next logical step to put open data in the hands of citizens” [ 18 ]. Since it is required the development of competencies for the effective use of public sector data analytics in the organizations [ 19 ]. Finally, Kassen [ 20 ] states that the reuse or processing of open data to develop third-party applications and projects requires skilled enthusiasts and tech-savvy citizens who are willing to contribute their time, knowledge and expertise to the creation or cocreation of products based on open data.

However, this technical definition of open data literacy focuses on technical skills. Another conception, critical data literacy, refers to the skills, knowledge, and attitudes to review the meaning of concepts, visualizations and operations carried out with the data that can put user groups at risk of inequity or ethical aspects.

“The value of openness in the fight against inequality should be emphasized, the equity should be placed at the center of data analysis, and practitioners should actively promote reflection on inclusion gaps in data and the harm those gaps can bring” [ 7 ].

“Data literacy is not just about open data, but open data can be an invaluable asset for inclusive and empowering data literacy development programs” [ 10 ]. Identifying the open data literacy framework and user skill gaps is crucial to understanding the types of professional learning contexts in which they can be developed. Montes and Slater [ 7 ] claim that the lack of a coherent and generally accepted definition of data literacy and requisite skill set leaves us without a real quantification of progress on open data literacy.

Theoretical frameworks refer to the critical theory and the socio-technical theory [ 21 ], applied to the studies on digital data, datadriven practices and their impact on society and education. Indeed, data literacy has become an essential part of digital competence as outlined in the DigComp Framework 2.1 [ 22 ]. Also, a critical approach to data is needed in an increasingly contested approach to the developments of data-driven practices [ 23 ].

The Data Skills Framework developed by the Open Data Institute (ODI) (https://theodi.org/article/data-skillsframework) is an initial reference for the technical data literacy approach. Also, in the Digital Competence Framework released by the European Commission (https://op.europa.eu/en/home), the concept of data literacy was introduced in 2017 alongside the information literacy dimension as an ability to search, read, and interpret data in several daily and academic contexts of communication [ 24 ].

On the other hand, critical data literacy will be studied in the light of the Data feminist principles developed in the book Data Feminism, which presents a new way of thinking about data science and data ethics, which is grounded in intersectional feminist thought. It debates about power, and how those differentials of power can be challenged and changed [ 25 ]. Likewise, other texts with a critical approach to data will be used as a frame of reference, such as Taylor [ 26 ] where the author posits that “just as an idea of justice is needed in order to establish the rule of law, an idea of data justice – fairness in the way people are made visible, represented and treated as a result of their production of digital data – is necessary to determine ethical paths through a datafying world”.

Further frameworks to be studied are Markham (https://futuremaking.space/criticalpedagogy-data-literacy/) who characterizes critical pedagogy as a vital part of building data literacy. The author identifies it as a research stance that can challenge quantification, datafication, and computational logic and it moves beyond the level of data critique to social action in response to datafication. Other approaches will be considered such as Raffaghelli [ 27 ], where the author provides a conceptual scheme to address further pedagogical reflection and practice to support social justice against datafication.

4. Aim of the research

The aim of this research is to identify a model of the open data literacy that professional learners must acquire to operate in advanced contexts of data usage. Once detected through the model, such literacy could be developed through different types of learning contexts. Moreover, the model could address professional learning recognition.

The research aims at developing an instrument that allows recognition and assessment of several levels of competence in open data literacy. Therefore, the stage of skills and knowledge within a context of usage of open data as digital resources.

This is an original purpose since most studies analyze data literacy centered in technical procedures relating data science abilities [ 6 ] but miss the political contexts and the critical approach to data [ 17 ].

This instrument can be used in the sphere of government, business, and universities, to assess and recognize the level of competences in open data usage in their employees or students. Also, to identify and understand the OGD competences’ gap in different contexts of professional practice.

Specific Objectives of this research: 1. To analyze current academic literature review to uncover the issues preventing open data usage, and within them, the role played by data literacy. 2. To identify what data literacy educational practices are currently available on the web there will be applied a mapping procedure of such pedagogical practices. 3. To validate such open data literacy dimensions by a panel of subject matter experts’ interviews. 4. To build and develop the measurement instrument. 5. To theoretically validate the instrument by determining the validity of the tool through the Delphy method. 6. To empirically validate the instrument through Circulation of the instrument as a survey, the estimation of Cronbach's alpha statistic and the confirmatory factor analysis. 7. To test the instrument in the context of ecological learning training by the application of it to the participants, as well as the application of a statistical analysis of the results to determine a diagnostic baseline in Open Data literacy and sensitivity to competence change.

5. Research hypothesis

The evaluation and recognition of skills and knowledge connected to open data usage could be supported by an open data literacy tool.

6. Research questions

In this context, the following research questions have been posed:

RQ1 What are the contexts of use and learning based on OGD? RQ2 What are the barriers that prevent the use of open data, and within those barriers, what role does technical and critical data literacy in open data play as one of the causes of the low use of OGD? RQ3 What are the current pedagogical practices available that can be used to develop the ODL required to make use of OGD? RQ4 What is the set of skills needed in OGD practice contexts required for professional learning? RQ5 How should be configured a measurement instrument that could be applied for the diagnostic baseline of ODL? RQ6 What is the applicability of such instrument for the self-analysis and/or external learning recognition on Open data literacy?

7. Methods

7.1. Design of research

To pursue the objective of this study, a mixed methods research approach will be applied. The design implies three phases to cover the objectives.

The first phase will be devoted to the analysis of the problem and the existing corpus of research. To this regard, a systematic review of the literature will be undertaken based on the methodological workflow called PRISMA [ 27 ] and it is a transparent report of systematic reviews and meta-analyzes. This method attempts to control for investigator bias in data collection and analysis [ 28 ].

The main PRISMA steps that will be carried out in this research are: 1. Select scientific databases, 2. Search the databases with keywords of interest for several articles, 3. Select articles using predefined exclusion criteria based on in the research objectives. 4. Analyze the selected articles by reading them in full.

The systematic review of the literature will be integrated with an analysis of existing pedagogical practices (benchmarking study/desk research), which will support the analysis of type of competences focused and trained as part of an underlying ODL approach.

Based on this selection, quantitative analysis methods will be applied that allow better identification of emerging issues and problems in a general and specific way, with respect to the research questions posed.

Also, an exploratory research, mapping and gap analysis is going to be performed to identify what data literacy educational practices are currently available in the web.

Finally, there will be a panel of experts interviews to identify dimensions as a base to the development and operationalization of the measurement instrument.

The second phase will be devoted to the development of a self-reported measurement instrument, over the basis of the theoretical assumptions emerging from the literature review.

After identification of the dimensions, from the theoretical frameworks review, for the theoretical validation, a Delphi study will be conducted. The panel of experts is going to be used for building the open data literacy set of skills and knowledge and the Delphi method to validate the measurement instrument. The Delphi method is defined as “a panel communication technique by which researchers collect expert opinions, enable experts to communicate anonymously with one another and then explore the underlying information collected” [ 29 ].

The panel of experts will be invited to review the instrument through the technique of interviews, developed in two stages. Therefore, the results will be assembled, and a second cycle of consultation will be enacted. [ 30 ]. A measurement instrument is going to be designed and created to assess open data literacy in the contexts of OGD. As for the empirical validation of the instrument, it is going to be circulated as a questionnaire to professionals working in either public administration or industry with a stratified sampling design by sector.

The study is going to use the exploratory, descriptive, and explicative approaches in its different research phases.

Finally, the third phase will be devoted to the instruments’ consolidation and further validation in ecological training contexts, the developed scale will be applied in specific educational context to analyze the applicability to: 1. Evaluate the development of ODL in ecological training context. 2. Self-assess ODL in formal (undergraduate) and non-formal/informal (professional) learning contexts. 3. Recognize ODL in professional contexts.

7.2. Sample

7.2.1. First phase

The sample units will be the articles selected for the literature analysis. For the selection of articles, this research will apply the PRISMA method for the systematic literature review. The detail of what will be done in each step, for the selection of a sample of articles, is detailed below: b. The research focuses on the type of open data and applications (Discipline, Type of Open Data, Applications of open data) c. Types of learning generated and barriers of use (Types of learning generated using open data, Barriers that prevent the use of open data)

Finally, after consolidating the categories, the authors will analyze 10% of the total set of articles and the agreement between evaluators will be estimated using Cohen's Kappa statistic (https://www.statisticshowto.com/cohenskappa-statistic/). A kappa higher than 0.60 can be considered a good agreement.

7.2.2. Second phase

In the initial task relating to the Panel of experts' interviews and Delphi study, the expert selection will be carried out in a non-random manner based on their expertise on the phenomenon being studied [ 31 ]. In this case are OGD subject matter experts. The sample size for the interviews and the Delphi study will be determined by the saturation point with a minimum of seven qualitative interviews to subject matter experts, active OGD users.

The target population is made up by 1. Quantitative units of analysis are current and potential OGD users around the globe that are available to fill out the instrument, 2. Qualitative units of analysis: are adult professionals identified as subject matter experts, and frequent users of OGD and ORD. Specifically, to test the questionnaire and to get data to validate and measure the reliability of the questions. The experts are professionals who have high experience on OGD usage. Professionals are current or potential users of OGD.

The sample size estimated for this study is 196 units of analysis, therefore 196 OGD users. It assumes a confidence level of 95%, a maximum error of 7% and a variance of 0.25. It assumes a big target population of ODG users.

Since currently there isn´t defined a sampling frame of the OGD user´s population, the type of sampling to be used in this study is non-probabilistic sampling defined as “a sampling technique in which some units of the population have zero chance of selection or where the probability of selection cannot be accurately determined” [ 31 ].

Measures of construct reliability and validity will be implemented, over the basis of classical test theory [ 32 ], [ 33 ].

7.2.3. Third phase

Two groups will be tested: 1. A group with at least 20 workers with none to high experience on the usage of OGD in both public and industry settings, for selfassessment and recognition of competences purpose. 2. A group of at least 20 undergraduate students in several disciplines, for selfassessment purposes, will be experimentally exposed or not exposed to OGD.

7.3. Data collection and instruments techniques

For data collection the research will adopt a mixed methods approach. A desk research approach will be applied to the first phase will adopt documental analysis and classification of pedagogical practices through a deductive scheme of analysis. Also, a synthesis report will be performed to identify ODL set of skills to define its dimensions. Then, in the second phase, a qualitative approach based on in-depth interviews will be adopted for the identification of dimensions and the instrument design and Delphi study for theoretical validation.

On the other hand, a quantitative approach will be adopted both for the instrument empirical validation (end of the second phase), and for the instrument testing (third phase). An electronic form with the instrument will be implemented and circulated for data collection. In the case of the third phase, there will also be a qualitative data collection and analysis. Indeed, the instrument will be embedded in a learning management system and the results will be made available for the respondents to react, reflect, and discuss upon them as the formative impact of the instrument implementation. 7.4.

Procedure

The procedure is going to be developed in three phases, as explained before, and it is summarized in table 1, which is located at Appendix 1. The summary table includes the phase, objective that is going to be pursued, the activity or task to be performed, the method to be applicable for pursuing the objective and the expected output or result for each task.

8. Current status and results

8.1. Systematic review literature of

In short, the PRISMA systematic review of literature reveals that the use of OGD seems to depend largely on the necessary technical and critical skills. Although there are many technological, structural, organizational, and cultural barriers, the skills of the stakeholders to use and obtain the expected benefits of open data is an obstacle that requires consideration.

The analysis of the corpus of literature uncovers that the lack of open data literacy arises as the main barrier, particularly in social sciences, OGD and governance. Our results reinforce the importance of data literacy, this is coherent with Matheus & Janssen [ 17 ] who imply that the same data that creates a higher level of transparency for the expert, creates less for someone with lack of knowledge of how to use it. re being considered.

Overall, what can be inferred from our analysis is that literacy opportunities are mostly technical; and that engagement with open data, when occurs, produces meaningful learning.

However, our analysis could not cover to what extent the collaborative and co-creative synergies between stakeholders can lead to innovation and governance. These are aspects that remain to be studied towards a holistic and critical data literacy.

Finally, the research outputs at this stage of the PhD are part of a literature review research, but the following phases relate online observations, interviews, the construction of an instrument based on a survey and the empirical validation in two phases.

9. Limitations of the study

This research is at a very early stage. In any case, the limitations foreseen relate a) the documented difficulties in analyzing adult learning and identifying patterns of learning activity (most learners follow informal learning pathways); b) the complex approach that the empirical validation will require, in terms of participants' recruitment; c) the complexity of identifying experts and contexts for empirical work.

In any case, risk management strategies are being considered. 10. Data management and ethics

This research plan was approved by the ethics committee of the UOC. For the approval of the ethical form, it was required to explain details about data curation policies, informed consent, how to proceed with the database once the study is concluded, etc. The data will be processed exclusively for the purposes for which they have been collected and for the time strictly necessary to fulfill the purposes for which they will be collected. 11. Acknowledgements

We would like to acknowledge UOC’s doctoral school on Education and ICT program, which offers the courses of social research design, Research methods, Directed Research subjects, as well as, workshops, and training sessions, during which we have been able to develop this research proposal. 12. References 2531–2560, 2018, 10.4236/psych.2018.911145. doi:

Analysis of the problem and the existing corpus of research and to map current pedagogical

practices

To develop and validate the measurement instrument

Activity

Academic literature review

Mapping of

pedagogical practices

Report of skills and knowledge required for open data literacy

To establish and validate the dimensions of ODL construct

Instrument development

To validate

dimensions of ODL by experts

Phase

3

To use the instrument in a context of ecological learning

Testing the

instrument in a context of ecological learning training

Circulation of the

instrument as a survey to a population of at least 196

persons

Cronbach's Alpha

Estimation and

Confirmatory Factor

Analysis

Report of results of the

instrument that includes baseline of diagnosis and sensitivity to change of competence.

References

1. Selection of Databases. SCOPUS, DOAJ and WOS will be selected to perform the bibliographic search .

2. Selection of articles using keywords. It is of interest to this research to know characteristics related to the use of open data, as well as to know aspects that prevent its use. Therefore, the following keywords will be searched in the selected databases:

3. SCOPUS and DOAJ: (open AND data) AND (government) AND (us *)

4. WOS : (open AND data) AND (government) AND (usa *)

5. Screening of articles abstracts will be read, and the following exclusion criteria will be used: a. Date before 2016, to have the latest knowledge on the topic of interest b . DOI absence c. Other Open Data issues that are not OGD d. It is not an article or review e. Not in English f. Related to OGD but not its use g . Not available

6. Analysis of the articles after reading them in full: each one will be read completely and will be coded and classified in variables defined in the codebook, which will be defined by the authors based on the objectives of the research, to generate a database of articles that will later be analyzed quantitatively to obtain their respective findings. Specifically, the articles will be coded and classified in the following categories: a. The identity of the research (Authors, Title, Year, Title of the source, No. of Citations, DOI, Type of Document, Abstract of the article , Author's keywords)

[1] M. Kassen , “ Understanding motivations of citizens to reuse open data: open government data as a philanthropic movement , ” Innovation , vol. 00 , no. 00 , pp. 1 - 27 , 2020 , doi: 10.1080/14479338. 2020 . 1738940 .

[2] A. Zuiderwijk , M. Janssen , and Y. K. Dwivedi , “ Acceptance and use predictors of open data technologies: Drawing upon the unified theory of acceptance and use of technology,” Gov. Inf. Q. , vol. 32 , no. 4 , pp. 429 - 440 , Oct. 2015 , doi: 10.1016/j.giq. 2015 . 09 .005.

[3] T. Sumitomo and N. Koshizuka , “ Progress and Initiatives for Open Data Policy in Japan,” Computer (Long . Beach. Calif)., vol. 51 , no. 12 , pp. 14 - 23 , Dec. 2018 , doi: 10.1109/ MC . 2018 . 2879993 .

[4] K. Haklae , “ Interlinking Open Government Data in Korea using Administrative District Knowledge Graph,” J. Inf. Sci. Theory Pract ., vol. 6 , pp. 18 - 30 , 2018 , doi: 10.1633/JISTaP. 2018 . 6 . 1 .2.

[5] J. E. Raffaghelli , “ Developing a framework for educators' data literacy in the European context: Proposal, Implications and debate .,” in International Conference on Education and New Learning Technologies EDULEARN , Jul . 2019 , pp. 10520 - 10530 , doi: 10.21125/edulearn. 2019 . 2655 .

[6] J. E. Raffaghelli , “ Developing a framework for educators' data literacy in the European context: Proposal, Implications and debate .” in EDULEARN19 Proceedings , 2019 , vol. 1 , doi: 10.21125/edulearn. 2019 . 2655 .

[7] M. G. Montes and D. Slater , “ Issues in open data: Data literacy . In T. Davies, S. Walker , M. Rubinstein , & F. Perini (Eds.), ” in The state of open data: Histories and horizons , Cape Town and Ottawa: African Minds and International Development Research Centre , 2019 , pp. 274 - 286 .

[8] M. Khayyat and F. Bannister , “ Towards a model for facilitating and enabling cocreation using open government data , ” Inf. Polity , vol. 22 , no. 4 , pp. 211 - 231 , Nov. 2017 , doi: 10.3233/IP-170406.

[9] World Bank , “ Data for Development An Evaluation of World Bank Support for Data and Statistical Capacity ,” 2017 . Accessed: Apr. 19 , 2021 . [Online]. Available: www.worldbank.org.

[10] T. Davies , S. Walker , M. Rubinstein , and F. Perini , The State of Open Data: Histories and Horizons . Cape Town and Ottawa: African Minds and International Development Research Centre , 2019 .

[11] Van Veenstra and Gromme, “ (6) (PDF) The Use of Public Sector Data Analytics in The Netherlands ,” Sep. 2020 . https://www.researchgate.net/publication/ 344202992_The_ Use_of_Public_Sector_ Data_Analytics_in_The_Netherlands (accessed Jan . 03 , 2021 ).

[12] B. Ubaldi , “ Open Government Data: Towards Empirical Analysis of Open Government Data Initiatives,” OECD Work . Pap. Public Gov., vol. NO. 22 , no. 22 , p. 61 , 2013 , doi: 10.1787/5k46bj4f03s7-en.

[13] L. Schmidthuber , F. Piller , M. Bogers , and D. Hilgers , “ Citizen participation in public administration: investigating open government for social innovation ,” 2019 .

[14] R. Sandoval-Almazan and J. R. GilGarcia , “ Toward an integrative assessment of open government: Proposing conceptual lenses and practical components,” J. Organ. Comput. Electron. Commer. , vol. 26 , no. 1-2 , pp. 170 - 192 , Apr. 2016 , doi: 10.1080/10919392. 2015 . 1125190 .

[15] M. Saleh Altayar , “ Motivations for open data adoption: An institutional theory perspective ,” 2018 , doi: 10.1016/j.giq. 2018 . 09 .006.

[16] C. Bonina and B. Eaton , “ Cultivating open government data platform ecosystems through governance: Lessons from Buenos Aires, Mexico City and Montevideo,” Gov. Inf. Q. , vol. 37 , no. 3 , p. 101479 , Jul . 2020 , doi: 10.1016/j.giq. 2020 . 101479 .

[17] R. Matheus , M. Janssen , and D. Maheshwari , “ Data science empowering the public: Data-driven dashboards for transparent and accountable decisionmaking in smart cities,” Gov. Inf. Q. , vol. 37 , no. 3 , p. 101284 , Jul . 2020 , doi: 10.1016/j.giq. 2018 . 01 .006.

[18] A. Degbelo , J. Wissing , and T. Kauppinen, “ A comparison of geovisualizations and data tables for transparency enablement in the open government data landscape,” Int. J. Electron . Gov. Res. , vol. 14 , no. 4 , pp. 39 - 64 , Oct. 2018 , doi: 10.4018/IJEGR.2018100104.

[19] A. F. van Veenstra , F. Grommé , and S. Djafari , “ The use of public sector data analytics in the Netherlands,” Transform . Gov. People, Process Policy, 2020 , doi: 10.1108/TG-09-2019-0095.

[20] M. Kassen , “ Adopting and managing open data,” Aslib J. Inf . Manag., vol. 70 , no. 5 , pp. 518 - 537 , Sep. 2018 , doi: 10.1108/AJIM-11-2017-0250.

[21] T. Fenwick , M. Nerland , and K. Jensen , “ Sociomaterial approaches to conceptualising professional learning and practice , ” J. Educ. Work , vol. 25 , no. 1 , pp. 1 - 13 , Feb. 2012 , doi: 10.1080/13639080. 2012 . 644901 .

[22] S. Carretero , R. Vuorikari , and Y. Punie , “ The Digital Competence Framework for Citizens With Eight,” Publ. Off. Eur. Union , no. May, p. 48 , 2017 , doi: 10.2760/38842.

[23] B. Williamson , “ Journal of Education Policy Digital education governance: data visualization, predictive analytics, and 'real-time' policy instruments ,” 2015 , doi: 10.1080/02680939. 2015 . 1035758 .

[24] J. E. Raffaghelli , “ Is data literacy a catalyst of social justice? A response from nine data literacy initiatives in higher education,” Educ . Sci., vol. 10 , no. 9 , 2020 , doi: 10.3390/educsci10090233.

[25] C. D'Ignazio and L. F. Klein , Data Feminism . 2020 .

[26] L. Taylor , “ What is data justice? The case for connecting digital rights and freedoms globally , ” Big Data & Society JulyDecember 2017 , 2017 . https://pure.uvt.nl/ws/portalfiles/portal/26 363266/taylor_17_ what_is_data_justice.p df (accessed May 09, 2021 ).

[27] A. D. Moher D , Liberati A , Tetzlaff J , “ PRISMA 2009 Flow Diagram,” The PRISMA statement , vol. 6 . 2009 .

[28] M. Petticrew and H. Roberts , Systematic Reviews in the Social Sciences: A Practical Guide . 2006 .

[29] Y.-F. Yeh , Y.-S. Hsu , H. - K. Wu , F.-K. Hwang , and T.-C. Lin, “ Developing and validating technological pedagogical content knowledge-practical (TPACKpractical) through the Delphi survey technique,” Br. J. Educ . Technol., vol. 45 , no. 4 , pp. 707 - 722 , 2014 , [Online]. Available: https://www.learntechlib.org/p/148366/.

[30] M. Adler and E. Ziglio , Gazing into the oracle: the Delphi method and its application to social policy and public health . Jessica Kingsley Publishers , 1996 .

[31] A. Bhattacherjee , Social Science Research: Principles, Methods, and Practices, 3rd ed. Textbooks Collection., 2012 .

[32] P. Mair , Modern Psychometrics with R. Cham: Springer International Publishing, 2018 .

[33] T. A. Kyriazos and A. Stalikas , “Applied Psychometrics: The Steps of Scale Development and Standardization Process,” Psychology, vol. 09 , no. 11 , pp.

13. Appendix 1 Phase 1 Phase 2