Competency model for open data literacy in professional learning within the context of Open Government Data (OGD) Eugenia Loría-Solano, Juliana Elisa Raffaghelli Universitat Oberta de Catalunya, Barcelona, 08018, Spain Abstract Research on Open Government Data (OGD) use reveals that the data is not being used as expected. Many governments have opened their data but lack the development of the capacities required for OGD usage. There is a need of having frameworks of reference for open data literacy (ODL). The initial screening of the literature uncovers there is a dearth of systemic interventions to develop ODL, and there is limited research on what works. This research will focus on understanding the contexts and barriers of OGD use to study the role of technical and critical data literacy concerning the current low usage. It will map practices to develop the ODL and expert’s knowledge to create an instrument that could be applied for the diagnostic baseline of ODL. Also, it will explore the applicability of such an instrument for the self-analysis and external learning recognition on ODL. This model can be used in the sphere of government, universities, and business, to assess the level of competencies in OGD usage in their employees or students, and to identify ODL competencies’ gaps in a different context of professional practice. Keywords1 Open government data, open data usage, open data literacy, critical data literacy, technical data literacy, professional learning. mission of “a just, free, and open future, where 1. Introduction all non-personal information is open and free for all to use”. Open data has great potential for use, The open data movement is an emerging specifically, Open Government Data (OGD) for political and socio-economic phenomenon that the development of public policies, democratic promises to promote civic engagement and dialogue, entrepreneurship, among others [2]. drive public sector innovations in various areas There are many benefits expected with the of public life [1]. The Open Data Handbook opening of government data to citizens and (https://opendatahandbook.org/guide/es/what- companies, such as improving transparency, is-open-data/) defines open data as data that can reliability in administration, promoting public be freely used, reused, and redistributed by participation and public-private collaboration, anyone, subject only, at most, to the as well as revitalizing the economy, with the requirement to attribute and share equally. recognition that public data is assets of people. The open data initiative initially arises from [3]. the universal declaration on human rights of However, while many open databases are 1948, where the right to information is already available, only a limited number of them are mentioned in Art.19 used [2], their active use is still limited because (https://www.un.org/en/about-us/universal- of issues with data quality and linkage [4]. In declaration-of-human-rights). Along the same addition, for the use of open data, users require lines, the Open Knowledge Foundation, a framework of open data literacy skills established in 2004, is recognized for its 1 Proceedings of the Doctoral Consortium of Sixteenth European Conference on Technology Enhanced Learning, September 20–21, 2021, Bolzano, Italy (online). EMAIL: eloria@uoc.edu (A. 1); jraffaghelli@uoc.edu (A. 2). ORCID: 0000-0003-1957-420X (A. 1); 0000-0002-8753-6478 (A. 2). ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) essential for advanced use of data in each literacy levels of knowledge. This tool can be context. Raffaghelli [6] has stated that reference used in the sphere of government, business and frameworks are needed for educators’ data in universities, to recognize and measure the literacy since after reviewing the literature level of competences in open data usage in their corpus it was detected that data literacy employees or students in different contexts of connected to OGD is never considered in the professional practice. adult´s data literacy educational frameworks even though it is a crucial dimension of educators’ professional competence. 3. Research problem The expected outcomes and significance of this Ph.D. research are to identify a set of skills and knowledge required to perform in an In the field of open government data, it is advanced level of usage of open government known that an effort was made to open data in data, thus finding the dimensions of ODL. As many governments, but not so much has been well as the development and deployment of a done for the development of the necessary measurement instrument to assess the level of capacities for the exploitation or the optimal use of the same for the taking of government ODL capacities for the quantification of decisions. The World Bank recognizes that its progress on ODL. current support models have focused more on data production and exchange than on building capacity to use data [9]. Furthermore, within the 2. Justification models developed for capacity building, only a few of them have been tested at scale [10]. For The relevance of this research relates the example, hackathons and local training need of having a set of skills of reference for activities within international cooperation such data literacy overall and for open data literacy, as Open Data Day (https://opendataday.org/). specifically. In the same line, one study of the use of the Data literacy, as a research topic, stems from public sector data analytics in The Netherlands numeracy and statistical literacy. However, the shows that the use of public sector data most recent developments connect data literacy analytics requires developing organizational with data-driven digital environments [6]. The capabilities to ensure effective use, foster research tries to identify the needed skills and collaboration, and scale-up [11]. Due to the knowledge concerning professionals and adults problem of the low use of open data, this in relation to open data. Open data is indeed a research focuses on studying the open data digital resource that can both trigger learning or literacy required for the effective and more be a product of formal, non-formal and informal frequent use of these by interested sectors learning. In this regard, Open Data can be including citizens. deemed part of technological environments and has the potential to enhance learning. As it appears from our initial screening of 3.1. Theoretical and empirical the literature, there is a dearth of systemic interventions to develop data literacy, and there antecedents is limited research on what works, as initiatives face funding and organizational challenges Open Government Data is characterized by limit scaling up training [7]. being data and information produced or According to Khayyat and Bannister [8], commissioned by public bodies [12]. Broadly OGD field experiments such as hackathons and speaking, the OECD (https://www.oecd.org/) competitions continue to be conducted, but defines Open Government Data as "a there has been no systematic research on the philosophy, and increasingly a set of policies, factors that contribute to a vibrant and that promote transparency, accountability and sustainable ecosystem of co-creation with civil value creation by making government data communities. available to all." This research is intended to contribute to Citizens’ participation in open government creating an instrument that allows the can improve their perceptions towards identification and assessment of open data government as a transparent, participatory, and collaborative institution and such participation that the reuse or processing of open data to of citizens increases operational capacity and develop third-party applications and projects trust [13]. It promises other benefits such as requires skilled enthusiasts and tech-savvy greater accountability and increased public citizens who are willing to contribute their time, participation, but few of these initiatives have knowledge and expertise to the creation or co- been evaluated in terms of their implementation creation of products based on open data. and results.[14]. And while many open However, this technical definition of open databases are available, only a limited number data literacy focuses on technical skills. of them are used [2]. Another conception, critical data literacy, refers A decade has passed since the first to the skills, knowledge, and attitudes to review International Data Conference the meaning of concepts, visualizations and (https://opendatacon.org/), which is designed to operations carried out with the data that can put bring the global open data community together user groups at risk of inequity or ethical aspects. to learn, share, plan and collaborate on the “The value of openness in the fight against future of open data and data for development. inequality should be emphasized, the equity Although efforts have been made to open should be placed at the center of data analysis, government data in many countries of the and practitioners should actively promote world, there has not been a similar effort to reflection on inclusion gaps in data and the develop the necessary capacities for the use of harm those gaps can bring” [7]. data by citizenship. “Data literacy is not just about open data, but Publishing OGD can lead to innovation open data can be an invaluable asset for since it allows external parties to access, inclusive and empowering data literacy explore and handle OGD, which in turn will development programs” [10]. Identifying the help to develop and build useful services, open data literacy framework and user skill products, and applications for the benefit of gaps is crucial to understanding the types of society [15]. However, Bonina & Eaton [16] professional learning contexts in which they state in their research on the governance of the can be developed. Montes and Slater [7] claim ecosystems of Government Open Data (OGD) that the lack of a coherent and generally platforms, that after a decade of open data accepted definition of data literacy and initiatives few economic and social benefits requisite skill set leaves us without a real have been achieved due to incomplete or low- quantification of progress on open data literacy. quality data, mismatches between the data that Theoretical frameworks refer to the critical are needed and those that are published, and the theory and the socio-technical theory [21], existence of technical barriers to participation, applied to the studies on digital data, data- besides lack of skills and training of users. driven practices and their impact on society and The use of open data, "is the activity that a education. Indeed, data literacy has become an person or organization performs to see, essential part of digital competence as outlined understand, analyze, visualize or in other ways in the DigComp Framework 2.1 [22]. Also, a use a set of data that a government organization critical approach to data is needed in an has provided to the public" [2]. This definition increasingly contested approach to the of use can be identified as technical literacy in developments of data-driven practices [23]. open data, delimited in this research as the The Data Skills Framework developed by competencies, knowledge, and skills necessary the Open Data Institute (ODI) to download, clean, order, analyze and interpret (https://theodi.org/article/data-skills- open data in a specific context. Just publishing framework) is an initial reference for the raw data, may not result in transparency, as technical data literacy approach. Also, in the without formatting the data may not be easy for Digital Competence Framework released by the most people to understand and use [17]. Some European Commission authors suggest helping users using visuals, (https://op.europa.eu/en/home), the concept of “geovisualizing open data seems the next data literacy was introduced in 2017 alongside logical step to put open data in the hands of the information literacy dimension as an ability citizens” [18]. Since it is required the to search, read, and interpret data in several development of competencies for the effective daily and academic contexts of communication use of public sector data analytics in the [24]. organizations [19]. Finally, Kassen [20] states On the other hand, critical data literacy will abilities [6] but miss the political contexts and be studied in the light of the Data feminist the critical approach to data [17]. principles developed in the book Data This instrument can be used in the sphere of Feminism, which presents a new way of government, business, and universities, to thinking about data science and data ethics, assess and recognize the level of competences which is grounded in intersectional feminist in open data usage in their employees or thought. It debates about power, and how those students. Also, to identify and understand the differentials of power can be challenged and OGD competences’ gap in different contexts of changed [25]. Likewise, other texts with a professional practice. critical approach to data will be used as a frame Specific Objectives of this research: of reference, such as Taylor [26] where the 1. To analyze current academic literature author posits that “just as an idea of justice is review to uncover the issues preventing needed in order to establish the rule of law, an open data usage, and within them, the role idea of data justice – fairness in the way people played by data literacy. are made visible, represented and treated as a 2. To identify what data literacy educational result of their production of digital data – is practices are currently available on the web necessary to determine ethical paths through a there will be applied a mapping procedure of datafying world”. such pedagogical practices. Further frameworks to be studied are 3. To validate such open data literacy Markham (https://futuremaking.space/critical- dimensions by a panel of subject matter pedagogy-data-literacy/) who characterizes experts’ interviews. critical pedagogy as a vital part of building data 4. To build and develop the measurement literacy. The author identifies it as a research instrument. stance that can challenge quantification, 5. To theoretically validate the instrument by datafication, and computational logic and it determining the validity of the tool through moves beyond the level of data critique to social the Delphy method. action in response to datafication. Other 6. To empirically validate the instrument approaches will be considered such as through Circulation of the instrument as a Raffaghelli [27], where the author provides a survey, the estimation of Cronbach's alpha conceptual scheme to address further statistic and the confirmatory factor pedagogical reflection and practice to support analysis. social justice against datafication. 7. To test the instrument in the context of ecological learning training by the application of it to the participants, as well 4. Aim of the research as the application of a statistical analysis of the results to determine a diagnostic baseline in Open Data literacy and sensitivity to The aim of this research is to identify a competence change. model of the open data literacy that professional learners must acquire to operate in advanced contexts of data usage. Once detected through the model, such literacy could be developed 5. Research hypothesis through different types of learning contexts. Moreover, the model could address The evaluation and recognition of skills and professional learning recognition. knowledge connected to open data usage could The research aims at developing an be supported by an open data literacy tool. instrument that allows recognition and assessment of several levels of competence in open data literacy. Therefore, the stage of skills 6. Research questions and knowledge within a context of usage of open data as digital resources. This is an original purpose since most In this context, the following research studies analyze data literacy centered in questions have been posed: technical procedures relating data science RQ1 What are the contexts of use and learning based on OGD? RQ2 What are the barriers that prevent the problems in a general and specific way, with use of open data, and within those barriers, respect to the research questions posed. what role does technical and critical data Also, an exploratory research, mapping and literacy in open data play as one of the gap analysis is going to be performed to identify causes of the low use of OGD? what data literacy educational practices are RQ3 What are the current pedagogical currently available in the web. practices available that can be used to Finally, there will be a panel of experts develop the ODL required to make use of interviews to identify dimensions as a base to OGD? the development and operationalization of the RQ4 What is the set of skills needed in OGD measurement instrument. practice contexts required for professional The second phase will be devoted to the learning? development of a self-reported measurement RQ5 How should be configured a instrument, over the basis of the theoretical measurement instrument that could be assumptions emerging from the literature applied for the diagnostic baseline of ODL? review. RQ6 What is the applicability of such After identification of the dimensions, from instrument for the self-analysis and/or the theoretical frameworks review, for the external learning recognition on Open data theoretical validation, a Delphi study will be literacy? conducted. The panel of experts is going to be used for building the open data literacy set of skills and knowledge and the Delphi method to validate the measurement instrument. The 7. Methods Delphi method is defined as “a panel 7.1. Design of research communication technique by which researchers collect expert opinions, enable experts to To pursue the objective of this study, a communicate anonymously with one another mixed methods research approach will be and then explore the underlying information applied. The design implies three phases to collected” [29]. cover the objectives. The panel of experts will be invited to The first phase will be devoted to the review the instrument through the technique of analysis of the problem and the existing corpus interviews, developed in two stages. Therefore, of research. To this regard, a systematic review the results will be assembled, and a second of the literature will be undertaken based on the cycle of consultation will be enacted. [30]. A methodological workflow called PRISMA [27] measurement instrument is going to be and it is a transparent report of systematic designed and created to assess open data reviews and meta-analyzes. This method literacy in the contexts of OGD. As for the attempts to control for investigator bias in data empirical validation of the instrument, it is collection and analysis [28]. going to be circulated as a questionnaire to The main PRISMA steps that will be professionals working in either public carried out in this research are: 1. Select administration or industry with a stratified scientific databases, 2. Search the databases sampling design by sector. with keywords of interest for several articles, 3. The study is going to use the exploratory, Select articles using predefined exclusion descriptive, and explicative approaches in its criteria based on in the research objectives. 4. different research phases. Analyze the selected articles by reading them in Finally, the third phase will be devoted to full. the instruments’ consolidation and further The systematic review of the literature will validation in ecological training contexts, the be integrated with an analysis of existing developed scale will be applied in specific pedagogical practices (benchmarking educational context to analyze the applicability study/desk research), which will support the to: analysis of type of competences focused and 1. Evaluate the development of ODL in trained as part of an underlying ODL approach. ecological training context. Based on this selection, quantitative 2. Self-assess ODL in formal (undergraduate) analysis methods will be applied that allow and non-formal/informal (professional) better identification of emerging issues and learning contexts. 3. Recognize ODL in professional contexts. b. The research focuses on the type of open data and applications (Discipline, Type of Open Data, Applications of open data) 7.2. Sample c. Types of learning generated and barriers of use (Types of learning generated using open 7.2.1. First phase data, Barriers that prevent the use of open data) The sample units will be the articles Finally, after consolidating the categories, selected for the literature analysis. For the the authors will analyze 10% of the total set of selection of articles, this research will apply the articles and the agreement between evaluators PRISMA method for the systematic literature will be estimated using Cohen's Kappa statistic review. The detail of what will be done in each (https://www.statisticshowto.com/cohens- step, for the selection of a sample of articles, is kappa-statistic/). A kappa higher than 0.60 can detailed below: be considered a good agreement. 1. Selection of Databases. SCOPUS, DOAJ and WOS will be selected to perform the 7.2.2. Second phase bibliographic search. 2. Selection of articles using keywords. It is of interest to this research to know In the initial task relating to the Panel of characteristics related to the use of open experts' interviews and Delphi study, the expert data, as well as to know aspects that selection will be carried out in a non-random prevent its use. Therefore, the following manner based on their expertise on the keywords will be searched in the selected phenomenon being studied [31]. In this case are databases: OGD subject matter experts. The sample size 3. SCOPUS and DOAJ: (open AND data) for the interviews and the Delphi study will be AND (government) AND (us *) determined by the saturation point with a 4. WOS: (open AND data) AND minimum of seven qualitative interviews to (government) AND (usa *) subject matter experts, active OGD users. 5. Screening of articles abstracts will be read, The target population is made up by 1. and the following exclusion criteria will be Quantitative units of analysis are current and used: potential OGD users around the globe that are a. Date before 2016, to have the latest available to fill out the instrument, 2. knowledge on the topic of interest Qualitative units of analysis: are adult b. DOI absence professionals identified as subject matter c. Other Open Data issues that are not OGD experts, and frequent users of OGD and ORD. d. It is not an article or review Specifically, to test the questionnaire and to get e. Not in English data to validate and measure the reliability of f. Related to OGD but not its use the questions. The experts are professionals g. Not available who have high experience on OGD usage. 6. Analysis of the articles after reading them Professionals are current or potential users of in full: each one will be read completely OGD. and will be coded and classified in The sample size estimated for this study is variables defined in the codebook, which 196 units of analysis, therefore 196 OGD users. will be defined by the authors based on the It assumes a confidence level of 95%, a objectives of the research, to generate a maximum error of 7% and a variance of 0.25. It database of articles that will later be assumes a big target population of ODG users. analyzed quantitatively to obtain their Since currently there isn´t defined a respective findings. Specifically, the sampling frame of the OGD user´s population, articles will be coded and classified in the the type of sampling to be used in this study is following categories: non-probabilistic sampling defined as “a a. The identity of the research (Authors, Title, sampling technique in which some units of the Year, Title of the source, No. of Citations, population have zero chance of selection or DOI, Type of Document, Abstract of the where the probability of selection cannot be article, Author's keywords) accurately determined” [31]. Measures of construct reliability and summarized in table 1, which is located at validity will be implemented, over the basis of Appendix 1. The summary table includes the classical test theory [32], [33]. phase, objective that is going to be pursued, the activity or task to be performed, the method to 7.2.3. Third phase be applicable for pursuing the objective and the expected output or result for each task. Two groups will be tested: 1. A group with at least 20 workers with none to high experience on the usage of OGD in 8. Current status and results both public and industry settings, for self- 8.1. Systematic review of assessment and recognition of competences purpose. literature 2. A group of at least 20 undergraduate students in several disciplines, for self- In short, the PRISMA systematic review of assessment purposes, will be literature reveals that the use of OGD seems to experimentally exposed or not exposed to depend largely on the necessary technical and OGD. critical skills. Although there are many technological, structural, organizational, and cultural barriers, the skills of the stakeholders 7.3. Data collection techniques to use and obtain the expected benefits of open and instruments data is an obstacle that requires consideration. The analysis of the corpus of literature For data collection the research will adopt uncovers that the lack of open data literacy a mixed methods approach. A desk research arises as the main barrier, particularly in social approach will be applied to the first phase will sciences, OGD and governance. Our results adopt documental analysis and classification of reinforce the importance of data literacy, this is pedagogical practices through a deductive coherent with Matheus & Janssen [17] who scheme of analysis. Also, a synthesis report will imply that the same data that creates a higher be performed to identify ODL set of skills to level of transparency for the expert, creates less define its dimensions. Then, in the second for someone with lack of knowledge of how to phase, a qualitative approach based on in-depth use it. re being considered. interviews will be adopted for the identification Overall, what can be inferred from our of dimensions and the instrument design and analysis is that literacy opportunities are mostly Delphi study for theoretical validation. technical; and that engagement with open data, On the other hand, a quantitative approach when occurs, produces meaningful learning. will be adopted both for the instrument However, our analysis could not cover to empirical validation (end of the second phase), what extent the collaborative and co-creative and for the instrument testing (third phase). An synergies between stakeholders can lead to electronic form with the instrument will be innovation and governance. These are aspects implemented and circulated for data collection. that remain to be studied towards a holistic and In the case of the third phase, there will also be critical data literacy. a qualitative data collection and analysis. Finally, the research outputs at this stage of Indeed, the instrument will be embedded in a the PhD are part of a literature review research, learning management system and the results but the following phases relate online will be made available for the respondents to observations, interviews, the construction of an react, reflect, and discuss upon them as the instrument based on a survey and the empirical formative impact of the instrument validation in two phases. implementation. 9. Limitations of the study 7.4. Procedure This research is at a very early stage. In any The procedure is going to be developed in case, the limitations foreseen relate a) the three phases, as explained before, and it is documented difficulties in analyzing adult learning and identifying patterns of learning Japan,” Computer (Long. Beach. Calif)., activity (most learners follow informal learning vol. 51, no. 12, pp. 14–23, Dec. 2018, doi: pathways); b) the complex approach that the 10.1109/MC.2018.2879993. empirical validation will require, in terms of [4] K. Haklae, “Interlinking Open participants' recruitment; c) the complexity of Government Data in Korea using identifying experts and contexts for empirical Administrative District Knowledge work. Graph,” J. Inf. Sci. Theory Pract., vol. 6, In any case, risk management strategies are pp. 18–30, 2018, doi: being considered. 10.1633/JISTaP.2018.6.1.2. [5] J. E. Raffaghelli, “Developing a framework for educators’ data literacy in 10. Data management and ethics the European context: Proposal, Implications and debate.,” in International Conference on Education and New This research plan was approved by the Learning Technologies EDULEARN, Jul. ethics committee of the UOC. For the approval 2019, pp. 10520–10530, doi: of the ethical form, it was required to explain 10.21125/edulearn.2019.2655. details about data curation policies, informed [6] J. E. Raffaghelli, “Developing a consent, how to proceed with the database once framework for educators’ data literacy in the study is concluded, etc. The data will be the European context: Proposal, processed exclusively for the purposes for Implications and debate.” in which they have been collected and for the time EDULEARN19 Proceedings, 2019, vol. 1, strictly necessary to fulfill the purposes for doi: 10.21125/edulearn.2019.2655. which they will be collected. [7] M. G. Montes and D. Slater, “Issues in open data: Data literacy. In T. Davies, S. Walker, M. Rubinstein, & F. Perini 11. Acknowledgements (Eds.),” in The state of open data: Histories and horizons, Cape Town and Ottawa: We would like to acknowledge UOC’s African Minds and International doctoral school on Education and ICT program, Development Research Centre, 2019, pp. which offers the courses of social research 274–286. design, Research methods, Directed Research [8] M. Khayyat and F. Bannister, “Towards a subjects, as well as, workshops, and training model for facilitating and enabling co- sessions, during which we have been able to creation using open government data,” Inf. develop this research proposal. Polity, vol. 22, no. 4, pp. 211–231, Nov. 2017, doi: 10.3233/IP-170406. [9] World Bank, “Data for Development An Evaluation of World Bank Support for 12. References Data and Statistical Capacity,” 2017. Accessed: Apr. 19, 2021. [Online]. [1] M. Kassen, “Understanding motivations of Available: www.worldbank.org. citizens to reuse open data: open [10] T. Davies, S. Walker, M. Rubinstein, and government data as a philanthropic F. Perini, The State of Open Data: movement,” Innovation, vol. 00, no. 00, Histories and Horizons. Cape Town and pp. 1–27, 2020, doi: Ottawa: African Minds and International 10.1080/14479338.2020.1738940. Development Research Centre, 2019. [2] A. Zuiderwijk, M. Janssen, and Y. K. [11] Van Veenstra and Gromme, “(6) (PDF) Dwivedi, “Acceptance and use predictors The Use of Public Sector Data Analytics in of open data technologies: Drawing upon The Netherlands,” Sep. 2020. the unified theory of acceptance and use of https://www.researchgate.net/publication/ technology,” Gov. Inf. Q., vol. 32, no. 4, 344202992_The_Use_of_Public_Sector_ pp. 429–440, Oct. 2015, doi: Data_Analytics_in_The_Netherlands 10.1016/j.giq.2015.09.005. (accessed Jan. 03, 2021). [3] T. Sumitomo and N. Koshizuka, “Progress [12] B. Ubaldi, “Open Government Data: and Initiatives for Open Data Policy in Towards Empirical Analysis of Open Government Data Initiatives,” OECD [22] S. Carretero, R. Vuorikari, and Y. Punie, Work. Pap. Public Gov., vol. NO.22, no. “The Digital Competence Framework for 22, p. 61, 2013, doi: Citizens With Eight,” Publ. Off. Eur. 10.1787/5k46bj4f03s7-en. Union, no. May, p. 48, 2017, doi: [13] L. Schmidthuber, F. Piller, M. Bogers, and 10.2760/38842. D. Hilgers, “Citizen participation in public [23] B. Williamson, “Journal of Education administration: investigating open Policy Digital education governance: data government for social innovation,” 2019. visualization, predictive analytics, and [14] R. Sandoval-Almazan and J. R. Gil- ‘real-time’ policy instruments,” 2015, doi: Garcia, “Toward an integrative assessment 10.1080/02680939.2015.1035758. of open government: Proposing [24] J. E. Raffaghelli, “Is data literacy a catalyst conceptual lenses and practical of social justice? A response from nine components,” J. Organ. Comput. Electron. data literacy initiatives in higher Commer., vol. 26, no. 1–2, pp. 170–192, education,” Educ. Sci., vol. 10, no. 9, Apr. 2016, doi: 2020, doi: 10.3390/educsci10090233. 10.1080/10919392.2015.1125190. [25] C. D’Ignazio and L. F. Klein, Data [15] M. Saleh Altayar, “Motivations for open Feminism. 2020. data adoption: An institutional theory [26] L. Taylor, “What is data justice? The case perspective,” 2018, doi: for connecting digital rights and freedoms 10.1016/j.giq.2018.09.006. globally,” Big Data & Society July– [16] C. Bonina and B. Eaton, “Cultivating open December 2017, 2017. government data platform ecosystems https://pure.uvt.nl/ws/portalfiles/portal/26 through governance: Lessons from Buenos 363266/taylor_17_what_is_data_justice.p Aires, Mexico City and Montevideo,” df (accessed May 09, 2021). Gov. Inf. Q., vol. 37, no. 3, p. 101479, Jul. [27] A. D. Moher D, Liberati A, Tetzlaff J, 2020, doi: 10.1016/j.giq.2020.101479. “PRISMA 2009 Flow Diagram,” The [17] R. Matheus, M. Janssen, and D. PRISMA statement, vol. 6. 2009. Maheshwari, “Data science empowering [28] M. Petticrew and H. Roberts, Systematic the public: Data-driven dashboards for Reviews in the Social Sciences: A transparent and accountable decision- Practical Guide. 2006. making in smart cities,” Gov. Inf. Q., vol. [29] Y.-F. Yeh, Y.-S. Hsu, H.-K. Wu, F.-K. 37, no. 3, p. 101284, Jul. 2020, doi: Hwang, and T.-C. Lin, “Developing and 10.1016/j.giq.2018.01.006. validating technological pedagogical [18] A. Degbelo, J. Wissing, and T. Kauppinen, content knowledge-practical (TPACK- “A comparison of geovisualizations and practical) through the Delphi survey data tables for transparency enablement in technique,” Br. J. Educ. Technol., vol. 45, the open government data landscape,” Int. no. 4, pp. 707–722, 2014, [Online]. J. Electron. Gov. Res., vol. 14, no. 4, pp. Available: 39–64, Oct. 2018, doi: https://www.learntechlib.org/p/148366/. 10.4018/IJEGR.2018100104. [30] M. Adler and E. Ziglio, Gazing into the [19] A. F. van Veenstra, F. Grommé, and S. oracle: the Delphi method and its Djafari, “The use of public sector data application to social policy and public analytics in the Netherlands,” Transform. health. Jessica Kingsley Publishers, 1996. Gov. People, Process Policy, 2020, doi: [31] A. Bhattacherjee, Social Science 10.1108/TG-09-2019-0095. Research: Principles, Methods, and [20] M. Kassen, “Adopting and managing open Practices, 3rd ed. Textbooks Collection., data,” Aslib J. Inf. Manag., vol. 70, no. 5, 2012. pp. 518–537, Sep. 2018, doi: [32] P. Mair, Modern Psychometrics with R. 10.1108/AJIM-11-2017-0250. Cham: Springer International Publishing, [21] T. Fenwick, M. Nerland, and K. Jensen, 2018. “Sociomaterial approaches to [33] T. A. Kyriazos and A. Stalikas, “Applied conceptualising professional learning and Psychometrics: The Steps of Scale practice,” J. Educ. Work, vol. 25, no. 1, pp. Development and Standardization 1–13, Feb. 2012, doi: Process,” Psychology, vol. 09, no. 11, pp. 10.1080/13639080.2012.644901. 2531–2560, 2018, doi: 10.4236/psych.2018.911145. 13. Appendix 1 Table 1 Procedure summary table Phase Objective Activity Method Expected Result ODL centrality needs Systematic literature Academic literature and the identification Analysis of the review with the review of professional learning problem and PRISMA Method needs. the existing Mapping of Map of current Phase corpus of Desk work and pedagogical pedagogical practices 1 research and to benchmarking practices and gap identification map current Report of skills and Identification of sets of pedagogical knowledge required skills and knowledge to practices Synthesis report for open data be included in the literacy instrument. To establish and validate the Panel of experts Established ODL dimensions of ODL (interviews) Dimensions construct Operationalization of Questionnaire Instrument the dimensions in prepared in document development Items with Likert scale and digital To validate To develop and Validated ODL dimensions of ODL Delphi Method Phase validate the Dimensions by experts 2 measurement Circulation of the instrument instrument as a First report of the Empirical validation survey to a population empirical validation of the instrument of at least 196 analysis persons Cronbach's Alpha Definitive validation Estimation and Instrument validation of the instrument Confirmatory Factor report Analysis Report of results of the Application to the instrument that participants of an includes baseline of ecological training diagnosis and context sensitivity to change of To use the Testing the competence. instrument in a instrument in a Phase Statistical analysis of context of context of 3 the results of the ecological ecological learning instrument to learning training determine a Measurement diagnostic baseline in instrument released. Open Data literacy and sensitivity to competence change.