Developing open educational resources to doctoral education training: an educational design research approach Lorena Sousa University of Aveiro, Campus Universitário de Santiago, 3810-193 Aveiro, Portugal Abstract This investigation aims at developing an Open Educational Resource (OER) to the learning of research methods in doctoral programs in Technology-enhanced Learning (TEL). Under the methodological approach of Educational Design Research, it consists of three phases: context analysis, development and formative evaluation, and semi-summative evaluation. Preliminary results from the first phase revealed that design-based research is the most used research method in TEL and also the one that PhD candidates and PhD holders need more training. Considering this, in the second phase, the OER prototype about design-based research was designed and developed using the H5P plugin on Moodle, where it was possible to insert some gamification strategies. At first, only a section about one research method was developed, so that this prototype can be formatively evaluated before developing the whole course on research methods. At the moment, the formative evaluation is being planned to encompass three cycles of assessment. In the first cycle, the scientific content will be validated by experts; in the second, a survey will be conducted with master’s and PhD students; and finally, there will be focus groups with professors, researchers and PhD students. In the last phase, a semi-summative evaluation of the whole course will be performed to make the last adjustments to the OER and finally implement and disseminate the results in doctoral programs in TEL. Keywords 1 Open Educational Resources, doctoral education, Educational Design Research 1. Introduction open licensing that permits their free reuse, remix and sharing for educational purposes [1]. These educational materials must be under As information technologies have advanced open licenses or reside in the public domain, and been more accessible, a vast number of free of copyright restrictions, to give users free digital resources have become more available permission to adapt and reuse them [2]. Wiley for those involved in education. Teachers have [3] claims that content is open not only when it been using the Internet to spread their materials is freely available to be used in other contexts. and courses, and content in digital format has It is open when it gives everyone permission to largely increased. Yet, most of these materials engage with the material through different are not open to be freely reused, shared or activities, known as the 5R: retain, reuse, remixed. With the purpose of overcoming these revise, remix, and redistribute. The types of barriers, the Open Educational Resource (OER) materials can vary from videos, images and movement was founded to encourage and textbooks, to podcasts, games, and courses [1]. enable anyone to reuse and share content in an There are some motivations for educators, open manner. institutions and governments to be involved OER can be defined as any teaching, with the development and sharing of OER [4, 5, learning or research materials that make use of Proceedings of the Doctoral Consortium of Sixteenth European Conference on Technology Enhanced Learning, September 20–21, 2021, Bolzano, Italy EMAIL: lorena@ua.pt (L. Sousa) ORCID: 0000-0001-5970-4027 (L. Sousa) ©️ 2020 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) 6]. Educators, for example, are able to share 1. to analyse the use of OERs in doctoral content as well as reuse and adapt it according education training; to their context, optimizing their time in 2. to identify which characteristics can creating materials from scratch [7]. contribute to make OERs an engaging Consequently, by reusing and sharing these solution to doctoral education training; resources, there might be an improvement in 3. to develop an OER to the learning of their quality, and the costs of content research methods in doctoral programs in development can be reduced, which can be a TEL; benefit to the institutions. From the 4. to evaluate the developed OER on governmental perspective, OER projects make research methods in doctoral programs in learning more accessible to society, particularly TEL. to nontraditional groups of learners, bridging the gap between non-formal, informal and In the following sections the methodological formal learning [4, 5, 6]. approach and some preliminary results are However, according to [8], educators from going to be presented. higher education institutions are still resistant to embracing the use of OERs and open 2. Research methodology educational practices. Besides, there is a lack of technical skills to select and remix OER appropriately and a lack of awareness regarding This investigation is being developed under copyright issues among academics [5, 7]. A the Educational Design Research survey conducted by the Doctoral Education for methodological approach. Its planning consists Technology-enhanced Learning (DE-TEL) of three phases [10, 11]: (1) context analysis, project has also shown that OERs are not (2) development and formative evaluation, and popular among PhD candidates and PhD (3) semi-summative evaluation. Each phase will be explained in the next subsections. holders. When asked which learning sources they used to deepen their knowledge on TEL topics, doctoral training topics and research 2.1. Context analysis methods, OERs were the least voted, being courses in the PhD program, academic In the first phase, named context analysis, a publications and supervisor help among the systematic literature review (SLR) is going to most voted learning sources. be carried out. The main goals of this SLR are Nine European universities and the to analyse the use of OERs in doctoral European Association for Technology- education training and identify the main Enhanced Learning (EA-TEL) created the challenges and barriers when adopting OERs in Doctoral Education for Technology-enhanced higher education, especially in doctoral Learning (DE-TEL) project with the aim at programs. identifying the best practices in doctoral Furthermore, a survey will be conducted programs in TEL, developing a proposal for a among PhD candidates, researchers and new program and developing sustainable Open practitioners who study and/or work in the field Educational Resources (OERs) [9]. The OERs of TEL, the context of this study, to identify will encompass modules on research methods which characteristics can contribute to make and key topics in TEL, such as Artificial OERs an engaging solution to doctoral Intelligence in education and mixed and education training. The survey will be divided Augmented Reality for TEL. into four sections. In the first section, there will This thesis, in particular, focuses on the be questions related to personal background. development of OERs to doctoral education The second section is going to be adapted from training and, as a contribution to the DE-TEL [12] and ask how relevant some characteristics project, the OERs are going to cover the are when searching for content for doctoral research methods for doctoral programs in education training. TEL. Our main research question is which Sections three and four will regard the characteristics should an OER developed for factors and formats that would make the doctoral education training have in order to participant more likely to select a particular make its use more engaging? and our objectives resource when searching for content. These are: sections are going to be adapted from a survey developed by the Hewlett-funded OER In the first cycle, there will be a validation Research Hub, an open research project based of the scientific content of the prototype. Three at The Open University (UK) [13]. At the end experts in research methods are going to be of each section, there will be an open-ended contacted via email to validate the content. The question in case participants need to add any main goal of this cycle is to verify if the content additional comments. is appropriate to the learning of research The data from the closed-ended questions methods in doctoral programs in TEL and make will be analysed through descriptive analysis on the necessary adjustments, before starting the SPSS, and the data from the open-ended second cycle. To guide the experts’ validation, questions will be analysed through content there will be a survey with questions asking if analysis on NVivo, if necessary. the content is useful, relevant, accurate, reliable, sufficient and if it meets learners’ 2.2. Development and formative needs. There will also be an open-ended question for additional comments. evaluation In the second cycle, the general aspects of the prototype will be evaluated with twenty In the second phase, called development and PhD candidates. They are going to answer a formative evaluation, the conceptualization, survey consisting of the User Experience design and development of the OER prototype Questionnaire (UEQ) and a self-design will be carried out considering the previous questionnaire. results from the first phase. Since one of the According to Díaz-Oreiro et al. [14], objectives of the DE-TEL project is to develop although AttrakDiff has appeared five years OERs on research methods and key topics in earlier than the UEQ and is the questionnaire TEL, this study will focus on the development that counts the most uses since 2006, UEQ has of OERs on research methods. surpassed AttrakDiff in uses per year in 2017 First, a prototype is going to be developed. and 2018. Conducting a quick search on Scopus This prototype will go through cycles of using the search query “user experience formative evaluation with PhD candidates, questionnaire” and “attrakdiff”, it is possible to professors and researchers in the field of TEL, visualize in Figure 1 that this tendency was kept and as feedback is being received, adjustments in 2019, 2020 and has also continued in 2021 and improvements will be carried out. (data until July). At the moment, we are studying the instruments and techniques and also searching for some validated questionnaires and scales to formatively evaluate the OER prototype. This formative evaluation will probably consist of three cycles, as presented in Table 1. As the feedback is being received from each cycle, adjustments and improvements will be carried out before starting the next cycle. Table 1 Figure 1: AttrakDiff versus UEQ Cycles of the formative evaluation Cycle What? Who? How? Díaz-Oreiro et al. [14] also report that 1 st scientific 3 experts message standardized user experience questionnaires content by email were used in combination with other evaluation 2 nd general 20 PhD UEQ + instruments, such as self-design questionnaires. As we intend to apply a self-design aspects students self-design questionnaire as a complement to the questions standardized questionnaire, we chose to use 3 rd general DE-TEL focus UEQ over AttrakDiff because AttrakDiff is aspects partners groups applied on its website (attrakdiff.de), without and PhD the possibility of adding other groups of students questions. Using UEQ, we will add it on LimeSurvey together with other questions from and make the last adjustments to the OER if our self-design questionnaire. On the website necessary. The data will be analysed using ueq-online.org, it is possible to download the SPSS and NVivo. Finally, the OER is going to UEQ in more than 30 languages and a tool is be disseminated in the doctoral programs in also available on an Excel file, completely free TEL in Europe. of charge, to facilitate the data analysis. Figure 2 summarizes the methodological As a complement to the UEQ, the self- design of this research: design questionnaire will encompass some open-ended questions, such as the most positive and negative aspects of the prototype in their opinion, and suggestions for improvement. These data will be analysed through content analysis on NVivo. The data collected in this second cycle will be triangulated with the data from the participants’ access to Moodle, where the OER is going to be integrated. On Moodle, it is possible to visualize if the participants accessed Figure 2: Methodological design the platform, how much time they spent on it, which pages they visualized and interacted with, which tasks they accomplished, etc. The 3. Preliminary results SPSS software is going to be used for statistical data analysis. This section will present some preliminary In the third cycle, focus groups are going to results regarding the development of the first be conducted. The purpose of these focus OER prototype on research methods in TEL. groups is to show some preliminary results and This first version is being developed using the shed light on the data collected in the second H5P tool and plugin on Moodle to test the cycle in order to identify the participants’ possibilities of tools that might be useful for the opinions about the prototype. There will be two development of OER prototypes. focus groups, one with professors and A survey was conducted by the DE-TEL researchers, and another with PhD candidates. project to collect information on the current Each group will have from six to eight practices and challenges of doctoral education participants, last from 60 to 90 minutes, and in TEL and find out what topics are useful but will be conducted online, through the Zoom have few educational resources. The survey platform. The questioning route will include was conducted among students, PhD questions related to the interactive content candidates, researchers, practitioners who study created with the H5P plugin on Moodle, the 5R and/or work in the field of TEL, between activities related to OERs, negative and positive December 2020 and March 2021, through the aspects, and others, to make improvements to online survey tool LimeSurvey whose link was the prototype. available on the DE-TEL webpage (ea- After the prototype goes through these tel.eu/de-tel/survey). The DE-TEL partners are cycles of formative evaluation, the final version using SPSS and Tableau to analyse the of the OER will go through a semi-summative collected data. evaluation before being disseminated in Preliminary results have revealed that 229 doctoral programs in TEL. participants from 40 different countries, most of them from Europe, answered the survey. When asked to select the item that best described the 2.3. Semi-summative evaluation general methodological approach of their PhD research, PhD candidates and PhD holders In the third and final phase, after we have a reported that design-based research was the final version of the OER, a semi-summative most used research method, followed by evaluation is going to be conducted among PhD quantitative and qualitative methods, as can be candidates, professors and researchers who seen in Figure 3. Regarding the research study and/or work in the field of TEL. A survey methods that they need more training, design- will be conducted to conclude this investigation based research was also the most selected research method by PhD candidates and PhD holders, followed by quantitative and qualitative methods as well. Figure 4: Example of the content type called drag the words Figure 3: Research methods use and training needs (source: data from the DE-TEL project) The content type that was adopted in the creation of this OER prototype on design-based As this research aims at developing OERs research was the interactive book. The H5P to the learning of research methods in TEL and interactive book content type allows authors to the preliminary results showed that design- create courses, books or tests, combining based research is the most used research various interactive content types inside of it, method and also the one which PhD candidates such as interactive videos, course presentations, and PhD holders need more training, a questions and much more, through multiple prototype of the OER module about design- pages. Figure 4 illustrates a part of the page based research is being designed and developed about the characteristics of design-based using H5P tool. research. On the left-hand side of the image, it H5P tool (h5p.org), an abbreviation for is possible to visualize the contents of this HTML5 Package, is a completely free and open interactive book. The numbers in the top right- technology, which enables anyone to create, hand corner of the image indicate the actual share and reuse interactive HTML5 content page and the total pages of the book, more efficiently, without the need for any respectively. technical knowledge. At the end of the interactive book, there is a H5P makes it easy to create rich interactive report displaying the learner’s progress content by providing several content types for throughout the book (Figure 5). There is the various needs. It is possible to create videos total score, which is the number of points enriched with interactions, presentations with scored by the learner from his/her correct interactive slides, drag and drop tasks with answers to the interactive questions; the book images and text, images with multiple progress, which is the percentage of the information hotspots, single or multiple choice visualized pages and the performed interactions questions, interactive books integrating several with the content (it is not possible to get 100% content types, and many others. of book progress only visualizing the pages); Figure 4 shows an example of the content and the interactions progress, which is the type named drag the words, where it is possible percentage of the content that the learner to create text-based drag and drop tasks. In this interacted with. On the last page of the activity, learners are asked to drag the interactive book, there is also a summary that characteristics of design-based research and shows the details of the interactions from each drop them into the correct explanation. Then, page. they can check their answers to see how many responses they got right and a score is generated. They can also choose between retrying the task or visualizing its solution. Figure 4, for instance, presents the solution with Figure 5: Report of an interactive book the correct responses. H5P content can be integrated into other platforms, such as Canvas, Blackboard and Moodle. This content about design-based research, specifically, was integrated on the open Tech4Comp platform (moodle.tech4comp.dbis.rwth-aachen.de/), on Moodle, since it is the platform that is being 6. References used by the DE-TEL project. It is also possible to create content using the H5P plugin on [1] UNESCO, Guidelines on the development Moodle. Thus, some editing and complement to of open educational resources policies, this material were performed directly on 2019. URL: Tech4Comp Moodle platform. http://oasis.col.org/bitstream/handle/1159 With this prototype, we intend to insert 9/3455/2019_Guideliness_OER_Policy_fi some OER characteristics according to the nal_COL_web.pdf?sequence=4&isAllow answers from the survey which is going to be ed=y carried out in the context analysis phase. PhD [2] D. Wiley, T. J. Bliss, M. McEwen, Open candidates, researchers and practitioners are Educational Resources: A Review of the going to identify which characteristics can Literature, in: J. M. Spector, M. D. Merrill, contribute to make OERs an engaging solution J. Elen, M. J. Bishop (Eds.), Handbook of to doctoral training and we are going to insert Research on Educational Communications these characteristics into the prototype to see and Technology, 4th. ed., Springer, New how they work. York, NY, 2014, pp. 781–789. doi: 10.1007/978-1-4614-3185-5. 4. Final considerations [3] D. Wiley, Defining the “open” in open content and open educational resources, With this investigation, we expect to 2014. URL: contribute to the practical and theoretical https://opencontent.org/definition/ understanding of the process related to the [4] J. Hylén, Open Educational Resources: development of technological solutions and Opportunities and Challenges, OECD’s innovations to the area of TEL. Design-based Centre for Educational Research and Research (or Educational Design Research) is Innovation, Paris, France, 2006. the most used research method in TEL and its [5] OECD, Giving Knowledge for Free: The process brings both theoretical and practical Emergence of Open Educational contributions to the area. Resources, Centre for Educational Regarding the practical results, we hope that Research and Innovation, Paris, France, the developed OER can be a useful tool to the 2007. teaching and learning of design-based research, [6] S. D’Antoni, Open Educational that it helps PhD candidates to deepen their Resources: reviewing initiatives and knowledge on this research method, and that its issues, Open Learning: The Journal of characteristics can contribute to making its use Open, Distance and e-Learning 24 (2009) more engaging. 3–10. doi: 10.1080/02680510802625443. Theoretically, we expect that the developed [7] C. Hodgkinson-Williams, Benefits and OER can raise awareness of the importance of challenges of OER for higher education the adoption and spread of OERs in education, institutions, Centre for Educational mainly among doctoral programs, and that this Technologies, University of Cape Town, study can support the development of other Cape Town, South Africa, 2010. solutions and innovations to the field of TEL. [8] A. García-Holgado, F. J. García-Peñalvo, C. Higuera, A. Teixeira, U. Ehlers, J. Bruton, F. Nascimbeni, N. P. Zea, and D. 5. Acknowledgements Burgos, Promoting Open Education Through Gamification in Higher I would like to express my gratitude to my Education: the OpenGame project, in: supervisors, Professors Luís Pedro and Carlos Eighth International Conference on Santos. This paper and the research behind it Technological Ecosystems for Enhancing would not have been possible without their Multiculturality, TEEM’20, ACM, New support, guidance and valuable comments. York, NY, 2020, pp. 399–404. doi: This project has received funding from the 10.1145/3434780.3436688. European Union’s Erasmus Plus programme, [9] M. Fominykh, E. Prasolova-Førland, DE- grant agreement 2019-1-NO01-KA203- TEL - A European initiative for Doctoral 060280. Education in Technology-Enhanced Learning, 2020, ERCIM NEWS 120, Special Theme: Educational Technology. [10] S. McKenney, T. C. Reeves, Conducting Educational Design Research, Routledge, New York, NY, 2012. [11] T. Plomp, Educational Design Research: an Introduction, in: T. Plomp, N. Nieveen (Eds.), Educational Design Research - Part A: an Introduction, Netherlands Institute for Curriculum Development (SLO), Enschede, the Netherlands, 2013, pp. 11– 50. [12] J. Henriques, Catálogo de características para análise e avaliação de Recursos Educacionais Abertos (REA): ferramenta de avaliação no formato checklist, Universidade Aberta, Lisboa, Portugal, 2016. [13] R. Farrow; R. Pitt; B. de los Arcos; L. Perryman; M. Weller, P. McAndrew, Impact of OER use on teaching and learning: data from OER Research Hub (2013–2014), British Journal of Educational Technology 46 (2015) 972– 976. doi: 10.1111/bjet.123410. [14] I. Díaz-Oreiro, G. López, L. Quesada, L. Guerrero, UX Evaluation with Standardized Questionnaires in Ubiquitous Computing and Ambient Intelligence: A Systematic Literature Review, Advances in Human-Computer Interaction (2021). doi: 10.1155/2021/5518722.