=Paper=
{{Paper
|id=Vol-3076/paper06
|storemode=property
|title=Developing open educational resources to doctoral education training: an educational design research approach
|pdfUrl=https://ceur-ws.org/Vol-3076/ECTEL2021_DC_paper06.pdf
|volume=Vol-3076
|authors=Lorena Sousa
|dblpUrl=https://dblp.org/rec/conf/ectel/Sousa21
}}
==Developing open educational resources to doctoral education training: an educational design research approach==
https://ceur-ws.org/Vol-3076/ECTEL2021_DC_paper06.pdf
Developing open educational resources to doctoral education
training: an educational design research approach
Lorena Sousa
University of Aveiro, Campus Universitário de Santiago, 3810-193 Aveiro, Portugal
Abstract
This investigation aims at developing an Open Educational Resource (OER) to the learning of
research methods in doctoral programs in Technology-enhanced Learning (TEL). Under the
methodological approach of Educational Design Research, it consists of three phases: context
analysis, development and formative evaluation, and semi-summative evaluation. Preliminary
results from the first phase revealed that design-based research is the most used research method
in TEL and also the one that PhD candidates and PhD holders need more training. Considering
this, in the second phase, the OER prototype about design-based research was designed and
developed using the H5P plugin on Moodle, where it was possible to insert some gamification
strategies. At first, only a section about one research method was developed, so that this
prototype can be formatively evaluated before developing the whole course on research
methods. At the moment, the formative evaluation is being planned to encompass three cycles
of assessment. In the first cycle, the scientific content will be validated by experts; in the second,
a survey will be conducted with master’s and PhD students; and finally, there will be focus
groups with professors, researchers and PhD students. In the last phase, a semi-summative
evaluation of the whole course will be performed to make the last adjustments to the OER and
finally implement and disseminate the results in doctoral programs in TEL.
Keywords 1
Open Educational Resources, doctoral education, Educational Design Research
1. Introduction open licensing that permits their free reuse,
remix and sharing for educational purposes [1].
These educational materials must be under
As information technologies have advanced
open licenses or reside in the public domain,
and been more accessible, a vast number of
free of copyright restrictions, to give users free
digital resources have become more available
permission to adapt and reuse them [2]. Wiley
for those involved in education. Teachers have
[3] claims that content is open not only when it
been using the Internet to spread their materials
is freely available to be used in other contexts.
and courses, and content in digital format has
It is open when it gives everyone permission to
largely increased. Yet, most of these materials
engage with the material through different
are not open to be freely reused, shared or
activities, known as the 5R: retain, reuse,
remixed. With the purpose of overcoming these
revise, remix, and redistribute. The types of
barriers, the Open Educational Resource (OER)
materials can vary from videos, images and
movement was founded to encourage and
textbooks, to podcasts, games, and courses [1].
enable anyone to reuse and share content in an
There are some motivations for educators,
open manner.
institutions and governments to be involved
OER can be defined as any teaching,
with the development and sharing of OER [4, 5,
learning or research materials that make use of
Proceedings of the Doctoral Consortium of Sixteenth European
Conference on Technology Enhanced Learning, September 20–21,
2021, Bolzano, Italy
EMAIL: lorena@ua.pt (L. Sousa)
ORCID: 0000-0001-5970-4027 (L. Sousa)
©️ 2020 Copyright for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
6]. Educators, for example, are able to share 1. to analyse the use of OERs in doctoral
content as well as reuse and adapt it according education training;
to their context, optimizing their time in 2. to identify which characteristics can
creating materials from scratch [7]. contribute to make OERs an engaging
Consequently, by reusing and sharing these solution to doctoral education training;
resources, there might be an improvement in 3. to develop an OER to the learning of
their quality, and the costs of content research methods in doctoral programs in
development can be reduced, which can be a TEL;
benefit to the institutions. From the 4. to evaluate the developed OER on
governmental perspective, OER projects make research methods in doctoral programs in
learning more accessible to society, particularly TEL.
to nontraditional groups of learners, bridging
the gap between non-formal, informal and In the following sections the methodological
formal learning [4, 5, 6]. approach and some preliminary results are
However, according to [8], educators from going to be presented.
higher education institutions are still resistant to
embracing the use of OERs and open 2. Research methodology
educational practices. Besides, there is a lack of
technical skills to select and remix OER
appropriately and a lack of awareness regarding This investigation is being developed under
copyright issues among academics [5, 7]. A the Educational Design Research
survey conducted by the Doctoral Education for methodological approach. Its planning consists
Technology-enhanced Learning (DE-TEL) of three phases [10, 11]: (1) context analysis,
project has also shown that OERs are not (2) development and formative evaluation, and
popular among PhD candidates and PhD (3) semi-summative evaluation. Each phase
will be explained in the next subsections.
holders. When asked which learning sources
they used to deepen their knowledge on TEL
topics, doctoral training topics and research 2.1. Context analysis
methods, OERs were the least voted, being
courses in the PhD program, academic In the first phase, named context analysis, a
publications and supervisor help among the systematic literature review (SLR) is going to
most voted learning sources. be carried out. The main goals of this SLR are
Nine European universities and the to analyse the use of OERs in doctoral
European Association for Technology- education training and identify the main
Enhanced Learning (EA-TEL) created the challenges and barriers when adopting OERs in
Doctoral Education for Technology-enhanced higher education, especially in doctoral
Learning (DE-TEL) project with the aim at programs.
identifying the best practices in doctoral Furthermore, a survey will be conducted
programs in TEL, developing a proposal for a among PhD candidates, researchers and
new program and developing sustainable Open practitioners who study and/or work in the field
Educational Resources (OERs) [9]. The OERs of TEL, the context of this study, to identify
will encompass modules on research methods which characteristics can contribute to make
and key topics in TEL, such as Artificial OERs an engaging solution to doctoral
Intelligence in education and mixed and education training. The survey will be divided
Augmented Reality for TEL. into four sections. In the first section, there will
This thesis, in particular, focuses on the be questions related to personal background.
development of OERs to doctoral education The second section is going to be adapted from
training and, as a contribution to the DE-TEL [12] and ask how relevant some characteristics
project, the OERs are going to cover the are when searching for content for doctoral
research methods for doctoral programs in education training.
TEL. Our main research question is which Sections three and four will regard the
characteristics should an OER developed for factors and formats that would make the
doctoral education training have in order to participant more likely to select a particular
make its use more engaging? and our objectives resource when searching for content. These
are: sections are going to be adapted from a survey
developed by the Hewlett-funded OER In the first cycle, there will be a validation
Research Hub, an open research project based of the scientific content of the prototype. Three
at The Open University (UK) [13]. At the end experts in research methods are going to be
of each section, there will be an open-ended contacted via email to validate the content. The
question in case participants need to add any main goal of this cycle is to verify if the content
additional comments. is appropriate to the learning of research
The data from the closed-ended questions methods in doctoral programs in TEL and make
will be analysed through descriptive analysis on the necessary adjustments, before starting the
SPSS, and the data from the open-ended second cycle. To guide the experts’ validation,
questions will be analysed through content there will be a survey with questions asking if
analysis on NVivo, if necessary. the content is useful, relevant, accurate,
reliable, sufficient and if it meets learners’
2.2. Development and formative needs. There will also be an open-ended
question for additional comments.
evaluation In the second cycle, the general aspects of
the prototype will be evaluated with twenty
In the second phase, called development and PhD candidates. They are going to answer a
formative evaluation, the conceptualization, survey consisting of the User Experience
design and development of the OER prototype Questionnaire (UEQ) and a self-design
will be carried out considering the previous questionnaire.
results from the first phase. Since one of the According to Díaz-Oreiro et al. [14],
objectives of the DE-TEL project is to develop although AttrakDiff has appeared five years
OERs on research methods and key topics in earlier than the UEQ and is the questionnaire
TEL, this study will focus on the development that counts the most uses since 2006, UEQ has
of OERs on research methods. surpassed AttrakDiff in uses per year in 2017
First, a prototype is going to be developed. and 2018. Conducting a quick search on Scopus
This prototype will go through cycles of using the search query “user experience
formative evaluation with PhD candidates, questionnaire” and “attrakdiff”, it is possible to
professors and researchers in the field of TEL, visualize in Figure 1 that this tendency was kept
and as feedback is being received, adjustments in 2019, 2020 and has also continued in 2021
and improvements will be carried out. (data until July).
At the moment, we are studying the
instruments and techniques and also searching
for some validated questionnaires and scales to
formatively evaluate the OER prototype. This
formative evaluation will probably consist of
three cycles, as presented in Table 1. As the
feedback is being received from each cycle,
adjustments and improvements will be carried
out before starting the next cycle.
Table 1 Figure 1: AttrakDiff versus UEQ
Cycles of the formative evaluation
Cycle What? Who? How? Díaz-Oreiro et al. [14] also report that
1 st
scientific 3 experts message standardized user experience questionnaires
content by email were used in combination with other evaluation
2 nd
general 20 PhD UEQ + instruments, such as self-design questionnaires.
As we intend to apply a self-design
aspects students self-design
questionnaire as a complement to the
questions
standardized questionnaire, we chose to use
3 rd
general DE-TEL focus UEQ over AttrakDiff because AttrakDiff is
aspects partners groups applied on its website (attrakdiff.de), without
and PhD the possibility of adding other groups of
students questions. Using UEQ, we will add it on
LimeSurvey together with other questions from and make the last adjustments to the OER if
our self-design questionnaire. On the website necessary. The data will be analysed using
ueq-online.org, it is possible to download the SPSS and NVivo. Finally, the OER is going to
UEQ in more than 30 languages and a tool is be disseminated in the doctoral programs in
also available on an Excel file, completely free TEL in Europe.
of charge, to facilitate the data analysis. Figure 2 summarizes the methodological
As a complement to the UEQ, the self- design of this research:
design questionnaire will encompass some
open-ended questions, such as the most positive
and negative aspects of the prototype in their
opinion, and suggestions for improvement.
These data will be analysed through content
analysis on NVivo.
The data collected in this second cycle will
be triangulated with the data from the
participants’ access to Moodle, where the OER
is going to be integrated. On Moodle, it is
possible to visualize if the participants accessed Figure 2: Methodological design
the platform, how much time they spent on it,
which pages they visualized and interacted
with, which tasks they accomplished, etc. The 3. Preliminary results
SPSS software is going to be used for statistical
data analysis. This section will present some preliminary
In the third cycle, focus groups are going to results regarding the development of the first
be conducted. The purpose of these focus OER prototype on research methods in TEL.
groups is to show some preliminary results and This first version is being developed using the
shed light on the data collected in the second H5P tool and plugin on Moodle to test the
cycle in order to identify the participants’ possibilities of tools that might be useful for the
opinions about the prototype. There will be two development of OER prototypes.
focus groups, one with professors and A survey was conducted by the DE-TEL
researchers, and another with PhD candidates. project to collect information on the current
Each group will have from six to eight practices and challenges of doctoral education
participants, last from 60 to 90 minutes, and in TEL and find out what topics are useful but
will be conducted online, through the Zoom have few educational resources. The survey
platform. The questioning route will include was conducted among students, PhD
questions related to the interactive content candidates, researchers, practitioners who study
created with the H5P plugin on Moodle, the 5R and/or work in the field of TEL, between
activities related to OERs, negative and positive December 2020 and March 2021, through the
aspects, and others, to make improvements to online survey tool LimeSurvey whose link was
the prototype. available on the DE-TEL webpage (ea-
After the prototype goes through these tel.eu/de-tel/survey). The DE-TEL partners are
cycles of formative evaluation, the final version using SPSS and Tableau to analyse the
of the OER will go through a semi-summative collected data.
evaluation before being disseminated in Preliminary results have revealed that 229
doctoral programs in TEL. participants from 40 different countries, most of
them from Europe, answered the survey. When
asked to select the item that best described the
2.3. Semi-summative evaluation general methodological approach of their PhD
research, PhD candidates and PhD holders
In the third and final phase, after we have a reported that design-based research was the
final version of the OER, a semi-summative most used research method, followed by
evaluation is going to be conducted among PhD quantitative and qualitative methods, as can be
candidates, professors and researchers who seen in Figure 3. Regarding the research
study and/or work in the field of TEL. A survey methods that they need more training, design-
will be conducted to conclude this investigation based research was also the most selected
research method by PhD candidates and PhD
holders, followed by quantitative and
qualitative methods as well.
Figure 4: Example of the content type called
drag the words
Figure 3: Research methods use and training
needs (source: data from the DE-TEL project) The content type that was adopted in the
creation of this OER prototype on design-based
As this research aims at developing OERs research was the interactive book. The H5P
to the learning of research methods in TEL and interactive book content type allows authors to
the preliminary results showed that design- create courses, books or tests, combining
based research is the most used research various interactive content types inside of it,
method and also the one which PhD candidates such as interactive videos, course presentations,
and PhD holders need more training, a questions and much more, through multiple
prototype of the OER module about design- pages. Figure 4 illustrates a part of the page
based research is being designed and developed about the characteristics of design-based
using H5P tool. research. On the left-hand side of the image, it
H5P tool (h5p.org), an abbreviation for is possible to visualize the contents of this
HTML5 Package, is a completely free and open interactive book. The numbers in the top right-
technology, which enables anyone to create, hand corner of the image indicate the actual
share and reuse interactive HTML5 content page and the total pages of the book,
more efficiently, without the need for any respectively.
technical knowledge. At the end of the interactive book, there is a
H5P makes it easy to create rich interactive report displaying the learner’s progress
content by providing several content types for throughout the book (Figure 5). There is the
various needs. It is possible to create videos total score, which is the number of points
enriched with interactions, presentations with scored by the learner from his/her correct
interactive slides, drag and drop tasks with answers to the interactive questions; the book
images and text, images with multiple progress, which is the percentage of the
information hotspots, single or multiple choice visualized pages and the performed interactions
questions, interactive books integrating several with the content (it is not possible to get 100%
content types, and many others. of book progress only visualizing the pages);
Figure 4 shows an example of the content and the interactions progress, which is the
type named drag the words, where it is possible percentage of the content that the learner
to create text-based drag and drop tasks. In this interacted with. On the last page of the
activity, learners are asked to drag the interactive book, there is also a summary that
characteristics of design-based research and shows the details of the interactions from each
drop them into the correct explanation. Then, page.
they can check their answers to see how many
responses they got right and a score is
generated. They can also choose between
retrying the task or visualizing its solution.
Figure 4, for instance, presents the solution with Figure 5: Report of an interactive book
the correct responses.
H5P content can be integrated into other
platforms, such as Canvas, Blackboard and
Moodle. This content about design-based
research, specifically, was integrated on the
open Tech4Comp platform
(moodle.tech4comp.dbis.rwth-aachen.de/), on
Moodle, since it is the platform that is being 6. References
used by the DE-TEL project. It is also possible
to create content using the H5P plugin on
[1] UNESCO, Guidelines on the development
Moodle. Thus, some editing and complement to
of open educational resources policies,
this material were performed directly on
2019. URL:
Tech4Comp Moodle platform.
http://oasis.col.org/bitstream/handle/1159
With this prototype, we intend to insert
9/3455/2019_Guideliness_OER_Policy_fi
some OER characteristics according to the
nal_COL_web.pdf?sequence=4&isAllow
answers from the survey which is going to be
ed=y
carried out in the context analysis phase. PhD
[2] D. Wiley, T. J. Bliss, M. McEwen, Open
candidates, researchers and practitioners are
Educational Resources: A Review of the
going to identify which characteristics can
Literature, in: J. M. Spector, M. D. Merrill,
contribute to make OERs an engaging solution
J. Elen, M. J. Bishop (Eds.), Handbook of
to doctoral training and we are going to insert
Research on Educational Communications
these characteristics into the prototype to see
and Technology, 4th. ed., Springer, New
how they work.
York, NY, 2014, pp. 781–789. doi:
10.1007/978-1-4614-3185-5.
4. Final considerations [3] D. Wiley, Defining the “open” in open
content and open educational resources,
With this investigation, we expect to 2014. URL:
contribute to the practical and theoretical https://opencontent.org/definition/
understanding of the process related to the [4] J. Hylén, Open Educational Resources:
development of technological solutions and Opportunities and Challenges, OECD’s
innovations to the area of TEL. Design-based Centre for Educational Research and
Research (or Educational Design Research) is Innovation, Paris, France, 2006.
the most used research method in TEL and its [5] OECD, Giving Knowledge for Free: The
process brings both theoretical and practical Emergence of Open Educational
contributions to the area. Resources, Centre for Educational
Regarding the practical results, we hope that Research and Innovation, Paris, France,
the developed OER can be a useful tool to the 2007.
teaching and learning of design-based research, [6] S. D’Antoni, Open Educational
that it helps PhD candidates to deepen their Resources: reviewing initiatives and
knowledge on this research method, and that its issues, Open Learning: The Journal of
characteristics can contribute to making its use Open, Distance and e-Learning 24 (2009)
more engaging. 3–10. doi: 10.1080/02680510802625443.
Theoretically, we expect that the developed [7] C. Hodgkinson-Williams, Benefits and
OER can raise awareness of the importance of challenges of OER for higher education
the adoption and spread of OERs in education, institutions, Centre for Educational
mainly among doctoral programs, and that this Technologies, University of Cape Town,
study can support the development of other Cape Town, South Africa, 2010.
solutions and innovations to the field of TEL. [8] A. García-Holgado, F. J. García-Peñalvo,
C. Higuera, A. Teixeira, U. Ehlers, J.
Bruton, F. Nascimbeni, N. P. Zea, and D.
5. Acknowledgements Burgos, Promoting Open Education
Through Gamification in Higher
I would like to express my gratitude to my Education: the OpenGame project, in:
supervisors, Professors Luís Pedro and Carlos Eighth International Conference on
Santos. This paper and the research behind it Technological Ecosystems for Enhancing
would not have been possible without their Multiculturality, TEEM’20, ACM, New
support, guidance and valuable comments. York, NY, 2020, pp. 399–404. doi:
This project has received funding from the 10.1145/3434780.3436688.
European Union’s Erasmus Plus programme, [9] M. Fominykh, E. Prasolova-Førland, DE-
grant agreement 2019-1-NO01-KA203- TEL - A European initiative for Doctoral
060280. Education in Technology-Enhanced
Learning, 2020, ERCIM NEWS 120,
Special Theme: Educational Technology.
[10] S. McKenney, T. C. Reeves, Conducting
Educational Design Research, Routledge,
New York, NY, 2012.
[11] T. Plomp, Educational Design Research:
an Introduction, in: T. Plomp, N. Nieveen
(Eds.), Educational Design Research - Part
A: an Introduction, Netherlands Institute
for Curriculum Development (SLO),
Enschede, the Netherlands, 2013, pp. 11–
50.
[12] J. Henriques, Catálogo de características
para análise e avaliação de Recursos
Educacionais Abertos (REA): ferramenta
de avaliação no formato checklist,
Universidade Aberta, Lisboa, Portugal,
2016.
[13] R. Farrow; R. Pitt; B. de los Arcos; L.
Perryman; M. Weller, P. McAndrew,
Impact of OER use on teaching and
learning: data from OER Research Hub
(2013–2014), British Journal of
Educational Technology 46 (2015) 972–
976. doi: 10.1111/bjet.123410.
[14] I. Díaz-Oreiro, G. López, L. Quesada, L.
Guerrero, UX Evaluation with
Standardized Questionnaires in
Ubiquitous Computing and Ambient
Intelligence: A Systematic Literature
Review, Advances in Human-Computer
Interaction (2021). doi:
10.1155/2021/5518722.