Since more and more Massive Open Online Courses (MOOCs) appear, constituting an innovative part of online education, they should also undergo quality assurance to assess their pedagogical design similarly to other online courses. Recently, new studies have appeared presenting quality assurance methods (QA) for assessing the instructional design in MOOCs with pedagogical innovations, like those implementing active learning pedagogies (e.g. collaboration, gamification), which provide designers and instructors with useful strategies in order to design more interactive MOOCs. However, it is not that clear how these efforts are being addressed and up to which point are appropriate to assess active learning pedagogies. Therefore, a Systematic Literature Review (SLR) was conducted to identify the most mature existing MOOC QA methods. Afterwards, an evaluative case study was carried out, based on the Evaluand-oriented Responsive Evaluation Model (EREM), to apply the selected methods to a MOOC implementing active learning pedagogies. As the results suggest, the instruments of the selected QA methods need enrichment to assess effectively the instructional design based on active learning pedagogies, by providing specific questions proper for this kind of MOOCs or, by stating the underlying pedagogical model clearly so that the designers could consider beforehand if it is appropriate or not for their case. The results of the study are a first step to define new, enriched quality assessment methods for MOOCs that apply active learning approaches.
Millions of people are learning in hundreds of Massive Open Online Courses
(MOOCs). MOOC learners are a vast online learning community with diverse
motivational interests; therefore MOOCs should also undergo quality assurance like other
online courses [
Following the RQ question of the present study, we firstly searched the terrain of QA
in MOOCs through an SRL [
In order to address the RQ in real contexts, we selected the “Innovative and Collaborative Learning with ICT” MOOC (CLAT MOOC) carried out by the University of 1 IEEE Xplore Digital Library, available at http://ieeexplore.ieee.org/Xplore/home.jsp . 2 Springer International Publishing, available at https://link.springer.com/ . 3 ACM Digital Library, available at https://dl.acm.org/ . 4 Google Scholar, available at https://scholar.google.gr/ . Pompeu Fabra5 and the University of Valladolid6. The course was designed based on the principles of active learning pedagogies, applying innovative elements in terms of collaboration and gamification, such as innovative group formation policies and collaborative quizzes leading to badge acquisition. CLAT MOOC was a 6-week-course, with 759 total enrollments, delivered by 4 teachers. The course was targeted to innovative pre-service and in-service teachers interested in incorporating collaboration with technology into their own teaching practices.
The assessment tools of the 3 finally selected QA methods were applied in a real case
study of the CLAT MOOC. The design of the evaluative case study was based on the
Evaluand-oriented Responsive Evaluation Model (EREM), which highly encourages
the plurality of data gathering techniques and informants in order to obtain different
perspectives about the Evaluand, thus enriching the evaluation process [
Multiple data gathering techniques were used (see table 1). Firstly, we analyzed
the design of the CLAT MOOC in order to understand its context, which was
enriched with data gathered through informal interviews with the CLAT MOOC
designers. Secondly, we carried out an evaluation of the quality tools, by conducting one
questionnaire and a focus group, addressed to seven (7) research experts in learning
design and assessment in MOOCs. Important findings emerged after the content
analysis of the transcription and the notes, which are presented in the following section
along with the supporting evidence [
N 10 2 7 7
Analysis of the CLAT MOOC instructional design to understand how the active learning strategies were designed and implemented.
Informal interviews with CLAT MOOC designers to analyze the instructional design of the CLAT MOOC. Questionnaires provided to research experts, to gather their initial opinion regarding the QA tools after their application on the CLAT MOOC.
Focus group with research experts to gather their overall opinion about the QA methods.
[CLAT] [INT] [QUEST] [FOC] The main findings derived from the data analysis in terms of the Issue and its corresponding topics to help illuminate the IQ are followingly presented. T1: According to participants, assessment tools could be useful since they: (i) provide guidelines to MOOC designers, (ii) help MOOC designers to identify gaps during the design process, (iii) could work as a good strategy while re-designing a MOOC, considering the results of the quality assessment (see Table 2, [QUEST]-A). T2: Participants were not satisfied with the questions referring to the collaborative activities and active learning pedagogies, supporting that there is a room for further improvement (see Table 2, [QUEST]-B). T3: The effort which was required while applying the quality tools was assessed according to the time consuming requirements, a factor which is affected either by the comprehensible vocabulary, or by the quality tools layout (see Table 2, [QUEST]-C). T4: Participants stated that a QA method by itself cannot help them assess the objectives that a designer sets (see Table 2, [FOC]-A). Also, it was stated the idea of forming more specific questions with special emphasis in the reflection on pedagogical aspects such as collaboration. In this way the quality of particular instructional design initiatives could be assessed more efficiently (see Table 2, [FOC]-B). T5: Most of the participants considered that QA methods were useful since they could reflect on important elements of their MOOCs. However there is space for improvement (see Table 2, [QUEST]-D). T6: Lastly, it was pointed out that one of the main challenges that MOOC teachers’ face is the assessment of students’ final products due to the massive scale of the courses (see Table 2, [FOC]-C).
[QUEST] [FOC] A. The QA tools were useful to identify aspects that should be taken into account when designing a MOOC, and possible weaknesses that have to be addressed. For example, in our case, the aspects related to assessment and the institutional support seems to be the weakest aspect.
B. The questions regarding active pedagogies were mainly focused on participation and collaboration, but there are many other strategies that promote active learning and were not evaluated (e.g., inquiry-based learning, problemsolving, role-playing, game-based learning or gamification).
C. For me Ten-Principle framework dealt with the instructional design and was easier for me to answer its questions. The other two had some questions that I did not find easy to answer.
D. I’d say they helped me to realize that the instructional design had taken into account some “basic” elements of any learning design (e.g., stating objectives, defining assessment criteria, etc.). On the other hand, the assessment of feedback is not very well developed. Important aspects regarding the assessment that affects the quality design in a MOOC are not informed in this instrument. A. I can’t expect from any instrument to say that this objective is correct, because it depends on each MOOC context.
B. For example, for the collaborative learning, I would expect questions such as: “Did you choose specific criteria for forming the groups?” C. Also, we couldn’t assess the student’s final products. We couldn’t provide real assessment for the objectives after the activities submission. 3
The work presented in this paper has explored the area of QA in MOOCs, and its findings have provided initial insights into how the most significant existing methods can be enriched to assess effectively the instructional design in MOOCs implementing active learning pedagogies. To sum up, there is indeed a need of QA methods in order to: (i) detect weaknesses and elements of the course that need improvement, (ii) assess elements that have not been taken into account while designing, (iii) acquire important information while re-designing a MOOC. The QA methods should report clearly the underlying pedagogical model, including clear and simple questions assessing as well specific elements of the active learning pedagogies. Thus, designers could end up with more valid and accurate conclusions about the quality of their MOOCs. A line of future work opened by this research is to consider the above mentioned insights and apply them to the definition of a new, enriched QA method for assessing MOOCs.
This research has been partially funded by the European Regional Development Fund and the National Research Agency of the Spanish Ministry of Science, Innovations and Universities under project grants TIN2017-85179-C3-2-R and TIN201453199-C3-2-R, by the European Regional Development Fund and the Regional Ministry of Education of Castile and Leon under project grant VA257P18, and by the European Commission under project grant 588438-EPP-1-2017-1-EL- EPPKA2-KA. The authors thank the rest of the GSIC-EMIC research team for their valuable ideas and support.