Introduction

Millions of people are learning in hundreds of Massive Open Online Courses (MOOCs). MOOC learners are a vast online learning community with diverse motivational interests; therefore MOOCs should also undergo quality assurance like other online courses [1]. Recently, some new studies have appeared presenting quality assurance (QA) methods assessing specific aspects of the instructional design in MOOCs, such us: collaboration, feedback course overview, learning objectives, as-sessment, instructional materials, learner interaction and engagement, learner support and accessibility [2,3]. However, most existing QA approaches for MOOCs do not cater for the increasing use of active learning pedagogies (e.g. collaboration, gamification), that aim at proposing richer learning experiences that go beyond onesize-fits all approaches [4][5][6]. This paper addresses this research gap in the QA for MOOCs implementing active learning pedagogies, by exploring the following research question (RQ): How can we assess the instructional design quality of a MOOC implementing active learning pedagogies?

In order to answer this RQ, we firstly carried out a Systematic Literature Review (SLR) to identify and select the most mature existing MOOC QA methods. Secondly, an evaluative case study was carried out, based on the Evaluand-oriented Responsive Evaluation Model (EREM), to apply the selected methods to a MOOC implementing active learning pedagogies and evaluate their instruments. Finally, the findings and the results of the data analysis are reported, followed by the main conclusions and recommended lines for future work.

Related Work

Following the RQ question of the present study, we firstly searched the terrain of QA in MOOCs through an SRL [7] aiming to find out: (i) which QA methods have been proposed so far? (ii) which are their main strengths and weaknesses while assessing the instructional design of a MOOC? Specific electronic databases were selected; IEEE Xplore Digital Library1 , Springer Link2 , ACM Digital Library3 and Google Scholar 4 , and particular search strings were used: "MOOCs quality", "pedagogical quality in MOOCs" and "instructional design QA in MOOCs", including journal publications, conference proceedings, books and book chapters. The inclusion criteria were publications: (i) written in English, (ii) referring to QA and accreditation, (iii) analyzing quality frameworks and (iv) presenting quality indicators. The exclusion criterion was not referring at all to online learning QA. The 10 initially identified QA methods were assessed to find out which of them better fulfilled the following criteria: (i) focus on MOOCs, (ii) inclusion of assessment instruments, (iii) focus on instructional design, (iv) evidence of testing of the framework, (v) inclusion of the process/methodology of analysis, and (vi) assessment of active learning pedagogies. 3 QA methods fulfilled most of the criteria and were finally selected in order to proceed with the case study; the Ten-principle framework [8], the OpenupED Quality Label [9] and Quality Matters [10].

Methodology

Context

In order to address the RQ in real contexts, we selected the "Innovative and Collaborative Learning with ICT" MOOC (CLAT MOOC) carried out by the University of Pompeu Fabra5 and the University of Valladolid6 . The course was designed based on the principles of active learning pedagogies, applying innovative elements in terms of collaboration and gamification, such as innovative group formation policies and collaborative quizzes leading to badge acquisition. CLAT MOOC was a 6-week-course, with 759 total enrollments, delivered by 4 teachers. The course was targeted to innovative pre-service and in-service teachers interested in incorporating collaboration with technology into their own teaching practices.

The case study

The assessment tools of the 3 finally selected QA methods were applied in a real case study of the CLAT MOOC. The design of the evaluative case study was based on the Evaluand-oriented Responsive Evaluation Model (EREM), which highly encourages the plurality of data gathering techniques and informants in order to obtain different perspectives about the Evaluand, thus enriching the evaluation process [11]. We assumed that this research method was the most appropriate, due to the mixed data gathering techniques and the multiple informants needed in this phase of the study. The Issue (the evaluative tension in EREM terminology) of the case study was defined as: To what extent did the selected QA methods help assess the instructional design quality and the active learning pedagogies of the CLAT MOOC? To help illuminate our research question, we performed an anticipatory data reduction process during the evaluation design (see figure 1), where the Issue was split into more concrete topics (T). Each topic was explored through a number of Informative Questions (IQ), in order to give answers to the issue of the CLAT MOOC study. Multiple data gathering techniques were used (see table 1). Firstly, we analyzed the design of the CLAT MOOC in order to understand its context, which was enriched with data gathered through informal interviews with the CLAT MOOC designers. Secondly, we carried out an evaluation of the quality tools, by conducting one questionnaire and a focus group, addressed to seven (7) research experts in learning design and assessment in MOOCs. Important findings emerged after the content analysis of the transcription and the notes, which are presented in the following section along with the supporting evidence [12].

Table 1. Data sources and labels used to quote them along the study

Data Sources

N Description of technique Label

CLAT MOOC design

10 Analysis of the CLAT MOOC instructional design to understand how the active learning strategies were designed and implemented.

[CLAT]

Informal Interviews 2 Informal interviews with CLAT MOOC designers to analyze the instructional design of the CLAT MOOC.

[INT]

Questionnaire

7

Questionnaires provided to research experts, to gather their initial opinion regarding the QA tools after their application on the CLAT MOOC.

[QUEST]

Focus group 7

Focus group with research experts to gather their overall opinion about the QA methods.

[FOC]

Findings

The main findings derived from the data analysis in terms of the Issue and its corresponding topics to help illuminate the IQ are followingly presented. T1: According to participants, assessment tools could be useful since they: (i) provide guidelines to MOOC designers, (ii) help MOOC designers to identify gaps during the design process, (iii) could work as a good strategy while re-designing a MOOC, considering the results of the quality assessment (see Table 2, [QUEST]-A). T2: Participants were not satisfied with the questions referring to the collaborative activities and active learning pedagogies, supporting that there is a room for further improvement (see Table 2, [QUEST]-B). T3: The effort which was required while applying the quality tools was assessed according to the time consuming requirements, a factor which is affected either by the comprehensible vocabulary, or by the quality tools layout (see Table 2, [QUEST]-C). T4: Participants stated that a QA method by itself cannot help them assess the objectives that a designer sets (see Table 2, [FOC]-A). Also, it was stated the idea of forming more specific questions with special emphasis in the reflection on pedagogical aspects such as collaboration. In this way the quality of particular instructional design initiatives could be assessed more efficiently (see Table 2, [FOC]-B). T5: Most of the participants considered that QA methods were useful since they could reflect on important elements of their MOOCs. However there is space for improvement (see Table 2, [QUEST]-D). T6: Lastly, it was pointed out that one of the main challenges that MOOC teachers' face is the assessment of students' final products due to the massive scale of the courses (see Table 2, [FOC]-C).

Conclusions and future work

The work presented in this paper has explored the area of QA in MOOCs, and its findings have provided initial insights into how the most significant existing methods can be enriched to assess effectively the instructional design in MOOCs implementing active learning pedagogies. To sum up, there is indeed a need of QA methods in order to: (i) detect weaknesses and elements of the course that need improvement, (ii) assess elements that have not been taken into account while designing, (iii) acquire important information while re-designing a MOOC. The QA methods should report clearly the underlying pedagogical model, including clear and simple questions assessing as well specific elements of the active learning pedagogies. Thus, designers could end up with more valid and accurate conclusions about the quality of their MOOCs. A line of future work opened by this research is to consider the above mentioned insights and apply them to the definition of a new, enriched QA method for assessing MOOCs.