Trends in Digital Education: Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd A Framework of Hybrid MOOC-based pedagogies Mar Pérez-Sanagustín, Isabel Hilliger, Pablo Schwarzenberg, Denis Parra School of Engineering, Pontificia Universidad Católica de Chile Av. Vicuña Mackenna, 4860, Macul, Santiago (RM), Chile {mar.perez, ihillige, pschwarzenberg, dparra}@ing.puc.cl Abstract. The School of Engineering at Pontificia Universidad Católica (PUC) started developing its MOOC initiative in September 2014. Since then, the school has been producing MOOCs based on specialized topics for Coursera, besides working on MOOCs to complement on-campus courses for Open edX. In parallel with the course production, the school started implementing MOOC- based hybrid pedagogies. In order to organize and systematically analyze these pedagogies, the school proposed a framework that classifies them into three categories: remedial, complementary and flipped. As a preliminary evaluation of the framework, this paper presents a case study of a remedial hybrid MOOC- based pedagogy implemented for a Calculus course. Adoption and learning benefits were defined as key principal indicators (KPIs) to understand the im- pact of this pedagogy at an institutional level, and inform future institutional decisions. Further work includes validating complementary and flipped catego- ries. Keywords: MOOCs, KPIs, Higher Education. 1 Introduction Researchers have reported how students experience hybrid MOOC-based pedagog- ies, such as flipped classroom or blended learning from the students’ perspective, reporting participant demographics, completion rates, students’ interaction patterns, and learning gains [1, 2]. However, few papers describe the variety of MOOC-based pedagogies derived from MOOC initiatives at an institutional level [3], disregarding the value of information used for guiding institutional decision-making. This paper presents the framework used by School of Engineering at PUC for classifying differ- ent MOOC-based hybrid pedagogies into three categories: remedial, complementary, and flipped. Also, it shows how institutional decision makers used KPIs for measur- ing the impact of these pedagogies on engineering students at PUC. 2 MOOC-based hybrid pedagogies framework and KPIs To organize and systematically analyze the implementation of different MOOC- based hybrid pedagogies, the school has defined a framework that classifies these 6 Trends in Digital Education: Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd pedagogies as a continuum of two factors (1) the level of hybridity of the MOOC with the traditional course, and (2) the level of support offered to the students in these courses (Fig. 1). Fig. 1. MOOC-based hybrid Pedagogies framework The level of hybridity indicates the degree of integration between the MOOC and the on-campus course at a curriculum level. A MOOC that is unlinked to a particular course but is always available to the students describes a low level of integration, despite of including resources related with on-campus courses topics. A MOOC that is used by professors as a complementary resource for the course describes a medium level of integration. A high level of integration implies that professors organize their classes around the MOOC, which is used as the main reference of the course. The level of support offered to students refers to the means (e.g. teacher interven- tions) that help students progressing on the MOOC. A low level of support implies that the MOOC is always available to students, but there are no teachers/tutors, nor support mechanisms providing feedback to students on their participation. A medium level of support implies that students can ask professors/tutors or institutional staff for help, and there are mechanisms for informing students about their progress. A course that includes classes or tutoring times specially organized to help students progress on their MOOC describes a high level of support. Accordingly, MOOC-based hybrid pedagogies can be organized into three groups: Remedial, Complementary or Flipped. PUC professional staff involved MOOC initiative proposed two KPIs for measur- ing the impact of hybrid MOOC-based pedagogies: Adoption and Learning Benefits. The Adoption KPI is defined as a combination of two metrics: students’ participation in the MOOC, and their activity patterns with its content. The students participation refers to the number of interactions that students have with the course videos and problems. The activity patterns refer to the number of times that students access the MOOC over time. This KPI was proposed as a proxy for the level of support provided 7 Trends in Digital Education: Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd to students. The Learning Benefits KPI refers to the learning gains of the student’s participating in the MOOC. Learning benefits will be measured differently depending on the category of the course, using the scores of the final exams or analyzing the students’ score within the MOOC content. This KPI was proposed as a proxy for the level of integration between the MOOC and the on-campus course. 3 Example of a MOOC-based hybrid pedagogy: a case study 3.1 Context and data collection As preliminary evaluation of the framework, we analyzed a hybrid MOOC-based pedagogy used for complementing a “Progressions and Summa- tions” unit within a Calculus course. This course consisted on a traditional mini- course of two days length. Freshmen with low scores at an institutional exam on Cal- culus have to take this remedial course before taking their first course in calculus. The MOOC was composed of a total of 9 videos, 11 lectures and 50 problems. Despite the MOOC was proposed as a complement of the traditional course, it was classified as Remedial because it was not integrated with the course. Although the course was available even before the students took the mini-course, professors did not consider the online resource in their classes and students took the course voluntarily without support. The school launched the first pilot this year (2015) between January 19th and 30th, in order to analyze the impact of this hybrid pedagogy with freshmen students. The MOOC was announced via e-mail and flyers on the first course presentation session, and it was available since then. All freshmen were registered in the MOOC provider platform by default, but their participation in the MOOC course was voluntary. 3.2 Participants, sample, data collection and data analysis Although the MOOC was open to anyone, the sample for data collection and anal- ysis was restricted to engineering students at PUC. 650 (N=650) students were admit- ted in engineering first year, from which 232 (N=232) had to mandatorily participate in the traditional Progressions and Summations course. At the end of this course, stu- dents had to take an exam to evaluate their progress. The data gathered from this sample of students were (1) students’ scores in the questions related to the content of Progressions and Summations in the institutional calculus exam; (2) scores obtained at the traditional course; (3) students’ activity and interaction patterns with the MOOC, and (4) students scores on the University Admission Exam of Mathematics (PSU). In order to study the students’ Adoption (KPI1), we calculated the percentage of students’ that were active in the MOOC and their activity patterns. First, we analyzed what percentage of the total of students was active on the course (accessed the course at least once and perform some of the problems and videos). Second, we analyzed what percentage of these active students had the requirement to take the mini-course. 8 Trends in Digital Education: Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd Then, we analyze the interaction patterns with the MOOC content of this last group. In order to understand the Learning Benefits of the initiative (KPI2), we compared the scores of those active MOOC students who had to take the Progressions and Summa- tions unit with the non-active group. Since the data were not normally distributed (Shapiro-Wilk turned out W=0.89, p<0.05), we performed a Wilcoxon rank-sum test between these two groups. 3.3 Results Adoption (KP1). The data in indicates that from all the students that needed to take the mini-course (N=232) the 37% (N=86) were active in the MOOC, and 63% (N=146) were not. To further investigate the motivations behind the inter- action of the students with the MOOC, we compared the students participating in mini-course using their scores in the PSU, since it works as an indicator of the previ- ous academic trajectory of the students. A Wilcoxon rank-sum test to compare the differences in score between these groups (Active: PSU (mean=770;sd=43); Non-active: PSU (mean= 748; sd=57) showed significant differences (W=4928, p<0.05), indicating that high performing students in the PSU were more prone to interact with the MOOC. In addition, Fig. 2 shows that the two days corresponding to the traditional mini-course was the most active period of the students in the MOOC, mainly for solving problems (10 problems per participant on average over the 2 video viewing). Fig. 2: Pattern of Activity in the MOOC of students participating in the MOOC. ‘Y’: total number of interactions with the MOOC; ‘X’: corresponds to the dates of the whole course. Learning Benefits (KPI2). We compare the performance of those who were active in the MOOC with those who were not. The results show that, although the average score of the MOOC active (mean= 0.81; sd=0.14; 95% CI = [0.79, 0.83]) students is slightly higher than the non-active (mean= 0.85; sd=0.11; 95% CI = [0.82, 0.87]), there is no statistically significant difference between the groups (W=5372, p=0.065). The effect size was r=0.17, estimated using rank-biserial correla- tion. At the end of the course, 230 out of 232 passed the final exam of the mini- course. 9 Trends in Digital Education: Selected papers from EC-TEL 2015 Workshops CHANGEE, WAPLA, and HybridEd 4 Conclusions This paper presents both the framework and the metrics used by the School of En- gineering at PUC in order to systematically organize and analyze the implementation of hybrid MOOC-based pedagogies. Three categories of pedagogies are defined, remedial, complementary and flipped, and two KPIs have been applied to a remedial hybrid pedagogy implemented on a calculus course to measure its impact at an insti- tutional level. The case study shows that the remedial category implies a MOOC adoption lower than 40%. It reports learning benefits, but these benefits are not statis- tically significant. Future work is needed in order to describe and analyze both flipped and complementary categories and make decisions about how to increase the KPIs. References 1. Zhang, Y. (2013) Benefiting from MOOC. World Conference on Educational Multime- dia, Hypermedia and Telecommunications, 2013(1): 1372-1377. 2. Griffiths R., et al. (2014). Interactive online learning on campus: Testing MOOCs and oth- er hybrid formats in the University System of Maryland. New York: Ithaka S+R. 3. Delgado Kloos, C. D., Muñoz-Merino, P. J., Alario-Hoyos, C., Ayres, I. E., & Fernández- Panadero, C. (2015). Mixing and Blending MOOC Technologies with Face-to-Face Peda- gogies, Global Engineering Education Conference (EDUCON): 967-971. 10