The School of Engineering at Pontificia Universidad Católica (PUC) started developing its MOOC initiative in September 2014. Since then, the school has been producing MOOCs based on specialized topics for Coursera, besides working on MOOCs to complement on-campus courses for Open edX. In parallel with the course production, the school started implementing MOOCbased hybrid pedagogies. In order to organize and systematically analyze these pedagogies, the school proposed a framework that classifies them into three categories: remedial, complementary and flipped. As a preliminary evaluation of the framework, this paper presents a case study of a remedial hybrid MOOCbased pedagogy implemented for a Calculus course. Adoption and learning benefits were defined as key principal indicators (KPIs) to understand the impact of this pedagogy at an institutional level, and inform future institutional decisions. Further work includes validating complementary and flipped categories.
Researchers have reported how students experience hybrid MOOC-based pedagogies, such as flipped classroom or blended learning from the students’ perspective, reporting participant demographics, completion rates, students’ interaction patterns, and learning gains [1, 2]. However, few papers describe the variety of MOOC-based pedagogies derived from MOOC initiatives at an institutional level [3], disregarding the value of information used for guiding institutional decision-making. This paper presents the framework used by School of Engineering at PUC for classifying different MOOC-based hybrid pedagogies into three categories: remedial, complementary, and flipped. Also, it shows how institutional decision makers used KPIs for measuring the impact of these pedagogies on engineering students at PUC.
To organize and systematically analyze the implementation of different
MOOCbased hybrid pedagogies, the school has defined a framework that classifies these
pedagogies as a continuum of two factors (
The level of hybridity indicates the degree of integration between the MOOC and the on-campus course at a curriculum level. A MOOC that is unlinked to a particular course but is always available to the students describes a low level of integration, despite of including resources related with on-campus courses topics. A MOOC that is used by professors as a complementary resource for the course describes a medium level of integration. A high level of integration implies that professors organize their classes around the MOOC, which is used as the main reference of the course.
The level of support offered to students refers to the means (e.g. teacher interventions) that help students progressing on the MOOC. A low level of support implies that the MOOC is always available to students, but there are no teachers/tutors, nor support mechanisms providing feedback to students on their participation. A medium level of support implies that students can ask professors/tutors or institutional staff for help, and there are mechanisms for informing students about their progress. A course that includes classes or tutoring times specially organized to help students progress on their MOOC describes a high level of support. Accordingly, MOOC-based hybrid pedagogies can be organized into three groups: Remedial, Complementary or Flipped.
PUC professional staff involved MOOC initiative proposed two KPIs for measuring the impact of hybrid MOOC-based pedagogies: Adoption and Learning Benefits. The Adoption KPI is defined as a combination of two metrics: students’ participation in the MOOC, and their activity patterns with its content. The students participation refers to the number of interactions that students have with the course videos and problems. The activity patterns refer to the number of times that students access the MOOC over time. This KPI was proposed as a proxy for the level of support provided to students. The Learning Benefits KPI refers to the learning gains of the student’s participating in the MOOC. Learning benefits will be measured differently depending on the category of the course, using the scores of the final exams or analyzing the students’ score within the MOOC content. This KPI was proposed as a proxy for the level of integration between the MOOC and the on-campus course. 3 3.1
As preliminary evaluation of the framework, we analyzed a hybrid MOOC-based pedagogy used for complementing a “Progressions and Summations” unit within a Calculus course. This course consisted on a traditional minicourse of two days length. Freshmen with low scores at an institutional exam on Calculus have to take this remedial course before taking their first course in calculus. The MOOC was composed of a total of 9 videos, 11 lectures and 50 problems. Despite the MOOC was proposed as a complement of the traditional course, it was classified as Remedial because it was not integrated with the course. Although the course was available even before the students took the mini-course, professors did not consider the online resource in their classes and students took the course voluntarily without support.
The school launched the first pilot this year (2015) between January 19th and 30th, in order to analyze the impact of this hybrid pedagogy with freshmen students. The MOOC was announced via e-mail and flyers on the first course presentation session, and it was available since then. All freshmen were registered in the MOOC provider platform by default, but their participation in the MOOC course was voluntary. 3.2
Although the MOOC was open to anyone, the sample for data collection and
analysis was restricted to engineering students at PUC. 650 (N=650) students were
admitted in engineering first year, from which 232 (N=232) had to mandatorily participate
in the traditional Progressions and Summations course. At the end of this course,
students had to take an exam to evaluate their progress. The data gathered from this
sample of students were (
In order to study the students’ Adoption (KPI1), we calculated the percentage of students’ that were active in the MOOC and their activity patterns. First, we analyzed what percentage of the total of students was active on the course (accessed the course at least once and perform some of the problems and videos). Second, we analyzed what percentage of these active students had the requirement to take the mini-course. Then, we analyze the interaction patterns with the MOOC content of this last group. In order to understand the Learning Benefits of the initiative (KPI2), we compared the scores of those active MOOC students who had to take the Progressions and Summations unit with the non-active group. Since the data were not normally distributed (Shapiro-Wilk turned out W=0.89, p<0.05), we performed a Wilcoxon rank-sum test between these two groups. 3.3
Adoption (KP1). The data in indicates that from all the students that needed to take the mini-course (N=232) the 37% (N=86) were active in the MOOC, and 63% (N=146) were not. To further investigate the motivations behind the interaction of the students with the MOOC, we compared the students participating in mini-course using their scores in the PSU, since it works as an indicator of the previous academic trajectory of the students. A Wilcoxon rank-sum test to compare the differences in score between these groups (Active: PSU (mean=770;sd=43); Non-active: PSU (mean= 748; sd=57) showed significant differences (W=4928, p<0.05), indicating that high performing students in the PSU were more prone to interact with the MOOC. In addition, Fig. 2 shows that the two days corresponding to the traditional mini-course was the most active period of the students in the MOOC, mainly for solving problems (10 problems per participant on average over the 2 video viewing).
Fig. 2: Pattern of Activity in the MOOC of students participating in the MOOC. ‘Y’: total number of interactions with the MOOC; ‘X’: corresponds to the dates of the whole course.
Learning Benefits (KPI2). We compare the performance of those who were active in the MOOC with those who were not. The results show that, although the average score of the MOOC active (mean= 0.81; sd=0.14; 95% CI = [0.79, 0.83]) students is slightly higher than the non-active (mean= 0.85; sd=0.11; 95% CI = [0.82, 0.87]), there is no statistically significant difference between the groups (W=5372, p=0.065). The effect size was r=0.17, estimated using rank-biserial correlation. At the end of the course, 230 out of 232 passed the final exam of the minicourse.
This paper presents both the framework and the metrics used by the School of Engineering at PUC in order to systematically organize and analyze the implementation of hybrid MOOC-based pedagogies. Three categories of pedagogies are defined, remedial, complementary and flipped, and two KPIs have been applied to a remedial hybrid pedagogy implemented on a calculus course to measure its impact at an institutional level. The case study shows that the remedial category implies a MOOC adoption lower than 40%. It reports learning benefits, but these benefits are not statistically significant. Future work is needed in order to describe and analyze both flipped and complementary categories and make decisions about how to increase the KPIs.