MOOC-based flipped courses are a new educational trend that has been on the rise over the last few years. However, experimental studies providing empirical evidence about the effectiveness of these educational approaches are scarce. This paper presents the results of a quasi-experiment of a MOOC-based flipped course. The study was conducted on a mandatory third year course on Organizational Behavior in the School of Engineering at Pontificia Universidad Católica de Chile with 316 students organized into experimental and control groups. Both groups had the same teacher, shared the same content and the assessment plan, but the experimental group followed a Flipped Classroom methodology and the control group the traditional lecture methodology. The objective of this quasi-experiment is to compare the learning outcomes of each group and analyze the experimental group's adoption of the initiative. The quasi-experiment lasted an entire semester, and the preliminary findings show that students who participated in the flipped course obtained statistically significantly better grades in the first course exam than students in the control group. Also, the interactions with the MOOC's content in the experimental group show a regular behavior, suggesting that they adopted the class methodology well.
To adapt to the demands and needs of current education landscape and market, lots of Higher Education (HE) institutions started producing Massive Open Online Courses (MOOCs). However, MOOC production has shown to be a resource-demanding activity that challenges current financial models [1]. To make this production sustainable, HE entities have started to explore different ways for benefiting from MOOCs and use them as the vehicle for learning innovation. With this aim, institutions started to implement blended learning initiatives of different types in which locally produced and thirdparty MOOCs are re-used within the traditional curricular activities [2].
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
One of the most frequent practices for MOOC re-use has been the Flipped Classroom. [3] defines the flipped classroom as “the inversion of expectations in the traditional lecture. That is, through the use of computer technology and the Internet (e.g. video recorded lectures), the information-transmission component of a traditional lecture is moved out of class time and replaced by a range of interactive activities designed to entice active learning” [4]. 1.1
Only few studies in the current literature provide empirical evidence about the effectiveness of this educational approach. [5] did a second-order meta-analysis to conclude that high-level, detailed research evaluating the efficacy of specific approaches of blended learning is rare [4]. Even so, studies in which the flipped classroom methodology is applied conclude that this teaching approach is at least as effective as a traditional class, having positive effects in students’ motivation and satisfaction, since students feel more flexible and autonomous. For example, the University of Washington introduced MOOCs for supporting a blended learning methodology in a traditional biology class. They were able to reduce its fail rate from 17% to 4% and the approval rates of the course increased from 14% to 24% since the initiative [6]. Also, at the University of Michigan at Ann Arbor, the math department has flipped its teaching of calculus since the mid-1990s, offering up to 60 small sections of introductory calculus, with a maximum of 32 students in each class, which meet for 80 minutes three days a week [3]. Finally, Eric Mazur, physics professor at Harvard University and one of the main references in this strategy worldwide, flips his courses to create a more active-learning environment [7], and he suggests that the flipped class results in significant learning gains when compared to traditional instruction [7][8].
With this study, we look forward to contributing to this body of literature with a quasi-experiment that evaluates the impact of a MOOC-based Flipped Classroom in terms of students’ adoption and learning outcomes. Specifically, we compare the learning outcomes of students participating in a MOOC-based flipped course (experimental group) with those of students participating in a traditional version of the same course (control group). The presented work is a quasi-experiment because it is an empirical intervention without random assignment between the control and experimental groups.
To evaluate the student’s adoption of the initiative in the experimental group, we analyze their interactions with the course content. Both courses share teacher, content and assessment activities.
The following sections detail the quasi-experiment. Section 2 presents the context and research questions, the course structure and experimental design, the participants sample and the data collection methods and analysis. Section 3 presents the results of the quasi-experiment until the first exam of the semester. Finally, Sections 4 and 5 discuss the obtained results and the main conclusions of this study, reflecting on how this work contributes to expand the literature on empirical studies in flipped classroom experiences.
The study was conducted in a mandatory course for undergraduate engineer students at Pontificia Universidad Católica de Chile called “Organizational Behavior”. The course aims at providing general knowledge of Organizations’ Management. It is mandatory for all engineering students and has 150 students per section in average, with two sections per semester. During the last few years, instructors of the course have tried different strategies to promote class participation, but given the class’s size, the results of these initiatives did not result in a significant increment of students’ motivation, participation nor learning outcomes.
To address this problem, during the second se
The course had three 90-minute sessions per week: Mondays, Wednesdays and Fridays. Monday and Wednesday were reserved for face-to-face sessions, and Fridays were either (1) Seminar Days, where both sections would join in the same classroom and the teacher would invite different people from outside the university to give a lecture; or (2) Exam Days, in which both sections took exams at the same time. Mondays and Wednesdays were flipped in the experimental group, and the same classes were taught through a traditional teaching methodology in the control group. The course structure was designed so as to keep the equivalence between both courses, in terms of content, exercises, and assessment activities to which the students were exposed. Table 1 shows the course’s structure for both sections, as a sequence of phases that consisted of activities for before, during and after each face-to-face session.
Class #1: Mon.
Aug. 21st Class #2: Wed.
Aug. 23rd Class #3: Mon.
Aug. 28th Class #4: Wed.
Aug. 30th Class #5: Mon.
Sept. 4th
A total of 317 students participated in the quasi-experiment, divided into a control group of 148 students (section 1) and an experimental group of 169 (section 2). The students were 21 years old in average. In the control group, there were 37 female and 111 male students, while in the experimental group there were 59 female and 110 males. The participant’s distribution in both groups was random, proposed by the university administration. The teacher selected by convenience which was the control and the experimental group depending on the course schedules. Also, all students were explained of this study, and were asked to sign a consent form allowing us to analyze the data obtained from the quasi-experiment. Students were explained that if they refused to sign, their participation in the course would not be affected in any way, and we would simply leave them out of the analysis. However, all students accepted to participate, and the consent forms were approved by the Ethical Committee of the University. 2.4
Several data gathering techniques for capturing data in and beyond the classroom were used.
To address the first research question (RQ1) about the experimental group students’ adoption, we defined what we called the “Online Metrics”. These metrics are used to understand how students in the experimental group used the Coursera MOOC content. The Online metrics were calculated by analyzing the Experimental Groups’ students’ movements in the MOOC from the beginning of the course until the first exam (from August 16th to September 8th). Specifically, we took the Coursera log-files and analyzed them differentiating between two different moments of the course: (1) before each of the six classes, and (2) during each class (the 90 minutes of the lecture). Students were classified into “more-active” and “less-active”. For this classification, we analyzed the number of movements that each student registered on the MOOC in each period. Less-active students are the ones who have between 5 and 70 movements in the MOOC, and more-active students have between 72 and 381 registered movements in the same period. In addition, we plotted the number of movements in the MOOCs in a bar graph from the beginning to the end of the study to understand the activity patterns in the different periods (see Figure 1 in Section 3.1).
To address the second research question (RQ2) about students’ learning outcomes, we define the “Success metrics” as: 1. The first course exam grades of both control and experimental groups. The exam was the same for both groups and was taken on the same day. 2. Students’ grades on the flipped classes, which averaged the grades each student obtained on the daily class quizzes, the weekly group assignments and the weekly coevaluations. 3. Students’ prior knowledge was determined by analyzing the students’ university grade point average (GPA) up to the semester before taking the course. All these individual scores have a scale from 1 to 7. The Success metrics were analyzed through different statistical analyses with Stata/IC. First, we performed Student t-tests to determine whether the average scores of moreactive students were higher than those of less-active students’ exam and flipped class grades. Then, we performed statistical matching by using propensity scores based on students’ prior knowledge to estimate the effect of being in the experimental group v/s being in the control group on their performance in the first course exam. GPA, sex and year of admission were considered as the covariates. As the treatment, we used the categorical variables of experimental or control group. Students’ scores in the first course exam were defined as the outcome variables. We paired the nearest neighbors with a caliper of 0.25. 3
This section reports on the results obtained from the analysis to address the two research questions. Subsection 3.1 presents an analysis of student’s adoption of the MOOC initiative in the experimental group. Subsection 3.2 presents the results about the effects on students’ learning outcomes in the control group and the experimental group. 3.1
time passed and was reactivated before the exam. Figure 1 shows the activity of the experimental groups’ students in the MOOC during the quasi-experiment up to the first course exam. Students mostly used the MOOC before Mondays’ classes, and the movements decreased by week. Even so, before the exam, the movements in the MOOC reached their highest number of 3.480 movements after Class #6 and before the exam. During each 90-minute class the movements were mainly for answering the corresponding quizzes, and before Wednesdays’ classes (#2, #4 and #6), the movements were mainly for revising Monday’s subject-matters.
“More-active” students spent an average of 54% more time interacting with the
shows that “more-active” students spent between 27% more time in class #5 and 133% more time in the MOOC before the exam than less-active students. Class #4 is not considered because of the few minutes spent on the platform.
the percentage of “more-active” and “less-active” students that fall in each of the quartiles by GPA. The results show that the percentages are similar independent to the quartile they belong to.
Less-Active More-Active
Exam Flipped Classes Exp. vs Control “More-Active” students obtained better scores in the exam and in the flipped class grades than “Less-active” students. Results in Table 5 indicate that there is a statistically significant difference in the scores of the exam between those students that were more-active in the MOOCs and those who were less-active.
exam score than their counterparts in the control group. The experimental group obtained, in average, 0.425 more decimals than the control group, and this difference does not depend on student’s GPA, as can be seen in Table 6. The lessons reported in this section were obtained from pondering on the quasi-experiment’s results on student’s adoption and learning outcomes.
the different courses’ evaluations. Students that were more-active in the MOOC during the three weeks of class had significantly more chances of obtaining better scores in the course exam and flipped class grades than students who did not use the MOOC as much. This result aligns with previous work, which shows that higher activity in the MOOC correlates positively with better grades [10].
Second, students that participated in the flipped classes had significantly more chances of obtaining better scores in the first course exam than students who attended a traditional version of the course. By comparing students with similar prior knowledge through their GPA, we observed that students who were in the experimental group would obtain better results than students in the control group. Although these results expand current knowledge on MOOCs’ effects, the lack of randomization limits the external validity of these findings. In order to test the effect of a flipped course in other educational settings, variables that signal prior knowledge should be identified for each particular context in order to build comparable groups of students.
Third, students tend to be active in the MOOC more intensively before the exam than during the class-period of the quasi-experiment. Also, interactivity patterns show that students tend to be active in the MOOCs more intensively before Monday’s classes that the rest of the week, but this activity is very different between the phases (weeks) of the study. 27% of the movements in the MOOC were registered after class #6 and before the first course exam, which makes us conclude that students probably found the MOOC useful for studying the course’s subject-matters. Even so, when going through a deeper analysis of the resources in the MOOC that students reviewed more in this period, the results show that 27% of the movements on the course were registered before and during class #1, 21% before and during class #3 and 16% before and during class #5. Since the entire MOOC was prepared by the same teachers and used the same resources, future work will be to better understand if this difference is due to the needs of the students on the different course topics, to the quality of the different sections of the MOOC, to students losing interest as they advanced in the course or if it is due to a change in the student’s adoption of the flipped class teaching methodology. 5
Regarding student’s adoption, in this study we have observed that at the start of the semester, students struggle with the new teaching methodology, but manage to adopt it successfully as the course evolves. Also, the analysis showed that although all the content of the course is available in the MOOC from day one, students access the content sequentially, in parallel with the face-to-face course curriculum. Regarding student’s learning outcomes this work concludes that students who were more active in the MOOC show better scores on the course evaluations than those less active. Also, the experimental group obtains better scores in the course’s evaluations than the control group.
This quasi-experiment provided a lot of data that has yet to be analyzed. Therefore, future work will consist on a deeper analysis of all the data that was gathered to obtain important results in student’s adoption and learning outcomes.
In conclusion, this paper has shown that a flipped course with MOOCs for an on-campus engineer course is a complex process that involves many variables and dimensions that need to be considered for the students to use the MOOCs and learn from them. However, the benefits of this effort give those students better chances of succeeding in the corresponding course exams and getting them more involved in their own learning process. This work enhances the empirical research in current literature on flipped courses with MOOCs, and the presented results are aligned with prior research in this area which also conclude that flipped courses are an effective teaching methodology [8].
Acknowledgments: This work was supported by FONDECYT (11150231), CONICYT Beca de Doctorado Nacional 2016, and the LALA Project (grant No. 586120-EPP-1-20171-ES-EPPKA2-CBHE-JP). The LALA project has been funded with the support from the European Commission.