=Paper=
{{Paper
|id=Vol-2437/paper6
|storemode=property
|title=Students’ Adoption and Learning Outcomes in a MOOC-based Flipped Course
|pdfUrl=https://ceur-ws.org/Vol-2437/paper6.pdf
|volume=Vol-2437
|authors=Josefina Hernández Correa,Julio Pertuzé,Isabel Hilliger,Mar Pérez-SanAgustín
|dblpUrl=https://dblp.org/rec/conf/ectel/Hernandez-Correa19
}}
==Students’ Adoption and Learning Outcomes in a MOOC-based Flipped Course==
https://ceur-ws.org/Vol-2437/paper6.pdf
1
Students’ adoption and learning outcomes in a MOOC-
based flipped course
Josefina Hernández Correa1, Julio Pertuzé1, Isabel Hilliger1, Mar Pérez-Sanagustín1, 2
1 School of Engineering, Pontificia Universidad Católica de Chile
Av. Vicuña Mackenna, 4860, Macul, Santiago (RM), Chile
2 Université Paul Sabatier Toulouse III, Institut de Recherche en Informatique de Toulouse
(IRIT), Toulouse, France
jmherna1@uc.cl,{jpertuze, ihillige}@ing.puc.cl,
mar.perez-sanagustin@irit.fr
Abstract. MOOC-based flipped courses are a new educational trend that has
been on the rise over the last few years. However, experimental studies providing
empirical evidence about the effectiveness of these educational approaches are
scarce. This paper presents the results of a quasi-experiment of a MOOC-based
flipped course. The study was conducted on a mandatory third year course on
Organizational Behavior in the School of Engineering at Pontificia Universidad
Católica de Chile with 316 students organized into experimental and control
groups. Both groups had the same teacher, shared the same content and the as-
sessment plan, but the experimental group followed a Flipped Classroom meth-
odology and the control group the traditional lecture methodology. The objective
of this quasi-experiment is to compare the learning outcomes of each group and
analyze the experimental group’s adoption of the initiative. The quasi-experiment
lasted an entire semester, and the preliminary findings show that students who
participated in the flipped course obtained statistically significantly better grades
in the first course exam than students in the control group. Also, the interactions
with the MOOC’s content in the experimental group show a regular behavior,
suggesting that they adopted the class methodology well.
Keywords: MOOCs, Higher Education, Adoption, Learning Outcomes,
Flipped Class, Flipped Course.
1 Introduction
To adapt to the demands and needs of current education landscape and market, lots of
Higher Education (HE) institutions started producing Massive Open Online Courses
(MOOCs). However, MOOC production has shown to be a resource-demanding activ-
ity that challenges current financial models [1]. To make this production sustainable,
HE entities have started to explore different ways for benefiting from MOOCs and use
them as the vehicle for learning innovation. With this aim, institutions started to imple-
ment blended learning initiatives of different types in which locally produced and third-
party MOOCs are re-used within the traditional curricular activities [2].
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
2
One of the most frequent practices for MOOC re-use has been the Flipped Class-
room. [3] defines the flipped classroom as “the inversion of expectations in the tradi-
tional lecture. That is, through the use of computer technology and the Internet (e.g.
video recorded lectures), the information-transmission component of a traditional lec-
ture is moved out of class time and replaced by a range of interactive activities designed
to entice active learning” [4].
1.1 Related work
Only few studies in the current literature provide empirical evidence about the ef-
fectiveness of this educational approach. [5] did a second-order meta-analysis to con-
clude that high-level, detailed research evaluating the efficacy of specific approaches
of blended learning is rare [4]. Even so, studies in which the flipped classroom meth-
odology is applied conclude that this teaching approach is at least as effective as a tra-
ditional class, having positive effects in students’ motivation and satisfaction, since stu-
dents feel more flexible and autonomous. For example, the University of Washington
introduced MOOCs for supporting a blended learning methodology in a traditional biology
class. They were able to reduce its fail rate from 17% to 4% and the approval rates of the
course increased from 14% to 24% since the initiative [6]. Also, at the University of Mich-
igan at Ann Arbor, the math department has flipped its teaching of calculus since the
mid-1990s, offering up to 60 small sections of introductory calculus, with a maximum
of 32 students in each class, which meet for 80 minutes three days a week [3]. Finally,
Eric Mazur, physics professor at Harvard University and one of the main references in
this strategy worldwide, flips his courses to create a more active-learning environment
[7], and he suggests that the flipped class results in significant learning gains when com-
pared to traditional instruction [7][8].
With this study, we look forward to contributing to this body of literature with a
quasi-experiment that evaluates the impact of a MOOC-based Flipped Classroom in
terms of students’ adoption and learning outcomes. Specifically, we compare the learn-
ing outcomes of students participating in a MOOC-based flipped course (experimental
group) with those of students participating in a traditional version of the same course
(control group). The presented work is a quasi-experiment because it is an empirical
intervention without random assignment between the control and experimental groups.
To evaluate the student’s adoption of the initiative in the experimental group, we
analyze their interactions with the course content. Both courses share teacher, content
and assessment activities.
The following sections detail the quasi-experiment. Section 2 presents the context
and research questions, the course structure and experimental design, the participants
sample and the data collection methods and analysis. Section 3 presents the results of
the quasi-experiment until the first exam of the semester. Finally, Sections 4 and 5 dis-
cuss the obtained results and the main conclusions of this study, reflecting on how this
work contributes to expand the literature on empirical studies in flipped classroom ex-
periences.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
3
2 The Quasi-Experiment
2.1 Context and Research Questions
The study was conducted in a mandatory course for undergraduate engineer students at
Pontificia Universidad Católica de Chile called “Organizational Behavior”. The course
aims at providing general knowledge of Organizations’ Management. It is mandatory
for all engineering students and has 150 students per section in average, with two sec-
tions per semester. During the last few years, instructors of the course have tried differ-
ent strategies to promote class participation, but given the class’s size, the results of
these initiatives did not result in a significant increment of students’ motivation, partic-
ipation nor learning outcomes.
To address this problem, during the second semester of 2017, the teacher of the Or-
ganizational Behavior course decided to flip one of the two sections to see if this teach-
ing methodology helped him give a more student-centered class instead of a traditional
expository lecture class. The teacher used an existing MOOC which he had created and
launched a year earlier in Coursera. The MOOC is aligned with the course’s content,
and therefore aims at a broad audience, with no prior knowledge required to enroll.
The quasi-experiment lasted an entire semester, from August 21 st to November 17th of
2017. However, this paper presents the results obtained up to the first course’s evalua-
tion, as a preliminary analysis to inform the institutional administration of the Univer-
sity of the Partial Results obtained so far. Specifically, two research questions were
addressed:
─ RQ1: What is the students’ adoption of the flipped class teaching methodology?
This question aims at studying the students’ use of the MOOC and their interactions
with the course’s content. The goal is to understand when and how they interact with
the MOOC in relation to the course’s structure planned by the teacher.
─ RQ2: What are the effects of participating in a flipped course in terms of stu-
dents’ learning outcomes? This question aims at understanding (1) whether stu-
dents that adopt the teaching methodology better have better scores in the courses’
exams compared with those that don’t adopt it as well; and (2) whether participating
in a flipped class helps students obtain better grades in the course than if assisting a
traditional version of the course.
2.2 Course Structure and components
The course had three 90-minute sessions per week: Mondays, Wednesdays and Fridays.
Monday and Wednesday were reserved for face-to-face sessions, and Fridays were ei-
ther (1) Seminar Days, where both sections would join in the same classroom and the
teacher would invite different people from outside the university to give a lecture; or
(2) Exam Days, in which both sections took exams at the same time. Mondays and
Wednesdays were flipped in the experimental group, and the same classes were taught
through a traditional teaching methodology in the control group. The course structure
was designed so as to keep the equivalence between both courses, in terms of content,
exercises, and assessment activities to which the students were exposed. Table 1 shows
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
4
the course’s structure for both sections, as a sequence of phases that consisted of activ-
ities for before, during and after each face-to-face session.
Table 1. Experimental and Control Groups’ class structure.
Before class During class After class
Monday
Students had to read a The teacher taught that day’s sub- Students had to read a case re-
lecture related to the ject-matter through a traditional lated to the class’s subject-matter
class’s subject-matter. expository methodology, promot- (the same case the experimental
The class was divided ing class participation by asking group read before class) and
Control
into 4 groups, and each questions related to the lecture, were given an individual assign-
group read a different etc. ment where they were to respond
lecture. 4 questions regarding the case
and the class’s subject matter.
This assignment was due before
Wednesday’s class.
Students had to read a Sessions were structured into 2 Students had to review the
case related to the parts: (1) Evaluation, where stu- weekly work performed by their
class’s subject-matter dents had to answer a graded quiz groupmates through online co-
and watch a video lec- to evaluate their work before evaluations.
Experimental
ture in the MOOC ex- class; and (2) the class followed
plaining the subject-mat- the Pyramid pattern [9]. Students
ter of that day’s class. started working in groups of 4,
debating and analyzing the case
they had read before class. Then,
they were regrouped in groups of
8 (from now on “class-groups”)
to compare and propose a final
analysis.
Wednesday
Students had to read the The teacher discussed the case Students had to revise their class-
aforementioned case and through a traditional expository mates’ individual assignments
turn in the individual as- methodology, fomenting class through a Peer Review Process,
signment. participation, asking questions, due before next Monday’s class.
Control
etc. Each student revised 2 class-
mate’s work using a specific ru-
bric created by the teacher. The
final grade was calculated by av-
eraging both Peer Review revi-
sions.
Students had to read a Sessions were structured into 2 Students had to turn in a group
lecture related to the parts: (1) Evaluation, where stu- assignment that reflected their
class’s subject-matter dents had to answer a graded quiz work in class (due every Friday).
(the same lectures the to evaluate their work before Also, students had to review the
Experimental
control group read be- class; and (2) the class would fol- weekly work performed by their
fore class). The class low the Jigsaw pattern. Students class-groupmates through online
was divided into 4 “lec- were assigned in class-groups of co-evaluations.
ture-groups” (different eight, where only two students
from the class-groups), had read each case. The group
and each lecture-group had to solve a case based on the
was assigned a different discussion generated by the dif-
lecture. ferent cases they each read
(hence: the term Jigsaw Puzzle).
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
5
Table 2 shows the contents (lectures, cases and video-lectures) worked in each of
the sessions analyzed in this paper to give an idea about the course’s subject-matters.
All lectures and cases are from Harvard Business Review. As can be seen in Table 2,
the selection of contents and learning objectives for the experimental and control groups
are the same and only differ in the order they are being taught throughout the week,
given the different methodological approaches. Each case and lecture’s complete bib-
liography can be found in:
https://drive.google.com/file/d/1exO94q9zsDADy2ItH4trH-
KizWzerpglo/view?usp=sharing
Table 2. Experimental and Control group’s lectures, cases and video-lectures.
Control Group (Section 1) Experimental Group (Section 2)
Class #1: Lectures: Video-Lecture:
Mon. • Lecture-Group 1: "Your strategy • MOOC Chapter “An Organization’s
Aug. 21st needs a strategy." Strategic Project” available in Coursera’s
• Lecture-Group 2: "Pipelines, plat- MOOC called “Effective Organization’s
forms, and the new rules of strategy." Management”.
• Lecture-Group 3: "The big lie of (https://www.coursera.org/learn/gestion-or-
strategic planning." ganizaciones-efectivas/home/week/4)
• Lecture-Group 4: "Bringing science Case:
to the art of strategy." • “Apple Inc. in 2015.”
Class #2: Case: Lectures:
Wed. • Same as Experimental Groups’ case • Same as Control Groups’ lectures for
Aug. 23rd for Monday, August 21st. Monday, August 21st.
Class #3: Lectures: Video-Lecture:
Mon. • Lecture-Group 1: "The multiunit en- • MOOC Chapter “Designing Effective
Aug. 28th terprise." Organizations” available in Coursera’s
• Lecture-Group 2: "How Strategy MOOC called “Effective Organization’s
Shapes Structure." Management”
• Lecture-Group 3: "Beyond the Ho- (https://www.coursera.org/learn/gestion-or-
lacracy HYPE." ganizaciones-efectivas/home/week/5)
• Lecture-Group 4: "First, let's fire all Case:
the managers." • “Appex Corp.”
Class #4: Case: Lectures:
Wed. • Same as Experimental Groups’ case • Same as Control Groups’ lectures for
Aug. 30th for Monday, August 28th. Monday, August 28th.
Class #5: Lectures: Video-Lecture:
Mon. • Lecture-Group 1: “Managing Your • MOOC Chapter “The key to organiza-
Sept. 4th Mission-Critical Knowledge.” tional learning” available in Coursera’s
• Lecture-Group 2: “Strategies for MOOC called “Effective Organization’s
Learning from failure.” Management”
• Lecture-Group 3: "Why Organiza- (https://www.coursera.org/learn/gestion-or-
tions Don’t Learn." ganizaciones-efectivas/home/week/6)
• Lecture-Group 4: "Is yours a learn- Case:
ing organization?" • “Managing knowledge and learning at
NASA and the Jet Propulsion Laboratory
(JPL)”.
Class #6: Case: Lectures:
Wed. • Same as Experimental Groups’ case • Same as Control Groups’ lectures for
Sept. 6th for Monday, September 4th Monday, September 4th.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
6
2.3 Participants and sample
A total of 317 students participated in the quasi-experiment, divided into a control
group of 148 students (section 1) and an experimental group of 169 (section 2). The
students were 21 years old in average. In the control group, there were 37 female and
111 male students, while in the experimental group there were 59 female and 110 males.
The participant’s distribution in both groups was random, proposed by the university
administration. The teacher selected by convenience which was the control and the ex-
perimental group depending on the course schedules. Also, all students were explained
of this study, and were asked to sign a consent form allowing us to analyze the data
obtained from the quasi-experiment. Students were explained that if they refused to
sign, their participation in the course would not be affected in any way, and we would
simply leave them out of the analysis. However, all students accepted to participate,
and the consent forms were approved by the Ethical Committee of the University.
2.4 Data Collection and Analysis
Several data gathering techniques for capturing data in and beyond the classroom were
used.
To address the first research question (RQ1) about the experimental group stu-
dents’ adoption, we defined what we called the “Online Metrics”. These metrics are
used to understand how students in the experimental group used the Coursera MOOC
content. The Online metrics were calculated by analyzing the Experimental Groups’
students’ movements in the MOOC from the beginning of the course until the first exam
(from August 16th to September 8th). Specifically, we took the Coursera log-files and
analyzed them differentiating between two different moments of the course: (1) before
each of the six classes, and (2) during each class (the 90 minutes of the lecture). Stu-
dents were classified into “more-active” and “less-active”. For this classification, we
analyzed the number of movements that each student registered on the MOOC in each
period. Less-active students are the ones who have between 5 and 70 movements in the
MOOC, and more-active students have between 72 and 381 registered movements in
the same period. In addition, we plotted the number of movements in the MOOCs in a
bar graph from the beginning to the end of the study to understand the activity patterns
in the different periods (see Figure 1 in Section 3.1).
To address the second research question (RQ2) about students’ learning out-
comes, we define the “Success metrics” as:
1. The first course exam grades of both control and experimental groups. The exam
was the same for both groups and was taken on the same day.
2. Students’ grades on the flipped classes, which averaged the grades each student ob-
tained on the daily class quizzes, the weekly group assignments and the weekly co-
evaluations.
3. Students’ prior knowledge was determined by analyzing the students’ university
grade point average (GPA) up to the semester before taking the course. All these
individual scores have a scale from 1 to 7.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
7
The Success metrics were analyzed through different statistical analyses with Stata/IC.
First, we performed Student t-tests to determine whether the average scores of more-
active students were higher than those of less-active students’ exam and flipped class
grades. Then, we performed statistical matching by using propensity scores based on
students’ prior knowledge to estimate the effect of being in the experimental group v/s
being in the control group on their performance in the first course exam. GPA, sex and
year of admission were considered as the covariates. As the treatment, we used the
categorical variables of experimental or control group. Students’ scores in the first
course exam were defined as the outcome variables. We paired the nearest neighbors
with a caliper of 0.25.
3 Results
This section reports on the results obtained from the analysis to address the two research
questions. Subsection 3.1 presents an analysis of student’s adoption of the MOOC ini-
tiative in the experimental group. Subsection 3.2 presents the results about the effects
on students’ learning outcomes in the control group and the experimental group.
3.1 Adoption of the flipped class teaching methodology
The activity in the MOOC of students’ in the experimental group decreased as
time passed and was reactivated before the exam. Figure 1 shows the activity of the
experimental groups’ students in the MOOC during the quasi-experiment up to the first
course exam. Students mostly used the MOOC before Mondays’ classes, and the move-
ments decreased by week. Even so, before the exam, the movements in the MOOC
reached their highest number of 3.480 movements after Class #6 and before the exam.
During each 90-minute class the movements were mainly for answering the corre-
sponding quizzes, and before Wednesdays’ classes (#2, #4 and #6), the movements
were mainly for revising Monday’s subject-matters.
4.000 3.480
3.500 3.063
3.000 2.468
Movements
2.500
2.000 1.758
1.500
1.000 409
500 172 220 239 52 276 241 285 229
0
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
8
Fig. 1. Total amount of movements in the MOOC performed by all the students in the experi-
mental group before each class, during each class and before the exam. In green, we show the
movements before the exam, in orange during the class and in blue before each class.
“More-active” students spent an average of 54% more time interacting with the
MOOC throughout the quasi-experiment than “less-active” students. Table 3
shows that “more-active” students spent between 27% more time in class #5 and 133%
more time in the MOOC before the exam than less-active students. Class #4 is not con-
sidered because of the few minutes spent on the platform.
Table 3. Average of time (in minutes) that “Less Active” and “More-Active” students spent in-
teracting with the course content in the different course periods.
Before Class (BC) During Class (DC)
Less- More- Total Less- More- Total Total
Active Active BC Active Active DC
34,5 49,1 7,1 7,8
Class #1 83,7 14,92 98,62
(41%) (59%) (48%) (52%)
11,2 15,5 6 6,4
Class #2 26,8 12,56 39,36
(42%) (58%) (48%) (52%)
37,2 55,6 4,3 4,3
Class #3 92,97 8,67 101,64
(40%) (60%) (50%) (50%)
1,4 8,9 6,3 6,2
Class #4 10,38 12,58 22,96
(14%) (86%) (50%) (50%)
23,4 30,3 5,8 5,7
Class #5 53,83 11,54 65,37
(44%) (56%) (50%) (50%)
14,2 19,3 5,7 6,1
Class #6 33,57 11,97 45,54
(42%) (58%) (48%) (52%)
32,9 77,8
Exam 110,76 110,76
(30%) (70%)
155,1 256,8 35,45 36,79
Total 412,01 72,24 484,25
(38%) (62%) (49%) (51%)
3.2 Students’ Learning Outcomes
The movements in the MOOCs do not depend on student’s GPA. Table 4 shows
the percentage of “more-active” and “less-active” students that fall in each of the quar-
tiles by GPA. The results show that the percentages are similar independent to the quar-
tile they belong to.
Table 4. Adoption rates according to GPA quartiles
Q1 Q2 Q3 Q4
Less-Active 56,1% 54,8% 52,4% 40,5%
More-Active 41,5% 42,9% 47,6% 59,5%
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
9
“More-Active” students obtained better scores in the exam and in the flipped
class grades than “Less-active” students. Results in Table 5 indicate that there is a
statistically significant difference in the scores of the exam between those students that
were more-active in the MOOCs and those who were less-active.
Table 5. Course grades regarding the students’ use of the MOOC
Score
Group N SD P-value
Mean
Less-Active 85 4,35 0.95
Exam 0,02161
More-Active 81 4,69 0.92
Flipped Less-Active 85 5,87 0.37
0,00888
Classes More-Active 81 6,02 0.34
Students in the experimental group had statistically higher marks in the course
exam score than their counterparts in the control group. The experimental group ob-
tained, in average, 0.425 more decimals than the control group, and this difference does not
depend on student’s GPA, as can be seen in Table 6.
Table 6. Course grades regarding the students’ use of the MOOC
AI. Stand. Confidence
Coefficient z P-value
Error Interval
Exp. vs
0.3130444 0.1307186 2.39 0.017 0.0568407 - 0.5692482
Control
4 Discussion
The lessons reported in this section were obtained from pondering on the quasi-exper-
iment’s results on student’s adoption and learning outcomes.
First, Students that better adopt the teaching methodology are more prepared for
the different courses’ evaluations. Students that were more-active in the MOOC during
the three weeks of class had significantly more chances of obtaining better scores in the
course exam and flipped class grades than students who did not use the MOOC as much.
This result aligns with previous work, which shows that higher activity in the MOOC cor-
relates positively with better grades [10].
Second, students that participated in the flipped classes had significantly more
chances of obtaining better scores in the first course exam than students who attended
a traditional version of the course. By comparing students with similar prior knowledge
through their GPA, we observed that students who were in the experimental group would
obtain better results than students in the control group. Although these results expand cur-
rent knowledge on MOOCs’ effects, the lack of randomization limits the external validity
of these findings. In order to test the effect of a flipped course in other educational settings,
variables that signal prior knowledge should be identified for each particular context in order
to build comparable groups of students.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
10
Third, students tend to be active in the MOOC more intensively before the exam
than during the class-period of the quasi-experiment. Also, interactivity patterns
show that students tend to be active in the MOOCs more intensively before Mon-
day’s classes that the rest of the week, but this activity is very different between
the phases (weeks) of the study. 27% of the movements in the MOOC were registered
after class #6 and before the first course exam, which makes us conclude that students prob-
ably found the MOOC useful for studying the course’s subject-matters. Even so, when going
through a deeper analysis of the resources in the MOOC that students reviewed more in this
period, the results show that 27% of the movements on the course were registered before
and during class #1, 21% before and during class #3 and 16% before and during class
#5. Since the entire MOOC was prepared by the same teachers and used the same re-
sources, future work will be to better understand if this difference is due to the needs of
the students on the different course topics, to the quality of the different sections of the
MOOC, to students losing interest as they advanced in the course or if it is due to a
change in the student’s adoption of the flipped class teaching methodology.
5 Conclusions and Future Work
Regarding student’s adoption, in this study we have observed that at the start of the
semester, students struggle with the new teaching methodology, but manage to adopt it
successfully as the course evolves. Also, the analysis showed that although all the con-
tent of the course is available in the MOOC from day one, students access the content
sequentially, in parallel with the face-to-face course curriculum. Regarding student’s
learning outcomes this work concludes that students who were more active in the
MOOC show better scores on the course evaluations than those less active. Also, the
experimental group obtains better scores in the course’s evaluations than the control
group.
This quasi-experiment provided a lot of data that has yet to be analyzed. Therefore,
future work will consist on a deeper analysis of all the data that was gathered to obtain
important results in student’s adoption and learning outcomes.
In conclusion, this paper has shown that a flipped course with MOOCs for an on-campus
engineer course is a complex process that involves many variables and dimensions that need
to be considered for the students to use the MOOCs and learn from them. However, the
benefits of this effort give those students better chances of succeeding in the corresponding
course exams and getting them more involved in their own learning process. This work
enhances the empirical research in current literature on flipped courses with MOOCs, and
the presented results are aligned with prior research in this area which also conclude that
flipped courses are an effective teaching methodology [8].
Acknowledgments: This work was supported by FONDECYT (11150231), CONICYT
Beca de Doctorado Nacional 2016, and the LALA Project (grant No. 586120-EPP-1-2017-
1-ES-EPPKA2-CBHE-JP). The LALA project has been funded with the support from the
European Commission.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)
11
References
[1] P. M. Nissenson and A. C. Shih, “MOOC on a budget: Development and
implementation of a low-cost MOOC at a state university,” Comput. Educ. J., 2016.
[2] M. Pérez-Sanagustín, I. Hilliger, C. Alario-Hoyos, C. D. Kloos, and S. Rayyan, “H-
MOOC framework: reusing MOOCs for hybrid education,” J. Comput. High. Educ.,
2017.
[3] D. Berrett, “How ‘Flipping’ the Classroom Can Improve the Traditional Lecture,” Educ.
Dig. Essent. Readings Condens. Quick Rev., 2012.
[4] L. Abeysekera and P. Dawson, “Motivation and cognitive load in the flipped classroom:
definition, rationale and a call for research,” High. Educ. Res. Dev., vol. 34, no. 1, pp.
1–14, 2015.
[5] R. M. Tamim, R. M. Bernard, E. Borokhovski, P. C. Abrami, and R. F. Schmid, “What
Forty Years of Research Says About the Impact of Technology on Learning: A Second-
Order Meta-Analysis and Validation Study A Second-Order Meta-Analysis and
Validation Study,” Source Rev. Educ. Res. Rev. Educ. Res., 2011.
[6] N. Aronson and K. Arfstrom, “Flipped Learning in Higher Education,” Flip. Learn.
Netw., pp. 1–4, 2013.
[7] K. P. Fulton, Time for Learning: Top 10 Reasons Why Flipping the Classroom Can
Change Education. 2017.
[8] C. H. Crouch and E. Mazur, “Peer Instruction: Ten years of experience and results,” Am.
J. Phys., 2001.
[9] D. Hernández-Leo, J. I. Asensio-Pérez, and Y. Dimitriadis, “Computational
representation of collaborative learning flow patterns using IMS learning design,” in
Educational Technology and Society, 2005.
[10] J. Hernandez, M. F. Rodriguez, I. Hilliger, and M. Perez-Sanagustin, “MOOCs as a
remedial complement: Students' adoption and learning outcomes,” IEEE Transactions
on Learning Technologies, 2018.
Copyright © 2019 for this paper by its authors. Use permitted under Creative Com-
mons License Attribution 4.0 International (CC BY 4.0)