Massive Online Open Courses (MOOCs) have shown potential for promoting learning at scale. A plethora of studies have tapped into in-course learner behaviors to predict learner success. Yet few studies have looked to the relation between performance and engagement during the course and career development after the course. As such, the present study collected and analyzed both in-course data reflecting learner achievement and engagement in a postgraduate-level MOOC, as well as post-course career development. The goal of this research is to examine how career advancers differ from the rest of learners in terms of their performance and engagement within the course. Results showed that career advancers earned better scores and were more likely to complete the course. Career advancers also engaged more frequently with all key course components such as course pages, lecture videos, assignment submissions, and discussion forums. However, while they read the forums, they were not significantly more likely to post, comment, or vote.
1.1
MOOCs have been credited as a disruptive innovation in education [
Despite the promising outlook, few empirical studies have delved into the links
between MOOCs and post-course career development. Much research in MOOCs focuses
on learner achievement and engagement during the course itself, leaving the area of
post-course student longitudinal development relatively untouched. It is observed that
the narrative on MOOCs has shifted from overwhelmingly optimistic from 2011 to
2014 to substantially more critical in 2015 [
The present study collects and analyzes three types of data reflecting both learner incourse achievement and engagement as well as post-course development in the context of a MOOC on educational data mining. This data is integrated to investigate the following question: How do MOOC learners’ performance and engagement during a course relate to their post-course development in an emerging STEM field? 1.2
In the following section, we review relevant literature and studies on the three sources of data utilized by the present study. We first introduce MOOC studies with an emphasis on learner achievement data. Then, we move on to studies that have tapped into learner engagement afforded by the availability of MOOC clickstream data. Lastly, we introduce studies emphasizing longitudinal impact on MOOC learners and present two key frameworks for thinking about MOOC post-course development.
As in traditional school and university settings, performance in MOOCs is generally
measured by calculating learner assignment and test scores [
However, unlike in traditional online learning platforms, many MOOC learners do
not consider completing a course their primary goal [
The availability of clickstream data allow for engineering a multitude of variables reflecting finer-grained learner engagement and revealing insights that would otherwise have stayed hidden.
Clickstream data have been used to derive measures of learner interaction with
course components such as videos, discussion forums, and assignments, which have
been correlated to other data. For example, Guo, Kim, and Rubin [
In addition to performance and interaction within the MOOC course platform, there has
been recent attention to whether a MOOC has longitudinal impact after the end of the
course [
We researched these issues within the context of the MOOC, “Big Data in Education”
[
Clickstream data from the system logs enabled the creation of variables on learners’ interaction with the components of the course environment. In the present analysis, we examined student-level interaction in four categories: page views, lecture videos, discussion forums, and assignments. The clickstream log data was obtained 3 months after the official conclusion of the course (i.e. analyses on student behavior after the official course end include behavior in the 3-month period following that date).
We calculated a global variable reflecting the total number of times each student viewed a page. Additionally, we computed a variable indicating how many times a student accessed the course’s syllabus page in specific.
Variables representing learner interaction with lecture videos were also included. First, we computed a global variable calculating the total number of times a student interacted with any lecture video. An interaction consists of starting, pausing, rewinding, or stopping a video, as well as changing the video speed. In addition, we computed how many times a student interacted with lecture videos for each week, from week 1 to week 8 as well as during the 3 months after the course officially concluded.
To differentiate types of forum interactions, we computed variables for the following forum actions: the number of times a student accessed and read a forum post, posted a new message, responded to an existing message, up-voted a message, or down-voted a message. For each of these five types of variables, we calculated the total number of actions taken during the entire duration covered by the log data, totals for each courseoffering week for all 8 weeks, and a count of interaction in the 3 months after the course officially concluded.
Since the course was designed to allow students to attempt the assignments multiple times, we calculated how many times a student attempted to submit an assignment. As above, we included a global variable to cover the entire course duration, weekly counts for all 8 weeks, plus a count of submissions in the 3 months after the course concluded.
This MOOC contains 8 weekly assignments. For the present analysis, we calculated 8 weekly scores and a final score. According to the course policy, the final score was calculated by averaging the 6 highest grades extracted out of the 8 assignments. Students who earned a final grade of 70% or above are eligible to receive a certificate and therefore are considered to have completed the course. A total of 638 students completed this MOOC and obtained a certificate. 2.4
Our goal in this analysis was to study the relationship between a student’s interaction with the course and their later participation in the community of practice. In partnership with members of the relevant scientific societies and under the oversight of our university’s Institutional Review Board, the first author was provided with a de-identified dataset linking interaction variables to indicators of post-course participation: whether the learner joined a relevant scientific society, and whether the learner submitted a paper to a relevant conference after taking the MOOC.
Learners who have enrolled in the MOOC and later joined the International Educational
Data Mining Society were coded as 1 = members; learners who did not join the society
were coded as 0 = non-members. The time window for joining the society was between
the end of the course and Spring 2016 – an earlier preliminary analysis [
During a two-year time range following the conclusion of the course in late 2013, three primary conferences in the fields covered in the MOOC offered an open call for paper submission and were held: The Seventh International Conference on Educational Data Mining, The Eighth International Conference on Educational Data Mining, and The Fifth International Learning Analytics & Knowledge Conference. Learners who submitted papers to any of the three conferences were coded as 1 = submitters; those who did not were coded as 0 = non-submitters. A total of 148 learners submitted a paper to one of these venues during this period. 2.5
As discussed above, data were collected in three categories: 1) Course interaction; 2) Course performance and 3) Post-course community participation. Our research goal was to investigate how course interaction and performance differ between students who have shown active participation and those who did not.
We conducted a set of two-sample independent t-tests (assuming unequal variance
in all cases, since this assumption was violated in almost all cases) in order to
investigate this question. As this comprises a large number of statistical analyses, we
controlled for multiple comparisons using Storey et al.’s [
The results of comparisons between eventual society members and non-members on their in-course page view actions showed that eventual members made statistically significantly more overall page views, t(47.01) = -4.00, q = .007, d = .800, and viewed the syllabus page t(47.01) = -2.92, q <.001, d = .569 more frequently than non-members. Similarly, paper submitters also viewed statistically significantly more pages, t(147.03) = -3.39, q = .002, d = .800, and also viewed the syllabus more times, t(147.13) = -4.01, q <.001, d = .436, than non-submitters. 3.2
Eventual members performed a statistically significantly higher number of actions related to viewing lecture videos (including, as discussed above, actions such as rewinding or pausing) in total than non-members, t(47.00) = -4.32, q < .001, d = .611. When examined on a weekly basis, members also conducted statistically significantly higher numbers of video watching-related actions than non-members from week 1 to week 7. There were no statistically significant differences for actions during week 8 and after week 8.
Similarly, paper submitters performed a statistically significantly higher number of actions related to viewing lecture videos than non-submitters, t(148.41) = -4.94, q<.001, d = .356. When examined on a weekly basis, paper submitters also conducted statistically significantly higher numbers of video watching-related actions than non-submitters from week 1 to week 7. There was not a statistically significant difference for actions during week 8 and post week 8. 3.3
Results of comparisons between members and non-members on discussion forum reading actions in total and per week show that members read the forums statistically significantly more frequently than non-members, t(47.01) = -3.65, q = .002, d = .705. When examined on a weekly basis (table 1), members also read the forums statistically significantly more than non-members from week 1 to week 8. There was not a statistically significant difference for forum posts after week 8.
Submitters read the forums statistically significantly more than non-submitters, t(147.05) = -2.90, q = .006, d = .327. When examined on a weekly basis (table 2), submitters also read the forums significantly more than non-submitters from week 1 to week 8. A statistically significant difference was also found for forum reading occurring after week 8. Statistically significant differences were not found, whether assessed in total or on weekly basis, when comparing actions such as initiating a post, responding to an existing post, or voting for an existing posts either between members and non-members or between paper submitters and non-submitters. 3.5
Members submitted assignments statistically significantly more frequently than nonmembers, t(47.01) = -3.75, q = .001 , d = .737. When examined on a weekly basis, members also submitted assignments statistically significantly more frequently than non-members from week 1 to week 8, though week 4 was marginally significant with a q value of 0.055. There was not a statistically significant difference in assignment submissions after week 8.
In addition, paper submitters submitted assignments statistically significantly more than paper non-submitters, t(147.12) = -5.10, q < .001 , d = .557. When examined on a weekly basis, paper submitters also submitted assignments more than paper non-submitters from week 1 to week 8. There was not a statistically significant difference for submitting assignments after week 8. 3.6
Members received statistically significantly higher final scores than non-members, t(47.01) = -3.43, q = .002 , d = .643. When examined on a weekly basis, members also received statistically significantly higher scores than non-members from week 1 to week 8.
In addition, paper submitters received statistically significantly higher final scores than non-submitters, t(147.10) = -4.35, q < .001 , d = .447 . When examined on a weekly basis, submitters also received statistically significantly higher scores than non-submitters from week 1 to week 8.
A chi-square test of independence was performed to examine the relation between course completion and post-course society membership status. Table 3 shows that the relationship between completion status and society membership status is significant, 2 (1, N = 49952) = 116.33, p < .001; students who completed the course were more likely to join the society, by more than a factor of ten.
A chi-square test of independence was performed to examine the relation between course completion and post-course paper submission. Table 4 shows that the relation between completion status and paper submitting status is significant, 2 (1, N = 49952) = 176.26, p < .001; students who completed the course were more likely to submit a paper – again, by more than an order of magnitude. The present study collected data reflecting learner post-course development and related it to indicators of their participation, engagement, and performance in the course. In this study, we investigated two post-course development variables: whether a learner joined a relevant scientific community, and whether the learner submitted a paper to a relevant conference, after taking a MOOC in an emerging discipline.
Overall, career advancers (of both types) earned higher scores in the course than non-advancers. They interacted more frequently with key course components including course pages, lecture videos, and discussion forums. However, somewhat surprisingly, career advancers did not post more to the forums or participate more often in reputation voting. They did, however, access the discussion forums more often to read posts than non-advancers. These results indicate that using post-course career development indicators such as joining a professional society and submitting a paper are related to students’ in-course interaction and performance.
Another somewhat surprising result is that posting to the discussion forums was not
a factor differentiating career advancers from non-advancers, even though career
advancers read the forums more often than their classmates. One possible reason is that
content posted by some learners involves basic topics that are not associated with the
types of advanced understanding and skill needed for career advancement. Another
possible explanation is that many of the posts in this class were off-topic or not
particularly professionally relevant, involving the color of the instructor’s shirt or criticizing
the video design [
An interesting – if less surprising – result was the strong link between course
completion and career advancement. Course completion, though widely adopted as a
metric, has received considerable skepticism. Many have noted that completion rates are
low [
Going forward, by understanding the role that MOOCs play in career development, and understanding which student behaviors are associated with positive developments, we can work to make MOOCs more effective at promoting learner success, and help MOOCs reach the high potential attributed to them at their very beginning. 5
This work was supported by the National Sciences Foundation, Award #DRL – 1418378, “Collaborative Research: Modeling Social Interaction and Performance in STEM Learning.” 6