In 2016 UNSW Australia (The University of New South Wales) designed and developed the Massive Open Online Course (MOOC) 'Through Engineer's Eyes: Engineering Mechanics through experiment, analysis and design' (TEE). Two iterations of TEE were run that year on the FutureLearn (FL) platform. The data generated from student engagement with the MOOC was examined after the first course offering, and this informed various design changes aimed to improve learner experience in the second and future offerings of the course. This paper provides useful and usable insight into MOOC design, development and ways that data analytics can inform the continuous improvement over time.
This paper examines the evolution of an engineering mechanics MOOC offered on the
FL platform across two course offerings. MOOC platforms have offered access to
courses to anyone in the world with an internet connection and an interest in learning.
MOOCs have traditionally attracted large enrolment numbers, usually in the tens of
thousands
FutureLearn data: what we currently have, what we are learning and how it is demonstrating learning in MOOCs. Workshop at the 7th International Learning Analytics and Knowledge Conference. Simon Fraser University, Vancouver, Canada, 13-17 March 2017, p. 63-73.
Copyright © 2017 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.
A key aim of TEE was to introduce learners to the world-view of an engineer by demonstrating how engineers use analysis to understand their surroundings and to predict the behaviour of the things they design. TEE course content was designed to be anchored in practical reality and to provide learners with experience on which to base their studies of classical analysis. The course was designed to be accessible to a global audience, this informed the design of experiments that learners could conduct in their own time. The experiments aimed to spark learner interest in the topics and ground these in physical reality. Experiments demonstrated the use of commonly available items, such as rubber bands, cardboard, string and toy vehicles to explain complex engineering concepts. Analysis activities helped to explain the experiments and lead learners through the design process. 2.1
Initial design considerations of TEE centred on what the potential target audience
would be for the course. A previous MOOC on a related subject suggested the
demographic for the course would cover an age range from 16 to over 65, a range generally
consistent with other related MOOCs. This provided a challenge in how to make TEE
accessible and interesting to this broad and eclectic global cohort. One early decision
was to focus TEE on teaching basic engineering mechanics, with a style that would be
friendly, authoritative and fun. Learners would however need knowledge of basic
trigonometry and algebra. Although this distinction has been challenged
The course was modularised following a seven-week structure that covered the topics in Table 1.
Learners were led through the course by short 1-4 minute videos (over 50 in total), accompanied by supporting text resources. Each week an introductory video set the scene, followed by a video/s on the week's experiment. If learners decided not to physically attempt the experiments themselves, they nevertheless could identify with the activities because of the familiar nature of the equipment used. Topic Elastic properties Forces that act at a point Forces on a rigid body Centre of gravity Friction Work and energy Impulse and momentum
The course design also incorporated several tools and resources into the FL platform.
These were designed to promote collaboration and sharing amongst learners, provide
rich interactive and adaptive courseware and promote learning consolidation. One of
these tools included ‘Padlets’, which were added to each experiment. These were virtual
walls that allowed learners to share images, videos and descriptions of any experiments
that they attempted. Links to these could also be added to the discussion forums to elicit
further discussion amongst learners. The structure of the course was also complimented
with the inclusion on-line SmartSparrow Adaptive Tutorials. SmartSparrow is a
learning design platform that enabled the incorporation of rich, interactive and adaptive
elearning courseware
Large amounts of data were generated from learners' interactions, both with the course and with fellow learners. TEE learner data was sourced from both the FL platform and SmartSparrow. The FL platform is a pioneer in providing near real-time data of its published courses. The data sets are updated daily, and this creates an opportunity to analyse learner interaction and behaviour while a course is active. The available data sets for the TEE course included campaigns, comments, enrolments, question response, step activity and team members. The purpose of each file is described in Table 2. These data sets are downloadable as CSV (Comma Separated Values) files.
There are two sources of demographic information in FL: a profile survey that asks learners for their basic information such as age, gender and level of education, and a pre-course survey that focuses on learner motivation to enrol and goals. As both surveys are optional, the information gleaned should be used with caution as the responding sample (approximately 10% in both iterations) might not be fully representative. This demographic information does however provide a useful portrait of learners.
File Campaigns Comments Enrolments Question Response Step Activity Team Members
It stores learners’ responses, its correctness and the timestamp associated when answering a question of any quiz.
learners in the course, e.g. the time when a step is first visited, and the last time a step is marked as completed.
names are stored in this file.
A MOOC dashboard was created at UNSW Australia for courses published in FL
platform
Heading Adaptive Tutorials (SmatSparrow) Demographics Activity Different types of visualisations to show the geographical distribution, gender distribution, gender vs. employment status, gender vs. age range, education Distribution, and employment area distribution.
Having multiple visualisations to analyse the step activities of learners. For example, the percentage of time spending on each week, finding the number of leavers at any step or any date of the course, a heat map to draw the step completion progress of learners, and transition networks between available materials of the course by step type or week number. 2.4
Approximately 7000 learners registered for the first run of the course with 40% actively
engaging with the course at some point while it was open. Similar to the patterns already
identified with the funnel of participation
'Active learners' are defined by FL as those who actively engage with some content while the course is open, and 'completing' refers to those who self-mark at least 90% of the steps in the course as complete. Due to the nature of the platform, active learners may have visited and completed learning activities, but may have not self-marked the step as completed, indicating a potential for under-estimation of the number of completers in the course. As anticipated in the design stage of the course, only a small proportion of learners obtained a paid certificate.
From the sample of survey responses, the typical learner in the TEE MOOC was male (71% of respondents), aged between 18-25 (26%), in full time employment (34%) and with an undergraduate degree (40%). The summary table below (Table 4) provides an overview of the distributions. This second run of the course revealed similar responses, with the typical learner being male (62% of respondents), aged between 1825 (34%), in full time employment (33%) and with an undergraduate degree (40%).
From the logs of interaction with the platform it is possible to identify several trends. Learners who engaged with the content spent between 90 minutes to two hours on average per week in the course. This equates to roughly 5-10 minutes per step. FutureLearn uses the concept of ‘step’ which can incorporate a variety of artefacts including articles, video, discussion, quiz, exercises etc. Table 5 provides a summary overview of the time spent in the course by active learners for both iterations.
In the first iteration of the course, learners spent more time in week 3 (114.78 average minutes) than in other weeks of the course (Table 5). This also correlated to a steeper drop in engagement during the first three weeks of the first iteration. The number and percentage of the leavers at any week is shown in Table 6 for both iterations. Typically, a large proportion of learners leave the course in the first week of any MOOC. The reasons for this will vary and require further research. Reasons may possibly relate to learner expectations not being met, or factors such as personal commitments hindering continuation and completion of a course.
Qualitative data in the form of discussion forum comments from learners in week 3
of the first iteration showed they experienced difficulty with some of the activities in
this week. The risk with problems being too easy is learners may lose interest quickly;
conversely problems that are too difficult may potentially place strain on the working
memory on novices
The story-line for week 3 was streamlined, and the design step for this week was also divided into two parts. The overall structure of the course was simplified by merging a majority of discussion forums that were originally separate steps into the step that they related to as ‘talking points’. The intention behind this was twofold. Firstly, it was hoped that fewer steps would make tasks in the course appear less intimidating to learners, secondly, a reduction in steps simplified the job of instructors by reducing the number of places they had to monitor in the course. This resulted in a general change in the transition of learners among materials of all weeks in the second iteration of the course (Figure 1). The Adaptive Tutorials were also revised for week 3. On a technical level the UI was improved in the second iteration by enhancing the accessibility of the adaptive tutorials for mobile devices such as iPads, creating several new drag-and-drop activities, improving the adaptive feedback, and adding better quality LaTex equations. General information screens were added at the beginning of each Adaptive Tutorial lesson to help orient learners to the features found in the adaptive tutorials. Qualitative feedback gleaned from the discussion forums in these steps also suggested too much complex information in some Adaptive Tutorials. In response, some Adaptive Tutorials were chunked, such as the Free-Body Diagram, into two smaller learning segments to make them easier for learners to understand and complete in less time. The result of these changes included learners spending less time in week 3 of the course in the second iteration as compared to the first (as seem in Table 5). Splitting the Free-Body Diagram Adaptive Tutorial also resulted in more learners achieving higher scores in the tutorial, as seen in Figure 2.
The TEE MOOC has reinforced for us how important it is to analyse the learning experiences of the courses we offer as part of the cycle of continuous improvement. Offering this MOOC has enabled thousands of learners to have access to a free course in the fundamentals of Engineering Mechanics, but this brings with it correspondingly increased responsibility to do it well.
By leveraging on the data we can make informed choices about the design of the course and thereby improve the learning experience of a global cohort of learners. In this way we have created a data-driven course development process that provides learners with the best learning experiences possible – wherever they are in the world. There are still challenges in accommodating broad and large demographics of learners. For example, mathematics was intentionally kept as simple as possible, however basic algebra and simple trigonometry challenged a number of learners, as evidenced in the discussions. The changes made to the course were implemented after the first iteration. A challenge lies in how agile this process can be, such as whether near real-time data can also be leveraged to inform course design changes in near real-time.
The overall aim in offering TEE has been simple: to offer to a wide range of people an understanding of engineering mechanics through experiments, analysis and design, whether for general interest or in preparation for an engineering future. We are offering them all a chance to see the world "Through engineers’ eyes". 4