MOOCs generate huge amounts of data. The provision of digital learning at scale to a global learner base provides access to a variety of data sets ranging from learner digital activities to learner contributions. This paper sets out the methodological case for adopting particular sense-making strategies and a variety of techniques to aid with data reduction, analysis and presentation with respect to interrogating MOOC data to support specific learning design or redesign objectives. The findings from the initial implementation of these strategies are provided based on a series of MOOCs delivered in the area of Irish Language and Culture by Dublin City University via the FutureLearn platform. Furthermore, it illustrates how the use of a mix of both qualitative and quantitative approaches can provide both rich and measured insights to give MOOC learners an active and vocal voice within the learning design process.
1 Dublin City University
Of the many ways to die, “death by data asphyxiation” [1, p.281] may be amongst the worst for those engaging in social scientific research. In an increasingly-connected and recorded world, our challenge, once how to define, apply and operationalise key constructs we wish to study, has arguably given way to a form of data saturation. In such a world, where we are confronted at every turn by sources of data unimaginable even 30 years ago, understanding how to build a coherent framework for analysing and making sense of such forms of information may be a key concern. What is the relationship between logs of behaviour /clickstream data and learners’ thoughts and emotions? What can we expect to understand from large scale analyses, given that each person behind the screen may be quite different, and interpret their own behaviour differently?
This challenge has been particularly-noted with regards to research into MOOCs:
in many cases, the embarras de riches that researchers have been presented with, in
addition to raising some ethical quandaries [
A coherent analytical framework is in many ways prerequisite for social research, in that it shapes our inquiry, yet the generally high-level nature of much existing research into MOOCs has not leant itself easily to such consideration. Indeed, the focus has perhaps been on describing at the macro-most level, who MOOC learners are, and what typical patterns of participation they may engage in, rather than asking how this research informs design and re-design, to provide a better learning experience. As argued by Reich [4, p.34] in reviewing studies attempting to analyse learner behaviour using an array of statistical metrics, “It does not require trillions of event logs to demonstrate that effort is correlated with achievement”. While understanding that the above correlational relationship exists can be useful, it tells us very little of the means through which this correlation may occur, when it is the latter which we would likely wish to explain. Veletsianos, Collier and Schneider [5, p.571] provide an equally-succinct take on the issue, arguing that though we might understand what is being done by learners, understanding the why and how of this behaviour are considerably more mercurial, if we rely on such data alone.
These concerns relating to lacking a coherent theoretical girding to research are not
anything particularly new. In a sense, all social research faces this difficulty from
balancing both an approach geared towards description and explaining [
Though the above may appear at first glance to map neatly onto a typical quantitative-qualitative divide, with a variance-based model linked with quantitative research, and process-based with qualitative, this need not be the case at all. Both perspectives are of use in analysing different forms of data, which speaks to the importance of being unafraid to mix methods. 3
Perhaps one of the most important considerations in making sense of a complex
phenomenon, is to combine many different forms of data to underpin a coherent
analysis. Adopting a mixed-methods research strategy provides a means through
which different forms of data can be incorporated into an analytical whole (or as
contrasting forms of information, equally) [
Regardless of exact design or philosophical stance, there is a general paucity of studies which appear to make use of the kind of explicit mixing which is increasinglycommon in other forms of research. Though many MOOC studies make use of terms such as ‘mixed-methods’ this is rarely explicitly linked to a research paradigm, logic or framework [14, p. 497]. This is despite many articulations of the importance of same to broadening the field and the types of topics that we may consider [15, p. 214].
Our research centres on a suite of Irish language and culture MOOCs. Our research interests span applied linguistics, educational psychology and broader aspects of design and redesign, and have thus been both inter-disciplinary and have drawn upon theoretical frameworks outside the field of MOOC research. We will now discuss the three principles that guided this process. 3.1
In explaining behaviour in the MOOC, we have typically made use of multiple sources of data, such as survey instruments and forum contributions, as complementary, yet distinct, way of sense-making. When measuring the affective reactions of learners to specific tasks, we were (and are) struck by the ways in which the statistical averages we observed (both positive and negative) contrasted with the sentiment expressed in forum contributions. An interesting example on a particular activity was of forum contributions yielding feedback considerably more negative than those expressed in an anonymous survey result on the same activity.
This example presents an actionable research hypothesis, in that seeing negative
responses in a forum may encourage others to post similar messages, while an
anonymous survey, with no social, external referents, may temper this dynamic
considerably. This highlights the potential role of forum participation as a mediator
and perhaps even amplifier, in certain contexts, of emotions. It also points to the
limitations of findings from this data collection method. Observing the survey results
alone, however, would also limit the findings as qualitatively-different types of
feedback expressed in forum posts would be removed and thus limit our
understanding of the organic and social nature of participation [
Triangulation is also possible, and indeed desirable, in analysis, particularly in
“enriching our understanding by allowing new or deeper dimensions to emerge” [17,
p.604]. Essential to this is considering how different theoretical conceptions and
levels may complement, enrich or perhaps even contradict each other. A vibrant
debate exists on the degree to which fully holistic data integration and interpretation is
possible, with Bryman [
A related point which follows is to under-utilised store of rich qualitative data
typically generated over the length of a MOOC, perhaps due to the comparative
difficulty of analysing same. For example, the above-mentioned example involving
discussion posts provides valuable contextual information in their own right, and can
be analysed not purely using a macro-level method such as content analysis, but also
in viewing the forum contributions as an ecosystem or socially-constructed form of
participation [
Understanding change clearly requires a strong awareness for temporal dimensions, and in this regard MOOCs may be perfect for capturing both harder forms of information (such as logs, timestamps, surveys) that can also be paired with the subjective elements of how learners construe their own participation over time, through interviews, open-ended comments or focus groups. Temporality is an important and necessary component of sense making, and is necessary to underpin and conceptualise event sequencing. It further enables us to move from viewing time as the trigger of change to a useful measure through which to explore change [21, p. 98]. Longitudinal research, and considering multiple forms and points of data collection, allows the tracking of how attitudes, beliefs and participation change over an extended period. It follows also that though much research on MOOCs has focused on learner dropout rates, mapping engagement with different types of tasks that make up a MOOC is an equally instructive way to consider how the structure of a MOOC may play a role in participation (and indeed what factors most influence observable dropout). This is particularly the case as several qualitative studies point to time itself as a major factor in whether learners continue [22, p. 143), suggesting that it is not merely satisfaction/dissatisfaction that has an effect but also wider social and personal life circumstances. Our research has involved using staggered survey instruments to measure how different tasks may provoke different types of emotional responses. Tracked over time, this can be used to consider whether patterns may relate to specific steps or tasks, and can then be analysed using supporting sources of data, such as subjective learner accounts or auto-narratives of their own participation. 3.3
Iterative research provides the researcher with the capacity to consider the impact of
interventions, changes and refinements on new groups of learners supporting a
multistaged research design. It also follows that small changes in a MOOC over course
runs might allow quasi-experimental conditions, which could be taken beyond a
simple pre or post-test to incorporate explicitly learner (and lecturer, see [
In our case, the above-mentioned example relating to negative emotional feedback, both articulated by learners and measured using statistical instruments, lead to direct redesign of several steps. We found that providing additional scaffolding, audio support and re-arranging activities, all suggested by a range of learners, useful in subsequent runs of the course, and a clearly more-positive response. This type of research is deeply practical, but can also be embedded strongly in deductive reasoning, experimental principles and seeking gradual improvements as a direct consequence of research. 4
We would argue that a more philosophically-grounded form of research and
sensemaking is required in MOOCs than what has typically emerged to this point, and to
link to wider disciplines which are readily-implicated in MOOC design, participation
and interaction. This may entail in particular forms of staggered, sequential data
collection, a mixing of different types of instruments and different paradigm stances.
This may even include looking at the tensions that emerge between different stances,
and what they may illuminate about the promise (and limits) of various methods and
ways of sense making [