Active pedagogies can improve the pedagogical effectiveness of MOOCs. Group formation, an essential step in the design of small-group learning activities, can be challenging in MOOCs given the scale and the wide variety of students' behaviors in such contexts. In this paper, we further analyze the suitability of applying the students' engagement in the course as grouping criterion to form small groups in MOOC contexts. The impact of a grouping strategy based on requiring homogeneity among the students' engagement of the group is examined in a real MOOC context. In a preliminary stage the results have been analyzed in terms of peer interactions, active students per team and students' satisfaction. These results have been also compared with those in prior interventions in real MOOCs, thus validating previous findings about the suitability of this approach. The role of the timing of grouping was also examined by carrying out two collaborative activities at different weeks (the fourth and the sixth). A consistent improvement of all indicators was observed in the second intervention and its possible causes are discussed.
Since their emergence in 2008, MOOCs (Massive Open Online Courses) have allowed
the free delivery of knowledge to millions of people around the world. Due to the need
of eventually hosting a huge number of enrollments, most MOOCs adopt pragmatic
instructional designs that scale easily and run smoothly with massiveness. The most
popular type of MOOC, called xMOOC, is based on an instructivist model that provides
educational content, in text or video format, and assesses the achievement of learning
by means of quizzes. In xMOOCs, the interactions among learners are mostly limited
to forums and peer reviews. This type of instructional design does not include
pedagogic methods which actively involve students in the learning process, such as
collaborative learning or project based learning [
Copyright © 2018 for this paper by its authors. Copying permitted for private and academic purposes
The management of groups in MOOC contexts is quite challenging [
In this paper, we present a study where the suitability of using the students’ engagement in the course as homogeneous grouping criterion is analyzed and discussed. The rest of the paper is structured as follows. Firstly, we present our prior work in order to introduce a conceptual and technological framework we have developed as well as our previous interventions. Then, we describe the study presented in this paper and discuss its preliminary results and findings. 2.
The present study is part of a wider research process aimed at supporting teachers in the management of collaborative groups in MOOCs. In the first iteration, we performed a literature review which, together with the gathering of experts’ opinions, produced a preliminary version of a framework intended to organize the available information about the issue of managing collaborative groups in MOOCs. In the second iteration of the process, we generated two instrumental artifacts (a guide and a tool prototype) that were tested in two exploratory studies. The components of the framework and the main findings of these studies are described in the next subsections. 2.1
MyGang (Mooc analYtics for Group Assignment, moNitoring and reGrouping)
framework [
Context (MyGang_C) presents, in a structured fashion, the intrinsic features (Massive, Open, Online and Course) of MOOC contexts, and their derived extrinsic properties. This information is used in the Design Guide (MyGang_DG), to make teachers reflect on the impact of the context on the management of groups.
Grouping Factors (MyGang_GF), shown in Figure 1, depicts a hierarchical classification of the factors that influence the management of collaborative groups in MOOCs. The possible factors are divided into two main subsets related to pedagogy and technology. The pedagogical factors are also split into three categories, i.e., (i) Learning Design, which are aspects related to the learning design decisions that affect the group composition; (ii) Dynamic Data, that include the information monitored and updated while the course is running, (mostly the trace data that emerge through students' learning activities and interactions); and (iii) Static Data, referred to the information about the students that is not updated during the course enactment (e.g., demographics, preferences, etc.). This information is collected usually through surveys at the beginning of the course. These pedagogic factors are also used in the Design Guide (MyGang_DG) to support the decision making on the design of the groups activity and the configuration of the teams.
Architecture (MyGang_A) of the envisioned supporting tool to manage groups in MOOCs, is shown in Figure 2. It presents a high-level design of the envisioned groupmanagement supporting tools’ structure. It uses the pedagogical Grouping Factors (i.e., Learning Design, Dynamic Data and Static Data) as data inputs for the system and depicts them in green in the figure. The system is composed of several modules including adapters to import/export data from/to the MOOC platform: (a) a Dynamics Processing Module, to gauge and estimate dynamic factors (such as the engagement, the emerging role or the dropout probability) using the raw dynamic data collected from the platform; and (b) a Grouping Module, to configure the group structures based on the collected data and on the specifications given by the teachers.
Design Guide (MyGang_DG is a questionnaire that includes guidance and recommendations. It consists of four sections. The first section is related to the MOOC context features depicted in MyGang_C in order to make teachers aware of the context aspects that affect group formation. It includes questions so that teachers can reflect and select concrete characteristics of the envisioned MOOC using the researcher recommendations. The following three sections correspond to the three dimensions of the pedagogical factors of MyGang_GF and should be filled out once for each group activity to be designed. Teachers have to configure the learning design characteristics of the group activity and elicit the static and dynamic data factors that can be considered to configure the groups by using them as grouping criteria. In its current state, the guide may be used in a co-design process in the form of interviews with the teachers in order to discuss every item included in it. The researcher should give advice about the possible advantages and drawbacks of every decision made by teachers based on prior experiences, literature and experts’ opinions.
Tool (MyGang_T) is an implementation of MyGang_A that, at the moment, includes an early version of an interface module, which receives the input (e.g., group size, grouping criteria, etc.) through a configuration file and produces scheduled reports about the groups’ performance. The adapters have been programmed to meet the Canvas Network platform requirements and the grouping module to implement the group configuration specifications provided by teachers in each intervention. The functionalities of the rest of modules have been developed in order to satisfy the concrete specifications of the three studies carried out until now. 2.2
In our first exploratory study [
In this intervention, the first version of MyGang_DG and MyGang_T were used in order to design and implement the configuration of the groups, thus validating their utility.
The second study [
In this section, we report on the design of a third study intended to accumulate evidence to validate prior findings. In addition, we also explored the usefulness of two of the aforementioned instruments, the design guide and the tool, developed within the framework (MyGang_DG and MyGang_T) in order to provide support to teachers. The study was carried out in the second edition of the MOOC used to deploy our first exploratory study. 3.1
This study was carried out to get additional data and evidence about the performance of the homogeneous-engagement grouping approach used in our first study with a different students’ population. Therefore, the main objective of this study was to validate the suitability of the homogeneous engagement grouping approach (i.e., application of homogeneous engagement criteria to form small groups) to produce successful teams.
The success of the resulting groups was measured in terms of: (a) participation level in the collaborative activity (i.e., number of posted messages and number of active participants in each team) and (b) satisfaction of students regarding the collaboration carried out in their team. The final goal is to validate if this approach is able to achieve teams with several active students which carry out many interactions within their group and also to minimize the number of teams with a single active student. The perception of the students about the collaboration within their teams, and its relationship with the grouping strategy is also covered in the study. 3.2
This study is part of a wider multidisciplinary project that involves education and
technology. The study is mainly guided by a DSRM (Design Science Research
Methodology) [
The study was carried out in a seven-week MOOC that taught translation of economy and finance-related texts from Spanish to English. The course was offered by the University of Valladolid, Spain and it was deployed in the Canvas Network platform between March the 12th and April the 30th, 2018. The enrollment was closed at the end of the first week to allow us to configure properly the groups for the collaborative assignments. A free certificate was granted to the students who completed the mandatory assignments (one per week) in addition to the two surveys.
The total number of enrollments was 1028, and 653 of these students fulfilled the mandatory survey that was a requirement to see the course content. To obtain the certificate it was necessary to complete a compulsory assignment per week together with a final satisfaction survey. 173 students achieved the certificate (almost 17% of the enrolled students and 26.5% of those who accessed to the course content). 3.4
To implement the homogeneous-engagement grouping approach, learning analytics were employed to track MOOC learners' activities using the platform API (Application Program Interface).
Three types of elements were taken into account to gauge student engagement:
engagement with course content, engagement with course assessment, and engagement
with course discussion [
The algorithm selected for implementing the homogeneous grouping was k-means
clustering as it has shown to be effective with large datasets [
This strategy was applied to the group formation process in two collaborative assignments planned for two different weeks of the course, i.e., at weeks four and six respectively. It is noteworthy that in both assignments, a window of 21 days was used to trace data about the students’ activity in the platform. For the first collaborative
activity, this length was the distance between the course start and the beginning of the activity. The same window length was also applied when obtaining the trace data in the second assignment.
To measure the experimental results, we gathered data about the activity carried out in each team (i.e., exchanged messages, active participants) using the Canvas Network API. We also collected information from four surveys deployed in the course. The first one was necessary to access the course content and the following surveys were intended to capture the students’ satisfaction. Furthermore, the messages sent from the students to the teachers through the platform during the collaborative assignments were also captured in order to detect potential complaints and issues. Table 1 shows the data sources used in the experiments and figure 4 depicts the timeline of this data gathering. to measure the effects of the grouping strategy employed.
Course surveys composed of open-ended and closed questions including 4-point Likert items (1 = strongly disagree, 2 = disagree, 3= agree, 4= strongly agree, + don’t know/no answer) were administered: -[Sur1]. – Mandatory survey at the beginning of the course to get ethnographic data and preferences of the students. -[Sur2]. - Optional mini-survey at the end of the 4th week activity to score satisfaction and gather positive and negative perceptions regarding the collaboration carried out in the teams. -[Sur3]- Optional mini-survey at the end of the 6th week activity to score satisfaction, and gather positive and negative perceptions regarding the collaboration carried out in the teams. -[Sur2]. - At the end of the course (mandatory) to obtain students’ satisfaction with the course.
Canvas LMS API was used to collect indicators about: -[Ana1], [Ana3]. - Students’ engagement variables (i.e., [sec_conn_time], [num_page_view], [num_subm_assi] and [num_post_mess]) used to inform the group formation process. -[Ana2], [Ana4]. - Activity carried out during the group assignments (active teams, activity carried out within a team), used to evaluate the impact of the strategy implemented.
the students to the teachers during the collaborative assignments (4th and 6th weeks). After a preliminary analysis of the data gathered from the Canvas API we observed that percentages of interactions and active students per team seemed to be in a similar range of values than in the first study.
The satisfaction of the students with the collaboration carried out in their teams was measured in a different manner than in the previous study. In the study reported in this paper the students were required to score this satisfaction in a 0 to 10 scale just when the assignment finished. In the fourth week, they scored it 6.64 and in the sixth the average score was 7.78. In the prior edition of this MOOC the students had to express their agreement or disagreement with the statement “the collaboration carried out in my team was satisfactory” and they agreed in a 55% in the fourth week and in 70% in the sixth.
Therefore, although a deeper analysis of this data must be carried out, we can share these preliminary findings: 1. The number of teams with one single active participant represents a low percentage of the total number of active teams, and it is below 10% in the sixth week. 2. The grouping approach results in groups with many active members (i.e., more than the half of the total number of members of the team). In the experiment of the sixth week this type of teams exceeds 75% of the active teams. 3. The number of interactions per team remained in the same range as in the previous intervention and it was more than the double of that in the random approach, used as control group in the first study. 4. The students’ satisfaction with the collaboration carried out in their team is positive. 5. The second experiment (carried out in the sixth week) achieved better results than the first one (carried out in the fourth week) in terms of peer interactions, number of active members per team and students’ satisfaction. This fact confirmed a finding of the prior study and we deem that it can be due to two reasons: a. The engagement of the students is more stable in the second half of the course and this approach based in engagement improves its accuracy b. The students are familiar with the mechanics to carry out a collaborative task (instructions, recommendations, available tools in the platform) and this information allows them to perform better thus increasing their satisfaction. 5.
This study served us to get additional evidence about the eventual advantages of applying homogeneous-engagement policies to form small-groups in MOOC contexts. After a preliminary analysis of data, several advantages of the homogeneousengagement grouping approach have been confirmed.
When the objective to create small groups in learning contexts is to carry out a collaborative task, an unavoidable requirement is to achieve more than one active student in the group. The approach validated in this paper has shown to achieve better results in this regard than a random grouping.
It was also shown that this approach achieves better results than a random grouping in terms of peer interactions and number of active students per team and it obtained a considerable percentage (40% in the fourth week and more than 75% in the sixth week) of teams, in which with more than half the students of the team were active. This feature does not guarantee an enriching collaboration but it is a first step towards achieving such objective. Also, the satisfaction of students with the collaboration carried out in teams formed with this approach is reported by them as positive.
Finally, all these positive results were even better when the experiment is carried out a second time and deployed using data analytics from the second half of the course.
In the short term, we plan to keep on exploring the usefulness of MyGang_DG and MyGang_T in other MOOC platform (MiriadaX) which offers different functionalities for team activities. We also want to check the differences between a random grouping and the application of heterogeneous criteria regarding different variables. In the future, we would like to deepen in the outcomes of applying homogeneity on Dynamic Factors by testing it with different indicators and variables like the students’ connection patterns. 6.
This research has been partially funded by the Spanish State Research Agency (AEI) and the European Regional Development Fund, under project grants TIN2014-53199C3-2-R and TIN2017-85179-C3-2-R, the Regional Government of Castilla y León and the European Regional Development Fund, under project grant VA082U16, the European Commission, under project grant 588438-EPP-1-2017-1-EL-EPPKA2-KA and the Universidad de Valladolid (UVa).