This paper describes the configuration, setup and initial analysis examples of an experience for introducing learning analytics in a MOOC of maths for adult high school education, using the flipped the classroom methodology. An overview of the MOOC of maths is provided, as well as an overview of the ANALYSE learning analytics tool which is used to analyze the learning process. We describe how the learning analytics tool can be useful in this setup and methodology and we illustrate specific examples of conclusions in the MOOC of maths.
Learning analytics is a useful technique to track and intervene in the learning
process in order to improve it. Massive Open Online Courses (MOOCs) have emerged as
a new paradigm that usually has an intensive use of videos and interactive exercises.
The type of rich users’ interactions in MOOCs makes learning analytics a very useful
technique in these scenarios. However, current support of learning analytics in
MOOC platforms is at an early stage. We have extended the learning analytics
support for the open edX platform [
Emerging and useful educational technologies such as learning analytics are usually designed, tested and experienced in research contexts. But there is a need of extending these educational advancements to the whole society. In this direction, we aimed at applying the ANALYSE learning analytics tool as well as the open edX platform in a MOOC of maths in the CEPA Sierra Norte de Torrelaguna High School Center (in the north of the Madrid Region). This center provides high school education for adults. The profile of the participants of this center is quite different from traditional high school students as there is a difference of age and old generation are e.g. not as used as new ones with technologies.
In this document, we give an overview of ANALYSE, the design of a MOOC for maths for high school education for adults or the methodology used in high school education for using the MOOC in combination with traditional class sessions and ANALYSE. In addition, we provide some initial examples of how to interpret some data of ANALYSE in this MOOC of maths for high school education for adults in order to improve the learning process. 2
The analytics visualizations that are provided by Open edX by default are very limited, therefore we have developed and installed ANALYSE1 (Add-on of learNing AnaLYtics Support for open Edx). This add-on extends the learning analytics support of Open edX with 12 new visualizations related to exercises, videos and course activity. These new indicators are obtained by processing the logs of Open edX and transforming them into useful information that can be used by the different stakeholders. The process of computing the different indicators is done by ANALYSE as a background job in regular intervals, which is transparent for the user; the indicators are updated in regular intervals of time, instead of being calculated each time that a visualization is requested.
ANAYSE provides of a different interface for instructors and students. Students can access the information related to their own activity only, whereas instructors can access the information related to the full aggregate of the class but also each one of the students taking the course individually. This can help students to gain selfawareness about how they are progressing in the platform and improve their motivation. Additionally, this dashboard helps instructors to keep track of all the students in the class to know who is doing what and who is having problems. They can also use this tool to detect problems in resources such as videos who are not transferring knowledge properly or problems with an elevate complexity compared to the rest of the course. The different visualizations of ANALYSE are grouped in the learning analytics dashboard in the following sections: • Exercise visualizations are related to the activity of students with the different problems of a course. There are three different visualizations which are the next: o Course Summary: Provides an overview of the percentage of students who have achieved proficiency, passed, failed or not accessed each category of resource o Students Grades: Provides the grade of each item of the course 1 http://www.it.uc3m.es/pedmume/ANALYSE/ o Problem Time Distribution: Time invested in each problem of the course Video visualizations are related about the interaction of students with the course videos. There are four different visualizations: o Video Time Watched: It shows the progress done in each of the videos from 0 to 100 by the students o Repetitions of Video Intervals: This visualization provides information about what segments of time of each video have been seen more times o Video Time Distribution: Shows the time invested in each one of the videos separately o Video Event Distribution: This visualization displays the position in the video of the different video events triggered by students when watching the video (pause, stop, play, seek…) Course visualizations are related to both course and exercise activity at the same time and other things: o Problem and Video Progression: This visualization shows the progress over time in both exercises and videos that students have done in a stepped area chart. o Daily Time on Problems and Videos: This visualization shows the time invested each day in exercises and videos separately. o Course Accesses: This visualization shows the number of accesses to each chapter of the course and their respective subsections and units. o Chapter Time: It shows the time elapsed in each chapter of the course and it divides this time in graded and ungraded. o Student Time Schedule: This visualization shows the amount of time invested in three different time intervals (morning, afternoon and night) 3
Open edX is one of the most widespread MOOC platforms due to the great quality of the software but also because is one of the few that has been released as an open source application that can be used for free. Open edX is composed of two main applications which are the Learning Management System (LMS) and the Course Management System (CMS) which is denominated as Studio. On one hand, the role of Studio is to provide of a friendly interface for instructors where they can build the course and create the different elements of each section of the course, set the grading policy, configure the different course deadlines or the timing in which each section of the course need to be released. Studio application allows organizing courses in sections, sub-sections and units. The contents can be diverse such as videos, HTML content or a big selection of assessment activities. Assessment activities include from the simple ones like checkboxes, multiple choice, or different types of input field problems, to those more advanced like the evaluation of programing expressions, drag and drop or peer assessment. On the other hand the role of the LMS is to display for each one of the courses the contents according to how the course has been configured in Studio. Students and instructors can navigate the course contents, use the different elements such as exercises and videos and interact socially with other peers and course instructors through the use of forums. The topics that have been covered and designed in the course have been divided into three main sections. Additionally, an introductory section about how to use the platform and the contents was included. The main sections of the course are the next: • Introduction: Introductory section so that students know how to interact with the contents, and other information regarding the MOOC. • Units of measurement: This section describes the different basic units of measurement and their use for mass, length, capacity, volume, time and surface area. • Algebra: This section starts describing monomials and how to operate with them, next it explains first grade and also quadratic equations by describing how to resolve different particular cases with exercises and videos. • Geometry: The last section describes the concept of angles in polygons, then it explains triangles and the Pythagorean theorem. Finally the calculation of areas for the different geometric shapes is explained. Each one of the three main topics of the course is distributed with different subsections, where each sub-section contains a unit that can be either HTML content, an exercise or a video. One of the decision design aspects that we took when building the course, was to only put one element (HTML content, exercises or videos) in each one of the units. The objective for this design is to be able to obtain individual analytics about the time spent in each course element separately, and that would have not been possible if we put several elements in the same unit. Additionally, the general idea is to provide short videos and after then some exercises to check if the students have learned the concept; this design also helps students not to get bored as it is more dynamic than other traditional methodologies where all the contents are delivered first and then there is an exam that includes everything.
There are a total number of 31 different problems. Most of the problems have a value of one point, but there are some others which are longer that score more because they require several answers instead of only one. Additionally a total number of 28 videos were recorded, edited and added to the MOOC. Videos are short, most of them within the range of 2 to 4 minutes and they explain a small piece of content of the course. Similarly, exercises evaluate small pieces of content that are closely related to the explanation at the previous videos. The main types of exercises that have been used are checkboxes, multiple choice and input field problems. 4
The MOOC of maths has been available for everyone interested in learning these concepts of maths, but not only for the students of the CEPA Sierra Norte de Torrelaguna. The MOOC is still available (activities in Spanish language) at: http://ela.gast.it.uc3m.es/
The students of the CEPA Sierra Norte de Torrelaguna were encouraged to use the
different videos and exercises of the MOOC. Educational materials (videos and
exercises) were released in sequence but not all at once, so that the educational activities
related to a face to face class session were available a few days before the face to face
class session. Students had to make the educational activities (watch the videos,
solver the exercises) before the correspondent face to face session. The face to face
sessions were devoted to reinforce the concepts, solve questions, etc. This is a flipped
classroom [
The role and importance of the learning analytics methodologies and the ANALYSE module relied on the following points:
- Self-regulated learning: Students were able to see their statistics, data, etc. They receive personalized feedback through visualizations so they can reflect about their own learning and they can take actions on the MOOC of maths to improve their learning experience.
- Augmented information for teachers for the flipped the classroom methodology. Teachers can know general tendencies about the students, know the specific problems of each student, etc. before the face to face lesson, so teachers can adapt the face to face lesson to be more effective based on the data provided by ANALYSE.
- Evaluation of materials, students and the learning process. The ANALYSE information permit the evaluation of material, students and the learning process. Based on these evaluations, different actions can be taken to improve the course for future editions. 5
ANALYSE can be used by both students and instructors with different objectives in the classroom. Students can track their own progress in the Open edX courses with visualizations that represent their own learning process. Therefore this feedback can help them maintain motivation towards achieving their goals and discover more about how they actually learn. They can know if they are progressing as expected and meeting the requirements that can be established in order to surpass a course. It also shows interest in self-regulated tools that can be related to the fact that they really want to learn the contents of the course. On the other hands, instructors can use ANALYSE in the classroom with several objectives. They can be tracking students individually, so that they can have time to intervene before one of them drops the course with no possibility of resuming it back. They can check for students who are struggling in certain areas of knowledge, i.e. a section of the course with deals with a specific topic, so that they are able to contact the students with problems and help them reinforce this problematic section of the course. Also they can try to detect problems in the educational resources for example exercise which hardness is about the rest of the course problems or they have been badly design attending to the concepts transmitted in the videos, and also videos who are not very well recorded and students are having problems to understand the concepts that are being delivered by the instructor in the recorded lesson. In order to illustrate some of these possibilities in a real experience, we propose here a couple of example obtained from the real data obtained from the maths MOOC which provides an idea about how to interpret some of the visualizations available through ANALYSE.
In the first case we are going to check about a concrete student denominated as “student A” to preserve their anonymity. What we are trying to represent here is the natural flow that an instructor could follow in order to understand what is happening with student A. First we check figure 2 for Students Grades visualization which shows us the grade from 0 to 100 in each one of the sections of the course. Then we can see that the scores in Units of measurement (100) and also Geometry (86) are good, and the student obtained good grades, however for the Algebra section (44) the student failed. So, the instructor might want to know more about this situation. A possibility is that student A failed Algebra because he/she did not put enough effort, another option is because he/she is really struggling with this topic. Therefore, in order to deepen into this matter we check figure 3 for Course Accesses visualization where we can check that the number of times that the student A has accessed Algebra section is 10, which is not a big difference compared to Units of measure and Geometry which are 12 and 17 respectively. Next we can check in figure 4 the Chapter Time visualization where we can see that student A has devoted 64 minutes to Algebra section, whereas he/she devoted to Units of measurement and Geometry, 48 and 81 minutes respectively (remember that student A completed these two other sections with high scores). Therefore we can see that despite student A has made an effort on Algebra section, he/she did not manage to surpass the problems. Therefore the instructor could conclude that it is probable that student A is actually having issues with Algebra section in order to intervene and help the student with the most appropriate method.
The second example that we want to propose here is related to find problematic parts of a course. Our first step as an instructor was to check by the use of Students Grades visualization the average grade of each one of the sub-sections on the course (each section has several sub-sections). After this analysis we found out that the subsection with the lowest average score is one called “Area calculation” which explains how to calculate the area of different shapes and belongs to the Geometry section. Therefore, it would be interesting to check this more in depth in order to detect if there are problems with this specific sub-section of the course. This sub-section is composed of 6 videos explaining the area calculation method of different shapes with their respective problems. The second thing we can check is the Video Time Watched visualization in figure 5. We notice that the last 6 videos which are the ones that belong to the “Area calculation” sub-section are some of the most watched of the entire course. We can check this detail by comparing the light blue bar with the ones of the rest of the course. Additionally, the Course Accesses visualization revealed that the “Area calculation” sub-section had 67 accesses by the entire class, which is actually the highest number of accesses of all the course sub-sections. Therefore we can conclude that despite students have watched a lot of videos and also accessed this concrete sub-section many times, which means that they have put a considerable effort in this sub-section, their grades in the “Area calculation” sub-section are still the lowest ones in the entire course. Thus it is possible that there are some issues with the contents of this sub-section, such as an elevate complexity of the topic, or too much content that might indicate that a revision and further analyses of the sub-section contents would be advisable.
This paper shows a way for combining a MOOC of maths, a flipped classroom methodology, with a learning analytics module we developed in the context of adult high school education. We show a general overview of how to take advantage of learning analytics in this context and some initial examples of use of the ANALYSE module in this MOOC. More analysis and use of visualizations are required for a complete evaluation of the course, students and materials.
This work has been supported by the “eMadrid” project (Regional Government of Madrid) under grant S2013/ICE-2715, the EEE project (Spanish Ministry of Science and Innovation, "Plan Nacional de I+D+I) under grant TIN2011-28308-C03-01, and the mapaTIC project (Spanish Ministry of Education, Culture and Sports).