The ubiquitous presence of cell phones in emerging economies has converted them in ideal platforms to cater services for underserved communities in areas like mobile learning. M-learning tools have proved successful in such challenging environments, specially for afterschool and vocational programs. A key component of that success is the personalization of the tools to the community and to particular individual needs. However, in order to tailor contents to the students, we first need a deep understanding of their learning abilities and preferences. Our findings show that there exist statistically significant differences in learning behaviors across gender and age. Finally, we propose a set of design suggestions that, based on the differences observed, attempt to personalize and adapt the mobile learning tool under study to enhance its educational impact in the low-income community.
There exists a large collection of mobile learning tools for low-income
communities adapted to all types of cell phones, mostly deploying SMS- or Java-based
solutions to deliver educational content [
EducaMovil is a system that has two main components: (1) a PC tool for
educational content creation and (2) a mobile game-based educational application
for Java-enabled cell phones [
The PC tool allows teachers to create the educational snippets that will be shown in the mobile game. Each educational snippet is composed of a lesson and a quiz, although only quizzes are also possible. The lesson typically contains an image and/or an explanation about a specific concept. A lesson is always followed by a quiz, which is a test question that the students are required to answer in order to evaluate their knowledge acquisition. For that purpose, student answers are compared against the correct quiz answers given by the teachers while creating the lessons. Additionally, teachers are required to label each educational snippet with its educational level (1st to 6th in primary school or 1st to 5th in secondary) and a complexity level (five levels from very easy to very difficult). 2.2
The mobile phone application that students run on their cell phones consists of two elements: the game and the educational snippets created by the teachers. EducaMovil offers the possibility of embedding the educational contents into different open-source games for cell phones like Snake, Tetris or Rally. The games are modified to add a module that handles the management of the educational contents. This module consists of three components: Game Model, Adaptation Model and User Model.
The game model is responsible for the interaction between the open-source game and the educational snippets created by the teachers. The game model is fired every time an event in the game allows players to win points (or lives) and introduces the educational snippets as the units that need to be solved before winning the points. After the student finishes exploring the lesson, s/he will be prompted with a quiz, which is based on the lesson’s content. If the learner provides a correct answer to the quiz, s/he receives an award in the form of lives or points.
The adaptation model determines the specific educational content that is going to be shown to the student at each step of the game. Recall that each educational snippet is labelled with its educational level and a complexity level. The adaptation model selects, for a specific educational level, the complexity of the educational snippet that is going to present to the student next, based on the students’ past learning evolution. In our strategy, students that answer correctly to at least 60% of the questions for a specific complexity level, are shown quizzes from the next level until the maximum complexity (very difficult ) is reached. If the student reaches the maximum level of complexity for her educational grade, the game ends. Questions for each complexity level are randomly selected across all academic subjects.
The user model stores the interactions of the student with the educational units and the game. The model keeps counters about the lessons explored by the student, whether the quizzes were answered correctly or not, time invested to answer quizzes, and the complexity level reached. This model is primarily used by the adaptation model to determine the complexity level of the lesson/quiz to be shown next. 3
The evaluation of EducaMovil consisted of one-on-one sessions where each student sat down and received an initial description of the game, its rules, the educational snippets and how to navigate through these with the cell phone. After the initial introduction, we let the student play for 20 minutes. During the session, students were shown educational snippets from their own educational grade starting with easy lessons which evolved in complexity based on the adaptation model. A total of 27 students from a low-resource school in Lima (Peru) tested EducaMovil: 5 from each 1st, 2nd and 3rd years and 6 students from each 4th and 5th years of the secondary school. These students had ages between 12 and 16 and in terms of gender, 15 were female and 12 male. At the end of each session, we collected a user model with the student interactions and computed a general performance model (GPM) for each student i as GP Mi = (C, T ) = (100 ∗ (Pjn=0 Aj)/n, (Pjn=0 Tj)/n) where Aj is the answer given to question j, Tj the time used to answer it and n represents the total number of lessons explored by the student. 4
We carry out two types of analysis: (i) analyze whether there exist differences
in the learning performance of the students based on gender or age and, (ii)
evaluate the types of learning behaviors observed across all students. The first
analysis will give us suggestions for gender- or age-based personalization. As for
the second, it will provide design guidelines to personalize education based on
stereotypes i.e., learning behaviors shared by groups of students [
To perform the age analysis we compute for each performance variable C and T one distribution per educational level (age). To understand whether there exist differences in the student performance across educational levels (age groups), we run Kruskal-Wallis tests for each group of five distributions representing the five age groups present in the pilot. We reject the null hypothesis with p <= 0.1. Our results determined that there exists a statistical significant difference between the percentage of correct answers among some of the five educational levels with p = 0.05. However, we do not observe any statistically significant differences on the average answering time. We observe that students from the 3rd grade outperform all their peers with an statistically significantly different percentage of correct answers (median = 85%), whereas students from the first grade show the worst performance with a median value of 29%. This analysis shows that although students are shown quizzes adapted to their own educational level, not all groups respond equally. In fact, we observe that students from educational levels 4th and 5th are statistically significantly outperformed by the students in the 2nd and 3rd grades. This might be related to learners in their last school years being allowed to move to the next educational level without making sure they have acquired the minimum required knowledge. We suggest to add quizzes from previous educational levels until the students show an improvement in their performance. This approach will allow them to review previous contents and to eventually reach their own educational level. Stereotypes In this section, we use the k-means clustering technique to identify common learning behaviors (stereotypes), independent of age or gender, among the students in our study. Given that the final k-means partition highly depends on the initial seeds selected, we run the algorithm 100 times for each value of k, and select the cluster distribution with the most compact and well separated clusters i.e., the one with the minimum cluster validity index computed as the ratio between the intra-cluster distance and the inter-cluster distance among all sample. Once the best selection of clusters has been identified for each value of k, we select the k with the smallest cluster validity index across all partitions evaluated (from k = 2 to k = 7). In practical terms, we attempt to find clusters of common performance behaviors across the 27 GPMs. For that purpose, we normalize across all GPMs, apply k-means and validate for each value of k. Although the minimum cluster validity index corresponds to k = 2, we discuss k = 3 since it provides more insight into the learning stereotypes than k = 2. Larger values of k have larger cluster validity indices and do not provide any other relevant information after exploration. Figure 1 shows the results for the clustering of the 27 GPMs with k = 3. Cluster 1 with centroid (73.2%, 65.1s) and three students represents a group with the highest percentage of correct answers and the largest answering times. Interestingly enough, this group contains only female students. Cluster 2 with centroid (65.1%, 24.3s) and 15 students, represents a group that employs little time to give the correct answers. This cluster probably groups the best students. Cluster 3 with centroid (29.1%, 26.2s) represents a group of 9 students that share a low percentage of correct anwers and low average answering times. This group probably represents students who answer randomly with their interest devoted to the game instead of the educational contents. We suggest that when this type of behavior is identified on a specific student, the game should be slowed and show more than one quiz at a time to force the student to focus on the quizzes.
Our analyses have identified learning behaviors that can be used towards the personalization of EducaMovil. Although these findings cannot be extended to other low income communities and learning tools, similar analytical techniques could be used to propose different personalization suggestions. The gender analysis highlighted that although women tend to perform better than men, they generally need longer answering times. This fact is not necessarily bad if it is related to women being more thoughtful, but could be harmful if connected to a lack of confidence. In an attempt to decrease answering times while maintaining performance, we suggest to put counters and timers in the quizzes so as to create a healthy competition between men and women not only in terms of correct answers but also in terms of answering times. The age analysis showed that students from the last years of secondary education (4th and 5th) performed worse than their younger counterparts. This finding could be related to students being promoted to the next level without having acquired the minimum knowledge for their current grade. In this sense, we suggest to add quizzes from previous educational levels until the students show an improvement in their performance. This approach will allow them to remember or review previous contents and to eventually reach their own educational level. Finally, the stereotype analysis revealed, among other things, that there exists a group of students that answer the quizzes almost randomly so as to advance on the game, without showing any interest on the educational content (identified as Cluster 1 in our analysis). We suggest that when this type of behavior is identified on a specific student, more than one quiz at a time should be shown before going back to the game; and the game itself should be slowed until it becomes boring and forces the student to focus on giving better answers to the quizzes. 6
A very important component of mobile learning services is the personalization of
the educational tools, specially for low-income communities. Research has shown
that personalization to tailor content and structure highly increases the quality of
the learning tools and enhances the learning process. However, in order to do so,
we need a deep understanding of the abilities and preferences of a learner. In this
paper, we have proposed a series of techniques to model learning performance
and understand the impact that gender, age or learning behaviors might have
on the learning process of the students in a low resource school in Lima, Peru.
For that purpose, we have run a pilot with EducaMovil, a game-based mobile
learning application for afterschool programs in low-income schools [