Video games have become one of the most popular mediums across cultures and ages. There is ample evidence that supports the benets of using games for learning and assessment, and educators are largely supportive of using games in classrooms. However, the implementation of educational games as part of the curriculum and classroom practices has been rather scarce. One of the main barriers is that teachers face to actually know how their students are using the game so that they can analyze properly the e ect of the activity and the interaction of students. Therefore, to support teachers to fully leverage the potential bene ts of games in classrooms and make data-based decisions, educational games should incorporate learning analytics by transforming click-stream data generated from the gameplay into meaningful metrics and present visualizations of those metrics so that teachers can receive the information in an interactive and friendly way. For this work, we use data collected in a case study where teachers used Shadowspect geometry puzzle games in their classrooms. We apply learning analytics techniques to generate a series of metrics and visualizations that seek to facilitate that teachers can understand the interaction of students with the game. In this way, teachers can be more aware of the global progress of the class and each one of their students at an individual level, and intervene and adapt their classes when necessary.
Playing games is one of the most popular activities for all ages of people around
the world, and its value as a learning opportunity has been widely accepted. In
U.S., for example, nearly three-quarters (74%) of parents believe video games can
serve an educational purpose for their children [
Copyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
increasing interest in using games in educational settings, not simply because
\it is what kids are paying attention to," but because well-designed games are
very closely aligned with the design of good educational experiences [
The bene ts of games to support student learning have been well documented
over past 10 years. In a recent meta-analysis study, Clark and his colleagues [
While ample evidence shows that games, in general, have a great potential
to support learning, only when combined with a thoughtful curriculum, games
can be successfully used to support learning in classrooms [
For this educational purpose, we are using Shadowspect, an interactive computer game where students can create geometrical primitives such as cones or spheres to solve 3D puzzles, developing their geometric, dimensional and spatial reasoning skills. Shadowspect features 30 levels of gradually increasing di culty where students need to move and rotate shapes in order to nd solutions to modeling problems. Using Shadowspect, students' interaction with the game is collected in order to analyze di erent metrics like active time, a number of puzzles completed or the total number of events performed while playing. These metrics can be presented graphically so that teachers can observe each students' progress in the game and nd problems that the whole classroom could be having with a speci c puzzle. More speci cally, for this research we have the following objectives: 1. To present a proposal of metrics that can help teachers to understand the interaction of students with Shadowspect. 2. To present a case study with two uses cases from data collected in K12 schools across the US using Shadowspect: (a) A rst use case using these metrics to understand the global progress in an entire classroom. (b) A second use case using these metrics to understand students' progress in a classroom at an individual level.
The rest of the paper is organized as follows. Section 2 reviews background literature on student engagement and learning analytics. Section 3 describes the methods, including Shadowspect, as well as the educational context and the data collection. Section 4 presents our proposal of metrics, and in Section 5 we introduce and describe both use cases using the previous metrics and visualizations. Then we nalize the paper with discussion, conclusions and future work in Section 6. 2
With current data collection technologies, we can collect large datasets from
students' interaction with educational games that need to be treated in order to
be understood [
Researchers often implement metrics to make sense of the collected. raw data
through a feature engineering process [
In some previous studies, the main goal has been to measure the engagement
of each student. Authors in [
Despite the proven bene ts of educational games in learning [
In this case study we use Shadowspect, a game-based assessment tool that aims to provide metrics related to geometry content and other behavioral and cognitive constructs. Shadowspect has been designed explicitly as a formative assessment tool to measure math content standards (e.g. visualize relationships between 2D and 3D objects), thus teachers can use it in their core math curriculum.
When students begin a puzzle, they receive a set of silhouettes from di erent views that represent the gure they need to create, which will be composed of other primitive shapes the student can put into the scenario. The primitive shapes that students can create are cubes, pyramids, ramps, cylinders, cones and spheres. Depending on the level and di culty, the puzzle may restrict the quantity or type of shapes they can create. After putting these shapes in the scenario, they can also scale, move and rotate the shapes in order to build a gure that solves the puzzle. Students can move the camera to see the gure they are building from di erent perspectives and then use the `Snapshot' functionality to generate the silhouette and see how close they are to the objective. Finally they can submit the puzzle and the game will evaluate the solution and provide them with feedback.
In the version of Shadowspect that we have used in this case study, we have 9 tutorial levels, 9 intermediate and 12 advanced. The tutorial levels aim to teach the basic functionality of the game, so the students can learn how to build di erent primitives, scale and rotate them, how to change the perspective, take snapshots and so on. The intermediate levels allow students more freedom so they will not receive so much help to solve puzzles and then the advanced levels pretend to be a real challenge for students who have gained experience with previous levels before. 3.2
The data used (N = 300) for this paper was collected as part of assessment machinery development that later will be implemented in Shadowspect. Due to the goal of ensuring that we have su cient data points to create robust assessment models, the team recruited 7 teachers who can use Shadowspect at least two hours in their 7th grade and 10th grade math and geometry classes. In this paper, we focus on a single class with 31 students to represent a real case scenario of how a teacher could use these visualizations to monitor the progress of their students in the classroom. All student interactions with the game were collected and stored in a MySQL database, we do not collect any identi able or personal data from the users except for a nickname. The data collection of the selected class with 31 students includes around 54829 events (an average of 1768 events per user); students were active in the game environment for 33 hours (an average of 65 active minutes per student), and students solved a total of 448 puzzles (an average of 14 puzzles per student). 4
For the analysis of the data within our game we have de ned four di erent metrics: Levels of activity and di culty, puzzle funnel and sequences between puzzles. With these metrics we obtain a detailed analysis of the students' interaction with the game, so the teacher can receive detailed information that can be used to evaluate and analyze students' interactions. With this analysis we cover the most important aspects in order to monitor and analyse students' activity in a simple way. In addition, we want to explore the a ordances of combining several metrics together to augment their signi cance. Next, we explain the implementation details of each metric.
{ Levels of activity: This metric implements a set of parameters that describe the levels of activity of the user with Shadowspect. These are straightforward metrics to compute based on a feature engineering process, such as the active time, number of events, di erent type of events, and number of di erent types of events like snapshots, rotations, movements, scaling, shape creations and deletions, among several others. However, for this case study we focus only on the two rst types of events that we mentioned because these are the most important to look at when analyzing students' interaction with the game, however we would like to denote that all of them are available for the teacher.
active time: Amount of active time in minutes establishing an inactivity threshold of 60 seconds (i.e. if the time between two events is above 60 seconds, the user is considered to be inactive during that time and this slot is omitted from the computation). n events: Total number of events triggered within the game (every action performed by a student in Shadowspect is recorded as an event). { Levels of Di culty: This metric provides a set of parameters that are related to the di culty of the puzzles: completed time: This parameter is computed by dividing the amount of time invested in the game (active time) by the number of completed puzzles. actions completed: This parameter is computed by dividing the number of actions (n events) by the number of completed puzzles. p incorrect: This parameter is calculated dividing the number of incorrect attempts by the total number of attempts (n attempts) multiplied by 100. p abandoned: This parameter is computed by dividing the number of started puzzles by the number of completed puzzles. norm all measures: First the four aforementioned parameters for this metric are standardized by computing the z -scores of each metric (i.e. z = x where is the mean and the standard deviation of x). Then, we make the sum of the standardized parameters: zall measures = zcomp time + zactions comp + zp incorrect + zp abandoned (1) Finally, the calculated parameter is normalized between 0 and 1. The parameters minz all measures and maxz all measures are the minimum and maximum of the zall measures.
normall measures = (zall measures
minz all measures) (maxz all measures minz all measures) (2) { Puzzle funnel: A conversion funnel is an e-commerce term that describes the di erent stages in a buyer's journey leading up to a purchase. We use this same metaphor to illustrate the stages that a student can go through in order to solve a puzzle. We de ne the following four stages for the funnel: started (if the student has started the puzzle), create shape (if the student has set up an object into this particular puzzle), submitted (if the student checked the puzzle solution) and completed (if the student has submitted the puzzle and the solution is correct). Then, we analyse the data to count the stages reached for each one of the puzzles and by each student. This funnel seeks to provide a quick overview of the current status for each student and puzzle in a class. { Sequence between puzzles: Although Shadowspect puzzles are divided in three di erent categories based on its di culty, they do not have to be completed in a linear sequence. Therefore, students can jump from any puzzle to another, regardless of its di culty, pursuing their own interests and exploring the game. In this metric our objective is to analyze the temporal interaction of students with the puzzles following a chronological order, so that we can reconstruct the sequence of puzzles for a given student. We provide an output with the sequence of puzzle attempts in chronological order and the funnel state the student reached in each attempt. 5
This section presents the case study that exempli es how teachers can use these learning analytics visualizations in two di erent situations: the rst use case in Subsection 5.1 applies the metrics and visualizations to understand the global progress in a classroom whereas the second use case in Subsection 5.2 is focused on the individual progress of students in a classroom. 5.1
The rst analysis implements visualizations at the classroom level, so that the teacher can see the overall progress of the class for each particular puzzle. This can be useful to detect which puzzles are harder or which contents areas require extra additional explanations or support from the teacher. Figure 2 represents the rst visualization with the puzzle funnel metric with its four stages as described in Section 4. We represent started (blue), create shape (yellow), submitted (red) and completed (green) in a circular chart for each one of the puzzles, which are arranged following the sequence order of appearance in Shadowspect game. For each one of the puzzles, the rst sentence on the top indicates its name and the second sentence the category of the puzzle.
Tutorial Levels One Box
Tutorial Levels Tutorial Levels Tutorial Levels Tutorial Levels Tutorial Levels Separated Boxes Rotate a Pyramid Match Silhouettes Removing Objects Stretch a Ramp Tutorial Levels Tutorial Levels Max 2 Boxes Combine 2 Ramps 88897770....5556%%%% 88887477....5455%%%% 88884444....4444%%%% 88884444....4444%%%% 88881144....2244%%%% 78788181....1212%%%% 77778885...111%%%% 77778885...111%%%% Tutorial Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels 9. Scaling Round ObtsDegree Rotations Angled Silhouette Bird Fez bBoxes Obscure Spheres Object Limits Pi Henge Pyramids are Strange
With this overview of the entire class, the teacher could quickly detect strange behaviours or problems in a particular puzzle. For this speci c class, at the tutorial level we see that the puzzle funnel is good, having in most cases an identical number of puzzles in stage completed and in started, which means that most puzzles that were started, were also completed. For the intermediate level, we can see that is more frequent to nd puzzles with a higher number of started than completed, which is indicative of the increased di culty. Once we review the advanced puzzles, we identify several puzzles that might represent that students are facing issues to solve them, for example for \Orange Dance" (46.9% started, 9.4% completed) and \Bear Market" (43.8% started, 3.1% completed), we nd that the percentage of started puzzles is much higher than the percentage of completed puzzles. Once these puzzles have been identi ed, the next step is to check whether this is due to the high di culty of the selected puzzles, conceptual issues or other factors. To do this, we analyze the puzzles with the levels of di culty metric.
completed_time 8 6 its4 e u n M 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 completed_actions s200 n o itc150 a f ro100 e b m50 u
N 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 p_incorrect p_abandoned 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 norm_all_measures 75 e g a tn50 rcee25 P 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 100 e75 g a ten50 rcP25 e
Fig. 3: Levels of di culty by puzzle in a classroom.
In the Figure 3, we represent the metric of levels of di culty with the four parameters and the composite measure. In the x -axis we have each one of the puzzles following the sequence order of Shadowspect, and the y-axis represent the di culty parameter as explained in Section 3. For both puzzles, \Orange Dance" and \Bear Market", that we identi ed as having a problematic funnel in Figure 2, we see that their di culty parameters are also high. In the time required to complete the puzzle, we have that \Orange Dance" has a lower time with 2 minutes on average, while \Bear Market" goes up to 8 minutes, so there is quite a di erence between both, as in the number of actions to complete the puzzle, which is of 50 actions and 200 respectively. However, for the percentage of incorrect attempts and abandoned we get almost equal values for both puzzles, with percentages close to 100%. Finally, the composite di culty measure shows what already appeared in the previous parameters, for \Orange dance" the di culty is around 0.8 and for \Bear Market" it is 1, the most di cult puzzle. With this last parameter of the metric, teachers can get a quick overview of the puzzles that are posing a major challenge among their students. 5.2
In this second use case we select a particular group of students in the classroom to see how they are progressing in the completion of the di erent game puzzles. Then, we will see how a teacher can observe students' progress and di culties at individual levels based on three of the metrics we de ned in Section 4.
Student 151
Student 155
Student 160
Student 188
Student 202 Student 220
Student 232
Student 238
Student 247
Student 250 Student 277
Student 278
Student 285
Student 287
Student 290 l e n n u F
As we can see in Figure 4, we have selected 15 students of the group. Using the funnel metric, the teacher can easily see the relationship between the number of started puzzles and the number of completed puzzles for every student. While some students show good progress as they have high completion rates (Student 188 has 96.7% completed of the total of puzzles available in the game), others reveal that they have been struggling with some puzzles. For example, Student 155 has started every puzzle in the game but only completed 46.7% of them. We are going to focus on this last student and gure out if the student is struggling with solving the tasks or if the student did not put enough e ort. Funnel
Completed 5.2 Number of seconds spent and number of events performed in each puzzle.
In Figure 5.1, we combine three di erent metrics at the same time: The x axis with the dots represent the sequence of puzzles of the student, while the color of the dots represents the funnel stage of each puzzle, and then we have incorporated the di culty metric of each puzzle by adjusting the position on the y -axis. The puzzles completed by the student are, in most cases, tutorial levels and levels with low di culty. Another thing we can gure out from this plot is that there are a lot of puzzles that the student has started but not even put a shape into it. We could think the student has been entering and exiting the di erent puzzles without doing anything else, but as it is not sure. To know what the student has done in each task, we introduce levels of activity metric for the teacher to review the actions of the student through puzzles.
From Figure 5.2, we can draw some conclusions about the student interaction. From the previous metric we know that puzzle level named as \Zzz" was submitted and then we see the active time and n events in this puzzle has been high. So we know the student has spent some time trying to solve it, and we could now say that the student has experienced di culties with this puzzle. As we could imagine, most part of tasks that were in a started stage, have insigni cant values of active time and n events performed so it can be assumed that the student has entered the puzzle but not even tried to solve it. 1.00 Funnel
Completed
Submitted 6.1 Sequence of puzzles 6.2 Number of seconds spent and number of events performed in each puzzle.
Now we have analysed a student experiencing certain issues with the game, let's focus on a student with good progress to see the sequence of puzzles and the levels of activity this other student has. In Figure 6.1, we now see a very linear progression, where the student has not entered a level until the previous one was already completed, so the student has been working ne but maybe needs more time than other students to solve the same amount of tasks. In Figure 6.2 we see that Student 247 has spent more than 300 seconds in task \45-degree rotation", but nally completed it, which shows high resiliency. The time spent by the student is regularly distributed in the di erent levels that have been solved. 6
The objective of this study was twofold: First to propose a series of metrics that can provide comprehensive information regarding the process of students with the puzzles in Shadowspect and second to achieve simple but detailed visualizations of these metrics that can allow teachers to track the students within their class, so that they can evaluate or detect problems quickly and e ectively. In Section 4, we proposed four metrics that can generate explainable insights regarding the interaction of students with puzzles; the main goal was to provide easy-tounderstand and actionable information for teachers. In the Subsection 5.1, we reported a set of visualizations at a classroom level for those metrics, so that teachers can evaluate the learning process of the class in a global way. Within the overall analysis, the teacher can identify the puzzles where students have found problems and analyse the cause with the di culty metric. This approach can help alleviate one of the main problems when implementing educational games in the classroom, which is the possibility to better understand how students interact with the game and the overall progress. Then, in Subsection 5.2 we presented a use case where teachers can monitor the individual activity of a student in Shadowspect using visualizations of the metrics. We analysed two di erent individual students, one of them with poor performance in the resolution of puzzles, and another showing a better puzzle funnel, but with a low percentage of completed puzzles in relation to the total number of levels available in the game. These two analyses exemplify how a teacher can assess the status of a student in the class and solve the possible problems a student might be having during a session. This represents an opportunity for educators to provide personalized attention to their students and help them in their learning process.
The next stage is to use this opportunity to implement just-in-time interventions that aim to provide support at the right time by adapting to the needs of each individual. One of the main limitations of this work is that these are o ine static visualizations, and thus in exible and not scalable. Therefore, as part of the future work we will be working on the co-creation of a dashboard for teachers that can provide greater speed and interactivity when displaying data from a class or individual students, and hence enabling just-in-time interventions during the sessions. Also we will be working on obtaining evidences of the interpretability of these visualizations and to make them explainable so that teachers can easily intervene. Shadowspect is designed as a formative assessment tool, and thus we can also use these visualizations for students so that they can receive feedback and improve their self-awareness. More nuanced metrics and visualizations will allow students to visualize their mistakes and areas of improvement. In this way we can use Shadowspect as a robust learning tool with that can be easily implemented by teachers in the classroom and that emphasizes the formative feedback to the student. This study has proposed a new dynamic approaches that can be helpful to facilitate systematic implementation of educational games in the classrooms of the future.
We want to acknowledge support from the MIT-SPAIN \la Caixa" Foundation SEED FUND and the Spanish Ministry of Economy and Competitiveness through the Juan de la Cierva Formacion program (FJCI-2017-34926).