Visualizing Educational Game Data: A Case Study of Visualizations to Support Teachers Pedro A. Martı́nez1? , Manuel J. Gómez1? , Jose A. Ruipérez-Valiente1,2 , Gregorio Martı́nez Pérez1 , Yoon Jeon Kim2 1 University of Murcia, Faculty of Computer Science, Murcia (Spain) 2 Massachusetts Institute of Technology, Playful Journey Lab, Cambridge (USA) Abstract. Video games have become one of the most popular mediums across cultures and ages. There is ample evidence that supports the bene- fits of using games for learning and assessment, and educators are largely supportive of using games in classrooms. However, the implementation of educational games as part of the curriculum and classroom practices has been rather scarce. One of the main barriers is that teachers face to actually know how their students are using the game so that they can analyze properly the effect of the activity and the interaction of students. Therefore, to support teachers to fully leverage the potential benefits of games in classrooms and make data-based decisions, educational games should incorporate learning analytics by transforming click-stream data generated from the gameplay into meaningful metrics and present visu- alizations of those metrics so that teachers can receive the information in an interactive and friendly way. For this work, we use data collected in a case study where teachers used Shadowspect geometry puzzle games in their classrooms. We apply learning analytics techniques to generate a series of metrics and visualizations that seek to facilitate that teachers can understand the interaction of students with the game. In this way, teachers can be more aware of the global progress of the class and each one of their students at an individual level, and intervene and adapt their classes when necessary. Keywords: Educational games · Learning analytics · Game-based as- sessment · Technology-enhanced Learning. 1 Introduction Playing games is one of the most popular activities for all ages of people around the world, and its value as a learning opportunity has been widely accepted. In U.S., for example, nearly three-quarters (74%) of parents believe video games can serve an educational purpose for their children [7]. This has prompted a rapidly ? Both first and second authors contributed equally to this work. Copyright c 2020 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). 98 P. A. Martı́nez et al. increasing interest in using games in educational settings, not simply because “it is what kids are paying attention to,” but because well-designed games are very closely aligned with the design of good educational experiences [16, 8]. That is, well-designed games pose cognitively complex and challenging problems that deeply engage learners, thus helping them to learn more [20]. The benefits of games to support student learning have been well documented over past 10 years. In a recent meta-analysis study, Clark and his colleagues [6] report that compared to nongame conditions, games had a moderate to strong effect for improving overall learning outcomes including cognitive and interper- sonal skills. Another review [4] similarly reports games are beneficial for learning of various outcomes such as knowledge acquisition, affect, behavior change, per- ception and cognition as well as so called 21st Century Skills [21]. While ample evidence shows that games, in general, have a great potential to support learning, only when combined with a thoughtful curriculum, games can be successfully used to support learning in classrooms [26]. Successful and meaningful integration of game-based learning in classrooms largely depends on teacher’s practices and classroom-contexts [15]. Several studies have pointed out the difficulties that teachers face to implement games in the classroom including the rigidity of the curriculum, the perceived negative effects of gaming, unpre- pared students, the lack of supportive materials, fixed class schedules and limited budgets. Time is another limited resource; spending class time on learning com- plex controls or watching long tutorials or teachers using their planning time to familiarize themselves with the educational components of the game can be limited [14]. Moreover, teachers often do not have the proper tools to understand how students interact with the game environment and in what areas students might need support. There are several options here, but one common approach has been to provide learning analytic dashboards that can represent the low level interactions in simple visualizations. This can bring opportunities for awareness, reflection, sense-making and, above all, about the potential to improve learning, that is, to get better at getting better [28]. For this educational purpose, we are using Shadowspect, an interactive com- puter game where students can create geometrical primitives such as cones or spheres to solve 3D puzzles, developing their geometric, dimensional and spatial reasoning skills. Shadowspect features 30 levels of gradually increasing difficulty where students need to move and rotate shapes in order to find solutions to modeling problems. Using Shadowspect, students’ interaction with the game is collected in order to analyze different metrics like active time, a number of puz- zles completed or the total number of events performed while playing. These metrics can be presented graphically so that teachers can observe each students’ progress in the game and find problems that the whole classroom could be having with a specific puzzle. More specifically, for this research we have the following objectives: 1. To present a proposal of metrics that can help teachers to understand the interaction of students with Shadowspect. Visualizing Educational Game Data 99 2. To present a case study with two uses cases from data collected in K12 schools across the US using Shadowspect: (a) A first use case using these metrics to understand the global progress in an entire classroom. (b) A second use case using these metrics to understand students’ progress in a classroom at an individual level. The rest of the paper is organized as follows. Section 2 reviews background literature on student engagement and learning analytics. Section 3 describes the methods, including Shadowspect, as well as the educational context and the data collection. Section 4 presents our proposal of metrics, and in Section 5 we introduce and describe both use cases using the previous metrics and visualiza- tions. Then we finalize the paper with discussion, conclusions and future work in Section 6. 2 Related Work With current data collection technologies, we can collect large datasets from students’ interaction with educational games that need to be treated in order to be understood [1]. Learning Analytics (LA) is a field of research and practice that aims to collect and analyze data generated by a learner in a given environment, which in this case can be applied to educational game data. This data analysis is not only useful for the evaluation of the students [13], but it can also be used for future improvements in the design of educational games, to personalize the difficulty of the scenarios according to the student’s abilities [2] or as in our case study, to identify strange behaviors or difficulties of the student when facing a task. Finally, one of the main advantages of data analysis is to increase student engagement and improve learning, as engagement and learning are linked [5, 9]. This personalized adaptation of scenarios and difficulty per student goes one step further with multimodal learning analytics, which aims to collect data external to the game such as the student’s heart rate [18], and can be used for multiple purposes, such as to adjust the game difficulty based on the identified problems and levels of concentration [27]. Researchers often implement metrics to make sense of the collected. raw data through a feature engineering process [17, 19], which are higher level information measures extracted from the data and according to the specificities of each met- rics’ purpose. In our case, the algorithms seek to obtain metrics related to the levels of activity, difficulty and other patterns, as other previous worked have attempted [11], so that we can predict behavior or assess the student based on what the instructor con learn from those metrics. Each environment might have specific metrics, however, there are some that are often implemented, such as the number of events or the time within the educational game. In some previous studies, the main goal has been to measure the engagement of each student. Authors in [22] differentiated four dimensions: the general activ- ity, social, exploration and quests, and they found four profiles of engagement: 100 P. A. Martı́nez et al. fully engaged, lone achiever, social explorer and non-engaged. In this study we implement similar metrics: for the general activity dimension, we implement a series of levels of activity and for the exploration dimension we analyze the funnel in the game puzzles. Another example of the importance of metrics for evaluation is described in this paper [12], which deals with the application of game-based learning to mathematics contents. Its aim is to study improvements with previ- ous training through play and to see if those metrics can be indicators of success. The study was carried out with students of about 10 years old that completed mathematical tasks about rational numbers, using the game Semideus School. The control of the behavior and performance of the students was recorded with different metrics, similar to our case but adapted to their type of game, and they proposed parameters such as time spent, the maximum level reached, number of games played or general performance. As a conclusion, the use of metrics in game-based learning as part of the evaluation process stands out. Despite the proven benefits of educational games in learning [21], their imple- mentation in schools remains rather limited. Problems such as the lack of both computational and additional human resources, teacher’s rejection of new teach- ing methods, the fact that some teachers still believe that the implementation of educational games is a complex process that is beyond their reach, make it very difficult to systematically expand the implementation of games in educational settings. That is why providing guidelines and facilitating a simplified deploy- ment of these games is so important, so that their implementation can greatly benefit teachers and students [3]. One of the reported methods to facilitate the adoption of educational games in the classroom is to provide visualization dash- boards that can represent easy, interactive and intuitive visualizations, a series of interactions of the student with the environment. Previous studies have made this proposal in other types of learning environments, such as for massive open online courses [23], or intelligent tutoring systems [10]. In our paper, we propose a set of visualizations of the data collected in the educational game Shadowspect [25], as a tool for teachers to detect problems within a class or with a particular student as proposed by previous work [24]. With a visualization dashboard sys- tem in place, the teacher can monitor what students are doing with the game during the class period and intervene during the development of the activity when appropriate or even use these metrics as part of formative assessment. 3 Methods 3.1 Shadowspect Tool In this case study we use Shadowspect, a game-based assessment tool that aims to provide metrics related to geometry content and other behavioral and cog- nitive constructs. Shadowspect has been designed explicitly as a formative as- sessment tool to measure math content standards (e.g. visualize relationships between 2D and 3D objects), thus teachers can use it in their core math cur- riculum. Visualizing Educational Game Data 101 When students begin a puzzle, they receive a set of silhouettes from different views that represent the figure they need to create, which will be composed of other primitive shapes the student can put into the scenario. The primitive shapes that students can create are cubes, pyramids, ramps, cylinders, cones and spheres. Depending on the level and difficulty, the puzzle may restrict the quantity or type of shapes they can create. After putting these shapes in the scenario, they can also scale, move and rotate the shapes in order to build a figure that solves the puzzle. Students can move the camera to see the figure they are building from different perspectives and then use the ‘Snapshot’ functionality to generate the silhouette and see how close they are to the objective. Finally they can submit the puzzle and the game will evaluate the solution and provide them with feedback. Fig. 1: Two puzzle examples in Shadowspect In the version of Shadowspect that we have used in this case study, we have 9 tutorial levels, 9 intermediate and 12 advanced. The tutorial levels aim to teach the basic functionality of the game, so the students can learn how to build different primitives, scale and rotate them, how to change the perspective, take snapshots and so on. The intermediate levels allow students more freedom so they will not receive so much help to solve puzzles and then the advanced levels pretend to be a real challenge for students who have gained experience with previous levels before. 3.2 Educational Context and Data Collection The data used (N = 300) for this paper was collected as part of assessment machinery development that later will be implemented in Shadowspect. Due to the goal of ensuring that we have sufficient data points to create robust assessment models, the team recruited 7 teachers who can use Shadowspect at least two hours in their 7th grade and 10th grade math and geometry classes. In this paper, we focus on a single class with 31 students to represent a real case scenario of how a teacher could use these visualizations to monitor the progress of their students in the classroom. All student interactions with the game were 102 P. A. Martı́nez et al. collected and stored in a MySQL database, we do not collect any identifiable or personal data from the users except for a nickname. The data collection of the selected class with 31 students includes around 54829 events (an average of 1768 events per user); students were active in the game environment for 33 hours (an average of 65 active minutes per student), and students solved a total of 448 puzzles (an average of 14 puzzles per student). 4 Metrics Proposal For the analysis of the data within our game we have defined four different metrics: Levels of activity and difficulty, puzzle funnel and sequences between puzzles. With these metrics we obtain a detailed analysis of the students’ in- teraction with the game, so the teacher can receive detailed information that can be used to evaluate and analyze students’ interactions. With this analysis we cover the most important aspects in order to monitor and analyse students’ activity in a simple way. In addition, we want to explore the affordances of com- bining several metrics together to augment their significance. Next, we explain the implementation details of each metric. – Levels of activity: This metric implements a set of parameters that de- scribe the levels of activity of the user with Shadowspect. These are straight- forward metrics to compute based on a feature engineering process, such as the active time, number of events, different type of events, and number of different types of events like snapshots, rotations, movements, scaling, shape creations and deletions, among several others. However, for this case study we focus only on the two first types of events that we mentioned because these are the most important to look at when analyzing students’ interac- tion with the game, however we would like to denote that all of them are available for the teacher. • active time: Amount of active time in minutes establishing an inactiv- ity threshold of 60 seconds (i.e. if the time between two events is above 60 seconds, the user is considered to be inactive during that time and this slot is omitted from the computation). • n events: Total number of events triggered within the game (every ac- tion performed by a student in Shadowspect is recorded as an event). – Levels of Difficulty: This metric provides a set of parameters that are related to the difficulty of the puzzles: • completed time: This parameter is computed by dividing the amount of time invested in the game (active time) by the number of completed puzzles. • actions completed: This parameter is computed by dividing the num- ber of actions (n events) by the number of completed puzzles. • p incorrect: This parameter is calculated dividing the number of incor- rect attempts by the total number of attempts (n attempts) multiplied by 100. Visualizing Educational Game Data 103 • p abandoned: This parameter is computed by dividing the number of started puzzles by the number of completed puzzles. • norm all measures: First the four aforementioned parameters for this metric are standardized by computing the z -scores of each metric (i.e. z = x−µ σ where µ is the mean and σ the standard deviation of x). Then, we make the sum of the standardized parameters: zall measures = zcomp time + zactions comp + zp incorrect + zp abandoned (1) Finally, the calculated parameter is normalized between 0 and 1. The parameters minz all measures and maxz all measures are the minimum and maximum of the zall measures . (zall measures − minz all measures ) normall measures = (2) (maxz all measures − minz all measures ) – Puzzle funnel: A conversion funnel is an e-commerce term that describes the different stages in a buyer’s journey leading up to a purchase. We use this same metaphor to illustrate the stages that a student can go through in order to solve a puzzle. We define the following four stages for the funnel: started (if the student has started the puzzle), create shape (if the student has set up an object into this particular puzzle), submitted (if the student checked the puzzle solution) and completed (if the student has submitted the puzzle and the solution is correct). Then, we analyse the data to count the stages reached for each one of the puzzles and by each student. This funnel seeks to provide a quick overview of the current status for each student and puzzle in a class. – Sequence between puzzles: Although Shadowspect puzzles are divided in three different categories based on its difficulty, they do not have to be completed in a linear sequence. Therefore, students can jump from any puz- zle to another, regardless of its difficulty, pursuing their own interests and exploring the game. In this metric our objective is to analyze the temporal interaction of students with the puzzles following a chronological order, so that we can reconstruct the sequence of puzzles for a given student. We pro- vide an output with the sequence of puzzle attempts in chronological order and the funnel state the student reached in each attempt. 5 Case Study This section presents the case study that exemplifies how teachers can use these learning analytics visualizations in two different situations: the first use case in Subsection 5.1 applies the metrics and visualizations to understand the global progress in a classroom whereas the second use case in Subsection 5.2 is focused on the individual progress of students in a classroom. 104 P. A. Martı́nez et al. 5.1 Use Case 1: Classroom Analysis The first analysis implements visualizations at the classroom level, so that the teacher can see the overall progress of the class for each particular puzzle. This can be useful to detect which puzzles are harder or which contents areas re- quire extra additional explanations or support from the teacher. Figure 2 rep- resents the first visualization with the puzzle funnel metric with its four stages as described in Section 4. We represent started (blue), create shape (yellow), submitted (red) and completed (green) in a circular chart for each one of the puzzles, which are arranged following the sequence order of appearance in Shad- owspect game. For each one of the puzzles, the first sentence on the top indicates its name and the second sentence the category of the puzzle. Tutorial Levels Tutorial Levels Tutorial Levels Tutorial Levels Tutorial Levels Tutorial Levels Tutorial Levels Tutorial Levels One Box Separated Boxes Rotate a Pyramid Match Silhouettes Removing Objects Stretch a Ramp Max 2 Boxes Combine 2 Ramps 87.5% 84.4% 84.4% 84.4% 81.2% 78.1% 75% 75% 87.5% 87.5% 84.4% 84.4% 81.2% 78.1% 78.1% 78.1% 87.5% 87.5% 84.4% 84.4% 84.4% 81.2% 78.1% 78.1% 90.6% 87.5% 84.4% 84.4% 84.4% 81.2% 78.1% 78.1% Tutorial Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels Intermediate Levels 9. Scaling Round ObtsDegree Rotations Angled Silhouette Bird Fez bBoxes Obscure Spheres Object Limits Pi Henge Pyramids are Strange 68.8% 56.2% 46.9% 62.5% 40.6% 43.8% 71.9% 53.1% 75% 56.2% 59.4% 71.9% 59.4% 56.2% 71.9% 59.4% 75% 59.4% 62.5% 71.9% 59.4% 56.2% 71.9% 59.4% 75% 65.6% 62.5% 71.9% 62.5% 56.2% 75% 65.6% Funnel Intermediate Levels Advanced Levels Advanced Levels Advanced Levels Advanced Levels Advanced Levels Advanced Levels Advanced Levels Square Cross-Sections Bear Market Bull Market Few Clues M nM. T. Meets Your Eyessye Not Bird Orange Dance Ramp Up and Can It 68.8% 3.1% 21.9% 15.6% 25% 28.1% 9.4% 31.2% 75% 28.1% 21.9% 18.8% 31.2% 28.1% 21.9% 34.4% 75% 37.5% 21.9% 25% 31.2% 28.1% 37.5% 34.4% 75% 43.8% 25% 25% 34.4% 31.2% 46.9% 37.5% Advanced Levels Advanced Levels Advanced Levels Advanced Levels Advanced Levels Advanced Levels Stranger Shapes Sugar Cones Tall and Small Unnecessary Warm Up Zzz 40.6% 46.9% 34.4% 25% 59.4% 21.9% 46.9% 53.1% 40.6% 31.2% 62.5% 28.1% 46.9% 53.1% 43.8% 31.2% 62.5% 28.1% 50% 53.1% 43.8% 34.4% 62.5% 28.1% Percentage Funnel Completed Submitted Create_shape Started Fig. 2: Funnel for each puzzle in a classroom. With this overview of the entire class, the teacher could quickly detect strange behaviours or problems in a particular puzzle. For this specific class, at the tutorial level we see that the puzzle funnel is good, having in most cases an identical number of puzzles in stage completed and in started, which means that most puzzles that were started, were also completed. For the intermediate level, we can see that is more frequent to find puzzles with a higher number of started than completed, which is indicative of the increased difficulty. Once we review the advanced puzzles, we identify several puzzles that might represent that students are facing issues to solve them, for example for “Orange Dance” Visualizing Educational Game Data 105 (46.9% started, 9.4% completed) and “Bear Market” (43.8% started, 3.1% completed), we find that the percentage of started puzzles is much higher than the percentage of completed puzzles. Once these puzzles have been identified, the next step is to check whether this is due to the high difficulty of the selected puzzles, conceptual issues or other factors. To do this, we analyze the puzzles with the levels of difficulty metric. completed_time completed_actions 8 200 Number of actions 6 150 Minutes 4 100 2 50 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 p_incorrect p_abandoned 100 75 75 Percentage Percentage 50 50 25 25 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 norm_all_measures 1.00 0.75 Difficulty 0.50 0.25 0.00 4. ota ed Box 9. . C . M a R cts s s a ota e p ll Zzz Be Da s R h S yr s 6. ovin oue id re O s al b 2 mp ua Ro 2 R xes ro O s Bi ons Bo ram ree i He z cu St s ec ts O Sph ge tL s ed r s ng C e or Ra Ta r S es an Up d S es ts Ca l ur It rd t M e t N ye ee d al O ew rke ke Py eg P Fe 5. atc a P xe St g tte C d mp bs re tion ec ere gl Wa mit e e xe id R ng ar nc Si m U ra ar tt em i am -S c Yo n Bi M an m e on n p ng Clu E St ug oue re ran 8 7 h je ss bje ar Sc om ax a M te Bo Sq ing ine Bo ti e m ll a ha F a i re un a rd R rat e tc b ot M 3. pa On S lh lh Bu Se 1. bj ra Th p An O -D 2. 45 M Fig. 3: Levels of difficulty by puzzle in a classroom. In the Figure 3, we represent the metric of levels of difficulty with the four parameters and the composite measure. In the x -axis we have each one of the puzzles following the sequence order of Shadowspect, and the y-axis represent the difficulty parameter as explained in Section 3. For both puzzles, “Orange Dance” and “Bear Market”, that we identified as having a problematic funnel in Figure 2, we see that their difficulty parameters are also high. In the time required to complete the puzzle, we have that “Orange Dance” has a lower time with 2 minutes on average, while “Bear Market” goes up to 8 minutes, so there is quite a difference between both, as in the number of actions to complete the puzzle, which is of 50 actions and 200 respectively. However, for the percentage of incorrect attempts and abandoned we get almost equal values for both puz- zles, with percentages close to 100%. Finally, the composite difficulty measure 106 P. A. Martı́nez et al. shows what already appeared in the previous parameters, for “Orange dance” the difficulty is around 0.8 and for “Bear Market” it is 1, the most difficult puz- zle. With this last parameter of the metric, teachers can get a quick overview of the puzzles that are posing a major challenge among their students. 5.2 Use Case 2: Individual Student Analysis In this second use case we select a particular group of students in the classroom to see how they are progressing in the completion of the different game puzzles. Then, we will see how a teacher can observe students’ progress and difficulties at individual levels based on three of the metrics we defined in Section 4. Student 151 Student 155 Student 160 Student 188 Student 202 33.3% 46.7% 90% 96.7% 23.3% 46.7% 66.7% 96.7% 100% 30% 46.7% 70% 100% 100% 33.3% 46.7% 100% 100% 100% 33.3% Student 220 Student 232 Student 238 Student 247 Student 250 Funnel 43.3% 23.3% 3.3% 46.7% 46.7% 63.3% 30% 13.3% 50% 50% 63.3% 30% 16.7% 50% 50% 73.3% 30% 16.7% 50% 50% Student 277 Student 278 Student 285 Student 287 Student 290 70% 3.3% 66.7% 16.7% 53.3% 70% 10% 70% 16.7% 60% 73.3% 10% 70% 20% 60% 73.3% 16.7% 70% 20% 60% Percentage Funnel Completed Submitted Create shape Started Fig. 4: Individual funnel for each student in a classroom. As we can see in Figure 4, we have selected 15 students of the group. Using the funnel metric, the teacher can easily see the relationship between the number of started puzzles and the number of completed puzzles for every student. While some students show good progress as they have high completion rates (Student 188 has 96.7% completed of the total of puzzles available in the game), others reveal that they have been struggling with some puzzles. For example, Student 155 has started every puzzle in the game but only completed 46.7% of them. We are going to focus on this last student and figure out if the student is struggling with solving the tasks or if the student did not put enough effort. Seconds Difficulty Se Se R par On O 0.00 0.25 0.50 0.75 1.00 300 100 0 200 ot at e a e pa ne 45 Mat te d B Bo R ra -D ch a P ox x e S y e ot ted Box R gre ilh ram s em e o a B o R ue id M te a ox St vin ota ttes re g O tio at e t n ch Py s Sc Co M ch bje s R S r al m ax a R cts em ilh am in bi 2 a g ne ov ou id R 2 Bo mp An ou x in e gl nd Ra es ed O m St g O ttes Bo S b ps re xe Be ilho jec tc bje s h O ar ue ts ct M bs M tt a s or cu B ark e e r e ird et C Ma R Th Sc om x 2 am an Bu Sph Fez l l e al bin B p M F M res ee e ar i o ts w C ke Sq ng R e 2 xe Yo lu t ur es ua o R s O N E r e un am O bje ot ye active_time Py ra ct Bi C d O ps ra ng L rd ro R m id e im Bo ss bje am s P Da its -S cts p a i nc xe Funnel Sq U re SHen e p t g s ec ua re an ra e O tio C d ng C e bs B n St ros Sa an cu ird s ra s- nd It re F ng Se bo Su er S tio x c S e O ph z Ta ga ha ns bj ll r C pe ec ere U and on s nn S es s Completed ec m Be t Lim a W ess ll ar its ar ar m y Su Ma U p ga rk Zz z R r C et am on Se #Events p p es R ar On M U 0 ot at e p 50 a e 100 150 or Submitted 45 Mat te d B Bo -D ch a P ox x e an Zz e S y e Th d z R gre ilh ram s an C em e o a o R ue id M Few n I St vin ota ttes re g O tio ee C t t n ts lu Sc Co M ch bje s al m ax a R cts O You es in bi 2 a g ne ra r 5.1 Sequence of puzzles. R 2 Bo mp An ou n gl d a s R xe St ng Ey ed O m ra e D e Create shape Bo S b ps ng xe s Be ilho jec e an O ar ue ts M bs M tt Ta r Sh ce or cu B ark e ll e r e ird et an ape Th an Bu Sph Fez d s l e Sm M F l M res ee e ar Started ts w C ke N all Yo lu t 45 o ur es -D B t B O N E eg ull ir n_events O bje ot ye Py ra ct Bi ng L rd re M d ra R mid e im An e R ark am s P Da its Py gl o e Sq p a i nc ra ed tat t U re SHen e ua p m io re an tra ge id Silh ns C d ng s C e for the teacher to review the actions of the student through puzzles. St ros Sa an ar oue ra s- nd It ng Se bo e c St tte Su er S tio x ra Visualizing Educational Game Data Ta ga ha ns Fig. 5: Sequence of actions and levels of activity for Student 155. ll r C pe W nge ar U and on s m nn S es ec m a Pi Up W ess ll 5.2 Number of seconds spent and number of events performed in each puzzle. ar a r m y U H en p ge Zz z what the student has done in each task, we introduce levels of activity metric the different puzzles without doing anything else, but as it is not sure. To know put a shape into it. We could think the student has been entering and exiting plot is that there are a lot of puzzles that the student has started but not even levels and levels with low difficulty. Another thing we can figure out from this the y-axis. The puzzles completed by the student are, in most cases, tutorial incorporated the difficulty metric of each puzzle by adjusting the position on color of the dots represents the funnel stage of each puzzle, and then we have axis with the dots represent the sequence of puzzles of the student, while the In Figure 5.1, we combine three different metrics at the same time: The x - 107 108 P. A. Martı́nez et al. From Figure 5.2, we can draw some conclusions about the student interac- tion. From the previous metric we know that puzzle level named as “Zzz” was submitted and then we see the active time and n events in this puzzle has been high. So we know the student has spent some time trying to solve it, and we could now say that the student has experienced difficulties with this puzzle. As we could imagine, most part of tasks that were in a started stage, have in- significant values of active time and n events performed so it can be assumed that the student has entered the puzzle but not even tried to solve it. 1.00 0.75 Difficulty 0.50 0.25 0.00 x s s ps id s s p ts ns z ge ns e s Bo Fe xe tte ct xe re am ng ec m am io io en e he Bo Bo ue ra ra bj bj at ct rd R ne H R Py Sp St O O Se ot o Bi a d 2 O Pi lh 2 R te g nd h e - ax a re Si ss in ne tc ar ra e e ou cu M ov re re ro ch pa at bi s R eg bs St id em C ot om at Se m g R O re -D M R in C ra ua s 45 al Py xe Sc Sq Bo Funnel Completed Submitted 6.1 Sequence of puzzles active_time n_events 300 100 Seconds #events 200 50 100 0 0 a dB x 45 tch Pyr s O s ro Str e a dB x id ov ota s St g O ns s Sc om x 2 mp R 2 R es cu ird s Sp ez Sq mid P res 45 tch Pyr s ou Ra s O s ro Str e -S ge ns re ou d ov ta s St g O ns s Sc om x 2 mp cu ird s Sp ez Sq mid P res - S ge ns R rate Bo R ate Bo e te ct nd amp B ct e em e R tte ct g e 2 xe nd mp B ct g i g -D S am -D S am F F M te a ox in tio in ine Box tio M te a ox in tio tio re are en ss an re are en ss an tc bje tc bje e e R ee uet C Ma Ra he C Ma Ra he o e bj bj e ne ec al bin B ec ua s i H H pa On o a o a O i eg ilh eg ilh em R h h re re ou ar re re ua s r b p R bs bs C C ot ot Se Se a a g O O R in ra ra s s al Py Py xe xe Bo Bo 6.2 Number of seconds spent and number of events performed in each puzzle. Fig. 6: Sequence of actions and levels of activity for Student 247. Now we have analysed a student experiencing certain issues with the game, let’s focus on a student with good progress to see the sequence of puzzles and the levels of activity this other student has. In Figure 6.1, we now see a very linear Visualizing Educational Game Data 109 progression, where the student has not entered a level until the previous one was already completed, so the student has been working fine but maybe needs more time than other students to solve the same amount of tasks. In Figure 6.2 we see that Student 247 has spent more than 300 seconds in task “45-degree rotation”, but finally completed it, which shows high resiliency. The time spent by the student is regularly distributed in the different levels that have been solved. 6 Conclusions and Future Work The objective of this study was twofold: First to propose a series of metrics that can provide comprehensive information regarding the process of students with the puzzles in Shadowspect and second to achieve simple but detailed visualiza- tions of these metrics that can allow teachers to track the students within their class, so that they can evaluate or detect problems quickly and effectively. In Sec- tion 4, we proposed four metrics that can generate explainable insights regarding the interaction of students with puzzles; the main goal was to provide easy-to- understand and actionable information for teachers. In the Subsection 5.1, we reported a set of visualizations at a classroom level for those metrics, so that teachers can evaluate the learning process of the class in a global way. Within the overall analysis, the teacher can identify the puzzles where students have found problems and analyse the cause with the difficulty metric. This approach can help alleviate one of the main problems when implementing educational games in the classroom, which is the possibility to better understand how students interact with the game and the overall progress. Then, in Subsection 5.2 we pre- sented a use case where teachers can monitor the individual activity of a student in Shadowspect using visualizations of the metrics. We analysed two different individual students, one of them with poor performance in the resolution of puz- zles, and another showing a better puzzle funnel, but with a low percentage of completed puzzles in relation to the total number of levels available in the game. These two analyses exemplify how a teacher can assess the status of a student in the class and solve the possible problems a student might be having during a session. This represents an opportunity for educators to provide personalized attention to their students and help them in their learning process. The next stage is to use this opportunity to implement just-in-time interven- tions that aim to provide support at the right time by adapting to the needs of each individual. One of the main limitations of this work is that these are offline static visualizations, and thus inflexible and not scalable. Therefore, as part of the future work we will be working on the co-creation of a dashboard for teachers that can provide greater speed and interactivity when displaying data from a class or individual students, and hence enabling just-in-time inter- ventions during the sessions. Also we will be working on obtaining evidences of the interpretability of these visualizations and to make them explainable so that teachers can easily intervene. Shadowspect is designed as a formative as- sessment tool, and thus we can also use these visualizations for students so that they can receive feedback and improve their self-awareness. More nuanced met- 110 P. A. Martı́nez et al. rics and visualizations will allow students to visualize their mistakes and areas of improvement. In this way we can use Shadowspect as a robust learning tool with that can be easily implemented by teachers in the classroom and that em- phasizes the formative feedback to the student. This study has proposed a new dynamic approaches that can be helpful to facilitate systematic implementation of educational games in the classrooms of the future. Acknowledgements We want to acknowledge support from the MIT-SPAIN “la Caixa” Founda- tion SEED FUND and the Spanish Ministry of Economy and Competitiveness through the Juan de la Cierva Formación program (FJCI-2017-34926). References 1. Alonso-Fernandez, C., Calvo, A., Freire, M., Martinez-Ortiz, I., Fernandez-Manjon, B.: Systematizing game learning analytics for serious games. In: 2017 IEEE Global Engineering Education Conference (EDUCON). pp. 1111–1118. IEEE (2017) 2. Alonso-Fernández, C., Cano, A.R., Calvo-Morata, A., Freire, M., Martı́nez-Ortiz, I., Fernández-Manjón, B.: Lessons learned applying learning analytics to assess serious games. Computers in Human Behavior 99, 301–309 (2019) 3. Alonso-Fernández, C., Perez-Colado, I., Freire, M., Martı́nez-Ortiz, I., Fernández- Manjón, B.: Improving serious games analyzing learning analytics data: lessons learned. In: International Conference on Games and Learning Alliance. pp. 287– 296. Springer (2018) 4. Boyle, E.A., MacArthur, E.W., Connolly, T.M., Hainey, T., Manea, M., Kärki, A., Van Rosmalen, P.: A narrative literature review of games, animations and simulations to teach research methods and statistics. Computers & Education 74, 1–14 (2014) 5. Chew, B.S.: An efficient framework for game-based learning activity. In: 2017 IEEE 6th International Conference on Teaching, Assessment, and Learning for Engineer- ing (TALE). pp. 147–150. IEEE (2017) 6. Clark, D.B., Tanner-Smith, E.E., Killingsworth, S.S.: Digital games, design, and learning: A systematic review and meta-analysis. Review of educational research 86(1), 79–122 (2016) 7. ESA: 2019 essential facts about the computer and video game industry. Tech. rep., Entertainment Software Association (2019) 8. Gee, J.P.: Are video games good for learning? Nordic Journal of Digital Literacy 1(03), 172–183 (2006) 9. Hamari, J., Shernoff, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., Edwards, T.: Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in human behavior 54, 170–179 (2016) 10. Holstein, K., McLaren, B.M., Aleven, V.: Intelligent tutors as teachers’ aides: ex- ploring teacher needs for real-time analytics in blended classrooms. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference. pp. 257– 266 (2017) Visualizing Educational Game Data 111 11. Kang, J., Liu, M., Qu, W.: Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior 72, 757–770 (2017) 12. Kiili, K., Moeller, K., Ninaus, M.: Evaluating the effectiveness of a game-based rational number training-in-game metrics as learning indicators. Computers & Ed- ucation 120, 13–28 (2018) 13. Kim, Y.J., Ifenthaler, D.: Game-based assessment: The past ten years and moving forward. In: Game-Based Assessment Revisited, pp. 3–11. Springer (2019) 14. Kirriemuir, J., McFarlane, A.: Use of computer and video games in the classroom. In: DiGRA Conference (2003) 15. Klopfer, E., Osterweil, S., Groff, J., Haas, J.: Using the technology of today in the classroom today: The instructional power of digital games, social networking, simulations and how teachers can leverage them. The Education Arcade 1, 20 (2009) 16. Koster, R.: Theory of fun for game design. ” O’Reilly Media, Inc.” (2013) 17. Mäses, S., Hallaq, B., Maennel, O.: Obtaining better metrics for complex serious games within virtualised simulation environments. In: European Conference on Games Based Learning. pp. 428–434. Academic Conferences International Limited (2017) 18. Ninaus, M., Tsarava, K., Moeller, K.: A pilot study on the feasibility of dynamic difficulty adjustment in game-based learning using heart-rate. In: International Conference on Games and Learning Alliance. pp. 117–128. Springer (2019) 19. Plass, J.L., Homer, B.D., Kinzer, C.K., Chang, Y.K., Frye, J., Kaczetow, W., Isbister, K., Perlin, K.: Metrics in simulations and games for learning. In: Game analytics, pp. 697–729. Springer (2013) 20. Prensky, M.: Digital game-based learning. Computers in Entertainment (CIE) 1(1), 21–21 (2003) 21. Qian, M., Clark, K.R.: Game-based learning and 21st century skills: A review of recent research. Computers in Human Behavior 63, 50–58 (2016) 22. Ruiperez-Valiente, J.A., Gaydos, M., Rosenheck, L., Kim, Y.J., Klopfer, E.: Pat- terns of engagement in an educational massive multiplayer online game: A multi- dimensional view. IEEE Transactions on Learning Technologies (2020) 23. Ruiperez-Valiente, J.A., Munoz-Merino, P.J., Gascon-Pinedo, J.A., Kloos, C.D.: Scaling to massiveness with analyse: A learning analytics tool for open edx. IEEE Transactions on Human-Machine Systems 47(6), 909–914 (2016) 24. Serrano-Laguna, Á., Martı́nez-Ortiz, I., Haag, J., Regan, D., Johnson, A., Fernández-Manjón, B.: Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces 50, 116–123 (2017) 25. Smith, S.P., Hickmott, D., Southgate, E., Bille, R., Stephens, L.: Exploring play- learners’ analytics in a serious game for literacy improvement. In: Joint Interna- tional Conference on Serious Games. pp. 13–24. Springer (2016) 26. Squire, K.: Video games and learning. Teaching and participatory culture in the digital age (2011) 27. Tsai, M.J., Huang, L.J., Hou, H.T., Hsu, C.Y., Chiou, G.L.: Visual behavior, flow and achievement in game-based learning. Computers & Education 98, 115–129 (2016) 28. Verbert, K., Duval, E., Klerkx, J., Govaerts, S., Santos, J.L.: Learning analytics dashboard applications. American Behavioral Scientist 57(10), 1500–1509 (2013)