Leveraging motion-based touchless games (MBTG) to support children's learning is appealing and technically challenging. The application of multimodal learning analytics (MMLA) can help researchers to better understand how children experience learning through movement by providing insights into children's cognitive, behavioural, interaction, and learning processes. However, there is limited knowledge about exploiting the integration of MMLA into the use of educational MBTG in children's learning. We present an in-progress study in which we conducted an experiment with 55 children, playing three different educational MBTG centred on the development of math and English competencies. We collected multimodal data from 6 different sources: eye-tracking glasses, video, wristband, game analytics, Kinect point cloud, and questionnaires. Future analysis will explore relationships between the various multimodal data, in pursuit of establishing a more holistic understanding of children's cognitive, behavioural, interaction, and learning processes experienced while engaged with MBTG for learning.
collaborations with interested parties, as well as encourage others to adopt this exciting area of research. 2
The central objective of this research is to explore how children experience educational MBTG. We aim to understand the cognitive, behavioural, and interaction processes experienced in this context and to investigate how these processes relate to the children’s learning, acceptance and perception of MBTG games, their interaction modes and knowledge to be acquired. Specifically, we suggest that when answering questions as part of the MBTG learning experience, children undergo the See-SolveMove-Select cycle (SSMS). During the SSMS cycle, children (1) see and understands the problem, (2) solve the problem mentally, (3) move their body to initiate the selection process, and finally (4) select and manipulates their answer via gestural interaction (see Figure 1). Using MMD capture, we aim to investigate the processes that occur during the different phases of the SSMS cycle, in pursuit of obtaining a holistic representation of the children’s learning experience.
Though technological advancements have only recently enabled the emergence of Motion-Based
Touchless (MBT) devices, their application in education has seen much traction, with research
permeating maths
Our study used three adaptable educational MBTG games from a commercial Kinect-based platform: Suffiz, Marvy Learns, and Sea Formuli. Each game was single player and focused on the development of math or English skills. Children interacted with the game content by moving their bodies and performing single hand mid-air hand gestures to move items on-screen. Though the focus of each game differed (Suffiz concentrated on English skills, Sea Formuli centred on arithmetic, and Marvy Learns targeted geometry or English depending on the grade setting), all of the questions presented were structured as a either a multiple-choice question or a sorting problem. Furthermore, the way that each child interacted with the game content was identical across the game play sessions. That is, to answer a question, the child needed to use a pre-defined gestural selection mode (i.e., a delay or a grab motion) to select the correct item from a collection of items and then move the selected item to a target destination. Both the delay and grab gestures were single hand movements and only recognised when performed by the child’s dominant hand. The delay gesture required the child to raise their hand, with palm facing forward, and hold it stable for a 1.5s. As the delay selection was progressing, visual feedback was displayed to the user. The grab gesture required the player to produce and maintain a grabbing gesture. In both cases, once the item was selected, it followed the child’s hand movement. Moreover, each game took a different approach to player representation within the game (i.e., level of immersion). In Suffiz, a hand shaped cursor tracked the movement of the player’s hand (low level immersion). In Marvy Learns, the players full body movement was mapped to a creature avatar (medium level immersion). Finally, in Sea Formuli, a video image of the player projected the child’s full body into the game setting (high level of immersion), see Figure 2. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
The context of our experiment takes place in two different venues. Namely, a children’s science centre and an elementary school in a European country. In both cases, researchers were present onsite during the game play sessions to assist children in understanding game play and gesture execution. Our sample was composed of 55 elementary school children with an average age of 10.27 years (min = 8, max = 11 years). 25 of the children were female and 30 were male. All of the children were typically developing. Furthermore, each child participated in 9 games play sessions (3 consecutive rounds of each of the aforementioned games) in the science centre or elementary school setting. 5.3
We conducted a four-phase within-between groups experiment to investigate the learning, behavioural and interaction processes experienced by children as they engaged with educational MBTG centred on developing math and English competencies (see Figure 3). The level of immersion (i.e., cursor, full body avatar mapping, and video of self) was the within-groups condition and the selection mode (i.e., delay, grab) was the between groups condition. We balanced the assignment of selection modes and the order of level of immersion (i.e., order in which the games were played). After obtaining parental written consent, children were given a pair of Tobii eye-tracking glasses, and an Empatica E4 wristband to wear (phase 1). Then, for each game, children played three consecutive sessions: a practice round, in which researchers assisted the child in understanding the game’s objective and rules (phase 2), and two non-practice sessions (phase 3). Finally, children filled out a questionnaire. None of the children had prior experience with MBT technologies, or Kinect games.
We collected six different data sources from each child throughout the duration of the sessions. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Eye tracking: Children wore Tobii eye-tracking glasses allowing us to capture both the eye-tracking data and the children’s field of view (objective camera on the nose-bridge).
Facial Video: We captured children’s facial expressions using a LogiTech HD web camera situated on top of the screen and directed at the child. The web camera was set to 200% zoom to enable clear capture of the child’s face.
Wrist Band: Participants wore an Empatica E4 wristband, from which we recorded 4 different measurements: 1) HR at 1Hz, 2) EDA at 64 Hz, 3) body temperature at 4 Hz, and BVP at 4 Hz. Kinect Skeleton: We collected the complete skeletal data provided by Kinect Point Cloud. Specifically, this includes information on the child’s joint movement (i.e., joint orientation, depth position), collected at successive time fixed intervals.
Game Analytics: We collected system log files containing event time stamps corresponding various child-computer interactions, such as when an item is selected and released or when a question is answered. As well, a report outlining various performance metrics, such as the child’s correctness and reaction time, was also obtained per session.
Questionnaire: This included basic demographic data, such as the child’s age, gender, and school grade, as well as 14 5-point Likert scale questions addressing their experience and emotions. 6
Our aim is to better understand how children experience educational MBTG for maths and English, by identifying and examining the cognitive, behavioural, and interaction processes that occur in learning. Specifically, we plan to investigate our proposed SSMS cycle, and how it relates to children’s learning, acceptance and perception of MBTG games, their selection modes, and the knowledge acquired. Our ongoing work on this experiment will exploit the use of MMLA. Narrowing the research scope considerably, we start by asking how the level of immersion in educational MBTG relates to student affect and behavioural processes. Our upcoming analysis will employ data captured from eye-tracking glasses, wristbands, video and Kinect sensor, to examine the relationships between various aspects of children’s embodied learning experience, such as levels of stress, arousal, fatigue, cognitive load, global and local information processing, on-task/off-task ratio, facial expression and amount of bodily movement. We hope such analysis may scaffold the understanding of processes that occur during children’s interactions with MBTG in educational contexts. As MBT technologies continue to establish themselves as rich resources for creating meaningful interactions in children’s education, we highlight the importance of this work’s relevance to the LAK community, in terms of exploring the design and assessment of learning experiences via MBTG.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).