=Paper=
{{Paper
|id=Vol-2844/games3
|storemode=property
|title=Towards Game-based Assessment of Executive Functions in Children (short paper)
|pdfUrl=https://ceur-ws.org/Vol-2844/games3.pdf
|volume=Vol-2844
|authors= Alexis Lueckenhoff, Callen Wessels, Maria Kyrarini, Fillia Makedon
|dblpUrl=https://dblp.org/rec/conf/setn/LueckenhoffWKM20
}}
==Towards Game-based Assessment of Executive Functions in Children (short paper)==
https://ceur-ws.org/Vol-2844/games3.pdf
Towards Game-based Assessment of Executive Functions in
Children
Alexis Lueckenhoff Callen Wessels
Computer Science and Engineering Department Computer Science and Engineering Department
The University of Texas at Arlington The University of Texas at Arlington
Arlington, Texas, USA Arlington, Texas, USA
alexis.lueckenhoff@uta.edu callen.wessels@uta.edu
Maria Kyrarini Fillia Makedon
Computer Science and Engineering Department Computer Science and Engineering Department
The University of Texas at Arlington The University of Texas at Arlington
Arlington, Texas, USA Arlington, Texas, USA
maria.kyrarini@uta.edu makedon@uta.edu
ABSTRACT
Executive Functions are very important mental skills that help us 1 Introduction
to coordinate, plan, pay attention, organize, and multitask, among
Executive Functions (EFs) are a set of cognitive skills that support
others. Weak executive functions may affect school or work
the regulation of thoughts, emotions, and behaviors. The role of
performance. Therefore, there is a need of identifying executive
EF is very important as they assist us to achieve goals in our daily
function deficits early during childhood and enable interventions
lives, whether planning an event, multi-tasking, or regulating
that could improve executive functioning skills. In this work, we
emotions. EFs are essential for school achievements, for the
present a game-based assessment system of executive functions in
preparation and adaptability of our future workforce, and for
children that could be performed at home. The proposed system
avoiding a wide range of health problems [1]. EFs are dramatically
utilizes machine learning techniques to detect and track head and
developed during infancy and childhood. Executive function
eye movements from image frames and fuses this data with game
deficits are common symptoms of some neurodevelopmental
performance. A novel variation of the Flanker task has been
disorders observed in children, such as Attention Deficit and
developed as a game to measure engagement, attention, working
Hyperactivity Disorder (ADHD), Learning Disability (LD), and
memory, and processing speed. In the future, the proposed system
Autism Spectrum Disorder (ASD) [2, 3]. In the U.S., according to
will be evaluated in a real-world study on children between 6 and
researchers, 9.26% of children between 6-11 years suffer from
14 years old.
ADHD, 8.02% from LD, and 1.75% from ASD [3]. Therefore, there
is a fundamental need to help children suffering from
neurodevelopmental disorders to overcome deficits of EFs. The
development of EFs requires proper assessment and intervention
CCS CONCEPTS at the appropriate time during childhood [4]. Traditionally,
• Human-centered computing ~ Human-computer interaction psychologists and medical experts have been assessing EFs
(HCI) ~ Interaction paradigms ~ Web-based interaction through written closed-ended questionnaires that the children,
their parents, and their teachers require to complete. However,
these assessments are subjective based on the personal feelings
KEYWORDS and opinions of the respondents and time-consuming as they
Game-based assessment, Executive Function, Flanker Task, Eye require multiple visits. Therefore, an objective system to assess
Gaze EFs is vital.
The NIH toolbox cognitive battery [5] is a set of computer-based
tests to assess EFs, such as working memory, inhibitory control,
GAITECUS0, September 02–04, 2020, Athens, Greece
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).
attention, and processing speed. When a test is completed, the end of the test. First, the GUI instructs the child to look at specific
NIH toolbox yields the measured scores. However, the NIH locations on the screen, as calibration is required to enable
toolbox calculates the score based on a child’s performance during accurate eye tracking. Both parent and child consent will be asked
the test. Nowadays, devices, such as smartphones, tablets, and for and is required for this functionality as well. Subsequently,
laptops, are part of the everyday life of children. Most of them the child starts playing the proposed game described in section 3.
play video games from a young age. Therefore, a child may not be During the game, image frames of the device’s camera are used to
engaged with the NIH toolbox tests and because of this, s/he may detect and track the head and eyes of the child. The head is
not perform well. detected and tracked by the framework developed in [9], which
detects the face, estimates the position and orientation of the head,
Another assessment system is the Activate Test of Embodied
and tracks the head’s pose in subsequent image frames. A
Cognition (ATEC) [6][7], which is designed to measure EFs in
recurrent Convolutional Neural Network (CNN) developed by
children through physically and cognitively demanding tasks.
[10] is used to detect the eye gaze. In parallel, game performance
Embodied cognition is a theory of cognitive psychology
is analyzed to measure game metrics, such as correctness and
suggesting that bodily actions can influence cognition [8]. The
response time. The game metrics and the head and eye
ATEC has 17 physical tasks with several variations and difficulty
movements are synchronized and a deep learning framework is
levels, designed to provide measurements of executive and motor
used for the fusion of the data. The output of the framework is the
functions. The ATEC is developed for school environments and
scores of attention, working memory, engagement, and
consists of two Kinect cameras, a large screen, and a table
processing speed, which are important EFs. The attention is
interface for the administrator. However, the ATEC system is not
scored based on the correct answers in the game combined with
suited for a home environment.
the eye gaze and head motion data. The working memory is
Moreover, children with weak EFs may stay undetected because scored based on correct answers according to the rules of the
of limited access to health professionals. Identifying issues with game and the engagement is computed by the eye gaze and head
EF early can be beneficial for the child’s development and could motion data. The processing time is computed based on the
improve the likelihood of success in school and later in life. response time in the game combined with the eye gaze. The
Therefore, it is crucial to have an assessment system of EF that is calculated scores are then grouped into three classes “low EF”,
engaging and can be conducted at home with widely-used “medium EF” and “high EF” and are sent to the parent with
everyday devices. In this paper, we propose a Game-based additional resources for EFs and contact information for experts.
Assessment Test of EFs (G-ATEF), which is web-based and
compatible with the most widely-used devices (e.g. smartphones,
laptops, tablets). Additionally, G-ATEF measures not only the
game performance metrics but also physiological measurements,
such as eye and head movements, from a camera already available
on the devices. The eye and head movements of the children
during the game can provide valuable information regarding
engagement and attention. Deep learning methods will be utilized
to identify the movements from the camera images and to
calculate the scores of attention, engagement, working memory,
and inhibition by combining the eye and head movements with
the game performance.
The rest of the paper is organized as follows; Section 2 presents
an overview of the G-ATEF system, section 3 discusses the
proposed game and section 4 concludes and provides future
directions.
2 Overview of the Proposed System
Figure 1 illustrates an overview of the proposed G-ATEF system. Figure 1: Overview of the Proposed Game-based
Assessment Test of Executive Functions (G-ATEF) System.
The G-ATEF consists of a web-based Graphical User Interface
(GUI) that is compatible with most smartphones, tablets, laptops,
etc. At the beginning of the assessment, the parents are required 3 Proposed Game
to give their consent and to provide an email so they can receive
the assessment scores and additional information about EFs at the The NIH Toolbox proposes a Flanker Inhibitory Control and
Attention Test (Flanker Task) in order to measure EFs. In the
flanker task, the subjects are required to indicate the left or right
orientation of a centrally presented arrow that is surrounded by
two arrows on either side (i.e. the flankers) [5].
In this paper, we present a variation on the Flanker Task that
strives to be more engaging for children to collect more accurate a b
results on EFs in children. In the proposed game-based assessment
task, various sharks are arranged across the screen facing left or
right. The child is directed to only focus on one. Their goal is to Figure 3: An example of the second level of the proposed
quickly identify its direction while ignoring the distractor sharks. flanker task – A spotlight will identify the focus-shark
briefly before the sharks appear (left image). The sharks
The task has different variations, or levels, in which the rules appear and the child has to identify the direction of the
slightly change. The first level is the closest to the traditional focus-shark (right image).
flanker task. The child is instructed to focus only on the center
shark and sequences of five sharks arranged horizontally or
vertically or nine sharks arranged in a grid are tested. An example
of the horizontal arrangement of the sharks is shown in Figure 2.
Level two uses a grid of nine sharks, but rather than focusing on
the middle shark, a spotlight will identify the focus-shark briefly
before the sharks appear. This spotlight-location changes every
round. Figure 3 shows an example of the second level of the
proposed game. Level three uses a grid layout of various-sized
sharks. The spotlight is used at this level as well. Figure 4
illustrates an example of level three.
Figure 4: An example of the third level of the proposed
flanker task – Grid layout of various-sized sharks.
Figure 2: An example of the first level of the proposed
flanker task – Horizontal arrangement of the sharks.
There is an additional long-term goal for the child to focus on. If
at any time during the task the child spots a dolphin anywhere on
the screen, they are to press the dolphin button rather than the
direction of the shark in focus. The child is told about the dolphin
at the beginning of level one and is not reminded for the Figure 5: An example of the long-term goal to spot a
remainder of the task. Figure 5 shows an example of the dolphin. dolphin.
In addition to collecting correctness and timing of each round,
head and eye movement data is used to discover trends in the 4 Conclusion and Future Directions
child’s attention and engagement. By analyzing the eye gaze data In this position paper, we have proposed a game-based assessment
we hope to be able to infer how the child approaches the task, why system of EFs in children. A web-based GUI has been developed
the child incorrectly identifies a shark’s direction, and for how to enable a child to play the game and the head and eye
long the child continues to look for the dolphin as the rounds movements are detected and tracked by a camera and advanced
progress. machine learning techniques. We have designed a novel game
based on the flanker task, which can measure EFs, such as
engagement, attention, working memory, and processing speed.
The proposed system has the potential to be used as a home
assessment tool, which provides parents initial indications to seek
further professional assistance. The next step of our research is to
conduct a real-world study with children in elementary school
(age range between 6 and 14 years old), to evaluate the proposed
system and its machine/deep learning algorithms.
ACKNOWLEDGMENT
This paper is based upon work supported by the National Science
Foundation under Grant No 1565328. Any opinions, findings, and
conclusions or recommendations expressed in this paper are those
of the author(s) and do not necessarily reflect the views of the
National Science Foundation.
REFERENCES
[1] Center on the Developing Child at Harvard University, 2012. Executive
Function (InBrief).
[2] Scandurra, V., Emberti Gialloreti, L., Barbanera, F., Scordo, M., Pierini, A., and
Canitano, R., 2019. Neurodevelopmental Disorders and Adaptive Functions: A
Study of Children with Autism Spectrum Disorders (ASD) and/or Attention
Deficit and Hyperactivity Disorder (ADHD). Frontiers in psychiatry, 10, p.673.
[3] Zablotsky, B., Black, L.I., Maenner, M.J., Schieve, L.A., Danielson, M.L., Bitsko,
R.H., Blumberg, S.J., Kogan, M.D., and Boyle, C.A., 2019. Prevalence and Trends
of Developmental Disabilities among Children in the United States: 2009–2017.
Pediatrics, 144(4), p.e20190811.
[4] McClelland, M.M., and Cameron, C.E., 2012. Self‐regulation in early childhood:
Improving conceptual clarity and developing ecologically valid measures. Child
development perspectives, 6(2), pp.136-142.
[5] Zelazo, P.D., Anderson, J.E., Richler, J., Wallner‐Allen, K., Beaumont, J.L., and
Weintraub, S., 2013. II. NIH Toolbox Cognition Battery (CB): Measuring
executive function and attention. Monographs of the Society for Research in Child
Development, 78(4), pp.16-33.
[6] Dillhoff, A., Tsiakas, K., Babu, A.R., Zakizadehghariehali, M., Buchanan, B., Bell,
M., Athitsos, V., and Makedon, F., 2019, September. An automated assessment
system for embodied cognition in children: from motion data to executive
functioning. In Proceedings of the 6th international Workshop on Sensor-based
Activity Recognition and Interaction (pp. 1-6).
[7] Ramesh Babu, A., Zadeh, M.Z., Jaiswal, A., Lueckenhoff, A., Kyrarini, M., and
Makedon, F., 2020. A Multi-modal System to Assess Cognition in Children from
their Physical Movements. In Proceedings of the 2020 International Conference on
Multimodal Interaction (pp. 6-14). ACM.
[8] Babu, A.R., Zakizadeh, M., Brady, J.R., Calderon, D., and Makedon, F., 2019,
August. An Intelligent Action Recognition System to assess Cognitive Behavior
for Executive Function Disorder. In 2019 IEEE 15th International Conference on
Automation Science and Engineering (CASE) (pp. 164-169). IEEE.
[9] Goldau, F.F., Shastha, T.K., Kyrarini, M., and Gräser, A., 2019, June. Autonomous
multi-sensory robotic assistant for a drinking task. In 2019 IEEE 16th
International Conference on Rehabilitation Robotics (ICORR) (pp. 210-216). IEEE.
[10] Palmero, C., Selva, J., Bagheri, M.A., and Escalera, S., 2018. Recurrent CNN for
3d gaze estimation using appearance and shape cues. arXiv preprint
arXiv:1805.03064.