• Human-centered computing ~ Human-computer interaction (HCI) ~ Interaction paradigms ~ Web-based interaction

Executive Functions (EFs) are a set of cognitive skills that support the regulation of thoughts, emotions, and behaviors. The role of EF is very important as they assist us to achieve goals in our daily lives, whether planning an event, multi-tasking, or regulating emotions. EFs are essential for school achievements, for the preparation and adaptability of our future workforce, and for avoiding a wide range of health problems [ 1 ]. EFs are dramatically developed during infancy and childhood. Executive function deficits are common symptoms of some neurodevelopmental disorders observed in children, such as Attention Deficit and Hyperactivity Disorder (ADHD), Learning Disability (LD), and Autism Spectrum Disorder (ASD) [ 2, 3 ]. In the U.S., according to researchers, 9.26% of children between 6-11 years suffer from ADHD, 8.02% from LD, and 1.75% from ASD [ 3 ]. Therefore, there is a fundamental need to help children suffering from neurodevelopmental disorders to overcome deficits of EFs. The development of EFs requires proper assessment and intervention at the appropriate time during childhood [ 4 ]. Traditionally, psychologists and medical experts have been assessing EFs through written closed-ended questionnaires that the children, their parents, and their teachers require to complete. However, these assessments are subjective based on the personal feelings and opinions of the respondents and time-consuming as they require multiple visits. Therefore, an objective system to assess EFs is vital.

The NIH toolbox cognitive battery [ 5 ] is a set of computer-based tests to assess EFs, such as working memory, inhibitory control, attention, and processing speed. When a test is completed, the NIH toolbox yields the measured scores. However, the NIH toolbox calculates the score based on a child’s performance during the test. Nowadays, devices, such as smartphones, tablets, and laptops, are part of the everyday life of children. Most of them play video games from a young age. Therefore, a child may not be engaged with the NIH toolbox tests and because of this, s/he may not perform well.

Another assessment system is the Activate Test of Embodied Cognition (ATEC) [ 6 ][ 7 ], which is designed to measure EFs in children through physically and cognitively demanding tasks. Embodied cognition is a theory of cognitive psychology suggesting that bodily actions can influence cognition [ 8 ]. The ATEC has 17 physical tasks with several variations and difficulty levels, designed to provide measurements of executive and motor functions. The ATEC is developed for school environments and consists of two Kinect cameras, a large screen, and a table interface for the administrator. However, the ATEC system is not suited for a home environment.

Moreover, children with weak EFs may stay undetected because of limited access to health professionals. Identifying issues with EF early can be beneficial for the child’s development and could improve the likelihood of success in school and later in life. Therefore, it is crucial to have an assessment system of EF that is engaging and can be conducted at home with widely-used everyday devices. In this paper, we propose a Game-based Assessment Test of EFs (G-ATEF), which is web-based and compatible with the most widely-used devices (e.g. smartphones, laptops, tablets). Additionally, G-ATEF measures not only the game performance metrics but also physiological measurements, such as eye and head movements, from a camera already available on the devices. The eye and head movements of the children during the game can provide valuable information regarding engagement and attention. Deep learning methods will be utilized to identify the movements from the camera images and to calculate the scores of attention, engagement, working memory, and inhibition by combining the eye and head movements with the game performance.

The rest of the paper is organized as follows; Section 2 presents an overview of the G-ATEF system, section 3 discusses the proposed game and section 4 concludes and provides future directions.

Figure 1 illustrates an overview of the proposed G-ATEF system. The G-ATEF consists of a web-based Graphical User Interface (GUI) that is compatible with most smartphones, tablets, laptops, etc. At the beginning of the assessment, the parents are required to give their consent and to provide an email so they can receive the assessment scores and additional information about EFs at the end of the test. First, the GUI instructs the child to look at specific locations on the screen, as calibration is required to enable accurate eye tracking. Both parent and child consent will be asked for and is required for this functionality as well. Subsequently, the child starts playing the proposed game described in section 3. During the game, image frames of the device’s camera are used to detect and track the head and eyes of the child. The head is detected and tracked by the framework developed in [ 9 ], which detects the face, estimates the position and orientation of the head, and tracks the head’s pose in subsequent image frames. A recurrent Convolutional Neural Network (CNN) developed by [ 10 ] is used to detect the eye gaze. In parallel, game performance is analyzed to measure game metrics, such as correctness and response time. The game metrics and the head and eye movements are synchronized and a deep learning framework is used for the fusion of the data. The output of the framework is the scores of attention, working memory, engagement, and processing speed, which are important EFs. The attention is scored based on the correct answers in the game combined with the eye gaze and head motion data. The working memory is scored based on correct answers according to the rules of the game and the engagement is computed by the eye gaze and head motion data. The processing time is computed based on the response time in the game combined with the eye gaze. The calculated scores are then grouped into three classes “low EF”, “medium EF” and “high EF” and are sent to the parent with additional resources for EFs and contact information for experts.

The NIH Toolbox proposes a Flanker Inhibitory Control and Attention Test (Flanker Task) in order to measure EFs. In the flanker task, the subjects are required to indicate the left or right orientation of a centrally presented arrow that is surrounded by two arrows on either side (i.e. the flankers) [ 5 ].

In this paper, we present a variation on the Flanker Task that strives to be more engaging for children to collect more accurate results on EFs in children. In the proposed game-based assessment task, various sharks are arranged across the screen facing left or right. The child is directed to only focus on one. Their goal is to quickly identify its direction while ignoring the distractor sharks. The task has different variations, or levels, in which the rules slightly change. The first level is the closest to the traditional flanker task. The child is instructed to focus only on the center shark and sequences of five sharks arranged horizontally or vertically or nine sharks arranged in a grid are tested. An example of the horizontal arrangement of the sharks is shown in Figure 2. Level two uses a grid of nine sharks, but rather than focusing on the middle shark, a spotlight will identify the focus-shark briefly before the sharks appear. This spotlight-location changes every round. Figure 3 shows an example of the second level of the proposed game. Level three uses a grid layout of various-sized sharks. The spotlight is used at this level as well. Figure 4 illustrates an example of level three. There is an additional long-term goal for the child to focus on. If at any time during the task the child spots a dolphin anywhere on the screen, they are to press the dolphin button rather than the direction of the shark in focus. The child is told about the dolphin at the beginning of level one and is not reminded for the remainder of the task. Figure 5 shows an example of the dolphin. In addition to collecting correctness and timing of each round, head and eye movement data is used to discover trends in the child’s attention and engagement. By analyzing the eye gaze data we hope to be able to infer how the child approaches the task, why the child incorrectly identifies a shark’s direction, and for how long the child continues to look for the dolphin as the rounds progress. a b

In this position paper, we have proposed a game-based assessment system of EFs in children. A web-based GUI has been developed to enable a child to play the game and the head and eye movements are detected and tracked by a camera and advanced machine learning techniques. We have designed a novel game based on the flanker task, which can measure EFs, such as engagement, attention, working memory, and processing speed. The proposed system has the potential to be used as a home assessment tool, which provides parents initial indications to seek further professional assistance. The next step of our research is to conduct a real-world study with children in elementary school (age range between 6 and 14 years old), to evaluate the proposed system and its machine/deep learning algorithms.

This paper is based upon work supported by the National Science Foundation under Grant No 1565328. Any opinions, findings, and conclusions or recommendations expressed in this paper are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.