A growing interest about alternative methods for the assessment and training of executive functions in children with neurodevelopmental disorders is emerging. Technology is considerably advancing, and the gamification is giving an important contribution in making the interventions more engaging. However, sometimes a “top-down” design process creates mismatches between technologies and both therapists and children's need. Thus, the analysis of requirements and user's feedback play a crucial role to identify their perspective regarding the technology and gauge its design and development. The present work aims to describe ASTRAS, a software for the assessment and training of executive functions in children, the requirements of therapists and children the software try to fulfil and, finally, the preliminary data reporting the experience of a sample of therapists about the application of software with their patients. In that regard, a survey was used to obtain the opinion of thirty-two therapists about the: a) usability (i.e., comprehensibility, simplicity of the software and the usefulness of its feature); b) the clinical validity of the tasks; c) the attractiveness of the tasks for the children; d) their general opinion about the software. Results demonstrated satisfying responses of the therapists about the software: they considered ASTRAS easy to use and clinically adapt to assesses and train executive functions in their patients. Furthermore, they reported that children well reacted to the software and specifically they were more likely to be engaged in training games than in assessment tasks. Bind together, our results show that ASTRAS is usable for therapists and engaging for their patients, which makes it promising for the assessment and training of executive functions in children.
Executive Functions (EFs) are an “umbrella term [
Although there is no agreement on the role played by EFs deficits in the etiopathogenesis of several neurodevelopmental disorders [5] and these differences are not enough to distinguish among different profiles [6], researchers agree that EFs impairment significantly affects the expression of symptomatology. EFs are involved in many neurodevelopment disorders [7, 8], including attention deficit and hyperactivity disorder [9, 10], autism spectrum disorder [11, 12], learning [13, 14], but also in genetic syndromes (i.e Prader-Willis)[15] or intellectual disabilities [16, 17].
Considering the importance of EFs in children with both typical and atypical development the assessment and training of these cognitive processes have received considerable attention. Importantly, the assessment and treatment of EFs should include several tasks to underline the difference among EFs [18, 19] and should pay attention to the ecological validity of such measures, proposing challenging tasks that reflect the actual cognitive demands of everyday life [20].
In literature, the use of alternative ways to assess and treat EFs, through appropriate software, is experiencing growing interest [21–26]. Gamification (i.e., the evaluation of skills through gaming activities) uses well-established procedures and classic tasks, adding all the typical game elements (scores, challenges, achievement of goals). These tasks/games are called serious games (i.e., games designed with a primary goal other than entertainment [27] and may provide a possible solution to difficulties relate to cognitive assessment. Cognitive tasks are typically viewed as effortful and repetitive, which often leads to participant disengagement. Furthermore, it negatively impacts data quality and/or reduces intervention effects. In contrast, serious game may make cognitive assessment funnier and easier[28].
Although an increasing number of technological platforms have been released, to our knowledge, no one has developed a computer-based tool for the assessment of EFs in children with neurodevelopmental disorder or at least a software in which there are both assessment and training.
Additionally, the adoption of computer-based tools has been limited since “top-down” design process is often used in creating this kind of technology [29]. This design often relies on technologists or preconceptions of the needs of therapists and children with little consideration of their perspectives and preferences. Few studies have addressed user-related issues in the design process of these software [29, 30] even though has been recognized that effective technologies are those that prioritize the needs and wishes of potential users, and they are crucial preconditions for their adoption [31]. Thus, an analysis of user’s requirements and feedbacks can reveal untapped areas to improve the design and development, fitting the technologies to user’s needs.
In the first part of the present work, we described an updated version of ASTRAS [32], a software aiming at assessing and training EFs in children with neurodevelopmental disorders. Specifically, we focused on the user’s requirements we tried to fulfil, the executive functions taken into account, the structure of the assessment and training tasks and the main features of the software (i.e., assignment/playing modality and data recorded by the software).
In the second part, we reported the results of a survey about the opinion of thirty-two therapists about the usability, the clinical validity of the tasks, the attractiveness of the tasks for the children and therapist’s general opinion about the software.
ASTRAS is designed as a useful tool supporting the therapists during their treatments and, at the same time, a pleasant gaming environment for children both engaging and challenging. We focused our attention on therapists and children’s needs during the development of the software.
Cognitive rehabilitation/training importantly relies on patient’s cognitive profile, which is the result of an in-depth neuropsychological assessment. It determines individual’s cognitive strengths and weaknesses. In other words, therapists require assessment tools to obtain an exhaustive profile about patient’s cognitive functioning and to plan an adequate rehabilitative/training program. Accordingly, in ASTRAS we sought to design a software for the assessment and training of the main executive functions in children with neurodevelopmental disorders. Thus, ASTRAS includes two different sessions: assessment and training. ASTRAS’ user interface (UI) has been designed to allow the therapist to set a training program taking into account the results of patients in the assessment session.
The outbreak of COVID-19 has highlighted the importance of providing a way to put patients and health professionals in contact when a consultation in person was not possible. Telerehabilitation has several advantages, chiefly among them: 1) it can cover situations in which it is complicated for patients to reach traditional rehabilitation infrastructures located far away from where they live.; 2) the active involvement of the patient in the health process; 3) it improves the motivation of the patient. Importantly, the telerehabilitation allows the extension of the treatments beyond the regular time and place of therapy.
We also tried to address this requirement through two types of use/access (therapist/tutor and parents) which allow the therapist to use ASTRAS in clinical context (vis-à-vis) or from remote (see below for a better description of these two types of use).
In literature there are several tools for the assessment and training of the executive functions in children (es. CogMed; CogniFit; Training Cognitivo). However, in some cases, the stimuli and procedures are just an adaptation from adults’ tests resulting not engaging at all for children with neurodevelopmental disorders. To make ASTRAS more attractive for children we used some gamification principles, namely the application of typical elements of game playing to other areas of activity [33, 34]. Assessment and training tasks share the characteristics of game elements such as: 1) design: the assets of each task (i.e., scenario and characters) are cartoon-like. Furthermore, training’s tasks share common themes (e.g. the space);
2) mechanics: tasks have been designed as challenges in which children can also compete with others;
3) components: assessment and training tasks have been designed with different levels of difficulty, in order to increase the load of the specific executive domain. Bind together, these game features aim to improve children engagement, motivation as well as learning and reduce the negative aspects of the cognitive assessment and training: the frustration and anxiety. Furthermore, ASTRAS hosts a wide range of tasks/games to make the gaming experience more variable and avoid delivering the same task.
Finally, from a technical standpoint ASTRAS is available for the main platforms (PC and tablets) and operative systems (Android, Windows e macOS). Additionally, has also been developed an online version which runs on all the main browsers (chrome, safari, firefox). Thus, therapist and children can use ASTRAS at their places of choice and not only in a controlled environment. 2.2.
In ASTRAS we targeted the executive functions. The rationale is that deficits of executive functions are more likely to be reported in children with neurodevelopmental disorders. Importantly, deficits of executive functioning are associated with: 1) a higher severity of the disorder [35]; 2) learning impairments (e.g. in ADHD patients) [5, 36, 37]; 3) problems with academic achievement [38–40]; 4) maladaptive and/or aggressive behaviors [41, 42]. On the other hand, atypically developing children seem to profit from working on executive functions since they acquire new strategies of self-regulation, goal-oriented behaviors, problem solving [43]. In other words, working on executive functions fosters the gains of the therapy.
ASTRAS aims to assess and train the main executive domains reported in literature. Specifically, we referred to Myake et al. [19] model as theoretical framework. This model merges the unitary and multicomponent view of executive functions. Miyake and colleagues have demonstrated that EFs tasks are best represented as three distinct, but interrelated domains. These executive domains were: 1) Inhibition: reflects the ability to control own behavior, thoughts, and/or emotions to override a strong internal predisposition. Inhibition can be divided in behavioral inhibition (i.e. the suppression of a prepotent motor reaction) and cognitive inhibition (i.e. the suppression of a prepotent mental representation); 2) Working memory (WM): reflects the ability to hold verbal (verbal WM) or spatial (spatial WM) information in mind and mentally working with/on it [44, 45]. Additionally, WM includes the updating, that is the incorporation of new information into the old ones [46]; 3) Cognitive flexibility: refers to the ability to switch between thinking about two different concepts or to think about multiple concepts simultaneously.
From these core EFs, is built the planning. Planning is a higher order EF and reflects the ability to set goals, develop action plans to achieve those goals, and to choose the most appropriate actions based on the anticipation of consequences. Additionally, in this model, selective attention, namely the ability to selectively focusing on a target stimulus and suppressing attention to other stimuli or external lure, has been conceived playing a crucial role in the correct development of EFs.
Below (table 1), we provide a more accurate description of each EF taken into account, their operationalization and the most common paper and pencil tests reported in literature aiming to assess that EF.
scenario finding and selecting a
target whilst ignoring non-target stimuli
selecting, in a sequence, the target sound whilst ignoring the nontarget stimuli.
working on
auditory information consistently with a goal.
working on visuospatial information consistently with a goal.
Updating auditory and visuo-spatial information in memory and retrieve them consistently with a goal.
suppressing an unwanted movement
of behaviours and predicting the consequences in order to reach a goal
condition (Inhibition test, NEPSY
2.3.
Assessment includes ten tasks, two for each executive domain (selective attention, working memory, inhibition, cognitive flexibility and planning). Additionally, other two tasks have been developed to assess the updating of working memory. Each task consists of trials, which progressively increase in difficulty. Tasks are preceded by four steps (Figure 1): 1) Starting: a traffic light which informs children that the task is going to start; 2) Pre-test: this step taps whether children have developed those cognitive processes mandatory to run the test (e.g. in the cancellation task, the pre- test verifies whether children are able to discriminate a target picture). If this session is failed twice the task ends; 3) Simulations: in this step a video shows an example of trial; 4) Practice: in this step, children are provided a practice trial. The aim of this step is twofold: a) to allow the participant to practice with the task; b) to ensure that children have clearly understood the instructions. If the patients fail this session once, the simulation will be provided again. If the patients fail twice, the task ends.
In each task, children are provided the instructions before the beginning of task by a human-like voice (software Speechelo, https://speechelo.com/).
Below a small description of the tasks for each cognitive domain has been provided (see figure 2 for an example of tasks to assess some of the five executive functions).
Cancellation. it assesses the visual selective attention. In this task, children are shown a clutter of items, and asked to tap with their finger the target item (e.g. a star), ignoring the other stimuli.
Listening. it assesses the auditory selective attention. In this task, children are provided a sequence of animal sounds. They have to tap a paw on the center of the screen when they hear a target sound (e.g. a frog).
Sequence of quantity. it assesses the working memory. In this task, a sequence of numbers appears on the screen. Children have to repeat the sequence either forward or backward.
Visual sequence. it assesses the visuo-spatial working memory. In this task, a sequence of flashing stars is visualized on the screen. The participant has to reproduce the sequence either forward or backward.
Auditory updating. it assesses the auditory updating of working memory. In this task, children listen to the tones in sequence. After each tone is heard children decide whether the new tone is the same from 1 (1-Back) or 2 trials (2- Back) before.
Visual updating. it assesses the visuo-spatial updating of working memory. In this task, children have to decide whether the position of a ghost is the same of 1 (1-Back) or 2 trials (2-Back) before.
Go-No Go. it assesses the motor inhibition. In this task, children have to move a penguin only when he/she listen to a GO sound.
Matching. it assesses the cognitive inhibition. In this task, children have to identify, from a subsequent set of stimuli, the one that “matches” the sample. Color-shape association: it assesses the cognitive flexibility. In this task, children have to choose, from two stimuli displayed on the top of the screen, the one that has the same target criterion of the sample (shape or color). The target criterion is provided by the software.
Trail. it assesses the cognitive flexibility. In this task, children have to tap in ascending order a clutter of numbers and letters.
Maze. it assesses the planning. In this task, children must trace through a maze without crossing the maze lines.
Planning. it assesses planning. In this task, children have to reproduce a picture (e.g. a cake) in a given number of steps.
Training includes 40 tasks/games (see figure 3 for an example).
Thirty-seven out of forty have been designed to address a specific executive function. Additionally, other three tasks have been designed in order to train more than one executive function in a daily life scenario (“life skills”).
For each training task the therapist chooses: 1) the executive domain to train; 2) the specific task/game for that executive domain; 3) the number of exercises; 4) the difficulty (1 to 4); 5) the prompt (i.e. the visual or auditory support to help children in accomplishing the task). Figure 4 reports the user interface of training.
ASTRAS has two modalities to assign/play both assessment and training tasks: in-person and from remote.
In-person. This modality is designed to use the same device to assign/play the assessment and training exercises. Therapist can assign the exercises from his/her screen and switch to the patient’s screen by clicking the relative button. When the session is concluded the therapist needs to point out the aims of that session and whether they have been accomplished or not.
From remote. This modality is designed to use two devices: one to assign the exercise and the other to play. This modality consists of two types of use/access: therapists/tutors and parents.
- Therapists/Tutors access is designed to: 1) register a new child, modify his/her data or delete his/her profile; 2) open an individual or group assessment/training session; 3) assign exercises to a singular child or a group of children; 4) visualize children’s scores.
- Parents: this type of use/access is designed for children. This type of use allows to visualize and run the assessment tasks or the training exercises assigned by the therapist. As for the “In-person” modality, when the session is concluded the therapist needs to point out the aims of that session and whether they have been accomplished or not.
“From remote” modality foster the feasibility of the e-rehabilitation providing an important opportunity to: 1) reach the children when they are physically unable to reach the therapist; 2) extend the therapy beyond the clinical setting (i.e. children can perform tasks at home). 2.5.2. Data
ASTRAS records both general and specific data to address clinicians’ requirements, but also for research purposes.
General data. They are about the accuracy of the performance in each task. Specifically, for the assessment session, the score in each task range from 0 to 100 and refers to the percentage of accuracy. These data are reported on the left side of the screen and a graphical representation can be visualize in the “statistics” section (see figure 5A).
For the training tasks general data report only whether the child had performed the task correctly or not. These data are reported in the section “diary” (see figure 5B).
A B by the therapists. Additionally, the number of correct trials and errors are reported. At the right of the screen are shown the aim of each training session and whether they it has been reached or not.
Specific data. This kind of data gives an in-depth view about the performance of each patient. For example, there are data about reaction times, planning time and kind of errors (e.g. commission and omission errors). Data can be visualized on a specific section of ASTRAS website www.astras.it. Data can also be exported in excel to clinical and research purposes.
The participants were recruited from ASTRAS users’ community via email. Thirty-two therapists (8 males and 24 females, MEAN age= 34, SD= 7) accepted to be included in this study and gave their informed consent. Of thirty-two participants 16 were speech therapists, 11 psychologists and 5 psychomotor therapists. 3.2.
This study employed the web survey method to gather data. A web survey with a Likert scale questionnaire and demographic information collection was distributed to the participants via email. The questionnaire was built ad-hoc by the authors and included fifteen 5-point Likert items measuring 5 main aspects about the interaction of therapists and their patients with the software: 1) the usability (i.e. comprehensibility, simplicity of the software and the usefulness of its feature; items 1-2-3); 2) the clinical validity of the tasks (items 4-5-6); 4) the attractiveness of the tasks for the children (items 7-89-10); 4) their general opinion about the software (items 11-15); 5) the professionalism of the staff (items 12-13-14). 3.3.
A reliability test was carried out in order to test the internal consistency of our questionnaire. Then, items about the professionalism of the staff were excluded because irrelevant to the aim of the study and a set of descriptive analyses were carried out for each item. Then, data about each aspect were merged resulting in five measures about usability, clinical validity, attractiveness for children and therapists’ general opinion. These data were submitted to a 4 x 3 mixed ANOVA with Aspects (usability, clinical validity, attractiveness for children, therapists’ general opinion) as a within factor and Professions (speech therapists, psychologists and psychomotor therapists) as a between factor.
Finally, in order to compare whether the children preferred more the assessment tasks rather than the training games or vice versa, we submitted the response to the items 7 (To what extent children appreciate the assessment tasks?) and 8 (To what extent children appreciate the training tasks?) to a repeated measure ANOVA. 3.4.
Reliability test reported a high level of internal consistency (Cronbach's Alpha= .94).
Descriptive analyses were reported in table 2.
For each item the mean was relatively high (general MEAN= 4.27, SD= 0.24). The lowest score was recorded in the item about the reaction of children to the assessment tasks (MEAN= 3.88, SD= 1.18). Interestingly, therapists recommended the software to other colleagues (MEAN= 4.75, SD= 0.72). Furthermore, the frequency of each point of the Likert scale highlighted that a score of five was more likely to occur (MEAN= 53%) as compared to the others. In other words, a high percentage of therapists showed a positive disposition about the software. However, two therapists -out of thirty-two- reported low scores because the software did not fit with their patients due the severity of their disorders.
Figure 6 shows the ANOVA’s results. The analysis highlighted a significant main effect of Aspects (F(3,87)= 7.43; p < .001; h2= .20), but neither a significant main effect of Professions (F(1,29)= 1.48; p= .24; h2= .09) nor a significant interaction Aspects x Professions (F(3,87)= .28; p= .94; h2= .02). A Bonferroni corrected comparison highlighted a greater score in items about the usability (MEAN= 4.42) as compared to items about the attractiveness for children (MEAN= 4.01; p= .02). Furthermore, items about general opinion showed a higher score (MEAN= 4.50) than validity (MEAN= 4.17; p= .03) and attractiveness for children (MEAN= 4.01; p= .001).
Finally, repeated measures ANOVA highlighted a significant difference between items with children appreciating more the training games (mean= 4.22) rather than the assessment tasks (mean= 3.88; F1,29=
The aim of this study was twofold: 1) to describe ASTRAS, a software for the assessment and training of executive functions in children with neurodevelopmental disorders; 2) to investigate the feedback provided by the therapist about the usability, the clinical validity of the tasks, the attractiveness of the tasks for children and their general opinion about the software.
A “top-down” approach is widely used in designing technologies for the assessment and training of cognitive functions [29]. However, this approach often results in significant mismatches between the needs and preferences of the users and the products [59, 60]. Thus, the analysis of users’ perspectives and feedback plays a crucial role in designing and developing software that fit with user’s requirements and improve the product as well as its adoption.
In this paper we focus on the analysis of therapists and children’s requirements in order to design and develop a software more sensitive to their needs. In fact, ASTRAS has been designed and developed in order to support the therapists during their treatments and, at the same time, it provides a pleasant gaming environment for children.
Firstly, therapists require assessment tools to obtain an exhaustive profile about patient’s cognitive functioning and to plan an adequate rehabilitative/training program. Thus, in ASTRAS we developed an assessment session to provide an extensive profile of executive functioning and highlight the executive domains which need to be trained.
We tried also to address the therapists’ need to extend the therapy beyond the regular therapy and the clinical setting (telerehabilitation). The “From remote” modality allow the therapist to assign both assessment and training tasks that children can perform at home.
Cognitive tasks are also typically viewed as effortful and repetitive, which often leads to participant disengagement, low quality of data and poor benefits from the therapy. We addressed this point by means of gamification, namely the application of gaming experience more variable and avoid delivering the same task.
The adoption of user’s requirements should foster the usability of the software, its adoption and provide a positive experience in users. We sought to explore these aspects through a web survey in which fifteen questions addressing usability, the clinical validity of the tasks, the attractiveness of the tasks for children and the general opinion about the software were provided to thirty-two different therapists (speech therapists, psychologists and psychomotor therapists). Results showed that a high percentage of therapists assessed positively all the main aspects of the software. They considered the software easy to use and clinically adapt to assess and train executive functions in their patients. Results also demonstrated satisfying positive responses during the interaction of children with the software. Specifically, children demonstrated to appreciate more training games rather than assessment tasks. This result was not surprising since assessment tasks require a high number of trials and a more distracting-free setting, which in turn contribute to make the application of gaming element difficult in this kind of tasks.
Interestingly, our results did not report a difference among professions about each aspect addressed by the questionnaire. Although we should be cautious to interpret null effects, results suggested that ASTRAS well fitted with the requirements of all the main professions involved in the rehabilitation of executive functions deficits in children with neurodevelopmental disorders.
In conclusion, our software can be thought as a platform hosting several tasks/games for the assessment and training of executive functions in children with neurodevelopmental disorders. ASTRAS offers the advantage to obtain an extensive assessment of executive functions and provide training tasks in-person or from remote and control patients’ score ongoing. ASTRAS may be conceived as a valid ally supporting the activity of the therapist. Furthermore, through the gamification our software increase children’s engagement and make the experience of the therapy more enjoyable as well as avoid drop-out. 5. References