Detecting attention breakdowns in robotic neurofeedback systems

Abstract

This paper focuses on the EEG measures suitable for robotic neurofeedback systems, able to detect and intervene in case of attention breakdowns. Such systems can be useful tools in cognitive remediation of children with ASD, in particular to improve their attention span. The proposed study focuses on a particular EEG measure, the Theta/Beta ratio, as convenient feature for recognizing attention states in scenarios of cognitive effort. A setup for the validation of this measure employing pupil dilation as attention baseline is introduced. Results show coherence between the Theta/Beta ratio and the pupil dilation.

Keywords

• Neurofeedback
• attention
• social robots

-

Attention is the cognitive process aimed to the selection of a specific perceptual or internal event (such as a thought), while discarding the others. “It is the taking possession by the mind, in a clear and vivid form of one out of what seem several simultaneously possible objects or trains of thought. Focalization, concentration of consciousness are of its essence. It implies withdrawal from some things in order to deal with others…”[ 1 ]. This cognitive process is a quite fluctuating skill that needs to be continuously maintained through a certain level of awareness, and it is often unconsciously letup during a task. Attention is a basic skill of human cognition: it is the process responsible for the allocation of the limited processing resources of human cognition to specific tasks[ 2 ].

Neurocognitive diseases can affect in different ways the attention skill. The case of children affected by Autism Spectrum Disorder[ 3 ] (ASD) is particularly interesting: they are able to maintain a firm and durable attention towards activities they like and motivates them. However, at the same time, in can be really hard for them to keep their attention on other, less attractive, activities. Children with ASD can develop their skills through joint activities with an expert clinician that can guide them on improving their attention span[ 4,5 ]. However, it is always difficult also for the most expert clinicians to infer mental states from the behavior of the children. The EEG technology, in this case, can help them by providing in real-time neuro-physiological measures resuming particular mental states, as the attention of the child towards the current joint activity[ 6 ]. A companion robot can be a helpful tool in these scenarios, with the role of conveying such neuro-physiological measures to clinicians while acting like a playfellow with the child[ 7 ]. At the same time, robots endowed with some kind of intelligent, social skills, can autonomously act according to particular mental states, providing feedbacks or rewards to the child[ 8,9,10,11 ].

In a recent work[ 12 ] (Fig. 1), we introduced a neurofeedback system that employs small humanoid robot to reinforce attention in autism spectrum disorder. This work aroused several questions, in particular on the nature of the EEG control measure. The goal of this paper is to verify if a particular EEG measure, the Theta/Beta ratio, can be employed as reference to track attention breakdowns in humans, and consequently if it can be employed as control measure of a robotic neurofeedback system.

Several works[ 13,14 ] reported of measurable changes on the eyes as effect of cognitive efforts. In particular, pupil’s dilatation seems an effective measure of cognitive load[ 15 ]. In scenarios in which cognitive effort is stimulated, this measure can be adopted as convenient baseline to evaluate EEG measures and identify attention breakdowns. 2

Materials and methods

In this work, we propose an experiment aimed at the evaluation of the Theta/Beta EEG waves ratio as measure of attention breakdowns in a scenario of cognitive effort, through its comparison with pupil dilatation. Theta/Beta ratios and the pupil dilatation are captured through synchronized EEG and Eye Tracker systems, while the participant to the experiment alternates relax phases to mental calculations, solving a simple arithmetic problem. This interleaving between different cognitive loads will simulate the attention breakdowns. Then, the measures obtained are off-line analyzed and compared.

The EEG employed is an Enobio from NeuroElectrics, while the Eye Tracker is a Tobii X1 Light.

Setup

As in Figure 2, the participant sits in front of a screen equipped with an Eye Tracker. The screen is employed to present the arithmetical problems that the participant should solve. Neurological measures are collected through an EEG. The two sensors, the EEG and the Eye Tracker, are both connected to the same PC. A hidden operator controls the correct evolution of the experiment. A standard camera conveniently placed on the side of the monitor, records the behaviors of the participant. The room of the experiment is a controlled environment with only controlled lights that were kept at the same intensity in all the performed experiments.

Attention during cognitive efforts can be modeled through Theta/Beta ratio. This ratio is calculated employing central, frontal and temporal EEG sensors (c3, c5, f3, f4, t7,t8)[ 16 ]. The average Theta/Beta ratio between the channels has been selected as attention feature. The attention model of the participant is learned at the beginning of the experimental protocol, during a training stage, by exploiting the data obtained during a cognitive effort. In this training stage it is supposed that the participant will be very focused towards the activity he is doing. To model the attention, an inhibit threshold for the Theta/Beta ratio activity defining the attentive state is set at a power level that the training activity fell over its range for the 50% of time[ 17 ]. This model will return a measure between -1 and 1, as attentive/not attentive to the current cognitive task. At the same time, the Eye Tracker will follow the eyes’ behaviors of the participant in terms of pupil’s dimension. Both system will calculate and store the data in real-time.

In a first setup stage, after a brief introduction of the system, a relaxing music is played and the participant is invited to seat, feeling comfortable and at ease. In this stage, during around 60 seconds, electrooculography is accomplished and a model of eye movements is created. Then, the operator starts the training session of the system, presenting to the participant an arithmetical problem for 60 seconds. In this stage, the system will model the attention of the participant by exploiting the data obtained during the cognitive effort stimulated by the resolution of the arithmetical problem.

After this step, the experiment will go through five stages, alternating a new arithmetical problems (max 2 minutes) with a relax period (1 minute). The hypothesis of this work is that the proposed measure should be able to reveal and identify this interchange between stages. 2.2

Participants

The participants involved in the experiments is composed by twelve healthy adults, eight males and four females. They were recruited among the member and the students of ISIR-UPMC and signed informed consent before the participation to the experiments.

During a first analysis of the captured data, the records of three participants were discarded due to technical problems: mainly, a wrong EEG calibration and a lack of eye tracking data. 3

Results

The results related to the pupil confirm the previous researches: pupil dilation follows the cognitive load stimulated by the proposed activity, becoming larger in case of cognitive effort, turning smaller while relaxing. This variation is statistically significant among to the different stage of the experiment, except the last couple. Similar coherence emerges by the EEG data that shows a statistical significance between the first proposed arithmetical problem and the subsequent relaxing stage. To explain the nonsignificant results it is possible to hypothesize the emergence of fatigue on the participants while solving the arithmetical problems.

Conclusions, limits and future works

The results illustrate a certain degree of coherence between the data captured by the two sensors, showing relevant differences between stages of cognitive effort and relaxation. However, such results are far from being general due to the dimension of the population participating to this experiment. In any case, such result encourage us on the implementation of neurofeedback systems that use robots as useful and natural tool to induce behaviors in humans. This setup will also give us the possibility of verifying the effectiveness on detecting attentions breakdowns of different kind of EEG measures as well as the efficacy of a large gamma of stimuli and in the case of populations with special needs as children with ASD.

Acknowledgments

This project was partially funded by the French Laboratoire d’Excellence SMART (ANR-11-IDEX-0004-02). Authors want to thanks R. Grassia MD, A. Tanet MD and O. Pallanca MD for their continuous support during the development of this project.

References

1. William James . The Principles of Psychology. Holt , New York, 1890 .

2. John Robert Anderson. "Cognitive psychology and its implications." A series of books in psychology . ( 1985 ).

3. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®) . American Psychiatric Pub, 2013 .

4. Geraldine Dawson , Sally Rogers, Jeffrey Munson, Milani Smith , Jamie Winter , Jessica Greenson, Amy Donaldson, and Jennifer Varley . Randomized, controlled trial of an intervention for toddlers with autism: the early start denver model . Pediatrics , 125 ( 1 ): e17 - e23 , 2010 .

5. Jean Xavier , Claude Bursztejn, Maitri Stiskin, Roberto Canitano, and David Cohen. Autism spectrum disorders: An historical synthesis and a multidimensional assessment toward a tailored therapeutic program . Research in Autism Spectrum Disorders , 18 : 21 - 33 , 2015 .

6. Robert Coben , Adam R Clarke, William Hudspeth , and Robert J Barry. Eeg power and coherence in autistic spectrum disorder . Clinical Neurophysiology , 119 ( 5 ): 1002 - 1009 , 2008 .

7. Salvatore M Anzalone , Sofiane Boucenna , Serena Ivaldi, and Mohamed Chetouani . Evaluating the engagement with social robots . International Journal of Social Robotics , 7 ( 4 ): 465 - 478 , 2015 .

8. Salvatore M. Anzalone , Elodie Tilmont, Sofiane Boucenna, Jean Xavier, Anne-Lise Jouen , Nicolas Bodeau, Koushik Maharatna, Mohamed Chetouani, David Cohen, MICHELANGELO Study Group, et al. How children with autism spectrum disorder behave and explore the 4-dimensional (spatial 3d+ time) environment during a joint attention induction task with a robot . Research in Autism Spectrum Disorders , 8 ( 7 ): 814 - 826 , 2014 .

9. Brian Scassellati , Henny Admoni, and Maja Mataric . Robots for use in autism research . Annual review of biomedical engineering , 14 : 275 - 294 , 2012 .

10. Joshua J Diehl, Lauren M Schmitt, Michael Villano , and Charles R Crowell. The clinical use of robots for individuals with autism spectrum disorders: A critical review . Research in autism spectrum disorders , 6 ( 1 ): 249 - 262 , 2012 .

11. Sofiane Boucenna , Antonio Narzisi, Elodie Tilmont, Filippo Muratori, Giovanni Pioggia, David Cohen, and Mohamed Chetouani . Interactive technologies for autistic children: a review . Cognitive Computation , 6 ( 4 ): 722 - 740 , 2014 .

12. Salvatore M. Anzalone , Antoine Tanet, Olivier Pallanca, David Cohen, and Mohamed Chetouani . A humanoid robot controlled by neurofeedback to reinforce attention in autism spectrum disorder . The 3rd Italian Workshop on Artificial Intelligence and Robotics at the XV International Conference of the Italian Association for Artificial Intelligence (AI*IA 2016 ), Genova, Italy, November 28 , 2016 .

13. Daniel Kahneman, Bernard Tursky, David Shapiro, and Andrew Crider . "Pupillary, heart rate, and skin resistance changes during a mental task." Journal of experimental psychology 79, no . 1p1 ( 1969 ): 164 .

14. Eckhard H. Hess . Attitude and pupil size . Scientific american 212 , no. 4 ( 1965 ): 46 - 55 .

15. Tepring Piquado , Derek Isaacowitz, and Arthur Wingfield . "Pupillometry as a measure of cognitive effort in younger and older adults . " Psychophysiology 47 , no. 3 ( 2010 ): 560 - 569 .

16. Martijn Arns , C. Keith Conners , and Helena C. Kraemer . "A decade of EEG theta/beta ratio research in ADHD: a meta-analysis . " Journal of attention disorders 17 , no. 5 ( 2013 ): 374 - 383 .

17. Steven M. Butnik , "Neurofeedback in adolescents and adults with attention deficit hyperactivity disorder . " Journal of Clinical Psychology 61 , no. 5 ( 2005 ): 621 - 625 .