Detecting attention breakdowns in robotic neurofeedback systems Parisa Nahaltahmasebi1 Mohamed Chetouani1, David Cohen2 and Salvatore Anzalone3 1 Institut des Systèmes Intelligents et de Robotique - Université Pierre et Marie Curie, Paris, Fr 2 Service de psychiatrie de l'enfant et de l'adolescent – Hôpital Pitié-Salpêtrière, Paris, Fr 3 Laboratoire de Cognition Humaine et Artificielle - Université Paris 8, Saint-Denis, Fr sanzalone@univ-paris8.fr Abstract. This paper focuses on the EEG measures suitable for robotic neu- rofeedback systems, able to detect and intervene in case of attention breakdowns. Such systems can be useful tools in cognitive remediation of children with ASD, in particular to improve their attention span. The proposed study focuses on a particular EEG measure, the Theta/Beta ratio, as convenient feature for recogniz- ing attention states in scenarios of cognitive effort. A setup for the validation of this measure employing pupil dilation as attention baseline is introduced. Results show coherence between the Theta/Beta ratio and the pupil dilation. Keywords: Neurofeedback, attention, social robots. 1 Introduction Attention is the cognitive process aimed to the selection of a specific perceptual or internal event (such as a thought), while discarding the others. “It is the taking posses- sion by the mind, in a clear and vivid form of one out of what seem several simultane- ously possible objects or trains of thought. Focalization, concentration of conscious- ness are of its essence. It implies withdrawal from some things in order to deal with others…”[1]. This cognitive process is a quite fluctuating skill that needs to be continu- ously maintained through a certain level of awareness, and it is often unconsciously let- up during a task. Attention is a basic skill of human cognition: it is the process respon- sible for the allocation of the limited processing resources of human cognition to spe- cific tasks[2]. Neurocognitive diseases can affect in different ways the attention skill. The case of children affected by Autism Spectrum Disorder[3] (ASD) is particularly interesting: they are able to maintain a firm and durable attention towards activities they like and moti- vates them. However, at the same time, in can be really hard for them to keep their attention on other, less attractive, activities. Children with ASD can develop their skills through joint activities with an expert clinician that can guide them on improving their attention span[4,5]. However, it is always difficult also for the most expert clinicians to infer mental states from the behavior of the children. The EEG technology, in this case, 2 can help them by providing in real-time neuro-physiological measures resuming par- ticular mental states, as the attention of the child towards the current joint activity[6]. A companion robot can be a helpful tool in these scenarios, with the role of conveying such neuro-physiological measures to clinicians while acting like a playfellow with the child[7]. At the same time, robots endowed with some kind of intelligent, social skills, can autonomously act according to particular mental states, providing feedbacks or re- wards to the child[8,9,10,11]. In a recent work[12] (Fig. 1), we introduced a neurofeedback system that employs small humanoid robot to reinforce attention in autism spectrum disorder. This work aroused several questions, in particular on the nature of the EEG control measure. The goal of this paper is to verify if a particular EEG measure, the Theta/Beta ratio, can be employed as reference to track attention breakdowns in humans, and consequently if it can be employed as control measure of a robotic neurofeedback system. Fig. 1. The prototype of neurofeedback system employing a robot to reinforce attention. Several works[13,14] reported of measurable changes on the eyes as effect of cognitive efforts. In particular, pupil’s dilatation seems an effective measure of cognitive load[15]. In scenarios in which cognitive effort is stimulated, this measure can be adopted as convenient baseline to evaluate EEG measures and identify attention breakdowns. 2 Materials and methods In this work, we propose an experiment aimed at the evaluation of the Theta/Beta EEG waves ratio as measure of attention breakdowns in a scenario of cognitive effort, through its comparison with pupil dilatation. Theta/Beta ratios and the pupil dilatation are captured through synchronized EEG and Eye Tracker systems, while the participant to the experiment alternates relax phases to mental calculations, solving a simple arith- metic problem. This interleaving between different cognitive loads will simulate the attention breakdowns. Then, the measures obtained are off-line analyzed and compared. The EEG employed is an Enobio from NeuroElectrics, while the Eye Tracker is a Tobii X1 Light. 3 2.1 Setup As in Figure 2, the participant sits in front of a screen equipped with an Eye Tracker. The screen is employed to present the arithmetical problems that the participant should solve. Neurological measures are collected through an EEG. The two sensors, the EEG and the Eye Tracker, are both connected to the same PC. A hidden operator controls the correct evolution of the experiment. A standard camera conveniently placed on the side of the monitor, records the behaviors of the participant. The room of the experiment is a controlled environment with only controlled lights that were kept at the same in- tensity in all the performed experiments. Fig. 2. The setup of the presented system. Attention during cognitive efforts can be modeled through Theta/Beta ratio. This ratio is calculated employing central, frontal and temporal EEG sensors (c3, c5, f3, f4, t7,t8)[16]. The average Theta/Beta ratio between the channels has been selected as atten- tion feature. The attention model of the participant is learned at the beginning of the experimental protocol, during a training stage, by exploiting the data obtained during a cognitive effort. In this training stage it is supposed that the participant will be very focused towards the activity he is doing. To model the attention, an inhibit threshold for the Theta/Beta ratio activity defining the attentive state is set at a power level that the training activity fell over its range for the 50% of time [17]. This model will return a measure between -1 and 1, as attentive/not attentive to the current cognitive task. At the same time, the Eye Tracker will follow the eyes’ behaviors of the participant in terms of pupil’s dimension. Both system will calculate and store the data in real-time. In a first setup stage, after a brief introduction of the system, a relaxing music is played and the participant is invited to seat, feeling comfortable and at ease. In this stage, during around 60 seconds, electrooculography is accomplished and a model of eye movements is created. Then, the operator starts the training session of the system, presenting to the participant an arithmetical problem for 60 seconds. In this stage, the system will model the attention of the participant by exploiting the data obtained during the cognitive effort stimulated by the resolution of the arithmetical problem. After this step, the experiment will go through five stages, alternating a new arith- metical problems (max 2 minutes) with a relax period (1 minute). The hypothesis of 4 this work is that the proposed measure should be able to reveal and identify this inter- change between stages. 2.2 Participants The participants involved in the experiments is composed by twelve healthy adults, eight males and four females. They were recruited among the member and the students of ISIR-UPMC and signed informed consent before the participation to the experi- ments. During a first analysis of the captured data, the records of three participants were discarded due to technical problems: mainly, a wrong EEG calibration and a lack of eye tracking data. 3 Results Figure 3 show the evolution of the median of the Theta/Beta attention measure and of the pupil dilatation. Detailed Results are reported in the Table 1. The interleaving be- tween cognitive efforts and relax stage simulates the attention breakdown. Index Pupil Median p-values EEG Median p-values 0. Cognitive effort 3.16 (±0.40) 0.09 (±0.43) 0.008** 0.039* 1. Relax 3.00 (±0.35) -0.04 (±0.38) 0.02* 0.39 2. Cognitive effort 3.12 (±0.33) 0.08 (±0.48) 0.09* 0.63 3. Relax 3.02 (±0.36) 0.00 (±0.47) 0.67 0.09 4. Cognitive effort 3.04 (±0.33) -0.19 (±0.41) Table 1. Average and standard deviation of pupil dilation median and of EEG Theta/Beta based attention measure. P-Values are introduced for each couple “Effort-Relax”. The results related to the pupil confirm the previous researches: pupil dilation fol- lows the cognitive load stimulated by the proposed activity, becoming larger in case of cognitive effort, turning smaller while relaxing. This variation is statistically significant among to the different stage of the experiment, except the last couple. Similar coher- ence emerges by the EEG data that shows a statistical significance between the first proposed arithmetical problem and the subsequent relaxing stage. To explain the non- significant results it is possible to hypothesize the emergence of fatigue on the partici- pants while solving the arithmetical problems. 5 Fig. 3. Among the different stages of the experiment: on the left, the evolution of the pupil di- mension median; on the right, the evolution of the EEG Theta/Beta based attention measure. 4 Conclusions, limits and future works The results illustrate a certain degree of coherence between the data captured by the two sensors, showing relevant differences between stages of cognitive effort and relax- ation. However, such results are far from being general due to the dimension of the population participating to this experiment. In any case, such result encourage us on the implementation of neurofeedback systems that use robots as useful and natural tool to induce behaviors in humans. This setup will also give us the possibility of verifying the effectiveness on detecting attentions breakdowns of different kind of EEG measures as well as the efficacy of a large gamma of stimuli and in the case of populations with special needs as children with ASD. Acknowledgments This project was partially funded by the French Laboratoire d’Excellence SMART (ANR-11-IDEX-0004-02). Authors want to thanks R. Grassia MD, A. Tanet MD and O. Pallanca MD for their continuous support during the development of this project. References 1. William James. The Principles of Psychology. Holt, New York, 1890. 2. John Robert Anderson. "Cognitive psychology and its implications." A series of books in psychology. (1985). 3. American Psychiatric Association. Diagnostic and statistical manual of mental disorders (DSM-5®). American Psychiatric Pub, 2013. 4. Geraldine Dawson, Sally Rogers, Jeffrey Munson, Milani Smith, Jamie Winter, Jessica Greenson, Amy Donaldson, and Jennifer Varley. Randomized, controlled trial of an inter- vention for toddlers with autism: the early start denver model. Pediatrics, 125(1):e17-e23, 2010. 6 5. Jean Xavier, Claude Bursztejn, Maitri Stiskin, Roberto Canitano, and David Cohen. Autism spectrum disorders: An historical synthesis and a multidimensional assessment toward a tai- lored therapeutic program. Research in Autism Spectrum Disorders, 18:21-33, 2015. 6. Robert Coben, Adam R Clarke, William Hudspeth, and Robert J Barry. Eeg power and co- herence in autistic spectrum disorder. Clinical Neurophysiology, 119(5):1002-1009, 2008. 7. Salvatore M Anzalone, Sofiane Boucenna, Serena Ivaldi, and Mohamed Chetouani. Evalu- ating the engagement with social robots. International Journal of Social Robotics, 7(4):465- 478, 2015. 8. Salvatore M. Anzalone, Elodie Tilmont, Sofiane Boucenna, Jean Xavier, Anne-Lise Jouen, Nicolas Bodeau, Koushik Maharatna, Mohamed Chetouani, David Cohen, MICHELANGELO Study Group, et al. How children with autism spectrum disorder behave and explore the 4-dimensional (spatial 3d+ time) environment during a joint attention induc- tion task with a robot. Research in Autism Spectrum Disorders, 8(7):814-826, 2014. 9. Brian Scassellati, Henny Admoni, and Maja Mataric. Robots for use in autism research. Annual review of biomedical engineering, 14:275-294, 2012. 10. Joshua J Diehl, Lauren M Schmitt, Michael Villano, and Charles R Crowell. The clinical use of robots for individuals with autism spectrum disorders: A critical review. Research in autism spectrum disorders, 6(1):249-262, 2012. 11. Sofiane Boucenna, Antonio Narzisi, Elodie Tilmont, Filippo Muratori, Giovanni Pioggia, David Cohen, and Mohamed Chetouani. Interactive technologies for autistic children: a re- view. Cognitive Computation, 6(4):722-740, 2014. 12. Salvatore M. Anzalone, Antoine Tanet, Olivier Pallanca, David Cohen, and Mohamed Chetouani. A humanoid robot controlled by neurofeedback to reinforce attention in autism spectrum disorder. The 3rd Italian Workshop on Artificial Intelligence and Robotics at the XV International Conference of the Italian Association for Artificial Intelligence (AI*IA 2016), Genova, Italy, November 28, 2016. 13. Daniel Kahneman, Bernard Tursky, David Shapiro, and Andrew Crider. "Pupillary, heart rate, and skin resistance changes during a mental task." Journal of experimental psychology 79, no. 1p1 (1969): 164. 14. Eckhard H. Hess. Attitude and pupil size. Scientific american 212, no. 4 (1965): 46-55. 15. Tepring Piquado, Derek Isaacowitz, and Arthur Wingfield. "Pupillometry as a measure of cognitive effort in younger and older adults." Psychophysiology 47, no. 3 (2010): 560-569. 16. Martijn Arns, C. Keith Conners, and Helena C. Kraemer. "A decade of EEG theta/beta ratio research in ADHD: a meta-analysis." Journal of attention disorders 17, no. 5 (2013): 374- 383. 17. Steven M. Butnik, "Neurofeedback in adolescents and adults with attention deficit hyperac- tivity disorder." Journal of Clinical Psychology 61, no. 5 (2005): 621-625.