th Companion Proceedings 10 International Conference on Learning Analytics & Knowledge (LAK20) Endowing Head-Mounted Displays with Physiological Sensing for Augmenting Human Learning and Cognition Evangelos Niforatos Norwegian University of Science and Technology (NTNU), Trondheim, Norway evangelos.niforatos@ntnu.no Athanasios Vourvopoulos Instituto Superior Técnico, University of Lisbon athanasios.vourvopoulos@tecnico.ulisboa.pt Michail Giannakos Norwegian University of Science and Technology (NTNU), Trondheim, Norway michailg@ntnu.no ABSTRACT: The EEGlass prototype is a merger between a Head-Mounted Display (HMD) and a brain-sensing platform with a set of electroencephalography (EEG) electrodes at the contact points with the skull. EEGlass measures unobtrusively the activity of the human brain facilitating the interaction with HMDs for augmenting human cognition. Among others, EEGlass is intended for collection of context-aware EEG measurements, supporting learning and cognitive experiments outside the laboratory environment. Thus, we expect EEGlass will promote the implementation and application of ecologically valid research methods (studies in the user's natural context). Keywords: Head-Mounted Displays, Electroencephalography, Brain-Computer Interfaces, Neuroadaptive Systems. 1 INTRODUCTION Human cognition is typically a composite notion we use for describing the states of the cognitive processes that underpin it. Namely, attention; memory recall; learning; decision-making; and problem-solving. Augmenting human cognition boils down to gauging the states of the underlying cognitive processes and deciding on an intervention. On one hand, electroencephalography (EEG), and other measures of physiological responses, have been extensively utilized for measuring attention, monitoring cognitive workload, assessing learning experience, and even evaluating software usability. On the other hand, contemporary HMDs, such as Augmented Reality (AR) smart glasses, are progressively becoming socially acceptable and ubiquitous by approaching the size and design of normal eyewear (Niforatos & Vidal, 2019). Thus, a merger between HMDs and EEG appears to be promising. HMDs bear a significant potential in hosting an array of physiological sensors in contact with the human skull, while situated in front of our most highly-esteemed perceptive organ: our eyes. In this work, we draw on the HMD form factor for designing, developing, and evaluating EEGlass, an EEG-Eyeware prototype for ubiquitous brain-computer interaction (Vourvopoulos et al., 2019). Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 th Companion Proceedings 10 International Conference on Learning Analytics & Knowledge (LAK20) 2 EEGLASS PROTOTYPE The latest version of the EEGlass prototype (see Figure 1) is comprised of a Vuzix Blades1 (Vuzix, Rochester, USA) HMD fitted with EEG electrodes that connect to a Cyton Biosensing Board by OpenBCI2 (OpenBCI, NY, USA). Vuzix Blades is a pair of AR smart glasses that features a monocular and transparent waveguide display, with a 19-degrees field of view, and a resolution of 480 x 853 pixels. Vuzix Blades is equipped with an 8MP camera, Bluetooth and Wi-Fi connectivity modules, a range of sensors (e.g., inertial measurement unit, microphones, etc.), and runs Android OS. OpenBCI is a popular and relatevely low-cost open hardware and software platform for the collection and analysis of biosignals such as EEG, EMG (Electromyography), and ECG (Electrocardiography), inspired by the grassroots movement of DIY (“Do It Yourself”). The Cyton board encompasses 8 biopotential input channels (for hosting up to 8 electrodes), a 3-axes accelerometer, local storage, Bluetooth connectivity module, while being fully programmable and Arduino compatible. Evidently, the EEGlass electrode topology is restricted by the form factor of Vuzix Blades and at the contact points with the skull. Thus, EEGlass utilizes 3 electrodes (plus 2 for reference and ground) based on the standard 10-20 EEG system (see Figure 1) for measuring brain activity: 1 electrode placed inwards at the top of the eyewear bridge touching the skull at glabella, and 2 more electrodes at the inner side of the eyewear temples, touching the left and right mastoids, behind the left and right ears, respectively. Both the Cyton Board and Vuzix Blades are connected to an external power source for enabling and prolonging mobile usage. 3 CURRENT STATE AND NEXT STEPS Our first aim is to investigate how reliably EEGlass can capture brain activity, particularly when featuring an electrode topology imposed by the form factor of an HMD. For this, we compare brain activity captured via EEGlass with that captured via a standard EEG system as baseline. So far, we have tested a previous version of the EEGlass prototype, implemented with eyewear frames. Limited trials with 1 participant indicated that the EEGlass is capable of capturing brain activity manifested in two modes of resting state: (a) eyes open and focused on a target, and (b) eyes closed. Brain activity recorded during resting state with EEGlass demonstrated similar variations in frequency and amplitude to when recorded with an established EEG system. Recorded brain activity linked to upper limb motor-action displayed significant differences when compared to that captured with an established EEG system due to the fundamentally different electrode topology of EEGlass. Nevertheless, EEGlass managed to capture upper limb motor-action relying on signal propagation over the skull through volume conduction (van den Broek et al., 1998). EEGlass also detected subtle eye movements in 4 basic directions, displaying an eye-tracking potential particularly useful for navigating in HMD interfaces. Low sample size (N=1) and stationary experimental settings are significant limitations that we will address over the next studies. However, human skull and brain anatomy is almost homogeneous, 1 https://www.vuzix.com/products/blade-smart-glasses 2 https://openbci.com/ Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 th Companion Proceedings 10 International Conference on Learning Analytics & Knowledge (LAK20) and the HMD form factor ensures a rather stable electrode contact with the skull, only somewhat influenced by movement. In future iterations, we will utilize machine learning for training algorithms to match input from EEGlass to that of established EEG systems. We believe a merger between EEG and HMDs bears an unprecedented potential to “close the loop” by increasing the communication bandwidth between human and machine and paving the way for cognition-aware systems (Niforatos et al., 2017). (a) (b) TP9 TP10 Cyton Board by Power source OpenBCI Nz Ref. GND Figure 1: (a) The EEGlass prototype comprised of a Vuzix Blades HMD fitted with 5 EEG electrodes, based on the 10-20 system, connecting to a Cyton OpenBCI board and a mobile power supply. (b) A user wearing EEGlass. 4 APPLICATIONS FOR LEARNING Besides the promising application areas in augmenting human cognition in general, we believe EEGlass also bears significant potential in facilitating learning. For example, after investigating EEGlass in reliably measuring cognitive activity in the wild, we will introduce it to the classroom. Although EEG can capture the subtle cognitive processes associated with learning (e.g., attention and concentration levels), performing EEG experiments in a classroom with the typical EEG headsets is deemed cumbersome and often inappropriate. Thus, we expect that EEGlass can be a viable alternative in collecting unobtrusively the brain activity of students related to learning. Moreover, the HMD component of EEGlass can be utilized for projecting information about the learning content in pre, post or during learning stage, and even on the go. We expect that by presenting our prototype to the CrossMMLA workshop, we will spark ideation and generate discussions about different applications and user scenarios for EEGlass about enhancing learning and the entire spectrum of human cognition. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 3 th Companion Proceedings 10 International Conference on Learning Analytics & Knowledge (LAK20) ACKNOWLEDGEMENTS This project is a product of a research collaboration between Norwegian University of Science and Technology (NTNU) and Institute for Systems and Robotics (ISR-Lisbon), receiving funding from Swiss National Science Foundation (SNSF) and the ERCIM “Alain Bensoussan” Fellowship. REFERENCES van den Broek, S. P., Reinders, F., Donderwinkel, M., & Peters, M. J. (1998). Volume conduction effects in EEG and MEG. Electroencephalography and clinical neurophysiology, 106(6), 522- 534. Niforatos, E., Vourvopoulos, A., & Langheinrich, M. (2017, September). Amplifying human cognition: bridging the cognitive gap between human and machine. In Proceedings of the 2017 acm international joint conference on pervasive and ubiquitous computing and proceedings of the 2017 acm international symposium on wearable computers (pp. 673-680). ACM Niforatos, E., & Vidal, M. (2019, March). Effects of a Monocular Laser-Based Head-Mounted Display on Human Night Vision. Retrieved from https://dl.acm.org/citation.cfm?id=3311858 Vourvopoulos, A., Niforatos, E., & Giannakos, M. (2019, September). EEGlass: An EEG-eyeware prototype for ubiquitous brain-computer interaction. In Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers (pp. 647-652). ACM Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 4