Research proposal: integration of EEG and oculometry methods for external cognitive load assessment in graphical user interfaces Paula Andrea Cesarino Vargas1, Luis Eduardo Bautista Rojas2, María Fernanda Maradei García3 1 Industrial University of Santander paulacesarino@hotmail.com 2 Industrial University of Santander luis.bautista@correo.uis.edu.co 3 Industrial University of Santander mafemar@correo.uis.edu.co Summary.Within the framework of the master's degree project in innovation and design, a research proposal is proposed that presents the integration of objective techniques to evaluate the cognitive load generated by the graphical user interface in an augmented reality environment, based on an experimental design unifactorial, cross-sectional, retrospective, at an explanatory level. This proposal is relevant as it will provide empirical evidence for the graphical user interface design process. It is expected, at the end of this project, to have a procedure manual for the evaluation of graphical user interfaces based on the level of external cognitive load. Keywords: GUI, Graphical user interface, External cognitive load, EEG, eye tracking, Augmente reality. 1. Introduction With the fourth industrial revolution, technologies such as augmented reality (AR) have been making headway in areas such as learning and skills training, given their ability to merge real and virtual objects in practice environments and the ability to provide information in time. real to apprentice (Akçayır & Akçayır, 2017). Several studies have shown that cognitive overload can be an important aspect of usability(Adams, 2007) and augmented environments are likely to be particularly sensitive to its effects. When a person is making use of an augmented environment, a series of cognitive processes are triggered directly linked to that activity. The level of cognitive load is an indicator that could be used to evaluate increased user-environment interaction. In the nineties, Sweller(Sweller, 1994)proposed the cognitive load theory (CLT), an instructional theory based on the knowledge of human cognitive architecture and focused directly on the limitations of working memory and the automation of long- term memory schemas. However, despite the fact that CLT is widely known from its scientific theory, its measurement for instructional materials (especially in multimedia teaching) is mainly based on indirect, subjective or both methods.(Brunken et al., 2010). This proposal focuses on the process of designing graphical user interfaces for augmented reality environments, specifically in the evaluation phase of design alternatives. Focused on defining a framework that allows the designer to objectively evaluate his interface from the level of extrinsic cognitive load generated by the visual stimulus. The foregoing, through a cross-technique approach of electroencephalography (EEG) and eye tracking, providing the designer with pertinent information for the redefinition of the interfaces. 1.1 Formulation of the problem The Cognitive Load (CC) can be defined as the total amount of mental activity processed, consciously, at the moment in which the subject completes a task (Paas et al., 2003). However, not all CC is the same type; There are three classes: a) intrinsic, b) extrinsic, and c) Germanic. Intrinsic QC is related to the complexity of the task itself, depending on the conceptual difficulty of the material and the skill of the learner to develop it; the extrinsic CC is responsible for contaminating Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1 or saturating the information that is presented, thus affecting working memory; generally it relates to multimedia materials or interfaces. Finally, Germanic CC is directly related to learning and the processes of abstraction and information processing towards long-term memory.(Sweller, 1994). The measurement of external cognitive load continues to be a challenge, since this construct cannot be measured directly, but rather requires the evaluation of dimensions related to it, which are: mental load, mental effort and performance(Andrade-Lotero, 2012). Many augmented reality applications involve tasks that require the use of a large amount of cognitive resources and since users have limited them, if an evaluation is not carried out to verify that the graphical interface design does not impose an external cognitive load on the user high, technology could add more complexity and influence the learning process. However, conducting this assessment during augmented reality experiences is especially complex, since learners are exposed to a large amount of interaction.(Akçayir et al., 2016). An essential question in this study is whether there are valid, reliable and practical methods that can measure external cognitive load in augmented reality environments. In the literature, two different approaches are generally found for such evaluation: a) requesting users to subjectively rate their perceived cognitive load or b) using objective measures, commonly physiological measures. In this proposal, an objective magnitude that is robust, continuously measurable and sensitive enough to solve the bad signal-to-noise ratio is sought. 1.2 Justification As human-computer interaction (HCI) systems are becoming ubiquitous and used to perform critical tasks in different domains, the need to measure the cognitive load caused by a purpose-designed HCI system is becoming increasingly common. increasingly important in stages prior to implementation (Neville et al., 2005). Although HCI evaluation methods have also advanced with the evolution of systems and techniques, such as verbal reports, observations of task fulfillment, concurrent verbalization, questionnaires, etc., they are being used to evaluate interactive systems for their effectiveness and efficiency, however, there is still a shortage of methods that can directly measure the cognitive load caused by the system. Current tools have allowed us to measure cognitive load from a weighted analysis; An interesting observation is that although researchers are continually trying to find or develop secondary task and physiological measures, the internal consistency of these measures requires further study. Especially to determine the extrinsic cognitive load, referring to when the learner is interacting with a material or interface whose design or execution has irrelevant elements that hinder the processes of both construction and automation of schemes, which will allow the learning of complex concepts. 2 Methodological framework To carry out the development of the project, a methodology based on the methodological approach of(Blessing & Chakrabarti, 2009)one of the popular models for design research; This method belongs to the field of research in design engineering and consists of 4 stages: definition of criteria, descriptive study I, prescriptive study and descriptive study II. Each of the objectives of the Design Throught Research (DTR) phase Table 1. Methodology for the fulfillment of the objectives related to the DTR methodology Target Research level Activities Method Subject Identify the variables that Typical descriptive Theoretical project Context analysis Cognitive load in determine the level of framework augmented reality cognitive activity Experimental EEG through EEG and Evaluation metrics analysis Oculometry oculometry analysis Assess cognitive load Descriptive Determine the level of Experimental EEG variables with respect to correlational reliability of the objective Oculometry brain waves and eye cognitive load External cognitive movement measurement with respect load scale to the subjective one Evaluate the integration Explanatory To determine the level of Experimental EEG of the two techniques as Descriptive and reliability of the cognitive Oculometry a tool for the objective inferential statistical load measurement with a Cross technique measurement of external analysis from the multimodal approach cognitive load. comparison of means. compared to the unimodal one 2 It is expected, at the end of the development of this research work, to have a procedure manual that describes the activities related to the evaluation of external cognitive load of graphical user interfaces applying crossed techniques; This manual will have a chronological narrative that systematically requires data capture, a guide for the treatment of electroencephalographic signals and a list of metrics that differentiate the levels of external cognitive load resulting from its interface, in this way, the designer will be able to evaluate objectively its design alternatives for augmented reality environments and obtain the necessary information for its redefinition, if required. The above, based on the case study for the evaluation of external cognitive load of a prototype for learning and training activities in an augmented environment, which allows to demonstrate the significant improvement of the evaluation of cognitive load of each of the techniques separately. compared to the application of integrated techniques. 2.1 Research design This work is framed as an investigation through design, experimental, retrospective and cross- sectional at an explanatory level. The hypothesis proposed by the researcher affirms that multimodal approaches that combine data from different sensors improve the specific evaluation of external cognitive load caused by the stimulus in graphical user interfaces. To confirm the previous hypothesis, the development of an experiment with a unifactorial design is proposed, in order to verify the values for the estimation of cognitive load from the frequency, voltage and amplitude of the waves. This information will be acquired with the EMOTIV EPOC + device, which allows us to recognize the report of performance metrics of stress, commitment and concentration. On the other hand, for the analysis of the percentage of pupil dilation, the SMI eye tracking device (SMI Eye Tracking Glasses SMI ETG) will be used; with the aim of verifying the information obtained in the literature and determining the level of reliability of the evaluation methods with respect to a subjective scale. Table 2. Scheme of the experiment Independent variable Dependent variable Controlled variables Augmented interface design Extrinsic cognitive load Training activity Number of treatments: 2 Range O1 - Related to EEG illumination Frequency, voltage and Treatment 1 (T1) amplitude of brain waves Low level of interactivity O2 Range - Related to eye Treatment 2 (T2) tracking High level of interactivity Number of blinks Percentage of pupil dilation The test will be carried out in person in a closed room with controlled lighting. The protocol begins with the explanation of the experiment, and the signing of the informed consent. The user is directed to a hair washing area in order to meet biosafety parameters and at the same time to control the humidity of the scalp that will facilitate subsequent data collection. Once the washing is done, the subject is positioned in front of the guide marks to start with the calibration of the SMI glasses, the user must follow the marks presented; Afterwards, the EPOC + electroencephalograph will be positioned and through the EMOTIV APP application, a quality of contact of the electrodes will be established not lower than 95%, to finalize the assembly of equipment, the Hololens augmented reality glasses will be positioned. For the analysis of results, the ordinal categorical variables will comply with a statistical analysis based on contingency tables and frequency diagrams. Likewise, a comparison of samples will be carried out to determine the significant differences between them, using the Chi-square test statistic with a p-value <0.05 to reject the null hypothesis of equality. In the case of continuous data, once the data has been obtained and filtered, it will be analyzed descriptively through the SPSS statistical software where the internal behavior of the data is analyzed from measures of central tendency and position, such as box plots and whiskers to establish consistency in the data. 3 Finally, the inferential analysis will be carried out, in this case, if the data are parametric, a mean comparison study will be carried out using ANOVA as a test statistic and it will be considered that there are significant differences for p-value <0.05. For non-parametric data: The Kruskal-Wallis test would be used initially to determine if there are differences between the groups and later a Wilcoxon signed rank test to compare the two related samples. With the analysis of the data, it is expected to accept the established hypothesis that it supposes an improvement in the specific evaluation of external cognitive load caused by the stimulus in graphical user interfaces from the use of multimodal approaches. References 1. Adams, R. (2007). Decision and stress: cognition and e-accessibility in the information workplace. Univ Access Inf Soc, 5, 363–379. https://doi.org/10.1007/s10209-006-0061-9 2. Akçayir, M., Akçayir, G., Pektaş, HM, & Ocak, MA (2016). Augmented reality in science laboratories: The effects of augmented reality on university students' laboratory skills and attitudes toward science laboratories. Computers in Human Behavior, 57, 334–342. https://doi.org/10.1016/j.chb.2015.12.054 3. Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, 1–11. https://doi.org/10.1016/j.edurev.2016.11.002 4. Andrade-Lotero, LA (2012). Cognitive load theory, design and multimedia learning: A state of the art. Magis: International Journal of Research in Education, 5 (10), 75–92. 5. Blessing, & Chakrabarti. (2009). DRM: A Design Research Methodology. In DRM, a Design Research Methodology (pp. 11–42). Springer US. https://doi.org/https://doi.org/10.1007/978-1-84882-587-1_2 6. Brunken, R., Plass, JL, Leutner, D., & Brünken, R. (2010). Direct Measurement of Cognitive Load in Multimedia Learning Direct Measurement of Cognitive Load in Multimedia Learning. Educational Psychologist, 38 (1), 53–61. https://doi.org/https://doi.org/10.1207/S15326985EP3801_7 7. Neville, AS, Salmon, PM, Walker, GH, Baber, C., & Jenkins, DP (2005). Human Factors Methods: A Practical Guide for Engineering. In Ashgate Publishing (Ed.), Angewandte Chemie International Edition (First Edit, Vol. 6, Issue 11, pp. 77–115). Taylor & Francis. 8. Paas, F., Tuovinen, JE, Tabbers, H., & Van Gerven, PWM (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38 (1), 63–71. https://doi.org/10.1207/S15326985EP3801_8 9. Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4 (4), 295–312. https://doi.org/10.1016/0959-4752(94)90003-5 4