Preliminary steps towards detection of proactive and reactive control states during learning with fNIRS brain signals ? Alicia Howell-Munson1[0000−0002−7390−9675] , Deniz Sonmez 2[0000−0003−3389−2466] Unal , Erin Walker2[0000−0002−0976−9065] , Catherine 3[0000−0001−6686−2189] Arrington , and Erin Solovey1[0000−0003−2423−4963] 1 Worcester Polytechnic Institute, Worcester MA 01609, USA 2 Pittsburgh University, Pittsburgh PA 15260, USA 3 Lehigh University, Bethlehem PA 18015, USA Abstract. This paper describes a two-pronged approach to creating a multimodal intelligent tutoring system (ITS) that leverages neural data to inform the system about the student’s cognitive state. The ultimate goal is to use fNIRS brain imaging to distinguish between proactive and reactive control states during the use of a real-world learning environ- ment. These states have direct relevance to learning and have been dif- ficult to identify through typical data streams in ITSs. As a first step towards identifying these states in the brain and understanding their effects on learning, we describe two preliminary studies: (1) we distin- guished proactive and reactive control using fNIRS brain imaging in a controlled continuous performance task and (2) we prompted students to engage in either proactive or reactive control while using an ITS to understand how the two modes affect learning progress. We propose in- tegrating the fNIRS datastream with the ITS to create a multimodal system for detecting the user’s cognitive state and adapting the environ- ment to promote better learning strategies. Keywords: Intelligent tutoring systems · functional near-infrared spec- troscopy · brain-computer interfaces. 1 Introduction Intelligent tutoring systems (ITS) allow students to get personalized assistance by collecting valuable information from their actions while learning. This infor- mation includes how the students navigate through the system, the correctness of their responses and the materials they struggle with [27]. This data is retrieved from the logs that are generated when students interact with the system; how- ever, these systems are not able to receive any input when students pause and do not produce log events. The states that students experience at these times may be indicative of beneficial behaviors such as self-monitoring and self-reflection or harmful behaviors such as mind wandering or going off-task [19, 6]. We propose ? The National Science Foundation under Grant Nos. (1835307 and 1912474). Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 A. Howell-Munson et al. to introduce neural data as an additional input source for an ITS to fill in the gaps of data during pauses. Long pauses in ITS log data contain a rich and complex set of possible cognitive activities that may or may not support the learning objectives of the ITS task. Defining these cognitive states is important for building a clear model of the students’ behavior. A key underlying process is the nature of cognitive control during and immediately after these pauses. Cognitive control describes the set of processes that coordinate thoughts and guide actions in support of goal-directed behavior. Prior work investigated this kind of behavior in various ways within ITS research from promoting self-regulated learning strategies [2, 5] to detecting and intervening to students zoning out [11]. Even though these behaviors have been heavily studied, their underlying mechanism of cognitive control has been less explored in this line of research. Our research is built on the dual mechanism of control (DMC) framework [7], which includes proactive and reactive control. Proactive control is the main- tenance of task-relevant information for sustain periods and elicits an early se- lection for the goal. An example of proactive control is making the goal to run errands after work and scheduling the rest of the day to accomplish this goal. Reactive control is the late-correction for a goal when stimuli in the environment triggers a just-in-time response. For reactive control, one would make the goal to run errands after work and then by the end of the day, realize they still need to run errands and are perhaps out of time now. The balance between proactive and reactive control can shift based on multiple factors and we hypothesize that it will influence the efficiency and accuracy of goal-directed behavior in learning environments. We explore this hypothesis more in Section 3. In this research, we bring a two-pronged approach to studying proactive and reactive control modes in ITS use. First, we examined behavioral and neural data from a simple continuous performance task that allows for identification of periods of proactive and reactive control (Section 2). This line of work is aimed at defining ground truth states of proactive and reactive control based on behavioral data and capturing neural signatures associated with each control mode. Second, we explored the manipulation of proactive and reactive control states during ITS use through instruction of strategy for participants and measured performance on the task in different control states (Section 3). While the results presented in this paper are preliminary, they provide a foundation for future multimodal intelligent tutoring systems using brain data. Our long term goals are to measure cognitive control states in real-world ITS use through behavioral and neural data. 1.1 Multimodal approach to an intelligent tutoring system Several techniques have been adopted to understand student behavior and needs in addition to analyzing log data from ITSs. Many use additional modalities such as audio, video, and/or sensor data. Eye tracking has been one of the most popular technologies within this line of research. It has been used to detect the lapses in students’ attention and reorient it on the learning activity [11], and to predict both positive behaviors such as self-explanation [10] or negative behaviors Towards detection of cognitive states during learning with fNIRS 3 such as mind-wandering [17, 18]. Other physiological sensors have been used to identify student affective states such as boredom, frustration, excitement, and concentration [28, 4]. Researchers have also investigated collaborative processes during learning using audio, video, and physiological measures [21, 12]. Within this work, we are interested in bringing in brain data to disambiguate the learner cognitive states in the pauses between logged events during ITS use. Previous research has shown that functional magnetic resonance imaging (fMRI) can be used to detect deep processing while problem solving [3]. Other studies have shown electroencephalogram (EEG) can be used to predict student perfor- mance [9, 15], and also to understand student emotions and engagement [16, 25]. Based on prior work, brain sensing, collected through use of fMRI, can detect more complex processing such as cognitive control states [8, 7]. Functional near- infrared spectroscopy (fNIRS) is a brain sensing technique that works similarly to fMRI and has been proven to work well in typical human-computer interac- tion environments [24]. We argue that data collected through fNIRS combined with the contextual information that log data provide will allow us to have a deeper understanding of students’ cognitive states. 1.2 Functional near-infrared spectroscopy (fNIRS) fNIRS is a brain-imaging tool that is safe, portable, easy to use, and quick to set up. These characteristics have led to an increased adoption in human-computer interaction research. It detects hemodynamic changes associated with neural activity in the brain while performing tasks [24]. Because fNIRS enables brain activity to be measured continuously during interactive computing tasks, it has promise for understanding user experience with systems such as an ITS. The fNIRS sensors are arranged on a mesh cap worn on the head and uses light to detect the hemodynamic response, changes in blood oxygen over time resulting in neural activity, from 1-3cm deep into the cortex [24]. The fNIRS signal reaches its peak between 4 and 7 seconds after a stimulus. fNIRS has been shown to be robust in typical human-computer interaction scenarios, including during typing, mouse clicking [24], and verbalization [23]. Real-time fNIRS brain data has been used to modulate interruptions [1] and enable attention-aware systems [22]. 2 Preliminary study 1: cognitive control task The purpose of this study was to explore the use of fNIRS to identify neural patterns of activation associated with proactive and reactive control in a simple continuous performance task. fMRI work shows that proactive control is associ- ated with larger responses that establish goals and reactive control is associated with larger responses to target stimuli. In the controlled task we used (Section 2.1), both modes of control may drive participant behavior and the shifts be- tween proactive and reactive control states can be identified by the types of errors that occur on particular trials. Our approach was to use these behavioral markers as a ground truth for proactive and reactive control. To assess whether fNIRS could identify patterns associated with each control state, we examined neural activity during the time leading up to a relevant behavioral marker. 4 A. Howell-Munson et al. 2.1 Experimental task The AX-continuous performance task (AX-CPT) presents a series of letters, as shown in Figure 2. Participants make a key press response for every letter. The letters appear in cue probe pairs; the first letter serves as a cue for the second letter. The AX pair represents the target trial where a unique response is made to the probe X, when it is preceded by cue A. All other cue probe pairs represent non target trials and are responded to with a different key press. Thus, for an appropriate response to the probe, the participants must maintain the cue context (A or not A) throughout the inter-stimulus interval (ISI) [7]. Responses made on non target trials are particularly important for identifying reactive and proactive control. Both AY (A cue is followed by a probe letter that is anything other than X) and BX (the cue is not an A, but the probe is an X) trials represent non target trials where probe response errors represent false alarms [7]. AY errors suggest proactive control because the participant was holding the A cue in mind and anticipating a target response, resulting in a false alarm when the non target probe appeared. BX errors suggest reactive control because the participant reacts to the X probe with the most common response to that probe even though it is not relevant in the B cue [8]. We will use these behavioral markers to examine fNIRS signals in preceding time windows to identify brain activity that is indicative of proactive and reactive control. Fig. 1: Example trial line for the AX-CPT paradigm. A single cross appeared for 500 ms before the cue, which appeared for 1000 ms. This was followed by three crosses during the inter-stimulus interval of 2000 or 6000 ms before the probe, which appeared for 500 ms. Participants had 1000ms to respond to the cue and probe. An AY non target, BX non target, and AX target trial are shown. 2.2 Procedure We recruited 23 participants (11 male) between 18 and 23 years old (M = 19.5, SD = 1.4). Two participants’ data was removed due to large amounts of noise across more than half of the fNIRS sensors. Participants were compensated with either coursework credit or payment of $15.00. The experiment was performed in a controlled laboratory environment with minimal distractions. The fNIRS signals were recorded using a NIRx NIRSport2 fNIRS device with a sampling rate of 8.7Hz [20]. The device was configured with a 21 channel design of the prefrontal cortex using eight sensors and eight detectors (Fig. 2). Each sensor and detector were approximately 3 cm apart, which allows measurement of 2-3 cm deep into an adult brain cortex [24]. Towards detection of cognitive states during learning with fNIRS 5 After reviewing and signing the informed consent, the fNIRS cap was placed on the participant’s head. Participants gave verbal confirmation that the cap was comfortable before proceeding. Participants began the full task once they got each trial type correct two times in a row in a practice AX-CPT block. Participants verbally told the researcher the AX-CPT rules to ensure that each participant understood the AX-CPT task. Each participant saw a total of 320 cue-probe paired trials in the following amounts: 96 AX trials, 64 AY, BX, and BY trials, and 32 “catch” trials. Catch trials did not require a response to the probe and encourage proactive control. Between the cue and probe, half of the trials had ISI of 2000ms and half had 6000ms. ISI timing was randomized and the order was different for each participant. 2.3 Brain data results The brain signals were divided into windows beginning one second before cue onset and lasting 18 seconds to allow the 4-7 second hemodynamic response from both cue and probe to peak. All analyses were conducted on change in oxygenated hemoglobin levels from an average of the first 8 frames of the window. To identify sensor locations in the brain that are relevant to the AX-CPT task, we took the average signal across all trials for each sensor and selected those sensors that showed a clear peak in the time period‘ following both cue and probe. To further consider the differences between proactive and reactive control, we used the key behavioral markers of error trials on AY and BX trials, respectively. Our data set contained 48 AY errors and 102 BX errors across all participants. We selected trials within a five trial window leading up to each error response that also had a 6 second ISI where the temporal separation between cue and probe allowed ample time for the hemodynamic response function to peak. We averaged the signal for trials leading up to AY and BX errors and analyzed the difference between the trial types. Trials leading up to an AY error should show a peak following the cue, but not the probe; whereas trials leading up to a BX error should show a peak to the probe, but not the cue. Our initial results show that one sensor located in the right dorsolateral prefrontal cortex (rDLPFC) had this pattern, distin- guishing between the two trials types (Figure 2). These patterns align with prior block-wise fMRI data patterns [8] associated with conditions conducive to proactive and reactive control. Here, we demonstrate that these states can also be distinguished in neural data from a string of trials within the same block where fluctuations between proactive and reactive control state occur naturally. By successfully identifying this dissociation between the two states with fNIRS data, future work can use these states to inform decisions made by an ITS. By reducing and expanding the number of trials leading up to an AY or BX error response, we could define the time prior to an error response that the participant is in a particular cognitive control mode. Additionally, after ascertaining the period in which a cognitive control state is most evident, we will confirm that the dissociation between the two cognitive states seen in the rDLPFC is significant through mixed-effects linear modeling. 6 A. Howell-Munson et al. Fig. 2: Left: fNIRS sensor positions. S signifies light sources and D signifies detectors. Right: Hemodynamic response functions, measured in oxygenated hemoglobin, show distinct patterns for trials leading up to AY errors (red) associ- ated with proactive control and trials leading up to BX errors (black) associated with reactive control. Solid line indicates the cue response period and dashed line indicates the probe response period. This sensor was located above rDLPFC. 3 Preliminary study 2: intelligent tutoring system In Section 2, we showed that we could identify proactive and reactive control using fNIRS in an abstract task. Within this section we aim to understand if proactive and reactive control can be induced during problem solving and if using one of these modes of cognitive control affects learning. In order to achieve this goal, we designed an experiment using ASSISTments [14] without integrating the neural data at this time. ASSISTments is an online tutoring tool that allows students to get help and feedback while also providing assessment data to teachers. ASSISTments provides a flexible environment to create problem sets with predefined hints and feedback for students. 3.1 Task Design We created a problem set that consists of nine probability problems. The prob- lems were covering three topics (basic probability, addition rule for probability of non-mutually exclusive events, and multiplication rule for probability of depen- dent events). All of the problems were divided into three to four substeps in order to create a goal maintenance scenario similar to [7]. When the participant was presented with a problem for the first time, they saw the full problem; however, they were not expected to solve it at first. The participant was only asked to rate their confidence level in solving the particular problem. After rating their confi- dence, the participant sees the first step to solve the problem. After completing each step, participants were given the next step until they reached the last one which would lead them to the solution of the problem. Within this scenario, one can think of the initial step as the time that the participant extracts the goal of the problem. How participants solved the substeps allows us to observe if they could maintain that goal in presence of other cognitively demanding events. Participants were randomly assigned to one of the two cognitive control modes. With these conditions, we manipulated how participants approached the Towards detection of cognitive states during learning with fNIRS 7 substeps of the problems. In the proactive condition, participants were prompted to think about how the substep they were solving was related to the goal of the full problem that they saw. In the reactive condition, the prompt in the substep instructed the participant to focus on the current substep. With these prompts, we hypothesized that proactive or reactive control [13] can be induced within a learning task as participants in the proactive condition would be practicing active goal maintenance while the participants in the reactive condition would only focus on the current step. This exact experiment design was used in an ear- lier thinkaloud study [26]. We replicated this experiment design excluding the thinkaloud protocol to get more accurate behavioral data. 3.2 Procedure We recruited 29 participants (5 male) between 18 and 22 years old (M = 19.8, SD = 1.13). The participants were recruited through emails that are sent to student mailing lists and announcements on online student bullet-in boards in a univer- sity in the Northeastern US. The inclusion criterion was not having completed more than two university level math courses. After providing informed consent, participants were introduced to ASSISTments and given time to practice an- swering problems. Then, they took a pre-test that had six probability problems on ASSISTments. After the pre-test, they solved another practice problem to get used to step by step problem solving and using proactive or reactive control based on the condition they were assigned to. After the practice, they engaged in problem solving activity while also using proactive or reactive control as the way they were shown during the practice. Participants took a post-test that is isomorphic to the pre-test after solving problems on ASSISTments. Participants answered a demographic questionnaire at the end of the study. This study was run completely online due to the COVID-19 outbreak. We communicated with the participants using Zoom. We asked participants to turn their camera off to protect their identities. We asked them to share their screen to be able to watch and record their actions during the study. 1 participant was excluded as they scored 100% on the pre-test. 3.3 Results Prior work [26] has shown success in inducing proactive and reactive control during problem solving when using thinkalouds. They found participants in the proactive condition made significantly more statements that include the goal of the problems. Without the thinkaloud protocol, one indicator that the partici- pants behaved as expected is the average time spent on the problem steps. We hypothesized that the participants in the proactive condition should spend more time on the problem steps because they need to think about how the current problem step they are solving helps them reach the goal of the main problem. Figure 3 pictures the average time spent on the problem steps by condition for both studies. We conducted a two sample t-test to determine if participants in the proactive condition spent more time on the problem steps because of their reflection. Results supported our hypothesis. We found that the participants 8 A. Howell-Munson et al. Fig. 3: Average time spent on problem steps by conditions (in seconds). Error bars represent the standard error of the means in the proactive condition spent significantly more time on the problem steps (t(24.05) = 2.36, p < .05). We confirmed that this manipulation of cognitive control was still effective without the thinkaloud protocol. In order to see if us- ing one mode of cognitive control had any effects on learning, we conducted a repeated measures ANOVA with condition as the between-group variable and the tests as the within-group variable. We found the main effect of test was sig- nificant (F (1, 26) = 51.33, p < .001) meaning that participants improved from pre to post test. However, results showed no significant interaction between the condition and test (F (1, 26) = 1.45, p > .05) indicating no significant difference in learning gain (post - pre) between proactive (M = 0.24, SD = 0.23) and reactive (M = 0.33, SD = 0.19) modes. 4 Discussion We have presented preliminary results on a two-pronged approach that is at- tempting to provide ground truth data on different modes of cognitive control based on the DMC framework [7] and induce these modes in a realistic learning environment for potential benefits during learning. We first showed that we can identify proactive and reactive modes of cognitive control using fNIRS with a similar methodology as [8], who identified these states with fMRI. We identified the rDLPFC as a region of interest in detecting these cognitive states. Further analysis is needed to understand if there are additional regions of interest that can detect only one of the cognitive control modes. In theory, if there is a region that is highly active in the brain for only proactive control, then it can be used as a confirmation when activation for proactive control is seen in the rDLPFC. Next, we showed that those two modes can be induced in students during more realistic ITS use replicating the experiment design described in [26]. We showed that the cognitive control manipulation was still successful without hav- ing participants think out loud. We detected no significant differences in the pre- and post-tests between students who engaged in proactive and reactive control. This could be due to the order and difficulty levels of the presented problems. Participants may have improved because they had a better understanding of Towards detection of cognitive states during learning with fNIRS 9 the problem types after seeing each three times, and that may be why we did not see any difference between proactive and reactive control. Using proactive or reactive control can still be beneficial in different levels of challenge. We will modify the experimental design to test it once more. We propose integrating the fNIRS datastream with the ITS to create a mul- timodal system for detecting the user’s cognitive state and adapting the environ- ment to promote better learning strategies. Our approach can impact a broad range of ITSs, which are collectively used by thousands of students. Interventions within tutoring systems can meet an individual student’s needs better through modeling cognitive states and their underlying mechanisms. To do so, we will integrate fNIRS into the ITS task that promotes proactive and reactive control in participants. This would confirm that the neural regions of interest identified in the AX-CPT transfers to using an ITS. References 1. Afergan, D., Hincks, S.W., Shibata, T., Jacob, R.J.: Phylter: a system for modu- lating notifications in wearables using physiological sensing. In: International con- ference on augmented cognition. pp. 167–177. Springer (2015) 2. Aleven, V.A., Koedinger, K.R.: An effective metacognitive strategy: Learning by doing and explaining with a computer-based cognitive tutor. Cognitive science 26(2), 147–179 (2002) 3. Anderson, J.R., Betts, S., Ferris, J.L., Fincham, J.M.: Cognitive and metacogni- tive activity in mathematical problem solving: prefrontal and parietal patterns. Cognitive, Affective, & Behavioral Neuroscience 11(1), 52–67 (2011) 4. Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion sensors go to school. In: AIED. vol. 200, pp. 17–24. Citeseer (2009) 5. Azevedo, R., Hadwin, A.F.: Scaffolding self-regulated learning and metacognition– implications for the design of computer-based scaffolds (2005) 6. Bixler, R., D’Mello, S.: Toward fully automated person-independent detection of mind wandering. In: User modeling, adaptation, and personalization. pp. 37–48. Springer, Cham (2014) 7. Braver, T.S.: The variable nature of cognitive control: a dual mechanisms frame- work. Trends in cognitive sciences 16(2), 106–113 (2012) 8. Braver, T.S., Paxton, J.L., Locke, H.S., Barch, D.M.: Flexible neural mechanisms of cognitive control within human prefrontal cortex. PNAS 106(18), 7351–7356 (2009) 9. Chaouachi, M., Heraz, A., Jraidi, I., Frasson, C.: Influence of dominant electrical brainwaves on learning performance. In: E-Learn: World Conference on E-Learning in Corporate, Government, Healthcare, and Higher Education. pp. 2448–2454. As- sociation for the Advancement of Computing in Education (AACE) (2009) 10. Conati, C., Merten, C.: Eye-tracking for user modeling in exploratory learning environments: An empirical evaluation. Knowledge-Based Systems 20(6), 557–574 (2007) 11. D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: A gaze-reactive intelli- gent tutoring system. IJHCS 70(5), 377–398 (2012) 12. Ezen-Can, A., Grafsgaard, J.F., Lester, J.C., Boyer, K.E.: Classifying student di- alogue acts with multimodal learning analytics. In: Proceedings of the Fifth Inter- national Conference on Learning Analytics and Knowledge. pp. 280–289 (2015) 10 A. Howell-Munson et al. 13. Gonthier, C., MAcnamara, B.N., Chow, M., Conway, A.R.A., Braver, T.S.: In- ducing proactive control shifts in the AX-CPT. Frontiers in psychology 7(1822) (2016) 14. Heffernan, N.T., Heffernan, C.L.: The assistments ecosystem: Building a platform that brings scientists and teachers together for minimally invasive research on human learning and teaching. International Journal of Artificial Intelligence in Education 24(4), 470–497 (2014) 15. Heraz, A., Frasson, C.: Predicting learner answers correctness through brainwaves assesment and emotional dimensions. In: AIED. pp. 49–56 (2009) 16. Heraz, A., Frasson, C.: Towards a brain-sensitive intelligent tutoring system: de- tecting emotions from brainwaves. Advances in Artificial Intelligence 2011 (2011) 17. Hutt, S., Krasich, K., Brockmole, J., D’Mello, S.K.: Breaking out of the lab: Miti- gating mind wandering with gaze-based attention-aware technology in classrooms. In: Proc. of ACM CHI’21 Conference. ACM (2021) 18. Hutt, S., Mills, C., Bosch, N., Krasich, K., Brockmole, J., D’mello, S.: “out of the fr-eye-ing pan” towards gaze-based models of attention during learning with tech- nology in the classroom. In: Proc. User Modeling, Adaptation and Personalization. pp. 94–103 (2017) 19. Kalyuga, S.: Enhancing instructional efficiency of interactive e-learning environ- ments: a cognitive load perspective. Educational Psychology Review (19), 387–399 (2007) 20. LLC, N.M.T.: Nirsport user manual (Sept 2015), https://support.nirx.de/wp- content/uploads/2016/06/NIRSport88-UserManual-2015-09-R30-inkl.- Baseplate.pdf 21. Malmberg, J., Järvelä, S., Holappa, J., Haataja, E., Huang, X., Siipo, A.: Going beyond what is visible: What multichannel data can reveal about interaction in the context of collaborative learning? Computers in Human Behavior 96, 235–245 (2019) 22. Peck, E.M., Carlin, E., Jacob, R.: Designing brain-computer interfaces for attention-aware systems. Computer 48(10), 34–42 (2015) 23. Pike, M.F., Maior, H.A., Porcheron, M., Sharples, S.C., Wilson, M.L.: Measuring the effect of think aloud protocols on workload using fnirs. In: Proceedings of the SIGCHI conference on human factors in computing systems. pp. 3807–3816. CHI ’14, ACM, New York, NY, USA (2014) 24. Solovey, E.T., Girouard, A., Chauncey, K., Hirshfield, L.M., Sassaroli, A., Zheng, F., Fantini, S., Jacob, R.J.: Using fnirs brain sensing in realistic hci settings: ex- periments and guidelines. In: Proc. ACM symposium on User interface software and technology. pp. 157–166. ACM (2009) 25. Stevens, R.H., Galloway, T., Berka, C.: Eeg-related changes in cognitive workload, engagement and distraction as students acquire problem solving skills. In: Inter- national conference on user modeling. pp. 187–196. Springer (2007) 26. Unal, D.S., Arrington, C.M., Solovey, E., Walker, E.: Using thinkalouds to under- stand rule learning and cognitive control mechanisms within an intelligent tutoring system. In: AIED. pp. 500–511. Springer (2020) 27. VanLehn, K.: The behavior of tutoring systems. International journal of artificial intelligence in education 16(3), 227–265 (2006) 28. Woolf, B., Burleson, W., Arroyo, I., Dragon, T., Cooper, D., Picard, R.: Affect- aware tutors: recognising and responding to student affect. International Journal of Learning Technology 4(3-4), 129–164 (2009)