This study is built upon a behavior-based framework for real-time attention evaluation of higher education learners in e-reading. Significant challenges in AI model developments for learning analytics have been 1) defining valid indicators and 2) connecting the analytics results to interventions, balancing the generalization and personalization needs. To address this, we utilized a public multimodal WEDAR dataset and trained a neural network model based on real-time features of learners, aiming at predicting learners' moment-to-moment distractions. Real-time features for model training include 30 learners' attention regulation behaviors annotated every second, reaction times to blur stimuli, and page numbers indicating various reading phases. Our preliminary model based on a neural network has achieved 66.26% accuracy in predicting self-reported distractions. Based on the model, we suggest a framework of a Behavior-based Feedback Loop for Attentive e-reading (BFLAe). It has text blur as feedback, a mechanism responsive to learners' distractions that also works as data for next-round feedback. The general feedback implementation rules are established on a statistical analysis conducted on all learners. In addition, we propose a strategy for personalizing feedback using a quartile analysis of individual data, promoting learner-specific feedback. Our framework addresses the high demand for an automated e-learning assistant with non-intrusive data collection based on real-world settings and intuitive feedback provision. The feedback system aims to help learners with longer attention spans and less frequent distractions, leading to more engaging e-reading.
A
With recent quantitative and qualitative growth in data and computing availability,
machine learning approaches are becoming more prevalent in learning analytics and
educational data mining [
These authors contributed equally.
performances and specific internal states of learners from their afective (e.g., arousal,
valence [
Regardless of their accurate prediction capabilities, sensor-based approaches are often criticized for being intrusive [12], changing the nature of learning experiences. Thus, various computer vision-based approaches [13, 8] have been suggested to make learning and system design more seamless for real-world applications. Especially behavior-based analytics is valuable in that particular behavior that machines recognize is also observable and semantically interpretable to humans to some extent [14, 15]. Common challenges in behavior-based machine learning applications in learning analytics have been 1) to find valuable features for model training [14] and 2) to specify the implementation conditions and parameters that best support the accurate recognition of targeted signals [16]. 3) Also, closing the feedback loop, considering generalization and personalization [12] in the analytics phases, and implementing the feedback has been dificult.
In this regard, our objective is to suggest a Behavior-Based Feedback Loop for Attentive
e-reading (BFLAe) framework, which involves 1) webcam-based video data collection,
2) computer vision-based learning analytics, 3) blur feedback implementation in text,
and 4) further cognitive&behavioral changes of learners as consequences of feedback
loop implementation. The framework is built upon a multimodal WEDAR dataset,
which provides valuable insight into learners’ behavior during e-reading activities. Our
approach involves training a neural network model on real-time features that reflect
learner behavior, including attention regulation behaviors, reaction times to blur stimuli,
and page numbers that reflect diferent reading phases from the public WEDAR dataset
[17]. These features provide a basis for predicting learners’ perceived distractions and
form a foundation for implementing feedback mechanisms. By implementing the blur
feedback on the screen-based e-reader, we aimed to close the feedback loop that enables
the further loops, which is not obstructive to the primary reading task and is semantically
intuitive. Feedback could potentially help learners reflect on their current state and
strategize for future reading [
We believe that this personalized, behavior-based feedback loop ofers a practical
solution to the challenges faced by the fields of Technology-Enhanced Learning (TEL)
and Multimodal Learning Analytics (MMLA), promoting more engaging, efective, and
individually tailored learning experiences [
Our contributions are as stated follows: • According to our best knowledge, it is the first framework to introduce a real-time feedback loop for attentive e-reading. Our webcam-based behavioral framework is non-obstructive and applicable to diverse e-learning scenarios which involve e-reading as a major learning activity. Our BFLAe framework with increasing digital reading in formal and informal learning with prevalent digital technologies will be more valuable. • It is a framework built upon WEDAR, a multimodal public dataset collected in an e-reading scenario. It ofers more relevant data specified for attention measurement for e-reading. With the implementation details depicted in our framework, the work can be reproduced and further elaborated for specific scenarios based on diferent tasks and implementation requirements. • By specifying the statistical values of diferent behavior labels that represent attentive (i.e., neutral) and distractive (i.e., attention regulation behaviors) learner states, we provide researchers and instructional designers with options to make choices on thresholds for the feedback trigger. As feedback necessities vary depending on the system goals, our analysis result can provide valuable ground for the feedback rules for diferent systems.
In this section, we briefly analyze the multimodal WEDAR dataset. By doing so, we tried to understand the dataset’s structure and attention regulation behaviors shown in e-reading and potential patterns that are shown together with the self-reported distractions.
We used the multimodal WEDAR dataset in our investigation [17]. This dataset comprises
human-labeled behavioral labels with five categories of attention regulation behaviors
and a neutral behavior as the label, all annotated in every second of the video data.
These videos were collected from 30 higher education learners. In particular, this study
used real-time distraction reports as the ground truth for distraction instances [
Eyebrow -raise -bring together
Blink -blink flurry -voluntary prolonged blink
Mumble -mumble reading
Hand -touch body -touch hand
Body -adjust torso -adjust arm -adjust head
Neutral -without attention regulation behaviors
We graphically represented the five categories of attention regulation behaviors and neutral behaviors along with distraction reports to discern potential visual patterns between attention regulation behaviors and self-reported distractions. As is evident in Figure 2, participants exhibited a wide range of reading speeds, ranging from 461 seconds (7.7 minutes) to 1661 seconds (or 27.7 minutes). Moreover, we noticed substantial variation in the use of attention regulation behaviors, as well as in the patterns of perceived distractions and the reporting of these distractions. Given this unobservability, the integration of machine learning becomes crucial. It also represents the limitations of human educators in detecting complex patterns hidden within the behavioral patterns of learners.
Video data collection
1 Cognitive & Behavioral Changes
Learner Distractions
Attention regulation behavior recognition using neural networks
2 Computer-based
E-reader
Computer Visionbased Learning
Analytics Webcam-based
Video Data
Collection Behavior-based
Feedback Loop for Attentive e-reading (BFLAe) Text Blur as
Intervention 4
Extending the attention span, decreasing the frequency of distractions Blur feedback application on the text area based on feedback rules 3
This section presents the system’s architecture, as shown in Figure 3. Drawing on previous research in the realm of multimodal learning analytics [16, 12], critical factors in forming a multimodal feedback loop for learning include 1) the alignment and integration of data streams, 2) the identification of learning requirements, 3) informed design decisions for multimodal feedback, and 4) the observation of implications within specific learning scenarios. Consequently, we propose a four-stage approach to BFLAe.
In the first stage, webcam-based video data is collected during e-reading. This method ofers an unobtrusive approach compared to other sensor-based strategies. The second stage involves learning analytics, which is based on a model developed from attention regulation behaviors and self-reported distractions. The following section will detail the specific features used in model training and the rules for triggering system feedback. In the third stage, a blur efect is applied to the reader’s screen for the feedback generation condition, which was decided in the previous phase. The blur efect can be deactivated by the learner clicking on the reading area. This stage not only aids learners by increasing arousal but also serves as additional data for further learning analytics since the reaction time provides crucial cues about the learners’ cognitive states. The final stage of the loop aims to induce cognitive and behavioral changes in learners. Specifically, the system’s objectives are: 1) extending the attention span between distractions and 2) decreasing the frequencies of distractions, as measured by attention regulation behaviors, reaction speed to the blur stimuli, and self-reported distractions.
This section introduces the features and computational model that we have established to predict attention levels: a prerequisite step integral to the subsequent feedback generation.
The WEDAR dataset provides behavioral attributes in real-time from 30 higher education
learners engaged in e-reading. As referenced in Table 1, eight distinctive features have
been harnessed for model training. Five attention regulation behaviors were used as
binary features (feature 1) and independent features (features 2-6). Reaction times to
secondary blur stimuli, activated at random intervals, have been implemented as another
feature (feature 7). Reaction time is a classical measure used to assess learners’ arousal
levels [
We utilized eight real-time features described in Table 1 for our model training. We initially partitioned our dataset into training and testing sets, comprising 80% and 20% of the data, respectively. We balanced the data set, using the synthetic minority oversampling TEchnique (SMOTE) to prevent an imbalance between distracted and attentive states so that neither state would dominate the other in proportion and provide suficient data points for the training. Subsequently, we applied min-max normalization to confine the data distribution between 0 and 1. This process was implemented to mitigate any potential bias from diferent data ranges. Furthermore, min-max normalization is acknowledged for its ability to accelerate training. It is particularly advantageous for our approach, which will have many data points from second-to-second recognition.
As shown in Figure 4, we employ a sequential neural network model with its linear stack of layers. Our network architecture comprises three hidden layers with a rectified linear unit (ReLU) activation function. To mitigate the risk of overfitting, we incorporated a dropout layer into our model, which is widely used for randomly nullifying a fraction of the layer’s output features during the training phase. In our case, the dropout layer is configured with a rate of 20%, omitting one-fifth of the input. The final layer of our model is a dense layer with a Sigmoid activation function, with an output range between 0 and 1. It is an optimal choice for our binary classification task. The loss function is designated as mean squared error (mse), the optimization algorithm is set as Adam, and the accuracy is selected as the metric for model evaluation during training. The model has reached an accuracy of 66.26%. This performance exceeds the 50.00% accuracy expected from random guess, which implies that the prediction capacity of the model is considerably better than the chance. The real-world implementation could be enhanced by integrating the feedback rules, which will be further elaborated on in the next section.
This section introduces the rationale for implementing blur stimuli, feedback rules, and Human-Computer Interaction (HCI) architecture. See Figure 5 for descriptions of HCI, showing the functions of components and blur feedback applied in response to learners’ distractions.
We suggest the implementation of blur on text area as automatic visual feedback, which
has also been used to measure reaction time in previous studies [
1) The blur stimuli serve a dual function: they trigger the learner’s arousal and simultaneously work as data points for future feedback loops. Diferent reaction times, behavioral features, and self-distraction reports are incorporated into the screen-based reader as next-round feedback, enabling more precise predictions and personalized feedback.
2) Critics often suggest that feedback interrupts the primary task by adding secondary
tasks to learners, inducing cognitive overload [
1 bPuatgtoens
(a) HCI components and functions: page, web-(b) Blur feedback is applied to the text area cam operation, blur deactivation, and dis- as an intervention triggered by recognized traction report buttons. distractions.
The window size in machine learning refers to the number of data points that are considered to capture information and contexts at each step, which is especially crucial for sequential data processing [4]. We propose tailoring diferent window sizes to diferent attention regulation behaviors to enhance the prediction of self-reported distractions. As evidenced in Table 2, derived from the WEDAR dataset, the minimum, maximum, average, median, standard deviations, and quartiles of behaviors exhibit variability of the duration of each state. The current distraction prediction model was designed based on second-to-second labeling for all attention regulation behaviors. However, incorporating diferent behaviors and applying a range of sliding windows could potentially improve the accuracy of the learners’ distraction predictions.
The system’s feedback mechanisms can be varied according to its specified objectives. For example, some may apply a window size spanning the third quartile to maximum values of specific behavior for attention prediction. On the contrary, those who require stricter self-regulation among learners may opt to utilize a window size between medium and maximum values for the same task. By establishing specific ranges that act as a foundation for feedback implementation, researchers and educational practitioners will benefit from devising their intervention rules, drawing on general learning behavior. Please note that our analysis has been performed on the WEDAR dataset. Thus, the predefined ranges may undergo further refinement with the accumulation of additional sample data in future studies.
The creation of personalized models can be facilitated by conducting quartile analysis
in individual data, considering individual diferences in relation to their own unique
behavioral status [
We propose a framework of behavior-based feedback loops for attentive e-reading. As established in previous research, the challenge of closing the feedback loop has been a recurring issue in the fields of TEL and MMLA. We leverage the multimodal WEDAR dataset in this work, which aids in developing behavior-based predictions of learners’ perceived distractions. Real-time features have been extracted to train a neural network that predicts learners’ perceived distractions. These features encompass attention regulation behaviors, reaction time to blur stimuli, and reading phases derived from page numbers. Our approach involves the implementation of blur feedback in response to learners’ distractions and establishing the foundation for feedback rules based on the statistical attention regulation behavior analysis derived from general data. Simultaneously, we propose a strategy for personalizing the feedback based on a quartile analysis of individual data. Our behavior-based model addresses the emerging need for an e-reader with automatic learning analytics and feedback mechanisms that can be applied to real-world scenarios.
Optimizing the window sizes of attention regulation behaviors for accurate distraction prediction A statistical analysis of learners’ data in e-reading has been performed in the current framework. Broad ranges of learners’ attention regulation behaviors have been derived, indicating learners’ states of attention and distraction. In future work, several ranges of diferent behavior recognition technologies will be applied and tested. Doing so will provide practical insights into real-time recognition and feedback generation that can best assist our feedback objectives.
Testing the efects of the automated feedback from an intelligent e-reading system Though the overall behavior-based feedback loop framework has been suggested, the efects of implementing automated feedback still need to be tested: investigating the attention span and frequencies of distractions. Our intelligent system can be further evaluated for subsequent efects, such as learning outcomes and perceived learning experiences, with various qualitative and quantitative measures. Our next step involves comparing the intelligent feedback loop based on the current BFLAe framework and time-based feedback.
Exploring the efects of feedback types and modalities In this work, we suggested blur feedback due to its intuitive actionability and less cognitive load than other feedback. However, with the same feedback timing, we still need to validate whether diferent types and modalities (e.g., speech-based feedback from conversational agents) of feedback provide additional value in learning. We will further test the efects of varying feedback with various types and modalities built into our current attention recognition mechanisms.
This work has been supported by the Leiden-Delft-Erasmus Center for Education and Learning (LDE-CEL). International Conference on Learning Analytics and Knowledge, Association for Computing Machinery (ACM), 2023. [4] O. Banos, J.-M. Galvez, M. Damas, H. Pomares, I. Rojas, Window size impact in human activity recognition, Sensors 14 (2014) 6474–6499. [5] L. Wang, Attention decrease detection based on video analysis in e-learning, Transactions on Edutainment XIV (2018) 166–179. [6] R. Bixler, S. DMello, Automatic gaze-based user-independent detection of mind wandering during computerized reading, User Modeling and User-Adapted Interaction 26 (2016) 33–68. [7] S. Hutt, K. Krasich, C. Mills, N. Bosch, S. White, J. R. Brockmole, S. K. DMello, Automated gaze-based mind wandering detection during computerized learning in classrooms, User Modeling and User-Adapted Interaction 29 (2019) 821–867. [8] M. Xuelin Huang, J. Li, G. Ngai, H. V. Leong, A. Bulling, Moment-to-moment detection of internal thought from eye vergence behaviour, arXiv e-prints (2019) arXiv–1901. [9] V. J. Flood, F. G. Amar, R. Nemirovsky, B. W. Harrer, M. R. Bruce, M. C.
Wittmann, Paying attention to gesture when students talk chemistry: Interactional resources for responsive teaching, Journal of Chemical Education 92 (2015) 11–22. [10] H. Chen, E. Tan, Y. Lee, S. Praharaj, M. Specht, G. Zhao, Developing ai into explanatory supporting models: An explanation-visualized deep learning prototype, in: The International Conference of Learning Science (ICLS), 2020. [11] Y. Lee, H. Chen, E. Tan, S. Praharaj, M. Specht, Flower: Feedback loop for group work supporter, in: The International Learning Analytics and Knowledge Conference (LAK demo session), 2020. [12] Y. Lee, G. Migut, M. Specht, Role of multimodal systems in technology enhanced learning (TEL): A scoping review (in press), in: 18th European Conference on Technology Enhanced Learning (EC-TEL23), 2023. [13] S. DMello, T. Jackson, S. Craig, B. Morgan, P. Chipman, H. White, N. Person, B. Kort, R. El Kaliouby, R. Picard, et al., Autotutor detects and responds to learners afective and cognitive states, in: Workshop on emotional and cognitive issues at the international conference on intelligent tutoring systems, 2008, pp. 306–308. [14] M. Cukurova, Q. Zhou, D. Spikol, L. Landolfi, Modelling collaborative problemsolving competence with transparent learning analytics: is video data enough?, in: Proceedings of the tenth international conference on learning analytics & knowledge, 2020, pp. 270–275. [15] H. Chen, H. Shi, X. Liu, X. Li, G. Zhao, Smg: A micro-gesture dataset towards spontaneous body gestures for emotional stress state analysis, International Journal of Computer Vision 131 (2023) 1346–1366. [16] D. Di Mitri, J. Schneider, M. Specht, H. Drachsler, From signals to knowledge: A conceptual model for multimodal learning analytics, Journal of Computer Assisted Learning 34 (2018) 338–349. [17] Y. Lee, M. Specht, Multimodal WEDAR dataset for attention regulation behaviors, self-reported distractions, reaction time, and knowledge gain in e-reading , 2023. doi:1 0 . 4 1 2 1 / 8 f 7 3 0 a a 3 - a d 0 4 - 4 4 1 9 - 8 a 5 b - 3 2 5 4 1 5 d 2 2 9 4 b . v 1 .
The WEDAR dataset, which has been used in the work, is available via