Multimodal measurement of human learning enables a data-driven approach to learning analysis and feedback generation. This position paper discusses the possibility of feedback based on multimodal learning analytics from how machine learning methods can be applied. In particular, we first discuss how (1) behavioral measurements, such as learner browsing logs and eye tracking, and (2) content analysis of learning materials can lead to (3) prediction and modeling of learners' states (e.g., performance) and (4) feedback generation, such as information presentation and content optimization, through some research examples. We then show future research directions of machine learning-based learners' state modeling for feedback generation.
Various measurements of learning behaviors, such as clickstreams of e-book manipulation [
The concept of feedback loops with machine learning models has been discussed in the field
of multimodal learning analytics (MMLA) (e.g., [
learner States: knowledge, attention, interest, learning style, etc.
Feedback
Update Feedback design for a target learner - Visualization - Recommendation, etc.
Dashboard, chatbot, e-books, texts, videos, e-portfolio, quiz, quizzes, etc. recommendation, etc. (2) Content
analysis e-Book System
Eye tracker /
Web camera (1) Behavioral measurement
- Page transition - Comment / mark - Gaze pattern, etc. (4) Feedback / content optimization Integrated analysis (3) Modeling
- Semantic features - Design features, etc.
Machine learning
- Performance prediction - State estimation, etc. technical components in our assumed feedback loop by limiting learning context to e-book and video viewing-based learning situations. Section 3 shows some examples of performance prediction models and slide-content generation, which have the potential to be used for feedback. We then discuss an important research question for feedback generation “what is an appropriate representation of learners’ state?” in Sec. 4.
The overall structure of the feedback loop envisioned in this paper is shown in Fig. 1. We
consider electronic text browsing logs (e.g., clickstreams) and eye-tracking data for the learners’
behavioral measurement in Fig. 1 (1). With the spread of learning management systems (LMSs)
and massive open online courses (MOOCs), learners’ behaviors, such as signing in/out and
assignment submission, are obtained by system logs. Besides, more detailed behaviors can now
be measured by e-book systems and online-course systems as operation logs of lecture materials
(e.g., page transitions, adding or deleting markers) [
Meanwhile, the potential and application range of content analysis in Fig. 1 (2) are now
increasing because of the advancement of machine learning. In particular, its media-analysis
capability covers a wide range of contents, including textbooks, quiz questions, lecture slides,
videos, and audio. Neural networks such as recurrent neural networks (RNNs) and
Transformers [
Behavioral data captured in (1) and the content features obtained in (2) are then used for learners’ modeling, such as performance prediction and the estimation of learners’ state (e.g., mental and cognitive states), using machine learning methods (Fig. 1 (3)). As many machine learning models lack interpretability or explainability, we need to consider how explainable the models should be, which depends on feedback objectives (see Sec. 4 for detailed discussion).
For the feedback step in Fig. 1 (4), it is important to decide whom to target and when and what
to provide. For example, it is possible to provide feedback to teachers on which slides students
browse during class [
This section introduces research examples we are working on toward realizing feedback to show some technical components described in Sec. 2 and to discuss how they can be combined.
In order to predict each student’s performance, we can utilize both (a) what learning content is used and (b) how each student behaves. We here hypothesize that combining the content information from (a) and behavioral features from (b) will enable us to achieve more accurate performance prediction compared to only using content-independent features (b).
Browsing-log data on an e-book system. Some universities use e-book systems to obtain
the log data of students’ browsing behavior when manipulating textbooks or slides [
Eye tracking data during video viewing. Eye tracking is used for a finer-grained
measurement of learners’ behavior during viewing lecture videos. The gazing point series can
be used to estimate learners’ various states, including “mind wandering” [13] and “actively
examining slide materials” [
The prediction of students’ performance (e.g., final grades, comprehension of each topic) can be used to generate various feedback for both students and teachers. Feedback forms include visualization on dashboards that learners can check themselves, recommendation of learning materials, interactive support (e.g., chatbots), and the detection of students who need help from teachers. The implementation and verification of these feedback methods need to be investigated in the future.
Research on feature extraction and automatic text summarization using neural networks is
rapidly advancing in the natural language processing (NLP) community. Text feature extraction
methods, including word embedding (e.g., Word2Vec [14]) and Transformer encoders (e.g.,
BERT [
Lecture-video emphasis using audio. Those vector representations and similarity measures are key to finding related text in learning materials. In our ongoing work, we utilize those vector representations of text data to realize automatic spotlighting of video content. Once the lecturer’s utterance is processed by text-to-speech, the similarity between slide text and speech is computed, and the corresponding regions can be emphasized to guide learners’ attention.
Slide generation from documents. Transformer-based automatic slide generation from documents [15, 16] has recently attracted the attention of researchers in the machine learning community. While the text layout in a slide is not considered in these studies, we try to generate slides from a Wikipedia page by automatically selecting text layouts to improve the slide readability. To select appropriate layouts, we train and utilize a BERT-based classifier for estimating the discourse relationship of a given sentence pair.
As our ongoing work on automatic content highlighting and generation are not individualized, it needs to be combined with learners’ performance modeling described in Sec. 3.1. In addition, not only the information of content but learners’ behavior data (Fig. 1 (1)) can also be used to ifnd important topics and pages that should be emphasized [ 17]. Therefore, the integration of (1), (2), and the learner model (3) is also an interesting challenge.
Machine learning techniques can be applied for various objectives, from predicting students’ performance to generating content, in the context of learning analytics loop as described in Sec. 2 and Sec. 3. However, it has not yet been elucidated on “what kind of learners’ information is required to provide appropriate feedback and how machine learning models can estimate such information.” Machine learning could contribute to learners’ state modeling in the following three levels: feature level, manually designed level, and automatically extracted level. In this section, we discuss the above questions regarding the learners’ state representation and present several challenges.
Feature-level representation. The most straightforward representation of a learner’s state is the amount of activity on learning materials or topics, such as the frequency the learner has viewed slide pages, texts, figures, and the count of correctly answering topic-related questions. Once the similarity among learning materials or topics is extracted automatically using machine learning techniques (e.g., Transformer encoders described in Sec. 3.2), the similarity structure of learning content provides essential information to infer learners’ knowledge in detail in behavioral and content-related feature space. This leads to the generation of meaningful feedback, including recommendations and optimization of learning content. Automatic or semi-automatic knowledge extraction from learning materials (e.g., [18]) may also facilitate this process.
Manually designed state representation. The second direction is to manually design a
learner’s state (e.g., emotion, attention, cognition, knowledge levels, skills, key competencies).
While these variables cannot be directly observed from behavioral data, it is possible to estimate
them using machine learning models once the model is trained using annotated datasets [
Automatically extracted state representation from data. Machine learning could contribute to extracting learners’ states automatically as a latent representation in a model through behavior analysis. This direction corresponds to the “end-to-end modeling of the learner’s internal state” described in Sec. 2. For example, the techniques of Knowledge Tracing [19], which model the knowledge state of individual learners from a series of questions and their answers, have been rapidly advanced with machine learning models. While it also predicts learners’ performance, similar to Sec. 3.1, a learner’s state is obtained as a latent variable through end-to-end model training. This opens up the possibility of finding a useful representation of learners’ states from the data without annotation, although there are dificulties in ensuring explanatory and interpretability. To make the latent variable in the end-to-end model more interpretable, we can further impose the model with prior or external knowledge as the design of model structures, parameter constraints, and regularizers. Graph neural networks [20] are examples of such model structures that would allow the smooth integration of knowledge in learning theory.
This position paper introduced how the recent machine learning techniques can be applied to the components of MMLA-based feedback loops, focusing on integrated content analysis with behavioral data and content generation. In particular, we discussed that the recent trend of representation learning would open up new possibilities for integrating multimodal behavioral and contextual data. To estimate learners’ states in deep and generate appropriate and detailed feedback, the field of learning analytics and machine learning can collaborate in many aspects, and this leads to a new framework of feedback loops in MMLA.
This work was supported by JSPS KAKENHI Grant Number JP19H04226. Warning System for Spotting at-Risk Students by Using eBook Interaction Logs, Smart Learning Environments 6 (2019). [12] N. Reimers, I. Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT
Networks (2019). arXiv:1908.10084. [13] S. Hutt, J. Hardey, R. Bixler, A. Stewart, E. Risko, S. K. D. Mello, Gaze-based Detection of Mind Wandering during Lecture Viewing, International Conference on Educational Data Mining (2017) 226–231. [14] T. Mikolov, K. Chen, G. Corrado, J. Dean, Eficient Estimation of Word Representations in
Vector Space (2013). arXiv:1301.3781. [15] E. Sun, Y. Hou, D. Wang, Y. Zhang, N. X. R. Wang, D2S: Document-to-Slide Generation Via Query-Based Text Summarization (2021) 1405–1418. doi:10.18653/v1/2021. naacl-main.111. arXiv:2105.03664. [16] T.-J. Fu, W. Y. Wang, D. McDuf, Y. Song, DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents, AAAI Conf. on Artificial Intelligence 36 (2022) 634–642. [17] A. Shimada, F. Okubo, C. Yin, H. Ogata, Automatic Summarization of Lecture Slides for Enhanced Student Preview-Technical Report and User Study, IEEE Transactions on Learning Technologies 11 (2018) 165–178. [18] A. Fiallos, X. Ochoa, Semi-Automatic Generation of Intelligent Curricula to Facilitate Learning Analytics, International Conference on Learning Analytics & Knowledge (LAK) (2019) 46–50. [19] C. Piech, J. Spencer, J. Huang, S. Ganguli, M. Sahami, L. Guibas, J. Sohl-Dickstein, Deep
Knowledge Tracing (2015) 1–12. arXiv:1506.05908. [20] H. Nakagawa, Y. Iwasawa, Y. Matsuo, Graph-based Knowledge Tracing: Modeling Student Proficiency Using Graph Neural Network, International Conference on Learning Representations (2019).