Understanding students' behavior and performance in the class is the primary concern for the teachers [1]. In face-to-face classes, teachers closely monitor students' behavior, concentration, and engagement relatively quickly, as they can see them sitting before them. However, those behaviors are challenging to monitor in online classes as the teachers cannot see the students. Hence, understanding students' behavior, engagement, and concentration levels could be tedious and directly affect lecturing. Furthermore, situations such as not concentrating, playing mobile phones, gaming, doing off-task activities, moving around, and sleeping while listening with a hands-free are complex to control by the teacher without any technological support. In those cases, teachers need technological assistance in controlling the class activities and evaluating the students accurately, as the instructor might be busy delivering lectures.
Learning analytics is a fast-growing area that focuses on measuring, collecting, analyzing, and reporting data associated with students' learning and their environment. So far, many learning analytics applications have been proposed and evaluated in education. A learning analytics dashboard (LAD) is a typical example of a learning analytics intervention that visualizes various actionable insights about students learning behaviors to empower teachers to make informed decisions about the learning process. For this reason, many learning analytics dashboards as innovative learning analytics products are used in higher education. Many higher educational institutions rely on teacher-facing learning analytics dashboards to improve lecture quality. For instance, to maximize student retention rates by identifying at-risk students as early as possible and initiating quick interventions by the institutions. Classis learning analytics dashboards, such as Early Alert Indicators (EAI) [2], Canvas Discussion Analytics Dashboard (CADA) [3], and Social Network Analysis Pedagogical Platform (SNAPP) [4] dashboards generate actionable insights based on the student's interaction data with the learning management systems, while a few used emotional data for analysis. For example, an Early Alert Indicator (EAI) dashboard is developed at Open University (OU) to identify students at risk and inform teachers who can proactively intervene [2]. This dashboard informs teachers about [5]: When did the tutors use the LA in the EAI dashboard? Which types of LA did the tutors use in the EAI dashboard? Why and how did the tutors use the LA in the EAI dashboard? What contribution did the EAI dashboard make to how tutors supported their students? However, the EAI dashboard does not reveal students' affective states. We address this as a limitation of classic dashboards as they only use LMS-produced data.
In learning and teaching, emotions are identified by the patterns in their ability to think, respond, communicate, or behave in an educational context. Emotions are an integrated part of learning. Emotions can reveal many hidden factors about learning and have the potential to provide actionable insights to teachers to increase the quality of teaching.
This study uses multimodal learning analytics methodologies to introduce a classroom monitoring system for teachers that visualizes students' affective and emotional states while taking online classes. Using this real-time system, teachers of online lectures could understand their students' affective states, engagement, and concentration levels while conducting online lectures.
In learning analytics, dashboards (known as learning analytics dashboards or LADs in literature) are used to monitor classrooms. By now, together with learning analytics dashboards, several classroom monitoring systems have been developed to improve the quality of teaching and learning. This section discusses some classroom monitoring systems found in recent learning analytics literature.
Smart Online Class Monitoring System (SOCMS) is a classroom monitoring system that aims to understand students' non-responsive behavior during an ongoing online class and reports to the lecturer [6]. In this system, the Fisher-Yates algorithm is used to create a suggestion list for the teachers generated during online classes, aiming to cover all the students in a random but non-repetitive pattern. This system has been implemented on a group of undergraduate course students and found effective results [6]. One of the limitations of this study is that the system does not analyze emotional attributes in classroom monitoring.
An IOT-supported classroom monitoring system is developed to perform classroom monitoring tasks such as taking attendance, identifying entering and leaving activities, and analyzing the student's concentration level [1]. This framework uses face recognition, motion analysis, and behavior understanding modules to reveal insights on students. One limitation is that the system's UI seems complicated to operate and needs a dashboard.
A classroom monitoring system based on facial expression recognition [7] has developed to identify eight kinds of emotions from the students, namely, positive emotions: "happy"; negative emotions: "disgust, Sadness, doubts, contempt, anger"; neutral emotion: "focus, surprise." However, this system is not applied to educational settings. An automated attendance system with audio output in lectures or classroom sessions by which the lecturer or faculty can record students' attendance is found in the literature [8]. Although this system applies facial recognition for face matching, it does not have the function of detecting emotional data by which students' affective states could be understood.
In conclusion, given the limitations in existing studies, it is essential to have a new classroom monitoring system that could leverage emotional data to reveal students' affective states, engagement, concentration levels, and ideal times for the teacher to intervene. Therefore, in this paper, we propose a classroom monitoring system. This classroom monitoring system is a new development to the MOEMO learning analytics framework [9,10].
In the next section, we briefly discuss the functions and features of the proposed classroom monitoring system.
MOEMO platform is an LMS-independent learning analytics framework. The platform could understand students' motions to detect affective aspects such as engagement and concentration from the emotional data. This platform reads data from the camera function. The camera could be the default web camera of a laptop; hence, the users do not need to prepare additional cameras to use this system. In Figure 1, we present the overview of the platform.
In addition, it clusters the highly-engaged and disengaged students in real-time. The system determines online learners' five types of engagement ("strong engagement", "high engagement", "medium engagement", "low engagement", and "disengagement") and two types of concentration levels ("focused" and "distracted") [9].
In developing the system [9], MTCNN, Mini-Xception, HaarCascade, and Pnp algorithms are used for face detection, emotion detection, eye detection, and eye gaze estimation, respectively. Matplotlib and Plotly are used to produce the visualizations on the teacher-facing dashboard. Data is processed and analyzed using pandas. Multiple web-service applications are used to produce the after-class reports on the teacher's window offline and in real-time.
The dashboard has an engagement prediction panel, a concentration prediction panel, a classroom overview panel, a notification panel, an engagement summarization graph, a concentration summarization graph, clustering visualization, and emotion distribution graphs. Also, it provides information on video processing duration, video quality check report, frame analysis results, and an after-class report generation. The features provide actionable insights to the teachers to understand the classroom and make informed decisions about their teaching. The dashboard is also deployed to support insight into learning and emotional data.
Table 1 summarizes the functionalities of the teacher-facing dashboard and the frequency of updating them.
The analytics of the MOEMO platform provide many insights into learning and teaching to the teacher by analyzing students' affective states. For example, Figure 2, which is the after-class report for the teacher to monitor the class, provides the flowing insight, are: The third pie chart shows the visualization of overall concentration during the class Figure 3 shows the learning analysis of an individual student. The MOEMO platform can check each student's learning analysis by clicking on the name shown on the system-generated report. This gives the teacher more insights into a student, including when to intervene. For example, the student in Figure 3 was disengaged between 4:13 and 4:36 minutes. The system assumes that this 25 second of disengagement is long, and the teacher needs to interview to engage the student. The system assumes that 25 seconds of disengagement is long, so a pop-up notification is shown on the teacher's screen so that the teacher can intervene. The same student was found to be disengaged between 6:31 and 6:54 minutes. With this classroom monitoring system, a teacher could understand and monitor the class.
In learning analytics, developing classroom monitoring systems using emotional data for the decision-making support of learning behavior is essential. Classroom monitoring systems help teachers take attendance by face recognition and understanding students' interest in the lectures by identifying their emotions and sitting postures. This type of sophisticated technology is also used for automatic note-taking during the class by using audio-to-text converters such as pyaudio and Halo. [11]. Our classroom monitoring technology that leverages students' emotional data through facial expression analysis. The analytics of this system can identify seven types of affective states, five types of engagement, two types of concentration, identify clusters of students, and create an after-class report for the teacher. A teacher can use these actionable insights to monitor the class and decide when to intervene.
This project is partially supported by JSPS's Fund for the Promotion of Joint International Research no. 21KK0184.