Classroom Monitoring using Emotional Data Mohammad Nehal Hasnine1, Ho Tan Nguyen1, Gökhan Akçapınar2, Ryugo Morita1 and Hiroshi Ueda1 1 Research Center for Computing and Multimedia Studies, Hosei University, Kajinocho 3-7-2, Tokyo, Japan 2 Department of Computer Education and Instructional Technology, Hacettepe University, 06800 Beytepe, Ankara, Turkey Abstract Emotions are an integrated part of learning. Emotions can reveal many hidden factors about learning and have the potential to provide actionable insights to teachers to increase the quality of teaching. This study uses multimodal learning analytics methodologies to introduce a classroom monitoring system for teachers teaching online courses. The system is an integrated component of the MOEMO (Motion and Emotion) learning analytics framework that visualizes students' affective and emotional states while taking online classes. Using this classroom monitoring system, a teacher could understand the moments when students were disengaged so that the teacher could intervene to make those disengaged students engaged. The system reports actionable insights on students' engagements and concentrations to the teacher. 1 Keywords Classroom monitoring, emotion analysis, engagement, affective states, teacher-facing dashboard, MMLA 1. Introduction Understanding students' behavior and performance in the class is the primary concern for the teachers [1]. In face-to-face classes, teachers closely monitor students' behavior, concentration, and engagement relatively quickly, as they can see them sitting before them. However, those behaviors are challenging to monitor in online classes as the teachers cannot see the students. Hence, understanding students' behavior, engagement, and concentration levels could be tedious and directly affect lecturing. Furthermore, situations such as not concentrating, playing mobile phones, gaming, doing off-task activities, moving around, and sleeping while listening with a hands-free are complex to control by the teacher without any technological support. In those cases, teachers need technological assistance in controlling the class activities and evaluating the students accurately, as the instructor might be busy delivering lectures. Learning analytics is a fast-growing area that focuses on measuring, collecting, analyzing, and reporting data associated with students’ learning and their environment. So far, many learning analytics applications have been proposed and evaluated in education. A learning analytics dashboard (LAD) is a typical example of a learning analytics intervention that visualizes various actionable insights about students learning behaviors to empower teachers to make informed decisions about the learning process. For this reason, many learning analytics dashboards as innovative learning analytics products are used in higher education. Many higher educational institutions rely on teacher-facing learning analytics dashboards to improve lecture quality. For instance, to maximize student retention rates by identifying at-risk students as early as possible and initiating quick interventions by the institutions. Classis learning analytics dashboards, such as Early Alert Indicators (EAI) [2], Canvas Discussion Analytics Dashboard (CADA) [3], and Social Network Analysis Pedagogical Platform (SNAPP) [4] dashboards generate actionable insights based on the student’s interaction data with the learning management systems, while a few used emotional data for analysis. For example, an Early Alert Indicator (EAI) dashboard is developed at Open University (OU) to identify students at risk and inform teachers who can proactively intervene [2]. This dashboard informs teachers about [5]: When did the tutors use the LA in the EAI dashboard? Which types of LA did the tutors use in the EAI dashboard? Why and how did the tutors use the LA in the EAI dashboard? What contribution did the EAI dashboard make to how LAK-WS 2024: Joint Proceedings of LAK 2024 Workshops, March 18–19, Kyoto, Japan © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings tutors supported their students? However, the EAI dashboard does not reveal students’ affective states. We address this as a limitation of classic dashboards as they only use LMS-produced data. In learning and teaching, emotions are identified by the patterns in their ability to think, respond, communicate, or behave in an educational context. Emotions are an integrated part of learning. Emotions can reveal many hidden factors about learning and have the potential to provide actionable insights to teachers to increase the quality of teaching. This study uses multimodal learning analytics methodologies to introduce a classroom monitoring system for teachers that visualizes students' affective and emotional states while taking online classes. Using this real-time system, teachers of online lectures could understand their students' affective states, engagement, and concentration levels while conducting online lectures. 2. Literature review In learning analytics, dashboards (known as learning analytics dashboards or LADs in literature) are used to monitor classrooms. By now, together with learning analytics dashboards, several classroom monitoring systems have been developed to improve the quality of teaching and learning. This section discusses some classroom monitoring systems found in recent learning analytics literature. Smart Online Class Monitoring System (SOCMS) is a classroom monitoring system that aims to understand students’ non-responsive behavior during an ongoing online class and reports to the lecturer [6]. In this system, the Fisher-Yates algorithm is used to create a suggestion list for the teachers generated during online classes, aiming to cover all the students in a random but non-repetitive pattern. This system has been implemented on a group of undergraduate course students and found effective results [6]. One of the limitations of this study is that the system does not analyze emotional attributes in classroom monitoring. An IOT-supported classroom monitoring system is developed to perform classroom monitoring tasks such as taking attendance, identifying entering and leaving activities, and analyzing the student's concentration level [1]. This framework uses face recognition, motion analysis, and behavior understanding modules to reveal insights on students. One limitation is that the system's UI seems complicated to operate and needs a dashboard. A classroom monitoring system based on facial expression recognition [7] has developed to identify eight kinds of emotions from the students, namely, positive emotions: "happy"; negative emotions: "disgust, Sadness, doubts, contempt, anger"; neutral emotion: "focus, surprise." However, this system is not applied to educational settings. An automated attendance system with audio output in lectures or classroom sessions by which the lecturer or faculty can record students' attendance is found in the literature [8]. Although this system applies facial recognition for face matching, it does not have the function of detecting emotional data by which students' affective states could be understood. In conclusion, given the limitations in existing studies, it is essential to have a new classroom monitoring system that could leverage emotional data to reveal students' affective states, engagement, concentration levels, and ideal times for the teacher to intervene. Therefore, in this paper, we propose a classroom monitoring system. This classroom monitoring system is a new development to the MOEMO learning analytics framework [9, 10]. In the next section, we briefly discuss the functions and features of the proposed classroom monitoring system. 3. MOEMO platform 3.1. About MOEMO platform MOEMO platform is an LMS-independent learning analytics framework. The platform could understand students’ motions to detect affective aspects such as engagement and concentration from the emotional data. This platform reads data from the camera function. The camera could be the default web camera of a laptop; hence, the users do not need to prepare additional cameras to use this system. In Figure 1, we present the overview of the platform. Learning Analytics Dashboard Lecturer, Administrator MOEMO Database Analytics Engine Online class Figure 1: The MOEMO platform In addition, it clusters the highly-engaged and disengaged students in real-time. The system determines online learners’ five types of engagement (“strong engagement”, “high engagement”, “medium engagement”, “low engagement”, and “disengagement”) and two types of concentration levels (“focused” and “distracted”) [9]. In developing the system [9], MTCNN, Mini-Xception, HaarCascade, and Pnp algorithms are used for face detection, emotion detection, eye detection, and eye gaze estimation, respectively. Matplotlib and Plotly are used to produce the visualizations on the teacher-facing dashboard. Data is processed and analyzed using pandas. Multiple web-service applications are used to produce the after-class reports on the teacher’s window offline and in real-time. 3.2. Features and functions The dashboard has an engagement prediction panel, a concentration prediction panel, a classroom overview panel, a notification panel, an engagement summarization graph, a concentration summarization graph, clustering visualization, and emotion distribution graphs. Also, it provides information on video processing duration, video quality check report, frame analysis results, and an after-class report generation. The features provide actionable insights to the teachers to understand the classroom and make informed decisions about their teaching. The dashboard is also deployed to support insight into learning and emotional data. Table 1 summarizes the functionalities of the teacher-facing dashboard and the frequency of updating them. Table 1 Features and functions of the dashboard [9] Function Update Interval Description Engagement prediction Realtime Overall engagement rate (range 0 to 100%) Concentration prediction Realtime Overall concentration rate (range 0 to 100%) Classroom overview Once (before class) Number of students in the class Processing duration Realtime Total processing time of the lecture video Notification panel Realtime Intervention Engagement graph Realtime Engagement level in each minute Concentration graph Realtime Concentration details Cluster panel Realtime Top engaged and disengaged students Emotion distribution Realtime Overall emotional rate of the class Frame analysis Realtime Shows the affective states After class report Once (after class) Report provides a detailed on affective states 4. Classroom monitoring using the teacher-facing dashboard The analytics of the MOEMO platform provide many insights into learning and teaching to the teacher by analyzing students’ affective states. For example, Figure 2, which is the after-class report for the teacher to monitor the class, provides the flowing insight, are: • Course information, including Course ID, Course name, and teacher’s name • Video quality check and calibration • Class information, including the number of students, class time, and intervals • Overall engagement, emotional, and concentration level during the class (0 to 100%) • The bar graph shows the summary of engagement level (every 30 seconds) • The multi-axis line chart the summary of concentration level (in every 30 seconds) • The first pie chart shows the distribution of engagement during the class • The second pie chart shows the distribution of students’ affective states during the class • The third pie chart shows the visualization of overall concentration during the class Figure 3 shows the learning analysis of an individual student. The MOEMO platform can check each student’s learning analysis by clicking on the name shown on the system-generated report. This gives the teacher more insights into a student, including when to intervene. For example, the student in Figure 3 was disengaged between 4:13 and 4:36 minutes. The system assumes that this 25 second of disengagement is long, and the teacher needs to interview to engage the student. The system assumes that 25 seconds of disengagement is long, so a pop-up notification is shown on the teacher’s screen so that the teacher can intervene. The same student was found to be disengaged between 6:31 and 6:54 minutes. With this classroom monitoring system, a teacher could understand and monitor the class. Figure 2: Classroom summary in the monitoring process 5. Discussion In learning analytics, developing classroom monitoring systems using emotional data for the decision-making support of learning behavior is essential. Classroom monitoring systems help teachers take attendance by face recognition and understanding students' interest in the lectures by identifying their emotions and sitting postures. This type of sophisticated technology is also used for automatic note-taking during the class by using audio-to-text converters such as py- audio and Halo. Figure 3: Learning analyisis of a student So far, artificial intelligence (AI)-based behavior recognition techniques are integrated with classroom monitoring environments to evaluate students' attention and engagement during classroom sessions [11]. Our classroom monitoring technology that leverages students' emotional data through facial expression analysis. The analytics of this system can identify seven types of affective states, five types of engagement, two types of concentration, identify clusters of students, and create an after-class report for the teacher. A teacher can use these actionable insights to monitor the class and decide when to intervene. Acknowledgements This project is partially supported by JSPS’s Fund for the Promotion of Joint International Research no. 21KK0184. References [1] J. H. Lim, E. Y. Teh, M. H. Geh, and C. H. Lim, “Automated classroom monitoring with connected visioning system,” in 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), IEEE, 2017, pp. 386–393. Accessed: Dec. 13, 2023. [2] A. Boroowa and C. Herodotou, “Learning Analytics in Open and Distance Higher Education: The Case of the Open University UK,” in Learning Analytics in Open and Distributed Learning, P. Prinsloo, S. Slade, and M. Khalil, Eds., in SpringerBriefs in Education. , Singapore: Springer Nature Singapore, 2022, pp. 47–62. doi: 10.1007/978-981-19-0786-9_4. [3] R. Kaliisa and J. A. Dolonen, “CADA: a teacher-facing learning analytics dashboard to foster teachers’ awareness of students’ participation and discourse patterns in online discussions,” Tech Know Learn, vol. 28, no. 3, pp. 937–958, Sep. 2023, doi: 10.1007/s10758-022-09598-7. [4] F. Froehlich, “Social network analysis as a progressive tool for learning analytics: a sample of teacher education students’ course interactions with Threadz on the canvas platform,” PhD Thesis, University of British Columbia, 2020. Accessed: Nov. 20, 2023. [5] T. Olney, S. Walker, C. Wood, and A. Clarke, “Are We Living In LA (P) LA Land? Reporting on the Practice of 30 STEM Tutors in their Use of a Learning Analytics Implementation at the Open University,” Journal of Learning Analytics, vol. 8, no. 3, pp. 45–59, 2021. [6] M. Sharma, “SOCMS : Smart online class monitoring system,” Journal of Statistics and Management Systems, vol. 24, no. 2, pp. 251–261, Feb. 2021, doi: 10.1080/09720510.2020.1833466. [7] B. Zhang, D. Wei, Q. Zhang, W. Si, X. Li, and Q. Zhu, “Classroom monitoring system based on facial expression recognition,” in 2021 20th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES), IEEE, 2021, pp. 108–111. [8] S. Poornima, N. Sripriya, B. Vijayalakshmi, and P. Vishnupriya, “Attendance monitoring system using facial recognition with audio output and gender classification,” in 2017 International Conference on Computer, Communication and Signal Processing (ICCCSP), Chennai, India: IEEE, Jan. 2017, pp. 1–5. doi: 10.1109/ICCCSP.2017.7944103. [9] M. N. Hasnine, H. T. Nguyen, T. T. T. Tran, H. T. Bui, G. Akçapınar, and H. Ueda, “A Real-Time Learning Analytics Dashboard for Automatic Detection of Online Learners’ Affective States,” Sensors, vol. 23, no. 9, p. 4243, 2023. [10] M. N. Hasnine, H. T. Bui, T. T. T. Tran, H. T. Nguyen, G. Akçapınar, and H. Ueda, “Students’ emotion extraction and visualization for engagement detection in online learning,” Procedia Computer Science, vol. 192, pp. 3423–3431, 2021. [11] Z. Trabelsi, F. Alnajjar, M. M. A. Parambil, M. Gochoo, and L. Ali, “Real-Time Attention Monitoring System for Classroom: A Deep Learning Approach for Student’s Behavior Recognition,” Big Data and Cognitive Computing, vol. 7, no. 1, p. 48, 2023.