=Paper=
{{Paper
|id=Vol-3667/DC-LAK24-paper-10
|storemode=property
|title=Classroom Monitoring using Emotional Data
|pdfUrl=https://ceur-ws.org/Vol-3667/DC-LAK24-paper-10.pdf
|volume=Vol-3667
|authors=Mohammad Nehal Hasnine,Ho Tan Nguyen,Gökhan Akçapınar,Ryugo Morita,Hiroshi Ueda
|dblpUrl=https://dblp.org/rec/conf/lak/HasnineNAMU24
}}
==Classroom Monitoring using Emotional Data==
https://ceur-ws.org/Vol-3667/DC-LAK24-paper-10.pdf
Classroom Monitoring using Emotional Data
Mohammad Nehal Hasnine1, Ho Tan Nguyen1, Gökhan Akçapınar2, Ryugo Morita1 and
Hiroshi Ueda1
1 Research Center for Computing and Multimedia Studies, Hosei University, Kajinocho 3-7-2, Tokyo, Japan
2 Department of Computer Education and Instructional Technology, Hacettepe University, 06800 Beytepe, Ankara,
Turkey
Abstract
Emotions are an integrated part of learning. Emotions can reveal many hidden factors about learning
and have the potential to provide actionable insights to teachers to increase the quality of teaching. This
study uses multimodal learning analytics methodologies to introduce a classroom monitoring system
for teachers teaching online courses. The system is an integrated component of the MOEMO (Motion and
Emotion) learning analytics framework that visualizes students' affective and emotional states while
taking online classes. Using this classroom monitoring system, a teacher could understand the moments
when students were disengaged so that the teacher could intervene to make those disengaged students
engaged. The system reports actionable insights on students' engagements and concentrations to the
teacher. 1
Keywords
Classroom monitoring, emotion analysis, engagement, affective states, teacher-facing dashboard, MMLA
1. Introduction
Understanding students' behavior and performance in the class is the primary concern for the
teachers [1]. In face-to-face classes, teachers closely monitor students' behavior, concentration,
and engagement relatively quickly, as they can see them sitting before them. However, those
behaviors are challenging to monitor in online classes as the teachers cannot see the students.
Hence, understanding students' behavior, engagement, and concentration levels could be tedious
and directly affect lecturing. Furthermore, situations such as not concentrating, playing mobile
phones, gaming, doing off-task activities, moving around, and sleeping while listening with a
hands-free are complex to control by the teacher without any technological support. In those
cases, teachers need technological assistance in controlling the class activities and evaluating the
students accurately, as the instructor might be busy delivering lectures.
Learning analytics is a fast-growing area that focuses on measuring, collecting, analyzing, and
reporting data associated with students’ learning and their environment. So far, many learning
analytics applications have been proposed and evaluated in education. A learning analytics
dashboard (LAD) is a typical example of a learning analytics intervention that visualizes various
actionable insights about students learning behaviors to empower teachers to make informed
decisions about the learning process. For this reason, many learning analytics dashboards as
innovative learning analytics products are used in higher education. Many higher educational
institutions rely on teacher-facing learning analytics dashboards to improve lecture quality. For
instance, to maximize student retention rates by identifying at-risk students as early as possible
and initiating quick interventions by the institutions. Classis learning analytics dashboards, such
as Early Alert Indicators (EAI) [2], Canvas Discussion Analytics Dashboard (CADA) [3], and Social
Network Analysis Pedagogical Platform (SNAPP) [4] dashboards generate actionable insights
based on the student’s interaction data with the learning management systems, while a few used
emotional data for analysis. For example, an Early Alert Indicator (EAI) dashboard is developed
at Open University (OU) to identify students at risk and inform teachers who can proactively
intervene [2]. This dashboard informs teachers about [5]: When did the tutors use the LA in the
EAI dashboard? Which types of LA did the tutors use in the EAI dashboard? Why and how did the
tutors use the LA in the EAI dashboard? What contribution did the EAI dashboard make to how
LAK-WS 2024: Joint Proceedings of LAK 2024 Workshops, March 18–19, Kyoto, Japan
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
tutors supported their students? However, the EAI dashboard does not reveal students’ affective
states. We address this as a limitation of classic dashboards as they only use LMS-produced data.
In learning and teaching, emotions are identified by the patterns in their ability to think,
respond, communicate, or behave in an educational context. Emotions are an integrated part of
learning. Emotions can reveal many hidden factors about learning and have the potential to
provide actionable insights to teachers to increase the quality of teaching.
This study uses multimodal learning analytics methodologies to introduce a classroom
monitoring system for teachers that visualizes students' affective and emotional states while
taking online classes. Using this real-time system, teachers of online lectures could understand
their students' affective states, engagement, and concentration levels while conducting online
lectures.
2. Literature review
In learning analytics, dashboards (known as learning analytics dashboards or LADs in literature)
are used to monitor classrooms. By now, together with learning analytics dashboards, several
classroom monitoring systems have been developed to improve the quality of teaching and
learning. This section discusses some classroom monitoring systems found in recent learning
analytics literature.
Smart Online Class Monitoring System (SOCMS) is a classroom monitoring system that aims
to understand students’ non-responsive behavior during an ongoing online class and reports to
the lecturer [6]. In this system, the Fisher-Yates algorithm is used to create a suggestion list for
the teachers generated during online classes, aiming to cover all the students in a random but
non-repetitive pattern. This system has been implemented on a group of undergraduate course
students and found effective results [6]. One of the limitations of this study is that the system does
not analyze emotional attributes in classroom monitoring.
An IOT-supported classroom monitoring system is developed to perform classroom
monitoring tasks such as taking attendance, identifying entering and leaving activities, and
analyzing the student's concentration level [1]. This framework uses face recognition, motion
analysis, and behavior understanding modules to reveal insights on students. One limitation is
that the system's UI seems complicated to operate and needs a dashboard.
A classroom monitoring system based on facial expression recognition [7] has developed to
identify eight kinds of emotions from the students, namely, positive emotions: "happy"; negative
emotions: "disgust, Sadness, doubts, contempt, anger"; neutral emotion: "focus, surprise."
However, this system is not applied to educational settings.
An automated attendance system with audio output in lectures or classroom sessions by which
the lecturer or faculty can record students' attendance is found in the literature [8]. Although this
system applies facial recognition for face matching, it does not have the function of detecting
emotional data by which students' affective states could be understood.
In conclusion, given the limitations in existing studies, it is essential to have a new classroom
monitoring system that could leverage emotional data to reveal students' affective states,
engagement, concentration levels, and ideal times for the teacher to intervene. Therefore, in this
paper, we propose a classroom monitoring system. This classroom monitoring system is a new
development to the MOEMO learning analytics framework [9, 10].
In the next section, we briefly discuss the functions and features of the proposed classroom
monitoring system.
3. MOEMO platform
3.1. About MOEMO platform
MOEMO platform is an LMS-independent learning analytics framework. The platform could
understand students’ motions to detect affective aspects such as engagement and concentration
from the emotional data. This platform reads data from the camera function. The camera could
be the default web camera of a laptop; hence, the users do not need to prepare additional cameras
to use this system. In Figure 1, we present the overview of the platform.
Learning
Analytics
Dashboard
Lecturer,
Administrator MOEMO Database
Analytics Engine
Online
class
Figure 1: The MOEMO platform
In addition, it clusters the highly-engaged and disengaged students in real-time. The system
determines online learners’ five types of engagement (“strong engagement”, “high engagement”,
“medium engagement”, “low engagement”, and “disengagement”) and two types of concentration
levels (“focused” and “distracted”) [9].
In developing the system [9], MTCNN, Mini-Xception, HaarCascade, and Pnp algorithms are
used for face detection, emotion detection, eye detection, and eye gaze estimation, respectively.
Matplotlib and Plotly are used to produce the visualizations on the teacher-facing dashboard.
Data is processed and analyzed using pandas. Multiple web-service applications are used to
produce the after-class reports on the teacher’s window offline and in real-time.
3.2. Features and functions
The dashboard has an engagement prediction panel, a concentration prediction panel, a
classroom overview panel, a notification panel, an engagement summarization graph, a
concentration summarization graph, clustering visualization, and emotion distribution graphs.
Also, it provides information on video processing duration, video quality check report, frame
analysis results, and an after-class report generation. The features provide actionable insights to
the teachers to understand the classroom and make informed decisions about their teaching. The
dashboard is also deployed to support insight into learning and emotional data.
Table 1 summarizes the functionalities of the teacher-facing dashboard and the frequency of
updating them.
Table 1
Features and functions of the dashboard [9]
Function Update Interval Description
Engagement prediction Realtime Overall engagement rate (range 0 to 100%)
Concentration prediction Realtime Overall concentration rate (range 0 to 100%)
Classroom overview Once (before class) Number of students in the class
Processing duration Realtime Total processing time of the lecture video
Notification panel Realtime Intervention
Engagement graph Realtime Engagement level in each minute
Concentration graph Realtime Concentration details
Cluster panel Realtime Top engaged and disengaged students
Emotion distribution Realtime Overall emotional rate of the class
Frame analysis Realtime Shows the affective states
After class report Once (after class) Report provides a detailed on affective states
4. Classroom monitoring using the teacher-facing dashboard
The analytics of the MOEMO platform provide many insights into learning and teaching to the
teacher by analyzing students’ affective states. For example, Figure 2, which is the after-class
report for the teacher to monitor the class, provides the flowing insight, are:
• Course information, including Course ID, Course name, and teacher’s name
• Video quality check and calibration
• Class information, including the number of students, class time, and intervals
• Overall engagement, emotional, and concentration level during the class (0 to 100%)
• The bar graph shows the summary of engagement level (every 30 seconds)
• The multi-axis line chart the summary of concentration level (in every 30 seconds)
• The first pie chart shows the distribution of engagement during the class
• The second pie chart shows the distribution of students’ affective states during the class
• The third pie chart shows the visualization of overall concentration during the class
Figure 3 shows the learning analysis of an individual student. The MOEMO platform can check
each student’s learning analysis by clicking on the name shown on the system-generated report.
This gives the teacher more insights into a student, including when to intervene. For example, the
student in Figure 3 was disengaged between 4:13 and 4:36 minutes. The system assumes that
this 25 second of disengagement is long, and the teacher needs to interview to engage the student.
The system assumes that 25 seconds of disengagement is long, so a pop-up notification is shown
on the teacher’s screen so that the teacher can intervene. The same student was found to be
disengaged between 6:31 and 6:54 minutes. With this classroom monitoring system, a teacher
could understand and monitor the class.
Figure 2: Classroom summary in the monitoring process
5. Discussion
In learning analytics, developing classroom monitoring systems using emotional data for the
decision-making support of learning behavior is essential. Classroom monitoring systems help
teachers take attendance by face recognition and understanding students' interest in the lectures
by identifying their emotions and sitting postures. This type of sophisticated technology is also
used for automatic note-taking during the class by using audio-to-text converters such as py-
audio and Halo.
Figure 3: Learning analyisis of a student
So far, artificial intelligence (AI)-based behavior recognition techniques are integrated with
classroom monitoring environments to evaluate students' attention and engagement during
classroom sessions [11]. Our classroom monitoring technology that leverages students'
emotional data through facial expression analysis. The analytics of this system can identify seven
types of affective states, five types of engagement, two types of concentration, identify clusters of
students, and create an after-class report for the teacher. A teacher can use these actionable
insights to monitor the class and decide when to intervene.
Acknowledgements
This project is partially supported by JSPS’s Fund for the Promotion of Joint International
Research no. 21KK0184.
References
[1] J. H. Lim, E. Y. Teh, M. H. Geh, and C. H. Lim, “Automated classroom monitoring with connected
visioning system,” in 2017 Asia-Pacific Signal and Information Processing Association
Annual Summit and Conference (APSIPA ASC), IEEE, 2017, pp. 386–393. Accessed: Dec. 13,
2023.
[2] A. Boroowa and C. Herodotou, “Learning Analytics in Open and Distance Higher Education:
The Case of the Open University UK,” in Learning Analytics in Open and Distributed Learning,
P. Prinsloo, S. Slade, and M. Khalil, Eds., in SpringerBriefs in Education. , Singapore: Springer
Nature Singapore, 2022, pp. 47–62. doi: 10.1007/978-981-19-0786-9_4.
[3] R. Kaliisa and J. A. Dolonen, “CADA: a teacher-facing learning analytics dashboard to foster
teachers’ awareness of students’ participation and discourse patterns in online discussions,”
Tech Know Learn, vol. 28, no. 3, pp. 937–958, Sep. 2023, doi: 10.1007/s10758-022-09598-7.
[4] F. Froehlich, “Social network analysis as a progressive tool for learning analytics: a sample of
teacher education students’ course interactions with Threadz on the canvas platform,” PhD
Thesis, University of British Columbia, 2020. Accessed: Nov. 20, 2023.
[5] T. Olney, S. Walker, C. Wood, and A. Clarke, “Are We Living In LA (P) LA Land? Reporting on
the Practice of 30 STEM Tutors in their Use of a Learning Analytics Implementation at the
Open University,” Journal of Learning Analytics, vol. 8, no. 3, pp. 45–59, 2021.
[6] M. Sharma, “SOCMS : Smart online class monitoring system,” Journal of Statistics and
Management Systems, vol. 24, no. 2, pp. 251–261, Feb. 2021, doi:
10.1080/09720510.2020.1833466.
[7] B. Zhang, D. Wei, Q. Zhang, W. Si, X. Li, and Q. Zhu, “Classroom monitoring system based on
facial expression recognition,” in 2021 20th International Symposium on Distributed
Computing and Applications for Business Engineering and Science (DCABES), IEEE, 2021, pp.
108–111.
[8] S. Poornima, N. Sripriya, B. Vijayalakshmi, and P. Vishnupriya, “Attendance monitoring
system using facial recognition with audio output and gender classification,” in 2017
International Conference on Computer, Communication and Signal Processing (ICCCSP),
Chennai, India: IEEE, Jan. 2017, pp. 1–5. doi: 10.1109/ICCCSP.2017.7944103.
[9] M. N. Hasnine, H. T. Nguyen, T. T. T. Tran, H. T. Bui, G. Akçapınar, and H. Ueda, “A Real-Time
Learning Analytics Dashboard for Automatic Detection of Online Learners’ Affective States,”
Sensors, vol. 23, no. 9, p. 4243, 2023.
[10] M. N. Hasnine, H. T. Bui, T. T. T. Tran, H. T. Nguyen, G. Akçapınar, and H. Ueda, “Students’
emotion extraction and visualization for engagement detection in online learning,” Procedia
Computer Science, vol. 192, pp. 3423–3431, 2021.
[11] Z. Trabelsi, F. Alnajjar, M. M. A. Parambil, M. Gochoo, and L. Ali, “Real-Time Attention
Monitoring System for Classroom: A Deep Learning Approach for Student’s Behavior
Recognition,” Big Data and Cognitive Computing, vol. 7, no. 1, p. 48, 2023.