The study of human engagement has significantly grown in recent years, particularly accelerated by the interaction with a growing number of smart computing machines [1, 2, 3]. Engagement estimation has significant importance across various domains of study, including advertising, marketing, human-computer interaction, and healthcare [4, 5, 6]. In this paper, we propose a real-time application that leverages a single RGB camera to capture user behavior. Our approach implements a novel method for estimating human engagement in real-world scenarios by extracting valuable information from the combination of facial expressions and gaze direction analysis. To acquire this data, we employed fast and accurate machine learning algorithms from the external library dlib, along with custom versions of Residual Neural Networks implemented from scratch. For training our models, we used a modified version of the DAiSEE dataset, a multi-label user afective states classification dataset that collects frontal videos of 112 diferent people recorded in real-world scenarios. In the absence of a baseline for comparing the results obtained by our application, we conducted experiments to assess its robustness in estimating engagement levels, leading to very encouraging results.
CEUR ceur-ws.org
In today’s rapidly evolving digital landscape, humanity interacts with a growing number of smart computing machines. This situation highlights the increasing trend of direct interactions with smart devices in various doand industrial applications. Despite this technological advancement, many devices lack algorithms capable of perceiving and responding to users’ attentional states.
tion of users’ engagement represents a huge usability improvement for Human-Computer Interaction (HCI) and Human-Robot Interaction (HRI) systems. This capability itive interactions, elevating system responsiveness, and enhancing overall user experience. In detail, engagement is a fundamental aspect of the human experience and nEvelop-O (C. Napoli) SYSYEM 2023: 9th Scholar’s Yearly Symposium of Technology, Engiment, focus, and interaction with their surroundings. For detecting it, facial expressions and gaze direction are crucial elements. In particular, the motion of the eyes is an important element to employ since it highlights the psychological mechanisms behind the human mind and naturally gravitates toward objects, people, or specific
In this paper, we propose a real-time application that combines gaze direction and face expression analysis to determine the engagement level of a person while interacting with intelligent systems. To achieve this, we defined two machine-learning pipelines leveraging RGB videos of a person interacting with the system. The first pipeline focuses on the user’s facial expressions analysis and employs a residual neural network architecture. The second pipeline concentrates on the user’s gaze direction estimation by combining pre-trained face and facial algorithm that we developed. Predictions of the user’s engagement level are ultimately calculated by merging the outputs of these two pipelines using a weighted linear
baseline for reference, our primary hurdle in handling this task involved creating an appropriate dataset for training our models. We opted to customize the Afective
hensive collection of multi-label videos designed for idenholds the promise of ushering in more advanced and intu- landmark detection models with a fast computer vision captures in depth the quality of an individual’s involve- interpolation formula. (S. Russo); 0000-0002-3336-5853 (C. Napoli)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License tifying user afective states. Given that the estimation of engagement levels necessitates both temporal and spatial ease of implementation and their proven efectiveness information, videos proved to be an ideal choice. How- in achieving accurate results. The prominence of such ever, to mitigate the high computational and resource techniques underscores the significance of visual cues, costs associated with treating videos as opposed to single particularly facial expressions, in gauging user engageimages, we implemented mandatory preprocessing steps ment levels during online interactions. to optimize memory and computational eficiency. The work presented in [18] investigates the suitability
Continuing to tackle the absence of a reference base- of three popular models: All-CNN [19], NiN-CNN [20],
line, we conducted experiments to assess the robustness and VD-CNN [21], along with a customized
Convoluand efectiveness of our application. This evaluation was tional Neural Network (CNN) [
In a comprehensive review treated in [17], the state- for predicting engagement level as a single scalar value of-the-art engagement detection techniques within the from RGB video streams captured by two cameras on the context of online learning are explored. The authors torso and head of an autonomous mobile robot, utilized classify existing methods into three primary categories: for tours at The Collection museum in Lincoln, UK. The automatic, semi-automatic, and manual. This classifica- model incorporates CNN and Long Short-Term Memory tion is based on the methods’ dependencies on learners’ (LSTM) [27] networks for video data analysis. Trainparticipation. Furthermore, each category is subdivided ing and evaluation of this regressor network were conbased on the type of input data used (e.g., audio, video, ducted using a dataset built from the recordings of the text). Among these, video-based methods in the auto- autonomous tour guide robot in the public museum. The matic category that leverage facial expressions emerge as dataset, manually annotated by three independent peothe most prevalent. These methods are favored for their ple, assigns scalar values in the range [0,1] to represent the user’s engagement level. The model demonstrates optimal engagement level predictions, achieving a Mean driver attention estimation, generating a heat map on the Squared Error (MSE) prediction loss of up to 0.126 on the images representing the road. The training dataset for test dataset. this model is constructed using virtual reality and a driv
The research conducted in [28] focused on investigat- ing simulator, incorporating images from the DR(eye)VE ing the Deep Facial Spatiotemporal Network (DFSTN). dataset [32] that depict the frontal view of the road obComprising two integral modules, namely the pretrained served by the driver. Experimental results showcase the SE-ResNet-50 (SENet) utilized for extracting facial spatial feasibility and superiority of the proposed method over features and an LSTM network with Global Attention existing approaches. for generating an attentional hidden state, the DFSTN synergistically captures both facial spatial and temporal information. This combined information is crucial 3. Dataset for enhancing engagement prediction performance. The model underwent testing on the DAiSEE dataset, achiev- The baseline dataset utilized for training our networks is ing an accuracy of 58.84%, showcasing its capability to a customized version of the Dataset for Afective States outperform numerous existing engagement prediction in E-Environments (DAiSEE), a large collection of multinetworks trained on the same dataset. label videos designed for identifying user afective states,
In [29], the estimation of human attention is based on including boredom, confusion, engagement, and frustrathe direction of the user’s face, considering five diferent tion in real-world scenarios. This dataset comprises 9068 directions: central, lateral to the left, lateral to the right, frontal view videos featuring 112 distinct individuals extowards up, and towards down. If the user looks in any di- pressing diferent levels of afective states. Each of these rection other than the central one, they are assumed to be states was manually ranked utilizing the following scale: distracted, with only the central gaze indicating full focus. very low, low, high, and very high. The authors created a dataset for training, comprising To create our customized dataset we initially modified 270 videos of approximately 20 seconds each from 18 dif- the task from which DAiSEE was originally built. We ferent individuals. To enhance data diversity, GAN-based switched from multi-label to multi-class classification, data augmentation techniques were employed to gener- associating only the level of engagement with each video ate new samples, diversifying somatic features in the and removing the labels for the other afective states. Exrecorded videos. Transfer Learning [30] was utilized to ample instances present inside our customized version construct the classifier. Specifically, a pre-trained VGG16 of the DAiSEE are displayed in Fig. 1. Subsequently, we [21] architecture was employed, with three additional divided the dataset into Training, Validation, and Test dense layers attached at the end for attention estimation. sets, with proportions of 60%, 20%, and 20%, respectively.
The approach presented in [31] ofers a novel method However, the resulting sets were highly unbalanced due for estimating driver attention. Departing from conven- to a small portion of videos classified as very low and tional methods that primarily focus on a single frontal low engagement. To address this issue, we downsampled scene image to analyze driver gaze or head pose, this the dataset in several ways to achieve a more balanced method introduces a dual-view scene. The additional distribution. First of all, redundancy in subjects was input data includes the frontal view of the car that the reduced by removing multiple videos of the same indidriver is observing. Specifically, the gaze direction is viduals. Then, through the use of a normal distribution, detected and transformed into a probability map of the we sampled the remaining data instances considering same size as the road view image, while salient features the frequency of labels in the videos with the following of temporal and spatial dimensions are extracted from the formula: road view images. These features are then combined and = ⋅ − ⋅ (1) fed into a multi-resolution neural network tasked with where is the reduction coeficient (that we have set to 0.25), represents the total number of samples in a given set, and denotes the frequency of label in that set. Table 1 displays information both before and after the preprocessing procedures on the dataset.
Since DAiSEE contains recordings captured in dynamic environments, each of these videos may present diferent and various disturbances, such as changing light conditions, visual occlusions, or unconstrained user motion.
To improve video quality, we applied manual color and intensity adjustments, focusing on enhancing contrast, brightness, and sharpness for optimal detail resolution.
Examples include adjustments to the gamma value, which efectively improves visibility in varying light conditions or exposure levels by normalizing image histograms, making videos more suitable for continuous analysis; Another example is the sharpness adjustments, which enhance fine details and edges, making facial features more prominent.
Despite these modifications, the dataset still demanded excessive memory requirements. Consequently, we opted for further adjustments. Considering that the majority of engagement information is likely derived from human expressions and gaze attention, with a smaller contribution from gestures, we decided to crop from each video only the user’s faces. This step also aimed to eliminate potential issues and biases arising from background data. The face cropping was automated using a pretrained Single Shot Multibox Detector (SSD) model from the Cafe framework [ 33].
To prevent the generation of unstable videos, we applied a stabilization algorithm (see pseudocode in Fig. 2) that facilitates smooth transitions between subsequently detected faces by stabilizing the position of their bounding boxes. At the start of each video, the size of the first detected face’s bounding box is stored. In all the following frames, this dimension is used to resize the bounding box of the subsequently detected faces. Additionally, if the distance between the centers of two consecutive detected faces is smaller than a manually adjusted threshold , the center of the newest detected face is replaced with the center of the bounding box of the previously detected face. Finally, each frame is converted to grayscale, and histogram equalization is applied to normalize the color information. frame is passed to the Gaze Direction Module and the Face Engagement Model. The predictions of these models are then combined to produce the actual output of our system. 4.
developed is illustrated in Fig. 3. Specifically, the system utilizes a single input video stream captured through a webcam reader module, implemented in the external library OpenCV [34], to feed two distinct models. The Face Engagement Model evaluates engagement based on facial expressions, while the Gaze Direction Model predicts engagement by analyzing where the user’s focal point. Lastly, the predictions of these two models are combined to derive the final engagement level estimated by our application.
level from frontal recording videos. We designed it as a
customized version of the ResNet architecture [
To address the human engagement level classification problem, our models needed to capture both spatial and temporal information. To enable the network to learn temporal information by analyzing multiple frames simultaneously in the same layer, we opted for 3D convolutional layers instead of the traditional 2D convolutional layers implemented in the original ResNet architecture. Learning temporal information is crucial for video analysis, as it allows the network to recognize complex patterns such as actions, gestures, or sequences of facial expressions. Due to this requirement, the model necessitates an initial period to populate a bufer of 60 frames, ensuring a suficient amount of data for the correct utilization of the 3D convolutional layers. Once the bufer reaches its capacity, the prediction of the engagement level can begin. Subsequently, with the arrival of each new frame, the bufer is updated, and the oldest frame is discarded. We tested three versions of this architecture, difering mainly in the depth and the internal structure of the convolutional block used. Specifically, we implemented the 18-, 34-, and 50-layer versions.
For all these architectures, we introduced 3D layers for batch normalization, max pooling, and average pooling. In detail, each convolutional block includes a batch normalization layer, and all convolutional layers employ the ReLU activation function. Only the last fully connected layer, responsible for the final prediction of the human’s engagement level, uses the Softmax activation function. During training, we utilized the He/Kaiming initialization technique [36], which initializes weights us ing a normal distribution with zero mean and a variance of 2 , where is the total number of inputs to the neuron.
employing the ReLU activation function, mitigating the vanishing or exploding gradient problem.
training function, opting for it over the conventional Categorical Cross-Entropy. The principal reason is that the
improvement on challenging examples, preventing the model from becoming overly confident with easy ones. We implemented the following Focal Loss formula: − (1 − ) ln( ) where represents the focusing parameter (typically a positive number) to be fine-tuned using cross-validation, and denotes the predicted probability of the correct class. Also, our training process incorporates early stopping with a learned patience value of 10 epochs, L2 regularization featuring a weight decay set to 1 −3, and an Adam Optimizer accompanied by a Learning Rate Scheduler [38] with a maximum learning rate of 1 − 4 and a Gradient Scaler to reduce the range of magnitudes in the gradients. All the implementation details of the tested models are reported in the Table 2. Following the training phase, we opted for the 50-layer version model, with a batch size equal to 16, as the engagement network for our application, as it demonstrated the highest accuracy among the tested versions.
This model is designed to extract attention information from a person’s gaze in frontal recording videos. The gaze direction provides valuable insights into a person’s engagement during a task. The complete workflow of this model is displayed in Fig. 4. To implement this model, we combined two pre-trained neural networks available in the dlib library [39].
The first network is the Face Cropping Model, a CNN trained for face detection in general images. It not only identifies faces but also provides their bounding box coordinates and converts the input image to grayscale. Although the use of this network may appear redundant considering the customized dataset that we have employed for training the engagement model, it plays a crucial role in the real-time application. Specifically, it crops faces from the live stream frames and passes these images to both the Engagement Model and the Facial Landmark Detector.
The Facial Landmark Detector, the second network x2 x2 x2 x2 Convolution
Block k=[3,3,3],f=128 k=[3,3,3],f=128 k=[3,3,3],f=128 k=[3,3,3],f=128 Convolution
Block k=[3,3,3],f=256 k=[3,3,3],f=256 k=[3,3,3],f=256 k=[3,3,3],f=256 Convolution
Block k=[3,3,3],f=512 k=[3,3,3],f=512 k=[3,3,3],f=512 k=[3,3,3],f=512 Average Pool
Dropout Linear
Output Size = 1x1x1
Rate = 0.4 Neurons = 1024 x3 x4 x6 x3 k=[1,1,1],f=64 k=[3,3,3],f=64 k=[1,1,1],f=256
x3 k=[1,1,1],f=128 k=[3,3,3],f=128 k=[1,1,1],f=512
x4 k=[1,1,1],f=256 k=[3,3,3],f=256 k=[1,1,1],f=1024
x6 k=[1,1,1],f=512 k=[3,3,3],f=512 k=[1,1,1],f=2048 x3 that we have employed from the dlib library, recognizes 68 2D facial landmarks (e.g., nose tip, corners of the mouth, and eyes) in a given face image. These facial landmarks serve two purposes: they are used to crop the eye regions based on the eye landmarks and to calculate the face orientation with respect to the vertical axis (yaw angle). This orientation is determined through the use of a vector starting from the midpoint between the eyes and terminating at the nose tip.
Estimating the focal point of the user is accomplished through the Gaze Direction Module, a simple computer vision pipeline. Initially, the eye landmarks outlining the eye contours are employed to create a mask that removes extraneous pixels from each cropped eye image. Subsequently, the Otsu’s method [40] is applied to automatically threshold the image, distinguishing between foreground (iris and pupil pixels) and background (sclera pixels).
The resulting image is then horizontally and vertically divided around its center to estimate the gaze direction. Both vertical and horizontal gaze directions are quantiifed as values within the range of [-1,1]. Regarding horizontal gaze direction, a value approaching -1 indicates the user is looking to the left, while a value approaching 1 suggests a rightward gaze. A value around 0 indicates the user is looking at the center of the screen. Similarly, for vertical gaze direction, a value nearing 1 signifies a downward gaze and a value nearing -1 indicates an upward gaze.
To compute these directions, the density of white pixels representing the sclera is analyzed. For each eye image, the total number of white pixels is calculated. If this value is zero, it implies incorrect eye detection, and the current frame is skipped. Otherwise, for each sub-image generated, the percentage of white pixels in relation to the total number of white pixels in the corresponding original eye image is calculated. Then, the percentages belonging to the same direction of both eyes are averaged (e.g., the percentage of white pixels in the left sub-image of the left eye is averaged with the percentage of white pixels in the left sub-image of the right eye). Finally, the diference between these averages produces the value within the range of [-1,1] described earlier.
To efectively use the estimated gaze direction, it’s crucial to consider the limits of the user’s field of view, which may vary based on the task. In our screen-based task implementation, we assume that a face orientation deviation exceeding 30 degrees from the camera-aligned orientation indicates the user is no longer looking at the monitor.
Initially, these limits are set at the task’s beginning and dynamically adjusted based on the user’s face position and orientation relative to the camera frame’s center. If the face orientation exceeds 20 degrees from the frontal position, the horizontal limits shift proportionally based on the sine of the face orientation. Updates related to face position involve calculating the distance between the face bounding box center and the frame center. If this distance exceeds one-sixth of the total frame dimension, the right and left limits are adjusted. The adjustment is determined by normalizing the distance between the face and frame centers between 0 and 0.5. If the face shifts to the right, the distance is subtracted from the limits; otherwise, it is added.
The engagement level, derived from the gaze direction, is within the range [0,1]. It is obtained by subtracting the sum of horizontal and vertical gaze errors from 1. A score of 1 indicates complete focus on the screen, with no gaze exceeding the defined limits. A score of 0 implies no face detection in the current frame. The closer the engagement level is to zero, the more the user surpasses the admissible field of view limits, indicating a lack of focus on the task. Specifically, when the gaze exceeds the limits, horizontal and vertical gaze errors are calculated as the diference in modulo between the estimated gaze direction and the corresponding limits.
To obtain the final detected engagement level, we combined the predictions from the Face Engagement Model and the Gaze Direction Model using a linear interpolation formula: ⋅ + (1 − ) ⋅ (3)
importance of the models’ predictions. In addition, to correctly apply this formula, the prediction of the Face Engagement Model needs to be converted from labels to a value within the range [0,1]. The conversion is performed according to the rules displayed in Table 3.
disparity between the predicted engagement scores and the ground truth values using two regression metrics: Mean Absolute Error (MAE) and Mean Absolute Percentage Error (MAPE). To facilitate the application of these 0.5, our system achieved an accuracy of approximately 58% (57.7%), closely aligning with the performance of state-of-the-art works in engagement level detection discussed in Section 2 that work with the original version of the DAiSEE.
level estimation by integrating two distinct machine learning pipelines focused on analyzing facial expressions and gaze direction. Noteworthy is our real-time application’s emphasis on cost-efectiveness and accessibility, achieved through the utilization of a single RGB Figure 5: Trend of the Mean Absolute Error (MAE) with vary- camera, fast and lightweight machine learning algoing values of the parameter in Eq. (3). rithms, and computationally eficient computer vision techniques.
In terms of system training, we customized the DAiSEE dataset to optimize memory usage, reduce class imbalance, mitigate bias introduced by repeated instances of the same individuals, and focus exclusively on facial cropping to eliminate potential background-related biases.
The achieved results underscore the potential of our system as a robust foundation, ofering a secure benchmark for the development of innovative applications integrating automatic user engagement recognition, thereby dynamically adapting to user interactions. This not only enhances overall usability but also heralds a new era in application interfaces, promising heightened levels of user experience and interaction.
Looking forward, future improvements to our system can be directed towards enhancing the accuracy, robustFigure 6: Trend of the Mean Absolute Percentage Error ness, and generalization capabilities by expanding the (MAPE) with varying values of the parameter in Eq. (3). dataset’s dimensions. This expansion may involve incorporating data from a more diverse group, encompassing individuals with varying demographic characteristics, metrics, we converted the engagement level labels associ- cultural backgrounds, and engagement patterns. ated with the samples in our customized dataset using the Also, exploring attention estimation in multi-face conconversion rules outlined in Table 3. This transformation texts, where multiple individuals are present simultaneefectively turned the multi-label class problem, designed ously, represents another intriguing avenue for future refor the DAiSEE dataset, into a regression problem. search. Lastly, a significant refinement to our application
During training, we experimented with diferent val- involves substituting the CNN layers in the Face Detecues for the parameter in Eq. (3) to maximize the sys- tion Model with Visual Transformers [41](ViTs), known tem’s accuracy. As illustrated in Figs. 5 and 6, the lowest for their excellence in image manipulation and longerror for both MAE and MAPE occurred when was set range dependency modeling compared to traditional conto 0.5. This indicates that both predictions from the Face volutional layers. This substitution could enhance the Engagement Model and the Gaze Direction Model carry precision of engagement level estimation from facial exequal importance and are essential for achieving accurate pressions, as diferent facial regions can be efectively predictions. combined at the same time.
Analysis of the scenarios where is 0 (using only the Face Model) or 1 (using only the Gaze Model) reveals References significantly higher errors in both performance metrics.
Independently, these predictions struggle to accurately gauge the user’s engagement level. With initialized to [1] G. Capizzi, G. L. Sciuto, C. Napoli, M. Woźniak,
G. Susi, A spiking neural network-based long