Monitoring the driver's attention is an important task to maintain the driver's safety. The estimation of the driver's gaze direction can help us to evaluate if the drivers are not focusing their attention on the street. For an evaluation of this type, comparing the inside view and outside scenery of the vehicle is essential, therefore we decided to create a specific dataset for this task. In this work, we realize a machine-learning-oriented approach to driver's attention evaluation using a coupled visual perception system. By analyzing the road and the driver's gaze simultaneously it is possible to understand if the driver is looking at the trafic signs detected. We evaluate if a determined Region Of Interest (ROI) contains a road sign or not through YOLOv8.
1. Introduction
the analysis of the vehicle cabin and the driver’s gaze is
conducted independently, without considering the
evaluArtificial Intelligence (AI) employed in assessing driver ation of the surrounding environment, road conditions,
attention within assisted driving scenarios is swiftly ad- and the driver’s reaction to specific events.
vancing, propelled by the evolution of autonomous ve- Several studies focus either on observing the driver’s
hicles and the integration of hybrid systems designed to behavior through internal vehicle cameras or analyzing
assist drivers. These systems encompass a range of func- external road conditions using external cameras and
sentionalities, including cruise control, lane-keeping assis- sors [
Literature primarily addresses driver attention by di- ment, machine learning is at the forefront of monitoring viding the internal and external components. Typically, and predicting environmental changes, enabling us to take timely action against potential threats [14, 15]. SoSYSYEM 2023: 9th Scholar’s Yearly Symposium of Technology, Engi- cial benefits are manifold, including improved healthcare neering and Mathematics, Rome, December 3-6, 2023 through predictive diagnostics, personalized education, $ fiani@diag.uniroma1.it (F. Fiani); ponzi@diag.uniroma1.it and efective public services, all contributing to an im(V. 0P0o0n9z-i0)0;0sa5m-0u39e6le-.7r0u1s9so(@F. uFniairnoim); a010.0i9t-(0S0.0R0u-2ss9o1)0-0273 (V. Ponzi); proved quality of life [16, 17, 18]. In the context of urban 0000-0002-9421-8566 (S. Russo) driving, machine learning is the driving force behind © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License autonomous vehicles [19]. These vehicles promise to sigCPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) nificantly reduce trafic accidents, improve trafic flow, and point of focus. Our novel approach involves the use and reduce carbon emissions, making our cities safer of a grid of nine cells to predict the Regions Of Interand more sustainable. Thus, machine learning is a key est (ROIs) of the driver’s gaze, as illustrated in Figure 1. enabler in our pursuit of a safer society. To achieve this, we employ a VGG16 network to extract
In this research, we merge various internal and exter- features from facial video frames, augmenting this innal techniques for gaze recognition and correlate them formation with head-pose data (i.e. roll, pitch, and yaw with external Regions Of Interest (ROIs) to develop an angles) to enhance gaze-position prediction [20, 21, 22]. easily applicable solution that comprehensively tackles The diference between tracking gaze-position when a the issue of driver attention. This approach holds sig- person is looking at a monitor and while they are driving nificant practical implications for everyday scenarios, is, in fact, substantial [21, 22]. When looking at a monitor, including: head movements are imperceptible, so the only discriminant is the position of the pupil. During driving, however, • Autonomous vehicle development: Understand- the driver tends to rotate their head to look at vehicles ing the driver’s focus during critical driving situ- and pedestrians or tilt it to see street names, signs, or ations, including the duration of their attention higher trafic lights. They also shift their gaze to look at to specific elements and their perception of ir- mirrors or to initiate a reverse maneuver. For these rearelevant factors, plays a pivotal role in the ad- sons, analyzing only pupil movement was insuficient for vancement of Advanced Driver Assistance Sys- our task and it was necessary to have additional informatem (ADAS) solutions. tion about head pose (rotation angles) and characteristics • Car crashes: Having information about driver of eyes or facial images, see Figure 2.
attention during a road accident could facilitate In addition to methodological research, another sigthe execution of investigations, checks, and in- nificant challenge we faced was sourcing an approprisurance procedures. By utilizing an afordable ate dataset for driver attention monitoring. We encouncamera system, video data on the driver involved tered existing datasets with comprehensive documentain the accident could be collected and provided tion of driver behavior, but they lacked corresponding to an application. real-world external observations. Furthermore, datasets • Emergency services: Emergency response vehi- focused solely on gaze analysis typically consisted of cles, including ambulances and fire trucks, of- images of individuals looking at points on a computer ten need to navigate through trafic quickly and screen, which did not align with our real-world driving safely. Driver attention monitoring systems can scenario. To address this gap, we decided to create our help emergency service providers ensure their own dataset, encompassing both internal and external drivers remain vigilant while responding to emer- videos captured during driving sessions. This approach gencies, minimizing the likelihood of accidents enabled our final application to process and correlate and delays. • Public transportation infrastructure: Driver attention monitoring systems can also be integrated into public transportation infrastructure, such as trafic lights and pedestrian crossings. By detecting instances of driver distraction or inattention, these systems can improve trafic flow and pedestrian safety, reducing the risk of accidents and congestion in urban areas.
To advance driver attention monitoring, we have directed our eforts towards computer vision-based methodologies, which are gaining traction over physiologybased approaches. Unlike physiological methods, in fact, vision-based techniques rely solely on cameras to observe and analyze driver behaviors, eliminating the need for intrusive devices such as eye-tracking glasses or brainwave recognition gadgets and consequently reducing the cost associated with experiments.
The most in-depth analysis in our work focused on ifnding the best method and features to extract from images to accurately determine the driver’s gaze direction information from multiple perspectives simultaneously. most classical metric being the gaze direction, generally
To train the two components of our application, we uti- assessed by analyzing facial features such as the face
lized two additional datasets. For the internal component, mesh. Other approaches are however available, for
inwhich involves predicting the driver’s gaze position, we stance, the position of the hands and arms, which can be
curated the HEAD-POSE dataset, featuring data from dif- used to assess whether the driver keeps their hands on
ferent subjects. Unlike many existing datasets that often the steering wheel or in other positions, such as holding
focus on a single subject, our dataset ofers a broader a phone [
This dataset comprises images sourced from the German Trafic Sign Detection Benchmark (GTSDB), available for download from Kaggle. of thousands of photos collected using a personal computer’s webcam, and then extract facial information from the given images to crop eye images and pass them to networks such as VGG-16 [22]. In addition, information related to head position (rotation angles - roll, pitch, yaw) can also be considered [20]. It is particularly of note that to improve regression on the viewpoint position it is fundamental to collect images from multiple subjects, in multiple vehicles, and under diferent weather conditions.
Finally, additional approaches make use of recordings
in simulated environments using various technologies,
from simulators to simple computer-played videos. For
example, the user’s gaze position can be recorded while
watching driving videos shortly before certain incidents,
in order to understand which objects the driver
(simulated in this case) would have focused on [
equipment setup as depicted in Figure 3. For both the Gaze Directions and Head Posture Dataset and People Driving Dataset we utilized a city car, while an iPhone 15 camera was employed to capture internal images and record internal videos. The iPhone was strategically positioned behind the steering wheel to ensure clear visibility of the driver while minimizing extraneous details. Furthermore, we positioned a GoPro Hero10 camera at the center of the car’s dashboard to capture external footage 3. Methods throughout the drive.
For the GDHPD, we compiled images from distinct This research explores an innovative method for recogniz- subjects, consisting of two males and two females. In ing gaze patterns while driving to evaluate driver atten- some cases, subjects wore glasses, while in the other tion. Subsequently, we focused on the internal aspect of they did not. The image collection process encompassed the vehicle, where we trained and tested neural networks various times of the day and diverse lighting conditions, for gaze classification. Our experimentation involved resulting in a total of 1012 images. Table 1 provides a various models, including SVM, ClassNET, VGG16-based breakdown of the distribution of these images. Net, and HEGClass Net. Additionally, we conducted a Subjects were positioned inside a car, with their seattraining phase for the external aspect using a custom ing adjusted to achieve a standard driving posture. Subdataset comprising trafic sign objects. Once we obtained sequently, images were captured while subjects varied results for both components, we merged the two mod- their gaze and head positions. To facilitate classificaules to conduct a comprehensive analysis of video record- tion, we devised a virtual grid dividing the external view ings obtained during real-world driving scenarios. This and the driver’s gaze into a 9-cell configuration. This integrated approach facilitated a more holistic comprehension of gaze behavior and its correlation with driver attention in typical driving situations. 3.1. Dataset A variety of images and videos were gathered and utilized at diferent stages of development to construct custom datasets tailored to our research objectives. These datasets can be categorized into four distinct collections: • Gaze Directions and Head Posture Dataset (GDHPD): This dataset comprises images captured by us, featuring individuals in a driving environment. The images are utilized to categorize the gaze position of individuals within a grid consisting of nine cells, including the exterior of the grid. • People Driving Dataset (PDD): This dataset consists of both external and internal videos, recorded by our team, showcasing the driving activities.
GDHPD dataset specifications Classes Male Female Total 0 50 64 114 1 50 64 114 2 51 71 122 3 51 71 122 4 48 60 108 5 48 60 108 6 50 50 100 7 51 50 101 8 51 61 112 9 50 61 111 Total 500 612 1012 grid facilitated the association of head and eye positions with specific regions of the external images, enabling the identification of Regions Of Interest (ROIs) during experimentation. 3.2. Gaze Classification
The dataset comprises ten classes: the nine cells of the grid, alongside an additional class representing situations Several algorithms, were analyzed to identify the apwhere the subject’s attention is not directed towards the proach with the best trade-of between accuracy in gaze road (e.g., face turned sideways, gaze directed upwards direction prediction and generalization capabilities, alor downwards, etc.). lowing to eficiently recognize images with varied con
The PDD dataset comprises videos captured during trast and/or brightness or diferent drivers. The structure driving sessions, utilizing the same recording setup as takes as input an image (either singular or an extracted the previous dataset. Subjects were filmed while driv- frame from the recording) and generates a label predicing under various conditions, capturing both the driver tion through two diferent but subsequent sub-models. and the road view simultaneously. To ensure synchro- The head-pose estimation algorithm is common for all nization, the videos underwent pre-processing using a tested approaches, while the classification algorithm has third-party software, DaVinci Resolve. Synchronization been severely varied in model type, structure, and input was achieved through voice cues, guaranteeing precise during the search for the optimal solution. alignment between internal and external footage.
Subsequently, the videos were segmented into sub- 3.2.1. Head-Pose Estimation Part
clips of 30 seconds each to streamline subsequent
processing steps. Each of the extracted sub-clips was anno- The face detection is performed through a pre-trained
tated with labels indicating whether the driver exhibited Multi-task Cascaded Convolutional Network (MTCNN)
a "CAREFUL" or "NOT CAREFUL" driving style, along model, used for both face detection and alignment in
with information regarding the driver’s use of glasses. literature [
For trafic sign detection and recognition, the primary eration with P-Net and refinement with R-Net and then dataset utilized was the Trafic Object dataset from the extracts the main 5 landmarks of the face (left and right Mapillary Trafic Sign Dataset, encompassing tens of eye, nose tip, and mouth corners) with O-Net. Among thousands of images sourced from roads worldwide. Fo- similar methods, such as Haar Cascade Classifiers [ 36], cusing solely on Italian/European trafic signs, around MTCNN has shown the best results even in the presence 3,000 images were selected from the dataset after filter- of glasses, partially occluded eyes and beards, and has ing out images with significantly diferent sign shapes therefore been selected as our chosen method. or contents. The chosen images ofer a varied range of The identified landmarks are then used to analytically brightness, positioning within frames, and contextual calculate the roll, pitch and yaw angles of the driver’s head, while the extracted pupil positions will be used network. This Classifier Network architecture consists of as input features for the final classifier to determine the two convolutional layers with ReLU activation functions, observed ROI. This feature is fundamental for our clas- followed by a max pooling layer, and culminates in two sification task compared to other facial features and is fully connected layers with ReLU and Softmax activation therefore particularly important to determine accurately. functions. Training of ClassNet spanned 300 epochs, employing the MSE loss function (Mean Squared Error) 3.2.2. Classification Part and Adam optimizer.
Finally, the last experimental iteration involved emOur approach involves classifying observed ROI and road ploying the VGG-16 architecture. Training data consisted signs through a classification method. The field of view of 1012 samples from the GDHPD dataset, each composed is divided into nine sections, with an additional label of the face image tensor, alongside head rotation angles for identifying any gaze position outside these sections (roll, pitch, yaw), and pupil centers. In this setup, fea(e.g., distracted driving or maneuvering). We’ve explored tures from each image were directly extracted within the various methods for analysis, ranging from traditional classification network from the RGB image of the entire SVM to CNNs, all accurately adapted for our application. face, rather than solely from the eyes. Additionally, other We will first introduce our novel method, followed by an features (roll, pitch, yaw, and pupil centers) obtained preoverview of other models considered in the analysis. viously through MTCNN were incorporated alongside
The standout classification model is HEGClass (Head- the 512 features of the image in the rfist fully connected Eyes-Gaze Classifier), a hybrid approach outlined in this layer. Training was executed over 100 epochs, using simpaper. It takes cropped face images from head-pose esti- ilarly to the HEGClass model the Cross-Entropy Loss mation, along with head rotation angles and pupil cen- function and Adam optimizer. ter coordinates, as inputs. This combined approach has yielded high precision in classifying the Region of Interest toward which the gaze is directed. In the HEGClass 3.3. YOLO Training for Trafic Signs network, as depicted in Figure 4, initial features are ex- For the detection and recognition of trafic signs, we start tracted from cropped face RGB images using a pre-trained from the pre-trained YOLOv8 model, with experimentaVGG-16 network. The features are then flattened and tion also conducted using the YOLOv5 model prior to concatenated with a normalized array containing head transitioning to the v8 version. Fine-tuning of YOLOv8 roll-pitch-yaw and pupil center coordinates. This com- was carried out using two distinct datasets: Trafic Obbined feature vector of dimension 4096+7 passes through jects and TSDY. The final set of weights chosen for the two fully connected linear layers, followed by ReLU ac- application was derived from the dual fine-tuning of tivation functions, and finally through a last fully con- YOLOv8 with both datasets. nected linear layer with Softmax activation to determine The initial fine-tuning with the Trafic Objects dataset class membership among the 10 possibilities (9 ROIs for involved 1802 images for training and 919 for validation. frontal regions and 1 for others). Model training utilized Despite starting with 3000 images, adjustments were our GDHPD dataset, with 10 epochs to mitigate overfit- made to the training and validation sets due to imbalting, 32 samples per batch, Cross-Entropy loss function, ance issues within the original dataset, which persisted and Adam optimizer. even after categorizing labels based on sign categories as
The first classical model used for comparison is Sup- described in the Dataset section. Subsequently, proceedport Vector Machine (SVM). In our scenario, where we’re ing from the fine-tuned weights, the model underwent classifying 10 distinct classes, we employed an SVM with retraining with images from the TSDY dataset, utilizing a polynomial kernel of degree 4, regularization param- 600 images for training and 141 for validation. eter set at 100, and coeficient set to 10. Training ex- The dual fine-tuning approach resulted in enhanced clusively used images from the GDHPD dataset. We ex- performance, as evidenced by improved final accuracy tracted roll, pitch, yaw, and pupil centers from the images and heightened generalization capabilities in detecting with MTCNN. Then, using Haar Cascade Classifier [ 36], road signs, even in images sourced from the PDD dataset we isolated eye patches from each grayscale image and and exhibiting varied lighting conditions. To further enpassed them through a pre-trained ResNet to obtain 2048 rich the diversity and generalization capabilities of the features for each eye. These two sets of features were ifne-tuned YOLO model, diverse image augmentation then averaged to create a unified array of 2048 elements techniques from the Albumentations library were emcontaining information from both eyes. The resulting ployed during training to simulate real-world conditions, samples underwent L2 normalization before being fed including Blur, MedianBlur, and CLAHE (Contrast Liminto the SVM for both training and testing phases. ited Adaptive Histogram Equalization). To streamline the
Using the same 2048 features extracted ResNet and roll, training process, the Stochastic Gradient Descent (SGD) pitch, yaw and pupil centers, we also trained the ClassNet optimizer was utilized, with an initial learning rate of 0.01. module. The image and information are then passed
Post-training, the model returns a text file for each through the network to classify the driver’s gaze posiimage processed, containing one line per detected sign tion. Upon obtaining the prediction of the observed cell alongside its position. By processing the pixel coordi- in a given frame, it is compared with the position of the nates from this file, sign position information was ex- corresponding sign. In frames with multiple detected tracted to reconstruct the sign’s center and edges within road signs, each corresponding ROI is considered active, grid cells. In instances where signs spanned multiple thus rendering the driver alert when focusing on any of cells, multiple coordinates were necessary for accurate them without favoring any type of signal. identification. Given that a single road sign can span multiple cells, 5 points characterize the object’s position: the four corners 3.4. Application: Merging the Methods and the center. If the driver looks at a cell containing a partial view of the sign, accounting for the peripheral The final application, depicted in Figure 5, comprises vision of human eyes we consider them attentive. Moretwo primary components and generates a CSV report over, in the absence of signals or when the driver’s gaze detailing the overall behavior of a driver. It requires is directed to cells 4/5/6 (representing the entire road as input an internal video capturing the driver and an surface), they are still deemed attentive to the street. If external video recording the street view. For ease of gaze is directed to cells 7 or 8, indicating focus on the analysis, synchronization of a 30-second video between car’s dashboard or infotainment system, the driver’s enthe two components is necessary. gagement is noted accordingly.
After inizialization, external frames undergo analysis Finally, a CSV file is generated to store the analysis using the YOLOv8 model trained on road signs, produc- results from the 30-second videos. Each row contains ing a text file detailing the detected signals along with data including the frame count (consistent across internal their specifications, including the Regions of Interest and external videos), the number of road signs detected (ROIs) where these signals are located. For frames with in that frame, the cell number(s) housing the detected at least one detected sign, the corresponding internal signals, the predicted cell value from the driver’s gaze image is used to extract information via the GDHPD network, the number of observed signals following ROI ifnal classification network. However, occasional failures in face detection or slight misplacements of landmarks introduce minor errors in this initial phase.
For predicting gaze direction, a zone-based classification approach was chosen over regression due to the dificulty in precisely determining the exact point on the road the individual is looking at, coupled with the human eye’s ability to perceive a broad area. Despite testing various methods, the SVM-based approach struggled to TFhigeuinrete5rn:aPliapnedlineextoefrnthalevaidpepoliscaarteiosnimpureltsaennetoeudsilny tphroecpeaspseedr. exceed a 70% accuracy level, likely due to the similarity to extract relevant features (facial landmarks and road signs in feature values across the 1012 samples, particularly bounding boxes respectively). The features are then used those derived from ResNet for eye images. to determine the active ROIs, which are then compared to Transitioning to neural network-based methods, the generate the attention level of the driver. ClassNet network yielded lower accuracy than SVM, even after experimenting with diferent feature combinations.
Training a network based on VGG-16 architecture from matching, and an indication of the driver’s attentiveness. scratch yielded better results with an accuracy level of 81%. However, the limited size of our dataset and computational constraints hindered achieving satisfactory 4. Results performance through this approach. Hence, we adopted the hybrid HEGClass approach, achieving an impressive The objective of this research is to develop a comprehen- 96% accuracy and 94.3% F1-score without additional data. sive system capable of analyzing an individual’s atten- Comprehensive accuracy and F1-score results are shown tion while driving using only two synchronized videos in Table 2. as input. Given the scarcity of references on the simultaneous analysis of internal and external perspectives, all 4.2. YOLOv8 Classification and Detection subsequent evaluations and comparisons will focus on the individual components constituting the final system.
Nonetheless, through extensive testing conducted with the PDD dataset, comprising approximately 194 videos each lasting 30 seconds, the final application demonstrates commendable performance.
Through dual fine-tuning of the YOLOv8 network by using the Trafic Object Dataset and the TSDY dataset, an impressive final F1-score of around 95% was achieved, with an example of prediction on the PDD dataset shown in Figure 6. We have observed an interesting phenomenon when training the network solely with the 4.1. Gaze Classification Trafic Objects dataset, where the F1-score is significantly lower. Specifically, the training of YOLOv8 with the For what concerns face detection and landmark extrac- Trafic Objects dataset yielded an overall accuracy of tion for facial rotation angle calculation, the MTCNN approximately 65%-70%. Performing the same process model outperformed the Haar Cascade Classifier. This with YOLOv5, instead, showed unexpectedly a higher superiority stems from MTCNN’s ability to handle vari- accuracy (around 80%), albeit with occasional misclassifious facial orientations, which is crucial for our GDHPD cations of elements such as empty spaces between tree dataset as it contains images with rotated or profiled faces. branches. In any case, for the scope of this project this Additionally, MTCNN’s prediction of landmarks, includ- comparison is not particularly relevant, given the much ing the center of the pupil, proved vital for training the higher accuracy with dual fine-tuning. dataset. Following initial evaluations, videos compromised by low light conditions (such as those filmed in almost night-time environments) or excessive blurring of frames, rendering accurate prediction unfeasible, were eliminated.
In instances where images lack clarity, are blurry, or exhibit excessive shaking, the predominant predicted class is 0, indicating the model’s failure to accurately identify the correct gaze position. Moreover, during nighttime or low-light scenarios, accurate gaze evaluation is significantly impeded by diminished brightness. Additionally, YOLO struggles with precise detection of relevant signs, often leading to confusion. Classes 5 and 6 are frequently identified as the gaze position during driving, aligning with the fact that these areas correspond to central regions of the windscreen. In some situations, such as when the vehicle is stationary at a trafic light or in trafifc congestion, the system may recognize the same signs across multiple frames. However, drivers may not consistently attend to them throughout, as they may have already observed them and they may not be of immediate significance at that moment.
Despite the substantial improvement in generalization capabilities achieved through dual training, errors in sign recognition persist. Certain objects along the road may be mistaken for road signs, such as advertisements containing elements that, with low resolution, could be confused. While this issue is present, its impact on the overall results remains manageable and could potentially be mit- 5. Conclusions igated with a wider variety of images. Another challenge arises from grouping signs of diferent shapes and colors This work aimed to develop and assess a comprehensive into the same class, creating a bias in their classification. system for evaluating attention to trafic signs in driving Additionally, signs containing other signs within them environments. We accomplished this by creating two may only become relevant in specific situations, such new datasets (GDHPD, PDD) and modifying two existas parking signs reserved for disabled individuals. For ing ones (Trafic Objects, TSDY) to better suit our task this reason, some signs were excluded from the training requirements. The final application was divided into two phase. parts, utilizing YOLOv8 for sign prediction and MTCNN
Given the high accuracy values in detection, adjusting + HEGClass for gaze position classification. the confidence threshold can help alleviate misclassifi- Despite encountering challenges during various traincation issues. Signs may be recognized even when ro- ing and testing phases, as described in the Results section, tated, facing the opposite direction of the lane, or located the overall accuracy of the final system remains very in irrelevant areas. In such cases, they are counted as high, notwithstanding the partial errors accumulated by points of inattention. Due to class imbalance, accurately its constituent parts. classifying the type of road sign remains a challenge. These challenges serve as valuable insights for future Consequently, for our purposes, only information related research endeavors. Opportunities for improvement into the bounding box defining the sign’s position is ex- clude implementing mechanisms to track seen and untracted, without specifying the type of sign. Despite seen signals, enhancing prediction accuracy in diverse attempts to simplify the dataset to recognize only one lighting and atmospheric conditions through dataset augclass, "TRAF_SIGN", challenges persisted between detect- mentation or pre-processing techniques, and expanding ing signs and identifying unrelated environmental areas. datasets to ensure greater completeness. Therefore, the decision was made to revert to using the Overall, this work presents significant potential for furoriginal labels. ther refinement and advancement, promising avenues for enhancing the performance and robustness of attention 4.3. Overall Analysis evaluation systems in driving contexts.
diction of the driver’s average attention while viewing a video, exhibit high performance across most cases, with a few notable exceptions. Tests were conducted on 191 videos, each lasting 30 seconds, sourced from the PDD