Distracted driving is a leading cause of road accidents globally, posing significant challenges to road safety. According to the National Highway Traffic Safety Administration (NHTSA) 1 approximately 3,308 people lost their lives in the United States in 2022 due to distracted driving, and nearly 290,000 people were injured. Almost 20% of those killed in distracted driving-related crashes were pedestrians, cyclists, and others outside the vehicle. In addition to the loss of lives and injuries, the financial burden from distracted driving crashes collectively amounts to $98 billion in 2019 alone, highlighting the urgency of developing effective detection methods.
The task of identifying distracted driving involves reliably detecting and classifying various forms of driver distraction, such as texting, eating, or using other objects/devices from in-vehicle camera feeds. This task is challenging due to the need for robust models that can generalize to a diverse set of driver behaviors without requiring extensive annotated datasets. Traditionally, the DDD task has been solved using various end-to-end learn-1 https://www.nhtsa.gov/speeches-presentations/distracteddriving-event-put-phone-away-or-pay-campaign ing and computer vision techniques, including, but not limited to, object detection, pose estimation, and action recognition. On the other hand, recent advancements in knowledge infusion [1] and Neurosymbolic AI [2] provide new opportunities for challenging tasks in scene understanding [3,4,5] and context understanding [6]. Hence, we posit that there is valuable auxiliary knowledge that can be either computed/ derived from the visual inputs. Specifically, we hypothesize that by infusing such knowledge with current computer vision models would improve the overall detection capabilities and robustness while not requiring the heavy computation demands of ultra-high parameter models.
To this end, we propose KiD3, a novel, simplistic method for distracted driver detection that infuses auxiliary knowledge about inherent semantic relations between entities in a scene and the structural configuration of the driver's pose. Specifically, we construct a unified framework that integrates scene graphs and the driver's pose information with visual information to enhance the model's understanding of distraction behaviors (see Figure 1).
Conducting experiments on a real-world, open dataset, our results indicate that incorporating such auxiliary knowledge with visual information significantly improves detection accuracy. KiD3 achieves a 13.64% accuracy improvement over the vision-only baseline, demonstrating the effectiveness of integrating semantic and pose information in DDD tasks. This improvement highlights the potential of our method to contribute to safer driving environments by providing a more reliable, effi-
Object Information Scene Graph Information cient and scalable solution that does not demand the use of expensive high-parameter models.
Contributions of this paper are as follows:
1. A novel, simple method for distracted driver detection that incorporates the auxiliary knowledge computed/estimated with vision inputs without the need for high-parameter, computational heavy models. 2. A demonstration of the effectiveness of infusing different types of auxiliary knowledge over vision-only baselines using real-world distracted driving data. Existing Methods for DDD: Vats et al. [7] proposes Key Point-Based Driver Activity Recognition that extracts static and movement-based features from driver pose and facial features and trains a frame classification model for action recognition. Then, a merge procedure is used to identify robust activity segments while ignoring outlier frame activity predictions.
In their work, Tran et al. [8] utilize multi-view synchronization across videos by training an ensemble 3D action recognition model on each view and taking the average probability over all the views as the final output. The outputs are then post-processed for predicting the action label and temporal localization of the predicted action. This work utilizes the X3D family of networks [9] for video classification instead of relying on manual feature engineering. Wei Zhou et al. [10] improve upon this work by fine-tuning large pre-trained models instead of training from scratch and by empirically selecting specific camera views for specific distracted action classes.
Previous works mainly focus on the use of sophisticated post-processing algorithms, use of larger encoder-decoder architectures and multi-view synchronization to improve action recognition and TAL performance. In contrast, our work aims to improve classification performance by incorporating auxiliary knowledge (e.g., semantic entities/relationships of a frame, pose information) that can be derived and infused as graphs into the encoder side of our architecture. Next, we will explore the state-of-the-art methods for scene graph generation.
Scene Graph Generation (SGG) refers to the task of automatically mapping an image or a video into a semantic structural scene graph, which requires the correct labeling of detected objects and their relationships [11]. Yuren Cong et al. [12] pose SGG as a set prediction problem. They propose an end-to-end SGG model, RelTR, with an encoder-decoder architecture. In contrast to most existing scene graph generation methods, such as Neural Motif, VCTree, and Graph R-CNN, [13,14,15] which RelTR used as benchmarks, RelTR is a one-stage method that predicts sparse scene graphs directly only using visual appearance without combining entities and labeling all possible predicates. Due to its simplicity, efficiency and SOTA performance, we selected RelTR to generate SGGs for our experiments.
Additionally, inspired by the work of Pen Ping et al. [16] we incorporate atomic action information extracted from the objects detected in the scene and the estimated pose of the driver.
In this section, we formally define the DDD problem, the datasets used, preprocessing steps, and delve deep into the technical details of each sub-component in the proposed approach (see Figure 3).
Given a video frame x β R πΓπΓ3 sampled from a video where π denotes the height of the frame, π denotes the width of the frame, and 3 corresponds to the color channels (RGB), the learning objective is to classify it into one of 18 predefined activities π = {πΆ1, πΆ2, . . . , πΆ18}.
We define a classifier model π : R πΓπΓ3 β [0, 1] 18 that maps a video frame to a probability distribution over the 18 activities. Specifically, π (x) = p, where p = [π1, π2, . . . , π18] and ππ represents the probability that the frame x belongs to class πΆπ, such that βοΈ 18 π=1 ππ = 1 and 0 β€ ππ β€ 1 βπ β {1, . . . , 18}. The predicted class πΆ Λfor the frame x can therefore be determined by: πΆ Λ= arg maxπΆ π βπ ππ.
The real-world datasets for distracted driver identification typically include annotated video sequences from cameras mounted inside the vehicle. While several open datasets are available, such as StateFarmDataset2 , we have selected SynDDv1 [17] to be used for experiments due to the higher number of distracted behavior classes and the diversity, including variations in lighting conditions, driver appearances, and the use of objects and Picking up from floor (passenger) 13
Talking to passenger at the right 14
Talking to passenger at backseat 15 Yawning 16
Hand on head 17
Singing with music 18
Shaking or dancing with music people in the background. SynDDv1 consists of 30 video clips in the training set and 30 videos in the test set. The dataset consists of images collected using three in-vehicle cameras positioned at locations: on the dashboard, near the rear-view mirror, and on the top right-side window corner, as shown in Table 1 and Figure 1. The video sequences are sampled at 30 frames per second at a resolution of 1920Γ1080 and are manually synchronized for the three camera views. Each video is approximately 10 minutes long and contains all 18 distracted activities shown in Table 2. The driver executed these activities with or without an appearance block, such as a hat or sunglasses, in random order for a random duration. There are six videos for each driver: three videos with an appearance block and three videos without any appearance block.
From the dataset, we selected the Dashboard variant, resulting in 10 videos for training and 10 videos for testing. Sets of (frame, label) were created by sampling frames from the videos at regular intervals and obtaining the corresponding labels from the annotations. The publicly available dataset contains various inconsistencies in the annotation format provided as CSV files. These inconsistencies, such as different naming conventions, variations in capitalization, and extra spaces in names, have been resolved to ensure consistency across all data splits.
Next, we will outline the technical details for each sub-component in our approach, shown in Figure 3.
To classify video frames into one of the predefined activities, the first step is to obtain robust image embeddings that would effectively capture the visual features in raw pixel data into a more manageable and informative representation. Possible methods for this transformation include using pre-trained Convolutional Neural Networks (CNNs) like VGGNet [18], ResNet [19], or Inception [20]. Out of these methods, we selected VGG16, a variant of VGGNet, due to its simplicity and effectiveness in extracting deep features from VGG16 been extensively used and validated in various image classification tasks, making it a reliable choice for our purpose.
VGGNet, particularly VGG16, is a deep convolutional network known for its simple yet effective architecture, consisting of 16 weight layers. The network is structured with multiple convolutional layers followed by fully connected layers. Each convolutional layer uses small receptive fields (3x3) and applies multiple filters to extract features at different levels of abstraction. The fully connected layers then process these features for classification. VGG16's design focuses on depth and simplicity, making it an ideal candidate for transfer learning.
To adapt VGG16 for our task, we fine-tuned the model to obtain image embeddings. Specifically, we discarded the last 2 classifier layers of the pre-trained VGG16 model and retained the base model along with the first 4 classifier layers. This configuration results in a 4096-dimensional image embedding vector. The rationale for discarding the last 2 layers is that the final layer reduces the dimensionality to only 18, which is insufficient for our needs. Additionally, the earlier layers capture more general features, which are beneficial for transfer learning. These embeddings are then used for further processing and classification tasks.
Scene graphs structurally represent the relationships between various objects in a given image. Each node in the graph represents an object, while edges denote the relationships between these objects; for example consider the triple: "Β« man holding phone Β»". Scene graphs capture the high-level contextual and semantic information of the scene, going beyond pixel-level data. They are also essential for scene understanding and reasoning and allow us to explicitly inject knowledge into the pipeline. For example, considering DDD task, a scene graph containing the triple "Β« person drinking_from bottle Β»" might indicate distracted driving activity. Modeling such important relations can otherwise be achieved implicitly using methods such as convolutional-network-based image encoders, with some uncertainty.
To generate the scene graph for a given frame, we use the RelTr architecture [12]. Then, we use a Graph Convolutional Network (GCN) [21] layer followed by a π ππβ activation to obtain representations for each node in the graph. We take the mean of all the node embeddings to obtain a graph-level representation and treat this vector as the graph encoding.
A scene graph output from RelTr [12] is in the form of triplets of the form (ππππ, relation, ππππ). Essentially, we get a list of relations π π = (π1, r, π2) where π1 and π2 are nodes and r is the relation between them. This format is converted to a list of edges, where edges are represented as pairs of nodes. This is provided to the GCN encoder to obtain a graph-level representation.
Pose estimation is a critical component in understanding the spatial configuration of a subject's body, which in this case is the driver. By capturing the positions of key body parts, pose estimation provides valuable information about the driver's posture and movements. This information is essential for accurately classifying the driver's activities. Various methods can be employed for pose estimation, including 2D and 3D approaches. We opted to use a state-of-the-art 2D pose estimation technique to effectively capture the required spatial data.
We utilized OpenPose [22], a state-of-the-art 2D pose estimation model, to extract pose information. OpenPose can detect and output a set of key points corresponding to various body parts, such as the head, shoulders, elbows, and hands. These key points are represented as coordinates in a 2D space. The process involves detecting the spatial locations of these joints and constructing a pose structure that reflects the driver's body configuration. Mathematically, each key point can be represented as: kπ = (π₯π, π¦π) where kπ denotes the π-th key point with π₯π and π¦π being its coordinates in the image frame.
To adapt the pose estimation data for our task, we preprocessed the key point coordinates obtained from Open-Pose. The key points were normalized and structured to consistently represent the driver's pose. Additionally, we derived features such as the distance between the hands and eyes/face, the angle formed by the eyes with the neck, and the distance between the hands and objects like a phone or bottle (if detected using YOLO [23]). These features were crucial for enhancing the model's ability to accurately interpret and classify the driver's activities.
We construct a simple machine-learning pipeline to combine the latent encodings of the above modules. Each module takes an image as input and processes it into a meaningful vector representation. We then concatenate these representations using a feed-forward MLP to classify the input image. Algorithm 1 succinctly outlines the main steps of this pipeline.
abstraction. The fully connected layers then process these features for classication. VGG16's design focuses on depth and simplicity, making it an ideal candidate for transfer learning.
3.4.3 Pre-processing and Adaptation. To adapt VGG16 our task, we ne-tuned the model to obtain image embeddings. Specically, we discarded the last 2 classier layers of the pre-trained VGG16 model and retained the base model along with the rst 4 classier layers. This conguration results in a 4096-dimensional image embedding vector. The rationale for discarding the last 2 layers is that the nal layer reduces the dimensionality to only 18, which is insufcient for our needs. Additionally, the earlier capture more general features, which are benecial for transfer learning. These embeddings are then used for further processing and classication tasks.
3.5.1 Background. Scene graphs structurally represent the relationships between various objects in a given image. Each node in the graph represents an object, while edges denote the relationships between these objects; for example consider the triple: "Β« man holding phone Β»". Scene graphs capture the high-level contextual and semantic information of the scene, going beyond pixel-level data. They are also essential for scene understanding and reasoning and allow us to explicitly inject knowledge into the pipeline. For example, considering DDD task, a scene graph containing the triple "Β« person drinking_from bottle Β»" might indicate distracted driving activity. Modeling such important relations can otherwise be achieved implicitly using methods such as convolutional-networkbased image encoders, with some uncertainty.
To generate the scene graph for a given frame, we use the RelTr architecture [2]. Then, we use a Graph Convolutional Network (GCN) [5] layer followed by a )0=β activation to obtain representations for each node in the graph. We take the mean of all the node embeddings to obtain a graph-level representation and treat this vector as the graph encoding.
3.6.2 Technical Details. We utilized OpenPose [1], a state-of-the-art 2D pose estimation model, to extract pose information. Open-Pose can detect and output a set of key points corresponding to various body parts, such as the head, shoulders, elbows, and hands. These key points are represented as coordinates in a 2D space. The process involves detecting the spatial locations of these joints and constructing a pose structure that reects the driver's body conguration. Mathematically, each key point can be represented as:
where k 8 denotes the 8-th key point with G 8 and ~8 being its coordinates in the image frame.
3.6.3 Pre-processing and Adaptation. To adapt the pose estimation data for our task, we pre-processed the key point coordinates obtained from OpenPose. The key points were normalized and structured to consistently represent the driver's pose.
Additionally, we derived features such as the distance between the hands and eyes/face, the angle formed by the eyes with the neck, and the distance between the hands and objects like a phone or bottle (if detected using YOLO [9]). These features were crucial for enhancing the model's ability to accurately interpret and classify the driver's activities.
We first fine-tune the pre-trained image encoder on the distracted driver classification task to obtain task-suitable embeddings. During training, we freeze the Image Encoding and Pose Information modules and only train the linear classifier and the GCN graph encoder in the Scene Graph Encoding module. We use πππ π‘πππ₯ activation in the final layer of the feed-forward MLP and use the Cross-Entropy loss function.
We outline the following experimental setup to evaluate the proposed approach's overall performance and the contribution of each sub-component.
In the first experiment, we utilized existing computer vision (CV) models to establish a baseline performance for the frame classification task. We fine-tuned the VGG-16 model to assess the performance of traditional CV models. To achieve this, we froze the weights of the entire model and unfroze only the classification layers (model.classifier [1...6]). The sixth classification layer nn.Linear(4096, 1000) was replaced with nn. Linear(4096,18) to match the number of activity classes. The modified model was then fine-tuned on our classification task, allowing the classification layers to adapt to the specific features of our dataset.
In the second experiment, we use the VGG-16 similar to how it was used in Method 1; however, out of the last six classifier layers, we discarded the last two layers and used the base model with the first four classifier layers to obtain a 4096-dimensional image embedding vector. The rationale is that the final layer could not be utilized because it reduces the image embedding to only 18 dimensions, which is insufficient for capturing the rich features needed for our task. Moreover, earlier layers in the network capture more general features beneficial for transfer learning. Then, we integrate image embeddings with scene graphs encoded using a Graph Convolutional Network (GCN) [21]. The embeddings derived from the GCN are concatenated with the image embeddings obtained from the VGG-16 model. Linear layers are used as a head to combine these information streams, forming a unified representation. This combined model was trained on the same classification objective, leveraging both the visual and relational features present in the data.
In the final experiment, we further enrich the scene representation by incorporating pose information, enhancing its ability to understand the driver's activities. The pose details included the location of objects via bounding boxes and the outline of the human skeleton with coordinates of key points such as the eyes, nose, and fists. We engineered additional features based on external knowledge, including the distance between the hand and face and the distance between the hand and a phone or bottle (if detected using YOLO [23]). These engineered features were added to the concatenation of image embeddings and scene graph embeddings. The model is then re-trained on the classification task with these additional features, providing a holistic understanding of the driver's activities.
Table 2 summarizes the results of our experiments on the test set and the ablation studies across different method variations. We evaluate the performance using two metrics: accuracy and the F1 score. The vision-only model achieves 79.64 overall accuracy and 0.81 F1 score, respectively. With the inclusion of scene graphs, the accuracy and the F1 score increased by 11.88% and 9.88%, respectively. Finally, the complete model incorporating both scene graphs and pose information achieves the peak performance of 90.5% accuracy and 0.91 F1 score, respectively. We have observed (see Figure 4) that our methods are particularly effective in identifying classes such as Eating (class 5), Adjusting Control Panel (class 10), and Singing with Music (class 17). We interpret this as evi-dence that our approach successfully incorporates auxiliary knowledge, enhancing our model's performance for these classes.
Our results clearly support the initial hypothesis that the inclusion of valuable auxiliary knowledge with visual features would enhance the performance of the DDD task. The ablation study further establishes each auxiliary knowledge type's role in the overall performance. Scene graphs provided the most significant auxiliary knowledge, highlighting the importance of explicitly encoding semantic information and infusing it with visual features. By incorporating pose information of driver actions, we were able to further enrich overall accuracy and robustness. However, several limitations to our approach warrant further investigation.
One limitation is the reliance on annotated data for training. While we used a combination of supervised and unsupervised learning techniques to mitigate this issue, the availability of annotated data remains a key constraint. Additionally, our method may struggle with complex and highly variable driving scenarios where the relationships between objects and actions are less clear. Finally, we have not considered using foundation models like Vision Language Models (VLMs) for our experiments. Our main focus in this work is to evaluate the impact of auxiliary knowledge on the DDD task without the need for complex, high-parameter models.
In this paper, we proposed a novel, simple approach to distracted driver detection by infusing two types of auxiliary knowledge with visual information. Our method leverages scene graphs and estimated pose information with visual embeddings to comprehensively represent driver actions. Our experimental results showcase the effectiveness of infusing each type of auxiliary knowledge with visual features to achieve 90.5% peak performance on the DDD task.
Future work will address the limitations mentioned above, such as the reliance on annotated data and the handling of complex driving scenarios. Additionally, we plan to explore the integration of other types of knowledge representations, such as temporal graphs, to further enhance the performance of distracted driver detection systems Further, we plan to investigate the role of VLMs in this task.