This paper gives an overview of the Driving Road Safety Forward: Video Data Privacy Task organized as part of the Benchmarking Initiative for Multimedia Evaluation (MediaEval) 2021. The goal of this video data task is to explore methods for obscuring driver identity in driver-facing video recordings while preserving human behavioral information.
The lifetime odds for dying in a car crash are 1 in 107 [
Video footage from driver-facing cameras presents a unique
opportunity to study driver behavior. Indeed, in the United States,
the Second Strategic Highway Research Program (SHRP2) worked
with drivers across the country to collect more than 1 million hours
of driver video [
The dataset consists of both high- and low-resolution driver video
data prepared by Oak Ridge National Laboratory (ORNL) for this
Driver Video Privacy Task. The videos were captured using the
same data acquisition system as the larger SHRP2 dataset mentioned
above, which currently has limited access in a secure enclave. For
the data in this Task, there are drivers in choreographed situations
designed to emulate diferent naturalistic driving environments.
Actions include talking, coughing, singing, dancing, waving,
eating, and various other actions that are typical among drivers [
To assess the de-identification of faces and measure the consistency in preserving driver actions and emotions, there will be a preliminary automated evaluation as well as a human evaluation. The scores for each of the automated and human evaluations will be combined for an overall assessment, prioritizing the human assessment of de-identification and action preservation. This Task is heavily reliant on human evaluation, and we encourage participants to include in their submission any ideas, methods, and results from their own evaluation approaches. This includes any available data from participants, descriptions of methodology, assumptions, and results. This information will be shared with reviewers and the project organizers for additional discussion and opportunities for seed funding for further research.
Although we encourage all Task participants to think creatively and holistically about how the expectations of privacy, the risk from potential attackers, and various threat models may evolve, our starting assumptions are that: (1) The drivers are not known to the potential attacker; there is no relationship between the attacker and the driver; the driver is not a public figure. (2) Information from the driver’s surroundings will not influence the attacker’s ability to identify the driver. (3) Access to the data is limited to registered users who have signed a Data Use Agreement specifying they will not attempt to learn the identity of individuals in the videos. (4) Attackers have access to basic computational resources. (5) There is a low probability of attackers launching an efective crowd-sourcing strategy to re-identify the drivers, in part due to the Data Use Agreement and context in which the data were collected. 4
Human evaluation is adapted from the methodology as described by
Baragchizadeh et al. in Evaluation of Automated Identity Masking
Method (AIM) in Naturalistic Driving Study (NDS) [
Undergraduate student volunteers will be recruited from the University of Texas at Dallas (UTD) to participate in the study in exchange for research credit. All procedures will be approved by the Institutional Review Board (IRB) of UTD. In all cases, a minimum of 10 students will evaluate each video for identity masking success. 4.2
Evaluations will be conducted using the masked videos submitted by the Task participants. For each submission1, a subset of at least 116 masked videos will be evaluated by human participants using an identity-matching procedure. The selected video subset will be identical for all Task participants and will be chosen by the organizers of the evaluation. The particular set of videos to be used in the evaluation will not be revealed until all submissions have been processed.
On each trial, the participant will be asked to match the identity of the person shown in a masked video to one of 5 high-resolution static facial images presented simultaneously at the top of the screen. The participants will be ofered responses to indicate which of the static images shows the person pictured in the video, or to indicate that the person pictured in the video does not appear in the set of the static images. The participant will have the option of replaying the video as many times as they want before entering a response.
The static face images will be matched demographically to the person in the video so that gender, race, and age cannot provide cues to the identity of the person in the video. Each static face image will be cropped to show only the internal face, so that identification cannot be based on peripheral face cues such as hair style. 4.3
Identification of the original (unmasked) videos from this dataset was assessed in a previous study at the Univ. of Texas at Dallas, using the methods described here for the Task evaluation. The identification accuracy results for these unmasked videos will be used to assess the success of Task participants in masking the face identities. It is important to note that matching the identity of the faces between the unmasked videos and the face images is not perfect. This is due to diferences in the image/appearance conditions between the static face images (high resolution, cropped to show only the internal face) and the videos (inside the car, highand low-resolution, variable expression, etc.). Therefore, masking success will be measured for each trial as the diference between the identification of the unmasked video (from the previous evaluation at UTD) and the identification of the masked video (from the human evaluation to be conducted on the masked videos submitted by Task participants). 1subject to the constraint that the algorithm is submitted by the deadline published on the MediaEval website
Task participants will be given summary data on the overall accuracy of their submission, as well as complete data on their performance for the individual videos. These data should be helpful for troubleshooting and improving the performance of the masking algorithm. We will also make available summary data on the performance of the other participants, so that individual Task participants can determine how their algorithm performed relative to other submissions.
The computational face recognition evaluation consists of face recognition and face detection steps. In the face recognition step, masked faces from selected frames are compared with the gallery face of the driver. We log the number of instances where the matching metric between the gallery face and the unmasked face indicate a match. In the face detection step, we attempt to detect faces in the masked video, and compute the intersection of union (IoU) score. We compute the cumulative score for IoU across tested frames. 5
The human evaluation procedures and results analysis for action preservation will be similar to those described for de-identification testing, with the following changes. On each trial, a masked video will be presented alongside a list of possible actions. The participant will be asked to select the “most obvious” action they detect in the video. Again, the results will be compiled as the diference between the accuracy of identifying the action in the unmasked video (from the previous UTD evaluation) and the masked video (from the Task participant) submission.
The automated approach to measuring action preservation will
use a deep-learning based gaze estimator [
With the increased availability, prominence, and applicability of data in our daily lives, multidisciplinary connections and engagement are critical to harnessing societal benefit from advances in technology and methodology. This focused video de-identification task serves as a key example of how data science collaborations designed to bridge research and practice can simultaneously help address a pragmatic need, while sparking new lines of inquiry and research trajectories. The Driving Road Safety Forward: Video Data Privacy Task strives to raise awareness about transportation fatalities and how data might enable thoughtful discussion, analysis, and actions for the betterment of our community safety.
Special thanks to our collaborators and advisors, including David Kuehn, Charles Fay, Daniel Morgan, Natalie Evans Harris, Lauren Smith, René Bastón, Mae Tanner, David E. Culler, the U.S. Department of Transportation (USDOT), and the National Science Foundation (NSF) Big Data Hubs network. This efort is made possible through community volunteers, NSF Grants 1916573, 1916481, and 1915774, and an inter-agency agreement between NSF and USDOT.