The 2021 MediaEval Multimedia Evaluation introduces a new data de-identification task, which goal is to explore methods for obscuring driver identity in driver-facing video recordings while maintaining visible human behavioral information. Interested in the challenge, our HCMUS team participate in searching for diferent ideas to tackle the problem. We propose two novel approaches as our main contribution: one is based on generative adversarial networks and the other is based on adversarial attacks. Moreover, a specific combination of evaluation metrics is also included for the later method for a fair comparison. The source code is available at https://github.com/kaylode/mediaeval21-drsf
Car accidents are an urgent problem for many countries today.
Currently, we develop vehicles to become safer, more durable, but
the number of accidents is still worrisome. In [
With the desire to study driver behavior, it is required to collect
more driving data. This leads to concerns about privacy and security.
At the "Driving Road Safety Forward: Video Data Privacy"
competition, we need to perform de-identification on SHRP2 dataset [
In this work, we want to focus on hiding their full face, which is the most important identifier in the given dataset. Specifically, we propose two approaches: • A simple process that swaps the driver’s face with an anonymous face to keep the most facial information. • An adversarial pipeline to perturbate the face identity while preserving main facial attributes in form of embedding features. This approach is followed by specific evaluation functions for appropriate assessment.
* Equal contribution 2.1
In this run, we implement the idea of swapping face to hide the
real face of the driver while keeping all other facial features like
gaze, eyes, mouth, nose, and head pose. The proposed procedure is
shown in Figure 1. We first extract face from the given RetinaFace
[
Overlay
FaceSwap 2.2
This run focus on hiding a person’s identity in the image from human view and preventing unauthorized deep vision algorithms from extracting useful information while ensuring correct prediction for authorized algorithms only. In our research, we only consider the position with rotation of the human face and its eye gaze vector as principal information for studying the person’s action and behavior.
The specific approach is a process consisting of two main steps: (1) Safeguard identified information from being inferred by unauthorized models. (2) Guarantee that the model with a defined set of weights can extract information with low error.
In the first step, we apply a simple identity masking technique pixelation and blurring to anonymize the driver’s faces. This step Minh-khoi Pham, Thang-Long Nguyen-Ho, Trong-Thang Pham et al.
Prediction on original images De-identified images
on de-identified images
provides strong perturbation to the original image such that the faces may not be identified by humans nor any vision models.
Secondarily, to ensure that the hidden attributes, which are the
bounding box, landmarks, and gaze vector of the face, can be
revealed only to model with a defined set of weights, we utilize
Iterative Fast Gradient Sign Method (I-FGSM) [
In general, I-FGSM works by exploiting the gradient of the cost function with respect to the input image to create a new image that maximizes the loss such that it drives the model’s outputs towards the desired target. In our case, I-FGSM is used in the opposite trend to the common goal, which is to modify the input image and minimize the loss, resulting in changing the model prediction on the adversarial sample from false to true.
The targeted models (and their default model weights) which
we choose to attack are listed below:
• For face detection, we experiment on the Retina Face [
ETH-XGaze [
We carry out the backpropagation process and find the corresponding quantity to change on the image. Our objective is to minimize function as described follows:
= + +
• is the box proposal loss of the detection model. • is the L2 error between the predicted heatmap and the ground truth heatmap of the landmarks estimator. • is the L2 error between the gaze vector predicted by the model and the true gaze vector.
With the proposed loss function, we compute the network gradient and use it as a perturbation to update the current input image.
+1 = , { + (∇ )} [
Given the 0 = the raw input image, we iteratively add the perturbation to X until becomes smaller than a predefined threshold or until meets the maximum number of iterations.
In the concept described above, the goal of the problem is to ensure that the facial attributes extracted by authorized models between the original image and its de-identified version have a slight deviation. Therefore, we propose the following assessment method, which indicates whether the attributes are well hidden or not: (, ) = 1 ∑︁
(1 − ( , )) + ( , ) + Θ ( , )
Where , is the adversarial and raw video sequence consisting of frames. , is the predictions of model on two adversarial and original frames at the same timestamp respectively. The is the Intersect-over-Union between the two faces location. The is measured based on the euclidean distance between landmark points of the faces, Θ is the angle between two gaze vectors. The ifnal is calculated by summing all diferences between pairs of frames across the timestamp.
Consequently, we expect is low for authorized algorithms and higher for unauthorized ones. 3
In the second run, we pass consecutive video sequences as a batch with a size equal to 64 into our de-identification pipeline to generate perturbated videos whose facial attributes have been hidden. We perform a full adversarial pipeline on 720 videos from the SHRP2 dataset and demonstrate some visual results as shown in Figure 2. 4
Conclusively, we present diferent strategies to address the data privacy issues for MediaEval Challenge 2021. In the future, we aim to study the performance of our adversarial attack for the information preservation method on several deep vision models regarding facial attributes. We are intent to analyze our proposed metrics on these experiments as well.
This work was funded by Gia Lam Urban Development and Investment Company Limited, Vingroup and supported by Vingroup Innovation Foundation (VINIF) under project code VINIF.2019.DA19.