Thanks to the tremendous success of deep generative models, face forgery becomes an emerging research topic in very recent years and various methods have been proposed [1,2]. Depending on the manipulated region, they can be roughly categorized into two types: full-face synthesis [3,4] that usually swaps the whole synthesized source face to a target face, and local face region forgery [5,6] that only modifies partial face region, e.g., modifying the lip shape to match the audio content. Especially when the lips of politicians have been tampered with to make inappropriate speeches, it can lead to serious political crisis.
To alleviate the risks brought by malicious uses of face forgery, many detection methods have been proposed [7,8,9]. These methods usually consider the forgery detection from different aspects and extract visual features from the whole face region, achieving impressive detection results on public datasets FF++ and DFDC, in which most of the fake videos are tampered in a full-face synthesized manner. But this type of detection methods struggle to handle the local region forgery cases like lip-sync [5]. Recently, [10] attempt to detect lip-sync forgery video with single phoneme-viseme matching for The lip shapes of speaking the word "apple" in real (top) and fake (bottom) video. In the real video, the lips are more widely opened with clear teeth texture, while opposite in the fake. specific targets. [11,12] employ features such as audio and expression to detect synchronization between different modalities.
To address the problem of local region forgery detection, in this paper, we proposed a complete multiphoneme selection-based framework. To take full advantage of the particularity of lip forgery videos that contain audios, we need to establish a robust mapping relationship between the lip shapes and the audio contents. Prior studies in the realm of Audio-Visual Speech Recognition have demonstrated that the phoneme is the smallest identifiable unit correlated with a particular lip shape. Motivated by [13], we divide audio contents into 12 phoneme classes and classify all the video frames. For each phoneme-lip set, we measure the deviation on openclose amplitude between real and fake lip shapes, and train a sub-model for real/fake classification.
Usually, a large deviation represents the obvious discrepancy between the real and fake lip shapes, which also indicates the great difficulty in synthesizing the lip shape for the corresponding phoneme. Simultaneously, it shows the robustness of correlated phoneme-lip mapping against physical changes in different videos, e.g., volume and face angle. This precisely provides a distinguishing feature for forgery detection. By selecting the phonemes with the top-5 deviations, we integrate the corresponding 5 well-trained sub-models into an ensemble model for maximizing the discriminability of real and fake videos.
To verify the effectiveness, we have conducted extensive experiments on both the public DFDC dataset and a self-organized lip forgery video dataset which contains four sub-datasets. The experimental results demonstrate that our method outperforms the current state-of-the-art detection methods on cross-dataset evaluation and multiple class classification. In addition, our method is also competitive on single dataset classification.
According to different forgery regions, existing methods can be divided into two categories: full-face synthesis and local region forgery. Full-face synthesis usually synthesizes a whole source face and swaps it to the target. Typical works are [4,14].
Local region forgery is a more common type, focusing on slight manipulation of partial facial regions, eg, eyebrow locations and lip shapes. Lip-sync [5] is able to modify the lip shapes in Obama's talking videos to accurately synchronize with a given audio sequence. [15] leverages 3D modeling for specific face videos to make the control of lip shapes more flexible. First Order Motion [16] uses video to drive a single source portrait image to generate a talking video. The detection of local region forgery is more challenging due to the subtle and local nature.
Early works explored visual artifacts, eg, the abnormality of eye blinking and teeth. Learning-based detection methods have become mainstream in very recent years. [7] uses XceptionNet [17] to extract features from spatial domain. F 3 -Net [9] achieves state-of-the-art using frequency-aware decomposition. However, since the audios are lacking in most public deepfake datasets, these methods are designed in a universal manner with no consideration of audios matching. They perform well in full-face synthesis detection but is not adequate to recognize the subtle artifacts in local region forgery.
Recently, [11,12] utilize Siamese network to calculate the feature distances in multi-modalities. If manipulation is conducted on a small segment of the video, this will weaken the inconsistency among these modalities at the video level, leading to a decrease in detection performance. [10] establishes one single phoneme-viseme mapping for a specific person, which severely restricts the application scenario. To address the above limitations, we propose a multi-phoneme selection based framework for lip forgery video detection.
In this section we will elaborate the multi-phoneme selection based framework. Before that, an important observation of lip forgery will be introduced first.
Lip forgery modifies a specific person's lip shape to match arbitrary audio contents, thus establishing a close relationship between them. However, due to imperfections in the manipulation, uncontrollable artifacts may be generated to hinder the matching.
As shown in Figure 1, when saying the word "apple", the lips in the forgery videos are more blurred to open well. Although this nuance is not easy to perceive by human eyes, a well-designed detector can capture it. Nevertheless, the lip shape itself fluctuates in a certain range under different expressions, large fluctuation indicates poor robustness.
Based on this observation, it is necessary to establish a robust mapping from audios to lip shapes. Inspired by recent works in Audio-Visual Speech Recognition [18], we divide all audio contents into 12 phonemes categories as the smallest identifiable units. Each phoneme set consists of various vowels, consonants and quiet soundmark, which can be used to train sub-model independently to distinguish real/fake lips. Eventually, we select several sub-models to integrate the final classifier considering the trade-off between efficiency and performance. The framework is depicted in Figure 2.
For a given talking video, we use OpenFace [19] to align each frame and crop the lip area to 128Γ128. These lip images will be categorized into different phoneme set and used as training/testing data for real/fake classification.
To establish the mapping from phonemes to lip shapes, we first process all the real videos. According to the International Phonetic Alphabet (IPA) we divide the lip shapes into 48 classes. For a given lip shape, we calculate the Mahalanobis distance ππ of the open-close amplitude between the current lip shape x and mean xc of each class.
Next, we estimate the probabilities of it belonging to each class, and assign the sample to the class with the highest normalized probability ππ:
Here, π(π | x) is the probability of x belongs to class c, which is computed as the ratio between the in-class and the out-of-class distribution from the previous distance ππ, following the Gaussian distribution with means ππ, ππ Λand variances ππ, π ΜοΈ π , respectively :
After obtaining the mapping, a multi-class LDA classifier pre-trained on [20] is utilized for classification. However, different classes may share the same lip shape appearance, e.g., m,b,p. By iteratively merging similar phonetic symbol classes, we obtain 12 distinguishable real lip shapes named "phoneme" (from W1 to W12) with robustness. A visual example is given in Figure 3.
In fake videos, the lip shapes have been manipulated. As illustrated in Figure 1, the opening amplitudes of fake lips are quite different from real ones, thus directly using the phoneme classifier trained on real lips may lead to misclassification. Since the audio contents in fake videos are not modified, we decide to use them as the guidance for fake lips classification. First, Google's Speech-to-Text API is used to obtain the corresponding transcribed texts from the audios. Both the texts and audios are then fed into the P2FA toolkit [21]. By conducting forced alignment on phonemes and words, we get the start and end time for each phoneme, the lip images during this period will be categorized into the current phoneme. In Figure 2, the P2FA section clearly shows the alignment procedure.
Although the lip shapes in one phoneme set are similar, the open-close amplitudes among phonemes are quite different. We use dlib 68 face landmarks detector [22] to compute the vertical axis value between the 63th and 67th landmarks: π· = (π¦63-π¦67). Here π· represents the open-close amplitude of the current lip shape. Using the number of frames as the horizontal axis, we calculate π· for each frame during the period of the phoneme. In Figure 3, we plot two average amplitude curves for each set, the red curves represent the real videos while the blue for fake.
In W1 and W2, the real and fake curves are widely separated with almost no overlap, while in W3 and W6, there are partially stacked areas. This observation indicates that the real and fake lips are more discriminative in certain phoneme sets. To select the most distinguishable phonemes π for classification, we calculate the differences between the maximum and minimum values π·π πππ₯ ,π·π πππ of real/fake curves, respectively. We define the amplitude deviation π·π to represent the discrepancy between real and fake in each phoneme π : π·π = 1 2 (π·π πππ₯ + π·π πππ ). Considering the potential differences in forgery methods, the amplitude deviations of a single phoneme are not identical. As listed in Table 1, the phonemes with top-5 amplitude deviations are in bold, and we will introduce the self-organized dataset in Section 4.
After selecting the phoneme-lip sets for each forgery method, we train sub-classification models based on them. Each sub-model can be used independently for real/fake lips discrimination. Here we adopt XceptionNet [17] as the backbone and transfer it to our task by resizing the input to 128Γ128 and replacing the final connected layer with two outputs.
To obtain a stronger detection performance, we integrate the sub-models into an ensemble one. The average weight for each is equal to ensure the contribution is maximized. Furthermore, phoneme units in the video will last for some duration, which contain several lip frames. Both the lip frame numbers π and sub-models π will influence the detection accuracy of the final ensemble model, hence we experiment on them respectively. The results in Section 4 demonstrate that when π = 4 and π = 5, the ensemble model can achieve excellent
The composition of our self-organized dataset, including the numbers of videos and frames. The whole dataset consists of four sub-datasets.
Real/Fake Total Frames Obama Lip-sync [
In this section, we initially introduce a new lip forgery video dataset organized by this paper. Several parameter studies can verify the optimality of our settings. Further experiments are provided to demonstrate the effectiveness of our proposed framework on DFDC and selforganized dataset, as well as the transferability between them.
Many datasets [7,23] have been public for deepfake detection task. Although with large scale and various forgery methods, most fake videos do not contain the audios, which still tampered in a full-face synthesized manner. So far, there is no dedicated dataset released for lip forgery detection. In this paper, we use one public audio-visual deepfake dataset and organize a new dataset targeting the lip forgery detection task. Public DFDC Dataset [24] has been published in the Deepfake Detection Challenge, using multiple manipulation techniques and adding audios to make the video scenarios more natural. To make a fair comparison, we align with the settings of [11], using 18,000 videos in the experiments.
New Lip Forgery Dataset To build the new lip fogery dataset, we adopt four state-of-the-art methods [5, 15,
As mentioned before, XceptionNet is the baseline. According to the particularity of the public DFDC dataset and self-organized dataset, we adopt different training strategies. On the large DFDC dataset, we train our model with a batch size of 128 for 500 epochs. Due to the distinctly smaller size of the self-organized dataset, we train with a batch size of 16 for 100 epochs on each sub-dataset.
For both datasets, we uniformly use the Adam optimizer with the learning rate of 0.001 and employ ACC (accuracy) and AUC (area under ROC curve) as evaluation metrics.
Frame Selection. As showed in Figure 2, a single phoneme unit will include several lip frames. We use π to represent the number of lip frames, the value of π has an impact on the competence of the model. Few lip frames result in missing lip features of the current phoneme, while extra frames may overlap with others.
In order not to introduce disturbances from other factors, we experiment on the Obama Lip-sync dataset. We integrate all the 12 phoneme sub-models into one and take the beginning time of each phoneme as the center to select the surrounding frames π . Table 3 displays the accuracy of π from 3 to 8. The accuracy reaches 97.73% when π = 4, 7 and 8. Considering the tradeoff between accuracy and complexity, we finally choose π = 4.
Phoneme Selection. Still executing on the Obama Lip-sync dataset, we use π to denote the number of selected phonemes. Referring to the amplitude deviations ranking listed in Table 1, we integrate the sub-models from 2 to 12, the highest accuracy is achieved when π = 5. Thus we choose phoneme sets with the top 5 amplitude deviations to train sub-models.
In this section, we compare our method with previous deepfake detection methods on DFDC. The ratio of train-
Comparison of our method(Xception) with other techniques on the DFDC dataset using the AUC metric. We select submodels of W2, W5, W7, W10, and W11 for integration, and our result is competitive against Syncnet and Siamese-based methods.
Xception-c23 [17] 72.20 Video Meso4 [25] 75.30 Video DSP-FWA [26] 75.50 Video MBP [10] 80.34 Audio & Video Siamese-based [11] 84.40 Audio & Video Syncnet [12] 89.50 Audio & Video Ours (Xception)
Audio & Video ing and testing sets is 85:15. Even though we only crop the lip region of the face, we still achieve a competitive performance. In Table 4, our method achieves 91.6% on AUC, which outperforms not only the vision based fullface method but also the audio-visual based multi-modal method. Among them, Syncnet [12] detects the synchronization from audios to video frames, achieves 89.50% on AUC, while ignoring the content matching between them. The improvement in ours mainly benefits from the establishment of the phoneme-lip mapping, where the selected phonemes W2,W5,W7,W10 and W11 are robust to various external disturbances in DFDC such as face angle, illumination, and video compression, boosting the detection capability of the ensemble model. Moreover, we respectively visualize the Gradientweighted Class Activation Mapping (Grad-CAM) [28] for the baseline and ours, as shown in Figure 4. It shows that our method can significantly include the surrounding regions such as the upper and lower lips, which facilitates the network to focus on the open-close amplitudes and is in line with our motivation. In contrast, the baseline model mainly concerns the internal teeth regions, losing the edge information.
In this section, we conduct experiments on self-organized dataset to verify the performance of real/fake classification and multiple classification.
For each sub-dataset, We use different phonemes to integrate the final classification model, the selections are listed in Table 5. The baseline model (Xception) is directly trained on all continuous frames of real/fake videos. Further, to verify that our method is not restricted by the backbone, we adopt another network architecture ResNet-50 [29] which performs well in image classifica-
Evaluation of Real/Fake Classification. For each dataset, the performance of our approach surpasses baselines (Xception/ResNet-50) and existing state-of-the-art detection methods.
Obama Lip-sync [5] Audio Driven [15] First Order [16] Wav2lip
To further distinguish different forgery methods, in the 4 sub-datasets, we label all real lips with 0 and fake lips with 1 βΌ 4 individually. W2, W3, W4, W7, W8 are chosen to train the classification model.
Evaluation on cross-dataset. The testset is self-organized dataset. Ours (W2,W5,W7,W10,W11) achieves better results.
MBP [10] 57.94 59.12 Siamese-based [11] 59.51 60.68 Syncnet [12] 60.11 61.79 ResNet-50 [27] 54.74 57.67 Xception [17] 56 Table 6 verifies that the ensemble model can be applied to multiple classification scenarios. We also intuitively visualize the t-SNE [30] feature distributions from Siamese-based to ours. As shown in Figure 5, our method is superior to find latent dissimilarity in high-dimensional space with fewer outliers.
Transferability is evaluated by training on DFDC but testing on self-organized dataset where all lips are labeled as real/fake. Table 7 shows better transferability of ours in detecting universal artifacts in various datasets.
Lip forgery detection is an extremely challenging task in deepfake detection due to the subtle and local modifications. In this paper, we present a multi-phoneme selection based framework. Varying from existing deepfake detection, it takes full advantage of the particularity of lip forgery videos, establishing a robust mapping from audio to lip shapes. 12 categories of phonemes are determined as the smallest identifiable for various lip shapes and the phonemes with top-5 distinguishability are selected to train sub-classification models. In addition, we organize a new dataset consists of four sub-datasets, which is the first one organized for lip forgery detection task. Extensive experiments demonstrate the effectiveness of our framework, including the challenging task of cross-dataset evaluation.
This work was supported in part by the Natural Science Foundation of China under Grant U20B2047, U1636201, 62002334, by the Anhui Science Foundation of China under Grant 2008085QF296, by the Exploration Fund Project of the University of Science and Technology of China under Grant YD3480002001 and the Fundamental Research Funds for the Central Universities under Grant WK2100000011.