The rapid progression of Generative Adversarial Networks (GANs) has raised a concern of their misuses for malicious purposes, especially in creating fake face images. Although many proposed methods succeed in detecting GAN-based synthetic images, they are still limited by the need for large quantities of the training fake image dataset, and challenges for the detector's generalizability to unknown facial images. In this paper, we propose a new approach that explores the asynchronous frequency spectra of color channels, which is simple but efective for training both unsupervised and supervised learning models to distinguish GAN-based synthetic images. We further investigate the transferability of a training model that learns from our suggested features in one source domain and validates on another target domains with prior knowledge of the features' distribution. Our experimental results show that the discrepancy of spectra in the frequency domain is a practical artifact to efectively detect various types of GAN-based generated images.
Also, the checkerboard artifacts in spectrum generated
by up-sampling components of GAN model were also ex- images. Top: Real images and their corresponding
concurtensively investigated by Zhang et al. [
IJCAI 2021 (S. S. Woo) tional complexity, which can be prohibitively expensive
In recent years, there has been tremendous progress in
Generative Adversarial Networks (GAN), in which two
modules (generator vs. discriminator), play a minimax
game to produce highly realistic data. Unfortunately, in
addition to several fruitful GAN applications, attackers
can exploit GANs for malicious purposes, such as
spreading fake news [
Especially, Dzanic and Shah [
⎞ ⎛()
=1 where () is the number of channels of the ℎ layer’s output ,
∈ R××(+1)×() is a set of (+1) × () trainable 2 filters that have size of × . And (·), ⊛ and (·) denote the up-sampling operator, convolutional operator and activation function, respectively.
According to Khayatkhoei and Elgammal [11], we can simplify the Eq. 1 by restricting (·) to to rectified linear units (ReLU), which makes the () become locally piece-wise linear, and absorbing the up-sampling (·) into . In this way, we transform Eq. 1 to: +1 = () = ∑︁ ⊛ .
(2) =1 follows: By applying the 2D discrete Fourier transform (DFT) to +1, it is now viewed in the frequency domain as ˜ +1 = F(+1) = F (︃ ∑︁ ⊛ =1
)︃ =1 =1 =1 = ∑︁ F (︁ ⊛ )︁ = ∑︁ F (︁)︁ × F
(︁ )︁ = ∑︁ ˜ × ˜ = ⟨˜, ˜⟩, (linearity property of FT) (conv. property of FT) where ˜ = (˜ 1, ..., ˜ ) , and ˜ = (˜1, ..., ˜ ) . Based on this important observation, we propose the Equation 3 indicates that in the frequency domain, every following key statistical descriptive features to discrimichannel of in the next layer is decomposed into the com- nate the GAN images in the frequency domain: , bination all previous layer’s channels with diferent sets , , , , and , where of coeficients. If we consider +1 as the synthesized the details are presented below: output image and fix ˜ , every vector ˜,=1,..(+1) is trained independently to minimized the loss that applied • . We take the average of the channel-wise spectrum diferences: (6) (7) = + + ,
3
diferences between the spectra of the Red and Green channel in an image as follows: = 1
∑︁ ∑︁ ⃒⃒(, )−(, )⃒⃒, =1 =1 on each ˜+1. When we consider ˜ as a basic and ˜ as the coordinate of ˜+1 with respect to ˜, to synthesize a new image, the generative models expect that ˜ is also good enough so that each independent coeficient vector ˜ can produce corresponding single channel. In addition, these output channels should become as natural as possible in spacial domain after being stacked together in the order of three color channels: Red, Green and Blue. Small shifts of ˜,=1,..(+1) in the frequency domain may only change fine-grained details of visualization but can produce frequency-bias when there is no direct constraint between channels.
Let ℐ be a color image with three channels: Red, Green and Blue, which has width of and height of . To create its frequency representation, we firstly apply 2D DFT on each channel as follows:
Fℐ// (, ) = ∑︁ ∑︁ ℐ// (, )·−2( + ), =1 =1 and and can be similarly defined. • and . We take the maximum and min
imum values in {, , }. • . We calculate the correlation coeficient between and and transform it to positive range value by adding 1 to its negative values as follows: = −(, ) + 1, (8) where is the Pearson correlation coeficient, and and can be similarly defined.
Our end-to-end pipeline of extracting above frequency descriptive features from a given image is visually illustrated in Fig. 2. (4) where and , and denote the ℎ and ℎ slice in the width and height dimension of ℐ. For convenience, we use the notation Fℐ// to represent the function that is independently applied for each channel of the image. 2.3. Binary Classifier Note that Fℐ// (, ) is now a complex number, i.e.
Fℐ// (, ) ∈ C, and the spectrum of each channel is obtained as follows: // (, ) = (︁Fℐ// (, ))︁ , (5) where (·) denotes the modulus of complex number.
Although it would be challenging for human eyes to distinguish between real and GAN-generated fake images, we believe that their frequency spectra diferences can be possibly exposed, when we stack the three channels’ spectra of real vs. fake. Figure 1 presents our example of images’ spectra from VoxCeleb2 dataset [12] and Fake Head Talker dataset [13]. In particular, in the real images, we empirically find that the spectra of three color channels are mostly concurrent when stacking together, whereas they become noisy in the fake images, as shown in Fig. 1.
To demonstrate the characteristic-defining ability of the spectrum disagreement, we first employ the simple classifiers to classify real and fake images as below: • Gaussian Mixture Model (GMM). GMM is a probabilistic model that assumes the distribution of observed sampling data points is composed of a mixture of many Gaussian distributions, particularly, in our case is two distributions of real and fake class. To determine the means and variances of the two Gaussian distributions, ExpectationMaximization (EM) algorithm is used to iteratively estimate these parameters. Our descriptive features populate in the way that the higher spectrum agreement of an image will have the lower descriptive values. Therefore, we can expect that the Gaussian distribution in the mixture model, which has a smaller expectation will represent
the support vectors. In our experiment, we use SVM with the radial basis function (RBF) kernel to train with our six proposed features. the real images’ distribution, and the other repre- representations in the latent space with adversarsents the fake images’ distribution. By applying ial settings. The ALAE model can not only synEM, we can classify real and fake images in an thesize high-resolution images comparing with unsupervised manner in which the labels of a StyleGAN, but also can further manipulate or retraining dataset are not required. construct the new input facial images. • Support Vector Machine (SVM). SVM is a robust supervised learning method that maximizes Details of each dataset in our experiment are summathe margin of hyperplanes between diferent classes. rized in Table 1. In our experiment, the number of real and GAN fake images are equal in both training and test sets. We further provide the histograms to visualize the distributions of six descriptive features of these datasets in Supp. Section A.
To examine the efectiveness of our proposed frequency
features, we experiment four types of dataset: Fake Head
Talker [
Binary classification . Our experimental results are
shown in Table 2. We can observe that both
unsupervised and supervised methods are able to produce high
performance with our newly introduced frequency
features. The accuracy scores of the unsupervised method
on Fake Head Talker and ALAE dataset are competitive,
• Fake Head Talker dataset [
[
Resolution Require: Labeled source set {, }, unlabeled target set , where and includes the six proposed features [1, .., 6], respectively. The prior knowledge of Gaussian expectation values:
In particular, we first take the two Gaussian expectation values of two mixture distributions of each feature from both source and target dataset. These expectation values are kept as our prior knowledge about the target dataset. We then scale the source training set features such that their two Gaussian expectation values are normalized between 0 and 1 , to better fit the training dataset with the SVM model. In the testing phase, with our prior knowledge above, we can scale the testing features from the target dataset using the known expectation values and feed them to the pre-trained SVM model to make prediction. This adaptation learning process is summarized in the Algorithm 1.
We experiment with the four fake dataset and present the results in Table 4. We can observe that with our [︀,0, ,1]︀=1,...,6 and [︀,0, ,1]︀=1,...,6. 1: Step 1: Scale each feature in and : ¯ = (︀ − ,0)︀ / (︀,1 − ,0)︀, ¯ = (︀ − ,0)︀ / (︀,1 − ,0)︀ 2: Step 2: Fit source set {︀[¯1, .., ¯6], }︀ with SVM
model. 3: Step 3: Use pre-trained SVM to predict target set label
from [¯1, .., ¯6]. suggested features the pre-trained SVM shows its strong detection ability in the new target domain, where all the detection performance is above 80% of accuracy for any pair of source and target dataset. This preliminary experiment shows that our proposed features can be utilized in domain adaptation tasks with more complex learning models in the future.
Although GANs have significantly advanced in the past, we discover that there are some areas that GANs’ cannot mimic the real images efectively in the frequency domain. Thus, in this work, we propose a preliminary approach that reveals the asynchronous in frequency domain of the three channels in GAN images. By mining statistical features in frequency domain, our simple yet efective unsupervised and supervised learning methods can easily discriminate the real and GAN-based synthetic facial images without utilizing deep learning methods. Our extensive experiments demonstrates that the proposed features’ power in three scenarios: 1) unsupervised and supervised binary classification, 2) unbalanced training dataset, and 3) domain adaptation task. For future work, we plan to explore and exploit more on these aspects of GAN-generated images to combat against misuses from attackers, and extend our work to deepfake detection.
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00421, AI Graduate School Support Program (Sungkyunkwan University)), (No. 2019-0-01343, Regional strategic industry convergence security core talent training business) and the Basic Science Research Program through National Research Foundation of Korea (NRF) grant funded by Korea government MSIT (No. 2020R1C1C1006004). Additionally, this research was partly supported by IITP grant funded by the Korea government MSIT (No. 2021-0-00017, Original Technology Development of Artificial Intelligence Industry) and was partly supported by the Korea government MSIT, under the High-Potential Individuals Global Training Program (2019-0-01579) supervised by the IITP. sarial latent autoencoders, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14104–14113. [11] M. Khayatkhoei, A. Elgammal, Spatial frequency bias in convolutional generative adversarial networks, arXiv preprint arXiv:2010.01473 (2020). [12] A. Nagrani, J. S. Chung, W. Xie, A. Zisserman, Voxceleb: Large-scale speaker verification in the wild,
Computer Science and Language (2019). [13] H. Jeon, Y. Bang, S. S. Woo, Faketalkerdetect: Effective and practical realistic neural talking head detection with a highly unbalanced dataset, in: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 0–0. [14] Z. Liu, P. Luo, X. Wang, X. Tang, Deep learning face attributes in the wild, in: Proceedings of International Conference on Computer Vision (ICCV), 2015.
A. Distribution of Statistical Descriptive Features The histogram distributions of our six proposed statistical features in the frequency domains are present in the Fig. 3. We can observe that these feature distributions are highly separable between real and fake images across four datasets.
We provide example images from four datasets used in our experiment.