Robustness and Generalization of Synthetic Images Detectors Davide Alessandro Coccomini1,2 , Roberto Caldelli3,4 , Claudio Gennaro1 , Giuseppe Fiameni5 , Giuseppe Amato1 and Fabrizio Falchi1 1 ISTI-CNR, Pisa, Italy 2 University of Pisa, Pisa, Italy 3 CNIT, Florence, Italy 4 Universitas Mercatorum, Rome, Italy 5 NVIDIA AI Technology Center, Italy Abstract In recent times, the increasing spread of synthetic media, known as deepfakes has been made possible by the rapid progress in artificial intelligence technologies, especially deep learning algorithms. Growing worries about the increasing availability and believability of deepfakes have spurred researchers to concentrate on developing methods to detect them. In this field researchers at ISTI CNR’s AIMH Lab, in collaboration with researchers from other organizations, have conducted research, investigations, and projects to contribute to combating this trend, exploring new solutions and threats. This article summarizes the most recent efforts made in this area by our researchers and in collaboration with other institutions and experts. Keywords Deepfake Detection, Deep Learning, Super Resolution 1. Introduction other problem is that of adversarial attacks, strategies of camouflaging traces, enhancing fake content or ad-hoc Deepfakes and synthetic media are becoming more preva- manipulations designed to fool the detector, which can lent and realistic day by day, presenting society with an be used to make detection even more complex. Deepfake increasingly urgent challenge, learning to distinguish detection models must therefore be designed so that they reality from fiction effectively. These fake content can provide a high degree of robustness to possible adver- and are continually being used to spread disinformation, sarial attacks and also be able to effectively distinguish create smear campaigns, and manipulate reality with po- deepfakes without raising false alarms. For this reason, tentially devastating impacts for anyone who may end AIMH Lab at ISTI CNR has carried out numerous re- up a victim. To contrast this phenomenon, research has search attempts to explore new innovative techniques to been conducted in recent years creating detectors, often advance this field but also to highlight possible hidden based on deep learning techniques, that can classify a dangers that may damage the efforts made in previous piece of content (such as an image) as realistic or fake. research, representing dangers that detection systems Despite many efforts, this discrimination capability is may encounter in the real world. In particular, this paper still insufficient today with many open problems in the summarizes the efforts made in [3] and [4]. field of deepfake detection. One example above all is that of generalization[1, 2]. In fact, deepfake detectors, although particularly effective in detecting images gener- 2. Research Works in Deepfake ated or manipulated by the same methods they are trained Detection on, fail when using different and novel techniques. An- In this section, we present our most recent works in the Ital-IA 2024: 4th National Conference on Artificial Intelligence, orga- nized by CINI, May 29-30, 2024, Naples, Italy field of Deepfake Detection, highlighting the contribu- * Davide Alessandro Coccomini tions and discoveries made. $ davidealessandro.coccomini@isti.cnr.it (D. A. Coccomini); roberto.caldelli@unifi.it (R. Caldelli); claudio.gennaro@isti.cnr.it (C. Gennaro); gfiameni@nvidia.com (G. Fiameni); 2.1. Super-Resolution as an Adversarial giuseppe.amato@isti.cnr.it (G. Amato); fabrizio.falchi@isti.cnr.it Attack for Deepfake Detection (F. Falchi)  0000-0002-0755-6154 (D. A. Coccomini); 0000-0003-3471-1196 Super-resolution (SR) algorithms are a set of techniques (R. Caldelli); 0000-0002-3715-149X (C. Gennaro); designed to improve the resolution of an image. Starting 0000-0001-8687-6609 (G. Fiameni); 0000-0003-0171-4315 with a low-resolution one, through deep learning tech- (G. Amato); 0000-0001-6258-5313 (F. Falchi) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License niques, it is scaled up to a higher resolution. During this Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Model Forgery Attacked Accuracy ↑ FNR ↓ FPR ↓ AUC ↑ ✖ 95.3 5.9 3.6 99.1 Deepfakes ✓ 90.7 6.1 12.4 97.4 ✖ 87.1 12.9 12.8 94.9 NeuralTextures ✓ 81.5 13.2 23.9 90.4 ✖ 95.2 6.3 3.3 98.9 Swin Face2Face ✓ 87.0 24.4 1.7 96.1 ✖ 95.2 4.9 4.6 98.6 FaceSwap ✓ 86.4 21.9 5.3 93.9 ✖ 94.4 7.2 4.1 98.7 FaceShifter ✓ 89.0 18.9 3.1 97.4 ✖ 95.6 5.5 3.2 99.2 Deepfakes ✓ 95.0 6.9 10.1 98.9 ✖ 87.1 12.9 12.8 94.9 NeuralTextures ✓ 81.5 13.2 23.9 90.4 ✖ 95.9 5.0 3.2 98.9 Resnet Face2Face ✓ 91.2 14.4 4.7 97.6 ✖ 95.9 6.4 1.9 99.1 FaceSwap ✓ 88.5 21.1 2.6 95.6 ✖ 95.3 6.1 3.4 98.8 FaceShifter ✓ 85.9 24.8 3.3 95.2 Table 1 Evaluation on Faceforensics++[5] test set. Each set is composed of half pristine and half fake images. The Attacked column indicates if the SR-attack has been applied to the images. The attack is applied to fake and pristine images. process, some aspects of the image may change. For ex- ample, some previously visible details may become more blurred, totally disappear, or, conversely, be emphasized and brought to light. Deepfake generation algorithms commonly tend to introduce some more or less visible artifacts. In the case of human faces, these artifacts can be, for example, anomalies in pupils, contours of lips, eyes or ears, or accessories. Typically, deepfake detection models learn to recognize the specific anomalies intro- duced in manipulated images and, because of them, can discriminate between pristine and fake content. In [4], we explored whether Super Resolution techniques can be used as an adversarial attack to camouflage artifacts introduced by deepfake generations approaches. To do this, we proposed an SR-attack pipeline, whose purpose is to disguise artifacts present in deepfake images while still trying to keep the appearance as unaltered as Figure 1: Example of the impact of SR attack on fake images. possible. The pipeline begins with the detection of a face On the left, an example of manipulated face with a zoom on from the deepfake image that is subsequently downscaled the artifacts around the mouth. On the right the same face but after the application of the SR-attack. From the zoom on by a factor 𝐾 using interpolation techniques. The result- the second one it can be seen how the artifacts are drastically ing image is restored to its initial resolution through a smoothed. super-resolution approach. The resulting face can even- tually be reinserted into the original image, resulting in the camouflaged image. An example of the impact of the proposed SR-attack one we often rely on observing these artifacts and oddi- is shown in Figure 1. As the figure shows, the attack ties that can be introduced by the manipulation process. leads to the removal of artifacts introduced by deepfake The fact that super-resolution leads to their removal or generation techniques, such as noise around the mouth, attenuation qualifies it as a potentially effective attack and thus makes their detection extremely complex. In against deepfake detectors but also against the human fact, to distinguish a counterfeit image from a pristine eye itself. The usage of the SR-attack conduct to a blurring effect algorithm and the value of the 𝐾 factor appropriately in on the artifacts introduced in the fake images and this order to achieve maximum effectiveness from the attack. makes them more difficult to detect. This is pretty evident in terms of performance; in fact, the use of the attack 2.2. Future Works drastically degrades the performance of classifiers trained to do deepfake detection. In this section we expose some of the future works we are Table 1 shows the accuracies of a Swin Transformer[6] working on either as extensions of previously presented and a Resnet50[7] on images manipulated with different works or as new applications and solutions for effective techniques, considering them before and after SR-attack. deepfake detection. The dataset used is FaceForensics++[5] and the deepfake generation methods considered are Deepfakes[8], 2.2.1. Robustness of Deepfake Detectors Face2Face[9], FaceSwap[10], FaceShifter[11] and NeuralTextures[12]. This allows us to evaluate the The fact that Deepfake Detectors are susceptible to the effect of the super-resolution attack on different types use of SR techniques on images, whether fake or pristine, of manipulations, thus highlighting on which it is exposes a serious problem in their use in the real world more or less effective. Both the models are trained and therefore requires that more studies be conducted to on the FaceForensics++ training set considering for make them robust to this kind of content. It is necessary the construction of the fake class, the same deepfake to find an effective way to make the models robust to generation method used for the test. this attack for example by introducing super-resolution According to our experiments, for both the considered as data augmentation during training. models, when images are attacked with the proposed The attack itself can also be further explored and im- approach, there is an increase in False Negative Rate, par- proved by going to identify the optimal 𝐾 value and ticularly on some methods namely Face2Face, FaceSwap corresponding SR method as well as experimenting with and FaceShifter. Others, however, are found to be more different strategies for applying the attack, such as focus- robust to attack, namely Deepfakes and NeuralTextures ing on a frame rather than a detected face. on which, however, there is an increase in False Positives. This behavior indicates that in some cases, the use of 2.2.2. Deepfake Detection without Deepfakes super-resolution techniques could lead to the elevation As stated before, one of the most stringent problems in of false alarms, leading models to identify legitimately the field of deepfake detection is that of generalization. enhanced images through these approaches as deepfakes. Indeed, there is ample evidence that detectors tend to The latter result highlights a problem that could prove learn to effectively recognize deepfake content obtained crippling to traditional deepfake detectors and could pre- through methods used to construct their training set, but vent their deployment in the real world. Indeed, it is fail when they need to classify content obtained through plausible to think that on social networks it will become novel techniques. This leads to a total inadequacy of increasingly common to improve the quality of one’s pho- conventional deepfake detectors in being used in the real tos through Super-Resolution techniques. If this were world. In fact, new deepfake generation techniques are to happen and in parallel the deepfake detectors were continually being created, and it would be impractical unable to understand that these are legitimate images but to retrain the model each time to introduce every single instead end up mistaking them for deepfakes, the number possible method. In the context of synthetic images, this of false alarms would be such as to prevent their effective tendency of deepfake detectors stems from the fact that deployment on a large scale. It is therefore necessary each generator introduces a specific fingerprint into the on the one hand to defend against the malicious use of image[15, 16, 17]. It tends to be invisible to the human super-resolution to disguise artifacts introduced by ma- eye but involves the presence of structured patterns in nipulation techniques but also to make detectors robust the frequency domain (grids, symmetric peaks, halos, so that they are able to recognize legitimately augmented etc.). images. From the observation of this phenomenon, as a fu- In these experiments we used only EDSR[13] as the ture work we are exploring a new training technique basis of our attack but the proposed attack can be con- for deepfake detectors that tries to stimulate the model ducted using different types of SR algorithms (such as to recognize the presence of structured patterns in the BSRGAN[14]), and depending on the peculiarities of each, frequency domain and not to learn a specific fingerprint. greater or lesser effectiveness can be achieved on each The preliminary results of this approach can be found in specific deepfake generation method. The choice of the [3]. 𝐾 factor also has an impact in that as it increases, the de- We propose to reproduce prototype structured pat- tectors’ errors increase but the quality of the image itself terns inspired by what we observed from the fingerprints also deteriorates. Therefore, it is crucial to choose the SR actually introduced by generative patterns of various [2] D. A. Coccomini, R. Caldelli, F. Falchi, C. Gennaro, types. These patterns, in frequency, are injected in pris- On the generalization of deep learning models in tine images and considered as "fake" in the training phase. video deepfake detection, Journal of Imaging (2023). During the training, we show the model pristine images doi:10.3390/jimaging9050089. and others on which a pattern has been applied, indicat- [3] D. A. Coccomini, R. Caldelli, C. Gennaro, G. Fi- ing them to the model as fakes. ameni, G. Amato, F. Falchi, Deepfake detec- From our preliminary experiments, we demonstrated tion without deepfakes: Generalization via that models trained on this proto-task, are extremely synthetic frequency patterns injection, 2024. effective at identifying synthetic images despite never arXiv:2403.13479. really seeing one in the training phase. [4] D. A. Coccomini, R. Caldelli, G. Falchi, Amato, The use of synthetic patterns may also be used to sup- F. Falchi, C. Gennaro, Adversarial magnification port traditional training of deepfake detectors, introduc- to deceive deepfake detection through super res- ing them only occasionally and still maintaining deep- olution (to appear), in: Proceedings of European fakes in the training set. In addition, it will be possible to Conference on Machine Learning and Principles experiment with a virtually infinite number of patterns and Practice of Knowledge Discovery in Databases, by searching for the most effective ones and studying 2023. their impact. [5] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, M. Niessner, Faceforensics++: Learning to detect manipulated facial images, in: Proceed- 3. Conclusions ings of the IEEE/CVF International Conference on Computer Vision, 2019. Carrying out research in the field of deepfake detection [6] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, is an increasingly pressing need because of the multitude S. Lin, B. Guo, Swin transformer: Hierarchical of techniques that now make it possible to produce syn- vision transformer using shifted windows, in: 2021 thetic or manipulated content with an increasing degree IEEE/CVF International Conference on Computer of credibility. As shown in our recent research, it is criti- Vision (ICCV), 2021. doi:10.1109/ICCV48922. cal to explore both innovative methods to try to improve 2021.00986. the capability of deepfake detectors looking for new train- [7] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learn- ing approaches and techniques. This could be needed to ing for image recognition, in: 2016 IEEE Confer- overcome the pressing problem of generalization. On the ence on Computer Vision and Pattern Recognition other hand, it is important to find new solution to face (CVPR), 2016. doi:10.1109/CVPR.2016.90. the possible risks and unexpected situations that these [8] Deepfakes, 2018. URL: https://github.com/ models might encounter in the real-world that could in- deepfakes/faceswap. validate their potential, such as the adversarial attacks. [9] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, M. Nießner, Face2face: Real-time face capture and 3.0.1. Acknowledgments reenactment of rgb videos, Commun. ACM (2018). This work was partially supported by the follow- doi:10.1145/3292039. ing projects: Tuscany Health Ecosystem (THE) (CUP [10] K. M., Faceswap, 2017. URL: https://github.com/ B83C22003930001) and SERICS (PE00000014, MUR PNRR MarekKowalski/FaceSwap/. - NextGenerationEU), AI4Media (EC H2020 - n. 951911) [11] L. Li, J. Bao, H. Yang, D. Chen, F. Wen, Faceshifter: and AI4Debunk (Horizon EU n. 101135757), FOSTERER Towards high fidelity and occlusion aware face (Italian MUR PRIN 2022). We acknowledge the CINECA swapping, 2020. arXiv:1912.13457. award under the ISCRA initiative, for the availability of [12] J. Thies, M. Zollhöfer, M. Nießner, Deferred neural high-performance computing resources and support. rendering: Image synthesis using neural textures (2019). doi:10.1145/3306346.3323035. [13] B. Lim, S. Son, H. Kim, S. Nah, K. Mu Lee, En- References hanced deep residual networks for single image super-resolution, in: Proceedings of the IEEE Con- [1] D. A. Coccomini, R. Caldelli, F. Falchi, C. Gennaro, ference on Computer Vision and Pattern Recogni- G. Amato, Cross-forgery analysis of vision trans- tion (CVPR) Workshops, 2017. formers and CNNs for deepfake image detection, [14] K. Zhang, J. Liang, L. Van Gool, R. Timofte, Design- in: International Conference on Multimedia Re- ing a practical degradation model for deep blind trieval Workshop, 2022. doi:10.1145/3512732. image super-resolution, in: IEEE International Con- 3533582. ference on Computer Vision, 2021. [15] D. A. Coccomini, A. Esuli, F. Falchi, C. Gennaro, G. Amato, Detecting images generated by diffusers, 2023. doi:10.48550/arXiv.2303.05275. [16] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, A. A. Efros, CNN-generated images are surprisingly easy to spot. . . for now, in: IEEE Conf. Comput. Vis. Pattern Recog., 2020, pp. 8692–8701. doi:10.1109/ CVPR42600.2020.00872. [17] R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi, K. Nagano, L. Verdoliva, On the detection of syn- thetic images generated by diffusion models, in: Int. Conf. on Acoustics, Speech and Signal Processing, 2023, pp. 1–5. doi:10.1109/ICASSP49357.2023. 10095167.