In recent times, the increasing spread of synthetic media, known as deepfakes has been made possible by the rapid progress in artificial intelligence technologies, especially deep learning algorithms. Growing worries about the increasing availability and believability of deepfakes have spurred researchers to concentrate on developing methods to detect them. In this field researchers at ISTI CNR's AIMH Lab, in collaboration with researchers from other organizations, have conducted research, investigations, and projects to contribute to combating this trend, exploring new solutions and threats. This article summarizes the most recent eforts made in this area by our researchers and in collaboration with other institutions and experts.
1. Introduction other problem is that of adversarial attacks, strategies of camouflaging traces, enhancing fake content or ad-hoc Deepfakes and synthetic media are becoming more preva- manipulations designed to fool the detector, which can lent and realistic day by day, presenting society with an be used to make detection even more complex. Deepfake increasingly urgent challenge, learning to distinguish detection models must therefore be designed so that they reality from fiction efectively. These fake content can provide a high degree of robustness to possible adverand are continually being used to spread disinformation, sarial attacks and also be able to efectively distinguish create smear campaigns, and manipulate reality with po- deepfakes without raising false alarms. For this reason, tentially devastating impacts for anyone who may end AIMH Lab at ISTI CNR has carried out numerous reup a victim. To contrast this phenomenon, research has search attempts to explore new innovative techniques to been conducted in recent years creating detectors, often advance this field but also to highlight possible hidden based on deep learning techniques, that can classify a dangers that may damage the eforts made in previous piece of content (such as an image) as realistic or fake. research, representing dangers that detection systems Despite many eforts, this discrimination capability is may encounter in the real world. In particular, this paper still insuficient today with many open problems in the summarizes the eforts made in [3] and [4]. ifeld of deepfake detection. One example above all is that of generalization[1, 2]. In fact, deepfake detectors, although particularly efective in detecting images gener- 2. Research Works in Deepfake ated or manipulated by the same methods they are trained Detection on, fail when using diferent and novel techniques. An
In this section, we present our most recent works in the ifeld of Deepfake Detection, highlighting the contributions and discoveries made.
Model Swin
Forgery Deepfakes NeuralTextures Face2Face FaceSwap FaceShifter Deepfakes
NeuralTextures Resnet
Face2Face FaceSwap FaceShifter
Attacked ✖ ✓ ✖ ✓ ✖ ✓ ✖ ✓ ✖ ✓ ✖ ✓ ✖ ✓ ✖ ✓ ✖ ✓ ✖ ✓ process, some aspects of the image may change. For example, some previously visible details may become more blurred, totally disappear, or, conversely, be emphasized and brought to light. Deepfake generation algorithms commonly tend to introduce some more or less visible artifacts. In the case of human faces, these artifacts can be, for example, anomalies in pupils, contours of lips, eyes or ears, or accessories. Typically, deepfake detection models learn to recognize the specific anomalies introduced in manipulated images and, because of them, can discriminate between pristine and fake content. In [4], we explored whether Super Resolution techniques can be used as an adversarial attack to camouflage artifacts introduced by deepfake generations approaches.
To do this, we proposed an SR-attack pipeline, whose purpose is to disguise artifacts present in deepfake images while still trying to keep the appearance as unaltered as possible. The pipeline begins with the detection of a face from the deepfake image that is subsequently downscaled by a factor using interpolation techniques. The resulting image is restored to its initial resolution through a super-resolution approach. The resulting face can eventually be reinserted into the original image, resulting in the camouflaged image.
An example of the impact of the proposed SR-attack is shown in Figure 1. As the figure shows, the attack leads to the removal of artifacts introduced by deepfake generation techniques, such as noise around the mouth, and thus makes their detection extremely complex. In fact, to distinguish a counterfeit image from a pristine one we often rely on observing these artifacts and oddities that can be introduced by the manipulation process.
The fact that super-resolution leads to their removal or attenuation qualifies it as a potentially efective attack against deepfake detectors but also against the human eye itself.
The usage of the SR-attack conduct to a blurring efect algorithm and the value of the factor appropriately in on the artifacts introduced in the fake images and this order to achieve maximum efectiveness from the attack. makes them more dificult to detect. This is pretty evident in terms of performance; in fact, the use of the attack 2.2. Future Works drastically degrades the performance of classifiers trained to do deepfake detection. In this section we expose some of the future works we are
Table 1 shows the accuracies of a Swin Transformer[6] working on either as extensions of previously presented and a Resnet50[7] on images manipulated with diferent works or as new applications and solutions for efective techniques, considering them before and after SR-attack. deepfake detection.
The dataset used is FaceForensics++[5] and the deepfake generation methods considered are Deepfakes[8], 2.2.1. Robustness of Deepfake Detectors Face2Face[9], FaceSwap[10], FaceShifter[11] and NeuralTextures[12]. This allows us to evaluate the The fact that Deepfake Detectors are susceptible to the efect of the super-resolution attack on diferent types use of SR techniques on images, whether fake or pristine, of manipulations, thus highlighting on which it is exposes a serious problem in their use in the real world more or less efective. Both the models are trained and therefore requires that more studies be conducted to on the FaceForensics++ training set considering for make them robust to this kind of content. It is necessary the construction of the fake class, the same deepfake to find an efective way to make the models robust to generation method used for the test. this attack for example by introducing super-resolution
According to our experiments, for both the considered as data augmentation during training. models, when images are attacked with the proposed The attack itself can also be further explored and imapproach, there is an increase in False Negative Rate, par- proved by going to identify the optimal value and ticularly on some methods namely Face2Face, FaceSwap corresponding SR method as well as experimenting with and FaceShifter. Others, however, are found to be more diferent strategies for applying the attack, such as focusrobust to attack, namely Deepfakes and NeuralTextures ing on a frame rather than a detected face. on which, however, there is an increase in False Positives.
This behavior indicates that in some cases, the use of 2.2.2. Deepfake Detection without Deepfakes super-resolution techniques could lead to the elevation As stated before, one of the most stringent problems in of false alarms, leading models to identify legitimately the field of deepfake detection is that of generalization. enhanced images through these approaches as deepfakes. Indeed, there is ample evidence that detectors tend to
The latter result highlights a problem that could prove learn to efectively recognize deepfake content obtained crippling to traditional deepfake detectors and could pre- through methods used to construct their training set, but vent their deployment in the real world. Indeed, it is fail when they need to classify content obtained through plausible to think that on social networks it will become novel techniques. This leads to a total inadequacy of increasingly common to improve the quality of one’s pho- conventional deepfake detectors in being used in the real tos through Super-Resolution techniques. If this were world. In fact, new deepfake generation techniques are to happen and in parallel the deepfake detectors were continually being created, and it would be impractical unable to understand that these are legitimate images but to retrain the model each time to introduce every single instead end up mistaking them for deepfakes, the number possible method. In the context of synthetic images, this of false alarms would be such as to prevent their efective tendency of deepfake detectors stems from the fact that deployment on a large scale. It is therefore necessary each generator introduces a specific fingerprint into the on the one hand to defend against the malicious use of image[15, 16, 17]. It tends to be invisible to the human super-resolution to disguise artifacts introduced by ma- eye but involves the presence of structured patterns in nipulation techniques but also to make detectors robust the frequency domain (grids, symmetric peaks, halos, so that they are able to recognize legitimately augmented etc.). images. From the observation of this phenomenon, as a fu
In these experiments we used only EDSR[13] as the ture work we are exploring a new training technique basis of our attack but the proposed attack can be con- for deepfake detectors that tries to stimulate the model ducted using diferent types of SR algorithms (such as to recognize the presence of structured patterns in the BSRGAN[14]), and depending on the peculiarities of each, frequency domain and not to learn a specific fingerprint. greater or lesser efectiveness can be achieved on each The preliminary results of this approach can be found in specific deepfake generation method. The choice of the [3]. factor also has an impact in that as it increases, the de- We propose to reproduce prototype structured pattectors’ errors increase but the quality of the image itself terns inspired by what we observed from the fingerprints also deteriorates. Therefore, it is crucial to choose the SR actually introduced by generative patterns of various [2] D. A. Coccomini, R. Caldelli, F. Falchi, C. Gennaro, types. These patterns, in frequency, are injected in pris- On the generalization of deep learning models in tine images and considered as "fake" in the training phase. video deepfake detection, Journal of Imaging (2023). During the training, we show the model pristine images doi:10.3390/jimaging9050089. and others on which a pattern has been applied, indicat- [3] D. A. Coccomini, R. Caldelli, C. Gennaro, G. Fiing them to the model as fakes. ameni, G. Amato, F. Falchi, Deepfake detec
From our preliminary experiments, we demonstrated tion without deepfakes: Generalization via that models trained on this proto-task, are extremely synthetic frequency patterns injection, 2024. efective at identifying synthetic images despite never arXiv:2403.13479. really seeing one in the training phase. [4] D. A. Coccomini, R. Caldelli, G. Falchi, Amato,
The use of synthetic patterns may also be used to sup- F. Falchi, C. Gennaro, Adversarial magnification port traditional training of deepfake detectors, introduc- to deceive deepfake detection through super resing them only occasionally and still maintaining deep- olution (to appear), in: Proceedings of European fakes in the training set. In addition, it will be possible to Conference on Machine Learning and Principles experiment with a virtually infinite number of patterns and Practice of Knowledge Discovery in Databases, by searching for the most efective ones and studying 2023. their impact. [5] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, M. Niessner, Faceforensics++: Learning to detect manipulated facial images, in: Proceed3. Conclusions ings of the IEEE/CVF International Conference on Computer Vision, 2019.
Carrying out research in the field of deepfake detection [6] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, is an increasingly pressing need because of the multitude S. Lin, B. Guo, Swin transformer: Hierarchical of techniques that now make it possible to produce syn- vision transformer using shifted windows, in: 2021 thetic or manipulated content with an increasing degree IEEE/CVF International Conference on Computer of credibility. As shown in our recent research, it is critical to explore both innovative methods to try to improve Vision (ICCV), 2021. doi:10.1109/ICCV48922. the capability of deepfake detectors looking for new train- [7] 2K0.H21e.,X00.Z9h8a6n.g, S. Ren, J. Sun, Deep residual learning approaches and techniques. This could be needed to ing for image recognition, in: 2016 IEEE Conferovercome the pressing problem of generalization. On the ence on Computer Vision and Pattern Recognition other hand, it is important to find new solution to face the possible risks and unexpected situations that these [8] (DCeVePpRfa)k,2e0s,16. d2o0i1:180..1U1R0L9:/CVhPtRtp.s2:/0/1g6it.h9u0b..com/ models might encounter in the real-world that could in- deepfakes/faceswap. validate their potential, such as the adversarial attacks. [9] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt, M. Nießner, Face2face: Real-time face capture and 3.0.1. Acknowledgments reenactment of rgb videos, Commun. ACM (2018). This work was partially supported by the follow- doi:10.1145/3292039. ing projects: Tuscany Health Ecosystem (THE) (CUP [10] K. M., Faceswap, 2017. URL: https://github.com/ B83C22003930001) and SERICS (PE00000014, MUR PNRR MarekKowalski/FaceSwap/. - NextGenerationEU), AI4Media (EC H2020 - n. 951911) [11] L. Li, J. Bao, H. Yang, D. Chen, F. Wen, Faceshifter: and AI4Debunk (Horizon EU n. 101135757), FOSTERER Towards high fidelity and occlusion aware face (Italian MUR PRIN 2022). We acknowledge the CINECA swapping, 2020. arXiv:1912.13457. award under the ISCRA initiative, for the availability of [12] J. Thies, M. Zollhöfer, M. Nießner, Deferred neural high-performance computing resources and support. rendering: Image synthesis using neural textures (2019). doi:10.1145/3306346.3323035. [13] B. Lim, S. Son, H. Kim, S. Nah, K. Mu Lee, EnReferences hanced deep residual networks for single image super-resolution, in: Proceedings of the IEEE Con[1] D. A. Coccomini, R. Caldelli, F. Falchi, C. Gennaro, ference on Computer Vision and Pattern RecogniG. Amato, Cross-forgery analysis of vision trans- tion (CVPR) Workshops, 2017. formers and CNNs for deepfake image detection, [14] K. Zhang, J. Liang, L. Van Gool, R. Timofte, Designin: International Conference on Multimedia Re- ing a practical degradation model for deep blind trieval Workshop, 2022. doi:10.1145/3512732. image super-resolution, in: IEEE International Con3533582. ference on Computer Vision, 2021.
[15] D. A. Coccomini, A. Esuli, F. Falchi, C. Gennaro, G. Amato, Detecting images generated by difusers, 2023. doi:10.48550/arXiv.2303.05275. [16] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, A. A.
Efros, CNN-generated images are surprisingly easy to spot. . . for now, in: IEEE Conf. Comput. Vis.
Pattern Recog., 2020, pp. 8692–8701. doi:10.1109/
CVPR42600.2020.00872. [17] R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi,
K. Nagano, L. Verdoliva, On the detection of synthetic images generated by difusion models, in: Int.
Conf. on Acoustics, Speech and Signal Processing, 2023, pp. 1–5. doi:10.1109/ICASSP49357.2023. 10095167.