=Paper=
{{Paper
|id=Vol-3762/503
|storemode=property
|title=Robustness and Generalization of Synthetic Images Detectors
|pdfUrl=https://ceur-ws.org/Vol-3762/503.pdf
|volume=Vol-3762
|authors=Davide Alessandro Coccomini,Roberto Caldelli,Claudio Gennaro,Giuseppe Fiameni,Giuseppe Amato,Fabrizio Falchi
|dblpUrl=https://dblp.org/rec/conf/ital-ia/CoccominiCGF0F24
}}
==Robustness and Generalization of Synthetic Images Detectors==
https://ceur-ws.org/Vol-3762/503.pdf
Robustness and Generalization of Synthetic Images
Detectors
Davide Alessandro Coccomini1,2 , Roberto Caldelli3,4 , Claudio Gennaro1 , Giuseppe Fiameni5 ,
Giuseppe Amato1 and Fabrizio Falchi1
1
ISTI-CNR, Pisa, Italy
2
University of Pisa, Pisa, Italy
3
CNIT, Florence, Italy
4
Universitas Mercatorum, Rome, Italy
5
NVIDIA AI Technology Center, Italy
Abstract
In recent times, the increasing spread of synthetic media, known as deepfakes has been made possible by the rapid progress
in artificial intelligence technologies, especially deep learning algorithms. Growing worries about the increasing availability
and believability of deepfakes have spurred researchers to concentrate on developing methods to detect them. In this field
researchers at ISTI CNR’s AIMH Lab, in collaboration with researchers from other organizations, have conducted research,
investigations, and projects to contribute to combating this trend, exploring new solutions and threats. This article summarizes
the most recent efforts made in this area by our researchers and in collaboration with other institutions and experts.
Keywords
Deepfake Detection, Deep Learning, Super Resolution
1. Introduction other problem is that of adversarial attacks, strategies of
camouflaging traces, enhancing fake content or ad-hoc
Deepfakes and synthetic media are becoming more preva- manipulations designed to fool the detector, which can
lent and realistic day by day, presenting society with an be used to make detection even more complex. Deepfake
increasingly urgent challenge, learning to distinguish detection models must therefore be designed so that they
reality from fiction effectively. These fake content can provide a high degree of robustness to possible adver-
and are continually being used to spread disinformation, sarial attacks and also be able to effectively distinguish
create smear campaigns, and manipulate reality with po- deepfakes without raising false alarms. For this reason,
tentially devastating impacts for anyone who may end AIMH Lab at ISTI CNR has carried out numerous re-
up a victim. To contrast this phenomenon, research has search attempts to explore new innovative techniques to
been conducted in recent years creating detectors, often advance this field but also to highlight possible hidden
based on deep learning techniques, that can classify a dangers that may damage the efforts made in previous
piece of content (such as an image) as realistic or fake. research, representing dangers that detection systems
Despite many efforts, this discrimination capability is may encounter in the real world. In particular, this paper
still insufficient today with many open problems in the summarizes the efforts made in [3] and [4].
field of deepfake detection. One example above all is
that of generalization[1, 2]. In fact, deepfake detectors,
although particularly effective in detecting images gener- 2. Research Works in Deepfake
ated or manipulated by the same methods they are trained Detection
on, fail when using different and novel techniques. An-
In this section, we present our most recent works in the
Ital-IA 2024: 4th National Conference on Artificial Intelligence, orga-
nized by CINI, May 29-30, 2024, Naples, Italy
field of Deepfake Detection, highlighting the contribu-
*
Davide Alessandro Coccomini tions and discoveries made.
$ davidealessandro.coccomini@isti.cnr.it (D. A. Coccomini);
roberto.caldelli@unifi.it (R. Caldelli); claudio.gennaro@isti.cnr.it
(C. Gennaro); gfiameni@nvidia.com (G. Fiameni); 2.1. Super-Resolution as an Adversarial
giuseppe.amato@isti.cnr.it (G. Amato); fabrizio.falchi@isti.cnr.it Attack for Deepfake Detection
(F. Falchi)
0000-0002-0755-6154 (D. A. Coccomini); 0000-0003-3471-1196 Super-resolution (SR) algorithms are a set of techniques
(R. Caldelli); 0000-0002-3715-149X (C. Gennaro); designed to improve the resolution of an image. Starting
0000-0001-8687-6609 (G. Fiameni); 0000-0003-0171-4315 with a low-resolution one, through deep learning tech-
(G. Amato); 0000-0001-6258-5313 (F. Falchi)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License niques, it is scaled up to a higher resolution. During this
Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Model Forgery Attacked Accuracy ↑ FNR ↓ FPR ↓ AUC ↑
✖ 95.3 5.9 3.6 99.1
Deepfakes
✓ 90.7 6.1 12.4 97.4
✖ 87.1 12.9 12.8 94.9
NeuralTextures
✓ 81.5 13.2 23.9 90.4
✖ 95.2 6.3 3.3 98.9
Swin Face2Face
✓ 87.0 24.4 1.7 96.1
✖ 95.2 4.9 4.6 98.6
FaceSwap
✓ 86.4 21.9 5.3 93.9
✖ 94.4 7.2 4.1 98.7
FaceShifter
✓ 89.0 18.9 3.1 97.4
✖ 95.6 5.5 3.2 99.2
Deepfakes
✓ 95.0 6.9 10.1 98.9
✖ 87.1 12.9 12.8 94.9
NeuralTextures
✓ 81.5 13.2 23.9 90.4
✖ 95.9 5.0 3.2 98.9
Resnet Face2Face
✓ 91.2 14.4 4.7 97.6
✖ 95.9 6.4 1.9 99.1
FaceSwap
✓ 88.5 21.1 2.6 95.6
✖ 95.3 6.1 3.4 98.8
FaceShifter
✓ 85.9 24.8 3.3 95.2
Table 1
Evaluation on Faceforensics++[5] test set. Each set is composed of half pristine and half fake images. The Attacked column
indicates if the SR-attack has been applied to the images. The attack is applied to fake and pristine images.
process, some aspects of the image may change. For ex-
ample, some previously visible details may become more
blurred, totally disappear, or, conversely, be emphasized
and brought to light. Deepfake generation algorithms
commonly tend to introduce some more or less visible
artifacts. In the case of human faces, these artifacts can
be, for example, anomalies in pupils, contours of lips,
eyes or ears, or accessories. Typically, deepfake detection
models learn to recognize the specific anomalies intro-
duced in manipulated images and, because of them, can
discriminate between pristine and fake content. In [4],
we explored whether Super Resolution techniques can
be used as an adversarial attack to camouflage artifacts
introduced by deepfake generations approaches.
To do this, we proposed an SR-attack pipeline, whose
purpose is to disguise artifacts present in deepfake images
while still trying to keep the appearance as unaltered as Figure 1: Example of the impact of SR attack on fake images.
possible. The pipeline begins with the detection of a face On the left, an example of manipulated face with a zoom on
from the deepfake image that is subsequently downscaled the artifacts around the mouth. On the right the same face
but after the application of the SR-attack. From the zoom on
by a factor 𝐾 using interpolation techniques. The result-
the second one it can be seen how the artifacts are drastically
ing image is restored to its initial resolution through a smoothed.
super-resolution approach. The resulting face can even-
tually be reinserted into the original image, resulting in
the camouflaged image.
An example of the impact of the proposed SR-attack one we often rely on observing these artifacts and oddi-
is shown in Figure 1. As the figure shows, the attack ties that can be introduced by the manipulation process.
leads to the removal of artifacts introduced by deepfake The fact that super-resolution leads to their removal or
generation techniques, such as noise around the mouth, attenuation qualifies it as a potentially effective attack
and thus makes their detection extremely complex. In against deepfake detectors but also against the human
fact, to distinguish a counterfeit image from a pristine eye itself.
The usage of the SR-attack conduct to a blurring effect algorithm and the value of the 𝐾 factor appropriately in
on the artifacts introduced in the fake images and this order to achieve maximum effectiveness from the attack.
makes them more difficult to detect. This is pretty evident
in terms of performance; in fact, the use of the attack 2.2. Future Works
drastically degrades the performance of classifiers trained
to do deepfake detection. In this section we expose some of the future works we are
Table 1 shows the accuracies of a Swin Transformer[6] working on either as extensions of previously presented
and a Resnet50[7] on images manipulated with different works or as new applications and solutions for effective
techniques, considering them before and after SR-attack. deepfake detection.
The dataset used is FaceForensics++[5] and the deepfake
generation methods considered are Deepfakes[8], 2.2.1. Robustness of Deepfake Detectors
Face2Face[9], FaceSwap[10], FaceShifter[11] and
NeuralTextures[12]. This allows us to evaluate the The fact that Deepfake Detectors are susceptible to the
effect of the super-resolution attack on different types use of SR techniques on images, whether fake or pristine,
of manipulations, thus highlighting on which it is exposes a serious problem in their use in the real world
more or less effective. Both the models are trained and therefore requires that more studies be conducted to
on the FaceForensics++ training set considering for make them robust to this kind of content. It is necessary
the construction of the fake class, the same deepfake to find an effective way to make the models robust to
generation method used for the test. this attack for example by introducing super-resolution
According to our experiments, for both the considered as data augmentation during training.
models, when images are attacked with the proposed The attack itself can also be further explored and im-
approach, there is an increase in False Negative Rate, par- proved by going to identify the optimal 𝐾 value and
ticularly on some methods namely Face2Face, FaceSwap corresponding SR method as well as experimenting with
and FaceShifter. Others, however, are found to be more different strategies for applying the attack, such as focus-
robust to attack, namely Deepfakes and NeuralTextures ing on a frame rather than a detected face.
on which, however, there is an increase in False Positives.
This behavior indicates that in some cases, the use of 2.2.2. Deepfake Detection without Deepfakes
super-resolution techniques could lead to the elevation As stated before, one of the most stringent problems in
of false alarms, leading models to identify legitimately the field of deepfake detection is that of generalization.
enhanced images through these approaches as deepfakes. Indeed, there is ample evidence that detectors tend to
The latter result highlights a problem that could prove learn to effectively recognize deepfake content obtained
crippling to traditional deepfake detectors and could pre- through methods used to construct their training set, but
vent their deployment in the real world. Indeed, it is fail when they need to classify content obtained through
plausible to think that on social networks it will become novel techniques. This leads to a total inadequacy of
increasingly common to improve the quality of one’s pho- conventional deepfake detectors in being used in the real
tos through Super-Resolution techniques. If this were world. In fact, new deepfake generation techniques are
to happen and in parallel the deepfake detectors were continually being created, and it would be impractical
unable to understand that these are legitimate images but to retrain the model each time to introduce every single
instead end up mistaking them for deepfakes, the number possible method. In the context of synthetic images, this
of false alarms would be such as to prevent their effective tendency of deepfake detectors stems from the fact that
deployment on a large scale. It is therefore necessary each generator introduces a specific fingerprint into the
on the one hand to defend against the malicious use of image[15, 16, 17]. It tends to be invisible to the human
super-resolution to disguise artifacts introduced by ma- eye but involves the presence of structured patterns in
nipulation techniques but also to make detectors robust the frequency domain (grids, symmetric peaks, halos,
so that they are able to recognize legitimately augmented etc.).
images. From the observation of this phenomenon, as a fu-
In these experiments we used only EDSR[13] as the ture work we are exploring a new training technique
basis of our attack but the proposed attack can be con- for deepfake detectors that tries to stimulate the model
ducted using different types of SR algorithms (such as to recognize the presence of structured patterns in the
BSRGAN[14]), and depending on the peculiarities of each, frequency domain and not to learn a specific fingerprint.
greater or lesser effectiveness can be achieved on each The preliminary results of this approach can be found in
specific deepfake generation method. The choice of the [3].
𝐾 factor also has an impact in that as it increases, the de- We propose to reproduce prototype structured pat-
tectors’ errors increase but the quality of the image itself terns inspired by what we observed from the fingerprints
also deteriorates. Therefore, it is crucial to choose the SR
actually introduced by generative patterns of various [2] D. A. Coccomini, R. Caldelli, F. Falchi, C. Gennaro,
types. These patterns, in frequency, are injected in pris- On the generalization of deep learning models in
tine images and considered as "fake" in the training phase. video deepfake detection, Journal of Imaging (2023).
During the training, we show the model pristine images doi:10.3390/jimaging9050089.
and others on which a pattern has been applied, indicat- [3] D. A. Coccomini, R. Caldelli, C. Gennaro, G. Fi-
ing them to the model as fakes. ameni, G. Amato, F. Falchi, Deepfake detec-
From our preliminary experiments, we demonstrated tion without deepfakes: Generalization via
that models trained on this proto-task, are extremely synthetic frequency patterns injection, 2024.
effective at identifying synthetic images despite never arXiv:2403.13479.
really seeing one in the training phase. [4] D. A. Coccomini, R. Caldelli, G. Falchi, Amato,
The use of synthetic patterns may also be used to sup- F. Falchi, C. Gennaro, Adversarial magnification
port traditional training of deepfake detectors, introduc- to deceive deepfake detection through super res-
ing them only occasionally and still maintaining deep- olution (to appear), in: Proceedings of European
fakes in the training set. In addition, it will be possible to Conference on Machine Learning and Principles
experiment with a virtually infinite number of patterns and Practice of Knowledge Discovery in Databases,
by searching for the most effective ones and studying 2023.
their impact. [5] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess,
J. Thies, M. Niessner, Faceforensics++: Learning
to detect manipulated facial images, in: Proceed-
3. Conclusions ings of the IEEE/CVF International Conference on
Computer Vision, 2019.
Carrying out research in the field of deepfake detection
[6] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang,
is an increasingly pressing need because of the multitude
S. Lin, B. Guo, Swin transformer: Hierarchical
of techniques that now make it possible to produce syn-
vision transformer using shifted windows, in: 2021
thetic or manipulated content with an increasing degree
IEEE/CVF International Conference on Computer
of credibility. As shown in our recent research, it is criti-
Vision (ICCV), 2021. doi:10.1109/ICCV48922.
cal to explore both innovative methods to try to improve
2021.00986.
the capability of deepfake detectors looking for new train-
[7] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learn-
ing approaches and techniques. This could be needed to
ing for image recognition, in: 2016 IEEE Confer-
overcome the pressing problem of generalization. On the
ence on Computer Vision and Pattern Recognition
other hand, it is important to find new solution to face
(CVPR), 2016. doi:10.1109/CVPR.2016.90.
the possible risks and unexpected situations that these
[8] Deepfakes, 2018. URL: https://github.com/
models might encounter in the real-world that could in-
deepfakes/faceswap.
validate their potential, such as the adversarial attacks.
[9] J. Thies, M. Zollhöfer, M. Stamminger, C. Theobalt,
M. Nießner, Face2face: Real-time face capture and
3.0.1. Acknowledgments reenactment of rgb videos, Commun. ACM (2018).
This work was partially supported by the follow- doi:10.1145/3292039.
ing projects: Tuscany Health Ecosystem (THE) (CUP [10] K. M., Faceswap, 2017. URL: https://github.com/
B83C22003930001) and SERICS (PE00000014, MUR PNRR MarekKowalski/FaceSwap/.
- NextGenerationEU), AI4Media (EC H2020 - n. 951911) [11] L. Li, J. Bao, H. Yang, D. Chen, F. Wen, Faceshifter:
and AI4Debunk (Horizon EU n. 101135757), FOSTERER Towards high fidelity and occlusion aware face
(Italian MUR PRIN 2022). We acknowledge the CINECA swapping, 2020. arXiv:1912.13457.
award under the ISCRA initiative, for the availability of [12] J. Thies, M. Zollhöfer, M. Nießner, Deferred neural
high-performance computing resources and support. rendering: Image synthesis using neural textures
(2019). doi:10.1145/3306346.3323035.
[13] B. Lim, S. Son, H. Kim, S. Nah, K. Mu Lee, En-
References hanced deep residual networks for single image
super-resolution, in: Proceedings of the IEEE Con-
[1] D. A. Coccomini, R. Caldelli, F. Falchi, C. Gennaro, ference on Computer Vision and Pattern Recogni-
G. Amato, Cross-forgery analysis of vision trans- tion (CVPR) Workshops, 2017.
formers and CNNs for deepfake image detection, [14] K. Zhang, J. Liang, L. Van Gool, R. Timofte, Design-
in: International Conference on Multimedia Re- ing a practical degradation model for deep blind
trieval Workshop, 2022. doi:10.1145/3512732. image super-resolution, in: IEEE International Con-
3533582. ference on Computer Vision, 2021.
[15] D. A. Coccomini, A. Esuli, F. Falchi, C. Gennaro,
G. Amato, Detecting images generated by diffusers,
2023. doi:10.48550/arXiv.2303.05275.
[16] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, A. A.
Efros, CNN-generated images are surprisingly easy
to spot. . . for now, in: IEEE Conf. Comput. Vis.
Pattern Recog., 2020, pp. 8692–8701. doi:10.1109/
CVPR42600.2020.00872.
[17] R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi,
K. Nagano, L. Verdoliva, On the detection of syn-
thetic images generated by diffusion models, in: Int.
Conf. on Acoustics, Speech and Signal Processing,
2023, pp. 1–5. doi:10.1109/ICASSP49357.2023.
10095167.