Generative AI, mainly through Difusion Models, has revolutionized art creation, blurring the distinction between human and AI-generated art. This study introduces a novel dataset comprising human-made and AI-generated art and employs Deep Learning models (VGG-19, ResNet-50, ViT) to distinguish between them. We also use eXplainable AI techniques to derive insights. Our results highlight the potential of AI to detect machine-generated content, with implications for art authentication.
Art has undergone a profound transformation with the emergence of generative Artificial
Intelligence, notably driven by technologies such as Generative Adversarial Networks [
The distinction between genuine human-made art and its synthetic counterparts carries extensive implications, influencing aspects such as art authentication, valuation, and preservation while igniting debates concerning technology’s role in the creative process. While conventional methods of art connoisseurship traditionally relied on expert human judgment, the rapid evolution of Deep Learning models and the availability of extensive art datasets present new avenues for addressing this challenge. One intriguing approach to detecting instances generated by machines, in fact, involves leveraging the capabilities of machines themselves. This concept is rooted in the idea that the same AI technologies responsible for creating synthetic content can also be employed for their detection and diferentiation from authentic human-made counterparts. Deep Learning and Computer Vision algorithms can also be trained on large datasets containing authentic and AI-generated examples. These models can learn to identify nEvelop-O ∗Corresponding author. subtle patterns, inconsistencies, or artifacts that may indicate machine generation. By utilizing AI-powered classification models, we can automate the process of detecting machine-generated content, achieving both eficiency and accuracy.
Given these considerations, this paper distinguishes between authentic and synthetically
generated artworks. This efort begins with creating a novel dataset and proceeds to conduct
experiments that evaluate well-established Deep Learning algorithms, including VGG-19 [
The rest of this paper is structured as follows. Section 2 reviews the existing literature on recognizing synthetic images. Section 3 introduces our proposed dataset. Section 4 details the classification method proposed for this task. Section 5 provides insights into the experiments conducted in this research. Finally, Sec. 6 concludes the article while highlighting potential directions for future research eforts.
In recent years, there has been a significant surge in interest in Deep Learning-based models,
particularly in image analysis. Notably, the emergence of Difusion Models, as exemplified in
the review paper by Croitoru et al. [
Consequently, various Computer Vision techniques have been harnessed to facilitate the
automated detection of the factors mentioned above. Guo et al. [
In the digital humanities domain, identifying forgeries, often manifesting as synthetic images,
holds paramount significance for art experts. This task tests whether a given artwork is an
authentic painter’s creation or a generative algorithm’s output. Nevertheless, the scrutiny
of fine arts presents demanding challenges due to the intrinsic variability and subjectivity
inherent in the data. Notably, similar objects may be rendered in divergent artistic styles,
while conversely, a singular artwork can evoke disparate emotional reactions among diferent
observers. In response to this complexity, numerous models tailored for artwork analysis have
embraced a multimodal approach, accommodating various data inputs such as images, text
descriptions, or structured knowledge graphs. For example, in [
However, it is noteworthy that a scarcity of scholarly literature addresses the automated
detection of AI-generated artworks. In light of this, the present study contributes by
introducing a novel dataset, which encompasses both authentic pieces of art, obtained through
ℎ [
In order to conduct valuable experiments to recognize synthetic works of art, we designed a new
large-scale dataset consisting of authentic and generated artworks. Specifically, we exploited
two datasets: ℎ [
To collect data, we first extracted all the synthetic images of art within the ArtiFact dataset. This way, we gathered a set of images consisting of 37, 775 synthetic works of art generated by multiple algorithms (e.g., Stable Difusion [ 17] and StyleGAN [18]). Whereas, for authentic artworks, we sampled 50, 000 artworks from ℎ , stratifying this process on the style attribute to avoid extreme class imbalance. Figure 1 shows two images representing each class within our dataset. Interestingly, there appears to be little distinction between the authentic and generated images. This observation underscores the primary motivation driving our study. “Landscape” by Janos Mattis-Teutch “Nude on green sofa”
by M. H. Maxy
This research presents a comprehensive Computer Vision framework centered around binary
classification algorithms. To thoroughly evaluate the characteristics of this task, we conducted
experiments using three cutting-edge techniques: VGG-19 [
Figure 2 depicts the architecture of VGG-19, which we selected due to its capability to acquire hierarchical features from input images. This characteristic allows it to capture complex patterns and representations efectively. Moreover, the VGG-19 architecture is characterized by its simplicity and uniform structure, comprising convolutional and max-pooling layers. In Fig. 3, we present the conceptual design of the ResNet-50 model. This neural network introduces the concept of “residual learning”, incorporating skip connections that facilitate the smoother flow of information throughout the network and address the vanishing gradient problem. Lastly, Fig. 4 provides an overview of the ViT architecture, a state-of-the-art model recognized for its exceptional performance in various Computer Vision tasks, including image classification. ViT’s scalability makes it suitable for tasks involving images of varying sizes, as it allows for adjustments to the patch size and the number of layers, adapting to diferent input dimensions and computational resources.
In all cases, we leverage transfer learning to train the models mentioned above by freezing all feature extraction layers and letting the model optimize just the last fully connected layers, distinguishing whether the given image is an authentic artwork or a synthetic one. Accordingly, once visual information is computed by each backbone, the feature vector is fed into a sigmoidactivated fully connected output layer, representing the probability that the given image is authentic. Then, to optimize the model, we use a binary cross-entropy loss function: ℒ = 1
=1
∑ ℓ ( , ̂ ), where is the number of instances in the training set, is the true label, ̂ is the model outcome, and ℓ ( , ̂ ) = −( log( ̂ ) + (1 − ) log (1 − ̂ )) is the classic binary cross-entropy loss.
We employed Grad-CAM, a technique introduced by Selvaraju et al. in [
We performed some experiments for all the models mentioned above from both quantitative and qualitative points of view. In this section, we describe the experimental setup and discuss the obtained results.
The experiments were conducted using the Google Colab framework. Its resources include an Intel Xeon processor, 12 GB of RAM, and an NVIDIA T4 GPU with 15 GB of VRAM. All the models were implemented using the popular PyTorch library, which is well-known and suited for Computer Vision research.
We adopted an 80/10/10 splitting criterion, stratifying the partitioning on the target class. In this way, we ensured that the data distribution was preserved among training, validation, and test sets. It is worth noting that we used the same splitting to train all three models to have a fair comparison. Furthermore, all the images were preprocessed via center cropping. We resized them to a standard size of 224 × 224 pixels and performed normalization using the mean and standard deviation values of the ImageNet dataset. It is worth noting that all images (real and synthetic) were processed in the same way to conduct a fair experiment. Regarding the optimization stage, we used the Adam optimizer with a learning rate of 10−3 and a batch size of 32. An early stopping callback was introduced in the training loop with a patience of 3 and a learning rate schedule based on reducing it on the plateau to avoid overfitting.
To evaluate the efectiveness of the models, we used diferent performance metrics. Accuracy was calculated to show the percentage of well-classified instances. Furthermore, we calculated precision, recall, and F1-score to evaluate the model’s trustworthiness in identifying the authentic and the generated art pieces precisely.
The quantitative results of the experimental analysis are shown in Table 1. The table shows that all three models are good predictors, with all metrics above 95%. The best model is ViT, but VGG-19 and ResNet-50 also perform well. These results highlight the ability of these models to detect subtle and imperceptible patterns that allow them to correctly distinguish between authentic and synthetic artworks, even though this would be a very challenging task for the naked eye.
Furthermore, in terms of qualitative evaluation, we assessed the models by analyzing their Grad-CAM outputs. Firstly, in Fig. 5, we present examples where the models correctly classified the images. Notably, VGG-19 emphasizes broad, coarse-grained features that span extensive areas within the given image. In contrast, ResNet-50 and ViT focus more on fine-grained details,
Input
VGG-19
ResNet-50
ViT distinguishing whether a patch appears authentic or generated. In most cases, the models highlight image regions containing color discrepancies or lines that deviate from established patterns found elsewhere in the image. These discernible patterns are also noticeable in misclassified images, as depicted in Fig. 6. VGG-19, in these instances, emphasizes nearly half of the image, indicating confusion in its analysis. ResNet-50 directs attention to details in the sky where the pattern does not match the specific patch. Finally, ViT focuses on irregular lines within the eye. These observations highlight the inherent challenges in this task. Factors such as degradation over time or imperfections in the original artwork, such as inaccurately drawn lines by the artist, may contribute to these challenges.
From a conceptual standpoint, the suggested model accentuates image patches that deviate from the overall visual context. It considers broad and subtle features, including those that might not be immediately apparent to the naked eye. This discrepancy can be attributed to artifacts left in the image by generative models, which other algorithms can discern.
Grad-CAM heatmap
In this paper, our focus was recognizing artificial artworks generated by AI systems using Deep Learning models. To achieve this, we curated a novel dataset that included both authentic and generated artworks. We then experimented with various neural network architectures and conducted a comprehensive evaluation, considering both quantitative and qualitative aspects. Specifically, our quantitative analysis revealed that Vision Transformers exhibited strong predictive capabilities for this particular task, surpassing the performance of well-known models like VGG-19 and ResNet-50. Furthermore, it is worth noting that none of the three models misclassified the same test instances. However, upon closer examination of misclassified cases, we acknowledged that subtle imperfections at a fine-grained level could influence the models’ ability to identify synthetic artworks correctly.
Future work could systematically compare the proposed method against other techniques
designed for fake image detection, as the current work is exploratory in nature. Additionally,
analyzing feature importance from a semantic perspective would improve model interpretability
and move beyond pure visual interpretations of activation maps. Also, integrating contextual
cues could help performance, as prior work has shown this modality to be efective for analyzing
artistic metadata [