Robustness for Cross-Depiction Through Neural Style Transfer

Abstract

Modern neural networks models for computer vision are trained on millions of images. The idea is that models are able to increase generalization when the dataset contains well diversified images, e.g. with varied illumination and environmental conditions of the same objects. Generalization is particularly relevant in object detection, especially for what concerns the cross-depiction problem. In this work we explore the use of Neural Style Transfer as a novel technique to morph the original data, with the aim to enhance model generalization. To verify the efect on performances for object detection models, we selected the Faster R-CNN model to be applied on the Pascal VOC 2012 dataset. A number of tests were performed through style variations on images and by tuning Neural Style Transfer parameters to maintain the content of the original images. The experiments showed promising results, which efectively provide a foundation for future studies on cross-depiction via ICYRIME 2023: 8th International Conference of Yearly Reports on Attribution 4.0 International (CC BY 4.0).

1. Introduction

Object detection is a challenging task in computer vision which has a wide range of possible real-life applications, ranging from autonomous driving and healthcare to entertainment [ 1, 2 ]. This problem, while relatively new, has already been tackled in literature with several different approaches [ 3, 4 ]. The solutions are mainly classifiable in conventional methods, which are comprised of three phases (region selection, feature extraction and classification), and deep learning based methods [ 5 ]. The most advanced approaches focus on the use of deep neural networks, in particular convolutional neural networks (CNN), with the most popular solution to object detection being YOLO [ 6 ], developed in the years up to YOLOv8 [ 7 ].

Achieving high performance in this task is fundamental

sics or real-time usage (e.g. for autonomous driving). In order to improve the efectiveness of object detection models, various solutions to enhance generalization in unforeseen situations have been developed, the main

Data augmentation encompasses many diferent basic techniques, such as linear transformations, rotations and lfipping, random cropping, random noise and brightness modulation. By applying these transformations to the

nEvelop-O which is the pivotal concept on which Neural Style Transfer is built on. This process has been called image-based artistic rendering (IB-AR) [ 11 ]. Modern Neural Style for several applications, with some examples being foren- image [ 8 ]. For this reason, diferent data augmentation ones being data augmentation and Neural Style Transfer. age to another. Before the advent of neural networks,

© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Transfer, instead, makes use of two diferent starting imCEUR

ceur-ws.org ages, the content image and the style image. The first one defines the context on which the artistic style depicted in the second must be applied, thus generating a new hybrid image with the content of the content image but the style of the style image. The generation is performed by a CNN, with the first tests being performed with a classic VGG19 model [ 12, 13 ]. From then, a whole taxonomy of algorithms for Neural Style Transfer was developed, broadly divisible in Image-Optimization-Based Online Neural Methods, which rely on multiple executions of the image optimization and gradual online style transfer, and Model-Optimization-Based Ofline Neural Methods, which perform a single forward pass after optimizing the model ofline. Starting from Neural Style Transfer, several sub-applications were derived. Some examples are Visual Style Modeling, which aims at synthesizing textures from images, and Image Reconstruction, which instead tries to reconstruct whole images from extracted fragments. This paper, instead, tackles a particular case of interdisciplinary task between object detection and Neural Style Transfer referred to as the cross-depiction problem [ 14 ]. Cross-depiction consists of recognising visual objects regardless of their form and style, and it’s still an under-researched problem in computer vision.

This translates to the capability of a neural network to correctly identify objects portrayed in artistic styles that are more or less diferent from their realistic representation in photographs. A neural network trained in the usual way will struggle to recognize a dog painted in an abstract way. To perform cross-depiction, the network will have to consider less specific features and focus on the shape of the dog itself, as well as other features that are not necessarily typical of realistic photos. Our aim is to train and fine-tune our object classification model to be more focused on the shape of the objects and on more generic features that would not be considered, or would be considered with a minor weight, in a conventional environment. In this work we show how applying style transfer on a particular dataset with diferent hyperparameters can increase the performance of a model like Faster R-CNN on a object detection classification task.

By augmenting the data already present in the dataset with Neural Style Transfer transformations, the model is made more robust to outliers and edge cases, therefore rendering it applicable to more general situations. In particular, we focus on the application of Faster R-CNN on the Pascal VOC 2012 dataset, performing diferent tests to verify the preservation or improvement of the performances of the model after the application of Neural Style Transfer on the dataset. A subset of the total images was chosen to apply Neural Style Transfer on and be used as a test set. The experiments show that the style variation during training positively afects the performances of object recognition on a dataset of artistic images, cementing our approach as a possible solution to the cross-depiction problem.

2. Related Works

Several data augmentation techniques have been presented in modern deep learning as an eficient solution to improve model performances and limit overfitting during training [ 15 ]. Models, however, require substantial amounts of data in order to learn to classify images correctly, and the inability to provide this data usually correlates with poor performances during inference. The idea of using Neural Style Transfer as a form of data augmentation is not new, and it has already been verified as a domain-agnostic approach, making it suitable for various image classification tasks with several models (ResNet, VGG19 and Inception) [ 16 ]. One of the main problems in the original paper on Neural Style Transfer was the time needed by the algorithm to apply the style transfer, among the longest in all available Neural Style Transfer approaches [ 12, 9 ]. Following papers therefore showed how to increase the speed at which the style transfer is applied to the original image using a feed-forward approach, reducing the strain on the resources available for training purposes [ 17 ]. However, this method is only able to reproduce one style per model, and new, more lfexible models were proposed to solve both problems. The category of Arbitrary-Style-Per-Model algorithms (ASPM MOB-NST) eficiently solves the scalability problem, with also the possibility of completely removing learning limitations through feature transform [ 18 ], but introducing less impressive results compared to more specific approaches [ 19, 20, 21, 18 ]. It has also been veriifed that Neural Style Transfer can be used to reduce bias, and a novel pipeline for Antibody Mediated Rejection classification has provided an implementation faster than current SOTA approaches [22]. One of the most robust choice for object detection is R-CNN, or Region-based Convolutional Neural Networks, which marked a significant breakthrough in object detection performance, outperforming many rival algorithms [23]. The key concept behind Region-based Convolutional Neural Network architectures is region proposals (RPNs), regions in the image that could contain an object of interest, which are then fed to a Convolutional Neural Network, typically a ResNet or a VGG. The extracted features are finally passed to a series of fully connected layers for the final predictions of the classification and the object detection. The largest drawback and bottleneck of the original RCNN architecture is its computational expensiveness, as it requires running the CNN separately for each object proposal. Moreover, the selective search algorithm is fixed, which means that no learning happens at that stage. A whole family of state-of-the-art models spanned from R-CNN to address these issues, with architectures 15–20 such as Fast R-CNN [24] and Faster R-CNN [25] building upon the previous model’s success to improve object detection accuracy and speed. These models replace the separate CNN for each proposal with a shared CNN used to extract features for all the proposals, allowing faster processing. Also, instead of feeding the region proposals to the CNN, the same CNN generates both object proposals and detection. The diference between Fast R-CNN and Faster R-CNN is that the latter, instead of using the slower selective search algorithm on the feature map to draw the region proposals, utilizes a separate network to get the region proposals, further reducing execution time. Models like Faster R-CNN are able to perform relatively well when presented with images that resemble the ones seen during training, showing the capability to generalize and opening to the possibility of being fine-tuned for custom datasets.

3. Implementation

Our work aims at presenting a novel approach and solu

tion for the cross-depiction problem, with Faster R-CNN posed of 1000 classes. In order to be trained on the 20 being a particularly good fit for our task. More precisely, Pascal VOC classes, the model is initialized replacing the the model that we used is the Faster R-CNN ResNet50 last layer responsible for the regions of interest (RoI) with FPN from the Torchvision models, which combines the a new one that has 21 output features, the 20 classes plus ResNet50 model as feature extraction backbone with a the background. Before starting the fine-tuning, which is Feature Pyramid Network (FPN). This way, object de- performed on the Pascal VOC dataset, a pre-processing tection performance is improved by generating a set of phase in which the images are resized to the standard feature maps at diferent scales, which helps the model format of 256x256 pixels and the pixel values are normaldetect objects of varying sizes and aspect ratios. The ized in the range [ 0, 1 ] was necessary. The dimensions experiments performed in our work are aimed at under- of the bounding boxes’ labels have also been adapted standing how a CNN performs on unusual abstract im- accordingly to keep the ratio with the resized images. Afages under various conditions, and how much it is able to ter this, the network goes through the fine-tuning. This generalize in the presence of non-realistic features, with process is performed on 80% of the dataset, leaving the the goal of achieving object detection in artwork-like im- remaining 20% for evaluating it. The model uses stochasages. This would present a solution to the cross-depiction tic gradient descent (SGD) as optimizer, with a starting problem by making an object identifiable regardless of learning rate of 0.001, a momentum of 0.9, and weight the style of the image. To perform the task, we employed decay of 0.0005. The fine-tuning lasts 10 epochs. The the Neural Style Transfer methods previously described learning rate first goes through a warm-up period of 1000 to widely augment a well known dataset, Pascal VOC iterations in order to get to the starting learning rate in [26], used as a standard benchmark for evaluating object a gradual way. Then, it is adjusted over the fine-tuning detection models. In particular, we used the 2012 version, period following a learning rate scheduler, with step size the latest available. It contains 17,125 images annotated 4 and gamma 0.1, which are the values that best optifor object detection, as well as object classification and mized the performances while also avoiding overfitting. image segmentation. The images consist of 20 object We first created a variety of subsets of the initial Pascal classes, including animals, vehicles, and common house- VOC dataset through Neural Style Transfer. Specifically, hold items. Some examples of images contained in the 12 diferent artistic styles of diferent time periods were dataset are shown in Figure 1. A similar data augmenta- selected (e.g. Cubism or Puntinism), and the NST was tion has already been presented in previous works [27], applied in two versions, one with a lighter stylization but we won’t focus solely on people recognition and the and another with a stronger one. The intuition is that people class, instead employing the whole dataset. by performing the NST, one is able to produce a dataset

Faster R-CNN ResNet50 FPN is deployed in its ver- of a desired style which is already labelled, since the sion pre-trained on ImageNet [28], a large-scale image position and the dimension of the bounding boxes of database widely used in computer vision research, com- the objects remain unchanged. The parameters used to ability to detect objects of diferent sizes. The model pre-trained and fine-tuned only on the original Pascal VOC performs very poorly, with an AP50 of 0.318 on the light stylization and with an AP50 0.147 on the strong one. With just 10 epochs of training on the stylized images, however, the evaluation gets to AP50 0.549 on the light stylization and 0.356 on the strong one, which is already a good result compared to similar experiments [29]. We found that the best way to train the model is to conduct the training with a group of subsets of stylized images and a group of normal photographs at the same time. This keeps the object recognition grounded to a certain degree of reality, reducing weight assignment to some features and maintaining a slightly better ability of generalization. The AP50 score with the mixed training Figure 2: Examples of images from the Pascal VOC dataset set is 0.553, with respect to the 0.530 of the model trained with NST applied on them. The images on the left are the only on the stylized images, and a better score over the style images and the images on top are the images from the original test set was also maintained. It is possible to Pascal VOC dataset. The combinations are the results of NST achieve even better results by training for more epochs, application. but to avoid overfitting on the training images we will use the fine-tuned model weights with 10 epochs as a starting point. The next experiments aimed at evaluatobtain the enhanced datasets are total_ steps = 35 and ing the performances of the aforementioned models over learning_rate = 0.02 for the lighter stylization one and other images of diferent styles. We trained the model total_steps = 55 and learning_rate = 0.05 for the stronger on eight of the subsets, leaving the remaining four styles stylization one, with alpha = 0.8 and beta = 0.3 in both for the test, with subsets composed of images unseen versions. These values have been chosen as a compro- during the training. The final results evaluated in AP50 mise between a recognizable adaptation of the applied are 0.525 as the average of the scores obtained with light style and the preservation of the objects in the image, NST, and 0.247 with strong NST. In both cases the model although in the stronger version objects of smaller di- fine-tuned on the light stylization has been used. For mensions are often distorted and unrecognizable. Some the model fine-tuned on a training set of strong stylexamples of results of NST application to the Pascal VOC ization, instead, we got an AP50 of 0.519 and 0.316 on dataset are shown in Figure 2. Afterwards, we performed light and strong NST respectively. These results show several experiments to verify how our model acts when that correspondence between the fine-tuned models and lighter or stronger stylized images are fed to it. We also the training set positively reflects on the performance study how it performs when trained in diferent ways on of the object detection. In Table 1 we show an overview the previously produced subsets of stylized images, eval- of the results for each subset obtained from the model uating it both on the light and the strong stylization, and ifne-tuned on the strong stylization and tested on light both on seen and unseen styles. To evaluate the results stylization comparing the diferent metrics (AP, AP50 of the experiments, we used a group of average precision and AP75). It is possible to observe that after this type and recall measurements that can estimate the perfor- of fine-tuning the model obtains a certain degree of genmance of the object detection at various levels of overlap eralization, showing detection performances on the last between the predicted bounding boxes and ground truth four unknown styles which are in line with the results ones. The standard metric is the mean average preci- obtained for the other classes. sion (mAP) with a 50% bounding box overlap with the labelled box [26]. The performance has been evaluated 4. Conclusion with average precision AP, AP50 and AP75 as they are defined in the COCO detection evaluation metrics. AP The analyzed results confirm the concrete possibility to is the average precision value at diferent thresholds of achieve data augmentation on images with varied artistic intersection over union (IoU), respectively 0.50 for AP50, styles for any given dataset. We also demonstrate that 0.75 for AP75 and 0.50 to 0.95 for AP, evaluated for maxi- a CNN is able to generalize under the presence of difmum detection of 100% for all areas. Separate AP scores ferent features derived from diferent styles, therefore are also available for diferent area sizes, divided into confirming the efectiveness of this method. This opens small, medium, and large objects, to measure the model’s up to several possible applications, such as performing mass object detection in large datasets such as museum collections or online databases of artworks (both amateur and professional), resulting in an automatic extraction of metadata related to the identification and localization of objects. This could also be extended to the automatic creation of new datasets of non-photographic images for object detection. In such a process, the analyzed technique can be employed as a starting point to outline the bounding boxes of the objects in the scene, which can be then verified and adjusted. Many fields of possible applications can derive from the ability of neural network models to accurately perform object detection in whatever form of non-realistic representation. By providing a model the possibility to track objects and monitor their behaviours in diferent environments with a coherent artistic style, one could apply it also to animation, videogames, etc. Another interesting application coming from the achievement of higher levels of abstraction in object detection is the capacity of future AI agents, like generative agents, to behave socially and simulate human patterns [30, 31, 32, 33],. This approach, if extended to textual tasks, can also allow agents to perform tasks which require the understanding of unlabelled and unseen representations of various types, for example to navigate online forums or any kind of website to perform data scraping in a more comprehensive way. With regards to possible future works, our biggest limitation was the lack of big annotated datasets of artwork images for object detection, and it would be likewise insightful to see the results of tests of a model trained on such images as well. Finally, it would be also useful to extend the style transfer method illustrated here to other computer vision tasks, such as image segmentation or pose estimation, and see how much of what has been commented also applies to these problems.

Acknowledgements

This work has been developed at is.Lab() Intelligent Sys

tems Laboratory at the Department of Computer, Control, and Management Engineering, Sapienza University of Rome (https:// islab.diag.uniroma1.it). The work has also been partially supported from Italian Ministerial grant PRIN 2022 “ISIDE: Intelligent Systems for Infrastructural Diagnosis in smart-concretE”, n. 2022S88WAY - CUP B53D2301318, and by the Age-It: Ageing Well in an ageing society project, task 9.4.1 work package 4 spoke 9, within topic 8 extended partnership 8, under the National Recovery and Resilience Plan (PNRR), Mission 4 Component 2 Investment 1.3—Call for tender No. 1557 of 11/10/2022 of Italian Ministry of University and Research funded by the European Union—NextGenerationEU, CUP B53C22004090006.

References

[1] V. Ponzi , S. Russo , V. Bianco , C. Napoli , A. Wajda , Psychoeducative social robots for an healthier lifestyle using artificial intelligence: a case-study , volume 3118 , 2021 , pp. 26 - 33 .

[2] V. Marcotrigiano , G. D. Stingi , S. Fregnan , P. Magarelli , P. Pasquale , S. Russo , G. B. Orsi , M. T. Montagna , C. Napoli , C. Napoli , An integrated control plan in primary schools: Results of a field investigation on nutritional and hygienic features in the apulia region (southern italy) , Nutrients 13 ( 2021 ). doi: 10 .3390/nu13093006.

[3] F. Bonanno , G. Capizzi, C. Napoli , Some remarks on the application of rnn and prnn for the chargedischarge simulation of advanced lithium-ions battery energy storage , 2012 , pp. 941 - 945 . doi: 10 . 1109/SPEEDAM. 2012 . 6264500 .

[4] G. Capizzi , F. Bonanno , C. Napoli , A wavelet based prediction of wind and solar energy for longterm simulation of integrated generation systems , 2010 , pp. 586 - 592 . doi: 10 .1109/SPEEDAM. 2010 . 5542259 .

[5] R. Kaur , S. Singh , A comprehensive review of object detection with deep learning , Digital Signal Processing 132 ( 2023 ) 103812 . URL: https://www.sciencedirect.com/science/ article/pii/S1051200422004298. doi:https: //doi.org/10.1016/j.dsp. 2022 . 103812 .

[6] J. Redmon , S. Divvala , R. Girshick , A. Farhadi , You only look once: Unified, real-time object detection , arXiv ( 2015 ). arXiv: 1506 . 02640 .

[7] D. Reis , J. Kupec , J. Hong , A. Daoudi , Real-time lfying object detection with yolov8, arXiv ( 2023 ). arXiv: 2305 . 09972 .

[8] H. Xu , Y. Ma , H. Liu, D. Deb , H. Liu, J. Tang , A. K. Jain , Adversarial attacks and defenses in [22] X. Zheng , T. Chalasani , K. Ghosal , S. Lutz , A. Smolic, images, graphs and text: A review, arXiv ( 2019 ). Stada: Style transfer as data augmentation , in: arXiv:1909.08072. Proceedings of the 14th International Joint Confer-

[9] Y. Jing , Y. Yang , Z. Feng , J. Ye , Y. Yu , M. Song , Neu- ence on Computer Vision , Imaging and Computer ral style transfer: A review , IEEE Transactions Graphics Theory and Applications , SCITEPRESS - on Visualization and Computer Graphics 26 ( 2020 ) Science and Technology Publications, 2019 . URL: 3365 - 3385 . doi: 10 .1109/TVCG. 2019 . 2921336 . http://dx.doi.org/10.5220/0007353401070114. doi:10.

[10] F. Bonanno , G. Capizzi, G. L. Sciuto , C. Napoli, 5220 /0007353401070114. Wavelet recurrent neural network with semi- [23] R. Girshick , J. Donahue , T. Darrell , J. Malik, parametric input data preprocessing for micro-wind Rich feature hierarchies for accurate object detecpower forecasting in integrated generation sys- tion and semantic segmentation , arXiv ( 2014 ). tems, 2015 , pp. 602 - 609 . doi: 10 .1109/ICCEP. arXiv: 1311 . 2524 . 2015 . 7177554 . [24] R. Girshick , Fast r-cnn, arXiv ( 2015 ).

[11] J. E. Kyprianidis , J. Collomosse , T. Wang , T. Isenberg, arXiv: 1504 .08083. State of the ”art”: A taxonomy of artistic stylization [25] S. Ren , K. He , R. Girshick , J. Sun , Faster r-cnn: techniques for images and video, IEEE Transactions Towards real-time object detection with region proon Visualization and Computer Graphics 19 ( 2013 ) posal networks , arXiv ( 2016 ). arXiv:1506.01497 . 866 - 885 . doi: 10 .1109/TVCG. 2012 . 160 . [26] M. Everingham , L. Van Gool , C. K. I. Williams ,

[12] L. A. Gatys , A. S. Ecker , M. Bethge , A neu- J. Winn , A. Zisserman , The Pascal Visual Obral algorithm of artistic style , arXiv ( 2015 ). ject Classes (VOC) challenge , International JourarXiv:1508.06576. nal of Computer Vision 88 ( 2010 ) 303 - 338 . URL:

[13] S. Pepe , S. Tedeschi , N. Brandizzi , S. Russo , L. Ioc- https://doi.org/10.1007/s11263-009-0275-4. doi:10. chi , C. Napoli, Human attention assessment us- 1007/s11263-009-0275-4 . ing a machine learning approach with gan-based [ 27] D. Kadish , S. Risi , A. S. Løvlie , Improving object data augmentation technique trained using a cus- detection in art images using only style transfer, tom dataset , OBM Neurobiology 6 ( 2022 ). doi:10. arXiv ( 2021 ). arXiv:2102.06529 . 21926/obm.neurobiol. 2204139 . [28] J. Deng , W. Dong , R. Socher , L.-J. Li , K. Li , L . Fei-

[14] H. Cai , Q. Wu , T. Corradi , P. Hall, The cross- Fei, Imagenet: A large-scale hierarchical image depiction problem: Computer vision algorithms for database , in: 2009 IEEE Conference on Computer recognising objects in artwork and in photographs , Vision and Pattern Recognition , 2009 , pp. 248 - 255 . 2015 . arXiv: 1505 .00110. doi: 10 .1109/CVPR. 2009 . 5206848 .

[15] T. Kumar , A. Mileo , R. Brennan , M. Bendechache , [29] N. Gonthier , S. Ladjal , Y. Gousseau , Multiple inImage data augmentation approaches: A com- stance learning on deep features for weakly superprehensive survey and future directions, 2023. vised object detection with extreme domain shifts , arXiv:2301.02830. Computer Vision and Image Understanding 214

[16] A. Krizhevsky , I. Sutskever, G. E. Hinton , Imagenet ( 2022 ) 103299 . URL: http://dx.doi.org/10.1016/j.cviu. classification with deep convolutional neural net- 2021 .103299. doi: 10 .1016/j.cviu. 2021 . 103299 . works, Communications of the ACM 60 ( 2017 ) [30] N. Brandizzi , S. Russo , G. Galati, C. Napoli, Address84 - 90 . doi: 10 .1145/3065386. ing vehicle sharing through behavioral analysis: A

[17] J. Johnson , A. Alahi , L. Fei-Fei , Perceptual losses for solution to user clustering using recency-frequencyreal-time style transfer and super-resolution, arXiv monetary and vehicle relocation based on neigh( 2016 ). arXiv: 1603 .08155. borhood splits, Information (Switzerland) 13 ( 2022 ).

[18] Y. Li , C. Fang , J. Yang , Z. Wang , X. Lu , M.-H. doi: 10 .3390/info13110511. Yang, Universal style transfer via feature trans- [31] G. De Magistris , M. Romano , J. Starczewski, forms, arXiv ( 2017 ). arXiv: 1705 .08086. C. Napoli , A novel dwt-based encoder for human

[19] T. Q. Chen , M. Schmidt , Fast patch-based pose estimation , volume 3360 , 2022 , pp. 33 - 40 . style transfer of arbitrary style , arXiv ( 2016 ). [32] J. S. Park , J. C. O'Brien , C. J. Cai , M. R. Morris , arXiv: 1612 .04337. P. Liang, M. S. Bernstein , Generative agents: Inter-

[20] X. Huang , S. Belongie , Arbitrary style transfer active simulacra of human behavior, arXiv (2023). in real-time with adaptive instance normalization , arXiv:2304.03442 . arXiv ( 2017 ). arXiv: 1703 . 06868 . [33] G. De Magistris , R. Caprari , G. Castro, S. Russo,

[21] G. Ghiasi , H. Lee , M. Kudlur , V. Dumoulin , J. Shlens , L. Iocchi , D. Nardi , C. Napoli , Vision-based holisExploring the structure of a real-time, arbitrary tic scene understanding for context-aware humanneural artistic stylization network, arXiv ( 2017 ). robot interaction 13196 LNAI ( 2022 ) 310 - 325 . arXiv: 1705 .06830. doi: 10 .1007/978-3- 031 -08421-8_ 21 .