In this paper a novel approach for semantically robust unpaired image translation is presented. CLIPTraVeLGAN replaces the Siamese network in TraVeLGAN with a contrastively pretrained language-image model (CLIP) with frozen weights. This approach significantly simplifies the model selection and training process of TraVeLGAN, making it more robust and easier to use.
Currently generative deep learning has become the most promising direction in the development of
IT technologies, in which modern generative neural network models are being developed. There is a
lot of research going on in this rapidly growing field of machine learning, and a large proportion of it
is focused on generative adversarial networks [
Semantic robustness is an essential aspect of image translation. In the context of unpaired image
translation, semantic robustness is particularly important to ensure that the translations are accurate
and meaningful. Image translation from one domain to another is a challenging task that requires the
generated image to belong to the target domain while also retaining the individuality of the input
image. TraVeLGAN [
Our approach aims to solve the problems associated with cooperative learning in TraVeLGAN. In
TraVeLGAN, the generator and the Siamese network have the same goal, which leads to additional
difficulties in determining the effectiveness of each solution. Our approach eliminates the need for
choosing and training a Siamese network and thus avoids these difficulties. At the same time, the
generator still receives the TraVeL loss as proposed in [
The transfer of knowledge from CLIP to CLIPTraVeLGAN enables the generator to understand the relationships between words and images without any additional training. In this paper, we present the results of our experiments and compare CLIPTraVeLGAN with the original TraVeLGAN. Our results show that CLIPTraVeLGAN outperforms TraVeLGAN in terms of both stability and quality of the generated images. This paper is a contribution to the field of image translation and provides a promising new direction for further research.
The effectiveness of our proposed method, CLIPTraVeLGAN, is evaluated on a benchmark dataset for unpaired image translation. The results are compared with other methods to demonstrate the performance of our method.
Our code is available on www.kaggle.com/code/unfriendlyai/cliptravelgan-gta-cityscapes
The field of image translation has seen significant advancements in recent years with the
introduction of Generative Adversarial Networks. One such approach, CycleGAN [
Thus, this task of unpaired image translation consists of two components: the generated image must be a member of the target domain and have the individuality of the input image.
Membership in the target domain. The generator must ensure that GXY (X) ∈ Y. To do this, a standard GAN architecture is used, in which a discriminator DY tries to distinguish generated images from real samples from Y. The goal of optimizing the generator parameters is to maximize DY (GXY (X)), and the goal of optimizing the discriminator parameters is, conversely, to minimize DY (GXY (X)) and maximize DY (Y). The networks compete with each other.
Individuality. If xi, xj ∈ X, i ≠ j, then there must be a relationship between xi and GXY (xi), which
explains why GXY (xi) is a representation in the domain Y of the image xi, and not xj. In [
The authors [
TraVeLGAN used an additional Siamese network to encode high-level semantics between the source and target domains. This idea seemed like a breakthrough as the Siamese network was believed to outperform CycleGAN in terms of translation quality. However, TraVeLGAN has not received much development due to the difficulties in choosing the architecture of the Siamese network and the parameters of its training. This results in a large set of possible solutions and makes it difficult to determine the effectiveness of each of them. 2.2.
CLIP [
The authors discovered an unexpected feature that, on many data sets, knowledge transfer without any post-training performs better than adding logistic regression on the top of the frozen network and post-training in 4 epochs on a new data set. Even worse indicators were obtained when trying not to freeze CLIP, but to fit all layers on a new data set.
The internal representation of CLIP. One of the side effects of CLIP is that encoders learn the internal representation of images in a shared space with the internal representation of natural language texts. Although there is no consensus in the scientific community on what is a "perfect" representation, one common option for testing the quality of a representation is to train a linear classifier attached to a frozen model and determine the performance of that model on different datasets. According to the results of experiments, all CLIP models regardless of encoder and size outperform any other known models in this test.
Natural language encodes semantic content and hierarchical relations between concepts with words. Contrastive learning of a visual model using natural language texts as a learning cue led to the learning and generalization of such special knowledge about image elements as expressed in imagerelevant texts in human language. The extent to which the visual model learns the hierarchy of concepts that exists in a human language requires separate research. Currently, CLIP reflects the meaning of the image in hidden representation most effectively among other well-known models.
The vector into which CLIP transforms images is the best choice for finding similar images. Other options for using CLIP in the search task are finding images that are most relevant to the content of some text and finding the text that most relevantly describes the image.
Overall, CLIP's powerful internal representation of images and text make it a valuable tool for a wide range of applications, with potential future uses that have yet to be imagined.
In our work, CLIPTraVeLGAN, we use CLIP as a means of preserving high-level semantics between the source and target domains in unpaired image-to-image translation. 2.3.
In [
The robustness of CLIP under natural skew of data distribution was tested in [
The use of CLIP in our work adds the ability to preserve the high-level semantics between the source and target domains, making the translations semantically robust.
In conclusion, our work, CLIPTraVeLGAN, builds upon the idea of TraVeLGAN and adds the advantages of CLIP to improve the quality of unpaired image translation while preserving the highlevel semantics between the source and target domains.
The core of the CLIPTraVeLGAN approach is the use of a pre-trained language-image model (CLIP) as a Siamese network in TraVeLGAN setup. The proposed CLIPTraVeLGAN model is composed of a generator, a discriminator and pretrained CLIP model. The generator takes an image from one domain and generates an image that belongs to the other domain. The discriminator is responsible for distinguishing between real and fake images. In CLIPTraVeLGAN, we replace the Siamese network in TraVeLGAN with the pre-trained language-image model CLIP. The CLIP model is used to encode the high-level semantics of the input and target domains.
We train the CLIPTraVeLGAN model using the adversarial loss and the TraVeL loss. The adversarial loss is used to ensure that the generated image belongs to the target domain, while the TraVeL loss encourages the generator to preserve the high-level semantics of the input image. Thus, the final objective terms of the generator are:
= + , (1) where λ controls the relative importance of TraVeL loss.
TraVeL loss is the same as in TraVeLGAN: = ≠ [ ( ) − ( ), ( ) − ( ) ] , (2) where Dist is a distance metric, such as cosine similarity.
One advantage of our approach is that it eliminates the need for choosing and training a Siamese network, which can be complex and time-consuming. Instead, the transfer of knowledge from CLIP to CLIPTraVeLGAN enables the generator to understand the relationships between images without any additional training. This makes our approach simpler and more straightforward, while still ensuring the high-level semantics are captured in the generated image.
In this context, the use of CLIP in CLIPTraVeLGAN adds the ability to preserve high-level semantics between the source and target domains, making the translations semantically robust. Therefore, our work builds upon the idea of TraVeLGAN and leverages the advantages of CLIP to improve the quality of unpaired image translation while maintaining semantic robustness.
For experiments we use Kaggle environment with TensorFlow and TPU support. Pretrained CLIP
weights were loaded from https://huggingface.co transformers package openai/clip-vit-large-patch14.
The dataset contains images from two different domains that are not aligned with each other. We
preprocess the dataset by resizing all the images to a fixed size of 256x256 and normalising the pixel
values to lie in the range [
To test the effectiveness of the proposed approach, we used the models studied in the Kaggle
competition "I'm Something of a Painter Myself" [
The generators and discriminators of all models were identical. The purpose of the experiments was to establish the effect of using the proposed improvement, that is replacing the Siamese network with pretrained CLIP. We use the Adam optimizer to train both the generator and discriminator.
To eliminate the irrelevant effects associated with struggling with overfitting on a small number of Monet paintings in competition, as well as the difficulty of visually evaluating Monet paintings, models must generate realistic photos of the landscape from Monet paintings. Some of Monet's paintings did not participate in the training. It is designed to test the quality of image translation and determine the FID metric.
We evaluate CLIPTraVeLGAN on GTA (Grand Theft Auto) [
The GTA and Cityscapes datasets represent two different domains of real-world urban environments. The GTA dataset consists of images of urban scenes generated from a video game, while the Cityscapes dataset comprises real-world urban scenes captured by a camera mounted on a car. The images in these datasets differ in terms of lighting conditions, weather, time of day, and many other factors. The main problem is that GTA images have more sky than Cityscapes. The discriminator can easily distinguish fake image by that criterion. Cityscape images have more vegetation instead. Thus, models may hallucinate vegetation in open sky regions which is semantic mistake.
We made an ablation study using CLIP model without pretrained weights to understand if semantic CLIP knowledge is necessary for the correct translation of an image. TraVeL loss turned into cosine similarity of noise vectors instead of semantic vectors.
The results of the translation of Monet's paintings into photographs were the following. The basic CycleGAN model showed on the test dataset FID = 6.9. Basic TraVeLGAN had 7.7. The investigated CLIPTraVeLGAN showed a result between the two base models, FID = 7.3. An example of resulted images is shown in Figure 2.
We compare the results of CLIPTraVeLGAN with those of TraVeLGAN to evaluate the effect from using pretrained CLIP instead of the Siamese network using GTA – Cityscapes benchmark (Figure 3).
Yellow lane lines from GTA should be translated into white lane lines. All real Cityscapes images have Mercedes hood ornaments. GTA has more sky than Cityscapes. An example of semantic flipping is hallucinations of trees instead of the sky.
We compared results with different values of λ that controls the relative importance of TraVeL loss and a different number of updates. FID metric conflicts with semantic robustness. When the coefficient of the importance of TraVeL loss λ is small it still prevents the model from collapse mode but allows semantic flipping. The FID metric is the best and the output image fits target domain. Increasing the value of λ results in worse FID values, but helps control semantic flipping. When the λ value is too high, the model renders the input images unchanged. This is the easiest way to ensure TraVeL consistency between input and output pairs.
The results of the GTA-Cityscapes experiment are shown in Table 1 and in Figures 3,4. There is a trade-off between image quality and semantic robustness. In our experiments, we searched for the optimal value of λ to achieve satisfactory results according to both criteria.
lines are translated correctly into white lane lines 0.001 40 000 6.22
Model hallucinates multiple Mercedes hood ornaments. There are artefacts in regions out of CLIP control. The rest of the sky is translated correctly 0.001 6.81 There is no semantic flipping. Yellow lane lines are translated correctly into white lane lines
In Figure 5 we show the results of translation of the same images that were used to compare the
state-of-the-art VSAIT model against other models in [
In the Ablation study (Figure 6) we use CLIP model without pretrained weights. Semantic CLIP knowledge is necessary for the correct translation of an image.
We trained CLIPTraVeLGAN to translate GTA Label to Cityscapes image. Our method exhibits semantic flipping when λ is small. Increasing the value of λ leads to the disappearance of the gradient from the discriminator. This seems to be a limitation of our method when domain images (like semantic labels) were not found in the CLIP training dataset.
Our biggest impression from the experiments is that the CLIPTraVeLGAN is very easy to train compared to many image-to-image translation models we tried before. With this model, we did not meet the collapse mode. The only TraVeL loss from pretrained CLIP is enough to prevent it.
We evaluated our proposed method on a benchmark dataset for unpaired image translation and compared it with other methods. The results demonstrated the effectiveness of our approach and the potential for further research in this area. However, it should be noted that our approach has some limitations and there is still room for improvement.
The main problem is that perfect semantically robust translation and perfect membership in the target domain are incompatible. The FID metric itself is only an additional indicator of the quality of the generated images but has nothing to do with the accuracy of the translation. Even more, the accuracy of the translation conflicts with the FID metric - the more the image matches the target domain, the more semantic changes were made during the translation. Thus, an approximate correspondence of the FID metric different models is enough to consider the model as a candidate for further research. Trying to demonstrate SOTA by metric FID on any dataset for an image translation task is not proof of its superiority. The evaluation on the GTA to Cityscapes dataset showed that the proposed CLIPTraVeLGAN approach outperformed TraVeLGAN for unpaired image translation in terms of both stability and quality of the generated images. The FID score of CLIPTraVeLGAN is lower than that of TraVeLGAN when λ is small and translation is not perfect. But CLIPTraVeLGAN can provide such semantic stability of translation that is not available for TraVeLGAN. The results of the experiment demonstrate that CLIPTraVeLGAN is a promising approach to image translation and provides a new direction for further research.
Additionally, we evaluated our method on Kaggle competition dataset. Since the task of the Kaggle competition is not to translate images from one domain to another, but to generate images, the FID metric displays only quality generation and diversity of images of the target domain. Semantic robustness of the translation is not evaluated in any way.
One of the key features of CLIP is its internal representation of images and natural language texts in a shared space learned through contrastive learning. This shared space allows for finding similar images and relevant text descriptions of images, among other potential applications. Additionally, CLIP's encoders learn to represent images in a way that reflects the meaning of the image most effectively among other well-known models.
Specifically, we tried using CLIP to evaluate the similarity of the semantic content between the input image and the generated image. During training, the generator network tries to produce an output image that is not only visually similar to the input image but also semantically similar in the sense that it is classified similarly by CLIP.
To achieve this, we used a pre-trained CLIP model as a feature extractor and obtain the CLIP embedding of both the input and generated images. We then use the cosine similarity between these embeddings to evaluate the semantic similarity between the two images. Such straightforward method is not effective even for GTA to Cityscapes translation. It prevents translating yellow lane lines into white lane lines that are commonly found in Cityscapes images. Translating “men into women” or “horse to zebra” conflicts with such semantic identity too. To avoid such effect, we developed an idea using CLIP in the TraVeLGAN setup. By using CLIP in this way, we aim to improve the semantic robustness of our image-to-image translation model, allowing it to translate details that must be translated. There is one more limitation of our method. When domain images are very specific and were not found in the CLIP training dataset, our method is almost useless. In the example, we are trying to translate GTA labels into Cityscapes photos. We could not prevent semantic flipping in this task. Finally, we suggest exploring the use of multiple CLIP models with different pre-training data and architecture to improve the robustness and accuracy of the generated images. This could lead to even better performance and expand the range of applications for image translation. Possible use of two or three various CLIP models (for example based on ResNET and on visual transformers) to increase the reliability of Siamese network.
In this paper, we proposed a novel approach for semantically robust unpaired image translation, CLIPTraVeLGAN. CLIPTraVeLGAN simplifies the Siamese network selection and training process of TraVeLGAN by using a contrastively pretrained language-image model (CLIP) with frozen weights. To our knowledge, we were the first to utilize CLIP to enforce that the individual features and semantic content of the input image are reflected in the generated image during image-to-image translation. The proposed model CLIPTraVeLGAN proved to be much easier to train than the original TraVeLGAN. The training is quite stable, the results are quite comparable with other models built on the same generators and discriminators. Our results show that CLIPTraVeLGAN outperforms both CycleGAN and TraVeLGAN in terms of semantic robustness while being easier to train and producing comparable results in terms of quality.
The methodology for achieving semantic robustness in image translation typically involves
training a model to effectively preserve the high-level semantics between the source and target
domains. This can be done using a variety of techniques, including the use of specialized loss
functions and Vector Symbolic Architectures [
There is a trade-off between image quality and semantic robustness. In our experiments, we manually searched for the optimal value of λ to achieve satisfactory results according to both criteria. In future works, the automated estimation of the optimal value of λ during training has the potential to improve the results and stability of training. The proposed model showed promising results in terms of semantic robustness and ease of training and can be used as a starting point for future research on efficient image translation models. Possible future work includes exploring the use of multiple CLIP models and new solutions of generators and discriminators.
The work was performed as part of the state budget research project “Development of methods and algorithms for combined learning of deep neuro-neo-fuzzy systems under short training set conditions” (state registration number 0122U001701) of Artificial Intelligence Department of Kharkiv National University of Radio Electronics.