Text-to-image and image-to-text creation [1,2] is becoming very popular because of its vast use. The goal of this comparison analysis is to identify the advantages and disadvantages of various text-to-image creation techniques [3]. We may learn about the underlying mechanisms that contribute to their picture synthesis skills by investigating their architectural designs. Cogview (ELBO), discrete variational auto-encoders (dVAE), multi-stage AttnGAN, generative adversarial networks (GANs), LSTM+GAN, CycleGAN+BERT, DF-GAN, MirrorGAN, VQ-SEG (a modified VQ-VAE), StackGAN+fine-tuned BERT text encoding models, and DALL-E-2 are among the models investigated. We look at the datasets used by these models for training and assessment in addition to architectural comparisons. This includes well-known benchmarks like as COCO and CUB, as well as bespoke datasets created expressly for text-to-image creation [4]. The diversity and quantity of these datasets, as well as any pre-processing techniques used, have a significant impact on model performance. Various performance indicators have been used in the field to analyse the quality of produced photographs. Our study incorporates human assessments, user research, and additional qualitative evaluations that the analyzed models employed, along with perceptual similarity metrics like Frechet Inception Distance and Inception Score. This allows for a thorough assessment of each model's visual accuracy and realism. We hope that this comparative analysis will give scholars and practitioners a full grasp of the various text-to-image creation techniques. We provide vital insights for making educated decisions in picking the most appropriate model for certain applications by emphasising the strengths and drawbacks of each model based on architectural choices, dataset utilisation, and performance indicators. We give a deep study of the model designs, datasets, and performance indicators in the next sections of this work, as well as a comprehensive comparison analysis. We end by summarising the important findings and outlining potential future research avenues in text-to-image creation.
Text-to-image creation is a difficult problem that seeks to translate verbal descriptions into aesthetically realistic and semantically consistent images automatically. This task is critical in a variety of applications, including computer vision [5], multimedia content generation, and virtual reality. The objective is to bridge the gap between natural language and visual representations, making it possible for robots to interpret and produce visual content. Several models have been created to do this, including older approaches such as cogview and dVAE, as well as cutting-edge techniques such as different GAN models and BERT. These models use largescale picture datasets like MSCOCO, CUB, and Oxford 102 to understand the relationship between written descriptions and visual representations. These models help to improve human machine interactions and facilitate creative content development by creating high-quality visuals that correspond to the provided text. This review lays the groundwork for a more indepth examination and comparison of various models in the next sections of this study. Brief Introduction to Models:
1. Cogview: Cogview is a state-of-the-art text-to-image generating model that combines cognitive theories and deep learning methods.
Here are summaries of the mentioned datasets: A mix of supervised and reinforcement learning losses is used to train the model. To match the produced photos to the written descriptions, the supervised loss is utilised. The reinforcement learning loss is used to encourage the model to create aesthetically attractive pictures. A collection of text and picture captions is used to train the system. While the text encoder is trained using the written captions, the image decoder is trained using the images. The model is trained with the Adam optimizer, which has a 3e-4 learning rate. The CogView model has been demonstrated to be capable of producing realistic pictures from text descriptions. The model was tested on a range of datasets and found to be competitive with existing text-to-image creation techniques. details of the CogView architecture:
• The text encoder is a one-way Transformer that produces a series of latent codes after receiving a text caption as input.
• The text encoder's latent codes are used by the image decoder, a convolutional neural network, to create an image. • A dataset including 1.56 million Chinese text-image pairings is used to train the algorithm.
• The model is trained for 144,000 steps.
• The learning rate is decayed using a cosine annealing schedule.
• The batch size is 6,144.
• The Adam optimizer is used with a learning rate of 3e-4.
• The model is trained on a mix of 16-bit and 32-bit precision.
training. • The model uses a technique called Sandwich Layernorm to improve the stability of training.
dVAE (disentangled Variational Autoencoder) is a textto-image synthesis model that generates pictures from text descriptions using a disentangled latent space. The following is the model architecture: The text encoder takes a text caption as input and produces a sequence of latent codes. The image decoder takes the latent codes from the text encoder and produces an image.
Because the dVAE's latent space is disentangled, the latent codes reflect distinct features of the picture. This enables the model to provide more realistic and varied visuals. A mix of supervised and reinforcement learning losses is used to train the model. To match the produced photos to the written descriptions, the supervised loss is utilised. The reinforcement learning loss is used to encourage the model to create aesthetically attractive pictures. The algorithm is trained on a dataset of picture captions and text captions. The photos are utilised to train the image decoder, while the text captions are used to train the text encoder. The Adam optimizer is used to train the model, which has a learning rate of 3e-4. The dVAE model has been demonstrated to be capable of producing realistic visuals from text descriptions. The model was tested on a range of datasets and found to be competitive with existing text-to-image creation techniques [7]. Here are some of the details of the dVAE architecture:
• The text encoder is a bidirectional LSTM that takes a text caption as input and produces a sequence of latent codes. • The image decoder is a convolutional neural network that takes the latent codes from the text encoder and produces an image. • The latent space of the dVAE is disentangled into three factors of variation: pose, shape, and appearance. • The model is trained on a dataset of 100,000 text-image pairs. • The model is trained for 100 epochs.
• The learning rate is decayed using a cosine annealing schedule.
• The batch size is 64.
• The Adam optimizer is used with a learning rate of 3e-4.
• Here are some of the specific training details of the dVAE model.
• The model uses a technique called Wasserstein loss to improve the stability of training.
• The model uses a technique called KL annealing to gradually increase the importance of the KL divergence loss during training.
A text-to-image synthesis model called AttGAN (Attention GAN) [8] employs attention to regulate the creation of pictures from text descriptions. The model architecture is as follows:
The text encoder takes a text caption as input and produces a sequence of latent codes. The image generator takes the latent codes from the text encoder and produces an image. The image discriminator takes an image as input and produces a probability that the image is real or fake. When producing the picture, the attention mechanism allows the image generator to focus on select sections of the written description. As a result, the model may provide visuals that are more compatible with the written description. A mix of adversarial and supervised losses is used to train the model. The adversarial loss is used to teach the image generator and discriminator to compete. To match the produced photos to the written descriptions, the supervised loss is utilised. The algorithm is trained on a dataset of picture captions and text captions. The pictures are utilised to train the image discriminator, while the text captions are used to train the text encoder and image generator. The Adam optimizer is used to train the model, which has a learning rate of 3e-4. The AttGAN model has been demonstrated to be capable of producing realistic pictures from text descriptions. The model was tested on a range of datasets and found to be competitive with existing text-to-image creation techniques [9]. details of the AttGAN architecture:
• The text encoder is a bidirectional LSTM that takes a text caption as input and produces a sequence of latent codes. • The image generator is a convolutional neural network that takes the latent codes from the text encoder and produces an image. • The image discriminator is a convolutional neural network that takes an image as input and produces a probability that the image is real or fake. • The image discriminator is a convolutional neural network that takes an image as input and produces a probability that the image is real or fake.
Here are some specific training details of the AttGAN model:
• The model is trained for 100 epochs.
• The batch size is 64.
• The Adam optimizer is used with a learning rate of 3e-4. • The image discriminator is a convolutional neural network that takes an image as input and produces a probability that the image is real or fake. • The BERT model [10] is a Transformer-based model that takes the text caption as input and produces a sequence of hidden states. • The image generator is a convolutional neural network that uses the latent codes from the text encoder to create an image. • The model uses a technique called Wasserstein loss to improve the stability of training.
• The model is trained on a dataset of 500,000 text-image pairs. • The training goes on for 100 epochs and the learning rate is decayed using a cosine annealing schedule, with a size of 64 for each batch. • The Adam optimizer is used with a learning rate of 3e-4.
TThe DF-GAN [11] is made up of the generator, discriminator, and pre-trained text encoder. To guarantee the diversity of the pictures it generates, the generator needs two inputs: a phrase vector encoded by a text encoder and a noise vector sampled from a Gaussian distribution. The noise vector is first turned into a completely connected layer. The properties of the image are then upsampled using a sequence of UP-Blocks. An upsample layer, a residual block, and DF-Blocks make up the UPBlock, which combines the text and picture attributes throughout the image manufacturing process. Finally, an image feature is converted into an image using a convolution layer. Images are converted into attributes by the discriminator using a sequence of DownBlocks. After that, a copy of the vector phrase is mixed with the properties of the photo. To evaluate the visual realism and semantic coherence of the inputs, an adversarial loss will be anticipated. The discriminator aids the generator in producing pictures of superior quality and textimage semantic coherence by differentiating synthetic images from authentic examples. The bidirectional long short-term memory (LSTM) text encoder extracts semantic vectors from the text description. We employ AttnGAN's pre-trained model directly. Image scaling and normalisation are two ways to preprocess the image data.
• Images undergo resizing and normalizing the images. • Text undergoes tokenisation followed by verctorisation. • Text undergoes tokenisation followed by verctorisation.
• Train the image encoder using a dataset of real images.
• Update the image encoder's weights using a suitable loss function (e.g., Mean Squared Error). • Freeze the generator and discriminator.
• Train the text encoder using the vectorized text descriptions.
• Update the text encoder's weights using a suitable loss function.
• Unfreeze the generator, discriminator, and encoders.
• Generate fake images by sampling from random noise and text features.
• The discriminator part of the GAN is trained to distinguish between real and fake images.
• The generative part of the GAN is trained to fool the discriminator by generating realistic images.
The figure above depicts the MirrorGAN implementation [12], which includes a mirror structure that combines T2I (textto-image) and I2T (image-to-text) functions. MirrorGAN's fundamental idea is to use the concept of redescription to train T2I generation. MirrorGAN then reproduces the image's description, successfully matching the created image's underlying semantics with the given text description. The MirrorGAN model is made up of three major components: STEM, GLAM, and STREAM. Each module performs a distinct task in the model's overall operation.Pretrain a text encoder network using a large-scale text dataset (e.g., text corpus).
• Pretrain a text encoder network using a large-scale text dataset (e.g., text corpus).
• The text description is fed into this network, which then encodes it into a fixed-length feature vector. • The weights of the text encoder are updated using an appropriate loss function (such as Cross-Entropy Loss).
• The generator is trained to produce realistic images from text features and random noise.
• The discriminator is trained to discern between generated and real images.
• The discriminator is trained to discern between generated and real images.
• Turn off the discriminator and generator.
• Update the text encoder's weights using an appropriate loss function (such as Triplet Loss) after training it with the dataset's text descriptions. • Create false images by sampling from text features and random noise.
• Switch between training the discriminator and generator.
A version of the Variational Quantum VAE (VQVAE) architecture designed for image synthesis and segmentation tasks is called VQ-SEG. The architecture of VQ-SEG consists of several key parts. Using an encoder network, incoming images are first transformed into a representation in a lower-dimensional latent space. In order to capture hierarchical information at many scales, this encoder typically has convolutional layers followed by downsampling techniques like pooling or strided convolutions. The latent space representation is then obtained by the Vector Quantization (VQ) layer. The continuous latent coding is discretely quantized by the VQ layer. It uses a preset codebook that it learned during training to match each latent code to the closest codeword. Computation and storage are facilitated by this distinct form.
The VQ-SEG picture segmentation process now has an additional branch thanks to modifications made to the VQVAE architecture. This branch produces segmentation masks, pixel by pixel, that show the class labels of different regions inside the input image. VQSEG incorporates an extra decoder network to enable picture segmentation. This decoder produces pixel-wise predictions for every class label using quantized latent codes. Often, the decoder network consists of upsampling or transposed convolutions to improve the feature maps' spatial resolution. VQ-SEG integrates segmentation and reconstruction losses during training. The reconstruction loss promotes output images that are similar to the original input images, but the segmentation loss penalises differences between expected segmentation masks and ground truth masks. Typically, these losses are determined using pixel-by-pixel comparisons, such as mean squared error or cross-entropy loss. In essence, VQ-SEG is a modified VQVAE architecture that combines discrete quantization with segmentation branches to add image segmentation capabilities.
Diffusion models are used by the DallE-2 architecture to produce high-resolution images conditioned on optional text descriptions and CLIP image embeddings. CLIP embeddings are projected into and added to the current timestep embedding in this enhanced architecture. In addition, four additional context tokens are projected from CLIP embeddings and concatenated with the GLIDE text encoder's output sequence. It is discovered that the text conditioning pathway provides minimal assistance in this area, despite its attempt to capture natural language elements that CLIP might miss. In order to improve sample quality, training involves randomly removing the text caption 50% of the time and randomly setting CLIP embeddings to zero or applying learnt embeddings 10% of the time. Guidance on conditioning information is also used. It takes two trained diffusion upsampler models to produce high-resolution images. While the second upsampler further upsamples the photos to 1024×1024 resolution, the first upsampler concentrates on boosting the resolution from 64×64 to 256×256. By employing methods like Gaussian blur and varied BSR degradation to slightly contaminate the conditioned images during training, the robustness of the upsamplers is increased. The Dalle2 architecture does not include attention layers; it only uses spatial convolutionals. The model is applied immediately at the target resolution during inference, demonstrating its capacity to generalise to higher resolutions without the requirement for extra conditioning on the text caption. The upsamplers use the unconditional ADMNets technique and are not conditioned on the caption. To summarise, the GLIDE text encoder, CLIP image embeddings, and diffusion models are used in the Dalle2 architecture to produce high-resolution images. The resulting images' quality and resilience are enhanced by the processes of conditioning and upsampling.
1. The LSTM (Long Short-Term Memory) model [14], is known for its ability to capture long-range dependencies in data. A single LSTM memory cell, which consists of input gates (it), forget gates (ft), output gates (ot), cell state (ct), and cell input activation vectors.
The LSTM model utilizes composite functions to calculate these components based on input and previous hidden states. The model employs the logistic sigmoid function and the hyperbolic tangent function to process the inputs. The original LSTM algorithm used an approximate gradient calculation, but this paper adopts backpropagation through time for gradient calculation. However, training with the full gradient can lead to large derivative values. The LSTM unit receives inputs from external sources at each time step and updates its internal cell state and hidden state based on these inputs and previous states. 2. Recurrent neural networks (RNNs) composed of LSTM (Long Short-Term Memory) units are used in the LSTM Autoencoder Model, an unsupervised learning model. An encoder LSTM and a decoder LSTM are the two RNNs that make up the model. An image patch or set of features is a sequence of vectors that are fed into the model. Following the processing of this input sequence by the encoder LSTM, the decoder LSTM assumes control after all inputs have been read. A prediction for the target sequence-which is the input sequence in reverse order-is produced by the decoder LSTM. Both conditioned and unconditioned decoders are possible. The final created output frame is fed into a conditional decoder, but it is not fed into an unconditioned decoder.
3. The Future Predictor Model shares the same design as the Autoencoder Model, with the key difference lying in the decoder LSTM. While the Autoencoder Model predicts the target sequence that matches the input sequence, the Future Predictor Model goes a step further and predicts frames of the video that come after the input sequence. Essentially, this model is designed to forecast a longer sequence into the future, extending beyond the input sequence.
This section contains the analysis and findings from our investigation, which assessed the effectiveness of the eleven GAN and auto encoder models created for text-to-image conversion. One of the challenges encountered in the CogView framework is the slow generation process inherent to autoregressive models, as images are being generated token-bytoken. Additionally, blurriness is introduced as a substantial restriction by the use of VQVAE. When discretizing continuous data for use with discrete variational auto-encoders (dVAEs), there are disadvantages including limited expressiveness and possible information loss. Due to a lack of textimage pairs for each category and the inclusion of more abstract captions in the dataset, such as COCO, the multistage Attention-GAN model is constrained. Gaps exist in the ability of Generative Adversarial Networks (GANs) to produce coherent, high-quality images that are in line with the input data. While leveraging unsupervised learning, the LSTM+GAN technique struggles to produce clusters that truly reflect the truth, leading to restricted expressiveness regarding input information. CycleGAN+BERT's performance is hampered by insufficient training time and the absence of hyperparameter adjustment because of time restrictions. Due to its strong sensitivity to hyper-parameters, DF-GAN primarily relies on pre-trained models and lacks diversity in generated data. Basic text embedding techniques have a negative impact on STEM integration and the quality of the outcomes for MirrorGAN. The image quality of VQ-SEG, a modified VQVAE, could be better, however the improvements also cause losses in perceptual knowledge and awareness of a specific region. Further studies are required to construct complex loss functions and efficiently create images from text with little data for StackGAN+fine-tuned BERT text encoding models. Finally, Conditional Adversarial Networks (cGAN) still have difficulties in producing visually and semantically cohesive video sequences from textual descriptions.
The generated photos' quality could be evaluated subjectively using the Mean Opinion Score (MOS). On a numerical scale, human participants would be asked to score the generated images' perceived fidelity or quality. The MOS, which provides an overall assessment of the image quality, would then be calculated by averaging the ratings given by several people. Higher MOS values correspond to more visually attractive or realistic perceptions of the created images, whereas lower MOS values correspond to lower quality or fidelity. User happiness can be measured and text-to-image conversion model [15] enhancements can be directed via MOS evaluations.
Converting text to image generation models have come a long way, but there are still a number of areas that need more research and development. This section highlights potential future research directions based on the current state of models and identified areas for improvement Improved Semantic Understanding: Enhancing the semantic understanding of text is crucial for generating more accurate and contextually relevant images. Future research could focus on incorporating advanced natural language processing techniques, such as pre-trained language models or knowledge graphs, to capture a deeper understanding of text semantics. This could enable models to generate images that align more closely with the intended meaning of the input text. One area that could potentially be beneficial is generating knowledge graphs from text embeddings to improve context and positional understanding greatly.
Increased Resolution and Realism: Though current models have come a long way in producing high-quality photographs, resolution and photo-realism may still use some work. Future research could focus on developing techniques to generate images at higher resolutions, allowing for more detailed and visually appealing results. Additionally, exploring advanced loss functions or perceptual similarity metrics could further enhance the realism of generated images, making them indistinguishable from real photographs.
Fine-grained Control and Manipulation: Current text-to-image models often lack finegrained control over generated images. Future research could investigate methods to enable precise control and manipulation of image attributes, such as object positions, colors, and styles, based on textual input. This could involve exploring novel conditioning techniques or incorporating additional information during the generation process to produce images that align with specific user requirements.
Handling Ambiguity and Multi-modal Outputs: Textual descriptions often contain ambiguous or subjective elements that can lead to multiple plausible interpretations. Future research could explore methods to handle such ambiguity and generate diverse, multi-modal outputs that capture different interpretations of the same textual input. This could involve incorporating uncertainty estimation techniques, exploring variational approaches, or leveraging adversarial learning to encourage the generation of diverse image outputs. One possible approach is training the image generator module on a combination of parsed output from scene graph and the actual prompt for a more objective understanding leading to potentially reduce ambiguity to a reasonable extent.
Incorporating User Feedback and Interactive Generation: Interactive text-to-image generation systems that incorporate user feedback and preferences hold great potential for enhancing user satisfaction and enabling personalized image generation. Future research could focus on developing models that can adapt and refine their generation process based on user interactions, allowing users to provide feedback and guide the image synthesis process in real-time.
Ethical Considerations and Bias Mitigation: As text-to-image generation becomes more prevalent, it is important to address ethical considerations and mitigate potential biases in the generated content. Future research should explore meth-ods to ensure fairness, diversity, and inclusivity in generated images, avoiding the reinforcement of harmful stereotypes or biases present in the training data. This could involve developing bias detection [16] and mitigation techniques or incorporating fairness constraints during the training process. These new lines of inquiry could further the field of text-to-image generation and open up new avenues for producing contextually appropriate, high-quality images from textual input. Through the exploration of novel methodologies and resolution of these obstacles, scholars can facilitate the development of more advanced and adaptable text-to-image generation models that have wider applications across diverse fields.
In conclusion, our comparative study of 11 text-to-image generation models highlighted Stack-GAN as the top performer. StackGAN achieved a remarkable inception score of 4.44, indicating its ability to generate visually diverse and high-quality images. Additionally, StackGAN outperformed other models with a FID score of 37.7, demonstrating its superior ability to capture image fidelity and similarity to real images. While other models, such as cogview [17] and dVAE, showcased strengths in specific areas, they fell short in terms of overall performance compared to StackGAN. The models based on GAN architecture, including Multi-Stage AttnGAN, LSTM+GAN, and CycleGAN+BERT, exhibited promising results in capturing global and local image details, but StackGAN surpassed them in terms of both inception and FID scores.Our study also emphasized the impact of dataset selection on model performance. The MSCOCO dataset provided a diverse range of images, contributing to the evaluation and comparison of the models. The outcomes demonstrated that StackGAN could make good use of the dataset, producing better image creation results. These results offer insightful information to the text-to-image generating sector and help practitioners and researchers select models that are suitable for their particular requirements. Future studies can concentrate on developing StackGAN even more and investigating its possible uses in a range of fields, including as virtual reality, multimedia content generation, and computer vision.In conclusion, our comparison analysis shows that StackGAN is the best model for text-to-image creation, doing remarkably well with an origin score of 4.44 and a Fretchet Inception Distance score of 37.7. These outcomes demonstrate the effectiveness of StackGAN as a method for producing realistic and varied graphics from text descriptions.