Evaluation of generative models has been an underrepresented field despite the surge of generative architectures. Most recent models are evaluated upon rather obsolete metrics which sufer from robustness issues, while being unable to assess more aspects of visual quality, such as compositionality and logic of synthesis. At the same time, the explainability of generative models remains a limited, though important, research direction with several current attempts requiring access to the inner functionalities of generative models. Contrary to prior literature, we view generative models as a black box, and we propose a framework for the evaluation and explanation of synthesized results based on concepts instead of pixels. Our framework exploits knowledge-based counterfactual edits that underline which objects or attributes should be inserted, removed, or replaced from generated images to approach their ground truth conditioning. Moreover, global explanations produced by accumulating local edits can also reveal what concepts a model cannot generate in total. The application of our framework on various models designed for the challenging tasks of Story Visualization and Scene Synthesis verifies the power of our approach in the model-agnostic setting.
Image generation has been one of the most popular deep learning tasks, inspiring many
impressive state-of-the-art applications [
Explainability of generative models is another emerging field, which has currently received way less attention compared to discriminative models [16, 17]. The incorporation of explainable feedback in Generative Adversarial Networks (GANs) has demonstrated a promising research direction [18], while other works focus on interpreting GAN inner structure [19]. Overfitting in GANs can be tackled by determining the image areas that contributed to classifying a sample as fake/real, thus explaining the discriminator’s decision [20]. This limited literature impedes the development of explainable evaluation for generative models, even though related attempts have gained ground in other AI domains, such as Natural Language Processing [21, 22, 23].
We argue that resolving generative evaluation challenges calls for a conceptual approach to the evaluation process, diverging from the pixel-level route. Relying on concepts instead of pixels ofers the advantage of enhanced interpretability regarding the evaluation process and paves the way for explainable evaluation of generative models. Identifying concepts (objects or attributes) that can or cannot be generated reveals the capabilities and biases of the model at hand, thus driving potential architectural modifications. In this paper, we present the ifrst explainable evaluation technique targeting generative models. Specifically, we utilize counterfactual explanations to frame conditional generative evaluation as the answer to the following question: What concepts need to change in a generated sample X, for it to reach its conditioning c? Conceptual edits guided from external knowledge sources [24] have shown to eficiently indicate the shortest possible path to reach the conditioning concepts. Furthermore, existing works that combine explainability with image generation operate on specific models [18, 19, 20] and demand access to their inner structure (white-box techniques), while our approach only requires generated outputs along with their ground truth conditioning, yet still regarding the generative model as a black-box. We, therefore, contribute to the following: 1. We propose the first conceptual rather than pixel-based generative evaluation framework1, suitable for various tasks such as Scene Generation (SG) and Story Visualization (SV). 2. Our metrics are explainable by design, illustrating which concepts need to be inserted, deleted, or replaced in the generated images, for them to approach the ground truth conditioning. Those edit operations are applied in a model-agnostic setting, totally trespassing any access to the generative model inner workings. 3. Global explanations automatically reveal possible blind spots of generative models, i.e.
concepts that a model is intrinsically incapable of generating.
Generative Adversarial Networks (GANs) [
Several image generation cGANs [26, 27] perform well when it comes to generating images with distinct textures and colors. However, they tend to struggle with generating coherent overall object structures and other long-range dependencies, due to the limited nature of convolutional filters. The Self-Attention GAN (SAGAN) [ 28] was proposed as a solution to this problem; it utilizes a self-attention module in both and , as well as modern stabilization techniques such as Spectral Normalization of weights [29], while it leverages the two-timescale update rule [30] to impose diferent learning rates for and .
Difusion models are breaking new ground in the field of conditional image generation and
are becoming the state-of-the-art in that area [
StoryGAN [43] is a generative model that synthesizes images based on sequential input
(Story Visualization), using an RNN structure to encode the input text and provide context
information to the generator. The generator is trained adversarially against two discriminators:
the image discriminator, which evaluates image quality and text-image relevance, and the story
discriminator, which ensures consistency across images given the entire story context. Recent
work has focused on improving the baseline StoryGAN model [44] and exploring alternative
story encoding methods, such as using Transformer architectures [
Counterfactual explanations Contemporary AI research moves towards explaining a neural
network’s train of ’thought’, thus eXplainable AI (XAI) becomes a field of increasing interest [
Concept distances instruct the shortest path that connects two specific concepts. Concept
hierarchies are employed, deterministically defining the transition cost between concepts. We
explore both the option to use external hierarchical knowledge such as WordNet [
• Insertion (I) +(): A concept ∈ is inserted in .
Each edit operation inherits the concept distances imposed by the selected hierarchy. Therefore, R operation considers the path between and so that min((, )) is ensured. As in [24], also ensures actionability of edits, allowing semantically meaningful transitions (e.g. ’food’→’pasta’), while prohibiting meaningless ones (e.g. ’food’→’sky’). D and I operations regard the root node of the hierarchy as and respectively; in the case of WordNet, entity.n.01 serves as the root. Concept Set Edit Distance (CSED) ( → ) is obtained by aggregating all possible minimum cost edit operations so that → is finally achieved: , ,, = ( → ) = min ∑︁ ∑︁ (, ) ̸= (1)
The heart of our method consists of a pre-trained black-box generative model which receives a semantic description (in natural language or in symbolic format) as conditioning and produces an image corresponding to . We then use of-the-self automatic methods such as object detection, semantic segmentation, and others, in order to extract all the concepts depicted in the generated images and append them in the generated (or source) concept set . Similarly, concepts extracted from contribute to the real (or target) concept set . The format of defines the concept extraction technique that is followed, ranging from linguistic concept extraction, if is a textual sentence, to simple preprocessing, if is already in a set format. Ultimately, we aspire to answer the following: "What are the minimal required changes in order to traverse from to ?" The outline of our method is presented in Figure 1.
The counterfactual backbone described in Section 3 highlights our process for generative evaluation, which we employ on two dificult tasks of the generative literature: Story Visualization (SV) and Scene Generation (SG).
Story Visualization (SV) targets the sequential creation of images 1, 2, ..., that correspond, one-to-one, to a given sequential conditioning 1, 2, ..., of a total length . The generated images need not only to remain faithful to their conditioning, but to also maintain serial consistency. We therefore define the two desiderata applicable to SV: • Faithfulness: objects and attributes mentioned in should also appear in frame , for any story frame , where 1 ≤ ≤ . • Consistency: objects or attributes appearing in frame cannot disappear or change in later frames +1, ..., , for any story frame , where 1 ≤ < .
Since well-defined semantics are tied to counterfactual explanations [
shape (cube, sphere, cylinder), size (small, large), material (rubber, metal) and one of 8 colors (blue, cyan, brown, yellow, red, green, purple, gray). Each CLEVR-SV object contains ||=4 concepts that describe its shape, size, material and color. We handcraft a simple hierarchy to group object semantics to generic concept classes, demonstrating the following inclusion relationships: (large, small) ⊂ Size (blue, yellow, brown, grey, green, purple, cyan, red) ⊂ Color
Material (metallic, rubber) ⊂ (sphere, cube, cylinder) ⊂ Shape (2) CLEVR-SV contains stories of length =4, with the -th frame strictly containing objects. Any of the three available edit operations can be relevant per frame: D of a concept, when a generated frame contains more objects than its ground truth match; I of a concept in the opposite case; R equals to a D followed by an I, and can be applied on frames with proper number of objects when semantics difer. In the default case, we assign equal costs of 1 for all semantics, as well as for D and I operations (R cost is the sum of D and I costs).
To measure story faithfulness we propose the Story Loss (SL) metric, which sums up the per-frame Concept Set Edit Distance () for = 1, 2, ..., frames of the story. Generated CLEVR-SV semantics for shape, size, material and color for the k-th frame form the concepts set , while the semantics of the conditioning form , with their denoted as (, ). Thus, the cost for the transition {1, 2, 3, 4} → {1, 2, 3, 4} corresponding to the minimum cost R D I edits needed to transform the semantics of the generated sequence {1, 2, 3, 4} to the semantics of its conditioning {1, 2, 3, 4} can be expressed as: = = = ∑︁ = ∑︁ (, ), =0 =0 = 4 By scaling up the calculation of SL for a dataset containing stories, we obtain the Global Story Loss (GSL) metric:
= = ∑︁ = ∑︁ ∑︁ (, )
=0 =0 =1 As for story consistency, we propose the metric of Consistency Loss (CL): the frame is compared with − 1, = 2, .., frames of the generated sequences to capture changes of semantics. A challenging aspect of CL is that there does not exist a ground truth concept set. However, since it is known by task definition that the k-th frame contains k objects, and the cardinality || of dataset concepts is predefined ( ||=4 in the case of CLEVR-SV), we can assume that every previous frame constitutes the ’ground truth’ corresponding to the concept set . Commencing from the k=1 frame, we expect the cardinality of to be equal with || · = ||. Any discrepancy results in a penalty = | | − || · = for k=1. For later frames, we define as the concept set corresponding to the k-th frame, and as the ’ground truth’ set comprised of the k-1 frame concepts. Mathematically, CL can be written as: In the ideal case, when the k-th frame contains k objects with semantics, we expect that =1 = 0 and >1 = || · ( − 1). By extending CL to stories, Global Consistency Loss (GSL) evaluates the consistency capabilities of a generative model in total: (3) (4) (6) (7) (8) = = =1 + ∑︁ (, ), =2 = − 1, = 4 (5) = = ∑︁ = ∑︁{=1, + ∑︁ (, ), , = − 1,}
=0 =0 =2 Average values can be obtained for both local (SL/CL) and global (GSL/GCL) metrics: 1 Avg = ,
Avg =
[Avg ] = 1 1 For consistency, instead of exporting an average value over ∑︁ , it is more meaningful to count how many times the =1 = 0, >1 = || · ( − 1) requirement was not respected, averaged for = frames:
= Avg = =1 + 1 ∑︁[>1 ̸= || · ( − 1)], =1
Avg =
[Avg ] 1 SL and CL are by nature explanaible, as they do not only provide a measure of quality but also reveal the → edit paths. Those paths serve as local counterfactual explanations, highlighting the erroneously generated semantics for this particular story, either in terms of faithfulness or consistency. Overall, higher SL/GSL and CL/GCL values denote lower conceptual generation quality. GSL/GCL edit paths correspond to global counterfactual explanations: rule extraction techniques provide frequent patterns, summarizing the behavior of under investigation. Frequent GSL edit paths in fact contain common misconceptions, i.e. conditioning concepts that cannot easily generate. Similarly, GCL edit paths reveal frequent inconsistency patterns, showcasing concepts that arbitrarily change within the story frames. Hence, by researching the question "What has to minimally change in order to transit from to ?", we eventually answer a more generic one: "Which concepts cannot be generated or preserved by ?" Scene Generation (SG) aims to synthesize a visual scene based on a conditioning . The synthesized image comprises multiple objects which interact with each other. Scene objects are also accompanied by attributes. The given conditioning is more complex compared to conditionings provided for SV, since the concepts to be generated are numerous and not predefined; this yields a concept set of unknown but comparatively large cardinality.
COCO dataset [
In the concept extraction stage, YOLO-v8 [
5.1. Story Visualization
Since all semantics and D, I edit operations have an equal cost, we assign =1 for all semantics,
as well as for D, I. For example, deleting a color yields an edit cost of 1. Alternatively, by
substituting a color with another one induces an edit cost of 2, equal to deleting the source color
and then inserting the target color. The same logic applies to shape, size and material of objects.
Metric results over the best variants of selected SV models [
In general, we observe an agreement between pixel-level and conceptual metrics. This is
somehow expected, since the concept extraction stage depends on pixel-level image quality, with
better generated objects or semantics being more easily identifiable. Nevertheless, conceptual
evaluation ofers more explainable insights: percentages of losses per concept (Material, Size,
Shape, Color) are provided, highlighting strengths and shortcomings of investigated models
over diferent semantics. For example, higher Shape loss for all models (> 50%), indicates that
they synthesize objects of ambiguous shapes in most cases. On the other hand, relatively lower
Size losses reveal the models’ capability to generate objects having the right size.
Local explanations The transparency of the proposed SL/CL metrics is verified by obtaining
local explanations for [
GSL can also reveal patterns in the form of rules for the whole test set. We leverage the
apriori algorithm [
We observe that Material is the most common concept misconception, with both ’rubber’
and ’metallic’ semantics being frequently confused. Shape is the second most prominent
misconception, with ’cylinder’ appearing in the generated frames more often compared to the
’cylinder’ occurrence in the conditioning; ’cube’ and ’sphere’ shapes are sacrificed for ’cylinder’
to be generated. Since the rule support is not significantly high, with 26.77% being the maximum
value, we can safely assume that the SV model of Tsakas et al. [
We select the first 10K samples from COCO to reduce the inference time needed to extract visual concepts using YOLO-v8 and YOLOS object detectors. COCO provides 5 descriptive sentences per sample, which are paraphrases of each other. For this reason, we only regard the 1st out of the 5 sentences as the conditioning . We follow two separate processes for SG: actual generation conditioned on and retrieval of caption-image pairs based on captions similar to . Conditional generation on COCO captions For the generation experiment, we employ pre-trained difusion models without any further tuning, as mentioned in 4.1, which are all tested on the same conditionings . Each of the four difusion models required about 15 hours to synthesize 10K images using 2 T4 GPUs, therefore around 60 hours in total. Retrieval of COCO-related captions In order to obtain considerably more images conditioned on COCO-related queries without having to spend the time and resources to run many more thousand iterations of the difusion model, we utilized a Stable Difusion search engine (Lexica.art)3. The exact process we used was the following: we use of the first 10K COCO samples as the ’query’ caption. The search engine returned, for each of the 10k captions, 10 images that have been already generated by online communities with the closest input queries to our captions. This technique supplied us with 100.000 more Stable Difusion images, accompanied by their input queries. We then compare results between web-retrieved and generated images. Object detection We select a default threshold of =0.6 for detection; objects detected with confidence ≥ 0.6 are added in the generated concept set . This threshold is experimentally defined to maintain a valid trade-of between false positive and false negative objects; in fact, since no ground truth exists, even defining false predictions is untractable without human inspection. However, our approach can provide relevant hints regarding the probability of false detection, as a higher number of D operations may infer higher false positive rates (irrelevant objects being detected, if is too low), while more I operations can be correlated with higher false negative rates (relevant objects not being detected, if is too strict).
Metric results For comparative reasons we present results for =0.5, 0.6, 0.7 in Tables 5 (YOLO-v8) & 6 (YOLOS) for generated images, and in Table 7 for web images, reporting object extraction from both object detectors. Instances colored in blue denote the lowest scores, which are more desirable, while the highest scores are highlighted with red. We present number of edits (# I, # D, # R), as well as the total cost for each I, D, R operation for all images. Mean CSED is reported as an overall metric regardless of which operation was performed more often.
Regarding the selected threshold , our initial hypothesis is proven to be correct: more I operations are realized for higher threshold =0.7, suggesting that objects from the conditioning where not detected, while fewer I were performed for =0.5. Similarly, there are more D operations for the lowest =0.5, as spurious objects can be detected more easily. Additionally, stable difusion 0.5 stable difusion 2 protogen base protogen 5.8 more R operations are needed for lower thresholds, which is also expected, since more objects are extracted and added to the set. As for object detectors, results using YOLO-v8 are very homogeneous, indicating that the models under investigation follow a rather predictable behavior irrespectively of . Protogen 5.8 consistently yields the lowest mean CSED score, denoting cheaper transitions for all thresholds. This observation slightly changes for =0.7 and YOLOS object detector (Table 6), for which, surprisingly, protogen 5.8 produces the more expensive transitions. By comparing Tables 5 & 6, YOLOS results in higher mean CSED, less I operations, significantly more expensive D operations (even though the number of D operations is not substantially larger), as well as more and expensive R operations. Therefore, we can safely assume that YOLOS is comparatively more sensitive in detecting more objects, which may induce some noise in the detection process. All these results will become more interpretable should we delve into the explanations accompanying the evaluation. The patterns arising from evaluating generated images are also supported in Table 7 findings, verifying the threshold hypothesis, as well as the increased sensitivity of YOLOS. Nevertheless, web-retrieved images seem to miss objects mentioned in the query, as proven by the large number of I and R operations. Local explanations provide edit paths based on the I, D, R operations realized for a specific generated image. For this reason, we employ a scene depicted in Figure 3.
According to YOLO-v8 with the default threshold =0.6, the generated concepts are ={’car’, ’car’, ’trafic light’, ’car’, ’stop sign’}, and ground truth concepts are ={’light’, ’buildings’}. The edit operations of total minimum cost 59.00 for this → transformation are: I: { } D: {’car’, ’car’, ’car’}
When using YOLOS, the generated concepts are ={’car’, ’trafic light’, ’car’, ’stop sign’, ’trafic light’, ’car’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’car’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’car’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’car’, ’car’, ’trafic light’, ’trafic light’}, and the ground truth ones are ={’light’, ’buildings’}. By visually inspecting the image, YOLOS clearly overestimates the actual objects present, inducing noise in the generated concept set . Nevertheless, our evaluation strategy successfully captures this overestimation, by suggesting the deletion of multiple concepts. Specifically, we obtain the following transformations of total cost 104.04: I: { }
D: {’car’, ’trafic light’, ’car’, ’trafic light’, ’car’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’car’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’car’, ’trafic light’, ’trafic light’, ’trafic light’, ’trafic light’, ’car’, ’car’, ’trafic light’, ’trafic light’}
Global explanations for all images are presented in Table 8 for I, D edits and Table 9 for R edits. Results only involve YOLO-v8 extracted concepts, as YOLOS results in an overwhelming number of detected instances. Top-3 results are demonstrated, i.e. the 3 most frequent insertion, deletions and replacements. I and D refers to concepts inserted or deleted respectively, while Freq I, D denotes how many times a specific concepts was inserted or deleted within all images. I, D support indicates the frequency a specific edit happens among all I, D edits respectively. As for R, support denotes the frequency of a transformation rule among all produced rules. 0.5 0.6 0.7
stable difusion stable difusion 2 protogen base protogen 5.8 stable difusion stable difusion 2 protogen base protogen 5.8 stable difusion stable difusion 2 protogen base protogen 5.8
I street table tennis tennis street table tennis street table table tennis street street table tennis table street tennis street table tennis table tennis street table street tennis table street tennis street table tennis table street tennis
We can observe an obvious agreement between models; I edits include ’street’, ’tennis’ and ’table’ concepts. It seems that the selected cannot eficiently generate the I concepts, or generated concepts are of low visual quality, so that their detection is not feasible with =0.5, 0.6, 0.7. D edits mainly contain ’person’, ’sheep’, ’car’, ’umbrella’, ’donut’ concepts, indicating some bias towards generating spurious instances of those concept categories. Finally, R edits D refer to transforming ’person’ to ’people’, ’man’ or ’woman’. Since ’person’ is a YOLO category incorporating both genders, such transformations are somehow expected.
Freq
R Conceptual approaches in generative evaluation is an underexplored field, which can provide some novel insights regarding model quality and explainability of results. In our work, we propose a knowledge-driven explainable evaluation framework that suggests which concepts should be added, removed, or replaced for a generated image to approach its conditioning. Results on competitive tasks such as Story Visualization and Scene Generation illustrate the merits of such an approach, highlighting concepts that models cannot generate, or model biases towards generating excessive numbers of specific concept categories. As future work, we plan to expand our approach to other models and tasks and also incorporate alternative knowledge sources to examine how the produced edit paths conceptually deviate from the current ones.
The research work was supported by the Hellenic Foundation for Research and Innovation (HFRI) under the 3rd Call for HFRI PhD Fellowships (Fellowship Number 5537). 2021. URL: https://arxiv.org/abs/2103.11521. doi:10.48550/ARXIV.2103.11521. [15] Y. Benny, T. Galanti, S. Benaim, L. Wolf, Evaluation metrics for conditional image generation, International Journal of Computer Vision 129 (2021) 1712–1731. URL: https://doi.org/10.1007%2Fs11263-020-01424-w. doi:10.1007/s11263-020-01424-w. [16] K. Abhishek, D. Kamath, Attribution-based xai methods in computer vision: A review, 2022. URL: https://arxiv.org/abs/2211.14736. doi:10.48550/ARXIV.2211.14736. [17] V. Buhrmester, D. Münch, M. Arens, Analysis of explainers of black box deep neural networks for computer vision: A survey, Machine Learning and Knowledge Extraction 3 (2021) 966–989. URL: https://www.mdpi.com/2504-4990/3/4/48. doi:10.3390/make3040048. [18] V. Nagisetty, L. Graves, J. Scott, V. Ganesh, xai-gan: Enhancing generative adversarial networks via explainable ai systems, 2020. URL: https://arxiv.org/abs/2002.10438. doi:10. 48550/ARXIV.2002.10438. [19] A. Genovese, V. Piuri, F. Scotti, Towards explainable face aging with generative adversarial networks, in: 2019 IEEE International Conference on Image Processing (ICIP), 2019, pp. 3806–3810. doi:10.1109/ICIP.2019.8803616. [20] J. Kim, H. Park, Limited discriminator gan using explainable ai model for overfitting problem, ICT Express (2022). URL: https://www.sciencedirect.com/science/article/pii/ S240595952100179X. doi:https://doi.org/10.1016/j.icte.2021.12.014. [21] C. Leiter, P. Lertvittayakumjorn, M. Fomicheva, W. Zhao, Y. Gao, S. Eger, Towards explainable evaluation metrics for natural language generation, 2022. URL: https://arxiv.org/abs/ 2203.11131. doi:10.48550/ARXIV.2203.11131. [22] J. Opitz, A. Frank, Towards a decomposable metric for explainable evaluation of text generation from AMR, in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics, Online, 2021, pp. 1504–1518. URL: https://aclanthology.org/2021.eacl-main.129. doi:10.18653/v1/2021.eacl-main.129. [23] M. Lymperaiou, G. Manoliadis, O. Menis Mastromichalakis, E. G. Dervakos, G. Stamou, Towards explainable evaluation of language models on the semantic similarity of visual concepts, in: Proceedings of the 29th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 2022, pp. 3639–3658. URL: https://aclanthology.org/2022.coling-1.321. [24] G. Filandrianos, K. Thomas, E. Dervakos, G. Stamou, Conceptual edits as counterfactual explanations, in: Proceedings of the AAAI 2022 Spring Symposium on Machine Learning and Knowledge Engineering for Hybrid Intelligence (AAAI-MAKE 2022), Stanford University, Palo Alto, California, USA, 2022. [25] M. Mirza, S. Osindero, Conditional generative adversarial nets, CoRR abs/1411.1784 (2014).
URL: http://arxiv.org/abs/1411.1784. arXiv:1411.1784. [26] A. Odena, C. Olah, J. Shlens, Conditional image synthesis with auxiliary classifier gans, 2017. arXiv:1610.09585. [27] T. Miyato, M. Koyama, cgans with projection discriminator, 2018. arXiv:1802.05637. [28] H. Zhang, I. Goodfellow, D. Metaxas, A. Odena, Self-attention generative adversarial networks, 2019. arXiv:1805.08318. [29] T. Miyato, T. Kataoka, M. Koyama, Y. Yoshida, Spectral normalization for generative adversarial networks, 2018. arXiv:1802.05957. [30] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equilibrium, 2018. arXiv:1706.08500. [31] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, B. Ommer, High-resolution image synthesis with latent difusion models, 2021. arXiv:2112.10752. [32] A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning transferable visual models from natural language supervision, 2021. [33] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K.
Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, M. Norouzi, Photorealistic text-to-image difusion models with deep language understanding, 2022. URL: https://arxiv. org/abs/2205.11487. doi:10.48550/ARXIV.2205.11487. [34] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, Journal of Machine Learning Research 21 (2020) 1–67. URL: http://jmlr.org/papers/v21/20-074.html. [35] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, K. Aberman, Dreambooth: Fine tuning text-to-image difusion models for subject-driven generation, 2022. URL: https://arxiv.org/ abs/2208.12242. doi:10.48550/ARXIV.2208.12242. [36] S. E. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, H. Lee, Generative adversarial text to image synthesis, CoRR abs/1605.05396 (2016). URL: http://arxiv.org/abs/1605.05396. arXiv:1605.05396. [37] S. E. Reed, Z. Akata, S. Mohan, S. Tenka, B. Schiele, H. Lee, Learning what and where to draw, CoRR abs/1610.02454 (2016). URL: http://arxiv.org/abs/1610.02454. arXiv:1610.02454. [38] H. Zhang, T. Xu, H. Li, Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks, 2017, pp. 5908–5916. doi:10.1109/ICCV.2017.629. [39] H. Zhang, T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, D. Metaxas, Stackgan++: Realistic image synthesis with stacked generative adversarial networks, 2018. arXiv:1710.10916. [40] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, X. He, Attngan: Fine-grained text to image generation with attentional generative adversarial networks, in: CVPR 2018, 2018. [41] H. Tan, X. Liu, X. Li, Y. Zhang, B. Yin, Semantics-enhanced adversarial nets for text-toimage synthesis, in: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019, pp. 10500–10509. doi:10.1109/ICCV.2019.01060. [42] M. Zhu, P. Pan, W. Chen, Y. Yang, DM-GAN: dynamic memory generative adversarial networks for text-to-image synthesis, CoRR abs/1904.01310 (2019). URL: http://arxiv.org/ abs/1904.01310. arXiv:1904.01310. [43] Y. Li, Z. Gan, Y. Shen, J. Liu, Y. Cheng, Y. Wu, L. Carin, D. E. Carlson, J. Gao, Storygan: A sequential conditional gan for story visualization, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 6322–6331. [44] C. Li, L. Kong, Z. Zhou, Improved-storygan for sequential images visualization, Journal of Visual Communication and Image Representation 73 (2020) 102956. URL: https: //www.sciencedirect.com/science/article/pii/S1047320320301826. doi:https://doi.org/ 10.1016/j.jvcir.2020.102956. [45] A. Maharana, D. Hannan, M. Bansal, Improving generation and evaluation of visual stories
The following models were used for image generation: • Huggingface text-to-image generation models. • Stable difusion v1.4 • Stable difusion 2 • Protogen x3.4 • Protogen x5.8
The following web-page was used for retrieving already generated images with StableDifusion: • https://lexica.art/ • Huggingface YOLOS-base • Huggingface YOLO-v8
The following models were used for object detection:
In this section, we are going to provide a detailed analysis of the local properties of CSED per
frame for SV regarding the following sequence of Figure 2. This is an example of medium
dificulty, according to the analysis followed in [
For the 1st frame, Consistency Loss (CL) for the generated sequence is =1 = 0, since there are ||=4 semantics in total (Material, Size, Shape, Color), and 1 object containing =4 semantics is placed in k=1 position in the sequence: =1 = =1 = | | − || · =4-4=0. Frame k=2 Ground truth semantics: {[small, brown, metallic, sphere], [small, brown, metallic, sphere]} Generated semantics: {[small, brown, rubber, sphere], [small, brown, metallic, sphere]} There is a diference in the semantic of the 3rd position, highlighted in bold: while the ground truth semantic is ’metallic’, the generated is ’rubber’, therefore =2 proposes the replacement operation ’rubber’ →’metallic’ with Edit cost = 2 = =2. Moreover, as the transformation is an instance involving the Material semantic, one more generation failure is added to the Material Loss counter.
In the same time, CL will inevitably increase just by adding one more object containing || =4 semantics. Therefore, the minimum increase of CL for CLEVR-SV when one object is added can be 4. Other than that, if there are more inconsistencies between = 1 and = 2 generated frames, =2 will increase. Therefore, we compare = 1 generated sequence = − 1={[small, brown, rubber, sphere]} with the = 2 generated sequence = ={[small, brown, rubber, sphere], [small, brown, metallic, sphere]}, where no extra diferences are spotted. By applying equation 5 for k=2 we obtain:
=2 = =1 + (=2, =2) = 0 + I{, , , ℎ} = 0+4 = 4 Frame k=3 Ground truth semantics: {[small, brown, metallic, sphere], [small, brown, metallic, sphere], [large, blue, rubber, cube] } Generated semantics: {[small, brown, rubber, sphere], [small, brown, metallic, sphere], [large, blue, rubber, cube]} The diference in the 3rd position semantic remains, therefore =3 proposes the replacement operation ’rubber’ →’metallic’ with Edit cost = 2 = =3. Moreover, as the transformation is an instance involving the Material semantic, one more generation failure is added to the Material Loss counter.
CL will take into account the comparison between = 2 generated sequence [small, brown, rubber, sphere, small, brown, metallic, sphere] and = 3 generated sequence {[small, brown, rubber, sphere], [small, brown, metallic, sphere], [large, blue, rubber, cube]}, which only difer by the addition of the large, blue, rubber, cube in the third frame, thus yielding: =3 = =1 + (=2, =2) + (=3, =3) = 0 + 4 + I{, , , } = 0+4+4 = 8 Frame k=4 Ground truth semantics: {[small, brown, metallic, sphere], [small, brown, metallic, sphere], [large, blue, rubber, cube], [large, blue, metallic, cylinder]} Generated semantics: {[small, brown, rubber, sphere], [small, brown, metallic, sphere], [large, blue, rubber, cube], [large, blue, metallic, sphere]} Apart from the diference in the 3rd position semantic, for which =4 proposes the replacement operation ’rubber’ →’metallic’ with Edit cost = 2, there is also one diference in the last position semantic, indicating the transformation ’sphere’ →’cylinder’ with Edit cost = 2. By aggregating the two transformations together, we obtain the total transformation for = 4: {’rubber’, ’sphere’} → {’metallic’, ’cylinder’} with Edit cost = 4 = =4. Counters for Material Loss and Shape Loss will increase by 1 each.
For CL, the sequences corresponding to = 3 generated sequence {[small, brown, rubber, sphere], [small, brown, metallic, sphere], [large, blue, rubber, cube]} and = 4 generated sequence {[small, brown, rubber, sphere], [small, brown, metallic, sphere], [large, blue, rubber, cube], [large, blue, metallic, sphere]}, which only difer by the addition of the large, blue, metallic, sphere item. Therefore =4= 4.
=4 = =1 + (=2, =2) + (=3, =3) + (=4, =4) = 0 + 4 + 4 + I{, , , ℎ} = 0+4+4+4 = 12 By aggregating results, Story Loss (SL) as a sum of per frame CSED costs will be: and by averaging SL on all = 4 frames according to equation 7:
= 2 + 2 + 2 + 4 = 10 = 1
= 10/4 = 2.5 For consistency, we follow equation 8:
= = =1 + 1 ∑︁[>1 ̸= || · ( − 1)] = 0 + 0 = 0
=1 The generated story of Figure 2 is fully consistent as the Average CL equals to 0, which is the ideal case. Therefore, no semantics are inserted, deleted, or altered within the generated sequence. It is however interesting that CL cannot capture the faithfulness error between the new item inserted in the 4th frame: while the ground truth item is a large, blue, metallic, cylinder, the generated sequence inserts a large, blue, metallic, sphere, but CL does not penalize more the diference in the semantic of the last position. On the contrary, SL is responsible to penalize for this error. Of course, the opposite scenario could be applicable in a diferent example, where CL would indicate an error that SL could not capture. This observation concludes that both metrics can be important, with SL focusing on faithfulness between ground truth and generated stories, while CL focuses on consistency between consequently generated frames. The better the model, the lower both metrics should be on the global level.