How to Blend Concepts in Difusion Models

Abstract

For the last decade, there has been a push to use multi-dimensional (latent) spaces to represent concepts; and yet how to manipulate these concepts or reason with them remains largely unclear. Some recent methods exploit multiple latent representations and their connection, making this research question even more entangled. Our goal is to understand how operations in the latent space afect the underlying concepts. We hence explore the task of concept blending through difusion models. Difusion models are based on a connection between a latent representation of textual prompts and a latent space that enables image reconstruction and generation. This task allows us to try diferent text-based combination strategies, and evaluate them visually. Our conclusion is that concept blending through space manipulation is possible, although the best strategy depends on the context.

Keywords

• eol>Concept blending
• Generative AI
• Difusion models

1. Introduction

The field of knowledge representation deals with the task of representing the knowledge of a domain in a manner that can be used for intelligent applications [ 1 ]. Over the decades, most of the progress in the ifeld has focused on logic-based knowledge representation languages, and their reasoning capabilities. In this setting, concepts—the first-class citizens of any domain representation—are formalised by limiting the interpretations that they can be assigned to, and their connections with other concepts. A diferent, more implicit approach represents concepts as points (or sometimes volumes) in a multidimensional socalled latent space. This representation (or embedding) is built considering the semantic similarities and diferences between concepts. Although at an abstract level this representation is similar to Gardenfors’s conceptual spaces [ 2 ], there are essential diferences between the two; most notably, concept composition cannot be achieved through simple set operations. Hence, while the use of the latent space is becoming more common, it remains unclear how to navigate it and how to reason within this representation.

Our overarching goal is to understand the properties of the latent space and how diferent operations within it afect the underlying concepts. It is usually understood that every point in the latent space represents a concept, and thus, navigating it has the potential of creating new concepts. In this paper, we focus on the question of concept blending; briefly, the task of creating new concepts combining (“blending”) the properties of two or more concepts [ 3 ] (see Section 2 for more details). We explore the possibility of constructing such blends through (text-to-image) difusion models starting from textual prompts describing the concepts. This choice is motivated by, first, the easy access to the latent space through the textual prompts and, second, the ability to evaluate the quality of the results visually.

We study diferent strategies for concept blending which exploit the overall architecture of Stable

Difusion (SD) [ 4 ]. None of these methods relies on further training or fine-tuning, but they all focus on the topology of the latent space and the SD architecture. In general, visual depictions of concept blends can be automatically generated through these techniques, although their quality may vary. An empirical study was used to evaluate the relative performance of each method. The results suggest that there is no absolute best method, but the choice of blending approach depends on the combined concepts. The task of concept blending considered here is just a milestone towards our general goal of understanding the properties of the latent space and how navigating it afects the underlying concepts. This understanding will be useful towards an explainable use of latent spaces and embeddings in general.

2. Preliminaries

Human creativity has always been the key to the innovating process, giving us the possibility to imagine

things which are yet to be discovered, and diverging scenarios to explore. In recent years, the field of artificial intelligence (AI) has been revolutionized by generative models, which are capable of creating new and original contents by exploiting the countless examples these models have been trained on.

Among the multiple variants and possibilities to exploit generative AI, difusion models like Stable

Difusion [ 5], Dall-E, or Midjourney produce as output original images based on textual prompts or images given as input for the model. To provide clarity for this work, we introduce the fundamental concepts and components of Stable Difusion and the notions of concept blending.

2.1. Stable Difusion

Denoising U-Net Stable Difusion (SD) is a text-to-image generative model developed by Rombach et al. in 2022 [ 4 ], which follows the typical architecture of difusion models [ 6 ] comprising a forward and a backward process. In the forward prompt process, a clean sample from the data (in this case, an encoder image) is sequentially corrupted by random noise reaching, after a defined number of steps, pure random noise.

In the backward process, a neural network is trained to sequentially remove the noise, thereby restoring the clean data distribution; this is the main phase intervening during image generation. The Stable Difusion network architecture utilized during the backward phase is principally made up of (i) a Variational Autoencoder (VAE) [ 7 ], (ii) a U-Net [ 8 ], and (iii) an optional text encoder. The VAE characterizes SD as a Latent Difusion Model, mapping images into a lower-dimensional space through an encoder, followed by a difusion model to craft the distribution over encoded images. The images are then represented as points in the latent space (R). Afterwards, a decoder is needed to convert a point back into an image.

The U-Net is composed of an encoder-decoder pair, where the bottleneck contains the compact embedding representation of the images. The encoder maps the input samples according to the given prompt embedding into this latent embedding, then the decoder processes this latent embedding input prompt together with its prompt embedding to reconstruct a sample that is as close as possible to the original one. The U-Net and text embedding are crucial in conditioning the output generated by the model. At each step of the denoising process, the prompt embedding is injected into the three blocks of the U-Net via cross-attention mechanism. In this way, the textual prompt conditions the denoising process and in turn the generation of an image. The prompt embedding is generated by the text encoder, following the pipeline of SD 1.4. In our experiments, as text encoder, we adopt a pre-trained CLIP Vit-L/14.

With these details we can now establish the focus of this study through the research question: can difusion models produce visual blends of two concepts? Identifying each concept through a word, we want to create a new image that simultaneously represents a combination of both, simulating the human capacity for associative thinking. To address this problem, we present various methodologies leveraging

SD as the backbone of our experiments. But before, we explain the notion of concept blending.

2.2. Concept Blending

Blending represents a cognitive mechanism that has been innately exploited to create new abstractions from familiar concepts [ 9 ]. This process is often experienced in our daily interactions, even during a casual conversation. This conceptual framework has been studied over the past three decades [ 10 ], ofering a model that incorporates mental spaces and conceptual projection. It examines the dynamic formation of interconnected domains as discourse unfolds, aiming to discover fundamental principles that underlie the intricate logic of everyday interactions. In this context, a mental space is a temporary knowledge structure, which is dynamically created, for the purpose of local understanding, during a social interaction. It is composed of elements (concepts) and their interconnections. It is contextdependent and not necessarily a description of reality [ 11 ]. This general notion can be specified in diferent notions. For our purpose, we are interested in visual conceptual blending, which combines aspects of conceptual blending and visual blending.

Conceptual Blending constructs a partial match between two or more input mental spaces, and project them into a new “blended” one [ 11 ]. This blended space has common characteristics of the input spaces, allowing a mapping between its elements and their counterparts in each input space. Yet, it also generates a new emergent conceptual structure, which is unpredictable from the input spaces and not originally present in them. Therefore, blending occurs at the conceptual level. Representations of these blends are valuable and frequently employed in advertising [ 12 ] and other domains [ 13 ].

The Visual Blending process, instead, is essential to generate new visual representations, such as images, through the fusion of at least two existing ones. There are two primary options for visual blending, according to the rendering style employed: photo-realistic rendering and non-photo-realistic techniques, like drawings. Approaches that focus on text-to-image generation have as main goal the visual representation of concepts, and, in the case of blending, the topology could to be summarized as a bunch of visual operations, as analyzed by Phillips and McQuarrie [ 14 ]. One of these operators, called fusion, partially depicts and merges the diferent inputs to create a hybrid image, allowing for a higher coherence between the object parts of the object(s), and helping the viewers in perceiving the hybrid object as a unified whole. In replacement, one input concept is present and its sole function is to occupy the usual environment of the other concept, or have its shape adapted to resemble the other input. Juxtaposition is a technique that involves placing two diferent elements side by side, to create a harmonious or provoking whole. Good example of Visual Blending and diferent approaches to the operations described (and others) can be found in [ 15 ]. Importantly, high-quality blends between concepts require that only some of the main characteristics of the input concepts are taken into account [ 16 ]. Exploiting the three main visual properties of color, silhouette, and internal details, helps the creator to obtain a great resulting blend. An image result from blending can be evaluated by taking in account the number of dimensions (or visual properties) over which the blend has been applied.

Visual Conceptual Blending introduces a model for creating visually blended images grounded in strong conceptual reasoning. Cunha et al. [ 17 ] argue that visual conceptual blending goes beyond simply merging two images: it emphasizes the importance of conceptual reasoning as the foundation of the blending process, resulting in an image and accompanying conceptual elaborations. These blends have context, are grounded in justifications, and can be named independently of the original concepts.

In contrast, standard Visual Blending focuses solely on image merging, and typically involves mapping concepts to objects and integrating them while maintaining recognisably and inference association.

We now rephrase our research question as: can Stable Difusion models merge two semantically distant concepts into a new image, practically performing a Visual (Conceptual) Blending operation? We investigate the eficacy of difusion models, which are supposed to recreate each image that should be imagined, in generating high-quality blended images. We assess existing approaches to perform blending with stable difusion, and propose novel methods. To the best of our knowledge, this is the first investigation that evaluates the performance of diferent blending techniques with difusion models using only textual prompts. We initially operate on the latent space where the textual prompts are embedded, and then explore alternative methods by directly manipulating the specific architecture of the difusion model; more precisely, the U-Net conditioning phase is manipulated to edit the textual prompt that is injected (Section 3). To evaluate the results, we conducted a user survey where the subjects were asked to rank the outcomes of diferent blending tasks, divided in multiple categories.

3. Blending Methods with Stable Difusion

In this section, we briefly review some of the existing approaches for blending concepts with difusion models. Some of these methods were already published in previous work [18], while others are available on public implementations, but without a full description of their details. We mention explicitly whenever we are unsure if our implementation matches exactly the one proposed in the reference.

Experimental setup We fix the generative network as Stable Difusion v1.4 [ 4 ] with the UniPCMultistepScheduler [ 19 ] set at 25 steps. This version uses a fixed pretrained text encoder (CLIP Vit-L/14 [ 20 ]).

All images are generated as 512x512 pixels with the difusion process carried in FP16 precision in a

space downscaled by a factor of 4. The conditioning signal is provided only in the form of textual prompts, and the guidance scale is set to 7.5. We focused on Stable Difusion as a good trade-of between quality and computational cost; however, the blending methods analyzed can be implemented in other difusion models with no latent downscale. Our entire implementation of the blending methods in their respective pipelines together with some of the generated samples is openly available.1

An important feature of many generative methods, which allows them to produce varying outputs on the same prompt, is the use of a pseudo-random number generator (and pseudo-random noise) which can be established through a seed. Given an input textual prompt , and a seed , we denote as , = (, ) the image generated by the model given the input prompt and the seed . Prompts will be usually denoted with the letter , sometimes with additional indices to distinguish between them; e.g., 1 and 2 when two diferent prompts are used simultaneously. Given a prompt , * denotes its latent representation; that is, the multi-dimensional vector obtained from the encoding operation. Similarly, *1 and *2 denote the latent representations of 1 and 2, respectively.

3.1. Blending in the Prompt Latent Space (TEXTUAL)

The first method examined was recently proposed by

Melzi et al. [ 18 ]. It exploits the relationship between conceptual blending and vector operations within the prompt latent space. Given the two input prompts 1 and 2, we first compute their latent representations *1 and *2 through the prompt encoder. The blended latent vector is the Euclidean mean between *1 and *2. The blended image is generated by conditioning SD with the blended latent vector.

1Project repository: https://github.com/LorenzoOlearo/blending-difusion-models

Denoising U-Net n a e m

Importantly, blending in the latent space representing the prompts does not correspond to blending images directly, as in a visual blending process. Instead, it means generating an image representing a specific fusion of the concepts provided as the input textual prompts. Indeed, the Euclidean mean between the two representations is a (potentially unexplored) point of the latent space which intuitively represents the concept that is closest to both input concepts, thus defining an “in-between” characterisation. Although in this paper we only consider the mean of the two latent representations of the input prompts, we highlight that Melzi et al. consider also other linear combinations of *1 and *2 to avoid fully symmetric constructions. A similar technique was implemented in the Compel open source library,2 which performs the weighted blend of two textual prompts.

3.2. Prompt Switching in the Iterative Difusion Process (SWITCH)

Denoising U-Net

This blending technique involves switching the textual

prompt during the iterative process of the difusion model. The inference process first starts with a single prompt 1 and then, at a certain iteration, the prompt prompt is switched to 2 until the end of the difusion process. encoder

The generation is thus conditioned on both prompts The first N denoising iterations

leading to an image that, when the switch is executed at The last M denoising iterations the right timestep, blends the two concepts. Intuitively, SWITCH starts by generating the general shape of 1, but then fills out the details based on 2 thus producing a visual blend of the two concepts.

It is crucial to choose the right iteration to switch the prompt. Unfortunately, this is an intrinsic

challenge for each new image and does not depend only on the geometric distance between the *1 and *2 embeddings. From our experiments, it was observed that the optimal iteration for this switch is directly related to the spatial similarity between the image generated by the model conditioned only on 1 and the one generated by 2. This technique was also implement in the Stable Difusion web UI developed by AUTOMATIC1111.3 Among its numerous functionalities, this implementation allows prompt editing during the mid-generation of an image.

3.3. Alternating Prompts in the Iterative Difusion Process (ALTERNATE)

In general difusion models, at each timestep defined by

the scheduler of the difusion process, the noise in the sample is estimated by the U-Net model. This estimation is performed by the model with knowledge of the prompt timestep and the conditioning signal (i.e., the prompt). encoder The Alternating Prompt technique conditions the U-Net itEervaetnion with a diferent prompt at each timestep: the prompt iteOrdatdion 1 is shown to the U-Net at even timesteps, while 2 is shown at odd timesteps. By performing this alternating prompt technique, the difusion pipeline can successfully generate an image that blends the two given prompts. Even though at diferent timesteps, the U-Net is conditioned by both prompts during the difusion process. The blending ratio can be controlled by adjusting the number of iterations in which each prompt is shown to the U-Net. One can intuitively think of this approach as an alternating superposition of the generation process between 1 and 2. This method is also implemented in the Stable Difusion web UI developed by AUTOMATIC1111. Denoising U-Net

2Compel: https://github.com/damian0815/compel

3Stable Difusion web UI: https://github.com/AUTOMATIC1111/stable-difusion-webui

4. Method

We now propose a diferent blending paradigm to visually combine two textual prompts in the difusion pipeline. In a standard difusion architecture, given a single input prompt , its corresponding embedding * is injected with a cross-attention mechanism in the three main blocks of the U-Net: the encoder, the bottleneck, and the decoder. During the encoding and bottleneck steps, the * embedding is used to guide the compression of the input sample into a latent representation that accurately maps the concept that is being generated. Then, during the decoding phase, the * embedding is used to guide the reconstruction of the sample towards the distribution of the concept that is being generated. Our idea arises from this compression and reconstruction operation and it is described in the following subsection. To the best of our knowledge, this method has not been proposed before.

Diferent Prompts in Encoder and Decoder Components of the U-Net (UNET)

We implement our new method using text-based con- Denoising U-Net ditioning, but it can theoretically be extended to other conditioning domains. As describe above, the U-Net architecture contains three main blocks: the encoder, the prompt bottleneck and the decoder. Each of these block receives encoder the prompt embedding * as input together with the sample from which the noise has to be estimated.

The key idea our method involves guiding the com

pression of the sample into the bottleneck block with a first prompt embedding *1. Then, guide its reconstruction towards the distribution of the second prompt 2 by injecting into the decoder block the embedding *2 as visualized in the figure. This allows the U-Net to construct a latent representation for the sample matching the concept described by 1 and then reconstruct the sample with the features of the second prompt 2.

The expected result from this technique is to obtain an image that globally represents or recalls the concept described by 1 and simultaneously shows some of the features that typically describe the concept of the second prompt 2. From our findings, changing the prompt embedding in the bottleneck block does not significantly afect the final result. Consequently, we keep the prompt 1 in the decoder and bottleneck blocks while we change the prompt 2 in the encoder block. input prompts 1 2

5. Validation and Results

We now describe the experimental setting and analysis made to evaluate the four blending approaches presented in the previous sections, applied over two simple conceptual prompts. The outputs of these models can be visualized in Figures 2 and 3. The experiments aimed to assess previously proposed blending methods across four distinct macro-categories, which are visually explained in Figure 2. The four categories are pair of animals, object and animal, compound words, and real-life scenarios. These were selected to showcase diferent kinds of blending of concepts, which are expected to showcase diverging properties. For pairs of animals, we expect that the shared characteristics between the concepts will aid the blending process; the use of object and animal concepts in the second category is expected to widen the semantic gap between the input prompts, leading to more “creative” artifacts. The third category considers objects representing compound words, ofering a more conceptual blending challenge. Here, we observed how the methods responded to prompts comprising the compound’s constituent parts, which are not literal descriptions of the target object but are rather interpretable as a figure of speech or metaphor. We aimed to investigate whether the models should learn the necessary abstractness to perform a blending similar to the concept associated with the compound word, or reach a new visual blending that merges the characteristics of the two prompts. The last category draws inspiration from real-world visual blend examples, regardless of their underlying concepts, deriving prompts to condition the models, allowing us to investigate their adaptability and ability to reconstruct well-known blends.

TEXTUAL SWITCH ALTERNATE UNET lsa ioLn imra n G A + t c je in b a O r

B e l t r u T y l F rsd ttre o u WB d n u o p To impartially evaluate the quality of the methods, we conducted a survey on 23 participants involving 24 images from the four categories described. The survey was constructed as follows. We first selected 24 concept pairs covering examples from the four macro-categories; each concept in the pair was described through a simple prompt. Then, the four diferent blending methods were used to generate the visual conceptual blend of each pair. All images were generated with the same size and quality, and presented to the users, with the instruction to rank them according to their blending efectiveness from best to worst. Our participant pool was carefully selected to ensure they had no prior experience with blending theory. While the two prompts used to generate the images were provided to the subjects, we deliberately withheld information regarding the model responsible for each image, eliminating potential bias. Additionally, to further mitigate bias, the order of images within each question was randomized for each participant. This approach aimed to discern whether a superior blending method existed among the four proposed and whether certain methods outperformed others within specific categories.

For each question, the top four images proposed were selected by the authors from a pool generated

using ten diferent seeds. Given that blending quality across all methods is not entirely independent of seed choice, we aimed to minimize this dependency by carefully selecting the best results. For a better understanding of the evaluation approach, Figure 2 shows some of the images that were presented to the subjects for ranking, along with the methods that produced them.

Table 1 summarises the results of the survey, indicating the mean and mode (i.e., most frequent) rank given to each method for each prompt pair and summarizing the results by category and globally. In both cases, a higher value means a lower quality blend perceived by the subjects of the survey. The goal of this analysis is to understand which blending method performs better in general (for the global summary) and in a more fine-grained manner by category and by prompt pair. We emphasise that the mean value should be handled with care, as a few low rankings (value 4) can greatly skew to mean of a ranking that is typically considered of high quality. Indeed, in the last row of the table we can observe that the average ranking of all methods throughout the whole experiment is quite similar, even though SWITCH is most frequently selected as the best method, and UNET as the worst. Worth noticing is also that the mode does not necessarily provide a full ranking between methods.

In the next section we will discuss the merits of the presented blending methods and the results of the user survey; yet, for the moment we can already see that, at least from the perspective of the ranking given, there is no clear best blending approach, but quality varies between images, and more broadly between categories. For instance, UNET was ranked fourth in three categories, but second for the category of real-life scenarios. Similarly, although UNET’s mode rank in compound words was 4, it was also the highest ranked in three of the prompt pairs in this category.

6. Discussion

Figure 3 shows the results of the four diferent blending methods with the prompts Frog-Lizard, ButterFly, Kung fu-Panda, Tortoise-Broccoli, and Tea-Pot. To better understand the behavior of each method, all images in each row were generated using the same seed and thus starting from the same random noise.

Moreover, the blending ratio between the two prompts was kept constant at 0.5 across all methods.

We measure the visual distance of two concepts by visually evaluating the spatial similarity of the images generated conditioning the pipeline on them. This is a key aspect to consider when evaluating the quality of the blend as, with the exception of the TEXTUAL which instead focuses on the semantic blend, it influences the performance of the blending methods.

When it comes to logical blends, one often considers a main concept which is modified by a secondary one. That is, the blended concept is primarily an instance of the main concept, but with some characteristics that recall the secondary concept. With the exception of TEXTUAL, the blending methods presented in this paper are not symmetric, meaning that the order of the prompts in the blend afects the final image. This is particularly important when dealing with compound words like pitbull, although s l a m i n A f O r i a P l a m i n A + t c e j b O s d r o W d n u o p m o C l a e R

Prompt Elephant-Duck

Lion-Cat Frog-Lizard Fox-Hamster

Rabbit-Dog CATEGORY TOTAL

Turtle-Brain Pig-Cactus

Garlic-Swan Coconut-Monkey Tortoise-Broccoli

Turtle-Wood

Turtle-Pizza CATEGORY TOTAL

Butter-Fly Dragon-Fly

Bull-Pit Blimp-Whale

Jelly-Fish Fire-Fighter

Tea-Pot Snow-Flake

Cup-Cake CATEGORY TOTAL

Kung fu-Panda

Man-Bat

Beaver-Duck CATEGORY TOTAL GLOBAL TOTAL this word commonly refers to a specific breed of dog, its intrinsic semantic and historical meaning refers to bull in a pit. When visually blending the two concepts pit and bull with the methods illustrated in this paper, it is important to take into account which of the two concepts is the main one and which is the modifier . By analyzing the results in Figure 3, it is evident that this primary-modifier relationship is not coherent across all the analyzed methods. In TEXTUAL and ALTERNATE, the main concept of the blend appears to be the second prompt while its modifier is the first one. The opposite behavior is instead what characterizes SWITCH and UNET where the main concept of the blend is the first prompt and the modifier is the second one. This behavior was not expected; to keep the experiments straightforward all blends were generated considering the first prompt as the main concept and the second as its modifier . This is the reason why when blending the words that make the compound word Pitbull, the blend is generated as a Bull-Pit instead of Pit-Bull.

As expected from the work by Melzi et al. [ 18 ], performing the blending operation in the latent space of the prompts, as in the case of TEXTUAL, does not always lead to an image that visually blends the two concepts. This is particularly evident in the case of Kung fu-Panda, where the generated image is a conceptual blend of the two prompts. From our findings, TEXTUAL usually produces inconsistent results, although the conditioning embedding given to the pipeline always remains the same, the balance d r a z i L g o r F y l F r e t t u B a d n a P u f g n u K i l o c c o r B e s i o t r o T t o P a e

T between visual and semantic blending changes drastically from one seed to another. An instance of this behavior can be observed in its Kung fu-Panda sample at Figure 2. In this case, the model generated possibly the best visual blend out of the four methods, however, out of all the other seeds tested, no other sample was able to achieve the same level of blending.

As mentioned already, results from SWITCH vary considerably depending on the timestep at which the prompt is switched; finding the right timestep is crucial to achieve a good visual blend. This is evident in the case Tea-Pot and Butter-Fly shown in Figure 3: the images generated from the prompts Butter and Fly are visually distant even though both of them start from the same initial noise. When in the middle of the difusion process the prompt is switched, the model is unable to shift and correct the existing distribution towards the one of the new prompt and only the first prompt is retained in the blend. Another undesired behavior of SWITCH is the cartoonification of the produced blend.

The difusion pipeline, when unable to shift the pixel distribution towards the new prompt, corrects

the existing noisy image latent by progressively removing the high-frequency details, resulting in a cartoonish image. This behavior can be clearly observed in the Kung fu-Panda blend produced by SWITCH in Figure 3. From our experimental results, this behavior does not afect the other methods.

The ALTERNATE method, which alternates between the two prompts at each timestep, tends to produce consistent results when the two blended concepts are visually very diferent. What is arguably even more interesting is the type of visual blend that this technique produces when the two concepts are both visually and semantically very diferent. This is the case of Tea-Pot and Butter-Fly, where the model creates an image that literally and spatially contains both the first and the second prompt. This is also evident in the Bull-Pit blend in Figure 2, where the ALTERNATE generates what could be described as a bull in a pit. TEXTUAL also seems to produce a similar results but once again, it is too inconsistent across the seeds space to state it as a general rule.

Compared to the other approaches, the UNET method which encodes in the U-Net the image latent conditioned with the first prompt and then decodes it with the second one, produces more subtle blends but generally consistent results. This might be the reason why this is the blend method that performs worse in the survey, as the visual blend is not as evident as in the other methods. Interestingly, on the Kung fu-Panda blend, UNET seems to slightly change the visual representation of the first prompt while matching the colors of the second one. This subtle blending is also evident in the Bull-Pit blend of Figure 2, where surprisingly the pipeline creates an image that somewhat resembles a pitbull.

The results of the survey summarized in Table 1 show that the most preferred method is SWITCH, however, this comes with some considerations. In order to better represent each method, in the survey we have chosen the best settings for each method, in the case of SWITCH this translates into using the optimal timestep in which to switch the prompt for each blend. Finding this value is a tedious process made by trial and error, with no clear and empirical way to determine it. Although UNET ranked the lowest in the survey, while comparing its results with the ones of SWITCH with a fixed switch timestep in the middle of the difusion process (Figure 3), it is evident that the visual blend produced by these two methods are generally similar if not better in the case of UNET.

7. Conclusions

Through this paper we tried to answer a novel research question: is it possible to produce visual concept blends through difusion models? We compared diferent possible solutions to force a difusion model (more specifically Stable Difusion [ 4 ]) to generate contents that represent the blend of two separated concepts. We collected three diferent alternatives from existing publications and from the web. Additionally, we propose a completely new method, which we call UNET that exploits the internal architecture of the adopted difusion model. We collected the outputs of the diferent methods on 4 diferent categories of test; namely, pairs of animals, animal and object, compound words, and real life scenarios. For each of these categories we produced various diferent pairs of concepts, and generated all blends (in total, four blended images for each pair of prompts).

The quality of a blend, as any creative endeavor, has a subjective component on it. Thus, to evaluate

which approach is more adept at this task (in relation to human perception) we devised a user study that was run by 23 subjects. In it, participants were asked to rank the results of the blending methods.

It is worth noting that two participants did not rank all methods, but 21 full surveys were submitted.

We still used the partial surveys to compare those pairs where the ranks were available.

From the user study it results that there is no single best blending method, but the perceived quality

varies from pair to pair and, more importantly, from category to category. And yet, from a positive perspective, we can answer our research question on the afirmative: is it possible to produce visual conceptual blends through difusion models, and the results are often quite compelling (see the samples in Figure 2. Indeed, the survey participants expressed surprise with some of them.

An important point to make is that, for this work, we used the latent space from Stable Difusion

directly; that is, without any kind of fine-tuning or added training. Thus, our results are less fragile towards model updates, and do not require significant efort to implement and execute. This is consistent with our original stated goal of understanding how to manipulate the latent space as a representation of concepts. This work only scratches the surface of this topic and we hope that it can inspire new discussion and further analysis.

For future work, note that our blends are based on very simple (mainly one-worded) prompts. This

allows us to better understand the impact of the operations (in contrast to the subtleties of promptengineering) but has the disadvantage of working over very general concepts, in general, and more in particular is prone to ambiguities and misinterpretations. It would thus be interesting to explore ways to guarantee a more specific identification of concepts selected for blending.

Work funded by the European Union–Next Generation EU within the project NRPP M4C2, Invest

ment 1.,3 DD. 341-15 march 2022–FAIR; Future Artificial Intelligence Research – Spoke 4-PE00000013

D53C22002380006. Part of this work was supported by the MUR for REGAINS, the Department of

Excellence DISCo at the University of Milano-Bicocca, the PRIN project PINPOINT Prot. 2020FNEB27,

CUP H45E21000210001, and by the NVIDIA Corporation with the RTX A5000 GPUs granted through the Academic Hardware Grant Program to the University of Milano-Bicocca for the project “Learned representations for implicit binary operations on real-world 2D-3D data.”

References

[1] R. Brachman , H. Levesque , Knowledge representation and reasoning , Morgan Kaufmann, 2004 .

[2] P. Gardenfors , Conceptual Spaces: The Geometry of Thought, A Bradford book , MIT Press, 2004 . URL: https://books.google.it/books?id= FSLFjw1EcBwC .

[3] G. Fauconnier , M. Turner , The Way We Think: Conceptual Blending And The Mind's Hidden Complexities , Basic Books, 2008 . URL: https://books.google.it/books?id=FdOLriVyzwkC.

[4] R. Rombach , A. Blattmann , D. Lorenz , P. Esser , B. Ommer , High-resolution image synthesis with latent difusion models , in: Proc. IEEE/CVF conf. on comp. vision and pattern recog. , 2022 , pp. 10684 - 10695 .

[5] D. Podell , Z. English , K. Lacey , et al., Sdxl: Improving latent difusion models for high-resolution image synthesis , arXiv preprint arXiv:2307 . 01952 ( 2023 ).

[6] J. Sohl-Dickstein , E. Weiss , N. Maheswaranathan , S. Ganguli , Deep unsupervised learning using nonequilibrium thermodynamics , in: Proc. ICML'15 , volume 37 , PMLR, Lille, France, 2015 , pp. 2256 - 2265 . URL: https://proceedings.mlr.press/v37/sohl-dickstein15. html .

[7] D. P. Kingma , M. Welling , Auto-encoding variational bayes , arXiv preprint arXiv:1312.6114 ( 2013 ).

[8] O. Ronneberger , P. Fischer , T. Brox , U-net: Convolutional networks for biomedical image segmentation , in: Proc. MICCAI 2015 , Springer, 2015 , pp. 234 - 241 .

[9] R. Confalonieri , A. Pease , M. Schorlemmer , et al., Concept Invention : Foundations, Implementation, Social Aspects and Applications, Computational Synthesis and Creative Systems , Springer, 2018 .

[10] F. J. Costello , M. T. Keane, Eficient creativity: Constraint-guided conceptual combination , Cognitive Science 24 ( 2000 ) 299 - 349 .

[11] G. Fauconnier , Mental spaces: Aspects of meaning construction in natural language , CUP , 1994 .

[12] A. Joy , J. F. Sherry Jr, J. Deschenes , Conceptual blending in advertising , Journal of business research 62 ( 2009 ) 39 - 49 .

[13] O. Kutz , J. Bateman , F. Neuhaus , T. Mossakowski , M. Bhatt , E pluribus unum: Formalisation, use-cases, and computational support for conceptual blending , in: Computational Creativity Research: Towards Creative Machines , Springer, 2014 , pp. 167 - 196 .

[14] B. J. Phillips , E. F. McQuarrie , Beyond visual metaphor: A new typology of visual rhetoric in advertising , Marketing theory 4 ( 2004 ) 113 - 136 .

[15] P. Xiao , S. Linkola , et al., Vismantic: Meaning-making with images ., in: ICCC, 2015 , pp. 158 - 165 .

[16] L. B. Chilton , E. J. Ozmen , S. H. Ross , V. Liu, Visifit: Structuring iterative improvement for novice designers , in: Proc. 2021 CHI Conf. on Human Factors in Computing Systems , 2021 , pp. 1 - 14 .

[17] J. Cunha , P. Martins , P. Machado , Let's figure this out: A roadmap for visual conceptual blending , in: Proc. of International Conference on Innovative Computing and Cloud Computing , 2020 .

[18] S. Melzi , R. Peñaloza , A. Raganato , Does stable difusion dream of electric sheep? , in: Proc. ISD7 , volume 3511 of CEUR, CEUR-WS.org , 2023 . URL: https://ceur-ws. org/ Vol- 3511 /paper_09.pdf.

[19] W. Zhao , L. Bai , Y. Rao , J. Zhou , J. Lu , Unipc: A unified predictor-corrector framework for fast sampling of difusion models , 2023 . arXiv: 2302 . 04867 .

[20] A. Radford , J. W. Kim , C. Hallacy , et al., Learning transferable visual models from natural language supervision , in: Proc. of International conference on machine learning, PMLR , 2021 , pp. 8748 - 8763 .