Latent Difusion Models have recently emerged as the state-of-the-art approach for synthetic image generation. In the Web context, their adoption may significantly impact the way it is currently approached, from both sides of content generation and exploration. For example, future Web platforms may create alternative and personalised images for individual users or improve the accessibility for users with disabilities. However, due to the nascent stage of this research area, there remains a knowledge gap in efectively utilising these models, which can clutter the digital space with poor-quality AI-generated, thus diminishing the overall perceived impact and the user experience. To address this issue, we propose a novel methodology aimed at generating high-quality prompts with minimal user efort. In particular, we present BLACK (Background, Lighting, Amenities, Context, and Kinesis), a prompt generation model directly designed for achieving high-quality images satisfying a proposed set of five desiderata. We supplement our methodology proposal with illustrative examples, intended to clarify its mechanisms for the reader. As a second contribution, we publicly release a structured resource of prompts along with expected results.
Prompting in automatic image generation is a process that involves providing a textual
description or instruction as input for image generation [
Similarly to web search, where users must eficiently explore a vast information space to retrieve the most useful or relevant information, prompting addresses the problem of orienteering. Orienteering involves exploring search results to determine their relevance to the query, potentially requiring iterative query reformulation.
In the context of automatic image generation, prompting can be viewed as a form of orienteering that helps users navigate the space of possible images that can be generated automatically. However, refining keywords for textual search involves immediate human comparison, whereas exploring an image space can be more challenging, as the links between textual prompts and generated images can be less explicit. CHITALY 2023 https://evilscript.eu/ (F. Torrielli)
This paper aims to model prompting in the context of image generation, identifying patterns of positive cases and typical pitfalls. However, there are diferences between searching within an information space of existing elements and generating through prompting. In the former case, once results are obtained, the user can reformulate based on what has been retrieved. In the latter case, reformulation is based on what has been generated, introducing an additional element of uncontrolled variability. This creates a more significant gap between the threefold space of prompting engineering, expectation, and reformulation.
The widespread adoption of latent difusion models, such as Stable Difusion [
In this work, we present a novel methodology for image generation that builds upon previous literature and recent studies on generative models. Our approach leverages precise tokenisation, it minimises the use of irrelevant tokens, and it incorporates small, incremental changes to produce consistent outputs. We think that this approach might become one possible standard practice for prompt engineering, involving factual descriptions, style changes, etc. within a structured pipeline we named BLACK (Background, Lighting, Amenities, Context, and Kinesis). Our methodology draws upon the latest advancements in the field and provides a systematic framework for generating images. Furthermore, we created a structured lexical resource of prompts following the proposed pipeline and examples of expected results2. 1https://www.midjourney.com/ 2https://docs.google.com/spreadsheets/d/1xer8JvcsVYEVsRMhksJD--JaAXzZuJ6QJjyT3b5bpJc
This proposal aims at placing some fundamental bricks towards the creation of bases and guidelines to systematically generate a large number of diferent images. Our newly introduced method, called BLACK, relies on a fixed token structure and a permutation of single or multiple terms to achieve results ranging from simple and straightforward to complex and intricate. The prompting phase of the proposal derives from the current literature on text-to-image generation and can be divided into two parts: the factual description (Subject + Verb + Object) and the style embellishment.
A working template that is commonly utilised for this prompt is ” A / T h e [ s u b j e c t ] [ v e r b ] [ s u p p o r t t o k e n s ] [ o b j e c t ] ” where the verb and the support tokens (e.g. conjunctions) can be optionally included or excluded [9]. These groups of tokens are comma separated and can be weighted diferently. The use of templates is justified because tests have shown that maintaining focus on keywords is more impactful than rephrasings during the prompt engineering process [8].
The definition of a style is another fundamental aspect of image generation. A style encompasses both the content and the form of an artistic piece: it determines how the content is represented, illustrated and conveyed [10]. Since style depends heavily on the weights of the model, there are checkpoints3 that are either tailored to a particular style or general enough to handle diferent tasks. Synthetic style is defined in one or few words that are unique to artists, art movements and everything that is radically recognisable: everything that has influential aesthetic can be a style (e.g. cyberpunk) [9]. An example of a working prompting template is the following: [ f a c t u a l p r o m p t ] i n t h e s t y l e o f [ s t y l e n a m e ] . When discussing style, it is optimal to be concise and direct, and this approach should always be considered. In all observed cases, using single-term style names has been found to be more impactful than using style definitions.
In this paper, we introduce a novel method called BLACK for text-to-image prompting techniques, which aims to improve the generation of synthetic images using tools such as Stable Difusion. The name BLACK is an acronym for its five key components: Background, Lighting, Amenities, Context, and Kinesis.
Background represents the environment where the subject is located, while Lighting groups light, shadows, and everything in between. Amenities are model-specific tokens essential for highly-detailed generation. Context tokens work alongside style, and Kinesis tokens are used for movement, motion blur, and dramatic representation of reality.
When using tools like Stable Difusion, prompts are employed to add features to an image, with the model navigating the latent space and collecting features that are direct mappings of the prompts. Negative prompts [11] can be utilised to instruct the model on what to avoid while 3In Stable Difusion, checkpoints are pre-trained weights or models that can generate images of a specific genre or style. These checkpoints are created by training the model on a dataset and can be used to generate images of a particular genre or style. navigating the latent space. This can be particularly useful for constraining any generative model, as specifying that unwanted colours, elements, or styles should be avoided or reduced can help eliminate undesirable variations and steer the output towards the desired themes.
The background context provided in a prompt can encompass the environment surrounding a character or object. This context can be minimally specified with a simple phrase like ’ f l a t b l a c k b a c k g r o u n d ’ or more intricately described in vivid detail. However, more lengthy background descriptions are discouraged, as they can inadvertently ’bleed’ into and influence the generated subject. For instance, a prompt specifying ’ p i n k - g o l d a r a b i c e r a v e r y i n t r i c a t e c i t y b a c k g r o u n d ’ may result in the generated subject adopting pink or gold colours approximately 20% of the time, as the background context exerts an unintended efect on the subject’s attributes.
Lighting and shadows are crucial components of high-quality art generation. The choice of appropriate lighting depends on the background context and subject matter. As discussed in the Experiments section that follows, lighting selection holds a greater impact than may be intuitively evident. For instance, pairing a dark, ominous background with ’ e e r i e l i g h t s ’ or a cheerful, natural subject with ’ c i n e m a t i c l i g h t i n g ’ or ’ s o f t l i g h t i n g ’ can significantly influence the tone, interpretation, and perception of the generated art, as will be demonstrated.
Despite its name, the ’amenities’ component of a prompt is critical to generating high-quality outputs. ’Amenities’ refers to unique keywords that enhance the final product, and for that reason Oppenlaender calls them ’Quality Boosters’ [9]. These keywords are drawn from the training dataset and reflect its content. For example, prompts for models trained on the LAION dataset 4 or DeviantArt5 data might include fundamental terms like ’ 8 k , b e s t q u a l i t y , m a s t e r p i e c e ’ . Prompts targeting 3D-rendered art might instead include phrases like ’ t r e n d i n g o n C G S o c i e t y , e x t r e m e l y d e t a i l e d C G u n i t y w a l l p a p e r ’ . In general, selecting amenities relevant to the target style or dataset is essential to successful generation.
The ’context’ refers to a set of categories of tokens that include color palettes, level of detail, setting, and complementary style tokens (e.g. ’ v i b r a n t ’ , ’ m u t e d ’ , etc.). These tokens encode information such as time of day, whether the art is an illustration, 3D rendering or photograph, camera angles, and other relevant details. Additionally, context tokens may represent season (e.g. ’ w i n t e r ’ ), time period (e.g. ’ m e d i e v a l ’ ), subject matter (e.g. ’ l a n d s c a p e ’ ), and mood (e.g. ’ m e l a n c h o l y ’ ). 4https://laion.ai/blog/laion-5b/ 5https://www.deviantart.com/
The ’kinesis’ component refers to tokens that encode motion and pose attributes, such as movement (e.g. ’ w a l k i n g ’ , ’ r u n n i n g ’ , ’ j u m p i n g ’ ) or stance (e.g. ’ s i t t i n g ’ , ’ c r o u c h i n g ’ , ’ l y i n g d o w n ’ ). These tokens are often used in conjunction with context tokens to specify the nature of the movement, such as ’ s p r i n t i n g t h r o u g h a f o r e s t ’ or ’ m o t i o n b l u r ’ . Including kinesis tokens can help generate dynamic art with a sense of action or energy. The specific kinesis tokens selected will depend on the desired style and subject matter. For example, prompts for sports photography may include additional athletic motions like ’ k i c k i n g ’ , ’ t h r o w i n g ’ , or ’ s w i n g i n g ’ , while prompts for portraits could include more subtle poses like ’ t i l t i n g h e a d ’ or ’ c r o s s i n g a r m s ’ .
In this section, we demonstrate the ability of our methodology to generate art that is faithful to the prompt. To achieve this, we utilise a trained checkpoint called DreamShaper [12], which is capable of generating art in styles ranging from realistic to fantastical. To ensure consistent results, we fix the random seed for each generation, then only modifying individual tokens in the prompt.
In generating high-quality images, the selection of appropriate parameters is crucial. In our experiment, we employed the DPM++ Keras solver [13] to sample 20 steps on images with dimensions ranging from 512 × 512 to 512 × 768 pixels. The CFG scale was set to 9, with outputs upscaled by 1.8× using a latent space upscaler that involved 20 steps and a denoising strength of 0.6. The initial seed is 526430048.
When determining parameters, several fundamental decisions must be made, starting with the choice of solver. Based on our findings, DPM++ appears to be the fastest, yielding satisfactory results in just under 25 steps. It is widely recognised that difusion models have a significant limitation in terms of generation speed. This issue is expected to be addressed by Consistency Models in the near future [14]. These models promise a streamlined, few-step process for generating high-quality images and support a one-step generation without the requirement of adversarial training.
We advocate for a generate-then-upscale approach as opposed to generating images directly at a higher resolution. This strategy helps circumvent issues such as visual clutter, errors, and residual noise, thus enhancing the overall quality of the generated images.
The top-left image in Figure 16 shows the initial generated image from our model. As can be seen, the output image exhibits some flaws: the colours are unrealistic, the contrast is overly exaggerated, and the background appears neither like a tundra nor a desert but somewhere in between. However, as we will demonstrate in this section, our proposed model and training methodology are capable of generating more natural and compelling images than this initial attempt. 4.2.1. *B*LACK The mid-left image 7 in Figure 1 shows the results of providing our model with additional context about the background scenario. Specifying that the image should depict a desert landscape leads to notable improvements, including more realistic colours and clearer shadows. There are more details in the pyramid and in the sky. Overall, this example demonstrates how giving our model more contextual guidance can significantly enhance the quality and coherence of the generated images. 4.2.2. B*L*ACK The bottom-left image 8 in Figure 1 demonstrates continued improvements in the quality of the generated images as we provide more context and feedback to our model. The lighting and shadows are more realistic, and additional details have been added, including shining elements atop the pyramid and smaller surrounding objects. The sunset appears more realistic and casts 6Initial prompt: a oil painting of a alien pyramid. 7Prompt (with Background): a oil painting of a alien pyramid, ancient egypt background. 8Prompt (with Lighting): a oil painting of a alien pyramid, ancient egypt background, cinematic lighting, sunset. a ’bloom’ efect onto the dunes. Each iteration reveals how our model can produce increasingly compelling results when given more guidance about the desired style, content, and context of the images. 4.2.3. BL*A*CK By incorporating amenities tokens in the top-right image9 of Figure 1, we achieve further improvements in image quality. The bloom efect from the previous image is stronger, and the smaller surrounding pyramids are moved into the foreground, giving the primary pyramid subject greater prominence. The ground is detailed and more striking, and additional nuances are visible on the pyramid itself. Overall, the image appears smoother and with less noise than the previous iteration, demonstrating how continued refinements to our training methodology can steadily enhance the results. 4.2.4. BLA*C*K The mid-right image10 in Figure 1 demonstrates some unexpected changes that emerge from our model as it continues to be instructed. There are now clouds, while the gradient has become more prominent and some stars are visible. The shadows are longer and smoother, and the sky takes on an oil painting-like quality rather than photo-realism. 4.2.5. BLAC*K* The bottom-right image11 in Figure 1 shows the results of incorporating ’kinesis’ tokens into our model prompting. To demonstrate how our model can represent dynamic objects, we included a pirate ship token part, which produces some ships in a now present lake in the final image. With the addition of the kinesis and pirate ship tokens, our image-prompting process using the BLACK methodology is now complete. As shown, this process is capable of generating diverse, compelling images with unique content, styles, and implied motion.
This research serves as an exploratory foundation aimed at developing and assessing standards in prompt engineering for generative AI. Ensuring comparability and reproducibility in LDMs presents ongoing challenges. Notably, even minor modifications to prompts can yield varying outcomes. In this section, we address potential questions from readers. 9Prompt (with Amenities): a oil painting of a alien pyramid, ancient egypt background, cinematic lighting, sunset, perfect, Beeple, concept art, fantasy art, trending on ArtStation, trending on CGSociety, Intricate, High Detail, photorealistic painting art by midjourney and greg rutkowski, 8k. 10Prompt (with Context): a oil painting of a alien pyramid, ancient egypt background, cinematic lighting, sunset, perfect, Beeple, concept art, fantasy art, trending on ArtStation, trending on CGSociety, Intricate, High Detail, photorealistic painting art by midjourney and greg rutkowski, 8k, summer, 4k landscape, wallpaper. 11Prompt (with Kinesis): a oil painting of a alien pyramid, ancient egypt background, cinematic lighting, sunset, perfect, Beeple, concept art, fantasy art, trending on ArtStation, trending on CGSociety, Intricate, High Detail, photorealistic painting art by midjourney and greg rutkowski, 8k, summer, 4k landscape, wallpaper, (pirate ship in the background), (flying pirate ship), dynamic.
1. Does the order of prompts significantly influence the results? Preliminary evidence suggests it does, although the exact impact is not yet quantifiable. Tokens positioned at the beginning of a prompt tend to receive more attention [15]. However, the importance of individual words is often outweighed by their contextual relationships. 2. How does placing amenities before background in a prompt afect the outcome? This question is intrinsically linked to the previous one. Our experiments indicate that altering a sequence of tokens (rather than just one) can significantly impact the result. However, a precise qualitative evaluation remains elusive. Future studies will delve deeper into this aspect. 3. Is BLACK compatible with all image generators? Yes, provided they support prompting and are trained on internet data.
In conclusion, our study presents a straightforward and replicable token-based prompting methodology, known as BLACK, that caters to both novice and advanced users. We have developed a comprehensive example to elucidate the thought process underlying our approach. To assist readers in crafting prompts using the BLACK method, we have compiled a spreadsheet featuring over 1000 unique words, each assigned to a corresponding BLACK category such as subject or style tokens. Additionally, we have incorporated a random generator and sample prompts within the document to further facilitate the prompt creation process. All the work is already available on our repository12. 12https://github.com/federicotorrielli/BLACK language models: a survey, 2023. URL: https://arxiv.org/abs/2302.07842. doi:1 0 . 4 8 5 5 0 / A R X I V . 2 3 0 2 . 0 7 8 4 2 . [7] S. Witteveen, M. Andrews, Investigating prompt engineering in difusion models, ArXiv abs/2211.15462 (2022). [8] V. Liu, L. B. Chilton, Design guidelines for prompt engineering text-to-image generative models, Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (2021). [9] J. Oppenlaender, A taxonomy of prompt modifiers for text-to-image generation, 2022.
a r X i v : 2 2 0 4 . 1 3 9 8 8 . [10] S. Ross, Style in art, The Oxford handbook of aesthetics (2003) 228–244. [11] Negprompt, 2023. URL: https://github.com/AUTOMATIC1111/stable-diffusion-webui/ wiki/Negative-prompt. [12] Lykon, Dreamshaper (revision 51416b0), 2023. URL: https://huggingface.co/Lykon/
DreamShaper. doi:1 0 . 5 7 9 6 7 / h f / 0 2 9 7 . [13] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, J. Zhu, Dpm-solver++: Fast solver for guided sampling of difusion probabilistic models, ArXiv abs/2211.01095 (2022). [14] Y. Song, P. Dhariwal, M. Chen, I. Sutskever, Consistency models, ArXiv abs/2303.01469 (2023). [15] Deane, Does prompt order matter in stable difusion? - prompt phantom, 2023. URL: https://promptphantom.com/does-prompt-order-matter-in-stable-diffusion/.