Humans are critical for the creation and maintenance of high-quality Knowledge Graphs (KGs). However, creating and maintaining large KGs only with humans does not scale, especially for contributions based on multimedia (e.g. images) that are hard to find and reuse on the Web and expensive to generate by humans from scratch. Therefore, we leverage generative AI for the task of creating images for Wikidata items that do not have them. Our approach uses knowledge contained in Wikidata triples of items describing ifctional characters and uses the fine-tuned T5 model based on the WDV dataset to generate natural text descriptions of items about fictional characters with missing images. We use those natural text descriptions as prompts for a transformer-based text-to-image model, Stable Difusion v2.1, to generate plausible candidate images for Wikidata image completion. We design and implement quantitative and qualitative approaches to evaluate the plausibility of our methods, which include conducting a survey to assess the quality of the generated images.
CEUR ceur-ws.org
Large knowledge bases (KBs) such as Wikidata are maintained by human editors in a
collaborative manner in order to provide structured data of high quality [
In this work, we examine the problem of missing images for a specific class of Wikidata entities:
ifctional characters . We motivate this choice by the fact that querying Wikidata shows that
only 7% out of the 83.7K instances of the fictional character class, including its sub-classes, have
an image 1. It is important to note that this class was specifically chosen due to ethical and
privacy concerns, as other classes (e.g. person) can have detrimental consequences if automatic
images are created to portray them. Although alternative methods for finding images for this
class of entities exist (e.g. fan-created images), they are unreliable in terms of the objective
representation of characters and demand thorough and manual research by editors to make
sure that they correctly align with Wikidata’s knowledge about each entity.
We propose a novel method of leveraging knowledge from Wikidata triples about each fictional
character entity in order to create a representative image for it using generative artificial
intelligence (AI) models. This is done by (1) extracting triples from Wikidata entities, (2) creating
English prompts to be fed into a generative text-to-image model such as Stable Difusion [
Our work addresses the following research questions (RQs): • RQ1: To what extent can diferent types of prompts based on triples be used in text-toimage models to produce high-quality images? • RQ2: To what extent can the output of generative AI be used for Wikidata image completion? • RQ3: How can generative text-to-image models be evaluated? To the best of our knowledge, no previous study has explored the realm of using Wikidata as a source for creating prompts for generative text-to-image models. Our work 2 ofers the following contributions: • A framework that generates prompts for a text-to-image model (Stable Difusion v2.1 3) with diferent levels of structure and natural language text based on Wikidata triples. • A dataset of generated images for fictional characters extracted from Wikidata that can potentially be used by editors for image completion. • An evaluation strategy showing evidence of relevancy and adequacy of using AI-generated images for our use case. 1This query was performed in June 2023.
2The project and dataset are available at https://github.com/helemanc/gryffindor and at https://huggingface.co/ gryffindor-ISWS, respectively.
3The model card is at https://huggingface.co/stabilityai/stable-diffusion-2-1
Generative AI: Current groundbreaking advances in AI enable machines to generate novel
and original content based on textual prompts. Such generative applications include
text-totext [
Prompt Engineering: Because of the aforementioned advances, a novel area of prompt
engineering has emerged, in which humans interact with AI in an iterative process to produce
the best prompt (i.e. textual input) for a specific desired output [
We conduct our study on instances of the class designated as fictional character with Q95074
item ID. The initial stage of our approach involves the extraction of relevant triples pertaining
to a specific entity. Subsequently, these triples are used to form various types of prompts in
English, functioning as inputs to Stable Difusion [
Generating an image for a specific character requires a description that can be gathered from its related triples. In Wikidata, these can be obtained through SPARQL queries 5, yielding all triples with the character as the subject. Moreover, since properties and entities might be represented 4https://sweet-hall-e72.notion.site/A-Traveler-s-Guide-to-the-Latent-Space-85efba7e5e6a40e5bd3cae980f30235f 5All the SPARQL queries with detailed explanations are available at https://github.com/helemanc/gryffindor/ blob/main/src/data-collection/wiki_query_service.py. by item IDs and property IDs, we also extract and translate each triple from these IDs to their corresponding labels when the triple has an entity and property ID.
In order to investigate if prompts closer to natural language work better at generating better images, we generate four distinct prompts for each character. The first three are created utilising the set of triples that have been extracted for the respective entity from Wikidata, while the last prompt utilises English DBpedia abstracts. These prompts are defined as follows (see Table 15 for prompt samples of an entity): 1. Basic Label: This prompt merely employs the “label” that Wikidata assigns to its entities. 2. Plain Triples: This prompt is derived by concatenating the subject, predicate, and object of a triple to form a single sentence, utilising all available triples linked to a specific entity.
Notably, sentences generated from plain triples may lack proper structure and grammar.
3. Verbalised Triples: Triple verbalisation is defined as the transformation of structured
data (i.e., triples) into human-readable formats (i.e., text). These serve as a summarised
paragraph of all input triples.
4. DBpedia Abstracts: We use DBpedia abstracts as prompts obtained by querying the
English chapter of DBpedia [
When examining the triples for a single entity, we observe that triples sharing the same predicate tend to contain redundant information. As a result, prompts generated directly from these plain or verbalised triples will repetitively state the same facts. However, the “instance of” predicate seems to provide distinct information for each triple. To avoid duplicating facts in prompt types (2) and (3), we remove duplicate predicates, except for “instance of”, for the input triples. Among the remaining triples that share the same predicate, we keep only the one with the longest object, since longer objects likely contain more detailed information than shorter objects with the same predicate.
To ensure reproducibility in image generation, we utilise Stable Difusion version 2.1 6, an
open-source text-to-image model developed by Stability AI limited to the English language.
We chose version 2.1 because it supports all input shapes up to 1024x1024 and has a better
performance according to benchmark evaluation results [
It is important to note that this particular model has inherent limitations when it comes to generating images related to the human body. To address this issue and enhance its image generation capabilities, we employ the implementation of negative prompts that have been suggested and shared on a public GitHub repository 7. By incorporating these negative prompts, we aim to mitigate malformations in images (e.g. crossed eyes, more than five fingers, etc.). Moreover, since Stable Difusion has a limitation on the number of tokens allowed in the prompt sentence(s), we embed the prompt by utilising the encoder and tokenizer from Stable Difusion, courtesy of the Compel library 8. The model runs positive prompts of 1500 fictional characters without existing images on Wikidata and 1500 with images on Wikidata, the latter to be used for building a ground-truth dataset for evaluation.
We construct an extensive dataset 9 comprising 1500 fictional characters with images, as well as 1500 fictional characters without images, which are randomly chosen from the entire set of fictional characters on Wikidata. Our motivation for collecting data on fictional characters rather than real people lies in our commitment to upholding ethical standards and safeguarding privacy. Also, there is no available dataset about fictional characters, and our data collection source codes 10 can be easily applied to diferent domains by changing parameter settings. In addition, we extend our data by fetching the Wikipedia abstracts of each fictional character from the English chapter of DBpedia. Although a majority of these fictional characters lack information in DBpedia because it is constructed using English Wikipedia, this is not a problem in our case since the Stable Difusion model can only use English text as input. Since most of the fictional characters on Wikidata (ca. 78% 11) do not have any English Wikipedia page, we only managed to gather DBpedia abstracts for 925 fictional characters with images on Wikidata and 341 fictional characters without images (see Table 1).
By analysing basic statistics from Table 1, we directly notice a big descriptive gap in terms of triples, the number of extracted unique relations, and the length of the prompts between the two datasets we constructed. Moreover, we notice that the length of the prompt is usually the shortest for verbalised triples and the longest for plain triples. After gathering the data about the fictional characters from Wikidata and DBpedia, four diferent prompts are automatically 6Stable Difusion v2.1 model card: https://huggingface.co/stabilityai/stable-diffusion-2-1 7The negative prompts: https://github.com/mikhail-bot/stable-diffusion-negative-prompts 8Compel encodes and decodes the portion of the prompt and available at https://github.com/damian0815/compel 9The entire dataset is available at https://huggingface.co/gryffindor-ISWS 10The data collection codes are available at https://github.com/helemanc/gryffindor/blob/main/src/ data-collection/wiki_query_service.py
11The percentage is computed on Sept. 6, 2023 constructed by using the approach described in section 3. One example is shown in Figure 2, which depicts the ground-truth image for the character Harlequin with its four generated images. Based on this example, it is instantly clear that some of the prompts can produce images more similar to the ground truth.
In order to understand whether the generated images can plausibly be used for representing ifctional characters based on their Wikidata triples, we employ two evaluation strategies. The ifrst is an automatic evaluation of image similarity using diferent metrics, while the second is a human evaluation survey. Since the task of identifying whether two images portray the same character is subjective and dificult, we consider both qualitative and quantitative evaluation approaches. This helps us better understand the efect of the prompt type on the quality of the diferent generated images. In this section, we first describe the evaluation framework used, explaining the diferent metrics we took into account. Then, we present the obtained results.
We utilise automated evaluation methods based on three image comparison metrics:
• UQI [
Since the computation of the FID metric is more time-consuming than the other two, we compute it only on a small subset of our dataset consisting of images generated for ten random characters. On the other hand, UQI and CLIPscores are computed on the entire dataset. Additionally, we employ statistical methods for evaluating if the metrics above can measure the impact of the prompt on the quality of the generated images. For this purpose, we perform ANOVA to measure the efect of the prompt on the metric. We also perform Tukey’s HSD (honestly significant diference) tests on the metrics to reflect the prompts’ efect on the generated images. These statistical methods were computed on two subsets of our dataset: characters that have DBpedia abstracts and characters that do not.
Finally, we performed several Student tests to evaluate if a given property (e.g., the gender and occupation of a character) could lead to better results, and we separately made the test only on the values of the instance of property (P31). To carry out these tests we extract the types and properties used more than 100 times. For each property, we build two subsets. The first one includes evaluation metric results of the characters that contain the property, and the second is built by randomly choosing characters that do not have the evaluated property.
Although the above-mentioned evaluation metrics can provide automatic measures to compare
images, they are still unreliable in comparing whether the generated images successfully portray
the same characters as the ground-truth images. This is because noise such as the image style
or its color can afect the results of the automatic metrics. Therefore, we conduct a human
evaluation study in which we ask participants to rate how likely it is that a pair of images
(consisting of 1. the ground-truth image and 2. the generated image) portray the same character.
Additionally, we ask participants to list the criteria they think about when comparing two
images. The latter was done to get an idea of important features to look for when generating
images of fictional characters. For evaluating the agreement of the participants we compute
Krippendorf’s Alpha [
The results we obtained from the automatic evaluation metrics show diferent outcomes. In terms of UQI, all four prompt types yield a similar average similarity score of ca. 0.5, concluding that this metric is not optimal for our purposes. FID results (Table 5) show that images created from DBpedia are more similar to the ground-truth. When it comes to CLIPscores, we see a hierarchy of prompt types in terms of the obtained average similarity scores, with images generated by basic labels being the least similar (with a score of 0.48), followed by plain triples (with a score of 0.55), verbalised triples (with a score of 0.56), and lastly DBpedia abstract prompt seem to generate the most similar images to the ground-truth with a CLIPscore of 0.6. Results for UQI ad CLIP are shown in Table 4. It is important to note that contrary to UQI and CLIPscores, the FID metric is performed only on a subset of images generated for ten fictional characters, which makes it hard to make any concrete conclusions about this method. The ANOVA conducted on the UQI and the CLIPscores is shown in Table 6 and Table 7. The results show that the UQI is not able to underline a significant diference between the prompt type as a main fixed efect on the quality of the generated image. In contrast, CLIPscores are able to reflect this efect with high confidence.
The results of Tukey’s HSD test are shown in Tables 8 and 9. They highlight that the basic prompt is generally the worst prompt strategy and that the DBpedia abstract prompt is always the best one. However, for characters that do not have a DBpedia abstract, the verbalised triples prompt is better than the basic label and the plain triples prompts (with a p-value of 0.05810). Additionally, in order to understand if the number of relations and unique relation type attached to a given entity in extracted triples has an efect on the generated image quality, we compute the correlation between these variables with the CLIPscores. Results indicate that there is no such correlation (see Table 10).
Finally, we present the results of the Student tests related to the efect of the values of the instance of properties attached to an entity on the quality of generated images in two parts. The first displays the efect of values of the instance of properties on the generated images in Table 11 for plain triples prompts, and Table 13 for verbalised triples prompts. We can see that characters that already have a widely known visual representation (e.g. characters from comics, cartoons, or movies) generally have low CLIPscores. On the other hand, characters that do not have a visual representation (e.g. from written works such as novels) are usually more similar to the ground-truth images. The second part of the Student tests deals with the efect of properties on the quality of the generated images. Results are shown in Table 12 and Table 14. These results show that the majority of relations impact CLIPscores negatively.
To measure how humans evaluate the similarity of generated and ground-truth images, we ran an evaluation survey in which each participant is presented with images of ten diferent fictional characters chosen randomly from our dataset (shown in Figure 5). For each character, four pairs of images were displayed. Each pair consisted of the ground-truth image and a generated image. Participants were asked to rate how likely it is that both images portray the same character on a scale of 1-5, 1 being very unlikely and 5 being very likely. Figure 3 shows the distribution of participant replies for all ten characters based on prompt types. We immediately notice that images generated based on the three prompt types of basic labels, plain triples, and verbalised triples are more likely to be evaluated as not similar to the ground-truth image (i.e. the most frequent response for all three prompt types is one). On the other hand, images generated with DBpedia abstract prompts are most frequently rated as 3 and 4, both having the same number of responses. When examining the high numbers on the scale that indicate a high similarity between the ground truth and the generated images (i.e. 4 and 5), we notice a specific trend. The least frequent prompt type for those numbers is the basic label, followed by plain triples, verbalised triples, and DBpedia abstracts. Additionally, we analyse participant responses to the open question of which criteria they consider when giving their responses. Figure 4 presents the top ten criteria mentioned by participants. The analysis was done by extracting nouns and adjectives, and filtering out stop words and generic terms such as ‘character’. We also manually grouped synonymous concepts such as clothes, clothing, and outfit .
In total, our survey had 101 participants ranging between the ages of 17-59 with an average age of 30. About 57% of participants were male, 41% female, and 2% non-binary. 48% of the participants had a master’s education level. We did not target any specific group since we wanted to receive general responses regarding the similarity of images. Thus, we distributed the survey among friends and colleagues both from within and outside the research community. The cultural backgrounds of participants ranged from South and North America (ca. 8%) to
Europe (ca. 63%), East Asia (ca. 10%), and the Middle East (ca. 20%). That being said, ca. 25% of participants were Italian. We received 37 responses for the open question of listing relevant criteria.
Finally, we measured the agreement of participants using Krippendorf’s alpha. The global score is equal to 0.17, meaning that no concrete agreement was found. The same conclusion could be drawn at the level of the images presented to the participants and at the level of the prompt types used for the image generation (cf. Table 2 and Table 3).
As a last step in our evaluation, we want to assess if there is an alignment between the score of the automatic metrics (UQI, CLIPscore, and FID) and the human evaluation. In order to be able to normalise the participant evaluations, we standardise the score given by each participant. For ∈ {1, .., 101} , the unique ID of a given participant, and ∈ {0, 39} , a given generated image, the standardised score is computed as follows:
, = , − The alignment between automatic metrics and human evaluation scores is shown in figure 6. We see that CLIPscores seem to be most correlated to the human scores with a Pearson correlation of 0.5 for the plain triples prompt, 0.6 for the verbalised triples prompt, and 0.7 for the basic label prompt. Concerning the DBpedia abstract prompt, none of the metrics seem to be correlated with the human evaluation. UQI and FID are not correlated to human evaluation, results, both having scores close to zero.
Results of most automatic evaluation approaches we used (CLIPscores, ANOVA, and Tukey’s HSD) as well as the human evaluation results suggest a clear trend: images generated using DBpedia abstracts as prompts were rated as most similar to the ground truth images, followed respectively by verbalised triples prompts, plain triples prompts, and basic label prompts. This implies that DBpedia abstracts, which are written by human editors and contain more natural, diverse, and fluent text, enable text-to-image generators to produce better results. The fact that verbalised triple prompts produce the second-best results further emphasises the importance of lfuent text on the quality of the generated image. These results directly answer RQ1. When further analysing the obtained CLIPscores, we observe that the maximum CLIPscore occurred for an image generated by using a basic label prompt, possibly indicating that the text-to-image model had “seen” this character during training. This enabled it to create a similar image to the ground-truth one without adding any additional context. However, the lowest CLIPscore also occurred when using the prompt type of basic label, further emphasising that for some characters, generating an image based only on their label is not enough. We conclude that in order to automatically generate images for fictional characters that correctly portray them, using natural text descriptions is the best option. When this text is available (e.g. in DBpedia abstracts), it is best to use it, however, as we have observed when creating our dataset, many entities of fictional characters (See Table 1) do not have a DBpedia abstract. To create images for those characters, the best method seems to be extracting knowledge about them in the form of triples, verbalising those triples using a large language model, and giving the verbalised text as input to a text-to-image generative model. In this case, the content of the triples is crucial for generating high-quality images. However, the quality of images is not related to the number of triples or the number of unique relations contained in the triples. But is highly dependent on object values, highlighting the impact of the value of instance of property of fictional characters on the quality of generated images. Answering RQ2, generated images can then be leveraged for completing missing images in Wikidata entities.
Finally, addressing RQ3, when comparing the three automatic evaluation metrics we see that only the CLIPscores align with the human evaluation scores. This is because, unlike the human and CLIP evaluations which assess semantic similarity, the UQI and FID metrics only focus on image quality. This limitation in evaluating semantic content likely explains the discrepancy in results between the three automatic evaluation metrics.
Our work is limited in many aspects. First, we are currently dealing only with English data due to the limitations of the verbalisation model and the Stable Difusion model we used. Future work will consider dealing with multilingual datasets as well.
Additionally, when designing prompts based on Wikidata triples, we had to make decisions such as extracting triples based on subjects without considering objects. We also treated all triples equally with no emphasis on properties or types of entities. As shown by the open question in our human evaluation survey, it is evident that some properties are more important than others when generating images to portray a specific character. Future work can potentially explore in more depth which properties lead to better representations of characters. Further, when encountered with triples that have the same predicate, we selected the one with the longest object assuming it would contain more information. We are aware that this decision might have removed important information for characters, and this can be addressed in future work by concatenating object strings or summarising them automatically.
Our usage of the Stable Difusion generative model means that our method is inheriting its biases as well. Although directly leveraging information about each character from its triples is supposed to limit biases when generating images, this cannot always be controlled (e.g., for some female entities, the model generated images of male characters). Additionally, using a predefined set of negative prompts for all characters (which includes terms such as mutilated and disfigured ) is a considerable limitation of the model to correctly portray characters. A possible solution for this could be to design specific negative prompts for each individual character in a semi-automatic manner or to use another type of text-to-image model that does not require negative prompting.
Our work is also limited in terms of the ground-truth dataset constructed based on Wikidata entities that already have images. This is because, for some of these entities, the images are not reliably portraying the character, but the actor depicting the character.
Finally, in order to mitigate any copyright and/or privacy risks, we stress that our method is not suggested to be directly deployed into Wikidata, as we think that using AI-generated images can potentially be very harmful. Should this method be used for image completion, we encourage clearly watermarking images as AI-generated.
In this paper, we investigate four diferent methods for generating prompts based on extracting
knowledge in the form of triples. We then generate images based on each prompt using
Stable Difusion, a generative text-to-image model. We evaluate the diferent prompt types by
automatic as well as human evaluation approaches and conclude that the best-generated images
are based on natural language text that includes the context and background of the character.
When possible, this text can be extracted from a human-edited source such as DBpedia abstracts,
however, most characters do not have a DBpedia entity. This brings to light the need to verbalise
triples (i.e. transform them into natural text based on large language models) and use them
as prompts in order to receive the best visual representation of their corresponding fictional
characters. To the best of our knowledge, our work is novel in terms of utilising triples for
prompt engineering in order to complete missing information on Wikidata. Possible future
work includes finetuning the last Stable Difusion model via a Lora adaptation [
This project is the result of a research task force team at the International Semantic Web Summer School (ISWS) 2023. We would like to thank the organisers and tutors. We especially thank our mentor Albert Meroño-Peñuela for his valuable advice and input throughout this work.
We made use of the central High Performance Computer system at Freie Universität Berlin to conduct the data collection and image generation parts, and we would like to express our gratitude for the resources provided.
The work of the author, Sefika Efeoglu, is funded by the German Federal Ministry of Education and Research (BMBF) and the state of Berlin under the Excellence Strategy of the Federal Government and the Länder over the project.
This paper has been developed within the HE project MuseIT, which has been cofounded by the European Union under the Grant Agreement No 101061441. Views and opinions expressed are, however, those of the authors and do not necessarily reflect those of the European Union or European Research Executive Agency.
This work has been supported by the French government, through the 3IA Côte d’Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002.
dif lower upper q-value p-value basic prompt plain prompt 0.05848 basic prompt verbalised prompt 0.06724 basic prompt dbpedia abstract prompt 0.10023 plain prompt verbalised prompt 0.00875 plain prompt dbpedia abstract prompt 0.04174 verbalised prompt dbpedia abstract prompt 0.03298
dif lower upper q-value p-value
Figure 5: The 10 random generated images used for the Human Evaluation Figure 6: Correlation plots of the automatic and the human evaluation