This study evaluates the potential of AI image generation for visualising historical events, focusing on two ancient Roman scenarios: the Roman triumph and theLupercalia festival. Using DALL-E 3, we generated 600 images based on 100 prompts derived from scientific texts. We then conducted a twopart evaluation: (1) A human evaluation by 21 history students, who compared image pairs and rated individual images on accuracy and prompt alignment, and (2) two automated analyses, one modelled after the human evaluation protocol and one using visual question-answering (VQA) techniques. Our results reveal both the promise and limitations of AI in historical visualisation. While DALL-E 3 produced many convincing images, there were notable discrepancies between human and automated assessments. We found that Large Language Models tend to rate images more favourably than human evaluators. We contribute a novel dataset for historical image generation, initial human and automated evaluation protocols, and insights into the challenges of using AI for historical visualisation, which is incredibly important for historians to reconstruct past events. Our findings highlight the need for refined evaluation methods and underscore the complexity of assessing historical accuracy in AI-generated imagery. This study lays the groundwork for future research on improving AI models for historical visualisation and developing more robust evaluation frameworks.
Historians, akin to criminologists, analyse primary sources and eyewitness accounts to extract meaning and understand the motives and circumstances of historical events. However, unlike criminologists, who can re-enact events, historians face the challenge of studying occurrences that cannot be replicated or reproduced in experiments. This presents a significant challenge in their work.
Criminologists have developed methods to mitigate the uncertainties involved. Re-enacting crucial moments of an action or crime using real people or AI-based simulations has become one of the most potent tools in criminology. These re-enactments allow us to visualise the action, providing a cinematic perspective that clarifies a crime’s sequence and spatial dynamics. This process enhances our understanding by enabling us to perceive previously unseen details and prompting further questions. By experiencing events as if we were witnesses, we gain a newfound clarity.
Surprisingly, despite the similar challenges faced by both professions, historians have yet to embrace this re-enactment approach fully. Our project aims to change that. Leveraging rapid advancements in AI development, we aim to introduce an innovative platform to redefine how we perceive and comprehend historical events. Our goal is to develop a web application that generates storyboards or individual images from historical texts.
This ‘re-experiencing’ of history will empower users to recapture seemingly ‘lost’ historical moments. Users can model specific actions or occasions from diverse perspectives by employing various scenarios with AI support. This approach will uncover performative dynamics, potentially revealing previously undisclosed aspects of historical events, much like the re-enactment in criminology.
However, we must question the suitability of such image-generation models. They require testing for historical accuracy before we can employ them for the previously mentioned purposes. We chose two specific Roman scenarios to test the capacity of DALL-E 3 to create historically accurate and engaging images: the Roman procession and tLhuepercalia festival.
Our study focuses on these two events due to their significance in Roman culture and the varying levels of textual and visual documentation available for each. The Roman triumph, a well-documented celebration of military victory, provides a rich base of textual descriptions. In contrast, the Lupercalia, an ancient fertility festival, ofers a more challenging scenario with fewer detailed contemporary accounts.
To assess DALL-E 3’s capabilities in this domain, we generated 600 images – 450 for the triumph and 150 for theLupercalia (see Section 3). Our evaluation process is twofold: 1. Human evaluation: We conducted a comprehensive review involving 21 advanced history students to assess the images’ historical accuracy. 2. Automated analysis: We employed computer vision techniques to analyse the images for prompt alignment.
This dual approach allows us to measure the generated images’ subjective impact on human viewers and their objective alignment with historical data. Our research contributes to the broader discussion of AI’s potential in historical visualisation and its limitations and contains the following items: 1. A novel, automatically generated dataset comprising 100 prompts and 600 images for historical image generation. 2. An initial human evaluation of a subset of these automatically generated images. 3. An initial automatic evaluation of the same subset. 4. An assessment of how well human and automatic evaluation correlate.
The evaluation of automatically generated images has recently gained traction, mainly due
to the increasingly sophisticated image generation models. Otani, Togashi, Sawai, Ishigami,
Nakashima, Rahtu, Heikkilä, and Satoh 2[
The advantage of human feedback is that it can improve text-to-image models, e.g., with
reinforcement learning from human feedback (as used in Natural Language Processi2n8g])[. Xu,
Liu, Wu, Tong, Li, Ding, Tang, and Dong 3[
While Xu, Liu, Wu, Tong, Li, Ding, Tang, and Dong3[
The research mentioned above has had access to large and heterogeneous datasets and results from extensive evaluation campaigns. In the context of historical image generation, such work does not yet exist. One exception is the investigation of Fareed, Bou Nassif, and Nofa8l] [ who tested the usage of Leonardo1 for teaching purposes in the field of “History of Architecture”. They evaluated the usability of Leonardo with a questionnaire after a workshop, which generally showed a need for the evaluation of AI-generated images for usage in the historical domain.
Next, we outline the methodology for data collection using DALL-E 3 to generate images
related to triumphal processions and thLeupercalia, which included the following steps:
1. Collecting Historical Documents: We collected resources (i.e., academic papers,
books, and other relevant documents) about the triumph and thLeupercalia in ancient
1See https://leonardo.a.i
Rome. Specifically, we included five documents related to theLupercalia [
Note that we did not force the model to produce realistic images. This led to a great variety of image styles, some of which are indeed life-like, while others are more in the style of a Renaissance painting or a black-and-white pencil sketch. All prompts, however, are based on scientific literature. See Figure1 for example images and prompts from the datase4t.
The following sections focus on the diferent evaluation scenarios employing human annotators and automatic evaluation measures.
We generated two evaluation scenarios to obtain feedback from human annotators.
Image Comparison (IC) The first scenario asks annotators to decide which of two images
better reflects the prompt. This is a cognitively easier task. Much in the manner of Xu, Liu,
Wu, Tong, Li, Ding, Tang, and Dong [
Image Rating (IR) The second task requires the participants to rate an image on a 5-point Likert scale with the following options:
1. The image does not match the prompt at all. 2See https://openai.com/index/openai-ap.i 3The image generation costs amount to $48.06. 4The whole dataset (images and prompts) is available on GitHub. Sehettps://github.com/AncientHistory-UZH/C HR2024_prompt-and-image-datase.t
2. The image barely contains aspects of the prompt. 3. The image catches some aspects of the prompt, but it is not very accurate. 4. The image catches most of the aspects of the prompt.
5. The image completely matches the prompt.
Additionally, we asked the users to describe which aspects of the image dniodt correspond to the prompt in a text field. In this scenario, which demands more time and efort, we need 600 ratings for one complete dataset annotation.
We set up a Prodigy interface5, which we used to obtain the assessment of the annotators. See Figure2 to get an impression of the annotation environment. We recruited 21 advanced history students for the annotations. We did not ask the participants to annotate a specific number of pairs. They were compensated with book vouchers of a value of $30. An online meeting was organised to explain the guidelines, emphasising that in the first scenario, they should judge based on the alignment of the images with the prompts rather than their visual appeal. They should consider visual features only if the two images reflect the prompts equally. The students spent approximately one afternoon annotating the data in both scenarios. 5See https://prodi.gy.
Table 1 gives an overview of the results from the human evaluation. In the IC setting, we received 1,569 comparisons. 103 samples were annotated more than once. For unknown reasons, 64 data points did not contain the human assessment, so we excluded them from further analysis. On average, each participant compared 74.71 (SD 43.32) image pairs.
The IR scenario received less feedback since the participants provided written feedback in a text field besides their rating. We obtained 568 ratings, of which 29 were double ratings—24 feedbacks without scores needed to be excluded.
We must note here that, due to a wrong parameter setting of Prodigy in both scenarios, the data samples to be evaluated were presented to the participants in sequential instead of a random order. This led to only marginal annotation overlap. For this reason, we cannot compute inter-annotator agreements (IAA) yet. However, since we plan to improve the models with the feedback obtained from the participants, we will have further evaluation rounds during which we can take care of this limitation. Still, to the best of our knowledge, this is the first “largescale” evaluation campaign dedicated to historical image generation. We can still analyse and compare the results obtained with the limitations in mind (see Sectio4.n1.3).
However, since previous studies reported low IAA in human evaluation scenarios 1(c6f].)[, we hypothesise a similar outcome on our dataset.
To mitigate the missing information on IAA and to evaluate the suitability of multimodal LLMs
for scoring tasks, we employed GPT-4o2[
For IC, Table2 shows the agreements of the human comparisons with GPT-4o’s comparisons. We see that in 57.51% of the cases, human annotators and GPT-4o agree on which of the two images better corresponds to the prompt.
Figure 3 summarises the results for the IR setting. The left graph shows the diferences between the human and the LLM ratings. The tendency is that LLMs rate images higher than human annotators. The right graph shows the LLM’s deviations from the human scores. E.g., in 164 (30.15%) ratings, GPT-4o agrees with the human scores. In 169 (31.07%) cases, GPT4o scores one point higher on the Likert scale than the human annotators (i.e., GPT-4o had rated an image a 3 when the human annotator rated it at 2). We see that Claude tends to rate images higher, especially. Overall, the deviations seem normally distributed, a fact that might be exploited for future evaluations.
Choosing two scenarios to evaluate allows us to test for diferences in assessing images 6See https://www.anthropic.com/news/claude-3-5-sonne.t 7This generated costs of $19.14 for GPT-4o, $2.49 for Gemini and $5.27 for Claude. Inter-annotator agreement between diferent groups using Krippendorf’s alpha. We used the same 544 images for which we have computed the t-test..
GPT vs. Gemini vs. Claude
GPT vs. human
Triumph between the triumph and theLupercalia scenario. Our null hypothes is0 is that there is no diference in the ratings of human annotators and, e.g., GPT-4o in the two historical scenarios. variation and (ii) unequal sample sizes. For the human evaluation (unifying the assessment results but excluding invalid samples), thpe-value does not allow us to reject0. The GPT and Gemini ratings show another picture. Thpe-values show a highly significant diference between ratings of the triumph and theLupercalia images. Claude’sp-value is on the brink of showing a statistically significant diference. The, on average, lower ratings by LLMs of the Lupercalia images could indicate DALL-E’s difÏculties in generating adequate imagery. Firstly, since the Lupercalia are not so much a described nor illustrated phenomenon, it is reasonable that images portraying the festival are not on the same standard as those generated for the triumphal procession. Secondly, the automatic evaluation poses problems for LLMs because they do not “know” as much as they do for the triumph.
Although we cannot provide IAA scores for the human evaluation yet, we can do so for the automatically generated ratings by the LLMs. Tabl4e shows the results when we compare the ratings for the LLMs (again split into triumph- anLdupercalia-related scores). The scores are all around 0, indicating low overlap, IAA. Unifying all human scores and comparing them against the ratings obtained via GPT-4o also shows low overlap. These results hint at the very diferent rating “strategies” of the LLMs. We need further evaluation to shed more light on the origins of the discrepancies.
For a further fully automatic evaluation procedure, we employed the Question Generation and
Answering (QG/A) [
Question Generation (QG) In our study, we utilised GPT-3.55[] for QG employing the
Davidsonian Scene Graph (DSG) [
Visual Question Answering (VQA) We employed GPT-4o for the VQA task. The following prompt instruction guides the model: “You are a helpful assistant. Please answer the question only with ‘Yes’ or ‘No’. Do not give other outputs. Questio{nq:uestion}.” To ensure precise control over the output, specifically responding with either ‘Yes’ or ‘No’, we set the parameter logit_bias to 100 for both ‘Yes’ and ‘No’ tokens. Logit bias modifies the likelihood of speciifed tokens appearing in the model-generated output. We also set thetop_p (nucleus sampling) parameter to 0.1 to restrict the model’s consideration to a subset of tokens (the nucleus) whose cumulative probability mass reaches a designated threshold (top-p). In the context of a 0.1 top_p setting, the model exclusively considers tokens constituting the top 10% of the probability mass for the subsequent token. The combination olfogit_bias and top_p configurations enables the outputs to adhere to predefined patterns (‘Yes’ and ‘No’), rendering the model more deterministic and particularly suitable for our image evaluation8taWske. assign a score of 1 for ‘Yes’ and 0 for ‘No’ and then compute an average score for each image. We observe that GPT occasionally generates questions such as “Is there an image?” or “Can you visualize a scene?” which are invalid in our context, as the input consistently includes an image and a set of questions. We excluded the scores of these invalid questions from our analysis.
Figure4 shows a histogram of the results of the VQA scores for all 600 images. We find most scores between 0.5 and 0.9, with over 60 images obtaining a perfect score of 1. This means that each ‘Yes-or-No’ question was answered with ‘Yes’. When we look at three results as presented in Figure5 in Appendix A, we find that VQA attributes a low score of 0.05 for imagea). The human evaluator and GPT, however, have scored this image with a 4 in the IR scenario. bI)n, we have a medium VQA score of 0.61, a human score of 5 and a GPT score of 4. Lastlcy),shows an image with a VQA score of 1, but a human annotator scored this image a 3 and GPT a 4. We already see discrepancies between the diferent scores from these three examples only. A comparison of VQA between the 450 images from the triumphal procession and the 150 images 8This evaluation scenario cost us $7.69. The whole experiment, i.e., image generation, LLM evaluation in the two scenarios from Section4.1.3 and the one mentioned in this section totalled at $80.67. from theLupercalia based on Welch’s t-test shows no significant diferences between the two ratings ( = 0.88 ). From this, we conclude that ratings based on VQA produce more reliable results than those produced with a Likert scale.
The most significant limitation of our work is the missing IAA scores. For future evaluation rounds, we will set up the evaluation to allow for their computation. In this way, we get reliable measures of how demanding the task of assessing the alignment of historical images and the prompts they produced is. However, we argue that the results we obtained from the human evaluation are still valuable and allow for fine-tuning models based on human feedback (preferences in the IC and textual input in the IR scenario), albeit in a low-resource setting.
Moreover, we will employ more models to generate images in future experiments. This approach enables us to decide which models are the most suitable for historical image generation. The stable prompt base also allows for comparable results. Still, the significant number of images we will generate in future endeavours also calls for automatic evaluation methods.
In conclusion, our study provides valuable insights into the potential and challenges of using AI for historical image generation. The evaluation of 600 AI-generated images of triumphal processions and theLupercalia revealed both promising capabilities and significant limitations.
Our findings hint at the discrepancies between human and automated assessments, underscoring the complexity of evaluating historical accuracy in AI-generated imagery. Ultimately, this study serves as a stepping stone towards more sophisticated use of AI in historical recreation and education while cautioning against over-reliance on automated systems for historical interpretation.
This research contributes a novel dataset and evaluation framework to the field, enabling future studies. As AI continues to evolve, our work suggests that while it holds promise for enhancing historical visualisation and understanding, it requires careful human oversight and interpretation. [23] I. Östenberg. “Triumph and spectacle. Victory celebrations in the Late Republican civil wars”. In: The Roman Republican Triumph Beyond the Spectacle. 2014, pp. 181–193.