This article presents the ongoing work on developing a methodology for the systematic analysis and narrative of heritage image archives, focusing on the photo archive by Robert Gerstmann of the Sacambaya Expedition in 1928. This work combines state-of-the-art artificial intelligence techniques, such as advanced algorithms used in computer visión like convolutional neural networks, with Large Language Models (LLM) for generation purposes. The intent is to establish a practical and accessible framework in this area for institutions and individuals. The proposed method incorporates human-generated image labels with LLMs to produce narratives that aid researchers and users in their sense-making process while they explore a large archive of images. Through this iterative process, we aim to contribute not only to the understanding of this specific historical photo collection but also to the broader development of scalable solutions for the exploration and interpretation of heritage image archives. We seek to achieve a deeper understanding of the contents and meanings of the analyzed files, suggesting and highlighting new clustering of these materials and thematic/narrative connections that may not have been considered by a human observer.
The emergence of new technologies and the availability of a vast photographic archive have
motivated a multidisciplinary project that explores the potential of these two elements in
expanding the scope of multidisciplinary research. This ongoing research in the field of
computational narrative extraction aims to develop a methodology for the analysis and
semiautomatic construction of meaning and narratives from historical image archives using Large
Language Models (LLMs). On the heritage side of this research, we aim to uncover and specify
narratives inherent in large banks of photos for which there is limited information and
dissemination [
His work has been preserved in its original physical format since 1964 in Antofagasta, Chile.
This material consists of 43,475 negatives and 15,054 positives in different formats,
representing a period of photographic capture of approximately 40 years [
In January 1928, Edgar Sanders, a Swiss engineer, established a company in London to
search for an alleged Jesuit treasure hidden in an old monastery located in a ravine in the
province of Inquisiví, Bolivia. The expedition comprised 21 individuals with diverse
professional and military backgrounds, including 19 English citizens, 1 German, and 1 North
American. The team scoured four different locations in the sector for five months, but their
efforts proved fruitless, and they returned to Europe in November of that year [
This research seeks to exploit the power of LLMs [
The ROGER Narrative Pipeline unfolds in a structured, multi-phased approach,
incorporating human expertise and AI in a collaborative narrative pipeline. Initially, images
are labeled through a combination of AI-driven algorithms and human judgment, resulting in
a curated and contextually enriched dataset. Subsequent phases involve the use of AI to
generate descriptive narratives and cluster these into thematic groups. The proposed pipeline
integrates the use of prompt engineering loops [
This paper presents the results from the application of the ROGER Narrative Pipeline to the 1928 Sacambaya Expedition historical archive, uncovering the implicit narratives embedded within historical image archives. The proposed approach contributes to the interdisciplinary dialogue on narrative extraction, advancing our understanding of computational narrative construction in historical research. This represents a novel initiative to systematically decipher the stories enshrined in historical visual records. Finally, in the conclusion section, it is noted that this is the first work that has attempted to computationally unearth the stories contained in any of Robert Gerstmann's historical photo archives.
The computational analysis and interpretation of historical image archives is an emerging
interdisciplinary field that integrates computer vision [
Labelbox is a machine learning annotation platform that simplifies the creation and management of annotated datasets, which are vital for AI development. It supports a variety of data types and provides tools for both manual and semi-automated annotation, aimed at increasing efficiency and accuracy. The platform encourages teamwork with its collaborative features and maintains high-quality annotations with robust quality control measures. Its user-friendly interface and integrative capabilities with machine learning workflows make it accessible for users of different skill levels, streamlining the annotation process from start to finish. 2.2
LLMs are advanced AI systems designed to process and generate human-like text. They have
been trained on a massive amount of data and can understand and generate text in a wide
variety of languages and styles [
Sensemaking of archives requires synthesizing across individual images to construct a
higher-level understanding. Computational techniques for visual storytelling aim to build
such narratives from image sequences [
We present the ROGER Narrative Pipeline for the extraction of computational narratives from visual datasets. The general methodology is presented in Figure 1. This process commences with a systematic labeling phase where a set of input images is semantically annotated using a combination of software tools and human oversight, producing a curated dataset. This curation involves the enrichment of the images with contextual metadata in the form of image labels. These labels enhance the depth and relevance of the information that will be used in narrative construction.
Central to our pipeline is the integration of an LLM, which generates textual descriptions and titles from the enriched image data. These descriptions are the bedrock upon which the narrative structure is built. Subsequently, we use the LLM to perform clustering on the textual descriptions to organize the images into clusters or coherent thematic groups (Figure 3), followed by the establishment of a timeline, ordering these clusters and images to create a draft narrative sequence. Integral to this process are the prompt engineering loops, where human operators iteratively refine the AI prompts based on the outputs to produce a final narrative (Figure 4). This iterative process is pivotal, allowing for the human operator's critical and creative inputs to sculpt the narrative, ensuring structural and thematic integrity.
The final stages of the pipeline revolve around the transformation of the AI-generated timeline into a narrative draft. This draft undergoes a human-led finalization process, where narrative/theme experts refine the storyline, ensuring linguistic precision, narrative flow, and overall coherence. The result is a final narrative that provides a textual representation of the visual data in a narrative format. The final output contains the ordered list of images, their titles, their description, and the associated narrative. Furthermore, the output is accompanied by a detailed cluster list that provides an overview of the narrative elements and their organization, thereby offering transparency into the narrative structure and content. Through this methodical and collaborative approach, the pipeline achieves a high-fidelity narrative extraction from visual inputs, demonstrating the potential for robust human-AI collaboration in this domain. 4.
The main results associated with each stage of our pipeline, around a subset of 12 heritage images, are presented here. 4.1.
To demonstrate the capabilities of our methodology, we present the results on a
representative subset of 12 pictures from the photo archive of the Sacambaya Expedition in
1928 [
Prior to this study, this photographic archive was sustained by historical sources and
the account of at least one of its members published in 1934 [22]. The historical narrative
suggested by these sources coincides with what is seen in the heritage images found in the
archive. These materials consist of a distinctive first phase of photos made on board a ship,
another category of materials showing means of transportation and human displacement
tasks, to finally settle into a last general category of images that exhibit the efforts of an
excavation and the logistics that this semi-industrial human operation entails [
Journey by ship: Subgroup identified as “London to Arica” (LTA) with 69 images. Journey by land: Subgroup identified as “Arica to Sacambaya” (ATS) with 187 images Excavation sites: Subgroup identified as “Sacambaya” (SAC) with 244 images.
From each of these thematic groups we intentionally selected 4 images to exemplify the general progress of the expedition, the methodology employed, and suggest a temporal order in the narration of the journey. 4.2
The enrichment of the original 500 images through the application of annotation tagging (both classification and object) by humans was facilitated by the use of the LabelBox [23] platform. The use of automatic classification algorithms was considered; however, the decision was made to prioritize establishing a baseline developed by humans. The procedure is comprised of several distinct phases. 4.2.1
A corpus of 500 images was curated, encompassing approximately 2.1GB of data. These images were standardized to a width of 1920 pixels and subsequently compressed using the cjpeg software [24] to reduce their file size, culminating in a consolidated dataset of approximately 370MB.
a) b) The ontology for the annotation process was crafted following an initial analysis of the photographs. This ontology comprised two sets of annotations: General Classification and Specific Objects. The development of this ontology was a critical step in ensuring that the annotations would be comprehensive and consistently applied across the entire image set by the research assistants. The detailed categorization was designed to facilitate nuanced analysis and to support the study's objectives by providing rich, structured data.
The general classification elements consisted of 22 labels (such as trees, road, city, boat, excavation, beach, square, etc.) that were defined based on a prior visual analysis by the authors. The specific elements were grouped into categories: person, animal, object, transportation, and landscape. Each of these categories encompassed between 5 to 7 labels. This structured approach to classification allows for a detailed and organized analysis of the photographs. By having both general and specific categories, the researchers could ensure that the labeling process was thorough and nuanced, capturing both broad and fine-grained details within the images. 4.2.3
Ten undergraduate students from the fields of journalism and computer engineering were recruited to serve as research assistants. These individuals were selected based on the criterion of having completed at least 50% of their academic program and having prior participation in various research projects. Over the course of two weeks, these individuals were tasked with the systematic manual delineation of bounding boxes and the subsequent assignment of labels corresponding to the identified objects within the images. Each photograph was labeled by at least two students and subsequently reviewed by the researchers. The team identified a total of 4,868 objects, of which 44.1% were classified as people, 22.17% as landscape details, 15.14% as various objects, with the remaining categories including animals and transportation. The average number of annotations produced by each student was 54.4. 4.2.4
The annotations were subsequently extracted from LabelBox in JSON format, which facilitated their processing and analysis through the Python programming language. In a detailed analysis of the dataset, 12 images were selected as representative samples from the collection of 500, and their annotations were obtained for further investigation. This structured approach to annotation not only enhances the reliability of the data but also ensures a level of granularity that is conducive to subsequent computational analysis.
Following the manual labeling process, we generated narrative descriptions of each picture in the dataset. These descriptions are generated using ChatGPT with GPT4. In particular, we prompt the model with minimal context about the Sacambaya Expedition, upload the image, and provide the human-generated labels described previously. Table 1 illustrates the prompt design used to generate a narrative description and title for the picture shown in Figure 2 and the output generated by GPT4. We note that our annotators created the tags in Spanish as this was their native language, they were left untranslated in the prompt. We note that there was not a significant difference in translating them beforehand.
Cluster 1: “Maritime Prelude”.
Cluster 2: ““Expedition Life and Challenges”. Cluster 3: “Industrial and Excavation Efforts”.
The minimum context information supplied to the model was the name, year, and purpose of the expedition. The prompt included the names of some key places in the historical narrative (Bolivia and Sacambaya) and the name and nationality of the photographer (see Tables 1 to 3). Following the construction of all the narrative descriptions, we used another prompt to ask the LLM to cluster the images based on their content and reorganize them chronologically. The prompt used all the image descriptions generated beforehand to generate these clusters. The prompt is shown in Table 2.
Following this clustering process, we asked the LLM to generate a timeline of the photos followed by a draft of the final narrative. We note that our work required extensive prompt design to ensure that the generated descriptions, clusters, and final narratives were coherent. We show the final version of the timeline and narrative extraction prompt alongside the corresponding narrative summary output in Table 3. The final timeline is shown in Figure 4 with the corresponding images. These two elements (timeline and narrative summary) represent the final output of the proposed pipeline.
We note that our proposed prompt structure forces the model to hypothesize a plausible chronological order before generating the narrative draft. The data set of historical photos does not contain explicit temporal or spatial information not sharing with the model the thematic grouping mentioned in point 4.1. Therefore the LLM must be able to infer the order of the photos based on the general context of the expedition. While a human could also help with the timeline generation process and provide further information via prompting, the amount of photos in a full archive can make this generally unfeasible and too complex to scale. Thus, for scalability purposes, we rely on the capabilities of the LLMs to make sense of the underlying temporal and spatial information.
In general, sorting an unordered set of photos in the correct temporal order is a complex problem. In this context, GPT4 had issues when generating the correct timeline with only limited contextual information, which is why we had to include additional context in our prompt design. In reviewing the results, we found that the LLM model presented historical events in the correct chronological order, as long as it was provided with minimal contextual information.
You are an expert documentary filmmaker and you are exploring the historical archive of photos of the Sacambaya Expedition in Bolivia from 1928, taken by the German photographer Robert Gerstmann. The expedition sought to find the lost treasure of the jesuits, hidden somewhere near Sacambaya. You have the following list of descriptions of photos. Your goal is to generate two outputs: a timeline of these photos from the provided descriptions so that it forms a coherent narrative and a written summary of the narrative. The timeline should explicitly reference the IDs of the photos. The narrative summary should also reference the IDs of the photos as needed between brackets. Note that the provided descriptions might not be shown in the correct order and the IDs are not associated with any ordering. Please avoid the use of flowery language, your narrative should be written in an academic style.
Do not explain your process.
The Sacambaya Expedition of 1928, documented by German photographer Robert Gerstmann, embarked on a challenging quest to uncover the Jesuit's lost treasure in Bolivia's Sacambaya Valley. It began with maritime scenes (IDs 65 and 60), capturing the tranquil ocean and coastal topography as the British expedition approached its destination. Aboard the ship, the crew is depicted in a moment of camaraderie and preparation (ID 30), followed by a brief sojourn at a riverine port, indicating the integration of transportation modes (ID 23). Upon reaching the Sacambaya Valley, the expedition faced the valley's rugged terrain (ID 40), fording rivers and navigating hilly landscapes (ID 85), and traversing dense thickets (ID 91). The photographs depict the challenging nature of their journey, highlighting both the physical and logistical hurdles encountered. As the expedition progressed, the team's burden of transporting heavy machinery through the dry valley is evident (ID 71), reflecting the blend of hope and hardship. Moments of rest and mechanical endeavors are captured (ID 46), as well as industrious activities amidst natural ruins (ID 37), underscoring the expedition's reliance on technology of the time. The concluding images (IDs 33 and 81) focus on the excavation and manual labor involved in the treasure hunt, set against a backdrop of natural and historical landscapes.
This narrative, woven through Gerstmann's photographs, tells a story of determination, challenge, and the enduring human spirit in the pursuit of historical treasures.
This ongoing research represents an advancement in the field of narrative extraction from heritage image archives. Through the iterative and incremental development of our methodology, a framework is established that not only enhances the understanding of historical events but also contributes to the broader discourse on computational narratives and artificial intelligence [25].
This experimental model facilitates a utilitarian approach to solving issues prevalent in numerous audiovisual archives. The methodology proposed herein aims to achieve quality control measures concurrently with narration. While beyond this work's scope, future initiatives will prioritize expanding the search range and implementing an integral quality control system to regulate labeling accuracy, clustering thoroughness, and narrative coherence levels. Additionally, we acknowledge the potential of Large Language Models (LLMs) to autonomously identify a broad spectrum of objects. Consequently, subsequent projects may explore creating narrations without human support, relying solely on object identification and computer vision capabilities. However, if pursuing this avenue, methodological and future discussion elements must be incorporated to ensure a minimum level of confidence in the results, rendering them relevant for anthropological, historical, and heritage discourse. Limitations of this case study and the need for a more comprehensive evaluation are identified, as its primary objective was to illustrate the methodology rather than validate it with an extensive data spectrum.
The successes of the proposed method in constructing coherent historical narratives suggest a potential paradigm shift in how narrative extraction from visual historical records can be approached. Thus, this ongoing research represents a significant contribution to the challenge of uncovering narratives concealed within historical image archives. Furthermore, our aim is to observe significant changes, trends, and prevalent elements in large groups of visual information that may not be readily apparent through individual observations [26]. This broader perspective facilitates the finding and construction of narratives that extend beyond individual images.
These experiments using easily available LLMs demonstrate the need to always maintain human control in the process, as shown by all the required prompt engineering. Future work will consist of applying the proposed pipeline to the entire collection of 500 images. We hope that our proposed methodology and technical pipeline streamline the work of expert catalogers, documentarians, and media creators, who can now have a minimal foundational basis to explore large, undisclosed photographic collections. Additionally, in the present case of the archive of the Sacambaya Expedition by Robert Gerstmann, we hope to have contributed to the historical and heritage enrichment of a part of this little explored collection.
In conclusion, we propose that the analysis of a specific photographic collection can
be further enriched through the organization and utilization of information in narratives [
The authors wish to acknowledge the contribution of the UCN Faculty of Humanities with its grant “Concurso de Incentivo Productividad Científica”, 2023 initiative, which contributed financially to the project, and the UCN Library for allowing access to and work on Robert Gerstmann's photo archive. The authors also wish to thank the team of research assistants, made up of students from the School of Journalism and the Department of Computing and Systems Engineering, who carried out the task of manual classification and annotation of the more than 500 photos of the group under study. [21] D. Buck, 2000 Tales of Glitter or Dust, accessed December 2023. URL: https://www.thefreelibrary.com/Tales of Glitter or Dust.-a073064246. [22] S. Jolly, "The Treasure Trail". John Long Limited, London 1934. [23] Labelbox, "Labelbox," 2024. URL: https://labelbox.com. [24] Wallace, Gregory K. "The JPEG Still Picture Compression Standard", Communications of the ACM, April 1991 (vol. 34, no. 4), pp. 30-44. [25] Keith Norambuena, Brian Felipe, Tanushree Mitra, and Chris North. "A survey on event-based news narrative extraction." ACM Computing Surveys 55, no. 14s (2023): 1-39. [26] Klingenstein, Sara & Hitchcock, Tim & DeDeo, Simon. (2014). The civilizing process in London's Old Bailey. Proceedings of the National Academy of Sciences of the United States of America. 111. 10.1073/pnas.1405984111. [27] Battad, Zev & Si, Mei. (2022). A System for Image Understanding using Sensemaking and Narrative. The Ninth Advances in Cognitive Systems (ACS) Conference 2021.