Significant recent advances in AI are progressively giving Digital Humanities a range of powerful tools to analyse and contextualise artworks using techniques from computer vision, pattern recognition, ontology engineering, natural language processing, and the semantic web. These tools help to analyse artworks and link them to insightful descriptions. However, to obtain the full potential of these tools we need to tackle two issues: (i) how to integrate the fragmented and sometimes contradictory information these various tools provide, and (ii) how to make it much easier for art historians, curators, and artists to use and extend these tools.
There has been remarkable progress the past decade in tools to analyse and contextualise artworks based on techniques from computer vision, pattern recognition, ontology engineering, natural language processing, and the semantic web.1 What is missing however are good ways CEUR Workshop Proceedings to combine the outcomes of these various tools into one coherent interpretation and to make the results available both to art historians and to viewers.
The human interpretation process strives for narrative-based understanding. When interpreting an artwork the viewer tries to construct a narrative that integrates the visual depictions, memory of past experiences, knowledge of artistic styles, the personal history and prior work of the artist, and general world knowledge, in order to answer a series of questions (e.g. Who is depicted? Why?) and to put an artwork in context (e.g. Who is the painter? What was the function of the work?).
Narrative-based understanding has to be conceived of as a spiraling process.[
When considering partial digital support for interpretation we need not work not only on the individual knowledge sources that can answer or raise questions but also on the integration process and this requires as a first, fundamental step a data structure that supports integration.
For this purpose, we present a data structure called Integrative Narrative Network (INN). An INN acts as a kind of blackboard on which diferent knowledge sources write partial descriptions and through which they can consult the information provided by other knowledge sources to advance understanding. We operationalise the understanding process in terms of narrative questions and answers, defining questions as open slots which can be filled by knowledge sources by (i) evoking new questions, (ii) introducing answers to questions, (iii) constraining the answers to questions, or (iv) shrinking the set of questions by realizing that the answers to two diferent questions are in fact the same.
We have already applied the methodology and techniques described in this paper to other
domains (e.g. history [
In this paper, we explore historical artworks from the late Renaissance period by the Venetian
painter Lorenzo Lotto and we focus on integrating results from semantic web resources. More
concretely we report on a use case for the painting “Venus and Cupid” by Lorenzo Lotto3 (see
Fig. 1, left). This painting was chosen as a feasible case study for art understanding since the lack
of textual documentation and its complex iconography constitute an interpretative challenge
for art historians [
At first glance, the observer’s attention is driven by some visual indicators, such as the brightness contrast, to the main characters, i.e. a putto (little angel) and a woman. If the observer is familiar with Italian Renaissance themes, s/he will recognise immediately these ifgures respectively as Cupid and Venus. Nevertheless, the curious actions performed and the presence of uncommon objects may lead the observer to raise new questions about their meaning. Supported by a possible knowledge of Lotto’s artistic practice of depicting symbols in his artworks, the observer, particularly if s/he is an art historian, will search for possible meanings of the objects.
Narrative closure is reached when answers to all these questions are coherently integrated with each other and grounded in external and past experiences. For example, the art historian Christiansen recognises that the characters’ actions and the majority of the objects belong to the sphere of marital love, concluding that the artwork expresses a wedding wish4.
The focus of the interpreter has gone beyond the main focal point of the painting (i.e. Venus and Cupid), to explore secondary details, which are now seen as signs, triggering a deeper interpretation of the scene.
As explained, we frame art interpretation as a process whereby the agent raises questions, finds answers to questions, and interlinks questions and answers. And we focus in particular on how semantic web resources can help to raise questions and find answers to them.
What are the questions? Questions are computationally operationalised as variables. Following AI tradition, the name of the question is written as a symbol with a question mark in front, as in ?question-name (e.g. ?Where).
What are the answers? The answers to a question are entities in the domain of discourse. Entities are objects, events or (reified) concepts. They either refer to real-world observational 4A more thorough artwork’s interpretation description is available on catalogue entry of the MET Museum. data (for example a physical painting, a region in an image), to virtual entities (which may or may not exist in reality), or to entities in a knowledge graph in which case we use the URI (Universal Resource Identifier) as a unique identifier.
INN makes the relation between a question and an answer computationally operational in terms of binding between questions (which are technically variables) and identifiers of entities or constants.
Where do questions come from? We use a frame-based approach which was pioneered
in research on knowledge representation and object-oriented programming, starting in the
mid-seventies with the proposal by [
Visualisation The INN represents all questions that have been posed during the understanding process and all entities that appear as answers to these questions. The questions are visualized with green or red diamonds. The diamond is green if the question could be answered and red if it is still open. The answers to questions are represented with squares and can be frame-instances or constants (e.g. a number or a boolean value).5
This section illustrates the INN and its use in interpretation for a concrete case study on the painting Venus and Cupid by Lorenzo Lotto introduced earlier in section1.2. The case study has been implemented in the sense that all the access to diverse knowledge sources is done through queries and then integrated into the INN.
We formalised the set of narrative questions raised during art interpretation with a top-down
approach from the didactic art literature6 [
6University of Manchester, Art historical research. Url: https://library.wcupa.edu/_art_history 7The SPARQL queries performed are available at https://inn-painting-art-interpretation.streamlit.app/Narrative_ Questions_and_Queries an answer bound to the question ?Who in the INN, and the colour of the variable is changed to green. The criteria for selecting KBs is the possible presence of data about artworks, their content, and/or possible meanings. Furthermore, results obtained from computer vision analysis were integrated into the Narrative Network through a test on Google API Cloud Vision8.
More concretely, we searched for the painting ID on the KBs through SPARQL queries looking for the artist’s name and title. We then performed the queries expressing the 5W questions on each KB. During the process, results were represented according to the INN data structure, showing a graph which progressively expanded from the initial artwork node. We visually distinguished the sources of the answers using diferent shades of yellow colour. Nodes retrieved from diferent KBs which were possibly referring to the same concept (e.g. the representation of the character ”Venus” in Wikidata and in Zeri&Lode) were detected through a label similarity fuzzy ratio, and through Wordnet synonyms, and through the presence of common URI to which they were already aligned.
In the initial phase, we retrieved the artwork ID, which was described in 3 of them (Wikidata, the Iconology Dataset, and Zeri&Lode), and we added a starting INN Artwork node to the graph. Following, the 5W narrative questions and their sub-questions were raised9. Figure 2(a) illustrates how the network visualisation appears at this stage. The following sections illustrate in detail the network expansion according to each main question.
?Who This question concerned the author since no information about other involved people (.e.g the patron) was available on the KBs. It was possible to retrieve the author’s date of birth and death, places and art movement. Since this information partially answered other open narrative questions, such as ?When and ?Art movement (sub-question of ?What, a relation between them was added. It was also possible to retrieve other paintings by the same author, which is relevant information for art historical research, e.g. for style and subject matter comparison.
?What This question included the retrieval of metadata, e.g. the title, type, genre, material, and art movement of the painting. All the sub-questions found an answer on Wikidata and Zeri. We then answered the narrative question ’?Subject, sub-question in the ?What variable, which
9The visualisation of the graph at each stage is available at https://inn-painting-art-interpretation.streamlit.app/ Network_Evolution (a) (b) retrieved a great amount of information. The graph is enriched with the subjects depicted, and some of them were aligned through the reference to a common Iconclass identifier (92C454 for Venus and Cupid). The Iconology dataset provided a thorough description of the subjects according to three layers of understanding. Also, the results obtained by CV analysis were integrated, despite only the concept of ”person” being recognised, and the algorithm erroneously recognising Venus’s diadem ad a hat. Following, we retrieved the sources of the painting (i.e. an epithalamium, according to Wikidata) and of various objects, e.g. the incense burner described as a proper bridal chamber decoration by the Roman poet Sidonius (source: Iconology Dataset).
?When The retrieval of information about the artist partially answered the question, restricting the range of the possible date of the artwork creation to the one of the author’s life. Nevertheless, a more precise date was provided by Wikidata (1530) and Zeri (1520-1556).
?Where The main questions raised are 1) where the artwork was created, and 2) where the artwork was intended to be displayed. Whereas they could be formally expressed as queries, no further information was available for Lotto’s painting. The question is only partially answered by the information about the author’s work location previously retrieved in the ?Who question. We consequently added a relation between the narrative questions ?Location and ?Where.
?Why The narrative question ?Why had as a main sub-question the retrieval of the patron, namely the person who commissioned the artwork. Although this sub-question can be formally expressed (Wikidata’s relation wdt:P88), such information is not available for the chosen case study. The narrative question remains unbound.
Second iteration We continued the exploration of further knowledge in which a potential user could be interested, discovering further connections or information about a retrieved node. For example, we retrieved more information about the depicted characters and their role, and we looked for objects’ potentially embedded symbolism, not yet discovered, by querying HyperReal, discovering, for example, that the conch is a symbol of fertility, a concept supporting the meaning of the artwork as a wedding wish.
Among the possible applications that can be developed around the INN data structure, we operationalised one application intended for art historians. It allows them to direct the expansion of the INN by indicating which questions should be preferentially explored and by choosing which knowledge sources should be utilised preferentially. Figure 1 shows a snapshot of this interface. The user-centered implementation of INN data structure helps resolve conflicting or erroneous information from various sources, as the user can remove erroneous information from his/her own network. Future implementation considers providing feedback to KBs about the correctness of the retrieved data and integrating the newly added experts’ knowledge into collaborative graphs (e.g. Wikidata) to improve data quality.
By implementing the questions raised by the literature, the major part of the questions could be expressed as SPARQL queries, performed and answered with currently available information on knowledge graphs. Some answers were bound to multiple questions, as they answered both. Despite multiple information could be aligned through the strategies of reconciliation with a common vocabulary, word similarity, and synonyms detection, challenges in information integration remain open (e.g. when pieces of information slightly difer, as happened with the date of creation). As a solution, we make the INN available as an interactive tool, in which the user can manually select the desired data and perform reconciliations.
This paper introduced an application to the art history domain of Integrative Narrative Networks (INN), a data structure for representing progress in interpreting artworks in the form of a graph that connects questions and answers for a specific work. Narrative closure occurs when the main questions of relevance to the human interpreter have been answered. The paper defined the INN and focused on how semantic web resources about art history and interpretation, which are becoming more and more available, can be marshalled to push the interpretation forward. We used a case study of a painting by Lorenzo Lotto to illustrate the proposed methods and techniques. In future work, we plan to integrate additional resources, conduct more case studies, expand the capabilities of the user interface for art historians with the aid of a user study, make advancements toward the generation of KB-independent queries, and develop another interface embedded in an augmented reality device so that viewers seeing an artwork in situ can also interactively explore the semi-autonomous expansion of the Integrative Narrative Network.
This research was supported by the EU Pathfinder project MUHAI (EU grant 951846) and by the international exchange program of Bologna University.