An Ontology Model for Narrative Image Annotation in the Field of Cultural Heritage Lei Xu1 , Albert Meroño-Peñuela2 , Zhisheng Huang2 , and Frank van Harmelen2 1 School of Information Management, Wuhan University, Wuhan, China 2 Department of Computer Science, VU University, Amsterdam, the Netherland Abstract. In the task of tagging narrative images, traditional event or story models are not suitable for temporal-spatial information modeling. These models are too coarse-grained to represent plots and actions infor- mation sufficiently in the particular field of culture heritage. In this pa- per, we design a narrative image annotation ontology (NIAO) model and a tool (NIA) to address these issues, by using ontology design patterns and other relevant models for reusability. The annotation model, combin- ing the OAC (Open Annotation Collaboration) framework and regarding the Plot as a core element, makes a mapping between annotated image regions and high-level image semantics. It has been embedded in NIA, which we successfully use in the task of annotating narrative paintings. This tool can record annotation region pixels and related property values according to NIAO, and these annotation data can be stored as various formats such as csv, json, and rdf. We have built a SPARQL endpoint, in which end users can make semantic queries based on these annotation data, and visualize the results with pictures rather than tables. Keywords: Image annotation, Narrative image, Plot ontology 1 Introduction and Motivation A narrative [2,6] is a general unifying framework used for relating real-life or fic- tional stories involving concrete or imaginary characters and their relationships. A narrative may consist of a work of speech, writing, song, film, television, video game, photography, theater, etc. According to this definition, narrative image is a kind of image with stories behind it, like the one shown in Figure 1. This is a mural image, and the original mural was located in No. 257 Mogao cave in China. The content of this image is about a Jataka tale about the nine-colored deer, and can be simply described with the following words. The nine-colored deer saved a drowning person when it walked along a river. The drowning person gave his thanks to the deer on his knees, and the deer told the drowning person not to leak its location. At the same time, in the palace, the queen talked about her dream about a nine-colored deer to the king. When the drowning person came back home, he told the whereabouts of the deer to the king and queen. Then the king made an 118 A. Adamou, E. Daga and L. Isaksen (eds.) order to hunt the deer. At last, the deer got caught and confronted with the king. It told the cause of the whole thing. Finally the drowning person got punished due to his dishonesty. Fig. 1. Jataka tale about the Nine-Colored Deer In order to provide a guide interpreter for tourists, artists, historians and painters who are not familiar with or interested in these kinds of paintings, more detailed information about their contents should be extracted and organized effectively. In this paper, we have designed an ontology model and annotation tool of narrative images to represent narrative knowledge in a semantic way to satisfy users’ requirements. 2 Related Work At present, studies about narrative images are mainly focused on iconography and culture and arts history. The study of semantic annotations in narrative images is rare. However, lots of event or story models are developed during these years in different fields, such as the Event ontology3 , the Stories Ontology [11], Timeline4 , Storytelling Ontology [10], Narrative ontology [3], Narrative and Action Ontology [4], ABC Ontology [8], BBC’s Storyline ontology [12], Bal’s layered view of narrative [2], SEM [16], the Activity ontology [9], LODE [14], Event-model F [13], Wikipedia’s Current Events Ontology [15] and other related ontology design patterns (ODP) like [7], EventCore5 and EventProcessing6 . The main differences between these event or story models and our approach in NIAO are that the NIAO we designed is suitable to model plot and action level content and vague information in images, where general event models are too coarse-grained to represent such fine-grained information at this level. In addition, in these models it is not easy to represent temporal-spacial information by using Time or Location interfaces only, particularly in dealing with continuity of plots and the actions of entities in marked areas in an image. 3 See http://motools.sourceforge.net/event/event.html 4 See http://motools.sourceforge.net/timeline/timeline.html 5 See http://ontologydesignpatterns.org/wiki/Submissions:EventCore 6 See http://ontologydesignpatterns.org/wiki/Submissions:EventProcessing 2nd Workshop on Humanities in the Semantic Web (WHiSe 2017) 119 3 Narrative Image Annotation Ontology A Plot, as a process of change, occurs in a specific situation of time and space, and can be seen as a crm:Concept Object collection in the CIDOC CRM model [5]. The OAC (Open Annotation Collaboration)7 as a bridge framework for media annotation is used in our model to connect the task of image annotation to plot modeling. A crm:Concept Object in an image can be connected to an Annota- tion’s Body in OAC through the niao:referTo property, so in NIAO, Plot was added as a reference to Body. The entire model of Plots for narrative images is shown in Figure 2. There are chronological and overlapping relations between plots and their related image regions. A plot occurs (hasSetting) in a particular situation (Context), and plots have some entities involved in them, such as Person under Agent, and other entities. A plot represented in an image region usually combines some dynamic elements (Dynamics) to show the development process of the plot, such as the Action of some participants in plots. Fig. 2. UML Diagram about the Core Classes and Properties defined in NIAO The temporal relations between Plots can be expressed through the follow- ing object properties. The niao:nextPlot and niao:prevPlot are widely used in temporal relations between plots, and the other temporal relations are referenced to Allen’s temporal relations of time [1], which are more strict with the beginning and ending of a plot’s time. In a Plot, there should be some Entities making this 7 See https://www.w3.org/TR/annotation-model/ 120 A. Adamou, E. Daga and L. Isaksen (eds.) plot happen. In our model, we use crm:Entity for reusability. Instances of Entity can be assigned a Role, which is relevant for characters, such as King, Actor, Recipient etc. We use sem:Role and sem:Type [16] to represent an entity’s role and its type. In art images, vague time information or time with uncertainty and inaccuracy like Morning, Summer, Dynasty and other abstract objects can be represented easily as entities. The same applies to locations or places in images, usually characterized by mountains, rivers, palaces and other objects. Therefore, we assign TemporalSpacialEntity to Entity in NIAO. Dynamics is referred to Rossana Damiano’s work [4] in our model as shown in Figure 2. Action is a type of Dynamics and Dynamics has different types, namely DynamicsType. Dynamic elements in images are better represented by Actions to connect different entities in plots. Context provides this model with extensibility, and it contains envi- ronment, background information, and other textual descriptions information of plots, which should normally be aligned with a separate information object patterns or models for specification. 4 Narrative Image Annotation Tool An tool that uses NIAO for embedding Narrative Image Annotations (NIA) has been developed. The web UI of this tool is shown in Figure 3. All properties from NIAO could be embedded into this tool as columns, and we differentiate two different properties, namely image level metadata like author, annotator, shot time of this image, and region level properties, like hasPlot, hasAction and so on from NIAO. Users can also add new columns at the end of the table to extend this model in their annotation task. This tool can help users choose NOUN or VERB conveniently from imported texts as Entity or Action values while filling fields in the annotation table (left part in Figure 3). Automatic image object recognition or annotation as a function can be integrated into NIA in the future. Annotation data is stored in a Graph Data Base and some interesting SPARQL queries, like shown in Figure 4, can be executed. The results can also be rendered in a storytelling way, as shown in the lower part of Figure 4, in which each plot is linked to its corresponding image region with a sequence number on them. 5 Conclusion and Future Work In this paper, we have designed a narrative image annotation model and annota- tion tool, NIAO and NIA, in the field of cultural heritage. We applied NIAO and NIA to annotating narrative images, which facilitated publishing the annotation results as Linked Data, and provided a more visual way of inspecting SPARQL query results. NIAO is a simple model with a small number of important elements accompanying with related properties. When embedded into the NIA tool, some properties like niao:meets and others in NIAO may be not necessarily needed according to actual tasks. These properties are included for completeness and to support more use cases. In the processing of image labels, we do not expect 2nd Workshop on Humanities in the Semantic Web (WHiSe 2017) 121 Fig. 3. Narrative Image Annotation Tool’s UI Fig. 4. Sparql Query for Finding all Plots in Figure 1 122 A. Adamou, E. Daga and L. Isaksen (eds.) annotators to be confronted with much manual work, which mostly consists of filling numbers into fields. At the same time, if entity recognition for text and images are enabled in the future, this will reduce time-consuming labor and further facilitate the labeling process. References 1. James F Allen. Maintaining knowledge about temporal intervals. Communications of the ACM, 26(11):832–843, 1983. 2. Mieke Bal. Narratology: Introduction to the theory of narrative. University of Toronto Press, 2009. 3. Fabio Ciotti. Toward a formal ontology for narrative. MATLIT: Materialidades da Literatura, 4(1):29–44, 2016. 4. Rossana Damiano and Antonio Lieto. Ontological representations of narratives: a case study on stories and actions. In OASIcs-OpenAccess Series in Informatics, volume 32. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2013. 5. Martin Doerr. The cidoc conceptual reference module: an ontological approach to semantic interoperability of metadata. AI magazine, 24(3):75, 2003. 6. Manfred Jahn. Narratology: A guide to the theory of narrative. English Depart- ment, University of Cologne, 28, 2005. 7. Adila A Krisnadhi and Pascal Hitzler. A core pattern for events. http:// ontologydesignpatterns.org/wiki/images/5/52/WOP2016_paper_04.pdf. Ac- cessed May 24, 2017. 8. Carl Lagoze and Jane Hunter. The abc ontology and model. Journal of Digital Information, 2(2), 2006. 9. Georgios Meditskos, Stamatia Dasiopoulou, Vasiliki Efstathiou, and Ioannis Kom- patsiaris. Ontology patterns for complex activity modelling. In International Work- shop on Rules and Rule Markup Languages for the Semantic Web, pages 144–157. Springer, 2013. 10. Arturo Nakasone and Mitsuru Ishizuka. Storytelling ontology model using rst. In Proceedings of the IEEE/WIC/ACM international conference on Intelligent Agent Technology, pages 163–169. IEEE Computer Society, 2006. 11. Michael O.Jewell, Paul Rissen, and Toby Harris. The stories ontology. http: //www.contextus.net/stories/. Accessed May 24, 2017. 12. Paul Rissen, Helen Lippell, Matt Chadburn, Tom Leitch, Dan Brickley, Michael Smethurst, and Sebastien Cevey. Storyline ontology. http://www.bbc.co.uk/ ontologies/storyline. Accessed May 24, 2017. 13. Ansgar Scherp, Thomas Franz, Carsten Saathoff, and Steffen Staab. F–a model of events based on the foundational ontology dolce+ dns ultralight. In Proceedings of the fifth international conference on Knowledge capture, pages 137–144. ACM, 2009. 14. Ryan Shaw, Raphaël Troncy, and Lynda Hardman. Lode: Linking open descrip- tions of events. In Asian Semantic Web Conference, pages 153–167. Springer, 2009. 15. Giang Tran, Mohammad Alrifai, and Eelco Herder. Timeline summarization from relevant headlines. In European Conference on Information Retrieval, pages 245– 256. Springer, 2015. 16. Willem van Hage, Véronique Malaisé, Roxane Segers, Laura Hollink, and Guus Schreiber. Simple event model. http://semanticweb.cs.vu.nl/2009/11/sem/. Accessed May 24, 2017.