In the task of tagging narrative images, traditional event or story models are not suitable for temporal-spatial information modeling. These models are too coarse-grained to represent plots and actions information su ciently in the particular eld of culture heritage. In this paper, we design a narrative image annotation ontology (NIAO) model and a tool (NIA) to address these issues, by using ontology design patterns and other relevant models for reusability. The annotation model, combining the OAC (Open Annotation Collaboration) framework and regarding the Plot as a core element, makes a mapping between annotated image regions and high-level image semantics. It has been embedded in NIA, which we successfully use in the task of annotating narrative paintings. This tool can record annotation region pixels and related property values according to NIAO, and these annotation data can be stored as various formats such as csv, json, and rdf. We have built a SPARQL endpoint, in which end users can make semantic queries based on these annotation data, and visualize the results with pictures rather than tables.
A narrative [
The nine-colored deer saved a drowning person when it walked along a river. The drowning person gave his thanks to the deer on his knees, and the deer told the drowning person not to leak its location. At the same time, in the palace, the queen talked about her dream about a nine-colored deer to the king. When the drowning person came back home, he told the whereabouts of the deer to the king and queen. Then the king made an order to hunt the deer. At last, the deer got caught and confronted with the king. It told the cause of the whole thing. Finally the drowning person got punished due to his dishonesty.
In order to provide a guide interpreter for tourists, artists, historians and painters who are not familiar with or interested in these kinds of paintings, more detailed information about their contents should be extracted and organized e ectively. In this paper, we have designed an ontology model and annotation tool of narrative images to represent narrative knowledge in a semantic way to satisfy users' requirements. 2
At present, studies about narrative images are mainly focused on iconography
and culture and arts history. The study of semantic annotations in narrative
images is rare. However, lots of event or story models are developed during
these years in di erent elds, such as the Event ontology3, the Stories Ontology
[
The main di erences between these event or story models and our approach in NIAO are that the NIAO we designed is suitable to model plot and action level content and vague information in images, where general event models are too coarse-grained to represent such ne-grained information at this level. In addition, in these models it is not easy to represent temporal-spacial information by using Time or Location interfaces only, particularly in dealing with continuity of plots and the actions of entities in marked areas in an image. 3 See http://motools.sourceforge.net/event/event.html 4 See http://motools.sourceforge.net/timeline/timeline.html 5 See http://ontologydesignpatterns.org/wiki/Submissions:EventCore 6 See http://ontologydesignpatterns.org/wiki/Submissions:EventProcessing
A Plot, as a process of change, occurs in a speci c situation of time and space, and
can be seen as a crm:Concept Object collection in the CIDOC CRM model [
The entire model of Plots for narrative images is shown in Figure 2. There
are chronological and overlapping relations between plots and their related image
regions. A plot occurs (hasSetting ) in a particular situation (Context ), and plot s
have some entities involved in them, such as Person under Agent, and other
entities. A plot represented in an image region usually combines some dynamic
elements (Dynamics) to show the development process of the plot, such as the
Action of some participants in plots.
The temporal relations between Plots can be expressed through the
following object properties. The niao:nextPlot and niao:prevPlot are widely used in
temporal relations between plots, and the other temporal relations are referenced
to Allen's temporal relations of time [
An tool that uses NIAO for embedding Narrative Image Annotations (NIA) has been developed. The web UI of this tool is shown in Figure 3. All properties from NIAO could be embedded into this tool as columns, and we di erentiate two di erent properties, namely image level metadata like author, annotator, shot time of this image, and region level properties, like hasPlot, hasAction and so on from NIAO. Users can also add new columns at the end of the table to extend this model in their annotation task.
This tool can help users choose NOUN or VERB conveniently from imported texts as Entity or Action values while lling elds in the annotation table (left part in Figure 3). Automatic image object recognition or annotation as a function can be integrated into NIA in the future.
Annotation data is stored in a Graph Data Base and some interesting SPARQL queries, like shown in Figure 4, can be executed. The results can also be rendered in a storytelling way, as shown in the lower part of Figure 4, in which each plot is linked to its corresponding image region with a sequence number on them. 5
In this paper, we have designed a narrative image annotation model and annotation tool, NIAO and NIA, in the eld of cultural heritage. We applied NIAO and NIA to annotating narrative images, which facilitated publishing the annotation results as Linked Data, and provided a more visual way of inspecting SPARQL query results. NIAO is a simple model with a small number of important elements accompanying with related properties. When embedded into the NIA tool, some properties like niao:meets and others in NIAO may be not necessarily needed according to actual tasks. These properties are included for completeness and to support more use cases. In the processing of image labels, we do not expect annotators to be confronted with much manual work, which mostly consists of lling numbers into elds. At the same time, if entity recognition for text and images are enabled in the future, this will reduce time-consuming labor and further facilitate the labeling process.