Iconography is the domain of understanding the meaning of historical visual artworks. A formalization of iconographic knowledge can provide a basis for a semi-automatic description of what the content shown on a historical image means without the need for a domain expert. Semantic Web standards can be applied for an iconographic knowledge representation using multiple levels of expressiveness to model the description. This formalized knowledge can then be leveraged to infer the meaning of images based on a description of the depicted visual elements. Having semantically described the content, this information can be used to compare images based on their meaning and nd related images with a similar semantic description. We present a novel approach for representing iconographic knowledge using a multi-level semantic model in combination with an iconographic inference rule set. This can be used as a basis for similarity search based on the meaning of the content of historical images.
DESIGN, EXPERIMENTATION
Iconography studies the meaning of what the content of
an image depicts, which is distinct from the artistic style.
This meaning is represented by using symbols and allegories.
Symbols are objects that mean something on a conceptual
level in addition to its literal meaning. Allegories stand for
more complex concepts and are usually represented by a
combination of multiple symbolic objects. The meaning of
the content is dependent on the cultural context of the
artwork's creation. The symbolisms used to represent it can
be highly complex and di cult to understand. Even if no
complex concepts are involved, knowledge about the
meaning may not be commonly known anymore. To understand
what is represented by the content of a historical image, the
viewer has to know about iconography. Even for art
historians understanding and interpreting the meaning of an image
is a complex task. Art historian Erwin Panofsky de ned a
methodology [
Even when using an established iconographic system like
Iconclass [
Furthermore, a formal iconographic description can also be used for nding similar images based on the meaning of their content. This can provide navigation options for exploratory search and it can be used for the automatic creation of collections based on a theme. Use-cases can be found in research, education and tourism.
We propose an approach for representing iconographic domain knowledge as a formal model including an inference rule set to semi-automatically create an iconographic description for historical imagery. The model also supports the comparison of images based on the content and can be used to nd similar images based on their meaning. 2.
The model for the Semantic Content Description for image content consists of three parts. First, we de ne a data model as a knowledge representation for describing the content. The model consists of multiple levels with di erent semantic expressive power to represent di erent aspects of the content description. Then a rule set is de ned that represents iconographic domain knowledge and can be used by a reasoning system to infer additional semantic descriptions. The last part is the application of similarity algorithms that work on the semantic descriptions to nd images with similar content. 2.1
As described above the gap between simple keywords and
a description using an iconographic classi cation system is
too wide to allow others than domain experts to correctly
classify images based on the content. Therefore, we
introduce a three level model with increasing semantic
expressive power as a knowledge representation for the semantics
of image content. We use Semantic Web standards for this
representation [
Physical level. The rst level, the physical level, represents physical objects, including plants, animals and persons, that can be seen on the image and that can be used to describe the content in a direct visual way. No domain knowledge is needed for such a description. We take a painting of the Last Supper as an example. If we describe it at the physical level, the artwork is seen as an image depicting 13 men around a table. No cultural knowledge is needed to describe it that way and we do not have any connections to a religious story. classi cation concept. This level is intended to be used by domain experts for a direct iconographic classi cation. On this level the Last Supper is represented as exactly one concept for the story's scene that the content depicts. 2.2
For each of the three levels we use a hierarchical
taxonomy as a representation. For the physical and the conceptual
level we use the Art & Architecture Thesaurus AAT [
Conceptual level. The second level, the conceptual level, represents immaterial concepts like religion, seasons or emotions. These cannot be represented directly in the image, but they can have a symbolic representation by an object or an allegorical representation as a combination of several objects. In contrast to the physical level, a description based on the conceptual level is a semantic description. Here domain knowledge is needed for knowing about the symbolism. With this knowledge we can understand that the painting showing 13 men around a table represents the Last Supper. On the conceptual level the meaning is represented by the concept of Christianity, or more general, that of a religious concept.
Iconographic level. The third level, the iconographic level, represents iconographic concepts and has the highest expressive power. Each iconographic scene is represented by a
For implicit use of iconographic knowledge we propose an iconographic rule set that contains entailments based on the asserted descriptions of the images. These rules can be used by a reasoning system to infer additional describing concepts, as shown in gure 2. Basically, two types of rules can be identi ed. The rst type creates a semantic description by inferring it from a physical description (1). This entails descriptions on the conceptual and iconographic level. The second type improves an existing semantic description by inferring additional concepts on the conceptual and iconographic level (2a, 2b). Both types of rules can be seen as lifting (1, 2a) and lowering (2b) between the three description levels. For improving the rule matching, the hierarchic structure of the taxonomies can be used to reduce the precision needed for matching atoms of the rules. On each rule atom we match not only the concept itself, but the whole subtree below this concept in the hierarchic taxonomy of a level. As some symbols may be interpreted di erently based on the cultural context we can add spatial and temporal constraints to the rules. These rules only apply to images that were created in a speci c region or during a speci c period. Furthermore, specifying the cardinality for rule atoms is important for de ning inferences based on the number of objects depicted in the image. As an example, we present rules that can be applied to semantically describe the Last Supper. Starting with a description on the physical level the content is described by 13 men and a table. As such a scene is likely to represent the Last Supper, we create a rule that infers the Iconclass concept for the Last Supper on the iconographic level based on the physical description of men with cardinality 13 and a table. This inferred description then further infers the concepts for Christianity and religion on the second level based on the Iconclass concept for the Last Supper. The result is a description on each of the three levels. 2.4
2.4.1
For comparing the content of an image we compare the
descriptions on each level and compute a similarity. The
three levels are represented by SKOS Thesaurii and we can
use the taxonomic structure as a basis for this computation.
For pairs of concepts we can use taxonomic similarity
algorithms like Wu & Palmer [
The disadvantage of using taxonomic algorithms is that a comparison of concepts not of the same level is not possible because they do not origin from the same taxonomy. We have to de ne an algorithm for computing this intertaxonomic similarity for concepts that are from di erent levels. An approach for a computation is a combination of the lifting and lowering rules of the iconographic rule set and the taxonomic reasoning. For comparing two concepts, we nd the nearest lifted or lowered two concepts in each of the two taxonomies that are used as atoms in the same rule. These concepts and the connecting rule can be interpreted as a relation for computing the similarity. Using this relation, distance measures can be applied for an intertaxonomic comparison of the two concepts. Figure 4 shows comparisons of concepts on all three levels. Again, we assume two images A and B to compare. Image A is classi ed with the concepts 1 and 3. Image B is class ed with the concepts 2 and 4. We compare the two images by comparing their classifying concepts on all levels. We compare the concepts 1 and 2, where concept 1 is on the physical level and concept 2 is on the conceptual level. Rule 1 is interpreted as an edge connecting the physical and the conceptual level. Using this connection a distance value can be computed for the concepts 1 and 2. Then we compare the concepts 3 and 4, where concept 3 is on the conceptual level and concept 4 is on the iconographic level. Rule 2 connects the conceptual and the iconographic level and is used for the computation of the similarity of the concepts 3 and 4. The result are similarity values for the two images spanning all three levels. 3.
We de ned a Semantic Content Description for representing iconographic knowledge as a three level model in combination with an iconographic inference rule set. It can be used to infer a semantic description based on what is depicted in an image and to further improve descriptions on the semantic levels. The result is a formal representation of the meaning of the content of images. It can be used in combination with similarity algorithms appropriate for the three level model to nd images with a similar meaning. The current state of the work de ned the data model and the reasoning system and implemented it as a semantic web application. The next step is the de nition of an appropriate iconographic rule set for the reasoning system which has to be done in cooperation with iconographic domain experts. Then a representative and signi cantly large data base with images and descriptions is needed. Following is the implementation and evaluation of the similarity algorithms in combination with the automatically inferred semantic descriptions. Evaluation of the similarity search will be done using comparisons of keyword-based retrieval, retrieval on each of the three levels and retrieval using the inter-taxonomic approach. 4.