=Paper=
{{Paper
|id=Vol-1481/paper10
|storemode=property
|title=Semantic Content Description for the Visual Arts
|pdfUrl=https://ceur-ws.org/Vol-1481/paper10.pdf
|volume=Vol-1481
|dblpUrl=https://dblp.org/rec/conf/i-semantics/David15
}}
==Semantic Content Description for the Visual Arts==
https://ceur-ws.org/Vol-1481/paper10.pdf
Towards a Semantic Content Description for the Visual
Arts
Robert David
Institute of Software Technology and Interactive Systems
Vienna University of Technology
Karlsplatz 13, 1040 Vienna, Austria
r.david@semantic-web.at
ABSTRACT the content is dependent on the cultural context of the art-
Iconography is the domain of understanding the meaning of work’s creation. The symbolisms used to represent it can
historical visual artworks. A formalization of iconographic be highly complex and difficult to understand. Even if no
knowledge can provide a basis for a semi-automatic descrip- complex concepts are involved, knowledge about the mean-
tion of what the content shown on a historical image means ing may not be commonly known anymore. To understand
without the need for a domain expert. Semantic Web stan- what is represented by the content of a historical image, the
dards can be applied for an iconographic knowledge repre- viewer has to know about iconography. Even for art histori-
sentation using multiple levels of expressiveness to model the ans understanding and interpreting the meaning of an image
description. This formalized knowledge can then be lever- is a complex task. Art historian Erwin Panofsky defined a
aged to infer the meaning of images based on a description of methodology [5] for analysing different levels of meaning to
the depicted visual elements. Having semantically described help in the analysis of the content. Formalizing some of the
the content, this information can be used to compare images simpler aspects of art-historic analyses can support users in
based on their meaning and find related images with a sim- understanding the conceptual meaning of historical images.
ilar semantic description. We present a novel approach for Even when using an established iconographic system like
representing iconographic knowledge using a multi-level se- Iconclass [7], it is very difficult to classify artworks with-
mantic model in combination with an iconographic inference out domain knowledge. Using keyword search for metadata
rule set. This can be used as a basis for similarity search in such a system may still result in several matches, from
based on the meaning of the content of historical images. which the correct one has to be identified. This seman-
tic gap from simple keywords to iconographic concepts has
to be bridged automatically in order to reduce the domain
CCS Concepts knowledge needed for a correct classification. This would
•Computing methodologies → Knowledge represen- provide non-experts with the possibility to reach a higher
tation and reasoning; •Applied computing → Fine level of art-historical understanding for an image. It would
arts; •Information systems → Clustering and classifica- also be helpful for domain experts as a research tool in the
tion; •Theory of computation → Automated reasoning; art-historical domain.
Furthermore, a formal iconographic description can also
be used for finding similar images based on the meaning
General Terms of their content. This can provide navigation options for
DESIGN, EXPERIMENTATION exploratory search and it can be used for the automatic cre-
ation of collections based on a theme. Use-cases can be
found in research, education and tourism.
1. INTRODUCTION We propose an approach for representing iconographic do-
Iconography studies the meaning of what the content of main knowledge as a formal model including an inference
an image depicts, which is distinct from the artistic style. rule set to semi-automatically create an iconographic de-
This meaning is represented by using symbols and allegories. scription for historical imagery. The model also supports
Symbols are objects that mean something on a conceptual the comparison of images based on the content and can be
level in addition to its literal meaning. Allegories stand for used to find similar images based on their meaning.
more complex concepts and are usually represented by a
combination of multiple symbolic objects. The meaning of
2. SEMANTIC CONTENT DESCRIPTION
The model for the Semantic Content Description for im-
age content consists of three parts. First, we define a data
model as a knowledge representation for describing the con-
tent. The model consists of multiple levels with different
semantic expressive power to represent different aspects of
the content description. Then a rule set is defined that rep-
resents iconographic domain knowledge and can be used by
. a reasoning system to infer additional semantic descriptions.
The last part is the application of similarity algorithms that
34
work on the semantic descriptions to find images with simi- classification concept. This level is intended to be used by
lar content. domain experts for a direct iconographic classification. On
this level the Last Supper is represented as exactly one con-
2.1 A model for knowledge representation cept for the story’s scene that the content depicts.
As described above the gap between simple keywords and
a description using an iconographic classification system is 2.2 Taxonomic representation
too wide to allow others than domain experts to correctly For each of the three levels we use a hierarchical taxon-
classify images based on the content. Therefore, we intro- omy as a representation. For the physical and the conceptual
duce a three level model with increasing semantic expres- level we use the Art & Architecture Thesaurus AAT [6]. It
sive power as a knowledge representation for the semantics is defined using the Simple Knowledge Organization System
of image content. We use Semantic Web standards for this SKOS [1]. AAT contains concepts for many physical things
representation [2]. They see a widespread use in the Cul- that can be used for representing the physical level. De-
tural Heritage domain and allow for a flexible semantic data scribing this level using the concepts of a taxonomy is more
description [3]. expressive than using keywords. Even though this level rep-
resents no meaning, a controlled vocabulary is needed for
Physical level. The first level, the physical level, represents defining the rules in the next part. The AAT also contains
physical objects, including plants, animals and persons, that abstract concepts in the Associated Concepts Facet, that can
can be seen on the image and that can be used to describe be used to represent the symbolisms of the conceptual level.
the content in a direct visual way. No domain knowledge is For the iconographic level we use the Iconclass classification
needed for such a description. We take a painting of the Last system [7], which also is available as a SKOS Thesaurus.
Supper as an example. If we describe it at the physical level, Having a hierarchy of classifying concepts on each level will
the artwork is seen as an image depicting 13 men around a be important for the next two parts.
table. No cultural knowledge is needed to describe it that
way and we do not have any connections to a religious story. 2.3 A rule set for iconographic reasoning
Figure 2: Three level reasoning
For implicit use of iconographic knowledge we propose an
iconographic rule set that contains entailments based on the
Figure 1: The Last Supper described by three lev- asserted descriptions of the images. These rules can be used
els: 13 men and a table (physical level), Christianity by a reasoning system to infer additional describing con-
and religion (conceptual level), and the Last Supper cepts, as shown in figure 2. Basically, two types of rules can
(iconographic level) be identified. The first type creates a semantic description
by inferring it from a physical description (1). This entails
descriptions on the conceptual and iconographic level. The
second type improves an existing semantic description by
Conceptual level. The second level, the conceptual level, inferring additional concepts on the conceptual and icono-
represents immaterial concepts like religion, seasons or emo- graphic level (2a, 2b). Both types of rules can be seen as
tions. These cannot be represented directly in the image, lifting (1, 2a) and lowering (2b) between the three descrip-
but they can have a symbolic representation by an object or tion levels. For improving the rule matching, the hierarchic
an allegorical representation as a combination of several ob- structure of the taxonomies can be used to reduce the pre-
jects. In contrast to the physical level, a description based cision needed for matching atoms of the rules. On each rule
on the conceptual level is a semantic description. Here do- atom we match not only the concept itself, but the whole
main knowledge is needed for knowing about the symbolism. subtree below this concept in the hierarchic taxonomy of a
With this knowledge we can understand that the painting level. As some symbols may be interpreted differently based
showing 13 men around a table represents the Last Supper. on the cultural context we can add spatial and temporal
On the conceptual level the meaning is represented by the constraints to the rules. These rules only apply to images
concept of Christianity, or more general, that of a religious that were created in a specific region or during a specific pe-
concept. riod. Furthermore, specifying the cardinality for rule atoms
is important for defining inferences based on the number of
Iconographic level. The third level, the iconographic level, objects depicted in the image. As an example, we present
represents iconographic concepts and has the highest expres- rules that can be applied to semantically describe the Last
sive power. Each iconographic scene is represented by a Supper. Starting with a description on the physical level
35
the content is described by 13 men and a table. As such a sible because they do not origin from the same taxonomy.
scene is likely to represent the Last Supper, we create a rule We have to define an algorithm for computing this inter-
that infers the Iconclass concept for the Last Supper on the taxonomic similarity for concepts that are from different
iconographic level based on the physical description of men levels. An approach for a computation is a combination
with cardinality 13 and a table. This inferred description of the lifting and lowering rules of the iconographic rule set
then further infers the concepts for Christianity and religion and the taxonomic reasoning. For comparing two concepts,
on the second level based on the Iconclass concept for the we find the nearest lifted or lowered two concepts in each
Last Supper. The result is a description on each of the three of the two taxonomies that are used as atoms in the same
levels. rule. These concepts and the connecting rule can be in-
terpreted as a relation for computing the similarity. Using
2.4 Finding images with similar meaning this relation, distance measures can be applied for an inter-
taxonomic comparison of the two concepts. Figure 4 shows
2.4.1 Taxonomic similarity comparisons of concepts on all three levels. Again, we as-
sume two images A and B to compare. Image A is classified
with the concepts 1 and 3. Image B is classfied with the con-
cepts 2 and 4. We compare the two images by comparing
their classifying concepts on all levels. We compare the con-
cepts 1 and 2, where concept 1 is on the physical level and
concept 2 is on the conceptual level. Rule 1 is interpreted
as an edge connecting the physical and the conceptual level.
Using this connection a distance value can be computed for
the concepts 1 and 2. Then we compare the concepts 3 and
4, where concept 3 is on the conceptual level and concept 4
is on the iconographic level. Rule 2 connects the conceptual
Figure 3: Similarity of concepts on each level: com- and the iconographic level and is used for the computation
paring 1 to 2, 3 to 4 and 5 to 6 of the similarity of the concepts 3 and 4. The result are sim-
ilarity values for the two images spanning all three levels.
For comparing the content of an image we compare the
descriptions on each level and compute a similarity. The 3. CONCLUSIONS
three levels are represented by SKOS Thesaurii and we can We defined a Semantic Content Description for represent-
use the taxonomic structure as a basis for this computation. ing iconographic knowledge as a three level model in com-
For pairs of concepts we can use taxonomic similarity al- bination with an iconographic inference rule set. It can be
gorithms like Wu & Palmer [4]. As we can describe each used to infer a semantic description based on what is de-
of the levels with multiple concepts we have to expand our picted in an image and to further improve descriptions on
taxonomic comparison to sets of concepts. Figure 3 shows the semantic levels. The result is a formal representation
the three levels and comparisons of concepts therein. We of the meaning of the content of images. It can be used
assume two images A and B to compare. Image A is classi- in combination with similarity algorithms appropriate for
fied with the concepts 1,3 and 5. Image B is classfied with the three level model to find images with a similar mean-
the concepts 2,4 and 6. We compare the two images by ing. The current state of the work defined the data model
comparing their classifying concepts on each level. On the and the reasoning system and implemented it as a seman-
physical level, the concepts 1 and 2 are compared, on the tic web application. The next step is the definition of an
conceptual level, the concepts 3 and 4 are compared and on appropriate iconographic rule set for the reasoning system
the iconographic level, the concepts 5 and 6 are compared. which has to be done in cooperation with iconographic do-
The result is a similarity value for the two images on each main experts. Then a representative and significantly large
of the three levels. data base with images and descriptions is needed. Follow-
2.4.2 Inter-taxonomic similarity ing is the implementation and evaluation of the similarity
algorithms in combination with the automatically inferred
semantic descriptions. Evaluation of the similarity search
will be done using comparisons of keyword-based retrieval,
retrieval on each of the three levels and retrieval using the
inter-taxonomic approach.
4. REFERENCES
[1] S. B. Alistair Miles. Skos simple knowledge
organization system reference.
http://www.w3.org/TR/skos-reference/, 2009. [Online;
accessed 31-March-2015].
Figure 4: Similarity of concepts on all levels: com-
[2] T. Berners-Lee, J. Hendler, O. Lassila, et al. The
paring 1 to 2 and 3 to 4
semantic web. Scientific american, 284(5):28–37, 2001.
[3] M. Doerr, S. Gradmann, S. Hennicke, A. Isaac,
The disadvantage of using taxonomic algorithms is that C. Meghini, and H. van de Sompel. The europeana data
a comparison of concepts not of the same level is not pos- model (EDM). In WORLD LIBRARY AND
36
INFORMATION CONGRESS: 76TH IFLA
GENERAL CONFERENCE AND ASSEMBLY 10-15
August 2010, Gothenburg, Sweden, Aug. 2010.
[4] R. Mihalcea, C. Corley, and C. Strapparava.
Corpus-based and knowledge-based measures of text
semantic similarity. In AAAI, volume 6, pages 775–780,
2006.
[5] E. Panofsky. Studies in Iconology: Humanistic Themes
in the Art of the Renaissance. Harper & Row, 1972.
[6] T. Petersen. Art & architecture thesaurus. Oxford
University Press, 1990.
http://www.getty.edu/research/tools/vocabulary/aat/.
[7] H. Van de Waal. Iconclass: an iconographic
classification system, volume 2. North-Holland, 1974.
37