In this paper we discuss a tool for annotation and search in a collection of art images. Multiple existing Thesaurus may be used to support this process, including the Art and Architecture Thesaurus, WordNet, ULAN and Iconclass. We discuss knowledge-engineering aspects such as the annotation methodology, and the use of taxonomic dynamic search. The annotation and search process are illustrated with an application scenario.
Historically, curators have retrieved images by first manually annotating them with keywords. Given a query, these annotations are used to retrieve appropriate pictures. Underlying this approach is the belief that the words associated (manually) with a picture essentially capture the semantics of the picture and any retrieval based on these keywords will, therefore, retrieve relevant pictures. Since manual image annotation is expensive, there has been great interest in producing automatic ways to retrieve images based on content.
Queries based on image concepts like color or texture, have been proposed for retrieving images by content, but most users find it difficult to query using such visual attributes. Most people would prefer to pose text queries and find images relevant to those queries. For example, one should be able to pose a query like find me sad people. This is difficult, if not impossible, with many of the current image retrieval systems. This paper aims to address the problem of art image retrieval (AIR) and introduce a methodology to intent to reduce the semantic gap in image annotation of art images from a large art image collection.
The explosive growth of digital media and its usage on the web has opened both great challenges and opportunities to computer science researchers. One such example is the lively research area content-based image retrieval (CBIR), which aims to effectively and efficiently locate a few relevant images in a large database according to a user’s query concept.
This paper will focus on another related, but different topic: Art image retrieval
(AIR), which is of great interests to us because of its application potentials and
technical challenges. In recent years, many museums, galleries and commercial
companies have put their art images on the web for online exhibitions or selling. The
WWW technology has created dramatic flexibilities in helping people freely share,
exchange, enjoy and purchase paintings. To facilitate the search of paintings, one of
the main standards for the cataloging of cultural objects is the one proposed by the
Getty Research Institute, namely, Categories for the Description of Works of Art
(CDWA) [
A second model considered was the Cataloging Cultural Objects model (CCO). A
guide to describe cultural works and their images was published as a draft in February
2005, and its final version was published in 2006 [
From a technical point of view, AIR encounters many challenges that are also common in CBIR, like high-dimension feature space, nonlinear distributions, insufficient training examples and the gap between low-level features and high-level contents.
Thus previous research on CBIR has provided a good basis for our work on AIR. Moreover, there are some other unique characteristics coming with AIR. In the following work we will discuss some major issues.
One of the objectives that our approach tries to achieve is the search and information retrieval through an analysis of a textual description carried out for different pieces from the database of the artistic objects.
In art works databases, information retrieval could be done by the cataloging descriptors: author, title, or pictorial tendency. However, in many cases, users would like to retrieve some or a group of works through the objects painted, photographed or drawn. In other words, if a user were interested in retrieving pieces of work that include women, as response, he would expect for pieces in which feminine figures appear: Girls, young women or adult women. The type of search mentioned implies the use of facets that allows proper retrieval of items searched. The types of query required should then allow retrieval of information, either through the alreadyexisting facets in the system or defined at the time of execution through search equations in which the user is able to include a series of words that may indicate the elements that define a concept, which must be found in the description of the piece, so that it can be retrieved (simple concept). For example,
Urban images = vehicles + buildings + people In both cases, the defined concepts may become permanent upon the user’s request. By doing so, some feedback can be obtained from the user. On the other hand, as an object can be a part belonging to more than one concept, it is the intention to be able to categorize the order of importance of the object regarding the concepts related to it.
To support the structure and efficient retrieval of the artistic objects, and given the complexity in the multiplicity of the possible interpretations and contents that can be linked to the objects, we recommend the use of a faceted classification process.
Under the previous premise, faceted taxonomies can take advantage of the ways in
which the metadata act. Metadata are the collection of structured information about
any type of object or any part of that object, e.g. the name (or title) of a work art, the
author, age, image, textual description (denotation), its interpretation (connotation), or
its gender. All these are some kind of metadata associated with a cultural object.
In our approach, we have implemented the faced classification, and thus, the metadata
model proposed in [
In this natural way, every element of the metadata structures can be incorporated
as a concept of the faceted taxonomy and can be retrievable through a search engine.
The importance of taxonomies is based on the possibility of their use as the triplet
{classification diagram, semantic interpretation, knowledge map} [
Sacco [
In the literature, we can find two interesting approaches to the treatment of textual objects. The first one presupposes the existence of several topics on which a taxonomy is to be developed, and in this way, the job of the developed algorithms is to extract the terms and concepts linked to the topics in question from the documents. In this approach, each of the several derived taxonomies requires a group of key terms, linked to the concepts based on which the classification of the documents of the target objective will be carried out.
The second approach, in contrast, focuses on determining the facets and defining
their taxonomy, departing from the textual analysis of the documents of the
collection. In this case, some techniques for text analysis are used. In particular, in
[
The purpose of indexing is to identify the concepts that represent the content and
translation of documents into a way that can be computably managed [
We would now present a brief description of the three facets, genre, denotation, and connotation, which we had already mentioned in the previous paragraphs. Using these three facets, we would extend the classification of the artistic objects.
Genre (Artistic subject): Each of the different categories or kinds that can be used to order the works depending on common features, such as shape and content. For example, the portraits and self-portraits, landscapes, religion, mythology, etc.
Denotation (Content): Concepts that are shown in an artistic work.
Connotation (Interpretation): Concepts reflected or transmitted by the artistic work. For example, the word rain has the connotative meaning of sadness or melancholy.
From these three facets, only the facet Genre was defined by an expert user. The other two facets were obtained automatically.
To develop an information system and a motor search for the retrieved objects in a
collection of 500 works of art using their textual description in Spanish [
Finally, the objects are indexed [
Figure 1. Activity diagram of the dynamic taxonomy generation process
The exploration by the user will be realized by means of a navigation based search
[
We proposed a methodology, using IR techniques, to extract concepts and structures and facets to manage information and enable further research and exploration.
Taxonomies were constructed in the following aspects: gender, connotation and denotation, which can serve as a standard for other cases in which the use of this type of hierarchy is required.
We made and implemented algorithms that allow the retrieval of images by relying on their contents and thereby contributing to solving the semantic labeling of images, which is one of the goals of the CBIR methodology. The concept extraction process and indexing between the facets and the works was designed and subsequently implemented in AWK. This module can be used on any set of textual descriptors. With regard to the construction of dynamic taxonomies and navigation, the following operations were implemented: Zoom-In, Zoom-Out, Pivot, Shift, and Query by Example (QBE).
Dynamic taxonomies generated, unlike those reported in the literature, are not on a closed domain. An example is the taxonomy denotation, in which there is an open domain, associated with a thesaurus for expanding the domain of the facets obtained from the analysis of unstructured text.
Among the extensions and improvements that we have referred to are that of incorporating: The Art & Architecture Thesaurus ® developed by the Getty Research Institute (GRI), and translated into Spanish by the Documentation Center of Patrimonial Assets (under the Directorate of Libraries, Archives and Museums Dibam), the Thesaurus Spanish Historical Heritage, the Multilingual Thesaurus of Nations, and the Extended Multilingual Wordnet (University Alicante).