=Paper=
{{Paper
|id=None
|storemode=property
|title=An Approach to Semantic Content Based Image Retrieval Using Logical Concept Analysis
|pdfUrl=https://ceur-ws.org/Vol-939/paper8.pdf
|volume=Vol-939
|dblpUrl=https://dblp.org/rec/conf/ecai/GuerinBR12
}}
==An Approach to Semantic Content Based Image Retrieval Using Logical Concept Analysis==
https://ceur-ws.org/Vol-939/paper8.pdf
An approach to Semantic Content Based Image
Retrieval using Logical Concept Analysis.
Application to comicbooks.
Clément Guérin, Karell Bertet and Arnaud Revel
L3I, University of La Rochelle, France
{cguerin,kbertet,arevel}@univ-lr.fr
Abstract. In this paper, we present an ongoing work aiming to improve
content based image retrieval performance with the help of logical con-
cept analysis. Domain semantic is formalized and used instead of classical
CBIR visual features. This is being applied to comicbooks using Sewelis.
Keywords: Comic Books, Description Logics, Semantic, Logical Con-
cept Analysis, Content-Based Image Retrieval.
1 Introduction
Web search engines usually give poor results when searching in multimedia
databases since they use the contextual web page, or the meta information at-
tached to the multimedia objects. The semantic meaning that the user usually
attaches to the content of the document is often very different from the text used
for indexing the image (semantic gap). Content Based Image Retrieval (CBIR)
has been proposed to search into huge unstructured image databases by giving
an example of what the user is looking for instead of describing the concept it
represents. Classically, visual features are extracted from the images and then
compiled into a signature [1]. To perform the retrieval, a similarity function is
computed to compare the index of the query to those of the collection. A ranking
of the results is produced according to the similarity and shown to the users. To
improve the quality of the retrieval, an interaction with the user, called relevance
feedback [2], can be added. These techniques work pretty well in the context of
searching visually similar images in unstructured image databases.
In this article, we are interested in CBIR in the context of comicbook databases.
In this case, databases cannot be considered as unstructured anymore since im-
ages can be grouped in terms of panels, pages and volumes which are themselves
associated with metadata concerning the author or the series they belong to.
We would like to benefit both from the search facilities given by CBIR tech-
niques with feedback and semantic information embedded in the structure of the
comicbooks documents. To do such a thing, Logical Concept Analysis (LCA),
an extension of Formal Concept Analysis (FCA) [3], is used through the Sewelis
implementation [4]. We will first go through the presentation of our comicbook
model and its transcription into LCA. Then we will explain how we can mix
classical CBIR and LCA techniques together to enhance retrieval relevance.
2 Semantic Content Based Image Retrieval
2.1 Model description
Comicbooks have a natural hierarchical structure that can be formalized. They
are made of pages which contain panels. These panels can eventually be gath-
ered in strips1 and contain different kind of objects, such as speech balloons,
characters, free text, etc. Balloons can be of many kinds (dialogue, thoughts
etc.).This knowledge can be used to deduce more information such as pieces of
the storyline. Fig. 1 illustrates the model we propose to formalize the comicbooks
domain. It has been described with more details in [5].
Comic
- hasLabel
- hasWriter
- hasDrawer
hasPart
Validation Page
- isCorrect - hasNumber
is_a is_a hasReference hasValidation hasPart hasImage
RegionOfInterest Image
PanelValidation NextValidation - hasX - hasWidth
- hasY - hasHeight
is_a is_a hasExtractor
TextRegion Panel
Extractor
- hasText hasNext is_a hasNext
- hasName
- hasRank - hasRank
hasBalloon is_a is_a
Balloon GroundTruth Automatic
Fig. 1. Part of our model concepts hierarchy and their properties.
2.2 Heterogeneous and complex data integration
Some works [6–8] already enhanced the classical CBIR techniques with an ontol-
ogy approach. The modelling was mainly focused on the description of segmented
areas though. We would like to go further and use the full power of description
provided by description logics.Indeed, the model presented previously is expres-
sive enough to allow the retrieval of similar panels considering different charac-
teristics like low-level image features, spatial relations or semantic information.
An input picture, picked from the database, being given, the system will not
only be able to retrieve similar pictures based on the classical image charac-
teristics (colors, shapes, textures...), but also based on the associated semantic
and the knowledge that could have been learnt previously. Considering that the
query is a Panel instance, the search can focus on:
(1) The panel’s characteristics (i.e. data properties of a Panel object). This
could be its rank, its shape, its size, its position, its shot type, its view angle,
1
A strip is defined as an horizontal sequence of panels. Traditionally, a strip is made
of 1 to 6 panels and a page can contain up to 4 stacked strips.
etc. Images of a very close shot of a character’s face or a landscape picture of a
valley being at the top of a page can be examples of queries.
(2) The panel’s relations (i.e. object properties). Properties of objects related
to the query panel can be used as well as its own characteristics. Therefore, there
are two directions to look at from a panel point of view.
– The search can focus on what is inside the panel, like similar amount of
objects in a scene (a dialogue between two characters for instance) or related
text content. The retrieval process can also rely on objects contained in the
panel, whether they are identified or not. Assume that the query picture
contains an identified character A whose visual signature is defined by the
set of features X. The system will not only look for panels containing an
instance of A, but also for those showing a spatial region matching X.
– Outside: the search can focus on panels sharing page’s or comic’s character-
istics (such as author, style, etc.)
These kinds of retrieval angles are not mutually exclusive and it is very
possible to combine two or more of them in order to narrow the result set. The
search possibilities are only limited by the completeness of the description.
2.3 Sewelis integration
In databases, information retrieval is classically performed by request queries
expressed in a specific request language, as SQL for example. However, the more
refined is the search, the more sophisticated is the request. Some information
retrieval systems offer a simpler search refinement by navigation in a predefined
static data structure, where each navigation step proposes to the user a more
refined query answer. For example, file systems can be considered as an infor-
mation retrieval system where data is organized in a static tree structure. A new
approach of information retrieval, both by request and by navigation in a Galois
lattice structure [9], has been proposed in [10, 11].
The concept lattice is a rich and flexible navigation structure automatically
derived from data, and can therefore be considered as a dynamic and complete
space search enabling data description while preserving its diversity. Querying
and navigation can be freely combined: to each user request corresponds a con-
cept of the lattice as answer ; the user can then improve its search either by
amending its request, or by on-line browsing around the concept in the lattice
structure. Such an approach was already proposed, for example in [10] with the
logical information systems (LIS) and has been implemented in Sewelis [4].
Sewelis is used to load the comics’ ontology and to create a bound between
the model and a concept lattice. The objects of the lattice match the classes of
the model, the attributes are their properties and each concept stands for a set
of classes’ instances sharing the same properties. It is then possible to navigate
all the way to any concept, using the flexible navigation structure provided by
the concept lattice.
2.4 Application
Let us illustrate this with a simple example. Let say we have a query panel and
we want to retrieve the strip it is coming from. While it only takes a quick look
to a human being to find the answer, it is not something obvious for a machine,
the strip concept not even being part of the model. Classical CBIR methods,
based on a similar visual features criterion, are helpless in that case. However,
if the knowledge related to the panels and their inside/outside relatives is used,
it becomes possible to return results that can be justified by the system and
iteratively refined with the relevance feedback brought by the user. Concerning
this request, the page number of the panel will first be considered (outside panel’s
relation) in order to focus on panels coming from the same page. Then, the y-axis
value of its centroid will be selected and only panel’s whose centroid corresponds
to the same y-value, within a predefined delta, will be kept. Finally, the hasNext
[5] relation can be used to sort output panels in order to rebuild the strip.
3 Conclusion and perspectives
This paper has presented an ongoing work about a Semantic Content Based
Image Retrieval system applied to comic books. The final aim would be to pro-
vide a complete system that would be able to (1) retrieve resources similar to a
query, based on the amount of mutual properties they share and the significance
of these properties guided by the user relevance feedback, and (2) explain to the
user why a returned resource is considered to be relevant to the query.
References
1. R.C. Veltkamp. Content-Based Image Retrieval System: A Survey. In Technical
report, University of Utrecht, 2002.
2. M.E.J. Wood, N.W. Campbell, and B.T. Thomas. Iterative refinement by relevance
feedback in content-based digital image retrieval. In ACM Multimedia 98, pages
13–20, September 1998.
3. B. Ganter and R. Wille. Formal concept analysis, Mathematical foundations.
Springer Verlag, Berlin, 284 pages, 1999.
4. S. Ferré and A. Hermann. Semantic search: reconciling expressive querying and
exploratory search. In Proceedings of the ISWC’11, 2011.
5. N. Tsopze, C. Guérin, K. Bertet, and A. Revel. Ontologies et relations spatiales
dans la lecture d’une bande dessinée. In IC, pages 175–182, Paris, 2012.
6. P. Stanchev, D. Green Jr, and B. Dimitrov. High level color similarity retrieval.
2003.
7. Y. Liu, D. Zhang, G. Lu, and W.Y. Ma. Region-based image retrieval with per-
ceptual colors. Advances in Multimedia Information Processing-PCM 2004, pages
931–938, 2005.
8. V. Mezaris, I. Kompatsiaris, and M.G. Strintzis. An ontology approach to object-
based image retrieval. In Image Processing, 2003. ICIP 2003. Proceedings. 2003
International Conference on, volume 2, pages II–511. IEEE, 2003.
9. G. Birkhoff. Lattice theory, volume 25. American Mathematical Society, 418 pages,
third edition, 1967.
10. S. Ferré and O. Ridoux. An introduction to logical information systems. Informa-
tion Processing & Management, 40(3):383–419, 2004.
11. C. Carpineto and G. Romano. Concept data analysis. Wiley Online Library, 2004.