In this paper, we present an ongoing work aiming to improve content based image retrieval performance with the help of logical concept analysis. Domain semantic is formalized and used instead of classical CBIR visual features. This is being applied to comicbooks using Sewelis.
Web search engines usually give poor results when searching in multimedia
databases since they use the contextual web page, or the meta information
attached to the multimedia objects. The semantic meaning that the user usually
attaches to the content of the document is often very di erent from the text used
for indexing the image (semantic gap). Content Based Image Retrieval (CBIR)
has been proposed to search into huge unstructured image databases by giving
an example of what the user is looking for instead of describing the concept it
represents. Classically, visual features are extracted from the images and then
compiled into a signature [
In this article, we are interested in CBIR in the context of comicbook databases.
In this case, databases cannot be considered as unstructured anymore since
images can be grouped in terms of panels, pages and volumes which are themselves
associated with metadata concerning the author or the series they belong to.
We would like to bene t both from the search facilities given by CBIR
techniques with feedback and semantic information embedded in the structure of the
comicbooks documents. To do such a thing, Logical Concept Analysis (LCA),
an extension of Formal Concept Analysis (FCA) [
PanelValidation
TextRegion - hasText - hasRank is_a is_a hasReference hasValidation
RegionOfInterest NextValidation - hasX
- hasY is_a is_a
Panel hasNext is_a
- hasRank Validation - isCorrect hasBalloon
Balloon
Semantic Content Based Image Retrieval
Comicbooks have a natural hierarchical structure that can be formalized. They
are made of pages which contain panels. These panels can eventually be
gathered in strips1 and contain di erent kind of objects, such as speech balloons,
characters, free text, etc. Balloons can be of many kinds (dialogue, thoughts
etc.).This knowledge can be used to deduce more information such as pieces of
the storyline. Fig. 1 illustrates the model we propose to formalize the comicbooks
domain. It has been described with more details in [
An input picture, picked from the database, being given, the system will not only be able to retrieve similar pictures based on the classical image characteristics (colors, shapes, textures...), but also based on the associated semantic and the knowledge that could have been learnt previously. Considering that the query is a Panel instance, the search can focus on:
(1) The panel's characteristics (i.e. data properties of a Panel object). This could be its rank, its shape, its size, its position, its shot type, its view angle, 1 A strip is de ned as an horizontal sequence of panels. Traditionally, a strip is made of 1 to 6 panels and a page can contain up to 4 stacked strips.
Comic - hasLabel - hasWriter - hasDrawer hasPart
Page - hasNumber hasPart hasExtractor hasNext -EhxatsrNacatmore is_a
is_a GroundTruth
Automatic hasImage Image - hasWidth - hasHeight etc. Images of a very close shot of a character's face or a landscape picture of a valley being at the top of a page can be examples of queries.
(2) The panel's relations (i.e. object properties). Properties of objects related to the query panel can be used as well as its own characteristics. Therefore, there are two directions to look at from a panel point of view.
{ The search can focus on what is inside the panel, like similar amount of objects in a scene (a dialogue between two characters for instance) or related text content. The retrieval process can also rely on objects contained in the panel, whether they are identi ed or not. Assume that the query picture contains an identi ed character A whose visual signature is de ned by the set of features X. The system will not only look for panels containing an instance of A, but also for those showing a spatial region matching X. { Outside: the search can focus on panels sharing page's or comic's characteristics (such as author, style, etc.)
These kinds of retrieval angles are not mutually exclusive and it is very possible to combine two or more of them in order to narrow the result set. The search possibilities are only limited by the completeness of the description.
In databases, information retrieval is classically performed by request queries
expressed in a speci c request language, as SQL for example. However, the more
re ned is the search, the more sophisticated is the request. Some information
retrieval systems o er a simpler search re nement by navigation in a prede ned
static data structure, where each navigation step proposes to the user a more
re ned query answer. For example, le systems can be considered as an
information retrieval system where data is organized in a static tree structure. A new
approach of information retrieval, both by request and by navigation in a Galois
lattice structure [
The concept lattice is a rich and exible navigation structure automatically
derived from data, and can therefore be considered as a dynamic and complete
space search enabling data description while preserving its diversity. Querying
and navigation can be freely combined: to each user request corresponds a
concept of the lattice as answer ; the user can then improve its search either by
amending its request, or by on-line browsing around the concept in the lattice
structure. Such an approach was already proposed, for example in [
Sewelis is used to load the comics' ontology and to create a bound between the model and a concept lattice. The objects of the lattice match the classes of the model, the attributes are their properties and each concept stands for a set of classes' instances sharing the same properties. It is then possible to navigate all the way to any concept, using the exible navigation structure provided by the concept lattice. 2.4
Let us illustrate this with a simple example. Let say we have a query panel and
we want to retrieve the strip it is coming from. While it only takes a quick look
to a human being to nd the answer, it is not something obvious for a machine,
the strip concept not even being part of the model. Classical CBIR methods,
based on a similar visual features criterion, are helpless in that case. However,
if the knowledge related to the panels and their inside/outside relatives is used,
it becomes possible to return results that can be justi ed by the system and
iteratively re ned with the relevance feedback brought by the user. Concerning
this request, the page number of the panel will rst be considered (outside panel's
relation) in order to focus on panels coming from the same page. Then, the y-axis
value of its centroid will be selected and only panel's whose centroid corresponds
to the same y-value, within a prede ned delta, will be kept. Finally, the hasNext
[
This paper has presented an ongoing work about a Semantic Content Based Image Retrieval system applied to comic books. The nal aim would be to provide a complete system that would be able to (1) retrieve resources similar to a query, based on the amount of mutual properties they share and the signi cance of these properties guided by the user relevance feedback, and (2) explain to the user why a returned resource is considered to be relevant to the query.