The present work addresses the challenge of integrating lowlevel information with high-level knowledge (known as semantic gap) that exists in content-based image retrieval by introducing an approach to describe images by means of spatial relations. The proposed approach is called Image Retrieval using Region Analysis (IRRA) and relies on decomposing images into pairs of objects. This method generates a representation composed of n triples, each one containing: a noun, a preposition and, another noun. This representation paves the way to enable image retrieval based on spatial relations. Results for an indoor/outdoor classifier shows that neural networks alone are capable of achieving 88% in precision and recall, but when combined with ontology this result increases in 10 percentage points, reaching 98% of precision and recall.
In this work we investigate the application of spatial relations in the image retrieval problem. The issue of representing the semantics existent in an image has been receiving great attention recently. Numerical methods (low level) are not able to fully integrate semantics (high level) due to the fact that the semantic content might be constituted by qualitative concepts. The challenge of integrating low-level information with high-level knowledge is a known problem in computer vision, often referred as the semantic gap. In this work, a multi-level approach, called Image Retrieval using Region Analysis (IRRA), is proposed to retrieve images by their semantics from a bottom-up knowledge representation procedure. The method proposed here ensembles a stack of distinct neural networks in order to estimate spatial relations, expressed by a spatial preposition between pairs of objects. This procedure permits a representation of an image by the objects depicted and the relations holding between them. Thus, a sparse representation is constructed taking into account these objects and their relations. The images are indexed based on this sparse representation in order to enable fast retrieval. Finally, we extend a public ontology with this data in order to infer new relations beyond the original binary relations. This representation enables the retrieval of images based on queries with respect to spatial arrangements.
To semantically interpret an image it is necessary to:
define the context, detect objects, define some similarity metric
and, finally, apply some method for knowledge
representation
The quantitative analysis tackles the semantic gap problem using a hierarchy of classifiers. We represent the semantics building a top-down approach which contains a specific classifier for each of the following tasks: scene recognition, object segmentation and preposition estimation. The proposed approach decomposes an image into scenes and then, for each scene, it segments the related objects; given a pair of these objects, the method estimates a spatial preposition.
The qualitative analysis is built on top of Suggested
Upper Merged Ontology (SUMO)1
For evaluation purposes, the method proposed is applied to image retrieval tasks. The results obtained show that our method outperforms recent approaches aiming at the retrieval of images by means of spatial relations. Results also show that the classification task is much improved with a combination of neural networks (working at low-level information) with an object ontology (representing high-level knowledge) in contrast to using the neural network classifiers alone.
2
One of the most common approaches for retrieving images
is the paradigm known as query by example. Bag of Visual
Words (BOVW) is a method widely used to perform
image retrieval in this context. BOVW extracts local features
and, with respect to a sparse representation, performs
image retrieval
One possible way to represent the semantics expressed
in images is by following the steps described in
In order to establish a spatial relation, it is necessary to
define a reference system. For instance, considering the
relation x is in front of y, three concepts should be defined: the
target object, the reference object and the reference system
There are distinct definitions of ontology found in the
literature
3
In this work we address the problem of the semantic gap in image retrieval by representing an image by the spatial relation among the objects existing in the image itself. Our aim is to combine information obtained from the pixel level (quantitative analysis) with information provided by specialists (qualitative analysis), in this work the specialist knowledge is defined in terms of spatial prepositions in a general ontology. The method proposed in this work is called Image Retrieval using Region Analysis (IRRA) and is summarized in Figure 1. Considering an input image, IRRA applies a neural network in order to segment objects in the image, that are further combined into pairs of objects. In the sequence, for each pair, a preposition between them is estimated. Thus, we decompose the image into n-triples containing pairs of objects and a spatial preposition. These n-triples could, thus, be used to represent (and reason about) the domain. Section 3.1 describes the quantitative analysis of IRRA in more detail. 4. Estimate a spatial relation for every detected object pair. In order to perform all these tasks, a stack of neural networks is applied. We apply three distinct neural networks to hierarchically detect context, objects and spatial relations.
Context identification: The first neural network is
applied in order to estimate context. In this work, context is
understood as the scene class an image belongs to. Two scene
classes are used in this work: indoor and outdoor. We
created a model whose architecture is identical to that described
in AlexNet
Object segmentation: Object (semantic) segmentation is the second step to represent high-level information in an image. This task is challenging since we want to classify an image pixel-wise; besides, object identification is dependent on the context that the objects are immersed in. The neural networks that are performing semantic segmentation in this work have the same architecture, although each one has its own set of weights. The main difference between them is the target classes that they use to construct the model.
The segmentation neural networks generate a set of class proposals for each pixel in the image. Additionally, similar pixels are grouped together in order to represent objects. In this work we focus on assigning spatial relations between pairs of these segmented objects. In order to define a spatial region, we map pairs of detected objects aiming at establishing a representation that might be expressed in topological terms. Then, another neural network is used to estimate a spatial preposition from the previous (topological) classification. Each image Ix generates a CIx = n2Ix , where nIx is the number of objects detected for image Ix, and CIx is the combination of nIx in pairs.
Preposition estimation: A sparse representation of CIx is created and is provided as input to a neural network whose task is to assign English prepositions to images. It is important to notice that object information is crucial at this point to disambiguate multiple (possible) assignments. Besides, this sparse representation provides a notion of the spatial topology with respect to a pair of objects.
Finally, this sparse vector combined with the estimated
spatial preposition enables the creation of an index that
can be accessed with a structured query in order to
retrieve relevant images. This index is similar to those used
by BOVW
Knowledge representation is built with respect to the spatial
arrangement of the identified structures in images. The
interpretation of spatial relations contributes to reducing the
semantic gap in images since relations tend to suffer less
with variations than the objects in their arguments
Aiming at representing the knowledge acquired through
the numerical analysis of images, we have opted to extend
an existing ontology: SUMO
The initial step to represent knowledge was to create instances of the domain. In this work, each detected object is considered as an unique and independent instance.
Considering an example image Ix with the detected objects Building, Floor and Sky, in this work, each of these terms is identified by the suffix corresponding to the original image, in this case x. Therefore the term Building is referenced by Buildingx and so on. Formula 1 shows the instance definition. ( i n s t a n c e Buildingx B u i l d i n g ) ( i n s t a n c e F loorx F l o o r ) ( i n s t a n c e Skyx Sky )
All the images and the detected objects are represented in a similar fashion. The next step of our framework is to construct spatial relations between the distinct objects. To execute the mapping of these relations to the segmented pair of objects, we have considered the prepositions estimated by a statistical classifier. Finally, our preposition domain contains the following relations: above, across from, behind, below, in, in front of, inside of, left of, on, right of and under.
By using an off-the-shelf ontology, the representation of these relations expressed by prepositions was simplified. In this work, only a punctual extension of SUMO was necessary.
The standard SUMO ontology provides tools to define spatial relations. In order to define a spatial relation in SUMO it is necessary to create an instance of the class PositionalAttribute. This class enables the stating of binary orientation relations between two objects. Additionally, it is important to mention that the semantics with respect to spatial relations might be expressed by more than one preposition, besides, distinct prepositions might be similar or complementary. For instance, two distinct objects arranged consecutively might be referenced by the following prepositions : in front of or behind, the proper term is defined according to the context and the observer’s position. This characteristic allows us to represent both relations using the double implication operator <=>. Consequently, it is possible to define the preposition in front of based on the preposition behind, or vice-versa. This definition is illustrated in Formula 2. It is worth pointing out that, although in front of or behind are both relative to an observer, this paper assumes that observer is the camera point of view and, thus, it is implicit in the definitions. (<=> ( o r i e n t a t i o n ?X1 ?X2 B e h i n d ) ( o r i e n t a t i o n ?X2 ?X1 I n F r o n t O f ) ) Formula 2: Double implication to behind and In front of According to the nature of spatial prepositions, this procedure is applied to other relations, for instance: under and above or left of and right of. Additionally, the transitivity of relations was also used in this context whenever possible. For instance, we might infer that if there is an object a above an object b and b is above a third object c, therefore a is above c.
In conclusion, through quantitative methods, qualitative information with respect to the domain was inferred. The impact of this procedure was evaluated in the tests described in the next section.
4
This section details the experiments executed in order to
evaluate the proposed method. To perform the experiments,
the publicly available data set SUN09
Overall 4,367 images were used for training and 4,317 images for testing. In these datasets there are 186,299 pairs of objects for training and 173,111 for testing, being 106 distinct objects considering data set 1 and 42 considering data set 2. The scene classification part of the present proposal (identifying what we use as context: indoor or outdoor scenes) was evaluated using data set 2, with manual annotations for training and evaluation purposes. The results of this binary classification are shown in Table 1, where we can see that the overall precision and recall for each of the considered classes was 87%, attesting for the suitability of the method applied for this task.
As described above, scene identification provides information to refine the object segmentation procedure, whose results are shown below. In this part of the system evaluation, we investigate the hypothesis of whether the information provided by the scene identification (Section 4.1) improves the segmentation. According to this premise we have manually separated the objects as indoor and outdoor. The indoor objects are: armchair, basket, bookcase, book, bottle, box, chair, closet, cupboard, curtain, desk, floor, flower, ground, mirror, plant, poster, refrigerator, seats, table, vase, wall and window. The outdoor objects are: airplane, balcony, bench, building, car, door, fence, gate, grass, path, road, rock, sign, sky, streetlight, tree, van, water. According to each scene class detected we apply one or the other segmentation models (i.e. one trained with indoor objects or the other trained with outdoor objects).
The semantic segmentation model applied in this
experiment was fine tunned with the weights provided by
IoU =
AS \ AL AS [ AL (1) thus, when the neural network segments exactly the same area as the label (AS = AL), IoU = 1 and when the segmented area AS is completely different from the label area AL (AS \ AS = ;), then IoU = 0. For any situation when the segmented area overlaps with the labelled area (AS \ AS 6= ;) but are not equal, 0 < IoU < 1.
Results show that the classification using context information had a higher IoU value overall (shown in line “Average” in Table 2). However, there were various cases in which the use of scene class information did not improve the results (such as the results related to Armchair, or Bench), this occurred due to the fact that such objects appear in both (indoor and outdoor) scenes. Thus, in these cases, assuming scene classes caused a reduction in the number of examples of some objects during the training phase. 4.3
With the object-segmentation neural network presented in the previous section, a second indoor/outdoor classifier was tested, in which we combine low-level information of the scene with high-level information provided by the description of the objects in SUMO.
For this classifier, each of the objects presented in Section 4.2 was described in SUMO as having one of three possible classes: indoor (e.g., armchair), outdoor (e.g., sky) or both (for objects that are both indoor and outdoor, e.g., chair). The complete list of object classes and the corresponding SUMO annotations are shown in Table 3. With the same segmentation method as used in Section 4.2, the objects recognised in each of the scenes were counted and the class with the highest amount of objects present was considered to be the scene class (e.g., a scene with 2 indoor objects, 1 outdoor and 1 both is considered to be an indoor scene). Scenes where there is an equal number of indoor and outdoor objects were classified as both. In this case, the object rank was used in order to take into account the relevance of the object with respect to the scene classification. For instance, a scene containing an object “sky” is considered to be and outdoor scene, since there can be no sky indoors2). We considered that the following outdoor objects outrank any other object in the domain: building, sky, sign, fence, grass, road.
Results obtained for this combination of low-level and high-level classification are shown in Table 4 where we obtained over 98% of precision and recall. An improvement of 10 percentage points with respect to the classifier without the ontology (whose results are shown in Table 1).
From 4,268 scenes used in this experiment 44 could not be classified into any of the three classes available, since they did not contain any object, or contained only one object recognised in the class both. These were not accounted in Table 4. 4.4
The main difficulty of assigning a preposition to a pair of objects in an image is the common overlapping of terms, i.e., there are several possible (consistent) preposition assignments to each spatial relation. In order to cope with this issue, before assigning a preposition, the topological relation between pairs of objects is classified, serving as a bridge to preposition definition. In our experiments, each image in the data set was transformed into a representation containing: (target object, reference object). This representation is shown in Figure 2, Figure 2a presents the original image and the object mask is presented in Figure 2b. It is important to mention the fact that the color of the objects indicates the target object (blue) and the reference object (red).
Each image available in the data set was segmented and combined with its relative object, generating a combination of CIx = N2Ix for images I x and segmented objects NIx for the image. Every image generated from CIx was classified according to the spatial preposition related to the reference and target objects. In this test we have used data set 2 with 11 spatial prepositions.
2In this experiment, a window is recognised as a single object, therefore no other object can be perceived within its contours.
Table 5 shows the precision, recall and f-measure for each of the tested prepositions, where the right-most column is the number of tested samples. The overall values are exhibited in the last line: for the 4,953 relations tested, the system reached an overall precision of 0:75 with a recall of 0:75 and a f-measure of 0:71.
Next section compares the performance of IRRA with other state-of-the-art methods in the task of image retrieval from structured queries. 4.5
In order to evaluate image retrieval using IRRA, the
annotations provided by
Figure 3 illustrates the obtained results and also displays
a comparison with other approaches using the same data set.
Only recall is presented since this is the measurement used
in
Analyzing IRRA results alone, we can see that in larger queries that do not include single objects (in scenarios Sc and Se, for instance), IRRA’s performance is not as good as in other scenarios. This behavior occurs due the fact that, when answering a query such as Sc or Se, segmentation or preposition detection errors are propagated to the retrieval task.
We have also evaluated the retrieval by using the second
data set which has a larger set of prepositions. In this
setting IRRA achieved a retrieval with mean average precision
(mAP) of 53:95, outperforming the recent results reported
in
The queries executed in the tests above were strictly unidirectional due to the fact that there is no high-level reasoning with respect to the spatial arrangement of the items in the document collection. To address this issue, we investigated the application of reasoning using SUMO.
We extended the annotations in
In order to evaluate this method, distinct queries were proposed, but keeping the same information to be retrieved from the set of images. To accomplish this task inverse relations were applied. For instance, for two objects (x, y) and the query (x-above-y), we also evaluated the retrieval for (ybelow-x) against the same annotation for above as used in the original query.
The instanciation of the whole ontology with SUN09 data
Query B
l
l
a
c
e
R
60
40
20
0
ti)2014 )a2012 .)a2014 IRRA
z .l l
rF te te
nd an ie
i (L iqu
a
sknow i(ddS
i
l
a
M
(
)201 )401 RRA
2 2 I
. .
l l
a a
t t
e e
an ie
(L iqu
d
d
i
S
(
)021 )041 RRA
2 2 I
. .
l l
a a
t t
e e
an ie
(L iqu
d
d
i
S
(
)012 )014 RRA
2 2 I
. .
l l
a a
t t
e e
an ie
(L iqu
d
d
i
S
(
)120 )140 RRA
2 2 I
. .
l l
a a
t t
e e
an ie
(L iqu
d
d
i
S
(
generated more than 13; 000 terms with respect to the
images and objects, and more than 18; 000 formulas
referencing the created relations. The retrieval task based on the new
set of queries was performed by evaluating every image
using the E first-order logic theorem prover
Every image was tested using the new annotated queries,
the mAp achieved in this case was 41:60, whereas he
original set for these prepositions obtained a mAp of 51. The
decrease in performance observed with the extended set of
relations (in contrast with the original) was due to the fact
that, by increasing the size of the knowledge base, errors
were possibly included in the process, whose detection
becomes increasingly complex to perform
This paper proposed a framework for image retrieval based on an ensemble of neural networks and spatial relations defined over an ontology. The proposed method decomposes the image in distinct levels in order to classify objects and their spatial relations in static scenes. The retrieval of an image is than done by specifying the existence of objects and spatial relations between two objects (e.g., “car on road, pedestrians” is a possible sentence to use to retrieve an image).
Although the number of objects and relations seems to be small (42 objects and 11 relations), the number of images used is over 9,000 and since a scene is divided in relations between objects in it, it can be used for training more than one relation and/or terms, thus over 186,000 pairs of objects in training and 173,000 for testing were generated. This resulted in more than 13,000 terms and 18,000 formulas for the ontology.
Results on preposition classification also show that the method proposed in this paper outperforms previous work in the retrieval of images using spatial relations, however, it did not perform as well as expected in the following cases: across from, inside of, left of, right of. This problem was probably due to the distinct competing prepositions that could be equally applied to the scenes in these cases. A deeper investigation of how to represent the object pairs in order to enhance the estimation of their spatial preposition is a task for future work. It is also within our future interests to augment the set of relations covered by the system, relaxing the present restriction to binary relations.
When classifying an image as an indoor or outdoor scene using ontology, it is possible to specify an objects class, e.g., (subclass Sky Outdoor) or describe it in relation to another object, e.g., (subclass Armchair Chair), making it easier to add new objects to the classifier and to reason about their properties than it is to train a new deep learning method to classify new objects in the database. As a result, an increase of 10 percentage points was observed in the results when using the ontology to classify object’s class found in the outputs of the neural networks, suggesting a successful combination of a knowledge representation tool with a state-ofthe-art machine learning algorithm with virtually no learning or classification running time increase.
6
This work investigated the semantic gap that exists in content-based image retrieval by introducing an approach that establishes relations between objects in images by means of spatial arrangements. The method proposed in this paper, called Image Retrieval using Region Analysis (IRRA), starts by decomposing images with respect to pairs of objects, where each pair is also combined with a spatial relation. Each spatial relation is related to a spatial preposition expressed in natural language. IRRA was evaluated on a public data set, whose results show that our approach outperforms previous (recent) work in the retrieval of images using spatial relations.
Results showed that by combining SUMO’s high-level description of objects with the output of a machine learning classifier, it is possible to increase in 10 percentage points the precision and recall of such classifier when the scene classification is uncertain. Although this increase is achieved with almost no increase in running time, there as still some scenes that could not be classified for the lack of context regarding the objects found in the image.
We believe that the proposed framework has two
compelling applications. The first is to improve statistical
classifiers, following an approach similar to
Danilo Nunes was partially supported by CAPES (grant 5131024). Leonardo Anjoletto Ferreira acknowledges that this study was financed in part by the Coordenac¸a˜o de Aperfeic¸oamento de Pessoal de N´ıvel Superior – Brasil (CAPES) – Finance Code 001 and also from FAPESP-IBM (grant 2016/18792-9). Paulo Santos acknowledges support from FAPESP-IBM (grant 2016/18792-9).