This paper describes the participation of the CIII UTN FRC team in the ImageCLEF 2012 Robot Vision Challenge. The challenge was focused on the problem of visual place classification in indoor environments. During the competition, participants were asked to classify images according to the room in which they were acquired, using the information provided by RGB and depth images only. We based our approach on the Fisher Vector representation -a robust signature recently proposed in the literature- and the use of efficient linear classifiers. In order to exploit the information provided by different information channels, we adopted a simple fusion strategy and generated classification scores for each image in the sequence. Two tasks were proposed during the competition: in the first, images had to be classified independently of one another while, in the second, it was possible to exploit the temporal continuity of the stream. For the first task, we adopted a simple threshold based classification scheme. For the second, we considered the classification of groups of images instead of single frames. These groups, i.e. temporal segments, were automatically generated based on the visual similarity of the images in the sequence. Our team ranked first on both tasks, showing the effectiveness of the proposed schemes.
This paper describes the participation of the CIII UTN FRC team in both
tasks. Our methods leverage recent advances in the fields of image classification
and retrieval, in which robust and efficient representations have been devised.
Particularly, we consider the state-of-the-art Fisher Vector (FV) representation
[
Before introducing the core components in our system, we first highlight some of the differences between the problem of visual place classification (VPC) in robotics and the more general problem of automatic image annotation (AIA)3, i.e. the problem of assigning labels to images based on its content. First, the labeling of images in VPC is based on the physical location of the robot instead of a visually well defined concept. This makes the labeling of training images in some cases ambiguous, as images acquired at a particular location might reflect a different visual concept than the one assigned to them (e.g. last images in the sequence of Fig. 1). Second, images acquired for VPC exhibit a great degree of redundancy due to the temporal continuity of the image stream, i.e. labels associated to images acquired close in time are likely to belong to the same concept and share similar appearance. These peculiarities, originated in the very definition of the problem, make the visual classification of places a very challenging task.
The paper is organized as follows. In Sec. 2 we give a high level description of the different stages in our system. In Sec. 3, 4 and 5 we describe in detail the representation we use as well as the different classification schemes we applied. In Sec. 6 we present our experimental setup and in Sec. 7 we show results using the training set provided by the organizers of the challenge. Finally, in Sec. 8 we draw some conclusions. 2
In this section we describe the core components of our system. The methods we applied in solving both the obligatory and optional tasks comprise the following processing steps: 3 The problem is also known in the literature as image classification, categorization or tagging. – Encoding : images must be robustly represented in order to capture high level properties of the scene. We rely on the state-of-the-art Fisher Vector image signature. As far as we know, this is the first time such a representation is applied in robotics. – Scoring : we generate, for each image and concept, a score that provide us with a measure on how likely is for an image to have been acquired at a particular location. We use simple linear classifiers, which are efficient both to train and to evaluate. – Classification: based on the scores obtained in the previous step, we generate a prediction of the robot actual location. We consider two cases: 1. Individual frames are classified without taking into account the temporal consistency of the stream. We treat this problem as a simple (baseline) image classification task using FV and efficient linear classifiers. 2. The image stream is automatically segmented into visually similar groups of images and the classification is performed in a segment-by-segment basis. We propose an efficient temporal segmentation algorithm based on representation properties of the FV signature. 3
We provide a brief overview of the representation in which our method is based,
namely, the Fisher vector image signature. Further details can be found in [
Let uλ : RD → R+ be a pdf of parameter vector λ modelling the generation process of low-level descriptors in any image. Let X = {xn, n = 1, · · · , N } be a iid sample of such D-dimensional descriptors extracted from a given image. The Fisher vector is defined as Here, GλX denotes the gradient of the (average) log-likelihood of X under uλ:
GλX = LλGλX .
GλX = 1 XN ∇λ log uλ(xn) ,
N n=1 and Lλ is a diagonal normalizer. We model uλ as a mixture of M Gaussians with diagonal covariances, i.e. uλ(x) = PiM=1 wiui(x), parametrized in terms of λ = {wi, μi, σi, i = 1, · · · , M }. Here, wi, μi and σi2 denote, respectively, the mixing weight, mean and variance vector corresponding to the ith component of (1) (2) with γn(i) representing the soft assignment of low-level descriptors to components of the mixture, i.e. γn(i) = wiui(xn) PjM=1 wj uj (xn). The image signature is the concatenation of partial terms5, i.e.
GX =
GμX1 T , · · · , GμXM T , GσX1 T , · · · , GσXM T T ,
resulting in a vector of dimensionality E = 2M D. Following [
X by 1. In what follows, we use Gnorm to denote the transformed signature. 3.2
We extract two sets of low-level feature descriptors per image, computed independently from the luminance (Y) and depth (D) channels of each input frame. These sets of descriptors are used to compute two separate FVs that we de
Xdepth respectively. Note that these two FVs originate from
note by GnXolrumm and Gnorm
different probabilistic models, i.e. ulum and udepth. The parameters for these
λ λ
models can be estimated using the expectation maximization (EM) algorithm
and a large set of descriptors.
the mixture. It can be shown that [
GμXi = GσXi = 1 N
X γn(i) N √wi n=1
1 N √
N
X γn(i) 2wi n=1 " xn − μi σi xn − μi σi 2 Let us denote by C = {1, . . . , C} the set of concepts, i.e. locations defining the problem. For each channel (luminance and depth), we learn a set of C binary classifiers that provide us with a measure of how likely is for a given image to have been acquired on a particular location in the environment. Concretely, we generate a set of C linear predictors per channel, of the form:
scξ = θcT GnXoξrm + bcξ,
4 Vector divisions must be understood as term-by-term operations.
5 We consider the gradient w.r.t. the mean and variance vectors only as the gradient
w.r.t. the mixing weights has shown to provide little discriminative information [
In this section we describe our approach for robust place classification for both tasks of the challenge. 5.1
For the obligatory task, images have to be classified without considering the temporal continuity of the image stream, i.e. frame-by-frame. In cases of uncertainty, the system is allowed to refrain from making a decision (thus avoiding penalization points).
We treat this task as a simple (baseline) classification problem. Deciding to which class an image belongs was done according to the following rule: cˆ = (c = arg maxi si, if sc > α 0, otherwise (7) Here, 0 denotes “no classification”, i.e. the image is left unclassified. The parameter α is set empirically by cross-validation. 5.2
For this task, participants were allowed to exploit the temporal continuity of the image stream. This task also characterizes by the presence of kidnappings: situations in which the robot abruptly changes its location from one room to another.
Before introducing the methods we applied for this task, let us first introduce some notation. Let I1:T = {It, t = 1 . . . T }, It ∈ I, be a sequence of images acquired by a mobile robot up to time T . With a slight abuse of notation, we denote by sc(t) the classification score computed for image It and class c ∈ C. We define the score vector s(t) := (s1(t), . . . , sC (t))T . Similarly, we use the notation
Xξ Gnorm(t) to represent the FV computed for image It using the set of descriptors extracted from channel ξ ∈ {lum, depth}.
Temporal Segmentation. Let us assume that for every t and t′ there exists a function m(t, t′) that provide us with a measure of the similarity between images It and It′ . Given a reference image Ia, we define a temporal segment as the sequence Ia:b = {It, t = a, . . . , b}, where b is the greatest integer such that Algorithm 1 Temporal segmentation and classification t0 ← 1 S ← {s(1)} for t = 2, . . . do if m(t0, t) < m0 then
Classify images in It0:t−1 according to MCT or MVT using the scores in S t0 ← t
S ← {s(t)} else
S ← S ∪ {s(t)} end if end for m(a, b) ≥ m0 and m0 a free parameter. Reference images are selected as the first image that follows a previously computed segment. The first of such reference frames is chosen as the first image in the sequence. The classification of images is performed segment-by-segment instead of frame-by-frame. Algorithm 1 provides an overview of our approach for temporal segmentation and classification.
As a proxy for m(t, t′) we use the dot product between FVs computed from either the luminance or depth channel features of It and It′ (Sec. 3.1). This provides us with a measure consistent with the classification model. It holds that |m(t, t′)| ≤ 1.
Using the above procedure, dealing with kidnapping situations becomes rather natural since, in such cases, the visual appearance of images is likely to change considerably from one frame to another. This abrupt change in appearance will trigger the generation of a new reference image and the classification of the segment extracted just before the kidnap point.
Segment classification. Based on the above definition, we classify all images in the segment Ia:b according to one of the following rules: a) Maximum confidence and threshold (MCT), which takes into account the confidence of the classifiers w.r.t. the best alternative hypothesis within a given segment; or b) Majority vote and threshold (MVT), which tries to exploit the temporal and semantic consistency of the images. Details for these rules are given next. Maximum confidence and threshold (MCT). Let c1, c2 ∈ {1, · · · , C} denote the indices to the best and second best scoring classifiers at time t and let d(t) := sc1 (t) − sc2 (t) ≥ 0 denote the difference between the corresponding scores. All images in the segment Ia:b are classified as belonging to class cˆ according to the following rule: cˆ = (c1, if sc1 (u) > β, u = arg maxt∈[a,b] d(t) (8) Majority vote and threshold (MVT). Let va:b(c) := #{sc(t) > β, t = a, . . . , b} denote the number of times the classification score for the cth classifier is above β for images in the segment Ia:b. We consider the following voting strategy for the classification of the temporal segment Ia:b: cˆ = arg max va:b(c) c (9)
As before, cˆ = 0 means that images in Ia:b are left unclassified. In both cases, the parameter β is set empirically by cross-validation. 6
In this section we provide a detailed explanation of our experimental procedure. 6.1
The training set for the Robot Vision Challenge 2012 consists of three sequences of 2667, 2532 and 1913 RGBD images respectively, acquired under different illumination conditions within the same floor of an office environment. They include motions in both clockwise and counter clockwise directions. Performance is measured based on the number of correctly and misclassified images in a given sequence and it varies from task to task.
Further details regarding the dataset and the evaluation methodology can be
readily found in [
Both RGB and depth images were reduced at half their original resolution
before computations. We extracted 128-dimensional SIFT descriptors [
For each channel, we trained a Gaussian Mixture Model (GMM) with M components under a Maximum Likelihood (ML) criterion using the ExpectationMaximization (EM) algorithm. We used 106 random samples from the training set. We initialized the EM iterations by running k-means and using the statistics of cluster assignments (relative count, mean and variance vectors) as initial estimates. 6.4
As base classifiers we used linear SVMs trained on the primal using Stochastic
Gradient Descent (SGD) [
In order to allow the system to cope with changes in illumination and the robot motion direction, we considered the following data augmentation strategies: i) adding new images by simulating uniform changes in illumination, i.e. generating darker/brighter versions of randomly sampled images; ii) generating mirrored (left-to-right) versions of the images provided as training material. In the first case, we did not observe any noticeable improvement while, in the second, we observed an increase of +30% (on average) w.r.t. a system trained using the original data only. This is to be expected, as our low-level features (i.e. SIFT vectors) are based on gradient information which makes them insensitive to uniform changes in the illumination. On the other hand, adding mirrored samples to the training set let the system learn up to some degree the symmetries originated from the changes in the robot motion. 7.1
In this subsection we evaluate the performance of our system in classifying images independently (frame-by-frame). In particular, we consider the following aspects: i) the impact of using increasingly complex models (i.e. number of Gaussians, M ); ii) the benefits of using different representation channels. For each configuration, we ran three experiments using different train/test splits of the data, using two of the sequences for training and the third for testing. Results are reported on Table 1. We show recognition performance6 for models with M = 256, 512 and 1024 Gaussians and systems based on single and multiple descriptor channels. Results on Table 1 were obtained by setting α = −∞ in Eq. (7), i.e. argmax rule without thresholding. The classification performance obtained with α = −0.5 is shown in parentheses.
If we consider the different train/test configurations, it can be observed a big drop in performance for the system trained on sequences 1 and 2. This drop can 6 The score was computed using the scripts provided by the organizers of the challenge. be explained by noting that sequence 3 was acquired under very poor illumination. A system trained using only images acquired under “normal” illumination does not generalizes well to this previously unseen scenario. In contrast, systems to which this sequence was shown during training exhibit much better performance (second and third row in the table). As expected, the system based on the luminance channel alone performs worse than the system using depth information when testing on sequence 3. On the contrary, the luminance channel shows better performance on test sequences 1 and 2. The combination of both channels brings large improvements in all scenarios. Increasing the model complexity (number of components in the mixture) can bring additional improvements at the cost of a greater computational cost.
The system we submitted during the challenge included both luminance and depth features and models with M = 1024 Gaussians. The threshold parameter was set to α = −0.5. Our system ranked first, achieving a score of 2071 points. 7.2
For this task, we first evaluate the influence of the parameter m0 in classification performance (Sec. 5.2, temporal segmentation). m0 controls the degree to which an image is considered similar to another of reference, i.e if it belongs to the temporal segment defined by the second. Fig. 2 (left) shows the average score as a function of m0 for different choices of the similarity measure (dotproduct between luminance or depth FVs) and classification rules (Sec. 5.2), e.g. Lum+MVT corresponds to the system using the dot-product between luminance FVs for segmentation and the MVT rule for classification. Fig. 2 (right) show the average length of the temporal segments obtained by using FVs from either channel.
It can be observed that for m0 above 0.2, using luminance FVs for segmentation leads to better results than with depth FVs. Using luminance FVs, performance reaches a peak at m0 = 0.3 (Lum+MCT: 1836, Lum+MVT: 1871). Within this range, MVT performs better than MCT. For values of m0 < 0.2, the segmentation using depth FVs leads to better results. In this case, a peak is observed at m0 = 0.1 (Depth+MCT: 1919, Lum+MVT: 1888) with MCT performing better. As a comparison, the average score obtained by setting m0 = 1
Lum+MCT Lum+MVT Depth+MCT Depth+MVT Lum
Depth 5 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 m0 m0 (frame-by-frame classification) on this task is 1801. The average segment length at the above points is 5.4 and 6.86 for luminance (m0 = 0.3) and depth (m0 = 0.1) features, respectively. For the same value of m0, luminance FVs lead to larger segments.
For the competition we submitted two systems: Depth+MCT and Depth+MVT using, as before, 1024 Gaussians. We set the segmentation and classification thresholds to m0 = 0.1 and β = −0.4, respectively. Our systems ranked first, achieving 3930 points on this task. 7.3
Finally, we report computation times for the system based on models with M = 1024 mixture components. Reported times were measured on a AMD Opteron machine (8 cores @ 2GHz) with 8 GB of RAM. Table 2 show estimated times for both offline and online processes. Additionally, we report FV computation times for M = 256 and 512 Gaussians.
This paper describes the CIII participation at ImageCLEF Robot Vision Challenge 2012. Our approach leverages recent advances in the fields of image classification and retrieval. We proposed a temporal segmentation methodology based on the visual similarity of images that allowed us to classify groups of images in a robust manner. Our team ranked first on both the obligatory and optional tasks of the challenge, showing the potentiality and effectiveness of our approach. Acknowledgments. Jorge S´anchez was partially supported by a grant from CONICET, Argentina.