Many novel applications in the eld of object recognition and pose estimation have been built relying on local invariant features extracted from selected key points of the images. Such keypoints usually lie on high-contrast regions of the image, such as object edges. However, the visual saliency of the those regions is not considered by state-of-the art detection algorithms that assume the user is interested in the whole image. Moreover, the most common approaches discard all the color information by limiting their analysis to monochromatic versions of the input images. In this paper we present the experimental results of the application of a biologically-inspired visual attention model to the problem of local feature selection in landmark and object recognition tasks. The model uses color-information and restricts the matching between the images to the areas showing a strong saliency. The results show that the approach improves the accuracy of the classi er in the object recognition task and preserves a good accuracy in the landmark recognition task when a high percentage of visual features is ltered out. In both cases the reduction of the average numbers of local features result in high e ciency gains during the search phase that typically requires costly searches of candidate images for matches and geometric consistency checks.
Given an image as query, a Content-Based Image Retrieval (CBIR) system returns a list of images ranked according to their visual similarity with the query image. When queried, it extracts the same features from the query image and compare their values with those stored in the index, choosing the most similar images according to a speci ed similarity measure. Many CBIR systems support general visual similarity searches using global features such as color and edge histograms. The adoption of descriptions based on local features based (e.g., SIFT and SURF), from the computer vision eld, provided multimedia information systems with the possibility to build applications for di erent tasks, like, e.g., object recognition and pose estimation.
However, the number of local visual features extracted from cluttered, realworld images is usually in the order of thousands. When the number is `too' large, the overall performance of a CBIR system may decline. If too many features are extracted from `noise', i.e., regions that are not relevant, not only the CBIR becomes slower in its computations, but also its matching accuracy declines due to many false matches among the features. The reduction of the number of visual features used in the image descriptions can thus be considered a central point in reaching a good overall performance in a CBIR system. If only the keypoints extracted from relevant regions are kept a great improvement might be reached both in the timings and the accuracy of the system.
In this work we present an approach concerning the application of a biologicallyinspired visual attention model for ltering out part of the features in the images. The human visual system is endowed with attentional mechanisms able to select only those areas in the eld of view that are likely to contain relevant information. The basic assumption of our experimental work is that the user chooses the query image according to its most salient areas and expects the CBIR system to return images with a similar appearance in their salient areas. The model we implemented has a strong biological inspiration: it uses an image encoding that respects what is known about the early visual system by mimicking the biological processes producing the neural representation of the image formed by our brain. Since the biological inspiration does not bias the system towards speci c features, the approach can be used in generic image recognition tasks.
In order to assess quantitatively the performance of the visual attention model, we tested it on two tasks: a landmark recognition task and an object recognition task using two publicly available datasets. The results show that the ltering of the features based on the image saliency is able to drastically reduce the number of keypoints used by the system with an improvement or just a slightly decrease in the accuracy of the classi er in, respectively, the object recognition task and the landmark recognition task.
The rest of this paper is organized as follows. The next section brie y discusses the biological inspiration of our model. Section 4 describes the model of visual attention and its relationships with the biological facts introduced in section 3. Section 5 presents the datasets we used in the current experimentation and the results we obtained. The last section discusses the pros and the cons of our approach and brie y delineates some research lines we will follow in the future. 2
Visual attention has been used to accomplish di erent tasks in the context of Content Based Image Retrieval. For example, some works used attention as a mean to re-rank the images returned after a query. However, since our focus is on image ltering, we will restrict our analysis to two recent approaches that introduce an attentional mechanism for reducing the number of features used by a CBIR system with the goal of improving both its speed and its accuracy.
[Marques et al., 2007] proposes a segmentation method that exploits visual attention in order to select regions of interest in a CBIR dataset with the idea of using only those regions in the image similarity function. They use the saliency map produced by the Itti-Koch model [Itti et al., 1998] for the selection of the most salient points of an image. The selected points are then used for segmenting the image using a region growing approach. The segmentation algorithm is guided by the saliency computed by the Stentiford model of visual attention, whose output allows an easier and more precise segmentation than Itti-Koch's model. They experimented their methods on a dataset containing 110 images of road signs, red soda cans, and emergency triangles. Since that dataset is well known and used in other published experimentations, we used it in order to test our ltering approach.
[Gao and Yang, 2011] propose a method for ltering SIFT keypoints using saliency maps. The authors use two di erent algorithms for computing the image saliency, the Itti-Koch model (for local-contrast analysis) and a frequency-based method (for global-contrast analysis) that analyzes the Fourier spectrum of the image [Hou and Zhang, 2007]. The nal saliency, corresponding to the simple sum of the saliency maps computed by the two methods, is used to start a segmentation algorithm based on fuzzy growing. They experimented their method on a dataset composed by 10 classes with more than 10 images per class, extracted from the ALOI image dataset and the Caltech 256 photo gallery. The original images were modi ed using various transformations. The authors show that their method has a precision that is lower than standard SIFT and comparable to PCA-SIFT (a ltering approach based on Principal Component Analysis). Even if the accuracy is not improved by the ltering, their approach is much faster than the other two and is thus suitable for use in CBIR systems.
In this work we experiment and evaluate a model of visual attention both on the dataset described above and on a more complex dataset. The harder dataset contains a large number of real, cluttered photographies of monuments located in Pisa. The dataset contains pictures downloaded from Internet (e.g., ickr images) that have not undergone any modi cation. 3
When we open our eyes we see a colorful and meaningful three-dimensional world surrounding us. Such visual experience results from a sequence of transformations performed on the light stimuli that starts in our eyes. The light is focused on the retinal surface, then processed and transferred to our thalamus, and nally routed to the cerebral cortex. The initial transformation is accomplished by the retina, starting from the photoreceptors. Photoreceptors are connected to bipolar cells in the middle retinal layer, which are then connected to the third and nal layer (the retinal output), populated by ganglion cells. Ganglion cells have a structured receptive eld, i.e., they are connected to well-de ned areas in the retina and do not react to the simple presence of a light stimulus. In particular, ganglion cells have a receptive eld with a center{surround organization [Ku er, 1953]. For example, an on{center, o {surround ganglion cell reaches its maximum activity level when the light hits and lls the central part of the receptive- eld and no light stimuli are present in the surround area. The output of the ganglion cells reaches the striate cortex or area V1. In this layer, the cells start computing complex features; for example, V1 cells show preference for speci c orientations. Colors are the results of a complex processing that takes place at various stages in the processing pipeline described above. Cells described by the opponent process theory can be found as early as in the last retinal layers: it is possible to nd bipolar, ganglion (and later LGN cells) that have a preferred wavelength with a center-surround organization. In particular, there are (R+, G-) cells, excited by a red centre and inhibited by a green surround, and (G+, R-), (B+, Y-), (Y+, B-), where `Y' stands for yellow and `B' for blue. These cells, together with the achromatic channel composed by (Wh+, Bl-) and (Wh-, Bl+) cells (where `Wh' stands for White and `Bl' stands for Black), allow our visual system to represent million of colors by combining the activation patterns of the photoreceptors. Furthermore, this antagonism in color processing makes the visual system responsive to discontinuities, as edges, that are what best describe the shape of an object. 3.1
The visual stimuli we receive contain a overwhelming amount of visual information, that is simply too 'large' for our brain to process. Evolution has endowed humans with a series of lters able to reduce the large amount of the incoming information. A recent de nition of visual attention can be found in [Palmer, 1999]. Visual attention is de ned as those processes that enable an observer to recruit resources for processing selected aspects of the retinal image more fully than nonselected aspects. Evidence gathered in several psychological experiments shows that our attentional system can be roughly subdivided into two main components that operate very di erently and at di erent stages. The rst system, called preattentive, starts operating as soon as the light strikes the retinal photoreceptors. It processes basic visual features, like color, orientation, size or movements, in parallel and over the entire eld of view. This system is responsible of the visual pop-out e ect, i.e., the situations where an image area attracts our attention due to its di erences with the rest of the other image parts. The second system, called attentive, correspond to focused attention. When the target is not recognized by the preattentive system, the attentive processing starts and uses information computed by the preattentive system in order to select spatial regions that might contain the target object. It necessarily operates sequentially since it needs to focus several spatial regions looking for speci c object features.
According to the "Feature Integration Theory" [Treisman and Gelade, 1980] (FIT), the parallel, preattentive processes build an image representation with respect to a single feature and encode the information in feature maps (color, orientation, spatial frequency, . . . ). The maps are combined and the their peaks of activity are inspected guided by a global map that summarize the information computed in the various dimensions. One of the most in uential detailed models was proposed in [Koch and Ullman, 1985]. Such model is similar to FIT in the description of the preattentive and attentive stages, but proposes some intermediate structures able to give a plausible answer to the attentional shifts, both in visual pop-out and in conjunctive search. 4
In this experimentation we implemented a bottom-up model of Visual Attention that extends [Itti et al., 1998]. It is part of larger model that includes topdown attentional mechanisms for object learning and mechanisms. The model performs a multiresolution analysis of an input image and produces a saliency map assigning a weight to each image pixel (area) according to the computed saliency. The model is biologically-inspired: it encodes the image according to what is known about the retinal and early cortical processing and elaborates the channels with algorithms that resemble the biological processes, even if only at a functional level. Biologically-inspired models use a less sophisticated image encoding and processing than other approaches, but are not biased towards any speci c visual feature. Less general approaches, that focus on speci c features or measures for computing the saliency, are well suited for application domains characterized by a low variability in object appearance, but may fail when the content of the images is not restricted to any speci c category. The bottom-up model performs a multiresolution image analysis by using in each processing step a pyramidal representation of the input image. After encoding the input values using ve di erent channels for intensity and colors and four channels for the oriented features, it builds feature maps using a center-surround organization and computes the visual conspicuity of each level in every pyramid. For each level of the conspicuity pyramids, the model builds a local saliency map that shows the saliency of the image areas at a given scale. The level saliency maps are then merged into a unique, low-resolution global saliency map encoding the overall saliency of image areas.
The input images are encoded using the Lab color space, where for each pixel the channels L, a, and b corresponds, respectively, to the dimensions intensity (luminance), red-green, and blue-yellow. The Lab values are then split into ve di erent channels: intensity, red, green, blue, and yellow. Each channel extracted from the image is then encoded in an image pyramid following to the algorithm described in [Adelson et al., 1984,Greenspan et al., 1994]. 4.1
The set of feature used by the model includes intensity and color, computed according to the center-surround receptive- eld organization characterizing ganglion and LGN cells, and oriented lines, computed in area V1. The raw l; a; b values are used to extract the color channels II, IR, IG, IB, and IY that correspond, respectively, to intensity, red, green, blue, and yellow. Local orientation maps are computed on the intensity pyramid by convolving the intensity image in each layer with a set of oriented Gabor lters at four di erent orientations 2 0; 4 ; 2 ; 32 . Such lters provide a good model of the receptive elds characterizing cortical simple cells [Jones and Palmer, 1987], as discussed in the previous section. The lters used in the model implementation are expressed as follows [Daugman, 1985]:F (x; y; ; ; ) = exp( x2o +2 22yo2 ) cos( 2 xo + ) where xo = x cos + y sin yo = x sin + y cos . Each image in the intensity image is convolved with Gabor lters of xed size, in the current implementation they are 15 15 pixels wide. The rest of the parameters is set as follows: = 0:3; = 3:6; = 4:6, since those values are compatible with actual measurements taken from real cells [Serre et al., 2007].
The model uses the center-surround organization as found in the ganglion cells for color and intensity information. The channel for intensity, for example, is encoded in two di erent contrast maps, the rst one for the on-center/o surround receptive elds, the second one for the o -centre/on-surround opponency. Both types of cells present a null response on homogeneous areas, where the stimuli coming from the centre and the surround of the receptive eld compensate each other.
The original model [Itti et al., 1998] uses double-opponent channels, meaning that the red-green and green-red image encoding are represented by a same map. We used single-opponent channels since such choice allows us to distinguish, for example, strong dark stimuli from strong light ones. In order to respect the biological inspiration we use radial symmetric masks and we do not perform across-scale subtraction as in the original model. Basically, given two pyramids of two di erent features f and f ?, corresponding to the excitatory and the inhibitory features of the contrast map, the feature corresponding to the center of the receptive eld is convolved with a Gaussian kernel G0 that provides the excitatory response. The feature corresponding to the surround of the receptive eld is convolved with two di erent Gaussians G1; G2 with di erent sizes, that virtually provide the response of ganglion cells with di erent sizes of their receptive elds. The results of the convolutions correspond to the inhibitory part of the receptive eld.
The feature maps are computed for the following couples of ordered opponent features: (R; G) and (G; R), encoding, respectively, red-on/green-o cells and green-on/red-o opponencies; (B; Y ) and (Y; B), encoding, respectively, blue-on/yellow-o and yellow-on/blue-o opponencies. Furthermore, we encode center-surround di erences for intensity in separate feature maps: Ion;off , Ioff;on. The two maps encode, respectively, on-centre/o -surround and o -centre/onsurround cells for intensity. The feature maps are hereafter denoted with RG, GR, BY , Y B, Ion;off , and Ioff;on. Since the oriented features are extracted using di erential operators, they do not need to be processed as the other maps.
Before building the saliency maps for each level of the image pyramid, we need to merge the feature contrast maps in the same dimension: color, intensity, and orientation. This step is inspired by the FIT model, where parallel separable features are computed in parallel, each one competing with features in the same dimension. For example, in order to build the feature conspicuity map for color, we need to merge in a single map the two contrast maps RG (obtained by merging the R-G and G-R opponent channels) and BY. Simple summation or the creation of a map with the average values among the various contrast maps are not suited for the goal of creating a saliency map. For example, a red spot among many green spot should be given a higher saliency value than the green ones: with a merging algorithm based on simple summation or on the average red and green spot would receive the same weight. There are several strategies that could be used for modifying a map according to its relevance. Each strategy tries to decrease the values in maps that contain many peaks of activation and to enhance the values in maps that have few regions of activity. We implemented a merging step based on Summed Area Tables (SATs). Each pixel (r; c) in a SAT contains the sum of the pixel values in the subimage with corners located at image coordinates (0; 0) and (r; c), where the origin is the upper left corner.
In order to enhance maps with small spots of activity, for each pixel (r; c), we read the SAT value for a squared box centered at (r; c) with size equal to 1% the minimum dimension of the feature map and the SAT value for the entire image. Then we set the value for the feature conspicuity map using the following formula: F CM (r; c) = cSAT +2 cSAT tanh (cSAT sSAT ), where r and c are the coordinates in the feature contrast map F CM , cSAT and sSAT are, respectively, the sum of the values in the box representing the center and the surround values read from the SAT. This normalization procedure is repeated several times in order to inhibit weak regions while enhancing peaks of activity. 4.2
The nal saliency map is created at the lowest resolution of the pyramid. Several options are available and we chose to set the value of each pixel p with the maximum value of the areas in the image pyramid that are mapped onto p by the subsampling procedure. With respect to other solutions (average over the maps, summation) the max pooling operation allows us to keep and highlight in the global saliency map also areas that are very salient at only a single scale. By looking at pixels in the saliency map with high values, we can navigate through the pyramidal hierarchy to access the level where the maximum activation is present and analyze the salient region. However, in this paper we limit our experimentation to the bottom-up part that limits its computations to the bottom-up part. 5
We tested the proposed VA-based ltering approach on one landmark recognition
and one objection recognition tasks using two di erent datasets:
{ the publicly available dataset containing 1227 photos of 12 landmarks
(object classes) located in Pisa
The experiments were conducted using the Scale Invariant Feature Transformation (SIFT) [Lowe, 2004] algorithm that represents the visual content of an image using scale-invariant local features extracted from regions around selected keypoints. Such keypoints usually lie on high-contrast regions of the image, such as object edges. Image matching is performed by comparing the description of the keypoints in two images searching for matching pairs. The candidate pairs for matches are veri ed to be consistent with a geometric transformation (e.g., a ne or homography) using the RANSAC algorithm [Fischler and Bolles, 1981]. The percentage of veri ed matches is used to argue whether or not the two images contain the very same rigid object.
The number of local features in the description of the images is typically in the order of thousands. This results in e ciency issues on comparing the content of two images described with the SIFT descriptors. For this reason we applied a ltering strategy selecting only the SIFT keypoints extracted from regions with a high saliency. Each image in the dataset was processed by the VA model producing a saliency map. Since the resolution of the saliency map is very low, each saliency map has been resized to the dimension of the input image. 5.1
In order to study how many SIFT keypoints could be ltered out by the index, we applied several thresholds on the saliency levels stored in the saliency map. The thresholds range from 0.3 to 0.7 the maximum saliency value (normalized to 1). The 0.3 threshold did not modify at all any of the saliency maps, meaning that all of the saliency maps had values larger than 0.3. SIFT keypoints were ltered out only when they corresponded to points in the saliency map with a value below the given threshold. In order to see how e ective the ltering by the VA 0.9 model was, we compared it against random ltering, in this second case, we kept from 10% to 90% of the original SIFT keypoints by incrementally removing some chosen randomly ones. Notice that standard feature selection algorithms cannot be directly applied since in our approach the keypoints cannot be considered object features.
We used accuracy in assigning the correct landmark to the test images (in the previously mentioned dataset) as the measure of performance. For each test image, the best candidate match between the training images is selected using the SIFT description and verifying the matches searching for an a ne transformation using the RANSAC algorithm.
The results of the experimentation are shown in gure 2. The x-axis shows the percentage of SIFT keypoints kept after ltering. The y-axis corresponds to the accuracy reached by the classi er after the ltering. The maximum accuracy is reached by not removing any keypoint and is equal to 0.935. The accuracy does not vary much till a 40% ltering, when it starts decreasing.
When all the saliency values are used, the ltering performed using the visual saliency maps reaches a 0.89 accuracy when it removes almost 57% of the original keypoints. The performance of the VA-based lter is very similar to the randombased one when 30% keypoints are kept. However, when the percentages of removed keypoints inreases, the VA-based ltering algorithm outperforms the random ltering.
The results of the model when on aggressive ltering levels are quite encouraging. The model is in fact able to preserve regions that are signi cant for the recognition of the speci c object. There is a decrease in the overall accuracy with respect to the SIFT classi ers, but the time needed to perform the classi cation is signi cantly lower. In fact, when the classi cation uses 100% of the SIFT keypoints (no ltering), the average time for classifying a single test 0 10 20 30 40 50 60 70 80 90 100
Percentage of SIFT keypoints kept images is 7.2 seconds. When we use only 30% or 20% of the original SIFT keypoints (VA-based ltering) the time needed for the classi cation of an image is, respectively, 0.78 and 0.6 seconds per image on average. Even when the random lter and the VA-based lter have the same accuracy, the use of saliency provides 'better keypoints. When only a 40% percentage of the original keypoints is kept, the average time needed to classify a single image is 1.07 and 0.97 seconds for, respectively, images preprocessed using the random lter and the VA-based lter.
However, this experimentation has also shown a relevant limitation of ltering approaches based on bottom-up visual attention. In fact, many test images misclassi ed by the classi er contain salient regions that are radically di erent from the other images in the same category. For example, since many pictures contain people in front of monuments, the visual attention lter is prone to remove (i.e., assign a low saliency to) the monument in the background and preserve the people as the most salient areas. This behaviour is particularly evident on very aggressive ltering levels, where only the most salient regions are kept. In many cases the monument simply disappears in the saliency map. 5.2
In the case of the STIM-DATASET the saliency maps were thresholded using values ranging from 0.1 to 0.9 the maximum value in the map. By applying that set of thresholds, the percentage of SIFT keypoints kept and used by the classi er ranges from 11% to 77% (on average) the number of keypoints originally extracted from images. In this dataset the relevant objects are well-separated by the background in almost every image. Furthermore, since they never ll the entire frame, their features are not considered too 'common' to be salient and are not suppressed by the attentional mechanism. From the graph shown in Fig. 3 it is clear that the VA-based ltering is able both to improve the accuracy and to decrease the time needed for the classi cation. By using only half the keypoints selected by the VA model, the classi er reaches 81% accuracy much greater than that obtained using 100% of the original keypoints or 90% randomly selected, that are equal to, respectively, 0.77 and 0.74. 6
In this paper we have presented a ltering approach based on a visual attention model that can be used to improve the performance of large-scale CBIR systems and object recognition algorithms. The model uses a richer image representation than other common and well-known models and is able to process a single image in a short time thanks to many approximations used in various processing steps.
The results show that a VA-based ltering approach allows to reach a better accuracy on object recognition tasks where the objects stand out clearly from the background, like in the STIM-DATASET. In these cases a VA-based ltering approach reduces signi cantly the number of keypoints to be considered in the matching process and allows to reach a greater number of correct classi cations. The results on the PISA-DATASET are encouraging: a faster response in the classi cation step is obtained with only a minor decrease in accuracy. However, the results need a deeper inspection in order to gain a better understanding of the model on cluttered scene where the object (or landmark) to be detected does not correspond to the most salient image areas.
After this experimentation, we still think that bottom-up attention might be useful in the context of image similarity computations. In the context of landmark recognition, Better results could be obtained if the bottom-up processes receive a kind of top-down modulation signal able to modify the computation of the image saliency according the searched object. In fact, without such kind of modulation, if a query image contains only a single object, that same object might not be salient in any other image in the dataset.
The experimentation suggests at least two research lines. The short term goal is to evaluate the model for searching and retrieving images visually similar to a given query image. However, such goal requires the construction of a good dataset enabling a quantitative evaluation of the results. Except in very simple cases, it is not very clear when and how to consider two images visually similar. The long term goal is to introduce a form of top-down attentional modulation that enables object searches in very large datasets. Since CBIR systems usually relies upon an image index, it is far from clear how the most common index structures might be modi ed for allowing the introduction of that modulation.