We have developed a visual attention algorithm that combines previous existing methods to solve the problem of segmenting moving objects in real time. Our approach allows us to select regions of interest over which we force the fixations on a vision mechanism. We create saliency maps with characteristics that highlight within the scene. The amount of maps that can be extracted in an image is huge, so we just use some to avoid high latencies that can harm the performance of our system. Our approach is of special interest when there is no specific object to look for by the system. Thus, a scene is explored in a more natural way compared to simply sweeping out in some order the scene point by point or to wait patiently that an object of interest appears. With this, we assure that all relevant visual information in the scene is taken into account according to the priorities and objectives of the system.
The outline of this paper is as follows. In Section 2, we describe related work on visual attention models. Section 3 briefly describes the artificial visual system we constructed. In Section 4 we give details of our visual attention approach. Section 5 describes the saliency map generated together with the experimental results. Finally, in Section 6 are the conclusions and future work. 2
There exist different methods in the literature for visual attention. We propose a visual attention approach
based on a combination of three existing methods. First, there is the approach used by von Helmholtz
[
We have mentioned about fixing the sight in a point of interest in the scene, but the next question
arises: How to choose the point of interest within an unknown scene? Humans and some animals solve
this problem by using a very powerful biological tool known as visual attention. As established by Itti
[
In addition to the characteristics mentioned above, there exist other similarities between our artificial
visual system and the human visual system. One is the geometric configuration of the cameras, i.e., we
have the coplanarity restriction (via software) between the optical axis of the cameras and a point of
interest within the scene. Other characteristic of particular importance is focused on the motor abilities
of the human visual system [
The first aspect to consider when designing an active vision system is the type of configuration we want
our system to have. We have implemented an active visual system based on the Fick architecture [
We have considered the distinct methods and ideas mentioned in Section 2 to develop our visual attention algorithm. The algorithm allows us to select regions of interest over which we forced the fixations of a vision mechanism. We then create saliency maps with characteristics that are highlighted within the scene. The amount of maps that can be extracted in an image is huge, so we just use some of them to avoid high latencies that can harm the performance of our system.
In order to select an object of interest within the image scene we have developed a typical bottom-up
visual attention model based on the Itti’s model [
Our active stereo system is capable of fovealizing points of interest or previously known objects within
the scene by using a conventional CCD and an exponential retinotopic mapping (explained in Section
4.2) to resample the images. Working with foveated images improves significantly the time of extracting
information from the images. However, its variable resolution makes eye movements necessary to drag
the projection of the object to the fovea at each retina (camera). One of the key characteristics of active
vision is its real-time requirement. Having cameras able to move allows searching for new strategies
to decrease the response time. This is the main motivation of working with foveated images. One of
the objectives of a system with foveated vision is to achieve a good combination between a big aperture
angle of the camera with a significant decrease in the number of pixels, reaching the maximum resolution
over the regions of interest (fovea) [
At a biological level, the term retinotopic means that near points to the scene that are projected into
the retina are mapped near the striated cortex, i.e., the retinal topography is respected. Although it
is presumed that in primates the retinotopic mapping is polar-logarithmic [
X = x + sgn(x)bKjxjc 1; (1) where bwc is the floor function of w; w 2 R; X is a vector that represents the coordinates of a pixel (U;V ) in the original image, x corresponds to the coordinates of the pixel in the compressed image (u; v), where fX ; x 2 Z2g; and K > 1 2 R is a constant. In this way, a pixel (u; v) given in the compressed image takes the corresponding value of the pixel (U;V ) obtained from Equation 1, as follows: [U;V ] = [u + bK1jujc 1; v + bK2jvjc
Equation 1 describes an infinite family of functions to resample the image. The exponential component establishes how the resolution diminishes according to the pixel distance to the fovea. Therefore, we need to define K carefully so the image is resampled including the borders and the aperture angle of the camera is preserved without going beyond the limits of the image.
A good way to define constant K in a general form is:
Kx = (Sx=2 sx=2 + 1)sx=2; Ky = (Sy=2 sy=2 + 1)sy=2: Where Sn and sn (see Figure 3) represent the size of the positive and negative axis of the original and compressed images when the origin is displaced to pixel (X1;Y1). Equation 3 is used to design a mapping that decreases the area of the images coming from the camera to a quarter of the total size, i.e., from 480 640 to 240 320, with the origin (fovea) in the center of the image. In this way, we obtain Kx = 161(1=160) and Ky = 121(1=120) . The complexity of this algorithm can be inferred from Equation 1, which selects the pixels from the buffer with which a new image is formed to be processed. Although this image is smaller, it preserves the same aperture angle of the original image. This allows us to have an effect as shown Figure 3b, by reducing the image while keeping the retinotopic property of the sensor. (2) (3) (a) (b) Figure 3: (a) Original image, (b) the compressed image. Note in (b) how objects near the fovea keep their original size while far objects decrease in size and resolution. 4.2.1
The ability of the human brain to process images is far superior to any computational algorithm created. For example, it is excellent extracting the color of an object even with the presence of external variables, such as illumination. When we segment objects many different cues or characteristics are extracted. Color is one of the most relevant and commonly used to segment objects. The color of an object not only depends on the chemical composition of its surface, but also on the conditions of environment: illumination, intensity, number and color of the illumination sources, location and shape of the object, the intrinsic and extrinsic of the sensors, etc. Based on this knowledge we have developed a simple but fast algorithm that keeps the color constancy of objects. The procedure consists on changing the RGB format of each color pixel (of the fovealized image) to its quaternion representation assuming linearity in the reception of the luminous spectrum of the accopled device of the camera charge upon 4.2.2
By using Equation 5, which implies that (x + y + z) = 1, we can project the color on one plane (RGB triangle) and form histograms like the one shown in Figure 5. To obtain these histograms, the first step is to locate each color in the plane, there exists infinite ways to do this. In Figure 5, it can be observed that the red color is located in coordinates (255; 0), the blue in (0; 255) and the green color in (383; 383). The easiest way is to put the blue in the origin, the red in (0; 255) and the green color in (255; 128). This way a pixel’s RGB components are located in the coordinate plane (x; y) = (g; b + g=2). The following step is to define the desired resolution in the RGB triangle. The parameter we choose to restrict the resolution illumination changes. This representation helps in the segmentation because the direction of the color vector xi + yj + zk is less sensitive to illumination changes, thus facilitating the segmentation using color thresholding. The quaternion representation is defined as:
w = xi + yj + zk; where w is the intensity of the pixel, and can be defined with the color components on the RGB triangle: w = pR2 + G2 + B2 p3 ; x =
R R + G + B ; y =
G R + G + B ; z =
B R + G + B ; where R,G,B, correspond to the red, green and blue components, respectively. Therefore, we say that a pixel Pu;v = wu;v + xu;vi + yu;vj + zu;vk, where u and v are the pixel coordinates within the image, will be of a color of interest if its vector of directions v = (x; y; z) is additive inverse of the color being searched P = 0 + xi + yj + zk, i.e.: (Ph;k
P) ' wu;v + 0i + 0j + 0k:
The result of converting an image to its representation in quaternions is a format not well understood by the computer as an image, but it can be seen if we decompose it in two: the magnitud w and the direction (xi + y j + zk) as it can be observed in Figure 4. It can be noted that the quaternion representation makes the colors less sensible to variations in illumination. Note that same color objects but with different illumination in the left image are seen practically identical in the right image. Although this does not solve completely the problem, it will facilitate in great measure the search of the object of interest. In other words, what we do is to classify colors by grouping colinear vectors in just one vector. (4) (5) (6) is the number of transitions (n) between two primary colors. The last step is to count how many pixels fall in each of these slots and make a graph. 4.2.3
The search of the object of interest is based on the local extracted characteristics by the visual attention algorithm. The algorithm first selects the point of interest within the scene and then extracts some characteristics of that area, such as its histogram, the contrast map and the rate of change (explained in the next Section). We describe here the algorithm used to extract geometric characteristics of an object. As an start we assume that the object has a characteristic or set of characteristics C, as color, intensity, texture, etc. The algorithm begins by sweeping the images from top to bottom in search of C. At the moment in which this characteristic is found in a pixel Pu;v, it starts surrounding the pixel by searching all 8-connected neighboring pixels in clockwise direction in order to obtain its silhouette.
(a) (b) (c) Figure 6: Experimental results of the polar searching of objects. (a) shows the area of the pixels to search; (b) presents in black the pixels that has characteristic C; and (c) shows the objects found “of relevant size” for which their silhouette was extracted.
In this way, we can obtain the contour of a set of pixels with a common characteristic in a fast way and without using derivative-based operators that do not guarantee a closed curve and require slim filters that are sensible to the borders that the object may have inside. Once the silouhoutte of the object is obtained in a vector form, we can extract useful information to achieve the recognition of an object model previously saved in memory, such as the area (moment of order zero), perimeter, shape (compact or regularity factor), centroid, etc. If we look for an object that has been found in previous frames, it is not necessary to search the whole image. If we know the sampling rate of the camera and the maximum velocity of the motors, we can generate a search radius from the last position where the object was seen. In our case, we make the search in a spiral form increasing the radius r by two pixels each 2P radians. The angular increment is given by r p1 + 0:01.
Continuing with the example of Figure 4, we show now experimental results of the search algorithm described above in Figure 6. In this case the characteristic C to search is the red color. It can be observed that the algorithm is robust enough to detect in a predefined search zone the objects that contain the characteristic to search. 5
On of the most successful models in computational visual attention was proposed by Koch and Ullman
[
As users we can define a priority or degree of interest for the different colors based on the task. We also want our robot to have a certain degree of preference for some color or set of colors. Therefore, we set a weight different to zero to this map only when there is a search task.
Contrast is defined as the difference in color and intensity between a region and its surroundings. We can literally use this definition to generate the contrast map, which can be thought as a 3D filter, either a high-pass filter or magnitude of the gradient. The contrast map, Mc, is the result of the convolution in the space domain for the image I 2 R3 in its quaternion representation with the filter F 2 R2 : [Mc = I F; 2 R2], where the value of each pixel of Mc is obtained by using the discrete convolution (Equation 7), using a filter of (n m 4) as follows:
n m 4 Mc(x; y) = å å å I(i; j; k) f (i; j): i=1 j=1 k=1 (7)
In Figure 7, the colors that seem to highlight more are marked in the RGB triangle with a white cross, these are however, the ones that are far away from the background color in the triangle (indicated with a circle surrounding the white cross). Figure 8 shows the contrast map implemented with a magnitude of the gradient filter. In the three cases the circles that highlight the most are the ones indicated with the white cross, and the ones that highlight least are those near the background color, marked with black cross. What we can infer from the saliency of the contrast can be in function of the absolute difference between pixel and its neighbors, either vectorial for colors or scalar for intensities. This is the reason we use a magnitude of the gradient filter, besides of being faster than a high-pass filter.
We now show the contrast maps of a scene full of high contrasts in Figure 9, where the regions of maximum contrast are in red. 5.2
The individual motion of pixels in an image is known as optical flow and is measured when corresponding
pixels are found in two consequent frames. On one hand, the measurement of the optical flow can be
highly complex due to the similarity between pixels, and with current hardware technology, is almost
impossible to implement this algorithm on a real-time system. On the other hand, the main attentional
map is the motion map as is related with the most important task of the system: tracking moving objects.
An easy way to obtain a motion map without solving the correspondence problem is by using a motion
filter of absolute rate [
Obtaining this map could be complicated when the vision system is active, as the motion of the camera generates a visual flow in the whole image, for this reason the use of this map will be done exclusively at the end of each saccadic motion, when the motion of the cameras is practically null. 5.3
Inhibition of return is one of the most important components in the visual attention process as it retains in memory the regions that were already attended, to incentive the recollection of information in regions that have not been attended. This inhibitory effects does not depend on visual cues but on their spatial locations, for which we store the position of fixation points and induce an artificial potential field built in the robot’s workspace, so the inhibition map is the projection of the sum of the potential fields on the image. We define the potential field on a point as:
U (q ) = 1 ån e sqi22 ; i=1 where q is the angle between vectors A and B, s is the constant used to restrict the dilatation of the potential field so that does not exceed a solid angle greater than that occupied by the fovea. And, in order to project the effect of the potential field to an inhibition of return map (IoR), we use the coordinates (u; v) of vector B: IoR(u; v) = U (q ). (8) (d) (e) (f) Figure 11: Saliency map based on the red sphere. (a-b) are the original and foveated images, respectively; (c-d) quaternion representation of (b); (e) histogram map slightly rotated on the horizontal axis; (f) the same map inclined and rotated on the Z axis. 5.4
Under this computational model, visual attention is seen as a cost function that varies over time. Each visual characteristic constitutes a variable that, depending on its manipulation, can make fluctuate the attentional focus of the system giving an appearance of an animal or human behavior before an unknown scene. The integration of all maps is the more difficult and challenging part of the whole design. In our system we assign the preferences (weights) to each map according to the priority of each of the tasks.
We show in Figure 11 an experimental result of the saliency map obtained after integrating all the maps described above. In the sequence of images we show a scene where the object of interest is the red sphere. It can be seen that the sphere gets far from the fovea (located first at the center of the image) and therefore we used an attention map based on the color of the sphere to find it. Despite that there exist more red objects in the scene, the histogram resolution is good enough to broadly distinguish the sphere over the other objects. Even though this map is enough for this case, there will be other cases in which there exist very similar objects and the difference is not so well marked. Those cases can be solved using information about the motion of the cameras in order to see the regions of interest in high resolution, obtain fine details and facilitate segmentation. 6
We have presented a visual attention algorithm that combines existing approaches for segmenting moving objects in real time. First, the proposed image compression technique (foveated vision) with multiple resolution allows us to process images four to seven times faster and accomplish the objective of scanning the complete field of view without having to use special cameras. Second, the use of a quaternion representation of the RGB space together with the color histograms allow us to identify objects in a robust manner before illumination changes; so, we can use the same algorithm during the day with ambient illumination and during the night with artificial light. One clear disadvantage, however, about using color is the increase in the processing time. This forces us to select carefully the maps to be integrated and reduce its number to three. We hope in the near future to improve the latencies of the processing algorithms to include other maps, such as depth maps to direct the sight to areas where no measure of depth has been done. 6.1
The authors thank the National Council of Science and Technology (CONACyT) for funding this project.