Recognizing objects from images becomes a more and more important research and application topic. There are diverse applications such as face recognition, analysis of aerial images from multicopters, object tracking, image-based web search, etc. Many existing approaches focus on shape retrieval from a single polygon of contour points, or they try to compare clouds of interesting points of an object. However, human object recognition concentrates on few points of the segments forming the object. Clearly, complex objects, strictly speaking the projections of their shape on the image plain, consist of several (polygonal) segments. Therefore, the procedure presented in this paper takes this into account, by composing objects hierarchically into a group of segments. We briefly introduce our procedure for semantic object recognition based on clusters of image segment contours and discuss the problem of recognizing objects from different perspectives.
Already in [
Each object in an image is decomposed into segments with different shapes
and colors. In order to recognize an object, e.g. a house, it is necessary to find out
which segments are typical for this object and in which neighborhood of other
segments they occur. A group of typical and adjacent segments for a certain
object defines the whole object in the image. Similar segments are clustered.
A hierarchical composition of these segment clusters enables model building,
taking into account the spatial relations of the segments in the image. The
procedure employs methods from machine learning, namely k-means clustering
and decision trees with boosting [
The rest of this paper is organized as follows: First, we introduce our procedure for semantic object recognition in some more detail (Sect. 2). After that, we consider the influence of perspective projection on object recognition (Sect. 3). Next, we provide a brief evaluation of the approach (Sect. 4) and discuss some other approaches on object recognition (Sect. 5). Finally, we summarize and conclude the paper (Sect. 6). 2
In our procedure for semantic object recognition, at first, we train models with sample images for each object category. After training, the models are used to recognize objects in other images. This is done as sketched in Fig. 1. Most of the steps and data in this process (marked in black in the figure) are identical for the training and the recognition phase. The steps and data that are relevant only for the training phase are marked with blue boxes and arrows, whereas the data and steps that are relevant only for the recognition phase are marked in green. 1. Image optimization and segmentation:
For each pixel in the image, similar neighboring pixels are colored with a
uniform color by a flood fill algorithm. With an image pyramid segmentation
algorithm, the shapes of the resulting blobs of uniform color are extracted
as image segments [
A feature vector is computed for each segment, using the data of four
normalized distance histograms, computed from the segment contour. A
distance histogram consists of a vector of distances computed with several
related methods: polar distance, contour signature, and ray distance [
During training, a decision tree model from the data of the segment tree for each object category is created. During recognition, the decision tree model of each category is used to recognize objects in the actual image. image, category data base copter camera actual camera image images, categories 1 .. n actual camera image optimized image image segments feature vectors (*) (*) (*) ... cm_n cm_2 s_2
... dt_2 cluster analysis category
2 st_2 s_2 category affiliation prediction marking category object ... ... s_n ... ...
ca_n 1 3 2
4 1 – minimal tangential contour signatur distance 2 – maximal polar distance 3 – maximal ray distance through centroid 4 – maximal tangential contour signatur distance Let us now consider the problem of perspective distortion in more detail, where objects are viewed from different viewpoints. In this context, the shape of an object is more or less defined by its surface. For the sake of simplicity, we assume that the surface is given as a polygon mesh. One question then is, what happens with the contour of a polygonal segment on the object surface, when it is viewed from different perspectives. Here, we restrict attention to the case where the segment is completely visible and not partially hidden. Let us consider the projective image of a cube as an example (Fig. 3). Clearly, by perspective projection, angles between lines and lengths of lines including their ratios may deviate significantly from their original values. In addition, parallelism of lines is not preserved, too.
Formally, the image of an object can be roughly described by central projection. Central projection (planar 3D projection) is determined in essence by the point of the observer, called central or focal point c and the position of the image plane, in particular its normal vector v, both consisting of 3D coordinates.
This gives us approximately 6 parameters. In contrast, a polygonal surface with n vertices is determined by n 2D coordinates, giving us 2n values. Since 2n > 6 for n ≥ 4, it follows that, although the shape of a polygon may vary widely by perspective distortion (cf. Fig. 4 and 5), for polygons with 4 or more vertices, there are clear limits for the distortion.
In fact, there are several invariants during perspective projection: First of all,
straight lines remain straight lines. This implies that polygons remain polygons
with the same number and order of vertices. Furthermore, left-right relations,
which are important for localization, navigation and exploration [
In principle, full perspective projection can be taken into account: For this, a point x of a segment contour, given as 3D column vector in homogeneous coordinates, i.e. with an additional component (cf. [7, Sect. 5.6]), is projected onto the 2D plane, at position y, that is a 2D column vector (also in homogeneous coordinates). The corresponding mapping M0 : x → y consists first of a rotation R, that is 3 × 3 orthonormal matrix, and a translation T , a 3D column vector, and then the actual projection P from distance −d, as follows (cf. [7, Sect. 6.4]): 1 0 0 0
P
Let now y1 and y2 be the projections of a given object point x, whose position
usually is not known, on two images. Therefore, we have yi = Mi · x for i = 1, 2,
where M1 and M2 are mappings as above, in general different. From this, we
obtain (1) λ y2 = M · y1 with M = M2 · M1†, where † denotes the Moore-Penrose
pseudoinverse operator and λ a scale factor, which is needed because the 3 × 3
matrix M , mapping the homogeneous 2D coordinates, is not affine in general, i.e.,
its last row may be different from [0 · · · 0 1]. Eq. (1) can be expressed equivalently
without reference to λ by the cross product y2 × (M · y1) = 0. This leads to three
linear equations in the 9 components of the matrix M , of which only two are
linearly independent however, for each projection point pair (y1, y2). Given n
such pairs, we arrive at the matrix equation A · m = 0, where A is a (2n) × 9
matrix and m is the matrix M reshaped as column vector (cf. [
Since A is overdetermined in general and it must be m = 0, the solution for A · m = 0 can be found by minimizing the squared (L2) vector norm (2) ε = |A·m|2 with respect to the condition |m| = 1. Eq. (2) is equivalent to ε = (A·m) · (A·m) = m ·(A ·A)·m, where denotes transposition, thus (A ·A)·m = ε m (after multiplication with m). Hence, the problem of determining whether two segment contours stem from the same object, taking perspective distortion into account, can be reduced to an eigenvalue problem. As distance measure in the clustering procedure, the smallest eigenvalue ε of the 9 × 9 matrix (A · A) can be used. Nevertheless, a normalization with respect to the starting point of the segment contour has to be done. 4
The object recognition method with distance histograms (as described in Sect. 2)
has been implemented in C++/OpenCV [
To test and improve the first implemented algorithm in a controlled
environment, it was used to classify images from the butterfly image dataset [
The problem of recognizing and locating objects is very important in applications
such as robotics and navigation. Therefore, there are numerous related works.
The survey [
[
[
With our approach, we can recognize objects in digital images, independent
of scaling, translation, rotation, and perspective distortions of the object. To
do this, we train and use models based on the segment shapes of the objects
and the topological and spatial relations of these segments. The next step is to
implement the approach as a real-time object recognition process on autonomous
multicopters (cf. [