We propose a learning-based image feature points detector. Instead of giving an explicit definition for feature point we apply the methods of machine learning to infer it inductively using a representative training set. This allows for a flexible tuning of the proposed detector to a specific problem that is described by a training set of desired responses. To increase feature points' repeatability and robustness to various image transformations the feature space of the learning algorithm includes raw image moments and image moment invariants. Experiments demonstrate high flexibility in tuning the detector to a specific task, acceptable repeatability of the feature points and robustness to various image transformations.
The proposed learning-based feature points detector is based on the idea of transforming detection task into a classification task as suggested in [13], which boils down to training the detector's classifier on a set of the desired responses. where norm and local mean ̅ are defined: ̅ = 811 ∑4=−4 ∑4=−4 [ + , + ], = √∑4=−4 ∑4=−4( [ + , + ])2. moments are defined [14]: used [15, 16]: = ∑4=−4 ∑4=−4 ∙ ∙
[ + , + ]. 1 81
The first step towards constructing our detector is to define the classifier's feature space, which is an ℝ15 vector space. Each pixel of an image [ , ] is mapped to a certain vector in this feature space using a locally defined operator 9×9 → ℝ15, where = { : 0 ≤ < 256} is a set of intensities of a grayscale image. The features of the feature space are described below.
The first two features are standard deviation of a standardized local area, 1, and standard deviation divided by the norm of the local area, 2:
The set of the features , 1 ≤ ≤ 15, defined by (
The next four features are chosen to be central image moments of a local image area: +3 = , 0 ≤ ≤ 3. The central
To induce invariance to rotation transformations the following Hu invariant image moments and Flusser moments are
15 = √( − )2 + ( − )2,
where = 10/ 00 and с = 01/ 00.
the feature vector space.
( 21 + 03)2],
( 21 + 03)2],
11 = ( 30 − 3 12)( 30 + 12)[( 30 + 12)2 − 3( 21 + 03)2] + (3 21 − 03)( 21 + 03)[3( 30 + 12)2 −
(
Moments calculation is an intensive computational task that requires of a lot of operations. To reduce the number of arithmetical operations we apply the recursive method of moments calculation based on the use of integer factorial polynomials [16].
The last feature that characterizes misalignment of centre of local area and its centre of mass is defined:
(
Tuning the detector requires a training set that consists of the desired detector's responses. Depending on the application there are various ways the set can be obtained:
manually, involving experts of the domain; automatically, using well-known feature points detectors such as Harris or Canny; combining the two.
In case there is a human involvement of any kind it is inevitable for a training set to contain a so called training noise [17]. Besides, in a typical scenario a number of feature points is small compared to the other points. To alleviate these negative effects the neighbouring points of the feature points can be considered feature points as well.
Provided an application requires high level of robustness to certain transformations, a training set can be enlarged to contain the so called virtual examples [18]. To this end every image used to form a training set is transformed according to some transformation. Since the parameters of that transformation are known, the elements of the original image can be mapped onto the transformed image, which makes it possible to extract feature vectors of the points of the transformed image that correspond to the feature points of the original image. These new feature vectors are the virtual examples that convey information about various effects the transformation have on the feature vectors.
With a training set at hand we can pose and solve a supervised learning problem. Since the number of the feature vectors in a
training set is typically quite large we chose to apply nonparametric probability density estimation approach. Let
=
corresponds to the other points. Then, an estimation of conditional probability density function is defined as follows:
{( , )} =1 denote training set, where is a feature vector, is its label, ∈ { 1, 2}. 1 corresponds to feature points and 2
where ̂ is an estimate of prior probability of ith class:
where is a kernel function, ℎ is kernel’s width parameter. By the Bayes’ Theorem:
(
0, ℎ ( ), ( ) > ( ) + ∀ ∈ ∖ { }
,
From (
is a set of all feature vectors from the local neighbourhood, is some threshold.
In order to smooth the detector’s response we filter the characteristic function ( ) using a local peak filter. The peak filter suppresses non-maximal values in a local 3×3 neighbourhood of the point :
To experimentally evaluate the proposed detector we built a set of images. The set contains a series of 10 overlapping images of 6 different scenes, 60 images in total. Figure 1 shows three images of one of these scenes. Each of the 6 groups of images was split in relation 8:2 to form training set and test set , respectively. We chose to use Harris [6] corner detector to detect feature points. The training set was enlarged by the virtual examples as described in section 2.2.1 and the transformations that were applied are described in section 3.3. 3.2. Evaluation of training accuracy ( ) = 1 ∑
=1[ ( ) = ].
Precision and recall are defined: ( ) = ( ) =
( ) ( )+ ( )
( ) ( )+ ( ) , .
Let = {( , )} =1 be training or test set. The primary criterion of detector’s performance on the set V is its accuracy: Besides the accuracy two more criteria are used: precision and recall [19]. Precision is the fraction of relevant instances over the retrieved instances, while recall is the fraction of relevant instances among the retrieved ones over the total number of relevant instances in the set.
Let
, and
denote false positives, false negatives and true positives, respectively. Then,
(
The proposed detector was first trained on the training set. Accuracy, precision and recall were evaluated on the training set and test set . The results are shown in table 1. Taking into account a fairly large size of the sets, the data suggests an adequate quality of training.
As mentioned in introduction, repeatability is one of the most important properties of the feature points. Along with its importance, repeatability allows for an objective and qualitative evaluation. Hence, we used repeatability to evaluate the performance of the proposed detector.
An original image is used to find a set of feature points . The original image is transformed by one of the transformations (cf. the next list below). The transformed image is used to find a set of feature points . Since parameters of the transformation are known, coordinates of the points of the original image can be mapped onto the transformed image. Thus, the points in the set are mapped onto the transformed image, forming a set . The sets and are matched. Two points ∈ and ∈ are considered equal if ∈ ( ), = 2.0.
Image Processing, Geoinformation Technology and Information Security / A. Verichev As a result of the comparison performed in the previous step we find three sets of points: are the points found on both sets, are new points that were not found on the original image but were found on the transformed image, are the missed points that were found on the original image and were not found on the transformed image. The cardinalities of these sets are, respectively, , and values of the proposed detector. These values are used to calculate the detector’s accuracy, precision and recall.
To evaluate repeatability we used the following transformations of the images: rotation by angle , −45° ≤ ≤ 45°, is increased by 3°; sub-pixel shift by , 0.25 ≤ ≤ 0.75, is increased by 0.05; scaling by , 0.5 ≤ < 1.5, is increased by 0.1
The results of the repeatability evaluation of the proposed detector that was trained on the training set are shown on fig. 2. The detector’s performance can be considered adequate on rotated images for −9° < < 9° and on scaled images for 0.8 ≤ ≤ 1.2. The performance on shifted images is high for the whole range of the parameter .
In this paper we investigated a relatively new approach to feature point detection. Contrary to the standard approach to the problem, we didn’t formulate any heuristics-based definition of the term feature point but tried to infer it inductively using the methods of machine learning and a representative training set. This enabled us to tune the proposed detector to a specific problem at hand. The results of the experimental evaluation of the detector verify that such a tuning is in fact possible. Moreover, the detector showed acceptable robustness to rotation and scaling transformation, and high robustness to sub-pixel shift transformation. This suggests a great potential of the learning-based approach to feature points detection.
The reported study was funded by RFBR according to the research project №17-29-03190-ofi.