In this paper, we propose the solution of visual objects recognition in computer vision problems using the classification of descriptors of image keypoints based on the training of Kohonen neural network on the description data of etalon images. According to the results of training within the set of etalons, the image classification method has been improved by defining a specific data space in the form of a statistical center for each etalon. We propose mathematical models for the bitwise analysis of multiple descriptors searching for the centers and the method for convolution of descriptions from multiple descriptors with the determining a posteriori probabilities for the system of bit centers. Methods of data space transformation of description bits are proposed for various options for Kohonen network training, processing and estimation of class centers. The software implementation of the changed classifier was performed as well as the processing time with different options for determining the space of training data was estimated. Experimental researches confirmed the high efficiency of classification preserving sufficient performance and the ability to use proposed methods in real-time applications.
Accuracy and performance are the main indicators in computer systems when
recognizing visual objects. Structural methods of image classification based on the use of
neural networks as a way to identify patterns on the set of features of structural
descriptions of the etalon base have become widespread in applied problems [
Modern structural methods are based on the detection of keypoints of images and
classification models in the space of binary vectors [
Copyright © 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
images effectively and quickly. Modern keypoint generation methods (detectors), for
example, ORB (Oriented FAST and Rotated BRIEF) [
The main advantages of modern ORB and BRISK detectors are that they provide high performance with comparative or better accuracy than SIFT (Scale Invariant Feature Transform) and SURF (Speeded up Robust Features), because of the significant simplification of the processing using binary descriptors.
The investigation of the adaptive properties and parametric characteristics of neural
networks for training in image classification problems with a variety of keypoint
descriptors [
It is also important to research the effectiveness of network training schemes that take into account the proximity of elements of various classes in the constructed feature space.
The implementation of Kohonen networks into the classification process makes it
possible to tune methods to process arbitrary sets universally and successfully [
Leading for the classification is the study of the possibilities and properties of the learning process of a neural network in terms of the most efficient use or transformation of the data space, and the research of in-depth training schemes.
The aim of the work is to improve the method for classifying images within the database of database of etalons on the detection of keypoint descriptors by defining a specific data space for network training and applying the classification to training results. 2
Let W x| x Rn, W R n be the space of structural features (descriptors of keypoints) of images. The base of descriptions of samples (etalons) Z W is given in the form of a set Z Z j Jj1 , s j card Z j , s card Z , s j s j . Moreover, each attribute xij Z j is associated with a certain etalon Z j of the base in terms of membership in it.
The main task of training the Kohonen network is to establish a classifier in terms of the formation of a system of centers for etalon data, the effective use of the etalon information placed in the descriptions to ensure high classification efficiency.
The purposes of the work are the use of the Kohonen neural network to train the classification system in a specific data space, analyze the options for training the network, study adaptive capabilities and assess the effectiveness of the functioning of the network and the effectiveness of classification by software modeling.
We know the use of various types of neural networks for pattern recognition of visual
objects for tasks such as getting key parameters or features of given patterns,
classifying patterns or characteristics already got from them [
It is known to use a multilayer neural network to classify images of faces
(features – nose, mouth, eyes) [
A neural network detector was used to detect a human image [19]. The training was based on images that contain and do not contain images of people. To increase the reliability of detection, a complex of neural networks trained in different initial weights was used. Each of the networks gives its error, so the final decision was made by voting.
A unique effect on the classification problem is provided by networks that can selforganize – Kohonen neural networks, they provide a topological ordering of the input image space. Unlike other methods, the topological ordering of classes preserves the similarity of input images [20], which is especially useful in classifying many classes. The change of the Kohonen network weights is carried out by the competitive training method with a sufficiently high processing speed.
The application of the gradient descent algorithm is presented when setting weights
in the Kohonen network [21]; the search for the minimum point is carried out in the
opposite direction to the gradient of the optimized function. Such an algorithm is
characterized by a “dip into the pit of a local minimum”, when the modification of the
parameters practically ceases, despite the presence of another deeper extremum. This
problem can be partially solved if we take into account the factors of change of each
weight [22] or take into account the value of second-order derivatives [20]. An
analysis of the sources [
The Kohonen network can recognize clusters in data and establish the proximity of classes [23]. Thus, it is possible to improve understanding of the data structure to further refine the neural network model by adjusting the existing rules for classifying objects. We can use the Kohonen network in classification problems where classes are already given [23]. It is important that, if, after recognition, the Kohonen network encounters a data set is unlike any of the samples known to it, then it classifies such a set as a new class [20]. The Kohonen network is studied by the method of successive approximations [23]. The training, in this case, does not consist in minimizing the error, but in fine-tuning the internal network parameters for a maximum match with the input data.
The author’s monograph [23] contains a variety of approaches to building and training networks, which allows you to choose the network structure by the data being analyzed. In general, the application of the Kohonen network in computer vision systems is aimed at identifying the most significant features of the image which are subsequently used for recognition [20-22].
The statement of the problem of cluster formation on a set of descriptors is studied
[
Researches related to the formation of initial centers on the basis of statistical
processing of multiple descriptors contribute to more efficient data self-organization [
In this paper, the keypoints binary descriptors dataset was generated using an ORB
detector, justified the choice of which in [
The purpose of training the network in this formulation is to form a system of centers
on a multitude of descriptors of the etalon base. Let us introduce the Kohonen
network learning procedure as a sequence of stages [
Stage 1. Choose the elements of the training set in the form of a set of Z Z j Jj1 descriptions of all elements of the database of etalons.
Stage 2. Initiate the matrix of weights M m j Jj1 , where the rows are the formed vectors of neurons of m j centers of classes, so m j xij , xij Z j , where i is the number of an arbitrary vector from the class of samples Z j .
Stage 3. Choose the current element z Z , for each j 1, 2, , J we calculate the distance q j (z, m j ) and determine the class d of the inner neuron: d arg min q j .
j
Stage 4. Calculate the changes in the weights for the neurons of the output network layer m j h( j, d , t) ( z m j ) (1) where is the learning rate, h( j, d , t) – the value of the neighborhood function for neuron number j at the time of training t .
Usually, h( j, d , t) exp (mj, md) / (t) is defined as a Gaussian function, and radius (t) 1/ exp(t 2 ) of the outskirts decreases with increasing parameter t , t 1, , s .
Stage 5. Adjust the matrix of weights M M M in step t .
Stage 6. Continue learning the Kohonen network (stages 3-5) until list Z is completed.
Stage 7. Verify the fulfillment of the termination condition of the Kohonen network. Traditionally, the criterion is the error value or the total distance between the centroid systems at steps t and (t 1) . If the stopping condition is not fulfilled, we continue training by stage 3, the selection of data from set Z is carried out in a fixed or random order.
In practice, we apply another common criterion for stopping network learning in
the form of a fixed number of iterations [
The choice of metric in stage 3 is determined by the space of descriptors and the method of forming the centers. For binary ORB descriptors, we will use the Hamming metric. To carry out the learning of the Kohonen network, the square of the Euclidean metric was used in the experiments, even though for binary features the value of the Hamming metric and Euclidean distance is identical.
The quality of classification directly depends on the learning outcomes of the system and the available set of structural descriptions of the training sample.
Let us focus on possible applied versions of the modified definition of the space of
educational data in the form of:
a) multiple descriptors for each sample;
b) sets of sample descriptors for which the statistical procedures for the formation
of class centers have been previously applied [
c) systems of centers for each sample class. This option differs from b) in that a class of centers is formed based on each of the samples;
d) implementation of the convolution procedure for the center system of each sample.
Options c) and d) carry out a more in-depth analysis, taking into account the characteristics of each of the samples as representatives of the class.
Training of the Kohonen network during the experiments, normalized data were used to maintain a sufficient distance between neurons to ensure effective separation of classes.
The effectiveness of the classification of visual objects according to a set of keypoints using the Kohonen network directly depends on such fundamental interrelated factors: a dataset as a set of descriptors, a data space for training, pre-processing methods are applied (the method of forming descriptors or the initial choice of centers), a metric for comparing descriptors, the size of the tuple adapted during the training of neurons.
As a criterion for assessing the quality of classification (error value), we choose a
value that calculates the proportion of the elements of the training sample Z , which,
according to the results of the classification, fell into the wrong class [
We analyze three basic options for constructing a classifier which differs in the number of neurons that are modified in the learning process: – only the winning neuron is configured; – three neurons that are closest in distance to the winner neuron are modified; – the complete network of neurons is modified by the distance to the winner’s neuron.
Let us evaluate the recognition performance using these training options for the Kohonen network depending on the data management parameters that are different in depth of data analysis.
The indicated training options for the classifier fundamentally differ only in
stage 4, where function h ( j, d , t) is calculated differently. To achieve convergence,
condition h ( j, d, t) 0, 1 must be fulfilled. However, the option when only the
winning neuron is tuned implements rough recognition. The variant, where three neurons
are modified that are closest in distance to the winner neuron, classifies each element
under research to one of the three nearest centers. The variant where the complete
network of neurons is modified in accordance with the distance to the winner’s
neuron performs the most complete processing, relating the considered descriptor to all
class centers at the same time, but with different weights proportional to the distance
[
Given the binary representation of ORB descriptors, we apply bitwise processing and,
at the pre-processing stage, for each etalon description Z j , we determine the class
center vector based on a logical rule that compares the total number of units for each
of 256 bits of the entire set of description descriptors and half of their number [
d 1 xd (b) s( j) / 2, s( j) d 1 xd (b) s( j) / 2, xd Z j , b 1, , 256, (3) where xd (b) is a bit with number b of the descriptor and number d in the etalons description.
According to (3), the values of each of the bits for center m j are determined by the bits of the total set of descriptors belonging to etalon number j . Center (3) reflects the statistical properties of the sample or class. During the research, centers (3) can be used to define the learning data space in the variants of classification b) and c).
An important characteristic for options a) and b) of learning the Kohonen network is the training time to ensure the classification is effective. The potential need is to increase the number of keypoints because ORB and other detectors sometimes select keypoints that do not contribute to the efficient classification of the image which introduces the requirement to consider alternatives by method (3), which reduces to comparing the obtained centers. This enables the neural network to significantly reduce the time of training and classification. Another option is to increase the number of generated keypoints and use pre-processing methods to obtain image descriptions.
The main function of neural networks is a generalization of the system of attributes for an image [20]. Most of all, this concerns convolutional neural networks, whereby targeted processing, which is typical for computer image analysis methods, a significant reduction in the total number of parameters of the network model is achieved to minimize network retraining [23, 27].
We interpret the image described by bitmaps of the descriptors of etalon Z j in the
form of a probability map
p j sd(j1)xd (b) / s( j),
(4)
that in the values of vector p j integrates the spatial information of description Z j
by adding the values of the bits of the descriptors to obtain a shortened model of the
input data [
We feed the normalized vectors (4) to the input of the Kohonen network and train the network only on them. The classification according to option d) is also based on the values of (4). Figure 1 shows an example of the values of the vector (4) for the test image.
Vectors p j can be used for any type of network, which significantly reduces the learning time of the Kohonen network by processing a single vector where hundreds of vectors have traditionally been processed. 6
To implement the proposed classifier modifications, the Visual Studio 2019 environment, and the OpenCV library tools were selected [29]. The OpenCV library has over 2500 software modules, which include a set of traditional and modern computer vision algorithms, as well as libraries of machine learning programs. For the experiment, four categories of images were selected, which are shown on the example of the Leeds Butterfly dataset [20, 30] (Fig. 2).
Figure 3 contains an example of the coordinates of the prevailing keypoints shown as circles. Images of butterflies is a unique object for scientific research because visually different butterflies have both significantly different and a number of common properties.
We will evaluate the effectiveness of the classification with network training options and different data analysis depths. The performance criterion is selected (2). During the experiment, 2 variants of the number of descriptors on each etalon 400 and 100 were tested to study their effect on performance. In combination with the parameter of the number of descriptors per etalon, 2 options were tested for the number of epochs of the Kohonen network 100 and 200 for searching and achieving a balance of performance and accuracy. The increase in the number of iterations to 300 or 400 was tested, the result showed a lack of growth in the classification efficiency, while the learning time of the Kohonen network significantly increases.
It has been established that options with tuning exclusively the winner neuron or
three neurons closest in distance to the winner neuron make it possible to perform
recognition faster or more accurately [
The proposed options a)-d) were researched for learning the Kohonen network and classifying it into 4 classes using 4 categories of etalons (Fig. 2).
It was revealed that the training time of the researched network depends on the following factors: – the number of epochs; – the number of descriptors per sample; – the number of neurons that change during network learning.
A series of experiments with a fixed number of iterations of learning the Kohonen network was carried out. The number of descriptors for each etalon ranged from 375...400.
The results of the research of option a) for the set of ORB descriptors for the test case (Fig. 2) showed that this approach does not make it possible to classify etalons efficiently since the number of keypoints defined by the classifier from the class description is not the maximum. The value of criterion 0.65 , while the estimated operating time of the improved method, was 14.94 s. This argues for the use of transforming the data space of the description in the form of pre-processing approaches or building centers to increase efficiency.
The training procedure for option b) was carried out on two different images of each of the 4 etalons (Fig. 2). The number of keypoints ranged from 378...400, and the number of iterations was 200. The total classification time was 4.98 s, the error value was 0.31 . Table 1 contains the number of votes of descriptors for input images of given classes from the test sample which is assigned according to the results of classification using training method b) to certain etalons. Having analyzed the data in Table 1, a high degree of difference was revealed: the maximum values of the votes are on the diagonals and significantly exceed other elements, that is, all etalons are classified correctly. As we see, the use of the Kohonen network with the center parameter (3) significantly reduces the classification error: from 0.65 for method a) to 0.31 for method b).
We will analyze the effectiveness of the classification using network training options that differ in the depth of analysis depending on the number of neurons to be configured, test a different number of descriptors for each etalon, and a different number of iterations of the network learning.
For variants with 400 descriptors per etalon, the classification error ranged from 0.33-0.28. For variants of 100 descriptors, the error naturally increased and amounted to 0.46-0.29. So, the main dependence is the effectiveness of the number of iterations of the network. The best results were obtained for 400 descriptors, where even with a few iterations of the Kohonen network (100); an error index of 0.28 was achieved.
A change in the number (1 or 3) of adapted neurons almost does not affect the learning indicators of the Kohonen network, but it is effective about the classification error and the number of erroneously classified data. Modification of one neuron is practical only for 200 iterations and a significant number of descriptors. Even under these conditions, the classification error is 10% larger than in the corresponding variants with the modification of 3 neurons, the number of errors increased from 0 to 2. The best result was obtained at 200 iterations of 3 neurons, while all classes of etalons with an error of 0.23 were correctly classified.
The estimated runtime for options with 400 descriptors was 13.7-16.21 s for options with 100 descriptors 6.2-7.9 s. It was found that method b) showed a high level of difference compared to others; however, it requires significant amounts of training time. In addition, experiments showed that the processing time in method c) was significantly reduced in comparison with other methods; it amounted to 2.27-2.77 s, the error ranged from 0.20-0.34.
It was revealed that the number of descriptors in the etalon and the number of iterations have the main influence on the effectiveness and time of classification. The best results were obtained at 200 iterations, even when using 100 descriptors, the error was 0.2.
It should be noted that a change in the number of adapted neurons practically does not affect either the operating time of the Kohonen network or the classification efficiency.
In variant d) we apply definition (4) and calculate for the descriptors of each etalon the probability vector of the appearance of a unit in each bit. The error for the best network operation options (400 descriptors and 200 iterations) significantly decreased to 0.06, only one etalon was incorrectly classified. It was found that with a decrease in the number of descriptors the error increases to 0.34, and a decrease in the number of iterations to 100 leads to a classification quality of 0.20.
Thus, the results showed a clear relationship between the classification indicators and the number of descriptors in the description of the etalon. The number of iterations is also an important parameter, enhances the classification capabilities of the network, and the number of adapted neurons under the given experimental conditions and the content of the analyzed images have almost no effect on the result.
Method d) based on (4) significantly reduces the operating time of the neural network compared to other methods without compromising quality. The classification quality for options c) and d) is almost the same. The processing time was reduced by almost 8 times (from 16.21 s to 2.82 s for options with similar quality), which allows us to use the proposed approach d) for large image datasets, in real-time applications, when processing video signals. The specified time potentially allows you to process every tenth frame of video with a refresh rate of 60 frames per second.
The obtained data on the network training time and the value of the classification error are summarized in Table 2. We see that the use of Kohonen network learning tools in the calculation of centers significantly reduces the classification error.
The experiments confirmed the ability to successfully adapt the Kohonen network to arbitrary sets of visual data, especially in the case of the successful formation of the initial centers of the class or the process of descriptor pre-processing. 7
The paper describes the results of a comparative analysis of the developed training methods and classification of images based on it for experimental images of the Leeds Butterfly dataset for various parameters of the Kohonen classification network. The research confirmed the ability to adapt network parameters universally for arbitrary visual data; transformations using statistical class centers and data convolution at the preliminary processing stage are especially effective.
The methods with the formation of a system of centers and the use of data convolution showed the best results for each class sample, and the use of convolution significantly increases the speed of data processing compared to other options.
The contribution of the research is the improvement of the structural classification methods using the Kohonen network by introducing a new description data space based on a system of etalon centers and data convolution, which helps to ensure high classification efficiency with sufficient performance and makes it possible to use modified methods in real-time applications.
The practical importance of the work is made by the obtained software models for assessing the effectiveness of classifiers in computer vision systems; the effectiveness of the development in the examples of image datasets is confirmed.
Further performance improvements can be achieved through supervised training approaches since the dataset descriptor classes are known. However, it is clear that such accounting for additional information potentially complicates the processing and affects performance. 18. Stanley, K.O., et al.: Designing neural networks through neuroevolution. Nature Machine
Intelligence. 1(1), pp. 24–35 (2019). doi:10.1038/s42256-018-0006-z 19. Islam, M., Chen, G., Jin, S.: An Overview of Neural Network. American Journal of Neural
Networks and Applications. 5(1), pp. 7–11 (2019). doi:10.11648/j.ajnna.20190501.12 20. da Silva, I.N., et al.: Self-organizing Kohonen networks. Artificial Neural Networks. pp.
157–172 (2017). doi:10.1007/978-3-319-43162-8_8 21. Galutira, E.F., Fajardo, A.C., Medina, R.P.: A Novel Learning Rate Decay Function of Kohonen Self-Organizing Maps Using the Exponential Decay Average Rate of Change for Image Clustering. In: Proceedings of the 2nd International Conference on Natural Language Processing and Information Retrieval, 7 September 2018, pp. 55–59 (2018). doi:10.1145/3278293.3278299 22. Hu, Z., Bodyanskiy, Y., Tyshchenko, O.: Kohonen Maps and Their Ensembles for Fuzzy Clustering Tasks’. Self-Learning and Adaptive Algorithms for Business Applications. pp. 51–77 (2019). doi:10.1108/978-1-83867-171-620191004 23. Kohonen, T.: Self-Organizing Maps. Springer-Verlag, Berlin, Heidelberg (2001).
doi:book/10.5555/558021 24. Ahmad M. Ayaz, Tvoroshenko Irina, Baker Jalal Hasan, Lyashenko Vyacheslav: Modeling the Structure of Intellectual Means of Decision-Making Using a SystemOriented NFO Approach. International Journal of Emerging Trends in Engineering Research. 7(11), pp. 460–465 (2019). doi:10.30534/ijeter/2019/107112019 25. Tvoroshenko, I.S., Kramarenko, O.O.: Software determination of the optimal route by geoinformation technologies. Radio Electronics Computer Science Control. 3, pp. 131–142 (2019). doi:10.15588/1607-3274-2019-3-15 26. Tvoroshenko, I.S., Gorokhovatsky, V.О.: Intelligent classification of biophysical system states using fuzzy interval logic. Telecommunications and Radio Engineering. 78(14), pp. 1303–1315 (2019). doi:10.1615/TelecomRadEng.v78.i14.80 27. Nasirahmadi, A., Ashtiani, S.H.M.: Bag-of-Feature model for sweet and bitter almond classification. Biosystems engineering. 156, pp. 51–60 (2017). doi:10.1016/j.biosystemseng.2017.01.008 28. Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016). doi:10.1109/CVPR.2016.278 29. OpenCV Open Source Computer Vision. https://docs.opencv.org/master/index.html 30. Yan, X., Yang, J., Sohn, K., Lee, H.: Attribute2image: Conditional image generation from visual attributes. In: European Conference on Computer Vision, 8 October 2016, pp. 776– 791 (2016). doi:10.1007/978-3-319-46493-0_47