This work aims to develop approximate depth estimation methods for three-dimensional computer vision in unmanned vehicles (UVs) based on a developed wavelet transformation-based optimization method. This method can assess informative features for matching and/or optimizing costs in image analysis, refining discrepancies, and more. Possible solutions are demonstrated for obtaining an approximate depth map by simplifying the calculation of disparity values, traditionally used for forming a depth map in intensity space and using edge description with adjustable detail based on wavelet transformation. The advantage of the developed optimization method over existing algorithms in the wavelet space is the increased speed due to the rational selection of the Haar wavelet support length in the extremum search area. Modeling confirmed the effectiveness of the proposed approach for constructing depth maps and allowed for recommending the proposed method for unmanned vehicles operating under limited computational and energy resources.
The primary source of information about the surrounding environment for solving a multitude of
tasks in various fields such as unmanned vehicles (quadcopters, unmanned cars), robotics, and vision
for the visually impaired (UVs) is video cameras [
Modern machine vision systems not only extract video information but also present it in a form and quantity that allows for the identification of significant, informative features of objects and processes. These systems recognize and assess their position and condition, select rational trajectories for UV movement, and, if necessary, generate corresponding control actions.
A distinctive feature of machine vision systems is not only the identification and recognition of an object and its position but also the consideration of the scene and object depth, i.e., the transformation of a two-dimensional image into a three-dimensional one, where information about the object is represented not just in units of brightness but with pixel/distance parameters. The analysis of the evolution of three-dimensional computer vision and the corresponding hardware and software designed for obtaining images and forming depth parameters has shown that the main problem is the high cost of the sensors and image processors used, making them largely inaccessible to the broader consumer market. Therefore, finding solutions for obtaining 3D depth information about objects in an image characterized by low cost, low power consumption, and acceptable
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). performance is an important and relevant scientific and practical task for a wide range of the aforementioned UVs.
When solving navigation tasks for UVs or collecting depth information about objects for them, the
acquisition of this information can be redistributed and/or transferred to backup subsystems and/or
systems operating on other physical principles [
These researchers have demonstrated the feasibility of using LiDAR for path planning, collision avoidance, orientation, and navigation.
However, image sensors (video cameras) are characterized by their lightweight, potential
decrease in accuracy under low lighting conditions, and loss (absence) of depth information, which
increases computational load during data processing. LiDAR allows for direct distance measurement,
but the result depends on the reflective properties of the surface and is energy-intensive [
Let's consider the method for constructing a depth map, implemented by the StereoBM function. The algorithm involves the following stages of data processing: 1. Loading images from both video cameras (often using the OpenCV library). 2. Converting images from color to grayscale. 3. Creating a StereoBM object with specified parameters. 4. Calculating the disparity map (the magnitude of the difference in the localization of corresponding image elements obtained from the right and left cameras). Traditionally, this is done by matching pixels in the left and right images to calculate a disparity map. 5. Outputting and saving the disparity map.
In the StereoBM algorithm, stages 1, 2, 3, and 5 are standard and do not provide opportunities to increase execution speed and thereby reduce energy consumption. Therefore, stage 4, which is aimed at calculating the disparity map, is more promising. This stage involves the following steps: 1. Selecting blocks. Blocks of a fixed size are highlighted in the image. 2. Comparing blocks. For each block found in the left image, the algorithm tries to find a matching block in the right image. 3. Calculating disparity. For each block in the left image, a displacement (disparity) is calculated, indicating how many pixels the corresponding right image is shifted. This disparity is stored in a disparity map.
Normalizing the disparity map. The resulting disparity map may contain values other than depth. Therefore, it is normalized so that the values are within [0, 255] for convenient display.
In this part of the algorithm, the most complex and crucial for the quality of the depth map is the
block comparison stage. The block comparison algorithm searches for a pattern on the left frame
corresponding to a block on the right frame. This is done by algorithmically moving along the
horizontal axis of the right image and comparing it with the corresponding block of the left image.
Various approaches can be used to assess the degree of block matching [
Differences, SSD). 3. Based on determining the minimum normalized cross-correlation coefficient (Normalized
Cross-Correlation). 4. Based on the Semi-Global Matching (SGM) method, which takes into account global and local features of low-quality images with different intensities or difficult or complex image acquisition conditions. 5. Based on adaptive window matching. Instead of a fixed block size, algorithms can use adaptive window mapping to account for changes in texture and intensities in images. 6. Based on the Graph Cuts method, which uses graph structures to calculate the disparity map.
These algorithms can take global and local image properties into account and improve accuracy in challenging imaging conditions (e.g., low-light conditions). 7. Based on machine learning methods. These methods rely on machine learning models trained to predict disparity based on feature detection in images. They can handle images with uneven texture and variations in illumination. 8. Based on deep learning methods such as Convolutional Neural Networks (CNN). These neural networks can be trained on large datasets and perform well in complex lighting conditions. 9. Joint use of sensors implemented on various physical principles, such as LiDAR or infrared cameras, to improve the accuracy of the disparity map.
Note that the approaches are numbered and sorted by the degree of increase in computational
costs. Approaches N2 to N6 require high energy consumption, which may be critically unacceptable
for BTS. Approaches N7 and N8 rely on neural network calculations, requiring computational
resources and a training "database," which can be challenging to provide for some BTS applications.
Approaches involving the joint use of sensors implemented on different physical principles (e.g., N9)
lead to increased energy consumption. Therefore, for further research, the method of comparing
image blocks in the right and left images (N1) was chosen as the base method, which can ensure the
speed of calculations. However, this method is characterized by low noise immunity, determining its
low quality. This property is especially noticeable in changing light conditions and/or low light
conditions, often encountered in applied problems solved by BTS. Additionally, the analysis allowed
us to establish the following. It should be noted that when developing algorithms for obtaining depth
information about objects and/or scenes, two assumptions about the structure of the observed scene
are used [
Therefore, to correctly determine the distance to objects during vertical calibration (along the
yaxis in the image), the position of the cameras is set such that the horizontal lines of both cameras
coincide. Horizontal calibration (along the x-axis in the image) is done by rotating the cameras
relative to each other until the x-coordinates of points at a distance of more than 10 meters coincide.
Consequently, this method requires pixel-precise positioning of the cameras both horizontally and
vertically. This complicates alignment and reduces the quality of object positioning [
The need for camera calibration. It should be noted that even with perfectly accurate camera positioning, obtaining a depth map in these systems (e.g., when avoiding obstacles) poses difficulties due to the necessity of pixel-by-pixel image matching.
Dependence on the quality of the initial images and/or incorrectly set camera parameters, lighting, and illumination for each camera.
Dependence of the number of computational operations and, consequently, performance on the size and quality of the image.
As the above analysis showed, a number of methods based on the assessment of local and global
image characteristics have been developed to evaluate disparity. Algorithms based on estimating
local image characteristics are fast but low in accuracy, while global methods can improve accuracy
and (often) noise immunity, but are characterized by low performance. The search for a compromise
between accuracy and speed can be aimed at developing methods for assessing disparity based on
optimization algorithms. When assessing disparity based on changes in the intensity of the pixels in
an image line, the coordinates of the extremum with a signal-to-noise ratio below 6 are estimated.
Known methods for searching for an extremum based on estimating the value of the first derivative,
which are traditionally used at this stage, do not work when the signal-to-noise ratio is less than 15.
To search for the minimum under such conditions, the authors developed and investigated an
optimization method based on the wavelet transform (WT) [
To reduce the impact of these disadvantages when constructing depth maps from stereo images, an approximate approach to depth information assessment is proposed. This approach considers the results of identifying areas of similar intensity or object contours in the images or points of maximum curvature in the contour description.
Figure 1 illustrates the main steps of in-depth assessment by identifying areas of similar intensity in a test image (see Figure 1, a). The proposed approach involves the following: ● ● ● ●
For a segment of the intensity row, determine the region of maximum intensity on the image from one of the cameras, for example, the left one (1) (see Figure 1, b).
Create a brightness envelope template: the intensity value corresponding to the extremum coordinate of the row and the intensity values of neighboring pixels (Figure 1, c), and locate a similar fragment in the image from the right camera (2).
Determine the distance between similar elements and the corresponding disparity value. Processing is performed row-wise and column-wise. If necessary, the results are combined using a logical OR scheme.
To illustrate, brightness envelope templates representing a block of 3 pixels are provided for the left and right frames of the image (see Figure 1, c). The templates show the intensity distribution around the extremum in the given image row for the left and right cameras (see Figure 1, b).
Both templates have a "peak" shape (other possible shapes include "trough," "plateau," "rise," "fall,"
etc.). Black squares highlight the brightness levels of each pixel in the blocks, while gray areas
indicate the brightness dispersion boundaries. The disparity between blocks is 4 pixels along the
Xaxis, and the absolute difference in brightness is 2. In real stereo images, the brightness difference
can reach up to 30. This approach is simpler than the well-known StereoBM method [
However, it should be noted that in this case, the quality of the depth map depends on the
effectiveness of the contour detection methods used, among which the most noise-resistant methods
are those using the wavelet transform (WT) [
Due to the frequency-selective properties of the WT, the ratio of the intensity of the contour of
an object having a smaller size to the intensity of the contour having a larger size decrease as the
scale of the WT increases [
The rows of the left frame image (see Figure 2, b) and the right frame image are convolved with wavelet functions of a given scale and contour areas are detected (see Figure 2, c); The distance between object contours and the corresponding disparity value are determined; Processing is performed along rows and columns as needed. Results are combined using a logical OR scheme; If necessary, the process is repeated for different scales.
a
Research has also shown that with a wavelet function support length s = 1, both fine and coarse image details result in peaks of similar amplitude in the intensity drop area after processing.
As the scale increases, the relative size of the peaks for fine details decreases, while the amplitude
of the peak at the boundaries of large-scale objects increases [
Furthermore, the proposed methodology allows for adjusting the level of image detail, ensuring high noise resilience, resolution of object contour extraction, and providing an approximate depth information assessment that meets the requirements for a range of tasks.
Thus, depth map construction methods can be categorized into local and global groups. Local method-based algorithms are characterized by their speed and lower accuracy, while global methods enhance accuracy but are slower.
To find a compromise between accuracy and performance, a group of methods based on optimization algorithms is actively being developed.
These methods can be used for evaluating informative features, matching fragments, and/or
optimizing costs in image analysis, as well as refining discrepancies and other related tasks [
Analysis has shown that the use of global optimization methods based on exhaustive search leads to
high computational costs, while local methods based on first and second derivative estimates exhibit
low noise resilience [
Therefore, for solving a range of optimization problems in image analysis for UVs, the wavelet
transformation (WT) - based approach has been developed, which enhances noise resilience while
providing sufficient accuracy for these tasks. In known optimization methods using WT, the
property of gradient estimation and processing with WT changes sign as it approaches an extremum
[
Continuous wavelet transformation is defined by convolution: ( , 0) = √| | −∞ 1
+∞ ∫ ( )Ψ∗ ( − 0) ,
ℎ =
∫ 1
( ) −
√ factor
1 introduced for normalization. partial wavelet transform:
where is the function being tra0nsformed (analyzed); Ψ , 0( )is a two-parameter basis function derived from the mother wavelet Ψ0( ) through scaling with a scaling factor ∈ + and translation with a parameter 0 ∈ ; and Ψ∗ is the complex conjugate function with respect to Ψ (where corresponds to the width of the wavelet and defines the position of the wavelet on the x-axis). The Iterative Extremum Search Method in Wavelet Transform Space. Let's define the one-dimensional Operator
( ) = (ℎ1, … , ℎ ) where ( ) is the objective function; is the parameter vector; − represents the parameters.
In known optimization methods using wavelet transform (WT), the property of gradient estimation and wavelet processing is employed, where the sign of the gradient changes as it approaches an extremum. In other words, the condition for an optimum is considered to be the equality of all partial wavelet transforms: WT(c) = 0 . regular iterative optimization algorithm in the wavelet transform space [19] is as follows: step size; iteration number; extremum coordinate. the optimization requires significant computational resources.
It should be noted that due to the direction of the search being assessed using wavelet processing, (2) (3) (4) (5)
This limits the applicability of the method for practical UV tasks, as it is necessary to ensure optimization efficiency, especially for multimodal objective functions with low signal-to-noise ratios existing wavelets is due to the fact that it is characterized by low computational complexity.
When implementing computational procedures using it, multiplication operations by +1 and -1
are performed, which can simplify the hardware implementation of this procedure and increase the
speed of operation [
However, with a large Haar wavelet function length and a high signal-to-noise ratio of the objective function, the search may deviate in the wrong direction, moving away from the global minimum towards a local minimum. This results in reduced search efficiency.
The proposed work aims to improve search efficiency by selecting the Haar wavelet function length in the search area.
To achieve this, it is proposed that after localizing the minimum area of the functional during the Haar wavelet function search phase Ψ1( ) and narrowing down the search area by defining constraints ≤ 0, the Haar wavelet function length should be selected judiciously, taking into account the characteristics of the extrema in disparity evaluation.
The method with iterative constraint evaluation using Haar wavelet functions is implemented in the following sequence:
Step 1. Execute steps 1-5 of the basic wavelet optimization method, taking into account the
iterative scheme [
∑ ( [ ], + ) ⋅ ( )
where s is the length of the Haar wavelet support; Ψ ( ) is the step size of the Haar wavelet
discretization; Ψ1 is the Haar wavelet at the k-th start; = 1, … ,
is the dimensionality of the
parameter vector. This approach allows for determining the range of variation in the extremum [18]
coordinates with 1 being the search error for the optimal start (determined during the preliminary
functional quality studies); 2 being the search error for the practical task's optimum. The search is
conducted considering the following parameters: [0] as the initial approximation to the optimum
coordinate; [
Step 2. The constraints of the search zone for the next algorithm start g_1(c[n]) and g_2(c[n]) are determined:
where ∗[ − 1]is the extremum coordinate at step [ extremum coordinate at step , where the sign changed (see Figure 1).
1( [ ]) ≥ ∗[ − 1]; 2( [ ]) ≥ ∗[ ] (9) is the
Step 3. The length of the Haar wavelet support for the subsequent start is determined by selecting from the range = {2,4,6} according to the condition ≤ | 2( [ ])− 1( [ ])| .
Support Length Ψ ( ) 1 = 10 Defined Using Haar Wavelet Ψ ( ) (see Figure 3). When the SNR (signal-to-noise ratio) is reduced to 2, the proposed method's performance (measured by timer) is on average 1.1 times higher (see stage of selecting the length of the Haar WF carrier (Fig. 2, b). The sensitivity of the developed optimization method to local extrema and the starting point of the search using the test Schwefel function ( ) = 418,9829 + (− ⋅ √| |) (10)
The sensitivity of the developed optimization method to local extrema and the starting point of the search using the test Schwefel function has been studied. This function has a false global minimum. The function at x ∈ (−500; 500) has a global minimum fs(x) = 0) at (x = 420.9829. During the research, the starting point was chosen randomly. The gradient descent method made it possible to find the minimum closest to the start, while the optimization method with the Haar wavelet function achieved the global minimum with an error ≤ 10−2 in 128 out of 150 cases.
This results in a probability of finding the extremum coordinate of 0.85. The global minimum was not found when the starting point values were chosen outside the interval ∈ (−420; 470). [ ] are marked with squares.
1
basic method [
Future research in this direction will focus on evaluating the impact of wavelet function support length on the performance of the maximum search procedure, as well as the effects of the wavelet discretization step and the step size in iterative searches with Haar wavelets ( ).
The developed optimization method on the wavelet transformation base was tested in the context of constructing a depth map from a test image in the image database [20, 21].
The length of the wavelet support used in the minimum search after investigation was chosen as 17, the wavelet discretization step was 1, and the step in the iterative Haar wavelet search was applied 0,9. The minimum image alignment error was found after 12 iterations (starting the search at [1; 1]).
Furthermore, simulation results indicated that modifying the known StereoBM method by incorporating wavelet transformation for disparity calculation enhances noise robustness, reduces error in locating the characteristic fragment on the intensity line, and also decreases energy consumption by nearly a factor of 2 (Table 1). 6. Conclusions The paper addresses the development of approximate depth estimation methods for three-dimension computer vision in autonomous vehicles based on an improved optimization method utilizing wavelet transform. This method can be used for evaluating informative features for fragment matching and/or optimizing costs in image analysis, as well as for refining inconsistencies, among other applications. The study demonstrates possible solutions for obtaining an approximate depth map by simplifying the calculation of disparity values, traditionally used for depth map formation, within the intensity space and using image contour descriptions with adjustable detail based on wavelet transform. The advantage of the developed optimization method over existing algorithms on the wavelet transformation base is its increased efficiency due to the rational choice of Haar wavelet support length in the extremum search area.
Future research in this direction will focus on evaluating the impact of wavelet function support length on the performance of the maximum search procedure, as well as the effects of the wavelet discretization step and the step size in iterative searches with Haar wavelets.
This improvement makes the proposed method suitable for autonomous vehicles operating under constraints of computational and energy resources. [18] Y. Bodyanskiy, N. Lamonova, I. Pliss, and O. Vynokurova, "An Adaptive Learning Algorithm for a Wavelet Neural Network," Expert Systems 22.5 (2005): 235-240. doi:10.1111/j.14680394.2005.00314.x. [19] G. Shcherbakova, H.-S. Shi, V. Krylov, N. Bilous, and S. Antoshchuk, "Estimation of the Duration of RR-Intervals of Electrocardiograms by Means of Multi-Start Optimization Based on Wavelet Transformation," in: IEEE 9th International Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, 21-23 September 2017, Bucharest, Romania. doi:10.1109/IDAACS58523.2023.10348849. [20] Middlebury Stereo Vision Dataset, "Full-Size Stereo Data and Scene Information," Middlebury Stereo Vision Project. URL: https://vision.middlebury.edu/stereo/data/scenes2005/FullSize/Art/Illum1/Exp1/. [21] Middlebury Stereo Vision Dataset, "Stereo Data Archive," Middlebury Stereo Vision Project.
URL: https://vision.middlebury.edu/stereo/data/scenes2005/.