Depth map fusion is an essential task in multi-view stereo vision and structure from motion methods, but there are some issues of the process of depth maps matching of large-scale outdoor scenes, which are acquired by unmanned aerial vehicles. First, they can have low depth resolution due to long distance from a scene to a camera, in the second, as a consequence of above problem and some camera defects, they can have some noise. In our work we propose an approach of matching the depth maps based on a local shape and a color similarity correspondences. The proposed idea is to nd similar regions by calculating local key points in small patches of two images by ORB feature detector. Then we nd corresponding regions assuming spatial proximity of movement vectors of key points, this is done by the search of the main vector direction and length in feature space. We test our approach with the help of video sequences acquired through a consumer multicopter.
At present there is an increase in quantity of stereo vision method applications in the production, robotics, in systems of virtual and augmented reality, geoinformation systems, computer games and simulating. This brings about the need for creation and development e cient methods and algorithms for solving the problems of stereo vision. The present paper represent a process of nding the e ective ways to solve the problem of comparing large-scale depth maps obtained by unmanned aerial vehicles. At the same time, the obtained depth maps can have a low depth resolution due to the large distance from the camera to the scene and, as a result, such depth maps can have noise.
There are several approaches for solving the problem of nding strong
correspondences of images and depth maps which aimed to increase accuracy of 3d
model reconstruction. Among them are various variations of the iterative
closest points (ICP) [
Given a set of n consecutive frames in F = f1; f2; :::; fn and a set of n 1 depth maps M = m1; m2; :::; mn 1, acquired from pairs of adjacent images in F . It is necessary to nd such pairs of points p; q 2 R3 that for each point pj 2 mi and the corresponding point qk 2 mi + 1; i = 1::n 1; j; k = 1::jmj, j j { number of pixels in the depth maps will be: qk = Apj + t, where A is the 3 3 rotation matrix, t is the 3 1 transfer vector. Alongside strict correspondences between points of neighboring depth maps are unknown in advance. Thus, the task of nding the transformation A and t can be divided into two sub-tasks: nding the indices j; k for the corresponding pairs of points p; q on the depth maps and comparison of the obtained point sets.
In our work, we proceed from the assumption that adjacent frames have
a small di erences of the direction of movement of points, which consists of
transference and rotation combination, while camera is signi cantly distant from
the scene, we assume a rectilinear motion. Based on these assumptions, it can
be expected that the movement of points in close sections of images will have
almost similar values of the motion direction. At the same time, the initial point
correspondences are found in the original RGB image space using the robust
search of feature descriptions, such as scale-invariant feature transform (SIFT)
[
We propose the following method for detecting strong correspondences
between points on large-scale depth maps:
1. We take set of images F as input. Each image has size W H.
2. We set the parameters for dividing images into local areas, which include
the width W and the height H of the image local part.
3. At each algorithm iteration, we take the images fi and fi+1 from F and the
corresponding to them depth maps mi from M . Thus, we form the structure
data = fi; fi+1; mi.
4. We split fi and fi+1 into t rectangular sections (patches) pj ; j = 1::t, Pi =
p1; p2; :::; pt, where t = bW=wc bH=hc, bxc - integer part of x.
5. The next step is de ning arrays of key points Ki and Ki+1 for each segment
pi+1 by the ORB algorithm, Ki = kj ; j = 1::KN - number of key points.
6. Initial mapping of points from Ki and Ki+1 is conducted. As a method of
mapping can be used any method of search approximate nearest neighbors
algorithm. We used the implementation from the FLANN library [
Let us demonstrate our approach to determine the exact matches of key points. For each pair of key points we set the vector vj = kpj+1 kpj , where kpj { coordinates of kj in the patch p. Each vector vj has an angle and length l and indicates the direction of key point movement between frames.
Assumption 1. In small areas of the image, correctly mapped points will give close values of a and l, and false-positive pairs of points will give arbitrary corresponding values.
Based on this assumption, it is possible to construct a classi er that will nd closely spaced points in the parameter space l. To speed up the nding process of close elements in l-space, we discretize it by splitting into a regular grid with the number of cells A along the O axis and the number of cells L along the Ol axis, thus the size of the grid cell is: subject to ; l 0
Then we build accumulator Acc array with size (A cell in it with zeros. For each vj we discretize it and l:
size = lsize = max
A max l
L j jdiscr = b A c lj ljdiscr = b L c
Acc[ jdiscr ][ljdiscr ] = Acc[ jdiscr ][ljdiscr ] + 1 Summary of our approach is provided on the Fig. 1. (1) (2) (3) (4) (5) Input images and depth maps
Key points detection (ORB or SIFT descriptors) Initial matching with FLANN lib Movement vectors of key points 90o
Discretization step (t=8) length cell with maximal number of vectors 45o 8
Hash-table7 building 6 0o length 315o 5
Due to Assumption 1 the cell with maximal value corresponds to the matched key point pairs and we mark such as a "good" matches. To accelerate computations we use a hash-table H to store indices of key points, the key value of the elements of H we calculate as follows: hash = jdiscr
L + ljdiscr (6) Therefore we can nd strong correspondences of key points and their consecutive directions. 3
We tested our algorithm on depth maps obtained from a video taken by a UAV with a large, relatively to the size of the seen, depth.
Image data and algorithm parameters: We have frames of video with 8 Mpx resolution, so size of our all images and depth maps is (W H) = (4096 2160) pixels each.
Subdivision parameters: w = W=8, h = H=8.
Number of key points for ORB detector KN = 100.
Number of discretized grid cells along axis: A = L = 20.
Result of depth map recti cation by our algorithm is demonstrated in Fig. 2.
In our paper we demonstrate an approach for estimation of strong correspondences between the key points in two sequential video frames. The proposed algorithm can eliminate false-positive occurrences from an array of key points. This is done by nding of major direction of key point movement in parameter space.
The limitation of our approach is that it can delete positive correspondences between the key points, thus, the future work is aimed to improving the quality of correct pairs matching. Another disadvantage is the ability of our algorithm to match key points only in the case of camera or scene movement.