The paper is dedicated to determination of equipped with a video camera unmanned mobile object (e.g., a mobile robot) orientation by natural landmarks. The problem is relevant for solving the problem of autonomous movement of the mobile object at a given point of navigating using natural landmarks linked to the map location. The algorithm for determining the orientation of an unmanned mobile object by natural landmarks in view of system conditioning at the point of calculation is proposed. The results of physical experiments of determining the orientation of an unmanned mobile object by natural landmarks in dynamics are presented.
The paper is dedicated to determination of an equipped with a video camera unmanned mobile object (eg, a mobile robot) orientation by natural landmarks. The problem is relevant for solving the problem of autonomous movement of the mobile object at a given point of navigating using natural landmarks linked to the map location. As natural landmarks we can use any objects on the scene, which the classifier was pre-trained and which have a binding to map of the scene. The minimum number of natural landmarks required to solve the problem is three. A modified cascade Viola-Jones detector was used as a classifier. As the map of the scene it can be used any scene image (such as a satellite image service "Yandex-map") with marked natural landmarks by which orientation is performed. To determine the orientation the scene metric characteristics are not required. It is understood that the camera calibration procedure has passed, ie, we know the direction to the natural landmarks relative to the central optical axis of the camera. This solves the flat task, ie coordinates of landmarks are taken in projection on a plane of motion. The result of work is the misalignment angle between real mobile object direction and calculated one. 2
A mathematical model for determining of an unmanned mobile object orientation by natural landmarks
The following mathematical model for determining of an unmanned mobile object orientation by natural landmarks was proposed:
Calculation of the required angle of an unmanned mobile object orientation (t): =
sin (d2) sin (d2 + d3) ∙ ∙ sin (d3)
sin (d2) 1 − cos (d2 + d3) ∙ ∙ sin (d3) − 1 −
Error investigations of an unmanned mobile object orientation determining by natural landmarks on computer and physical models
To check the proposed mathematical model adequacy the console application on C++ implemented allowing both to calculate the angle of an unmanned mobile object orientation in any system configuration and to perform the full run of all the possible natural landmarks positions on the coordinate plane relative to an unmanned object adjusted position and orientation with the orientation angle calculation for each configuration and comparison of the obtained result with the adjusted one.
In computer model investigation the following criteria of the mathematical model equations resolving were mounted: 1) Coordinates of natural landmarks should not coincide; 2) Directions to natural landmarks should not coincide; 3) Directions to natural landmarks should not coincide with the axis of motion of an unmanned mobile object; 4) Directions and angles ratio that determine their relative position should meet the criterion: a) ≠ − , in case
+ ≥ △ b) ≠ − − , in case
+ < △
Failure of criterion 4 can lead to unresolved equation. The study of the computer model showed 2.43% of natural landmarks configurations that do not meet the criterion 4.
The investigation of the error arising from the natural landmarks recognition mistakes was performed on the obtained computer model. The unmanned mobile object orientation measurement error was calculated by variation of the input system parameters (directions on the natural landmarks).
The results of an unmanned mobile object orientation error calculation are shown on fig. 2. Varying of directions to natural landmarks was carried out independently for each of the natural landmarks. Fig 3(a) 0,75 1,51 2,28 3,07 3,86 4,66 5,48 6,30 Fig 3(c) 0,91 1,83 2,76 3,71 4,66 5,63 6,61 7,59 Fig 3(f) 0,74 1,39 1,97 2,46 2,87 3,20 3,43 3,57 150,00 ) m rg100,00 u d ixm (ro 50,00 a r
r m e 0,00
Fig 3(d)
1 14,20
2 39,10
3 98,46
On fig. 2 variation of natural landmarks coordinates determining errors was performed within [-8,8] pixels. Fig. 2(a) shows the variants of natural landmarks configuration satisfying the criterion 4 in case of mobile object passage side or through the landmarks. Fig. 2(b) showss the variant, when the triangle formed by the natural landmarks is close to the degenerate.
Aside from investigations on the computer model the physical experiment was performed. A theodolite was used to determine the direction on the natural landmarks (replacing the calibrated camera). The distances and the angles between the landmarks were measured on the satellite image obtained with “Yandex Map” service. Centimeters were selected as a unit of measurement of the distance between the landmarks.
Fig. 3 represents six theodolite positions relative to the natural landmarks and the calculated value of the theodolite deviation from the zero point to each the position. The direction of the theodolite motion was selected in parallel with the model required trajectory (zero point), i.e. the expected result of the theodolite orientation angle calculation treal was equal to zero. The theodolite position on fig. 3 is marked by green arrow start, the arrow direction coincides the direction to the zero point.
The important conclusion according to the results of the error investigation of the unmanned mobile object orientation determining is strong dependence of the equations system response on the same input data changes depending of the natural landmarks geometry and the mobile object position relative to natural landmarks, i.e. different equations system conditionality depending of the input data. 4
The equation conditionality investigation in the unmanned mobile object orientation determining by natural landmarks As seen on Fig. 2(b) the certain combinations of formed by the natural landmarks triangle configuration with the unmanned mobile object position relative to the natural landmarks lead to the essential mobile object orientation determining error at small input parameters changes, i.e. to poor equation conditioning.
To study the conditioning of unmanned mobile object orientation determining equation the special application was developed.
The application allows to set any configuration of three natural landmarks, to set the adjusted error to input data, and to determine the zone of interest with any degree of detail. In addition, the program allows navigation in an area of natural landmarks through the image shift and zoom. The application allows checking error in any scene point.
Fig. 4 shows an example of the equation conditioning mapping result of the unmanned mobile object orientation determining by natural landmarks forming the isosceles right triangle.
On fig. 4 the natural landmarks are marked by red points. Yellow grid can be switched by user to simplify the perception of distances on the map and have an arbitrary step. The image on fig. 4 is corresponding to example with adjusted error equal to one pixel. Green areas on the image are corresponding to scene points with error in orientation determining less than 0.1˚. Blue areas on the image are corresponding to scene points with error in orientation determining more than 10˚. The points with error between 0.1˚ and 10˚ are marked grayscale. White color corresponds to 0.1˚, black is corresponds to 10˚. Red areas on the image are corresponding to unresolved cases.
Fig. 5 shows dependence of mobile object orientation calculation error in case when input error increases.
Green areas on the image are corresponding to scene points with error in orientation determining less than 0.5˚. Blue areas on the image are corresponding to scene points with error in orientation determining more than 10˚.
Fig. 6 shows the configuration of natural landmarks forming a triangle close to degenerate.
The physical experiment of the unmanned mobile object orientation determining by natural landmarks in the dynamics
The results of these studies were used in calculation of the unmanned mobile object orientation determining by natural landmarks in the mobile object test passages. The points with value of equation conditioning below the predetermined threshold were thrown out, which has greatly improved the final result.
The experiment included: 1) Classifier training by selected landmarks; 2) Marks arrangement on the scene and the calculation of their coordinates in the selected coordinate system; 3) The selection of the mobile object route (straight line, 90 degrees turn) and painting it on the physical surface; 4) Mobile object transfer on the planned route with simultaneous recording by video camera mounted on a mobile object, and directed straight to the course of its movement. The scooter was used as the mobile object. 5) The calculation of the resulting video sequence in a specially designed program that included: a. Recognition of natural landmarks; b. Landmarks splitting to triples; c. Calculation of mobile object orientation for each tri
ple; d. Checking equation conditioning for each triple at a
given point; e. Filtering triangles with poor conditioning; f. Filtering triangles by the geometrical reasons (already passed the point, broken geometry of points location, the point is detected clearly above / below expected position, etc.). 6) Displaying the results in easy-to-study format.
The result of this approach to the problem is the absence of manifest errors in the determining of the unmanned mobile object orientation. The maximum error in the determining of the mobile object orientation is in range of 5 degrees. The average error in the determining of an unmanned mobile object orientation is about 1 degree. An example of the test program is shown on Fig. 7.
(a)
On fig. 7 (a) colored points denote natural landmarks. Green triangle marks the natural landmarks by which the calculation is made. The maroon point shows the mobile object position and the white segment shows the direction of its movement. Fig. 7 (b) shows a corresponding frame from a video camera mounted on a mobile object with recognized landmarks. 6
The efficiency of the algorithm as a whole is provided by: 1) topological (not methodological) approach to the formation of the map, using only the relative coordinates; 2) A good detection; 3) To monitor the landmarks for several tens of frames (detection plus tracking);
4) Rejection due to bad decisions (instead the Jacobian is used more "natural" criterion);
5) The individual steps are based on the known approaches, the novelty is their aggregation.
The following results were reached during researching: 1) The mathematical model for determining the orientation of an unmanned mobile object by natural landmarks is described; 2) The results of the study of suggested mathematical model in determining the orientation of a computer model of an unmanned mobile object by natural landmarks are presented; 3) The results of the study of the error in determining the orientation of an unmanned mobile object by natural landmarks are presented; 4) The results of physical experiments of determining the orientation of an unmanned mobile object by natural landmarks in statics are presented; 5) The results of the equation conditioning study of the unmanned mobile object orientation determining by natural landmarks are presented; 6) The algorithm for determining the orientation of an unmanned mobile object by natural landmarks in view of system conditioning at the point of calculation is proposed. 7) The results of physical experiments of determining the orientation of the unmanned mobile object by natural landmarks in dynamics are presented.
This article was prepared with the financial support of the Ministry of Education, Agreement no. 14.578.21.0047 RFMEFI57814X0047 Agreement 2014-2020 Priority Technological Research and Development in the Russian Federation federal target program. 7 ural Visual Features with the Existing TV-cameras. Ed. by M.Yu. Khachay, N. Konstantinova, A. Panchenko, D.I. Ignatov, G.V. Labunets: Analysis of Images, Social Networks and Texts. Fourth International Conference, AIST 2015, Yekaterinburg, Russia, April 9-11, 2015, Revised Selected Papers. Communications in Computer and Information Science, Vol. 542, pp. 431–442, Springer.
4. A.V. Bakhshiev, A.M. Korsakov. Application of a TLD method to a problem of objects tracking in a task of space docking by TV picture // Extreme Robotics – robotics for work in hazardous environments. // Proceedings of the 7th International Symposium. St.Petersburg: publishing house “Politechniqual service”, 2013. – P. 293297.