Close-range photogrammetry is widely used to measure surface shapes and diagnose deformation. Usually, a stereo system of video cameras is used to register images of the measured object from several different angles. The surface shape is determined by triangulating a set of 2D points from these images. Triangulation uses the stereo system calibration parameters, which are determined before the experiment. Measurements during conditions with increased vibration loads can lead to a change in the relative position of the cameras of the stereo system (decalibration). This leads to a change in the actual calibration parameters and an increase in the measurement error. The decalibration problem can be solved using multidimensional optimization algorithms. To verify their calculation's results it is proposed to use a computer and physical modeling of decalibration of a video camera stereo system in laboratory conditions. The paper presents the implementation of the optimizing algorithm for the external parameters of a stereo system and the results of its performance during the experimental investigations.
photogrammetry, stereo system decalibration, stereo system calibration parameters optimization, computer modeling, physical modeling
Optical methods for measuring surface shape and deformations have many advantages over classical
contact and strain gauge methods. The main advantage of optical methods is the ability to obtain results
from a large area simultaneously, which allows you to get qualitatively better results during
measurements. Photogrammetry [
The main goal of photogrammetric methods is to find the position of spatial points with unknown three-dimensional coordinates. For this, it is necessary to find a pair of two-dimensional coordinates of these points in the two images that were recorded with two digital cameras. Further, the founded coordinates using triangulation according to the known internal and external parameters of the stereo system of video cameras give the desired three-dimensional coordinates of a point in space.
Several mathematical models are used to describe the transformation of three-dimensional coordinates into two-dimensional when registering images. One of the most popular in computer vision is the pinhole camera model. In it, the image is formed by projecting three-dimensional points on the image plane using a perspective transformation EMAIL: (V. Pinchukov); (E. Shmatko); (A. Poroykov);
2021 Copyright for this paper by its authors. 1 ( ) = ( 0 0 (1) where (X, Y, Z) is the three-dimensional coordinate of the point, (xc, yc) is the two-dimensional coordinate of the point on the image, r11...r33 are the coefficients of the rotation matrix, tx, ty and tz are the coefficients of the transfer matrix, (cx, cy) are the coordinates the principal point (the point of intersection of the optical axis and the plane of the matrices), fx, fy is the focal segment expressed in pixels, s is the scale factor arising from the use of homogeneous coordinates. The focal length of the optical system, the coordinates of the principal point, the coefficients of optical aberrations (internal parameters), and the coefficients of the transfer and rotation matrices (external parameters) are determined during the calibration procedure. This procedure is usually performed before or immediately after taking measurements.
Photogrammetric measurements can often be carried out during high vibration loads. Vibrations
cause the cameras to move relative to each other during measurements. They lead to a change in both
the external and the internal parameters of the stereo system. This process is called decalibration [
Displacement of the cameras relative to each other can be corrected by carrying out the recalibration
procedure, which consists of optimizing the parameters of the calibration matrix to minimize the
triangulation error. That approach was already used in other works. In [
As in other works devoted to this issue, the decalibration problem is proposed to be solved using
multidimensional optimization. When choosing a specific optimization algorithm, an analysis of
existing methods was carried out [
A regular simplex in an N-dimensional space is a polyhedron formed by N + 1 equidistant points the vertices of the simplex. An important property of the simplex is that a new simplex can be constructed on any face of the original one by reflecting a vertex relative to the center of gravity of all other vertices of the simplex.
Advantages of the Nelder-Mead algorithm are simplicity, a small number of preset parameters, a simple search strategy, calculating only function values, a small amount of memory required. Disadvantages of the method: the method works efficiently for N ≤ 6, the algorithm is based on cyclic motion along with coordinates. This can lead to the degeneration of the algorithm into an infinite sequence of exploratory searches without pattern matching.
The measurement error introduced by decalibration can be evaluated to estimate the possibility of optimizing the calibration parameters of a stereo system. For this, computer modeling was carried out. The decalibration modeling process is as follows:
the camera parameters are set, both internal (focal lengths, pixel size, and coordinates of the principal point) and external (vector of displacement and rotation of cameras relative to each other), all these parameters are determined in the experiment, at the stage of camera calibration;
the measurement object is formed as a set of points in 3-dimensional space; 3. without displacement (decalibration), by projecting 3D points onto the image plane of the cameras, 2D points are calculated, which act as actually received images from the cameras; 4. a known offset (decalibration) is set as the vector of displacement and/or rotation of the cameras relative to each other; 5. taking into account the changed external parameters of the cameras and the 2D points obtained in item 3, the 3D points of the object are calculated by triangulation; 6. then the obtained 3D points are again projected onto the image plane of the cameras, as a result of which we get reprojected 2D; 7. further, comparing the original 2D points and the reprojected ones, the magnitude of the reprojection error is estimated.
The following values were used in the computer modeling: the ratio of focus to pixel size is equal to 7500, the center point of the camera [1250, 980], displacement of one camera relative to the other Tx = -315 mm, Ty = 115 mm, Tz = 40 mm, rotation of one camera relative to the another one around the x-axis is 3.5°, the y-axis is 12°, the z-axis is -1°. Parameters of the measured object: distance to the object is 1.5 m, object size is 30×20 cm2, the number of points is 10×10, the angle of inclination of the object is 45°. The object is located opposite the left camera, which is taken as the origin of the coordinate reference system. These parameters were taken as an example of the actual calibration parameters of the stereo system and the measured object used in laboratory experiments.
Modeling showed that simple optimization by displacement or rotation by one axis will not lead to a global minimum of it, i.e. no correct or close to correct solution will be received. Therefore, in the optimization process, the objective function of the reprojection error depending on six parameters is minimized. The parameters are: Tx, Ty, Tz are the elements of the displacement vector, and α, β, γ are the angles of rotation of one camera relative to the other. In step 4, the offsets are performed by the Nelder-Mead method, and then steps 5, 6, 7, etc. are repeated until we reach the maximum iteration value or the reprojection error value is less than the specified one.
An example of the dependence of reprojection error on displacement along the x-axis and rotation around the x-axis (with and without optimization) is shown in Figure 1. The vertical axis (mean square deviation (RMS) in pixels) is displayed in logarithmic scale, in red the result without optimization, and in green the result with optimization. Optimization was performed by the Nelder-Mead method in conjunction with the penalty function method. The number of attempts for each of the displacements is equal to 12, the maximum number of iterations in each attempt is equal to 500, the steps for calculating displacements are 0.1 mm and 0.02°.
It can be seen from the Figure 1 that the reprojection error as a result of optimization has significantly decreased, by about two orders of magnitude. It is not worth counting on the same error reduction in physical modeling, since the error of reprojection without displacements in real cases is much higher.
For experimental verification of the optimization algorithm, an experimental setup was developed (see Figure 2). It is intended for physical modeling of the process of decalibration of a video camera stereo system. The decalibration is performed by moving one of the cameras using linear motorized movement modules controlled from a personal computer.
The movable camera is attached to a 3-axis positioner from Zaber. It consists of 3 separate XLSM050A miniature motorized linear stages with built-in controllers with the following characteristics: travel range 50 mm, position error 20 µm, and offset step less than 0.1 mm.
For experimental research, a test installation described in [
In experimental studies, the Image Pattern Correlation Technique (IPCT) [
Graphs of the dependences of the reprojection error on the displacement of one camera relative to the other were built. The optimization of the external parameters of the cameras to reduce the reprojection error was performed. In Figure 3, solid lines indicate reprojection errors without optimization, and dashed lines - with optimization; red color corresponds to displacement along the xaxis, green to the y-axis, blue to the z-axis. The x-axis is directed from the stationary camera towards the moving camera, the z-axis is oriented in the direction of the measurement object, respectively, the y-axis is perpendicular to the x-axis and z-axis and is directed downward, i.e. towards the base of the camera tripod. The measurements were carried out for various shapes of the object surface (a, b and c on Figure 3 respectively). The offset steps for all axes to either side of zero are 1, 5, and 10 mm in positioner coordinates. The position on the positioner x = 25 mm, y = 25 mm and z = 25 mm is taken as the offset zero. the positioner offset range is 50 mm.
From the data obtained, we can conclude that for all three surface shapes, the same dependence of the reprojection error is visible without optimization for the x, y, and z axes. The offset along the y-axis is most influential, slightly less along the x-axis, and displacement along the z-axis has little effect on the error.
Thanks to optimization for the first shape of the object surface, it was possible to reduce the error at all camera positions to values less than 0.1 pixels, for the second shape, less than 4.0 pixels, and for the third shape, less than 0.25 pixels. Although for the second shape of the object's surface, the minimum error is clearly not achieved, its decrease is significant at displacements of 5 and 10 mm. For the first and third shapes of the object's surface, it turned out to fall below the calibration values of the error and to maintain errors close to these values even at displacements of 5 and 10 mm.
In the above results, as in computer modeling, the Nelder-Mead optimization method was used in conjunction with the penalty function method, where the restrictions on the search for displacements relative to the beginning of the search are ±10 mm, and the rotation is ±2°. With the maximum number of iterations in one attempt 500 and the number of attempts equal to 12. The optimization used all the points in the image, which was obtained as a result of cross-correlation.
From Figure 3, we can conclude that the implemented algorithm successfully copes with reducing the reprojection error. We also note that if there is a need to speed up the operation of the algorithm, then you can use not all image points for optimization and try to reduce the number of attempts to find a solution.
The paper shows one of the approaches to optimizing the reprojection error function to find the actual parameters of the stereo system calibration. Actual parameters may change during measurements under high vibration loads, and this leads to increased measurement error. The reprojection error function is optimized with the Nelder-Mead algorithm for a function of six variables: displacement amplitudes and angles of rotation along three axes. It is impossible to determine the accuracy characteristic of the algorithm directly since it depends on many factors, but it is possible to estimate the achieved values of the objective function as a result of optimization. As a result of the optimization, the values of the reprojection error likely are the same as at the calibration stage and even smaller values.
In the work, a computer and physical modeling of the decalibration process in laboratory conditions was carried out with the subsequent application of the recalibration procedure. In the course of the latter, the reprojection error has decreased significantly. However, it cannot be said unequivocally that recalibration allows one to determine the ongoing changes in the position of the cameras. Further research will be aimed at increasing the number of experiments carried out for the collection of statistics and comparing the optimized calibration parameters with those specified in the modeling.
The investigation was carried out within the framework of the project "Development of photogrammetric surface shape measurement system for the conditions of high vibration loads" with the support of a grant from NRU "MPEI" for implementation of scientific research programs "Energy", "Electronics, Radio Engineering and IT", and "Industry 4.0, Technologies for Industry and Robotics" in 2020–2022.