=Paper=
{{Paper
|id=Vol-2210/paper52
|storemode=property
|title=Compensation of the complex geometric distortions induced by a car digital video recording equipment
|pdfUrl=https://ceur-ws.org/Vol-2210/paper52.pdf
|volume=Vol-2210
|authors=Evgeny Myasnikov
}}
==Compensation of the complex geometric distortions induced by a car digital video recording equipment==
https://ceur-ws.org/Vol-2210/paper52.pdf
Compensation of the complex geometric distortions induced
by a car digital video recording equipment
E V Myasnikov1
1
Samara National Research University, Moskovskoe Shosse 34, Samara, Russia, 443086
Abstract. In connection with the wide spread of mobile video recording equipment such as
dashcam (car DVR), the restoration of distortions introduced by such equipment is of particular
interest.Unfortunately, the nature of the geometric distortions introduced by such equipment is
sometimescomplex, and cannot be described using the classical radial models. For this reason,
in order to compensate for geometric distortions, an original method for compensating the
geometric distortions was developed in this paper.The advantage of the proposed approach is,
on the one hand, the possibility of a compact parametric description of the introduced
distortions, and on the other hand obtaining at the final stage an improved correction field that
takes into account the local inhomogeneities of distortions that occur when inexpensive
systems are used.
1. Introduction
A number of models for estimating and compensating a radial distortion are described in the
literature.These models include polynomial models, rational models, fisheye models and so on.In
general, we consider the problem of correcting the radial distortion at the present time as an
engineering problem. Nowadays this problem can be solved using calibration patterns and existing
software tools (see, for example, [1, 2]).
In connection with the wide spread of mobile video recording equipment such as dashcam(car
DVR) [3], the compensation of distortions introduced by such equipment is of particular interest.
These systems often aim to expand the field of view of the camera in the horizontal direction, which
allows recording information related to the area of interest (road, room, etc.), limiting the amount of
unnecessary information outside the area of interest. This approach allows reducing both the amount
of stored video information and the computational load on processors of mobile devices.
Unfortunately, the nature of the geometric distortions introduced by such equipment is sometimes
complex, andcannot be described using the classical radial models. For this reason, in order to
compensate for complex geometric distortions, an original method for compensating the geometric
distortions was developed in this paper. The developed method is based on polynomial models and
takes into account geometric distortions in general, regardless of their nature (whether they were
induced by an optical system, hard- or software).
Thispaperisorganizedasfollows. The next section briefly reviews classical distortion models. The
proposed approach for compensating the distortion induced by the considered mobile video recording
equipment is described in Section 3. An experimental study of the proposed approach is given in
Section 3. The paper ends up with the Conclusion and Refences.
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018)
Image Processing and Earth Remote Sensing
E V Myasnikov
2. Distortion models
The optical system of a modern camera, like many other optical systems, introduces nonlinear
distortions [4]. It is accepted to distinguish the radial distortion associated with lens curvature and the
tangential distortion associated with the inclination of the lens optical system with respect to the
photosensitive sensor.
Radial distortion consists in shifting a point on the captured image towards or away from the
optical axis with respect to its true position. If the displacement is directed towards the optical axis, the
resulting distortion is called barrel-like distortion (Fig. 2a). Otherwise it is called pillow-like distortion
(Fig. 2b).
(a) (b) (c)
Figure 1. Image distortion: (a) radial pillow-like distortion; (b) radialbarrel-like distortion; (c)
tangential distortion second picture.
The tangential distortion associated with the inclination of the sensor plane is expressed in
projection distortions of the acquired image (Fig. 2c).
The presence of distortion not only leads to the curvature of straight lines, but also leads to gross
errors in measurements taken with the captured image.
Elimination of distortion is usually performed in several stages:
1. The choice of a parametric distortion model,
2. Estimation of the parameters of the model,
3. Compensation of geometric distortions in accordance with the found parameters of the model.
It is essential that special calibration patterns with known parameters are used to estimate model
parameters. Such patterns can contain a chessboard pattern, a uniform grid of lines, and so on (see Fig.
3). It is usually necessary to perform a series of shots of the calibration pattern to calibrate the camera
using existing algorithms and software.
(a) (b) (c)
Figure 2. Calibration patterns often used for camera calibration: (a) uniform grid; (b) chessboard
pattern [5]; (c) asymmetric circles pattern [6].
The literature describes a number of models used in the evaluation and compensation of radial
distortion. Before proceeding to the description of models, let us introduce the following notations:
x=(x,y,z) is the point of the object in the coordinate system associated with the object;
u=(u,v) is the point on the image in the coordinate system associated with the image. The point of
the object is to be projected to this point (ideal position);
ud=(ud,vd) is the point on the image in the coordinate system associated with the image. The point
of the object is projected to this point due to geometrical distortions (real position).
Assuming that the center of distortion coincides with the origin, let us also introduce:
r u 2 v2 , rd ud 2 vd 2
,
where r is a distance from the origin to an ideally located point (u,v), rd is a distance from the origin to
the real point (ud,vd).
A quite complete review of models of radial distortion is given in [7]. The most well-known
polynomial model of radial distortion (1) [8] is considered among the existing polynomial models. A
special case of this model is a model with two coefficients (2) [9].
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 411
Image Processing and Earth Remote Sensing
E V Myasnikov
rd r (1 k1r 2 k 2 r 4 k3r 6 ...) (1)
rd r (1 k1r k 2 r )
2 4
(2)
For the latter model (2), the solution of the inverse problem is found in the form of an iterative
scheme.
The model of a more general form (3) [10] and its particular case (4) [11] are also considered.
rd r (1 k1r k 2 r 2 k3r 3 ...) , (3)
rd r (1 k1r k 2 r ) .
2
(4)
For the latter model (4), the solution of the inverse problem is obtained, both with the use of
approximating functions, and in the analytical form [7].
In rational models, the radial function has the form of a relation. Among the rational models, we
can mention model (5) [12], and also a rational model based on the ratio of simple polynomials (6),
proposed in [7].
1 4kr 2 1
rd
2kr (5)
1 k1r k 2 r 2
rd r
1 k3 r k 4 r 2 k5 r 3
(6)
Other models of radial distortion (for example, the fish-eye model) were proposed in the literature.
Tangential distortion is described using a model:
u d 2 p1uv p2 (r 2 2u 2 )
vd 2 p2 uv p1 (r 2 2v 2 ) (7)
It should be noted that methods for blind identification of radial distortion exist.
A number of works related to the evaluation of the quality of the above models are known from the
literature. From the analysis of such literature, it can be concluded (in particular, in [7]) that the use of
models containing more degrees of freedom (polynomial and rational models containing polynomial
functions of higher orders) makes it possible to obtain more accurate results in the sense of the
approximation quality. On the other hand, it was noted that using more degrees of freedom (functions
of higher orders) can lead to instability when using numerical methods to estimate the coefficients.
3. Methods
As it was outlined in the Introduction, in this paper we propose a technique for correction of the
complex geometric distortion induced by the considered mobile registration device.The developed
method is based on the use of the video recording of the calibration pattern (uniform grid with a
sufficient size). The developed method consists in the estimation of the field of correction of
geometric distortions using a frame with the subsequent aggregation over all evaluated frames.
Actually, the estimation of the field of correction of geometric distortions consists of the following
steps:
- Filtering the horizontal lines of the calibration grid using the Gabor filter bank;
- Search for the spanning lines of the calibration grid in the horizontal direction;
- Approximation of horizontal lines of the calibration grid using polynomials;
- Construction of the field of correction based on horizontal lines by interpolation of the received
polynomials;
- Filtering the vertical lines with the subsequent search for the spanning lines, and approximation
of vertical lines of the calibration grid using polynomials;
- Evaluation of the correction field for the frame by interpolation of the received polynomials.
An example of the distorted calibration grid is shown in Figure 3 (a). The image is slightly cut from
below to exclude the influence of auxiliary inscriptions. As it can be observed, the distortion has quite
a complex nature. Horizontal lines are substantially convexwith relation to the center of the
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 412
Image Processing and Earth Remote Sensing
E V Myasnikov
image(barrel-like), while vertical lines change from slightly concave (pillow-like) to almost straight.
In addition, local non-uniformities of the lines are noticeable.
At the first step of the method, for the input grayscale image I(x,y), we obtain filtered images
I1 ( x, y) and I1V ( x, y) by using Gabor filter banks, which are tuned to distinguish horizontal lines
H
(with orientation equal to 45, 67.5, 90, 112.5, and 135 degrees), and vertical lines (with orientation
equal to -10, 0, 10 degrees). The examples of the filtered images are shown in Figure 3 (b,c).
At the second step of the method, we perform the search of the local maxima of filtered images,
which may correspond to the spanning lines of the calibration grid:
H
i h..h
I ( x, y), if I1H ( x, y) max I1H ( x, y i) and I1H ( x, y) t
I 2H ( x, y) 1
0 , otherwise
,
V
I 2V ( x, y) 1
i w..w
I ( x, y), if I1V ( x, y) max I1V ( x i, y) and I1V ( x, y) t
0, otherwise
Then the tracing of horizontal and vertical lines is performed, and the sets
i
LH
xiH, j , yiH, j
j 1..nh
i
, i 1.. N H and LVi xiV, j , yiV, j j 1..nv , i 1.. NV of the pixels corresponding to the
i
vertical and horizontal lines are formed. Here NH is the number of horizontal lines, NV is the number of
vertical lines, nih is the number of pixels in the horizontal line LHi , and niv is the number of pixels in
the vertical line LVi .
Approximation of the horizontal LHi and vertical LVi lines is performed by the polynomials of
degree k:
P ( x) p0 p1 x ... p K x K
so that the coefficients piH,k and piV,k , k=1..Kof the polynomials are optimal in a least squares sense:
P x y
2
i min , i = 1..N
H
H
piH, k
( x , y )LH
i
P y x
2
i
V
min , i = 1..N V
piV, k
( x , y )LVi
The examples of the approximation by polynomials are shown in Figure 3 (d, e).
After construction of the distortion model, the correction of an input frame can be performed using
the interpolation of the coefficients of the polynomials of the constructed model. For example, in the
case of linear interpolation, the interpolated polynomial P() has the coefficients
pk pi,k (1 ) pi1,k , k=1..K.
Here polynomial Pi() correspond to the bottom (left) edge of the considered cell, and polynomial
Pi+1() correspond to the top (right), λ=(S-d)/S, where S is a linear size of the grid cell, and d is the
distance to the bottom (left) edge of the cell.
It is worth noting that horizontal and vertical lines do not necessary have to be approximated by the
polynomials of the same order.
Here we suppose that the optical axis is approximately orthogonal to the plane of the grid pattern
and the orientation of the camera and pattern is approximately known.
The advantage of the proposed approach is, on the one hand, the possibility of a compact
parametric description of the introduced distortions in the form of the coefficients of polynomials, and
on the other hand obtaining at the final stage an improved correction field that takes into account,
among other things, the local in homogeneities of distortions that occur when inexpensive systems are
used.
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 413
Image Processing and Earth Remote Sensing
E V Myasnikov
(a)
(b) (c)
(d) (e)
Figure 3. An example of the distorted calibration grid (a); the results of the Gabor filtering (b, c);
the approximation of lines by polynomials (d, e).
4. Experiments
In this section we provide experimental results of the developed technique. The experimental setup
contained the car digital video recorded system equipped with a wide-angle camera. The
characteristics of the equipment are as the following: 1920×1080 pixelsresolution, 30 frames per
second, 170 degreesview angle (diagonal, as stated by the manufacturer), MOV / H.264codec.
To evaluate the developed technique, we used 20 frames of the video, which were taken with 10
frames interval (total approximately 7 seconds of the observation of the grid pattern). For each frame
we performed the following operations:
- Undistortion of the image according to the technique described in the previous section;
- Filtering the horizontal and vertical lines of the calibration grid separately using the Gabor filter
bank;
- Searching for the spanning lines of the calibration grid using the correlation approach separately
in the horizontal and vertical directions;
- Search for the crosses of the vertical and horizontal lines;
- Evaluation of the projective transformation that compensates for the mutual mismatch between
the recording device and the calibration pattern;
-Transformation of all the points of the vertical and horizontal lines using the projective
transformation found at the previous step;
- Evaluation of the mean squared error over all the projected points with respect to their ideal
positions.
The dependence of the average reconstruction error evaluated using the above scheme on the
degree of the polynomials is shown in Figure 4. The estimated error in restoring geometric distortions
is approximately 0.7 pixels. The best value was obtained for the polynomials of the 10 degree. The
reported results were obtained for the model estimated using only one frame without aggregation.
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 414
Image Processing and Earth Remote Sensing
E V Myasnikov
Figure 4. The dependence of the reconstruction error on the degree of the polynomials.
5. Conclusion
A new technique is proposed in this paper for the correction of the complex geometric distortion
introduced by the mobile video registration device.
For the model constructed using only one frame, the estimated reconstruction error is
approximately 0.7 pixels. The advantage of the proposed approach is, on the one hand, the possibility
of a compact parametric description of the induced distortions in the form of the coefficients of
polynomials, and on the other hand obtaining at the final stage an improved correction field that takes
into account, among other things, the local inhomogeneities of distortions that occur when inexpensive
optical systems are used.
The future directions of research include the experiments with different mobile video recording
equipment, increasing the efficacy of the developed method, and the use of dynamic targets for the
evaluation of geometric distortions.
It is assumed that the results of research can be used potentially to improve the recognition of
vehicles and road signs [13], as well as to improve the quality of road assessment and traffic planning
[14].
6. References
[1] Camera calibration with OpenCV (Access mode: https://docs.opencv.org/2.4/
doc/tutorials/calib3d/camera_calibration/camera_calibration.html)
[2] Camera Calibrator (Access mode: https://www.mathworks.com/help/vision/ref
/cameracalibrator-app.html)
[3] Dashcam (Access mode: https://en.wikipedia.org/wiki/Dashcam)
[4] Myasnikov E 2017 Technique for processing imagery of plans and drawings obtained using
digital camera Procedia Engineering 201 353-362
[5] Open CV chessboard calibration pattern (Access mode: http://docs.opencv.org/
2.4.1/_downloads/pattern.png)
[6] Open CV asymmetric circles grid (Access mode: http://docs.opencv.org/2.4.1/_
downloads/acircles_pattern.png)
[7] Ma L, Chen Y and Moore K L 2004 Rational radial distortion models of camera lenses with
analytical solution for distortion correction International Journal of Information Acquisition
1(02) 135-147
[8] Slama C C 1980 Manual of Photogrammetry (American Society of Photogrammetry, fourth
edition)
[9] Zhang Z 1999 Flexible camera calibration by viewing a plane from unknown orientation IEEE
Int. Conf. on Computer Vision 666-673
[10] Hartley R and Zisserman A 2000 Multiple View Geometry (Cambridge University Press)
[11] Ma L, Chen Y and Moore K L 2003 Flexible camera calibration using a new analytical radial
undistortion formula with application to mobile robot localization IEEE International
Symposium on Intelligent Control
[12] Brauer-Burchardt C and Voss K 2001 A new algorithm to correct fish-eye- and strong wide-
angle lens-distortion from single images Int. Conf. on Image Processing 225-228
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 415
Image Processing and Earth Remote Sensing
E V Myasnikov
[13] Yakimov P Y 2015 Tracking traffic signs in video sequences based on a vehicle velocity
Computer Optics 39(5) 795-800 DOI: 10.18287/0134-2452-2015-39-5-795-800
[14] Agafonov A A and Myasnikov V V 2016 Method for the reliable shortest path search in
timedependent stochastic networks and its application to GIS-based traffic control Computer
Optics 40(2) 275-283 DOI: 10.18287/2412-6179-2016-40-2-275-283
Acknowledgments
The reported study was funded by RFBR according to the research project №17-29-03190.
IV International Conference on "Information Technology and Nanotechnology" (ITNT-2018) 416