Driving fraught with an uncontrolled risk of small collisions with other vehicles and pedestrians because of "blind spots": at parking, at exit from a parking lot, while driving in heavy trac of vehicles. For trucks the task of creating 360 around view systems complicated by large size, small overview of a cabin and the presence of turning trailers, and at the moment the sort of system is not produced. The 360 around view system being developed will solve specied problems, getting images from 4 cameras and displaying a single image of a vehicle and its environment as a "bird's eye" on a monitor in real time. The article describes the existing 360 around view system for passenger cars, given their advantages and disadvantages, describes the current development by Volvo Trucks. The article describes the proposed approaches for calibrating the camera, removing of barrel distortion. The article contains results of calibration and removal of barrel distortion for the camera 180degree Fisheye Lens 1080p Wide Angle Pc Web USB Camera.
Driving fraught with an uncontrolled risk of small collisions with other vehicles and pedestrians because of "blind spots": at parking, at exit from a parking lot, while driving in heavy trac of vehicles. For trucks the task of creating 360 around view systems complicated by large size, small overview of a cabin and presence of turning trailers, and at the moment such a system is not produced.
Every year on Europe’s roads in accidents involving trucks more than 7
thousand people dies and about 100 thousand receive damage [
1. 5% of accidents in which the victims are drivers and passengers of cars (clash of sides at evolution).
2. 35% of accidents in which the victims are unprotected road users (pedestrians, cyclists and motorcyclists).
Scientic and technical problems which are solved by our work are the development of software and hardware for the 360 around view system of large vehicles, including algorithms and mathematical models to display vehicle’s environment in real-time as bird’s-eye view on the monitor. The result of the work will be hardware-software complex consisting of 4 (or more, depending on the results of experiments) wide-angle sh-eye camera, hardware unit, monitor, interface and software that uses developed video processing algorithms. This complex will be display a single image of a vehicle and its environment as a "bird’s eye" on a monitor in real time, getting the picture from 4 cameras. The system will be able to estimate the distance to objects in the environment, and warns of the dangers of proximity. In addition, it will include the parking assist system with "parking lines".
The advantages of the complex will be using it at speeds higher than in the existing analogues, working at low light levels and applying for small and for large vehicles. In addition to dimensions, large-sized vehicles are special by rotary parts, like trailers of trucks and articulated buses. The complex being developed will take into account this feature. 2
around view systems
Currently there are systems such as being developed, but only for passenger
(small-sized) cars. In 2007, the rst such system Around View Monitor was
presented by Nissan Motor Co., Ltd ,[
Besides Inniti and Nissan, such automakers as Audi, BMW, Lexus,
Mercedes Benz and Toyota have their own 360 around view systems. Almost all of
them co-operate with external chip manufacturers such as Freescale
Semiconductor, whose industry-specic solutions can be easily congured to work with
the equipment of any automakers[
Some automakers also oer a wide and ultra-wide front and rear view, for
example, such as BMW SUVs. Land Rover oers a technology Transparent
Bonnet, which allows "to see through the hood", when you’re climbing over the hills
and other obstacles. In addition, it is possible to add a fth camera at the top
of the liftgate for a less distorted rear view[
VW also oers some additions in its technology Area View[
2. The system has 17 dierent virtual camera positions. It is concentrated so that it guarantees the view of the vehicle and the area around it in every conceivable perspective.
3. While driving at low speed over rough terrain or in poor underfoot conditions front camera mode "Oroad", which provides the ability to recognize such obstacles as large rocks, stumps or holes, is available.
Research works on designing of the 360 around view system for cargo
(largesized) car launched in 2010 by Volvo Trucks, the completion of works is planned
by 2020[
According to the report [
During our research of the existing analogs, we have identied some of their shortcomings.
In systems for small vehicles : 1. Remain blind spots .
2. The distortion in the periphery of camera visibility => The distortion of the form and the size of objects.
3. Its work at low vehicle speeds.
For large vehicles are added : 1. Large size => Increasing the amount of blind spots . 2. Rotary trailers => Increasing complexity of image stitching.
To solve the problems with the presence of blind spots and distortion, we decided to use ultra wide-angle sh-eye camera. Description of the methods for its use described in the next section. 3
Since the main task is to obtain a picture of the car from the top, and the
system being developed uses a wide-angle camera such as sh-eye (Fig. 1), then
we need to get from each of 4 cameras its orthographic view from above[
In its turn this procedure includes three sub-procedures: 1. Camera calibration. 2. Getting an image without barrel distortion using the calibration results.
In the study, we analyzed the following methods and algorithms: 1. Algorithm by Luis Puig and others[13]. It is based on the use of DLT (Digital Linear Tape) and is used for catadioptric cameras. This method uses a three-dimensional model of a calibration chessboard.
2. Algorithm by Joao P. Barreto and Helder Araujo[14]. The method is based on special properties of conical curves that allows to calibrate the camera using at least three straight lines. It is used for catadioptric cameras.
3. Algorithm by Christopher Mei and Patrick Rives[15]. It is used for omnidirectional cameras. Based on the generalized matrix of the projection of sh-eye camera. For the calibration a few pictures of the chessboard is used.
4. Algorithm by Ellen Schwalbe[16]. This method is used for the sheye cameras. It is based on their property that there is approximately linear relationship between the angle of incidence of the beam to the point of the object in the image and the distance to it. Calibration is carried using a set of points distributed around the room.
5. Algorithm by Tsudikov M.B.[17]. It uses the same properties sheye camera, as the previous algorithm, but it is used to get rid of the distortion.
6. Algorithm by Biryukov E.D.[18]. This method allows to obtain "top view" for fragments of images, located away from the optical axis of the camera. It is based on the construction of a grid at a fragment and applying to it the linear transformations that result is a rectangular area. The author acknowledges that barrel distortion is not completely removed.
7. Algorithm by Davide Scaramuzza.
The method of camera calibration by Davide Scaramuzza was selected on the basis of the comparative analysis and taking into account the data of the research presented in [19]. The advantage of this method is the absence of using of the parametric model, which individual for each specic camera that allows unication. To determine the distance to the point of the space in the image, this approach uses a function based on a Taylor series: f (u00; v00)==pa0 + a1 00 + a2 002 + ::: + aN 00N , where u002 + v002, u00 and v00 image point’s coordinates in the plane of the camera, a scaling factor.
This method is described in more detail in [20].
The methodology is implemented as a tool "OCamCalib" to the MATLAB[21]. This tool is used by organizations such as NASA, PHILIPS, BOSCH, DAIMLER. For the calibration we used a set of images of "chessboard" (Fig. 2). After downloading the images (minimum - 9) and entering the dimensions of the calibration board, the program automatically determines the contact points of angles of the grid. In this paper we use the board size of which is 9x7 cells with an edge length of 23 mm.
At the next stage center of the grid is calculated, and on the basis of all available data, distortion of the board and the distance to all corners of the grid with a margin error is calculated.
Because calibration accuracy is very important for our system, we determined on the basis of experiments the conditions under which its results the most adequate. The intermediate results of the algorithm, as the average reprojection error (hereinafter "err") of grid cells (that is the dierence between the calculated and the real size, position and shape of the cells), were used to determine the quality of calibration and its dependence on parameters such as:
Depending on the requirements to the value of the error it is possible to determine the minimum of required number of images (Table 1).
The experiment showed that the number of grid cells does not signicantly aect the results of the calibration (Table 2). Variant of illumination daylight room light table lamp room light + backlighting by a lantern err 11.390 1.879 5.289 0.478
As seen from the experimental results, the method is sensitive to the degree of illumination of the space and to the calibration board (Table 3). This dependence the results of the lighting is due by harshness the image at a lower level of illumination, that is the dierence between the white and black cells is reduced.
The result of the calibration is the above function, or rather its coecients, which are used in the second sub-procedure. It was also performed using the tools Davide Scaramuzza, but already implemented as an application in C ++ using library OpenCV.
Since we develop the 360 around view system, as the input image used image (Fig. 3), on which the calibration board occupied area, presumed to output on the monitor. The result of the program was corrected image without barrel distortion. Fig. 4 shows that distortion remained insignicant (indicated by green lines).
The paper presents the research and analysis of the existing 360 around view systems review for cars. It describes their strengths and weaknesses.
Also we analyzed the algorithms of omnidirectional camera calibration, and algorithms for removal of distortion. Based on the methodology Davide Scaramuzza we obtained calibration data for the camera 180degree Fisheye Lens 1080p Wide Angle Pc Web USB Camera. The paper shows the results of the calibration for various parameters. We obtained images without barrel distortion.
Results of the rst two stages will be used at the next stage, while receiving an orthographic top view. Research will be focused on the development of the algorithm, which allows to get a horizontal projection of the front view (and others) of the car. The constancy of this area makes it easier and allows making calculations once and further applying them. We are going to calculate the dependence of the conversion results on the height of the camera above the ground surface and angle of tilt to it. The quality of the results will be checked by comparison with photo above of the eld. 13. Puig, L., Bastanlar, Y., Sturm, P., Guerrero, J. J. and Barreto, J., 2011. Calibration of central catadioptric cameras using a DLT-like approach. International Journal of Computer Vision, 93(1), pp.101-114. 14. Barreto, J. P. and Araujo, H. , 2005. Geometric properties of central catadioptric line images and their application in calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 27(8), pp.1327-1333. 15. Mei, C. and Rives, P. , 2007, April. Single view point omnidirectional camera calibration from planar grids. In Robotics and Automation, 2007 IEEE International Conference on (pp. 3945-3950). IEEE. 16. Schwalbe, E., 2005, February. Geometric modelling and calibration of sheye lens camera systems. In Proc. 2nd Panoramic Photogrammetry Workshop, Int. Archives of Photogrammetry and Remote Sensing (Vol. 36, No. Part 5, p. W8). 17. Tsudikov, M. B., 2011. Reduction of the image from the type chamber "Fish eye" to the standard television. [Privedenie izobrazheniia iz kamery tipa "Rybii glaz" k standartnomu televizionnomu.] Izvestiia Tul’skogo gosudarstvennogo universiteta.
Tehnicheskie nauki, (5-1). (in Russian) 18. Biriukov, E. D., 2015. Image correction algorithm from the wide-angle camera of car rear view. [Algoritm korrektsii izobrazheniia s shirokougol’noi kamery zadnego vida avtomobilya.] Novye informatsionnye tekhnologii v avtomatizirovannykh sistemakh, (18). (in Russian) 19. Lazarenko, V. P., Dzhamiykov, T, S., Korotaev, V. V., Yaryshev, S. N. Transformation algorithm for images obtained by omnidirectional cameras. [Algoritm preobrazovaniia izobrazhenii, poluchennyh vsenapravlennymi optiko-elektronnymi sistemami.] Nauchno-tekhnicheskii vestnik informatsionnykh tekhnologii, mekhaniki i optiki 15, no. 1 (2015): 82-90. (in Russian) 20. Scaramuzza, D., Martinelli, A. and Siegwart, R. , 2006, January. A exible technique for accurate omnidirectional camera calibration and structure from motion. In Computer Vision Systems, 2006 ICVS’06. IEEE International Conference on (pp. 45-45). IEEE. 21. Scaramuzza, D. OCamCalib: Omnidirectional Camera Calibration Toolbox for Matlab. Available at: https://sites.google.com/site/scarabotix/ocamcalibtoolbox