An algorithm for the video panorama construction from distributed multispectral cameras data is described. It is shown that operations of vision enhancement (modified Multiscale Retinex algorithm) and multispectral image fusion are implemented for two independently chosen regions of interest with a frame size of 1024x768 pixels and 30 fps using CUDA technology.
The real-time automatic generation of high-resolution video panoramas from information of several cameras with partially overlapping fields of view (FoV) is one of the modern trends in the vision systems development. Generally, panorama navigation implies the presence of a user-controlled region of interest (RoI). This approach is an alternative for mechanical drive-based vision systems, as it ensures simultaneous operation of several users with an independent choice of a personal RoI without mechanically moving the camera system. Another advantage of distributed panoramic systems (DPS) is the integration of video cameras into the body / fuselage of the carrier object, what positively affects its aerodynamic properties.
There are two basic approaches for panorama stitching:
an approach based on finding the correspondence of homogeneous pixel coordinates on the
frames of cameras with i and j numbers by detecting and matching keypoints using their
descriptors and estimating the homography matrix Hij [
The advantage of the first approach is operability even in the absence of a priori information about the mutual placement of DPS cameras; the advantage of the second approach is operability in difficult observation conditions and in low-contrast observable scenes.
It is known [
The analysis of publications [
Restrictions on the use of these methods are situations where the image in one of the video channels (usually in TV) does not allow selecting of the important details of the scene for matches searching (for example, in near to zero illumination, as well as in dense smoke or fog).
The solution of the superimposition problem can also be achieved by mechanical alignment, which is to ensure the parallelism of the sighting lines and the same angular dimensions of the field of view of the each spectral range cameras, as well as their mutual placement minimizing parallax, but this approach is not applicable for the DPS.
Photogrammetric calibration of DPS cameras is the most universal approach for combining images in
the far zone, but requires the use of a test pattern that have high contrast simultaneously in several
spectral ranges. Examples of such patterns with typical "chessboard" image are considered in [
1. Initialization: calculating quaternions quv0, that specify the initial angular direction to the pixels
of RoI [
Main operation cycle 2. Estimation of the current angular position of the DPS reference camera by pitch and roll (for example, according to the data from the inertial measurement unit) and corresponding rotation matrix:
cos Rθ sin 0 sin cos 0 01 00 10
0 cos sin
0 sin cos .
(1) 3. Calculation of the rotation quaternion for a given angular position of the line of sight (center of RoI) qvis and quaternions quv, determining the current angular position of the RoI pixels (u, v):
quv = qvisquv0. (2)
4. Filling RoI for each spectral range with pixels from cameras by re-projecting points from the
surface of a unit virtual sphere to their matrices (with lens distortion compensation [
Since for each pixel of the RoI the processing according to the above algorithm is homogeneous, this allows us to apply the procedure of parallelizing the computations, for example, using the resources of the GPU.
The layout of the DPS is a development of previous authors’ work [
For filling of the RoI the computations are divided into parallel blocks (64 horizontally and 48 vertically) with 256 threads in each (16 threads horizontally and 16 vertically) using CUDA and CUDA C. As copying of the data of the CPU memory into the GPU memory and back is relatively slow, the number of such operations is minimized in our implementation of the video panorama.
In the DPS layout are implemented:
angular position control of the line of sight of the operator: according to the data from the head
tracking system or (if not) from the joystick;
independent display of two RoIs with a dynamic change of its field of view from 80°60° (wide
angle) to 10°7.5° (telephoto angle);
blending according to the algorithm [
As the DPS layout currently contains a single thermal camera, the fusion mode is realized only at angular positions of the RoI containing a part of area 6 (see figure 2). Otherwise, the user in this mode receives information only from the TV cameras. This is illustrated in figure 5, where in the thermal camera and fusion modes the lower rows of the RoI are filled with information from the TV cameras, because at the current position of the line of sight their angular coordinates do not fall in the field of view of the LWIR camera and therefore do not contain data in infrared spectral band.
On the NVIDIA GeForce GTX 560 Ti GPU (384 cores) with the maximum amount of calculations (blending, false-color fusion of contrasted TV and IR channels) and the RoI size 1024768 pixels for each of the two operators, the rate of updating the information is 32 Hz. The calculation speed for other modes is given in the table, where for comparison, information on the processing speed is also provided when implementing a video panorama on an Intel Core i5 CPU. All values in the table are rounded to the nearest whole number.
As can be seen from the table, the use of parallel computations increases the processing speed by an average of 16 times.
With the use of CUDA technology the algorithm for creating a video panorama based on information from multispectral cameras for two windows of interest with a resolution of 0.7 Mp implements an independent display of video information and vision enhancement functions (Multiscale Retinex and pixel-level fusion) with a frequency of at least 30 fps.