Panoramic images are widely used as a means of information support for various control systems. Creation of panoramic images can be performed using specialized hardware, but this is associated with high financial costs and requires serious professional photography skills. As an alternative to specialized devices, you can use ordinary cameras, turning the camera in the desired direction and taking a sequence of frames, and then using computer vision methods to stitch the original frames and get a panoramic image. Computer vision methods allow obtaining panoramic images in the general case according to the following algorithm [2]:
1. Detection of special points of images and their comparison; 2. Construction of a projective transformation for aligning images and transferring them to a common plane; 3. Stitching images aligned relative to each other.
To construct a feature description of an image, it is necessary to select the characteristic parts of the image, for example, corners, edges, regions corresponding to extrema of intensity, etc. Algorithms that highlight such features (key points) should be invariant to various transformations: displacement, rotation, zoom and illumination of the original image, as well as the position of the camera relative to the captured object (change in perspective). To search for interpreted information on the image, it is necessary to link to the local features of the image.
Different algorithms for selecting key points do not provide universal solutions for different images due to the specifics of determining local features. In this study, it is proposed to use various sensor data to select the most appropriate algorithm for searching for key points, depending on the scene in the image, the angular displacement between frames, illumination, and other parameters.
The technical literature is rich in new detection features and image description algorithms [1]. However, to this day, there is no ideal detector [4]. This is mainly due to the almost infinite number of possible computer vision applications (which may require one or more functions) [5], the discrepancy in the image conditions (zoom, viewpoint, lighting and contrast, image quality, compression, etc.) [2] and the possible scene [6]. The computational efficiency of such detectors becomes even more important when considered for real-time applications [3].
Three algorithms (SIFT, SURF, ORB) were studied in detail and the following conclusions were made:
1. ORB algorithm -the fastest algorithm, but with a lower percentage of matches among other algorithms. 2. The SIFT algorithm is the slowest, but at the same time it surpasses other algorithms in terms of percentage coincidence in most cases of frame distortion considered. 3. The SURF algorithm is close enough in percentage coincidence to the SIFT algorithm and is close in speed to the ORB algorithm. 4. It is important to note that the ORB algorithm finds key points mainly in the center of the image, while the SIFT and SURF algorithms are evenly distributed over the entire image.
When creating panoramas, you can use various auxiliary data of the device from which the shooting was carried out: based on a timer, an encoder, a gyroscope, or other sensors. Smartphones are often used to capture panoramic images, the image below shows the various sensors found on most smartphones. For general control of the initial set of images, control of overlapping between frames and offsets along the axes, angular displacements, you can modify the basic algorithm based on the OpenCV library as shown in the block diagram below. As can be seen from the proposed block diagram, before starting the algorithm, it is planned to analyze the position of the camera between images to warn the user about the uselessness of processing this set of frames. With further stitching of images, it is proposed to estimate the displacement between frames, and, accordingly, the total area of overlap between frames. This approach will allow full control of the original images for the suitability of stitching into a general panoramic image, as well as control between individual frames, stopping the algorithm when the general overlaps between images are lost, and at the output of the algorithm, the user will not be provided with a full panorama, but correct in terms of image integrity. Figure 1 below shows the minimum required equipment for using the developed method and briefly shows the algorithm.
To test the developed method, use the PASSTA datasets (ie, image sets) of Linkoping University. These image sets have a few functions:
1. Images were taken from a camera mounted on a tripod. 2. Between each subsequent image, the camera rotates around the vertical axis through the optical center. 3. Small enough.
The dataset includes three sets of images. Sets of images were used for the experiment:
1. Blue Dining Room: Contains 72 images captured with the Canon DS50, perspective lenses with a poor resolution of 1280 x 1920 pixels under lighting. The panoramic head was used to rotate approximately 5 degrees around the vertical axis around the optical center of the camera. 2. Dining Room: Consists of 72 images captured with Canon DS70 Samyang 2.8 / 10mm wide-angle lenses (about 105 degrees), with a resolution of 5740x3780 pixels. The panoramic head was used to rotate approximately 5 degrees of the vertical axis around the optical center of the camera. Figure 11 shows a comparison diagram with different input data for the developed method and the standard method library OpenCV using different methods for finding key points. Defects in the final images are indicated separately. Table 1 below shows the results obtained with various input data and methods used. The operating time of the developed method was estimated for various sets of initial images. Based on the data obtained, the following conclusions can be drawn:
1. The developed method, regardless of the displacement between frames, creates a panoramic image in approximately the same time 2. With an angular displacement between frames up to 45 degrees for light scenes and up to 40 for dark scenes, the developed method does not create obvious defects in the final image.
The analysis of existing methods for creating panoramic images carried out, methods for finding key points in images using computer vision are analyzed in detail. As a result of the analysis, it was concluded that the presence of different methods is dictated by the difference in applied problems and, accordingly, objects in the images that require the search for key points.
A method was developed for creating panoramic images using multisensory data based on the OpenCV library in the Python programming language. To improve the quality of the created panoramic images by using the most suitable key point search algorithm for scenes on the original frames, as well as to control the mutual overlap between frames, shifts and displacements, it was proposed to add data from position sensors (gyroscope and accelerometer) to the algorithm. Choosing the optimal algorithm for finding key points also allows you to reduce the total running time of the algorithm without losing quality. It can be concluded that multisensor data is useful for creating panoramic images. At the same time, the developed method can be implemented in an embedded system due to a decrease in the operating time due to the use of an optimal algorithm for finding key points. The developed method, using the PASSAT dataset as an example, made it possible to obtain a final panoramic image of about 1.33 Mb in size in 16 seconds, regardless of the number of frames used (11, 8 or 6) with an angular displacement of (25/35/45) degrees, respectively.
The developed method is adapted for expansion and use with other algorithms for finding key points, as well as the use of various sensors.