Image normalization for Blurred Image Matching Yann Donon Alexander Kupriyanov Rustam Paringer CERN Samara National Research University; Samara National Research University; Geneva, Switzerland Image Processing Systems Institute of RAS - Image Processing Systems Institute of RAS - Samara National Research University; Branch of the FSRC "Crystallography and Branch of the FSRC "Crystallography and Image Processing Systems Institute of RAS - Photonics" RAS Photonics" RAS Branch of the FSRC "Crystallography and Samara, Russia Samara, Russia Photonics" RAS akupr@ssau.ru rusparinger@ssau.ru Samara, Russia yann.donon@cern.ch Abstract—Blurred Image Matching (BIM) is based on image measurement. In this paper, we used colour vector length and pre-processing and Blob detection. BIM has been designed to base ourselves on arithmetic model mean [6]: function with images presenting a strong level of noise of 𝑛 different kinds. The technique shows an excellent robustness, 1 (𝑟𝑖 + 𝑔𝑖 + 𝑏𝑖 ) speed and unique features when compared to existing methods. 𝐵𝑟 = ∙ ∑ 𝑛 3 This article investigates the process BIM is based on, proposes a 𝑖=0 new way to improve the range of noise the technique can process Where n is the amount of pixels in the image and r, g and b, with a good range of success by adding image normalization. the value for each pixel in red, green and blue. Moreover, the article investigates the technique’s performances when confronted to different parameters, thus suggesting an ideal brightness for the blob detection to perform at the best of its capacities. Keywords—Features extraction, Noised images, key points detection, image processing, parameters selection I. INTRODUCTION Blurred Image Matching (BIM), is a key point detection method for images. Feature matching in images has been remains of major importance in current context [1] and has applications as images reconstruction [2] stitching [3].It is Fig. 1. Illustration of the objective of brightness normalization: Matching based on the comparison of large areas of interest in images, two images with different exposition. after a pre-processing involving Gaussian blurring and thresholding. The technique selects large areas identified using a connected component labelling algorithm and matches them from an image to another [4]. As shown in previous researches, one of the main issues in stitching images using BIM is their brightness level [4]. Thresholding algorithms are by essence heavily impacted by brightness levels. Therefore, it was difficult to find key points Fig. 2. Samples of images from the datasets used in the framework of this in images presenting different expositions. Different experiment. expositions can result from different angles of view, time of A. Define brightness equalization necessity for a set the day, metrological phenomena. Experimentations shows that BIM present optimal In this context, it was necessary, for many samples to first performance when the difference of brightness between normalize the images characteristics. This normalization images stays under 5%, especially when using shape would allow finding comparable shapes as used by BIM. contouring for comparison instead of quadrilaterals [7]. II. DATASET Therefore, the formula serving to assess the necessity of normalization presents itself as such The dataset used in this experiment are sets of aerial 𝐵𝑟1 − 𝐵𝑟2 pictures taken by drone, those images include a wide range of | | < 0.05 colorimetry and brightness in their original state. 917 Images 𝐵𝑟1 + 𝐵𝑟2 were used, divided in 4 categories. Those categories are sets B. Define ideal brightness of images representing the same area or an area of proximity Fig. 3 represents the amount of points found depending on to images brightness; it shows that the different sets of images It allowed us to determine whether a unified, ideal, with a significant range of characteristics, present comparable features range existed. These dataset contains only aerial areas of matching. Although colorimetry plays a role into views, however the experimentation on different kind of shape matching, it seems not to be a relevant feature images has already be reviewed and deemed insignificant [5]. regarding brightness [5]. It has been shown in previous The brightness experiments that specific colour channels has an influence of points matching after the thresholding operation [5]. Brightness is one of pixel’s most significant characteristics; however, there is no standard formula for its Meanwhile, Fig. 4 highlights that the totality of the points were found in between a brightness of 38 and 161 (23% of Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) Data Science the spectrum, hereafter referred as partial brightness Average brightness modification spectrum). Which highlight the importance of the brightness As shown on Fig. 6, where the X and Y axis are different for BIM processing. Moreover, the majority of the points, (- values of images’ average brightness, from -255 to 255. The 0.25σ to 0.25σ) are situated in a range between 89 and 113, results of successful matching are presented on a pair of or just under 7% of the full brightness spectrum and about diagonal Fig. 6.1.a, the two original images having different 32% of the partial brightness spectrum. brightness, there are two combination for each coordinate, for example (-21;33) and (-33;21) as shown on Fig. 5. A smaller brightness difference between pairs of images is showed on Fig. 6.1 and Fig. 6.2 (histogram 1 for the pair with the highest delta, histogram 2 for the pair with the lowest delta), the closer the two diagonals, the lower the delta, which results in a less significant brightness correction. The best results appearing when no brightness difference exists, then the two diagonals are merged into one. On Fig. 6, the area directly below (Fig. 6.1.b) the diagonal shows dispersed points; those are either noise or isolated points that are very distinctive shapes on an image. Brightness change has a lesser impact on such shapes; they are usually caused by a sudden change of colour in the landmark (such as a red roof in a green forest). The area directly above the diagonal (Fig. 6.1.c) is empty as it represents the part of the array where images are brightness Fig. 3. Comparison of amount of points found on the different datasets. Results are comparable between the different datasets. correction of both images diverge in opposite directions; any point is this area is extremely likely to be noise. 60 Average amount of points found 1.a 1.b 50 ) ) 40 30 20 10 2.a 2.b ) ) 0 0 20 40 60 80 100 120 140 160 180 200 220 Brightness Without histogram normalization Fig. 5. Pair of images having different, opposite brightness. They are "twin" Fig. 4. Amount of points found on an image relatively to its brightness level. points on the opposite diagonals of Fig. 6. The ideal interval is to be found where the average 1 2 1.a 1.c brightness of the final image lays in the interval defined in B Define ideal brightness and confirmed in 0. Therefore, the ideal interval is in a brightness range between 78 and 110, where shapes are the more distinctive 2.a 1.a and where brightness of both images is equal. Which leaves two areas, on the centre of both diagonals for which all images respect the following formula: 110 > 𝐵𝑟 > 78 III. HISTOGRAM NORMALIZATION Average brightness changes between pictures, whether they come from different exposition time or were taken at different time, with different environmental conditions. 1.b Concretely, our objective was to equalize histograms in order Fig. )6. Difference matching results' histogram. On the left, the image's to find the same objects of interest from an image to another average brightness difference is 120% higher than on the right. The original [5]. image is the same. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 128 Data Science Fig. 6.2.a also confirms the results obtained in II.B Define On this figure, the two source images (1.a and 1.b) are ideal brightness, it is then mentioned that 60% of the points similar. 2.a and 2.b shows both images respective brightness could be found in 32% of the partial brightness spectrum. The histogram. Follows 3.a and 3.b, which displays the images shape 2.a contains 33% of the given spectrum and contains after their treatment of brightness correction. There they give 62.8% of the points matched. This information also the impression to the human eyes to be perfectly similar. establishes a direct link bet ween the amount of points and However, it is possible to see slight differences on their their quality as defined in [4]. resulting histograms: 4.a and 4.b. Those differences are the result of an information alteration due to the process of 1. 1. brightness modification. Finally, 5.a and 5.2 show the resulting shape detection after BIM processing. The process of histogram normalization adjusts an image intensity and enhance its contrast [8]. Two features that are important for BIM to reach good performances. In most cases, this process is not necessary, but in the context of this research, the process of histogram normalization showed 2. 2. itself especially useful. By default, the histogram normalization function equalizes the images to their mean brightness according to the following formula [9]: 𝑓𝑖,𝑗 𝑔𝑖,𝑗 = 𝑓𝑙𝑜𝑜𝑟 ((𝐿 − 1) ∑𝑛=0 𝑝𝑛 ) 3. 3. This results in images having the same brightness, and consequently, as shown on Fig. 8, the two combinations presented previously (Fig. 5) are joined. Which is a significant advantage for us to find optimal points, as they are now joined on one area as shown on Fig. 8. However, as seen previously, histogram normalization uses the image’s mean to adjust brightness, resulting in a 4. 4. constant brightness of 128 [10]. Which as seen previously in B Define ideal brightness, is out of the recommended brightness range. It is therefore still necessary to define an ideal area C. Amounts of points found after normalization Normalization has two effects in the framework of our experiments, as highlighted previously it does correct 5. 5. brightness of images, equalizing their histogram. But also as it is one of the technique’s well known purpose is to improve an image’s contrast, which is extremely helpful to BIM’s processing [11]. As a result, on average 87% more points are found when histogram normalization is used in the pre- processing technique. Fig. 7. Process of image brightness normalization and matching, from the two source image with different luminosity, to the results, BIM’s shape 1 2 detection. IV. HISTOGRAM NORMALIZATION A. Equalization by channel Experiments showed that equalizing the source by channel (RGB) was not necessary and has indeed a negative impact as it was frequent that from an image to another the colour distribution would change (new building, area of terrain) and treat colours independently would in that context induce mistakes into the image. B. Process Fig. 8. Results of brightness correction and matching between a pair of The process of image matching with source image having images. The closer the color is to green; the more points were found with a combination of brightness correction. 1) a pair of images before histogram a brightness delta higher than 1.5% goes as displayed on Fig. normalization, 2) the same pair after histogram normalization. 7. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 129 Data Science D. Process V. COMPARISON In most ways similar to the process presented in chapter The following figure shows the distribution of point found III, the following process involves one more step, represented without and with histogram normalization. The second in Fig. 9 as 3.a and 3.b. This step corresponds to the image’s having a much higher variance and shows comparable histogram normalization by colour channel, resulting in the amount on its smaller sigma. brightness presented in 4.a and 4.b. The brightness of 3.a and As shown on Figure 10 histogram normalization by 3.b in 128, it is then corrected as in 5.a and 5.b. resulting in channel also flattens matching differences relatively to an average brightness ranged between -0.5σ and 0.5σ of the colours. As such colorimetry doesn’t have to be considered matched points distribution. in finding the ideal parameters. 1.a) 1.b) Although results are comparable with and without histogram normalization on the highest sigma, the brightness equalization and the independence taken from image’s colorimetry maintains the utility of this technique. However, our experiments showed steadier results when, using histogram normalization, images in the range of brightness between -0.25σ and 0.25σ were selected, which corresponds 2.a) 2.b) precisely to the same range showed in chapter 2.ranged between -0.5σ and 0.5σ of the matched points distribution. 60 Average amount of points found 50 40 3.a) 3.b) 30 20 10 0 4.a) 4.b) 0 20 40 60 80 100 120 140 160 180 200 220 Brightness Without histogram normalization With Histogram normalization 5.a) 5.b) Fig. 10. Represents the distribution of points found with and without brightness normalization. The curb using brightness normalization has a significantly higher variance. VI. RESULTS It occurs that the best approach to merge a set of images is to use image normalization, not only increases the amount of 6.a) 6.b) points found but most importantly equalizes all image’s brightness. However, this technique sets the image brightness to its mean, which is outside the ideal range for blob detection defined earlier in this article. Therefore, in order to keep to a minimum, the modifications inflicted to the image, it is best to bring images to the 0.3σ value of amount of points found relatively to brightness, 113. 7.a) 7.b) VII. CONCLUSION The experiment presented in this paper allowed to successfully determine sets of pre-processing parameters, while proposing a process modification allowing BIM to perform with a higher robustness than previously, introducing the treatment of new noise sources in BIM’s Fig. 9. Process of image brightness normalization and matching, from the two source image with different luminosity, to the results, BIM’s shape abilities range. detection. Using histogram normalization. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 130 Data Science ACKNOWLEDGMENTS [5] Y. Donon, R. Paringer, A. Kupriyanov and Y. Goshin, "Blur-robust image registration and stitching," Journal of Physics: Conference The research was supported by the Ministry of Science and Series, vol. 1368, no. 5, 2019. Higher Education of the Russian Federation (Grant # 0777- [6] S. Bezryadin, P. Bourov and D. Ilinih, "Brightness Calculation in 2020-0017) and partially funded by RFBR, project number # Digital Image Processing," International Symposium on Technologies 19-29-01135. for Digital Photo Fulfillment, San Francisco, 2007. [7] R. aringer, Y. Donon and A. Koupriyanov, “Modification of the blurry REFERENCES image matching method,” 2020, in print. [1] A. Zakharov and A.L. Zhiznyakov, "A method for feature matching [8] T.J. Priyanka Garg, "A Comparative Study on Histogram Equalization in images using descriptor structures," Computer Optics, vol. 43, no. and Cumulative HistogramEqualization," International Journal of 5, pp. 810-817, 2019. DOI: 10.18287/2412-6179-2019-43-5-810-817. New Technology and Research (IJNTR), vol. 3, no. 9, pp. 41-43, [2] V. Vlasov, A. Konovalov and S. Kolchugin, "Joint image 2017. reconstruction and segmentation: Comparison of two algorithms for [9] R.C. Gonzales and R.E. Woods, “Digital Image Processing,” few-view tomography," Computer Optics, vol. 43, no. 6, pp. 1008- Tennessee: Pearson International Edition, 2008. 1020, 2019. DOI: 10.18287/2412-6179-2019-43-6-1008-1020. [10] S.M. Pizer, E.P. Amburn, J.D. Austin, R. Cromartie, A. Geselowitz, [3] A. Ruchay, K. Dorofeev and V. Kalschikov, "A novel switching T. Greer, B.H. Romeny, J.B. Zimmerman and K. Zuiderveld, bilateral filtering algorithm for depth map," Computer Optics, vol. 43, "Adaptive histogram equalization and its variations," Computer no. 6, pp. 1001-1007, 2019. DOI: 10.18287/2412-6179-2019-43-6- Vision, Graphics, and Image Processing, vol. 39, no. 3, pp. 355-368, 1001-1007. 1987. [4] Y. Donon, R. Paringer and A. Kupriyanov, "Parameters selection for [11] J. Alkofer, "Contrast adjustment in digital image processing method Blurred Image Matching," TBP, 2020. employing histogram normalization," USA Patent US4731671A, 1988. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 131