Joint processing of images in two spectral channels for small objects detecting Valeriy P. Kosykh1 , Gennadiy I. Gromilin1 and Nikolay S. Yakovenko1 1 Institute of Automation and Electrometry SB RAS, Novosibirsk, Russia Abstract The article is devoted to the problem of detecting low contrast small-sized objects in two-color images with a powerful spatially non-stationary background. An increase of the detecting reliability is achieved through a combination of three factors: attenuation of the background based on the construction of its locally stationary model; improving the estimation of model parameters by excluding statistically significant outliers from the initial data; joint processing of two-color images with a weakened background component. A method of constructing a linear boundary for detecting a useful signal in a two-dimensional space is proposed. The performance characteristics of a two-channel detector of small-sized objects are presented. Keywords Low contrast small-sized objects detection, spatially non-stationary background suppression, linear separating boundary. 1. Introduction One of the urgent tasks of the Earth space monitoring is the early detection of potentially dangerous objects in near-earth space by analyzing images generated by onboard spacecraft equipment. A particular version of this problem is the detection of dim "point" objects, the sizes of which are determined mainly by the point spread function (PSF) of the equipment, against a random spatially non-stationary background created by the Earth’s surface and cloud cover. Preliminary suppression of the background, based on knowledge of its statistical characteristics, facilitates to effective selection of objects. In conditions of spatial non-stationarity, this is ensured by constructing a locally stationary background model from a fragment of an image near the supposed presence of an object [1, 2]. If the signal from the object has a wide spectral range, it is advisable to jointly process images of the same area of the observed space obtained in different spectral channels with different statistical characteristics of the background. The suppression of the main structural features of the background in each of the channels by its local models leads to a decrease in the interchannel correlation of the background component, while maintaining the correlation between the images of objects. Therefore, with joint processing, one can expect a decrease in the probability of a false alarm, or, for a given probability of a false alarm, an increase in the probability of detection. The most effective way of joint processing would be to build a multichannel local stationary SDM-2021: All-Russian conference, August 24–27, 2021, Novosibirsk, Russia " kosych@iae.nsk.su (V. P. Kosykh) Β© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 142 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 background model, however, even for two channels (i.e., when the size of the model is doubled), the computational costs for estimating its parameters increase manifold. Therefore, the work considers a compromise variant of joint processing of two-channel images, when at the first stage in each spectral channel the background component is suppressed using its locally stationary model, and at the second stage, as a result of the analysis of the experimentally estimated two-dimensional distributions of the residual background brightness, a useful signal is detected. In what follows, we shell assume that the analyzed images are discrete and in each channel can be represented in the form 𝐷(𝑖, 𝑗) = 𝐡(𝑖, 𝑗) + 𝐴 Β· 𝑂(𝑖 βˆ’ 𝑖0 , 𝑗 βˆ’ 𝑗0 ) + πœ‰(𝑖, 𝑗). (1) Here 𝑖, 𝑗 are the current coordinates, 𝐡 is the background component, 𝑂 is the normalized image of the object with the center located at the point (𝑖0 , 𝑗0 ) and with the amplitude 𝐴, πœ‰ is the random uncorrelated registration noise. Then, with the known background model 𝐡 ^ (𝑖, 𝑗), it is advisable to search for objects in the modified image ^ (𝑖, 𝑗) = 𝐴 Β· 𝑂(𝑖 βˆ’ 𝑖0 , 𝑗 βˆ’ 𝑗0 ) + πœƒ(𝑖, 𝑗), (2) ˜ (𝑖, 𝑗) = 𝐡(𝑖, 𝑗) + 𝐴 Β· 𝑂(𝑖 βˆ’ 𝑖0 , 𝑗 βˆ’ 𝑗0 ) + πœ‰(𝑖, 𝑗) βˆ’ 𝐡 𝐷 where πœƒ(𝑖, 𝑗) = 𝐡(𝑖, 𝑗) βˆ’ 𝐡 ^ (𝑖, 𝑗) + πœ‰(𝑖, 𝑗) is disturbance, consisting of the residual part of the background, in which, with a well-chosen model, the spatial correlation is significantly weakened, and the noise component of the original image. 2. Locally stationary background model If we can consider the background in some domain Ξ© as a realization of a stationary random process, then its model at the point (𝑖′ , 𝑗 β€² ) ∈ Ξ© can be represented as a linear combination of the image samples in its vicinity βˆ‘οΈ 𝐡^ (𝑖′ , 𝑗 β€² ) = 𝐷(𝑖 βˆ’ 𝑖′ , 𝑗 βˆ’ 𝑗 β€² )β„Ž(𝑖, 𝑗), (3) 𝑖,π‘—βˆˆπ‘Š where π‘Š ∈ Ξ© is the vicinity of point (𝑖′ , 𝑗 β€² ), β„Ž(𝑖, 𝑗) is a set of weight coefficients, which, according to the optimal linear prediction (OLP) [3], can be estimated from the condition ⎧ ⎫ ⎨ ⎬ βˆ‘οΈ [οΈ€ 2 ^ = arg min 𝐽(h) = 𝐷(𝑖′ , 𝑗 β€² ) βˆ’ h𝑇 d𝑖′ ,𝑗 β€² (4) ]οΈ€ h . ⎩ β€² β€² ⎭ 𝑖 ,𝑗 ∈Ω Here d𝑖′ ,𝑗 β€² is a vector consisting of lexicographically ordered samples of the vicinity π‘Š , β„Ž is a vector of similarly ordered weight coefficients. This approach allows the model to be adjusted to the current statistical characteristics of the background. 3. Reducing the influence of objects on the background model estimate The minimum of the functional 𝐽(h) is attained at ^ = (P𝑇 P)βˆ’1 P𝑇 c, h (5) 143 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 where the rows of the matrix P are the vectors d𝑇𝑖′ ,𝑗 β€² , and the components of the vector c are the values 𝐷(𝑖′ , 𝑗 β€² ), taken in the same order as the vectors d𝑖′ ,𝑗 β€² . This decision is optimal in the case, when all the samples of the image participating in (3)–(4) belong to the background. If the samples of the object are among them, the estimate of the background model will be distorted. In work [4], devoted to the recursive suppression of the background, this drawback is partially eliminated due to the weakening of the contribution of the image samples, the difference of which with the previously constructed model is large. We propose to introduce such a weakening directly into the procedure for estimating the coefficients of the model, which, of course, complicates the estimation, but makes it possible to exclude samples for which the model has not yet been built. Let h ^ be the set of coefficients obtained in (5), then the components ^ of the vector c = Ph are the β€œsmoothed” model values of the components of the vector c. We introduce a diagonal matrix K, the elements of which are calculated as {οΈ‚ 1, |c ^𝑖 βˆ’ c𝑖 | < 𝑑, K𝑖𝑖 = 0, |c ^𝑖 βˆ’ c𝑖 | β‰₯ 𝑑. The corrected expression for estimating the coefficients of the current background model will take the form ^ 𝑐 = (P𝑇 KP)βˆ’1 P𝑇 Kc, h (6) The threshold 𝑑 is determined by the current value of the residual between c ^ and c: βˆšοΈƒ ^ βˆ’ c)𝑇 (c (c ^ βˆ’ c) 𝑑=πœ… , 𝑁𝑐 βˆ’ π‘β„Ž where 𝑁𝑐 is the number of components in the vector c, and π‘β„Ž is the number of weight coefficients in (3). Rigorous justification of the coefficient πœ… requires certain assumptions about the type of residual distribution. In this work, this coefficient was selected experimentally according to the quality of background suppression near objects. By means of numerical simulation, it was found that the value πœ… = 2.5 provides a sufficiently effective suppression of the background and relatively small distortions of the amplitude of objects. 4. Detection of objects in two-color images. Linear detection boundary for a given false alarm probability As mentioned in the Introduction, it is assumed here that background suppression is performed in each spectral channel separately, and at the detection stage, the samples of the modified images with the same spatial coordinates are considered as vectors in the two-dimensional ˜ (1) Γ— 𝐷 space 𝐷 ˜ (2) (superscripts indicate belonging to the corresponding spectral channel): (︁ (1) (2) )︁𝑇 D𝑖𝑗 = 𝐷˜ ,𝐷˜ . 𝑖𝑗 𝑖𝑗 To construct in this space the boundary between the background and objects, which satisfies the given criterion of optimality of object detection, it is necessary to know the two-dimensional 144 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 probability density functions of the background and objects brightness. Unfortunately, in real conditions, instead of probability density functions, we can only obtain a two-dimensional histogram of the joint distribution of the background and object brightness. This histogram, on the assumption that the appearance of objects is a very rare event, can be considered as an estimate of the discretized probability density function of the background. As for the two- dimensional probability density function characterizing objects, it can be constructed only based on a priori data on their spectral and brightness characteristics. In this work, we assumed the presence of minimal information about objects, namely, only knowledge of the ratio of their average brightness in different spectral channels. This is not enough to determine the specific shape of the boundary, so here we propose to construct a linear boundary. This boundary should provide, for a given false alarm probability, the maximum detection probability, being guided by the experimentally obtained two-dimensional histogram of the background and the direction in two-dimensional space, given by the ratio of the average brightness of objects 𝐴 Β― (2) /𝐴 Β― (1) in different channels. 4.1. Evaluation of the boundary for a given false alarm probability According to expression (2), the modified images are obtained by subtracting the background component adjustable model from the original images; therefore, it should be expected that the average value of the brightness in them would be close to zero. Accordingly, we will construct a two-dimensional histogram 𝐻(𝐷 ˜ (1) , 𝐷 ˜ (1) ) of brightness in the space 𝐷 ˜ (1) Γ— 𝐷 ˜ (2) in the range [βˆ’π·max , 𝐷max ] with step 𝛿. To determine the linear boundary dividing the space 𝐷 ˜ (1) Γ— 𝐷 ˜ (2) in a given direction in accordance with a given false alarm probability 𝑃𝑓 π‘Ž , we calculate the 𝑅(π‘ π‘˜ , πœ™) β€” Radon transform [5] of normalized histogram. (Here π‘ π‘˜ = 𝑠0 + π‘˜π›Ώ is the distance from the origin in the space 𝐷˜ (1) Γ— 𝐷 ˜ (2) to the straight line along which the histogram is (1) integrated, πœ™ is the angle between the direction 𝐷 ˜ and the normal to this line). Then the distance 𝑠𝑑 from the origin to the straight line, corresponding to the given false alarm probability 𝑃𝑓 π‘Ž , is determined by the relations 𝑑 βˆ‘οΈ 𝑑+1 βˆ‘οΈ 𝑅(π‘ π‘˜ , πœ™) < 1 βˆ’ 𝑃𝑓 π‘Ž , 𝑅(π‘ π‘˜ , πœ™) β‰₯ 1 βˆ’ 𝑃𝑓 π‘Ž . (7) π‘˜=0 π‘˜=0 These relations show at what distance from the origin of coordinates the straight line given by the angle πœ™ should lie, providing the given probability of false alarm, and exhaustively determine the set of straight lines that can be considered as possible variants of the boundary. 4.2. Choosing the border direction based on the known ratio of the average brightness of objects Apparently, with such scant information on the distribution of the objects brightness, the only justified choice of the detection boundary direction is the choice of such an angle πœ™ at which ˜ (1) Γ— 𝐷 the straight line satisfying (7) intersects in the space 𝐷 ˜ (2) direction to objects at the 145 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 point closest to the origin. The normal equation of a straight line satisfying (7) has the form ˜ (1) cos πœ™ + 𝐷 𝐷 ˜ (2) sin πœ™ βˆ’ π‘ π‘Ž = 0, and the direction to objects, taking into account the above, is set by the straight line (2) ˜ (1) 𝐴 𝐷 Β― (2) ˜ 𝐷 = . 𝐴¯ (1) It is easy to show that the distance from the point of intersection of these straight lines to the origin is equal to ⎯ [οΈ‚ ]οΈ‚ ⎸ 2 (︁ Β― (1) )︁2 (︁ Β― (2) )︁2 ⎸ ⎸ π‘ π‘Ž 𝐴 + 𝐴 (8) ⎸ 𝐿(πœ™) = ⎷ (︁ ⎸ )︁2 . Β― (1) cos πœ™ + 𝐴 𝐴 Β― (2) cos πœ™ The minimum of this expression with respect to πœ™ determines the sought boundary for detecting objects. The second way to choose the direction of the border is to choose, taking into account (7), a straight line orthogonal to the direction to the objects. For this straight line, the angle πœ™ must satisfy the condition Β― (1) sin πœ™ + 𝐴 𝐴 Β― (2) cos πœ™ = 0. Since the angle πœ™ takes only discrete values, its estimate, provided that relations (7) are satisfied, is obtained by minimizing the expression βƒ’ βƒ’ βƒ’ Β― (1) Β― (2) cos πœ™βƒ’βƒ’ . 𝐽(πœ™) = ⃒𝐴 sin πœ™ + 𝐴 (9) We will compare these methods of constructing the boundary in the experimental part of the work. 5. Computational experiment For the computational experiment, we used multispectral images obtained by the Landsat-7 satellite [6]. The purpose of the experiment was to test the above technique for identifying small-sized objects in two-color images with a complex spatially non-stationary background. Pairs of images obtained synchronously in different spectral channels made up two-color images and served as a spatially non-stationary background, on which images of small-sized objects were additively applied. 5.1. Formation of objects images Since one of the objectives of this study is to assess the probability of detecting objects with an accuracy of 10βˆ’3 , the experiment should contain a representative (∼ 105 ) sample of their images. Therefore, firstly, more than 600 objects were simultaneously applied to the background 146 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 Figure 1: Dependence of the object shape on the position relative to the PSE lattice. image, and secondly, the application procedure was repeated many times (more than 100 times) with a change in the position of the objects. Let us note one, in our opinion, rather important feature of the formation of images of objects, which consists in the following. The dimensions of the images of β€œpoint” objects projected into the focal plane of the receiving equipment are determined by its PSF. As a rule, to maximize the signal-to-noise ratio, the dimensions of the photosensitive elements (PSE) of the receiver and the characteristic PSF size are matched in such a way that the integral radiation power from the object incident on the area of one PSE is approximately 60–80% of the total power falling from the object onto photodetector [7, 8]. Therefore, the shape of the image formed by the photosensitive elements of the receiver depends significantly on the position of the object relative to the elements recording it. In the experiment, when generating images, this fact was taken into account by specifying various displacements of objects, which are not multiples of the step of placing the PSE [9]. Figure 1 shows examples of objects images located in various ways relative to the PSE array of the receiver. 5.2. Background suppression using a locally stationary model In the experiment, the domain of the model for predicting the background value at the image point (𝑖′ , 𝑗 β€² ) was its square neighborhood π‘Š of 7 Γ— 7 pixels with a 3 Γ— 3 pixel square cut out in the center. As the region of stationarity of the background Ξ©, according to which the parameters of the model were estimated, the neighborhood of the current point with a size of 13 Γ— 13 pixels was considered. The sizes and shapes of the domains Ξ© and π‘Š were selected experimentally. Background suppression was performed according to expressions (2)–(6). Figure 2 illustrates the effectiveness of background suppression without correction and with correction of the objects occurrence in Ξ©. Figure 2, a shows a fragment of a satellite image obtained in the spectral range of 0.45 Γ· 0.515 πœ‡m with objects applied to it, the average brightness of objects is ∼ 2.2 RMSD of the background brightness calculated over the entire image. All data are presented in a single brightness scale. The background standard deviation in the original image is 100, the average object brightness is 212, and the object brightness standard deviation is 12.7. In Figure 2, b, this fragment is shown after suppressing the background based on the model (5). The background RMSD is 10.8, while the average brightness of objects is 158, 147 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 a b c Figure 2: Background suppression using a locally stationary model. and their RMSD is 23.8. In Figure 2, c β€” the same fragment after suppressing the background by applying the model (6) (the RMSD of the residual background is 10.6, the average brightness of objects is 183, RMSD is 21.9). Experiments with images of different spectral ranges confirm the assumption that the use of a locally stationary background model makes it possible to weaken the background component many times (∼10 times) when the quality of object images deteriorates by about half. 5.3. Estimation of the boundaries of a given false alarm probability using two-dimensional histograms The images of β€œpoint” objects to be detected, as shown in Figure 1, are small outliers with a brightness exceeding the brightness of neighboring pixels, therefore, after suppressing the background, it is advisable to search for them among the local maxima of the modified images. The first step in processing a pair of images belonging to different spectral channels is to split the images into regions of the background and objects. Since the objects were applied artificially, a b c Figure 3: Family of boundaries with 𝑃𝑓 π‘Ž = 10βˆ’4 for two-color images. 148 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 such a partition is easy to implement. Further, for each of the regions, two-dimensional histograms of the maxima are constructed. Pairs of maxima are selected in each of the images and have the same (or close) coordinates. Based on the histogram belonging to the background region, according to expression (7) and the given false alarm probability 𝑃𝑓 π‘Ž , the possible linear boundaries of the space division 𝐷 ˜ (1) Γ— 𝐷˜ (2) are calculated. Figures 3, a and b show fragments of two synchronous images of the same area of the terrain, obtained in the spectral ranges of 0.45Γ·0.515 and 1.55Γ·1.75 πœ‡m. Figure 3, c with blue color shows a two-dimensional histogram of the distribution of maxima in the background area of their modified images, green β€” a family of straight lines dividing the space 𝐷˜ (1) Γ— 𝐷 ˜ (2) in accordance with a fixed false alarm probability 𝑃𝑓 π‘Ž = 10βˆ’4 at different angles πœ™. Two straight lines are selected from this family, one of which, 𝐿𝑇 , provides a minimum to functional (8), the second, 𝐿𝑁 , to functional (9). 5.4. Evaluation of the object detection probability The second stage of the experiment is to estimate the detecting probability of objects that are applied on the original background images. Similarly to the histogram of the background maxima for the area of objects, a histogram of the maxima of objects is built and, taking into account the already found boundaries, the probabilities of detection 𝑃𝑑𝑒𝑑𝑇 and 𝑃𝑑𝑒𝑑𝑁 are calculated. In Figure 4 for the above frames are shown in blue β€” the histogram of the background maxima distribution, in red β€” the histogram of the objects maxima distribution, the green straight line indicates the direction determined by the a priori known ratio of the average brightness of objects 𝐴¯ (2) /𝐴 Β― (1) in different channels. Straight lines mark the boundaries corresponding to the false alarm probability 𝑃𝑓 π‘Ž = 10βˆ’4 . Boundaries drawn independently from the first and second frames are shown in white, the 𝐿𝑇 boundary is indicated in cyan, and the 𝐿𝑁 boundary is in magenta. The summary result of the experiment with this pair of images is shown in Figure 5, a. Here, the dependence of the objects detecting probability on the angle πœ™, which determines the orientation of the boundary, is shown for three values of the false alarm probability 𝑃𝑓 π‘Ž : 10βˆ’2 , 10βˆ’3 and 10βˆ’4 . The blue points on the graphs correspond to the boundaries determined Figure 4: Background and object maxima distributions with boundaries of a different type. 149 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 a b Figure 5: Dependence of the detection probability on the orientation of the boundary. according to the minimum of the functional (8), purple β€” to the minimum of the functional (9). A significantly worse detection quality with the same false alarm probability values is obtained for a pair of images with close spectral ranges of 0.45–0.515 and 0.525–0.605 πœ‡m (Figure 5, b). Here, at 𝑃𝑓 π‘Ž = 10βˆ’4 , the graph of the detection probability is raised by 0.8 to be in the range of 0.8 Γ· 1.0. Nevertheless, as in the previous case, for all three values of 𝑃𝑓 π‘Ž the boundaries corresponding to functional (8) provide a higher detection probability than the boundaries corresponding to functional (9). 6. Conclusion The paper proposes an approach to solving the problem of detecting small-sized objects in a pair of synchronous images of a scene with a powerful spatially non-stationary background, obtained in different spectral ranges. It is shown that the separate application of locally stationary models to describe the background to each of the images makes it possible to weaken their background component many times (up to 10 times). The exclusion of statistically significant outliers from the data used to estimate the parameters of the model makes it possible to significantly reduce (by an average of 20%) the distortion of the objects amplitude. A method for joint processing of a pair of images with a suppressed background is proposed, based on the analysis of two- dimensional histograms of the background brightness, which ensures the detection of objects with a given false alarm probability. Assuming that only the ratio of the average values of the brightness of objects in different spectral ranges is known, it is proposed to search for the boundary between the background and objects in a two-dimensional color space as a straight line intersecting the line of direction to objects at a minimum distance from the origin in this space. It has been shown by means of modeling that among all linear dividing boundaries that provide a given false alarm probability, this boundary allows one to obtain the maximum of the object detection probability. 150 Valeriy P. Kosykh et al. CEUR Workshop Proceedings 142–151 Acknowledgments The work has performed with the financial support of the Ministry of Education and Science of Russia (project 121022000116-0). References [1] David S.K. Chan, David A. Langan, and Daniel A. Stayer. Spatial processing techniques for the detection of small targets in IR clutter // Proc. SPIE 1305, Signal and Data Processing of Small Targets 1990, (1 October 1990); doi: 10.1117/12.2321786 [2] Ffrench P.A., Zeidler J.R., Ku W.H. Enhanced detectability of small objects in correlated clutter using an improved 2-D adaptive lattice algorithm // IEEE Transactions on Image Processing. 1997. Vol. 6. No. 3. P. 383–397. [3] Vaidyanathan P.P. The theory of linear prediction. Morgan & Claypool Publishers, 2008. 184 p. [4] Jin L., Hong-Bing J. Improved background prediction algorithm for IR small targets de- tection // Proceedings of 2nd International Congress on Image and Signal Processing (CISP 2009). Tianjin, China, 17–19 October, 2009. P. 3363–3367. [5] Bracewell R.N. Two-dimensional imaging. Englewood Cliffs, NJ: Prentice Hall, 1995. P. 505–537. [6] Global land cover facility. Available at: http://glcf.umiacf.umd.edu (accessed 23.09.2017). [7] Winick K.A. Cramer – Rao lower bounds on the performance charged-coupled-device optical position estimator // JOSA A. 1986. Vol. 3. No. 11. P. 1809–1815. [8] Ivanov V.A., Kirichuk V.S., Kosykh V.P. et al. Specific features of detecting point objects in images formed by a detector array // Optoelectronics, Instrumentation and Data Processing. 2016. Vol. 52. P. 113–120. [9] Gromilin G.I., Ivanov V.A., Kosykh V.P., Popov S.A. Specific features of scanning with FPA in time delay integration mode // Computational Technologies. 2016. Vol. 21. No. 1. P. 60–89. 151