Scene-based Non-uniformity Fixed Pattern Noise Correction Algorithm for Infrared Video Sequences

Abstract

-An algorithm for a fixed pattern noise correction for infrared sensors based on the analysis of the video sequence of a static or dynamic scene observed by the camera is considered. It is shown that on the assumption of the additive nature of the fixed pattern noise, the frame-to-frame accumulation of such noise by analogy with the radar problem of detecting a signal against a correlated clutter can successfully compensate for it with a video sequence of more than 500 frames. During experiments with the Xenics Bobcat 640 short-wave infrared and Xenics Goby 384 long-wave infrared cameras it was demonstrated that in contrast to the well-known non-uniformity correction algorithm for a single frame, typical for it halo artifacts near extended scene objects are not observed in the resulting image, when fixed pattern noise is estimated from the results of accumulation over a set of frames.

Keywords

• non-uniformity correction
• fixed pattern noise
• recurrent averaging

-

INTRODUCTION

Fixed pattern noise (FPN) is commonly understood as a set of fixed deviations of the values of the output signals from various photosensitive elements of the infrared (IR) photodetector device (PD) at the same intensity of the radiation incident on them. Visually, the FPN appears on the IR image in the form of horizontal or vertical stripes depending on the orientation of the photodetector arrays (PA) in the PD matrix.

In a number of practical tasks (multispectral image fusion [1], computation of no-reference image quality indexes [2], dominant direction estimation in the task of gradient-based technique for image structural analysis [3], automatic recognition of bar pattern test object positions in task of IR camera calibration [4], ect.) estimation the parameters of FPN and its compensation (non-uniformity correction, NUC) are important stages of digital IR image processing.

II.

MATHEMATICAL MODELS OF IR CAMERAS FPN

Despite the fact that FPN of IR cameras in the general case is analytically described by a nonlinear dependence on the intensity of the radiation incident on the PD [ 5, 6 ], the authors of [ 7-10 ] use a linear model to solve the NUC problem:  Iij = kijI0ij + bij,  where I0ij and Iij are respectively the brightness of the pixel at the intersection of the i-th row and the j-th column in the absence of FPN and in its presence, kij and bij are respectively multiplicative and additive components of FPN.

The additive component of the FPN bij is mainly determined by the non-uniformity of the distribution of the dark current of the PD, therefore, it depends on the temperature and the exposure time. The multiplicative

In [ 11 ] model (1) is simplified to only multiplicative model:  Iij = kijI0ij. 

For matrix-type PD (MPD) composed of vertically arranged PA, the FPN model (1) can be reduced to the form [7]:  Iij = kjI0ij + bj,  where kj and bj are respectively the multiplicative and additive components of the FPN in the j-th PA of the MPD.

In sources [ 12-16 ] it was shown that for solving the NUC problem, model (3) can be further simplified to a single parameter – the constant displacement in the j-th column bj:  Iij = I0ij + bj, 

In this case the compensation of the additive FPN is simply to subtract its estimates for each j-th column:  I0ij = Iij – bj,  which excludes from the processing according to (2) or (3) the operation of multiplication by the weight coefficient {1/kij} or {1/kj} respectively.

The problem of NUC involves the assessment of parameters of mathematical models (1) – (4). To develop a NUC algorithm we accept the hypothesis that FPN mathematical model is described by expression (4).

III.

KNOWN METHODS OF FPN EVALUATION AND NUC

A. Classification of NUC methods

NUC methods are usually divided into two categories [ 16, 17 ]: methods based on pre-calibration according to the test object (Calibration-Based NUC, CBNUC) and methods based directly on analysis of the observed scene (SceneBased NUC, SBNUC). The first category includes methods of single, double, and multipoint calibration [ 18 ]. A classic representative of CBNUC algorithms is the two-point calibration method (Two Point NUC, TPNUC procedure), which involves calibrating the camera in two frames with uniform brightness:   for mid-wave IR (MWIR, 3..5 μm) and long-wave IR (LWIR, 8..14 μm) cameras – according to two images of a black body with different temperatures (at two temperature points – “cold” and “hot”); for short-wave IR (SWIR, 0.9..1.7 μm) cameras – according to two images of a scene with uniform illumination at two different shutter speeds (usually 0.5 and 5 ms).

For models (1) and (3) this allows to find estimates of the parameters kij and bij from the solution of systems of pairs of the corresponding linear equations. At the same time, the use of CBNUC methods for uncooled thermal cameras does not allow to perform effective NUC due to the sensitivity of the parameters kij and bij to changes in the temperature of the camera body and MPD.

The SBNUC family of methods that operate only with the statistics of the brightness distribution of the observed scene and do not require specialized equipment for calibration don’t have this drawback. Their drawback, in turn, is NUC artifacts, which appear at the boundaries of images of extended objects [ 14-17 ].

B. SBNUC methods

When accepting the hypotheses that the FPN model is described by (4), and the MPD consists of columns of PA, an effective SBNUC algorithm for NUC is the spatially adaptive column FPN correction based on 1D horizontal differential statistics, which was considered in detail in [ 15 ]. This algorithm contains the following basic steps.

1) Row-by-row processing of the initial image with a 1D smoothing filter, which is a guided filter [ 19 ] with a regularization parameter  = 0.42 (high degree of smoothing and noise suppression). A guided filter has all the advantages of a bilateral filter and is without its drawbacks [ 19 ].

2) Estimation of the horizontal spatial high-frequency (HF) component of the original image:  IHF_h = I – ILF, where I is the original image and ILF is the filtering result of a 1D Gaussian low-frequency (LF) filter;

3) Separation of the HF component IHF_h on the HF signal component IHF and the additive FPN component b:

IHF_h = IHF + b.

This separation is based on the calculation of the HDS1D statistics (1D Horizontal Differential Statistics [ 15 ]) in each i-th row of the image IHF_h for each j-th column:

HDS 1D ( j )  1 N h / 2

 K 1 ( j ) k   N h / 2

 [ I HF j  I HF j  k ]2   I j  k , exp    2  r12  where

N h / 2  [ I H Fj  I HF j  k ] 2 

K 1 ( j )  k  N h / 2 exp   2  r1 2  is the normalization term, Ij+k is the computed local gradient in the horizontal direction, Nh is a horizontal window which defines a set of neighboring pixels of i, and σr1 is the range weight parameter, that determines the gravity of the module of the brightness difference of the j-th and the (j + k)-th columns. In [ 15 ] σr1 is selected 10 times greater than the standard deviation (SD) of the brightness gradient I along the row in accordance with the recommendations of [ 20 ].

4) FPN estimation for each j-th column is based on calculation of statistic:

b j  where 1 N v / 2

 K 2 ( j ) k   N v / 2

 exp      k 2 

 2  I HF j  k , HDS 1 D ( j )   2  s 2  (6)  K 2 ( j )  k NvN/v2/ 2 exp   HDS 1 D( j )    2 k s22 2    is the normalization term, χ is a small positive number to avoid division by zero when HDS1D(j) = 0. The parameters , s2 and the vertical size of the sliding window Nv in [ 15 ] are taken to be 0.5, 0.8H and H, respectively, where H is the frame height in pixels.

The choice of the Nv window height is made for  compromise reasons: for small Nv the NUC is better (especially in images with a uniform background), however, FPN estimate in this case is sensitive to the HF spatial component of brightness. On the contrary, for large Nv the statistics (7) is less sensitive to the features of the observed scene, but the error in the estimation of the FPN is also higher. This is illustrated in Fig. 1, which shows that the presence of extended vertical objects leads to the appearance of NUC compensation artifacts in the image of a SWIR video camera – the highlighted (halo) areas above the cell towers and pipes of the building. Moreover, in areas with a uniform background, FPN is effectively suppressed. a) b)

The basic idea of the algorithm developed by the authors is that based on the principles of estimating FPN [ 15 ] due to the accumulation of a series of frames it is possible to form an image with an approximately uniform background, which will increase the efficiency of FPN estimation and NUC.

IV.

PROPOSED METHOD OF NUC FOR A SERIES OF

FRAMES

The idea of the algorithm is based on the adoption of the hypothesis (4) on the additive character of FPN and the principle of quasi-optimal detection of burst-pulse signals against the background of correlated clutters in radar systems: rejection of LF clutter and accumulation of an HF signal [ 21 ]. At the same time, we consider that FPN is a useful signal to be detected, and the scene image is a correlated clutter.

The main stages of our SBNUC algorithm for a series of frames are the following.

1) The frame from IR video camera Ik (at the k-th moment of time) is received.

2) In the frame Ik all its rows are randomly permutated and a frame I*k is formed. As a result of row permutation, the pixels of the j-th column of the frame Ik corresponding to one j-th PA of MPD with the vertical direction of reading charge packets in the frame I*k still remain in the j-th column.

3) The auxiliary frame Nk is recurrently evaluated:

Nk = [(n – 1)Nk-1 + I*k]/n, where n is the number of previously received frames; (7) 4) The variance of the brightness gradient Dhk along a row (in the horizontal direction) over the frame Nk is estimated:

Dhk = M {(Nij – Ni,j–1)2}, i , j

Start

Read frame I from IR camera

Rows permutation in I

Auxiliary frame Nk

estimation by (8) Brightness gradient Dhk estimation

Dhk > Dhmax

no yes

EXPERIMENTAL RESULTS AND DISCUSSION

The experiments were carried out with a Xenics Bobcat 640 SWIR camera. To reduce the amount of computation when dividing the calibration frame K into LF and HF parts where M{  } denotes the calculation of the brightness mean. The background component (correlated clutter) in the horizontal direction is suppressed by analogy with the principle of operation of the radar single delay line canceler [ 21 ].

5) If upon receipt of a new frame k the estimate of the variance Dhk is greater than its previous maximum value Dhmax, this means that the FPN-to-background ratio in the auxiliary frame Nk has grown. Therefore, Nk is written to the calibration frame K and the value of Dhmax is updated:

K = Nk, Dhmax = Dhk.

6) The calibration frame K is divided into two additive components: LF part KLF (background) and HF part b which is the estimated FPN:

K = KLF + b.

7) NUC is performed according to (5) subsequent linear contrasting of the result.

(8) with the

Random permutation of rows during the forming of I*k frame with subsequent averaging of such frames ensures that the background brightness is equalized over the Nk frame even if there are areas of different brightness in the scene (for example, for outdoors shooting conditions these areas are the sky and underlying surface), which eliminates the SBNUC-like FPN compensation artifacts. With a large number of averaged frames with randomly rearranged rows, by virtue of the central limit theorem, it is fair to assume that the background brightness distribution will tend to normal. Therefore, the accumulation FPN according to (8) is equivalent to the problem of incoherent accumulation of a useful signal against the background of Gaussian noise in radar systems [ 21 ]. NUC algorithm scheme is shown in Fig. 2.

Dhmax = Dhk

Calibration frame forming:

K = N FPN estimation: b = K – KLF

NUC by (5)

Linear contrasting

Frame with NUC output End

according to (9), the authors applied the fast low-pass filtering procedure [ 22 ] with a 32-elements aperture BOX filter.

Fig. 3-5 show the selective frames (with a 1.25 s time interval between them) from the original video sequence with a duration of 15 s and a frame frequency of 50 Hz, the results of the recursive estimation of the FPN according to the developed algorithm and the NUC results respectively.

From the results of the experiment it follows that after approximately 500 frames the asymptotic convergence of the FPN estimate according to (8) and (9) to the true FPN value is ensured. Moreover, the NUC result doesn’t contain halo artifacts specific to SBNUC algorithms [ 13-16 ].

The authors also conducted an experiment with a LWIR Xenics Gobi 384 video camera based on the uncooled microbolometer in the mode with FPN off and bad pixels correction, which compared the results of FPN estimation obtained with a closed defocused camera lens according to CBNUC method [ 23 ] (Fig. 6-8) and according to the developed algorithm (Fig. 9).

With the visual similarity of frames with FPN in Fig. 8 and 9, the difference in their SDs approximately 1.5 times is explained primarily by the distribution (when rows are randomly permutated) of the irregular gain of the camera matrix according to model (1) over the entire height of the frame column (dark left and right frame edges in Fig. 9). Therefore, despite the forming of a subjectively comfortable image (without pronounced FPN), the considered NUC algorithm without preliminary flat field correction does not compensate not only for the gain irregularity of MPD, but can even enhance it, which will appear on images with extended objects of a uniform texture and uniform brightness.

Fig. 5. NUC results.

Fig.6. Raw LWIR frame.

Fig. 9. Our estimation of sensor FPN, SD = 14.12. [7] [ 8 ] [9]

References

X. Xue , F. Xiang and H. Wang , “ A parallel fusion method of remote sensing image based on NSCT,” Computer Optics , vol. 43 , no. 1 , pp. 123 - 131 , 2019 . DOI: 10 .18287/ 2412 -6179-2019-43-1- 123 -131.

S. Pertuz , D. Puig and M.A. Garcia , “ Analysis of focus measure operators for shape-from-focus,” Pattern Recognit ., vol. 46 , no. 5 , pp. 1415 - 1432 , 2013 .

D.G. Asatryan , “ Gradient-based technique for image structural analysis and applications ,” Computer Optics, vol. 43 , no. 2 , pp. 245 - 250 , 2019 . DOI: 10 .18287/ 2412 -6179-2019-43-2- 245 -250.

A.V. Mingalev , A.V. Belov , I.M. Gabdullin , R.R. Agafonova and S.N. Shusharin , “ Test-object recognition in thermal images ,” Computer Optics , vol. 43 , no. 3 , pp. 402 - 411 , 2019 . DOI: 10 .18287/ 2412 -6179-2019-43-3- 402 -411.

V.N. Borovytsky , “ Residual error after non-uniformity correction,” Semicond . Physics, quantum electron. & optoelectron., vol. 3 , no. 1 , pp. 102 - 105 , 2000 .

6th Mediterranean Conf. on Embedded Comput. (MECO) , Bar, Montenegro, pp. 159 - 162 , 2017 .

Fig. 8. Raw LWIR frame with flat field correction , SD = 9 . 25 .

D.L. Perry and E.L. Dereniak , “ Linear theory of nonuniformity correction in infrared staring sensors,” Opt . Eng., vol. 32 , pp. 1854 - 1859 , 1993 .

B.M. Ratliff and M.M. Hayat , “ An algebraic algorithm for nonuniformity correction in focal-plane arrays” , J. Opt. Soc. Am. A. , vol. 19 , no. 9 , pp. 1737 - 1747 , 2002 .

P. Narendra , “ Reference-free nonuniformity compensation for IR imaging arrays , ” Proc. SPIE , vol. 252 , pp. 10 - 17 , 1980 .

[10] M. Sheng , J. Xie and Z. Fu , “ Calibration-based NUC method in real-time based on IRFPA , ” Physics Procedia , vol. 22 , pp. 372 - 380 , 2011 .

[11] Y.S. Bekhtin , V.S. Gurov and M.N. Guryeva , “ Algorithmic supply of IR sensors with FPN using texture homogeneity levels , ” Proc. 5th Mediterranean Conf. on Embedded Comput. (MECO) , Budva, Montenegro, pp. 252 - 255 , 2014 .

[12] R. Hardie , M. Hayat , E. Armstrong and B. Yasuda , “ Scene-based nonuniformity correction with video sequences and registration ,” Appl. Opt., vol. 39 , no. 8 , pp. 1241 - 1250 , 2000 .

[13] C. Zuo , Q. Chen , G. Gu , X. Sui and J. Ren , “ Improved interframe registration based nonuniformity correction for focal plane arrays,” Infrared Phys . Technol., vol. 55 , no. 4 ., pp. 263 - 269 , 2012 .

[14] Y. Cao , M.Y. Yang and C.-L. Tisse , “ Effective strip noise removal for low-textured infrared images based on 1-D guided filtering , ” IEEE Trans. Circuits Syst. Video Technol. , vol. 26 , pp. 2176 - 2188 , 2016 .

[15] Y. Cao , Z. He , J. Yang and M.Y. Yang , “ Spatially adaptive column fixed-pattern noise correction in infrared imaging system using 1D horizontal differential statistics,” IEEE Photonics J ., vol. 9 , no. 5 , pp. 1 - 13 , 2017 .

[16] C. Liu , X. Sui , Y. Liu , X. Kuang , G. Gu and Q. Chen , “ FPN estimation based nonuniformity correction for infrared imaging system , ” Infrared Physics and Technol. , vol. 96 , pp. 22 - 29 , 2019 .

[17] L. Huo , D. Zhou , D. Wang , R. Liu and B. He , “ Staircase-scene-based nonuniformity correction in aerial point target detection systems ,” Appl. Opt., vol. 55 , pp. 7149 - 7156 , 2016 .

[18] K. Liang , C. Yang , L. Peng and B. Zhou , “ Non-uniformity correction based on focal plane array temperature in uncooled long-wave infrared cameras without a shutter,” Appl . Opt., vol. 56 , pp. 884 - 889 , 2017 .

[19] K. He , J. Sun and X. Tang , “Guided image filtering, ” IEEE Trans. on Pattern Anal. and Machine Intell ., vol. 35 , no. 6 , pp. 1397 - 1409 , 2013 .

[20] A. Buades , B. Coll and J.-M. Morel , “ A non-local algorithm for image denoising , ” IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit ., San Diego, USA, vol. 2 , 2005 .

[21] B.R. Mahafza , “ Radar systems analysis and design using MATLAB,” NY: Chapman & Hall/CRC, 2000 .

[22] A. Lukin , “ Tips & tricks: fast image filtering algorithms , ” 17th Int. Conf. on Comput. Graphics “GraphiCon” , Moscow, pp. 186 - 189 , 2007 .

[23] I.I. Kremis , “ Method of compensating for signal irregularity of photosensitive elements of multielement photodetector , ” patent RU 2449491, date of patent: 27.04 . 2012 .