=Paper=
{{Paper
|id=Vol-2665/paper31
|storemode=property
|title=Scene-based non-uniformity fixed pattern noise correction algorithm for infrared video sequences
|pdfUrl=https://ceur-ws.org/Vol-2665/paper31.pdf
|volume=Vol-2665
|authors=Igor Kudinov,Ivan Kholopov
}}
==Scene-based non-uniformity fixed pattern noise correction algorithm for infrared video sequences ==
https://ceur-ws.org/Vol-2665/paper31.pdf
Scene-based Non-uniformity Fixed Pattern Noise
Correction Algorithm for Infrared Video Sequences
Igor Kudinov Ivan Kholopov
Ryazan State Radio Engineering University named after
Ryazan State Radio Engineering University named after V.F. Utkin (RSREU)
V.F. Utkin (RSREU) Ryazan, Russia
Ryazan, Russia kholopov.i.s@rsreu.ru
i.a.kudinov@yandex.ru
Abstract—An algorithm for a fixed pattern noise correction component of FPN kij is associated with the non-uniformity
for infrared sensors based on the analysis of the video sequence of the integrated sensitivity of the elements of the PD.
of a static or dynamic scene observed by the camera is
considered. It is shown that on the assumption of the additive In [11] model (1) is simplified to only multiplicative
nature of the fixed pattern noise, the frame-to-frame model:
accumulation of such noise by analogy with the radar problem Iij = kijI0ij.
of detecting a signal against a correlated clutter can
For matrix-type PD (MPD) composed of vertically
successfully compensate for it with a video sequence of more
than 500 frames. During experiments with the Xenics Bobcat
arranged PA, the FPN model (1) can be reduced to the
640 short-wave infrared and Xenics Goby 384 long-wave form [7]:
infrared cameras it was demonstrated that in contrast to the Iij = kjI0ij + bj,
well-known non-uniformity correction algorithm for a single where kj and bj are respectively the multiplicative and
frame, typical for it halo artifacts near extended scene objects additive components of the FPN in the j-th PA of the MPD.
are not observed in the resulting image, when fixed pattern
noise is estimated from the results of accumulation over a set of In sources [12-16] it was shown that for solving the
frames. NUC problem, model (3) can be further simplified to a
single parameter – the constant displacement in the j-th
Keywords—non-uniformity correction; fixed pattern noise; column bj:
recurrent averaging
Iij = I0ij + bj,
I. INTRODUCTION In this case the compensation of the additive FPN is
Fixed pattern noise (FPN) is commonly understood as a simply to subtract its estimates for each j-th column:
set of fixed deviations of the values of the output signals I0ij = Iij – bj,
from various photosensitive elements of the infrared (IR) which excludes from the processing according to (2) or (3)
photodetector device (PD) at the same intensity of the the operation of multiplication by the weight coefficient
radiation incident on them. Visually, the FPN appears on the {1/kij} or {1/kj} respectively.
IR image in the form of horizontal or vertical stripes
depending on the orientation of the photodetector arrays The problem of NUC involves the assessment of
(PA) in the PD matrix. parameters of mathematical models (1) – (4). To develop a
NUC algorithm we accept the hypothesis that FPN
In a number of practical tasks (multispectral image mathematical model is described by expression (4).
fusion [1], computation of no-reference image quality
indexes [2], dominant direction estimation in the task of III. KNOWN METHODS OF FPN EVALUATION AND NUC
gradient-based technique for image structural analysis [3],
automatic recognition of bar pattern test object positions in A. Classification of NUC methods
task of IR camera calibration [4], ect.) estimation the NUC methods are usually divided into two categories
parameters of FPN and its compensation (non-uniformity [16, 17]: methods based on pre-calibration according to the
correction, NUC) are important stages of digital IR image test object (Calibration-Based NUC, CBNUC) and methods
processing. based directly on analysis of the observed scene (Scene-
Based NUC, SBNUC). The first category includes methods
II. MATHEMATICAL MODELS OF IR CAMERAS FPN of single, double, and multipoint calibration [18]. A classic
Despite the fact that FPN of IR cameras in the general representative of CBNUC algorithms is the two-point
case is analytically described by a nonlinear dependence on calibration method (Two Point NUC, TPNUC procedure),
the intensity of the radiation incident on the PD [5, 6], the which involves calibrating the camera in two frames with
authors of [7-10] use a linear model to solve the NUC uniform brightness:
problem: for mid-wave IR (MWIR, 3..5 μm) and long-wave
Iij = kijI0ij + bij, IR (LWIR, 8..14 μm) cameras – according to two
where I0ij and Iij are respectively the brightness of the pixel images of a black body with different temperatures
at the intersection of the i-th row and the j-th column in the (at two temperature points – “cold” and “hot”);
absence of FPN and in its presence, kij and bij are for short-wave IR (SWIR, 0.9..1.7 μm) cameras –
respectively multiplicative and additive components of FPN. according to two images of a scene with uniform
The additive component of the FPN bij is mainly illumination at two different shutter speeds (usually
determined by the non-uniformity of the distribution of the 0.5 and 5 ms).
dark current of the PD, therefore, it depends on the
temperature and the exposure time. The multiplicative
Copyright © 2020 for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
Image Processing and Earth Remote Sensing
For models (1) and (3) this allows to find estimates of [I 2
H Fj I HF j k ]
Nh /2
the parameters kij and bij from the solution of systems of
K 1( j) exp
2
pairs of the corresponding linear equations. At the same kNh /2 2 r1
time, the use of CBNUC methods for uncooled thermal
is the normalization term, Ij+k is the computed local
cameras does not allow to perform effective NUC due to the
gradient in the horizontal direction, Nh is a horizontal
sensitivity of the parameters kij and bij to changes in the
window which defines a set of neighboring pixels of i, and
temperature of the camera body and MPD.
σr1 is the range weight parameter, that determines the
The SBNUC family of methods that operate only with gravity of the module of the brightness difference of the j-th
the statistics of the brightness distribution of the observed and the (j + k)-th columns. In [15] σr1 is selected 10 times
scene and do not require specialized equipment for greater than the standard deviation (SD) of the brightness
calibration don’t have this drawback. Their drawback, in gradient I along the row in accordance with the
turn, is NUC artifacts, which appear at the boundaries of recommendations of [20].
images of extended objects [14-17].
4) FPN estimation for each j-th column is based on
B. SBNUC methods calculation of statistic:
When accepting the hypotheses that the FPN model is
1
Nv /2 k
2
described by (4), and the MPD consists of columns of PA, exp I
bj HDS
HF j k ,
an effective SBNUC algorithm for NUC is the spatially 1D ( j ) 2 s 2
2
K 2 ( j) k N / 2
v
adaptive column FPN correction based on 1D horizontal (6)
differential statistics, which was considered in detail in [15]. where
This algorithm contains the following basic steps.
Nv /2
2
1) Row-by-row processing of the initial image with a 1D exp
k
K 2 ( j) HDS
smoothing filter, which is a guided filter [19] with a ( j) 2 2
kN v /2 1 D s2
regularization parameter = 0.42 (high degree of smoothing is the normalization term, χ is a small positive number to
and noise suppression). A guided filter has all the avoid division by zero when HDS1D(j) = 0. The parameters
advantages of a bilateral filter and is without its drawbacks
, s2 and the vertical size of the sliding window Nv in [15]
[19].
are taken to be 0.5, 0.8H and H, respectively, where H is the
2) Estimation of the horizontal spatial high-frequency frame height in pixels.
(HF) component of the original image:
The choice of the Nv window height is made for
IHF_h = I – ILF, compromise reasons: for small Nv the NUC is better
where I is the original image and ILF is the filtering result of (especially in images with a uniform background), however,
a 1D Gaussian low-frequency (LF) filter; FPN estimate in this case is sensitive to the HF spatial
component of brightness. On the contrary, for large Nv the
3) Separation of the HF component IHF_h on the HF statistics (7) is less sensitive to the features of the observed
signal component IHF and the additive FPN component b: scene, but the error in the estimation of the FPN is also
IHF_h = IHF + b. higher. This is illustrated in Fig. 1, which shows that the
This separation is based on the calculation of the HDS1D presence of extended vertical objects leads to the appearance
statistics (1D Horizontal Differential Statistics [15]) in each of NUC compensation artifacts in the image of a SWIR
i-th row of the image IHF_h for each j-th column: video camera – the highlighted (halo) areas above the cell
towers and pipes of the building. Moreover, in areas with a
[I 2
uniform background, FPN is effectively suppressed.
HF j I HF j k ]
N /2 h
1
HDS 1 D ( j ) exp
2
I j k ,
K1( j) k N 2 r1
h /2
where
a) b)
Fig. 1. Frames from the Xenics Bobcat 640 SWIR camera in the mode without NUC and bad pixels correction: a – the original image, b – image after FPN
NUC according to [15].
The basic idea of the algorithm developed by the authors
is that based on the principles of estimating FPN [15] due to
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 136
Image Processing and Earth Remote Sensing
the accumulation of a series of frames it is possible to form where M{ } denotes the calculation of the brightness mean.
an image with an approximately uniform background, which The background component (correlated clutter) in the
will increase the efficiency of FPN estimation and NUC. horizontal direction is suppressed by analogy with the
principle of operation of the radar single delay line canceler
IV. PROPOSED METHOD OF NUC FOR A SERIES OF [21].
FRAMES
5) If upon receipt of a new frame k the estimate of the
The idea of the algorithm is based on the adoption of the variance Dhk is greater than its previous maximum value
hypothesis (4) on the additive character of FPN and the Dhmax, this means that the FPN-to-background ratio in the
principle of quasi-optimal detection of burst-pulse signals auxiliary frame Nk has grown. Therefore, Nk is written to the
against the background of correlated clutters in radar calibration frame K and the value of Dhmax is updated:
systems: rejection of LF clutter and accumulation of an HF
signal [21]. At the same time, we consider that FPN is a K = Nk, Dhmax = Dhk.
useful signal to be detected, and the scene image is a 6) The calibration frame K is divided into two additive
correlated clutter. components: LF part KLF (background) and HF part b which
is the estimated FPN:
The main stages of our SBNUC algorithm for a series of
frames are the following. K = KLF + b. (8)
7) NUC is performed according to (5) with the
1) The frame from IR video camera Ik (at the k-th subsequent linear contrasting of the result.
moment of time) is received.
Random permutation of rows during the forming of I*k
2) In the frame Ik all its rows are randomly permutated frame with subsequent averaging of such frames ensures
and a frame I*k is formed. As a result of row permutation, that the background brightness is equalized over the Nk
the pixels of the j-th column of the frame Ik corresponding frame even if there are areas of different brightness in the
to one j-th PA of MPD with the vertical direction of reading scene (for example, for outdoors shooting conditions these
charge packets in the frame I*k still remain in the j-th areas are the sky and underlying surface), which eliminates
column. the SBNUC-like FPN compensation artifacts. With a large
3) The auxiliary frame Nk is recurrently evaluated: number of averaged frames with randomly rearranged rows,
by virtue of the central limit theorem, it is fair to assume
Nk = [(n – 1)Nk-1 + I*k]/n, (7) that the background brightness distribution will tend to
where n is the number of previously received frames; normal. Therefore, the accumulation FPN according to (8) is
4) The variance of the brightness gradient Dhk along a equivalent to the problem of incoherent accumulation of a
row (in the horizontal direction) over the frame Nk is useful signal against the background of Gaussian noise in
estimated: radar systems [21]. NUC algorithm scheme is shown in
Fig. 2.
Dhk = M {(Nij – Ni,j–1)2},
i, j
Start
Read frame I from IR camera
Dhmax = Dhk
Rows permutation in I
Calibration frame forming:
Auxiliary frame Nk K=N
estimation by (8)
FPN estimation: b = K – KLF
Brightness gradient Dhk
estimation
yes NUC by (5)
Dhk > Dhmax
Linear contrasting
no
Frame with NUC output
End
Fig.2. NUC algorithm scheme.
V. EXPERIMENTAL RESULTS AND DISCUSSION according to (9), the authors applied the fast low-pass
filtering procedure [22] with a 32-elements aperture BOX
The experiments were carried out with a Xenics Bobcat filter.
640 SWIR camera. To reduce the amount of computation
when dividing the calibration frame K into LF and HF parts
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 137
Image Processing and Earth Remote Sensing
Fig. 3-5 show the selective frames (with a 1.25 s time results of the recursive estimation of the FPN according to
interval between them) from the original video sequence the developed algorithm and the NUC results respectively.
with a duration of 15 s and a frame frequency of 50 Hz, the
Fig. 3. Frames of a video sequence recorded by a SWIR Xenics Bobcat 640 camera.
Fig. 4. FPN estimation results (with an additive background of 128 units).
correction, which compared the results of FPN estimation
obtained with a closed defocused camera lens according to
From the results of the experiment it follows that after CBNUC method [23] (Fig. 6-8) and according to the
approximately 500 frames the asymptotic convergence of developed algorithm (Fig. 9).
the FPN estimate according to (8) and (9) to the true FPN
value is ensured. Moreover, the NUC result doesn’t contain
halo artifacts specific to SBNUC algorithms [13-16].
With the visual similarity of frames with FPN in Fig. 8
The authors also conducted an experiment with a LWIR and 9, the difference in their SDs approximately 1.5 times is
Xenics Gobi 384 video camera based on the uncooled explained primarily by the distribution (when rows are
microbolometer in the mode with FPN off and bad pixels randomly permutated) of the irregular gain of the camera
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 138
Image Processing and Earth Remote Sensing
matrix according to model (1) over the entire height of the compensate not only for the gain irregularity of MPD, but
frame column (dark left and right frame edges in Fig. 9). can even enhance it, which will appear on images with
Therefore, despite the forming of a subjectively comfortable extended objects of a uniform texture and uniform
image (without pronounced FPN), the considered NUC brightness.
algorithm without preliminary flat field correction does not
Fig. 5. NUC results.
Fig.6. Raw LWIR frame. Fig. 7. Gain irregularity. Fig. 8. Raw LWIR frame with flat Fig. 9. Our estimation of sensor FPN,
field correction, SD = 9.25. SD = 14.12.
REFERENCES [7] D.L. Perry and E.L. Dereniak, “Linear theory of nonuniformity
correction in infrared staring sensors,” Opt. Eng., vol. 32, pp. 1854-1859,
[1] X. Xue, F. Xiang and H. Wang, “A parallel fusion method of remote 1993.
sensing image based on NSCT,” Computer Optics, vol. 43, no. 1, [8] B.M. Ratliff and M.M. Hayat, “An algebraic algorithm for nonuniformity
pp. 123-131, 2019. DOI: 10.18287/2412-6179-2019-43-1-123-131. correction in focal-plane arrays”, J. Opt. Soc. Am. A., vol. 19, no. 9,
[2] S. Pertuz, D. Puig and M.A. Garcia, “Analysis of focus measure pp. 1737-1747, 2002.
operators for shape-from-focus,” Pattern Recognit., vol. 46, no. 5, [9] P. Narendra, “Reference-free nonuniformity compensation for IR
pp. 1415-1432, 2013. imaging arrays,” Proc. SPIE, vol. 252, pp. 10-17, 1980.
[3] D.G. Asatryan, “Gradient-based technique for image structural [10] M. Sheng, J. Xie and Z. Fu, “Calibration-based NUC method in real-time
analysis and applications,” Computer Optics, vol. 43, no. 2, pp. 245- based on IRFPA,” Physics Procedia, vol. 22, pp. 372-380, 2011.
250, 2019. DOI: 10.18287/2412-6179-2019-43-2-245-250.
[11] Y.S. Bekhtin, V.S. Gurov and M.N. Guryeva, “Algorithmic supply of
[4] A.V. Mingalev, A.V. Belov, I.M. Gabdullin, R.R. Agafonova and IR sensors with FPN using texture homogeneity levels,” Proc. 5th
S.N. Shusharin, “Test-object recognition in thermal images,” Mediterranean Conf. on Embedded Comput. (MECO), Budva,
Computer Optics, vol. 43, no. 3, pp. 402-411, 2019. DOI: Montenegro, pp. 252-255, 2014.
10.18287/2412-6179-2019-43-3-402-411.
[12] R. Hardie, M. Hayat, E. Armstrong and B. Yasuda, “Scene-based
[5] V.N. Borovytsky, “Residual error after non-uniformity correction,” nonuniformity correction with video sequences and registration,”
Semicond. Physics, quantum electron. & optoelectron., vol. 3, no. 1, Appl. Opt., vol. 39, no. 8, pp. 1241-1250, 2000.
pp. 102-105, 2000.
[13] C. Zuo, Q. Chen, G. Gu, X. Sui and J. Ren, “Improved interframe
[6] Y.S. Bekhtin, A.A. Lupachev and M.N. Knyazev, “Estimating registration based nonuniformity correction for focal plane arrays,”
impulse parameters from point sources in onboard IR-sensor,” Proc. Infrared Phys. Technol., vol. 55, no. 4., pp. 263-269, 2012.
6th Mediterranean Conf. on Embedded Comput. (MECO), Bar,
Montenegro, pp. 159-162, 2017. [14] Y. Cao, M.Y. Yang and C.-L. Tisse, “Effective strip noise removal
for low-textured infrared images based on 1-D guided filtering,” IEEE
Trans. Circuits Syst. Video Technol., vol. 26, pp. 2176-2188, 2016.
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 139
Image Processing and Earth Remote Sensing
[15] Y. Cao, Z. He, J. Yang and M.Y. Yang, “Spatially adaptive column [19] K. He, J. Sun and X. Tang, “Guided image filtering,” IEEE Trans. on
fixed-pattern noise correction in infrared imaging system using 1D Pattern Anal. and Machine Intell., vol. 35, no. 6, pp. 1397-1409,
horizontal differential statistics,” IEEE Photonics J., vol. 9, no. 5, 2013.
pp. 1-13, 2017. [20] A. Buades, B. Coll and J.-M. Morel, “A non-local algorithm for
[16] C. Liu, X. Sui, Y. Liu, X. Kuang, G. Gu and Q. Chen, “FPN image denoising,” IEEE Comput. Soc. Conf. Comput. Vis. Pattern
estimation based nonuniformity correction for infrared imaging Recognit., San Diego, USA, vol. 2, 2005.
system,” Infrared Physics and Technol., vol. 96, pp. 22-29, 2019. [21] B.R. Mahafza, “Radar systems analysis and design using MATLAB,”
[17] L. Huo, D. Zhou, D. Wang, R. Liu and B. He, “Staircase-scene-based NY: Chapman & Hall/CRC, 2000.
nonuniformity correction in aerial point target detection systems,” [22] A. Lukin, “Tips & tricks: fast image filtering algorithms,” 17th Int.
Appl. Opt., vol. 55, pp. 7149-7156, 2016. Conf. on Comput. Graphics “GraphiCon”, Moscow, pp. 186-189, 2007.
[18] K. Liang, C. Yang, L. Peng and B. Zhou, “Non-uniformity correction [23] I.I. Kremis, “Method of compensating for signal irregularity of
based on focal plane array temperature in uncooled long-wave photosensitive elements of multielement photodetector,” patent
infrared cameras without a shutter,” Appl. Opt., vol. 56, pp. 884-889, RU 2449491, date of patent: 27.04.2012.
2017.
VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 140