=Paper=
{{Paper
|id=Vol-3006/28_short_paper
|storemode=property
|title=Improving the spatial resolution of digital images and video sequences using subpixel scanning
|pdfUrl=https://ceur-ws.org/Vol-3006/28_short_paper.pdf
|volume=Vol-3006
|authors=Aleksandr L. Reznik,Aleksandr A. Soloviev,Andrey V. Torgov
}}
==Improving the spatial resolution of digital images and video sequences using subpixel scanning==
https://ceur-ws.org/Vol-3006/28_short_paper.pdf
Improving the spatial resolution of digital images and
video sequences using subpixel scanning
Aleksandr L. Reznik1 , Aleksandr A. Soloviev1 and Andrey V. Torgov1
1
Institute of Automation and Electrometry of the Siberian Branch of the Russian Academy of Sciences, Novosibirsk,
Russia
Abstract
High-performance method for improving the resolution of digital images and video sequences based
on minimum-variance signal reconstruction are considered. A distinctive feature of the developed
algorithms is that they allow (with the availability of modern computing power) to obtain improved
images and video in “real time”.
Keywords
Image reconstruction, high-performance algorithms.
1. Introduction
In the process of digital registration, the image is recorded into a two-dimensional digital array
using elements of a rectangular photomatrix. In this case, whatever technology is used, the
spatial resolution of the resulting image is determined by the size of a single photocell of the
matrix. And while there has been tremendous progress in high-resolution digital cameras, there
is still a need to improve the resolution of digital images. At the same time, a certain limit has
already been reached in improving the quality of images by purely technological methods, since
with a decrease in the size of elements of photo matrices, the cost of their production increases
significantly [1, 2].
That is why purely digital methods of image processing to improve their spatial resolution
are of great interest. Such a need arises, in particular, in a situation where it is not possible to
obtain a high-resolution image, but it is possible to obtain an excessive number of low-quality
images, which must then be processed in an optimal way.
The results of this work are important for solving applied problems of automatic scanning, in
most problems of digital image processing, in areas related to stereo reconstruction of optical and
thermal images, in the development of modern night vision devises, for processing surveillance
data from video cameras, and in other fields of science and technology.
And although theoretical developments in this area have existed for several decades, only
the rapid progress in the development of modern computer technology has made it possible to
successfully apply computational algorithms for solving problems of improving the quality of
digital images (their use was previously complicated by the enormous computational costs that
computers could not provide several decades ago).
SDM-2021: All-Russian conference, August 24–27, 2021, Novosibirsk, Russia
" reznik@iae.nsk.su (A. L. Reznik); soloviev@iae.nsk.su (A. A. Soloviev); torgov@iae.nsk.su (A. V. Torgov)
© 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org)
238
Aleksandr L. Reznik et al. CEUR Workshop Proceedings 238–245
It should also be noted that the choice of optimal processing algorithms (with their help
the quality of processed digital images will be improved), in each specific case, depends on
two components: first, on a priori knowledge of the statistical characteristics of images, and
secondly, on the required specific output image parameters that must be achieved as a result of
applying the developed processing algorithm. This explains the presence of a large number of
studies in the field of creating mathematical and software-algorithmic methods for improving
the quality of matrix images. Each of these studies has a specific purpose and is being developed
to address one of the many challenges in digital imaging.
2. Improving the resolution of digital images based on the
calculation of the image with the least variance
As we mentioned above, the possibility of increasing the resolution of images requires reducing
the size of the elements of the photomatrix. However, there are a number of technical difficulties
here. A possible way out of this situation is multiple acquisition of the same image by a “bad-
resolution” photo sensor, which changes its position during the shooting. In this case, using
algorithms that effectively process the results of such a subpixel scan, it is possible to obtain an
image with a higher spatial resolution.
2.1. Improving the spatial resolution of images — one-dimensional case
Let us consider a one-dimensional registration scheme (see Figure 1). Here the number 𝑁
corresponding to the dimension of the restored vector 𝑋 = (𝑥1 , 𝑥2 , . . . , 𝑥𝑁 ) is a multiple of
𝑙 — the number of resolution elements located into the integrating aperture, that is, the field
size (in one-dimensional in the case, the size of the interval) of scanning is an integer number
of times larger than the size of the aperture: 𝑁 = 𝑛 × 𝑙.
The described registration mode leads to an underdetermined problem, when the observed
data are insufficient for accurate reconstruction of the signal (image), and its statistically
valid estimate must be constructed. Various authors [3, 4] from the field of signal and image
processing have developed mathematical models, computational schemes and algorithms for
solving such problems, which allow constructing rather effective schemes for solving specific
applied problems. For example, in [5], such an estimate is obtained by algebraic methods by
means of pseudo-inversion of matrices. The resulting solution has a number of advantages,
but also has certain disadvantages. One of them is that the solution obtained with the help
Figure 1: Registration scheme of a one-dimensional signal. 𝑙 is the size of the scanning integrator
aperture 𝐼1 , 𝐼2 , . . . , 𝐼(𝑛−1)𝑙+1 — measurement results.
239
Aleksandr L. Reznik et al. CEUR Workshop Proceedings 238–245
of pseudo-inversion is generally unbalanced (the sum of the weight coefficients with which
the samples-observations are included in the solution is different for different elements of the
generated image signal), and this leads to an increase in the variance the restored field, which is
not always acceptable.
In contrast to the classical approach, which leads to a solution with the minimum norm, in
this work we are looking for a solution with the minimum variance (energy), which is not the
same in the general case. The solution with the minimum norm is the “least bright” signal
corresponding to the system of observations, while the solution with the minimum variance
selects the “smoothest” solution from all images that satisfy the observation system.
Let us write the system of equations corresponding to the observation vector
𝐼 = (𝐼1 , 𝐼2 , . . . , 𝐼(𝑛−1)𝑙+1 ):
⎧
⎪ 𝑥1 + 𝑥2 + · · · + 𝑥𝑙 = 𝐼1 ,
⎪
⎨ 𝑥2 + 𝑥3 + · · · + 𝑥𝑙+1 = 𝐼2 ,
⎪
.. (1)
⎪
⎪
⎪ .
𝑥(𝑛−1)𝑙+1 + 𝑥(𝑛−1)𝑙+2 + · · · + 𝑥𝑛𝑙 = 𝐼(𝑛−1)𝑙+1 .
⎩
It is easy to see that in this case the mean value of the signal
𝑥1 + 𝑥2 + · · · + 𝑛𝑁
⟨𝑥⟩ = =
𝑁
(𝑥1 + · · · + 𝑥𝑙 ) + (𝑥𝑙+1 + · · · + 𝑥2𝑙 ) + (𝑥(𝑛−1)𝑙+1 + · · · + 𝑥𝑛𝑙 )
= = (2)
𝑁
𝐼1 + 𝐼𝑙+1 + 𝐼2𝑙+1 + · · · + 𝐼(𝑛−1)𝑙+1
=
𝑁
is a constant expressed in terms of the elements of the observation vector 𝐼 and independent of
the variables 𝑥𝑖 . Let’s write the expression for the variance
𝑁 𝑁
1 ∑︁ 2 1 ∑︁ (︀ 2
𝑥𝑖 − 2⟨𝑥⟩𝑥𝑖 + ⟨𝑥⟩2 =
)︀
𝐷𝑥 = (𝑥𝑖 − ⟨𝑥⟩) =
𝑁 −1 𝑁 −1
𝑖=1 𝑖=1
(︃ 𝑁 )︃
1 ∑︁ 1 (𝐼1 + 𝐼𝑙+1 + 𝐼2𝑙+1 + · · · + 𝐼(𝑛−1)𝑙+1 )2 (3)
= 𝑥2𝑖 − =
𝑁 −1 𝑁 −1 𝑁
𝑖=1
1
= ||𝑥||2 − const ⇒ min .
𝑁 −1
To find this solution, we will do the following. Let us fix free variables 𝑥1 , 𝑥2 , . . . , 𝑥𝑙−1 and
240
Aleksandr L. Reznik et al. CEUR Workshop Proceedings 238–245
express through them the remaining variables 𝑥𝑙 , 𝑥𝑙+1 , . . . , 𝑥𝑁 :
𝑥𝑙 = 𝐼1 − 𝑥1 − · · · − 𝑥𝑙−1 ,
⎧
⎪
⎪
⎪
⎪
⎪
⎪ 𝑥𝑙+1 = 𝑥1 + (𝐼2 − 𝐼1 ),
𝑥 𝑙+2 = 𝑥2 + (𝐼3 − 𝐼2 ),
⎪
⎪
.
⎪
⎪
⎪ ..
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪ 𝑥2𝑙−1 = 𝑥𝑙−1 + (𝐼𝑙 − 𝐼𝑙−1 ),
⎪
⎪
⎪
⎪ 𝑥2𝑙 = 𝐼1 − 𝑥1 − · · · − 𝑥𝑙−1 + (𝐼𝑙+1 − 𝐼𝑙 ),
𝑥2𝑙+1 = 𝑥1 + (𝐼2 − 𝐼1 ) + (𝐼𝑙+2 − 𝐼𝑙+1 ),
⎪
⎪
⎪
⎪
𝑥 = 𝑥2 + (𝐼3 − 𝐼2 ) + (𝐼𝑙+3 − 𝐼𝑙+2 ),
⎪
⎨ . 2𝑙+2
⎪
⎪
⎪
.. (4)
⎪ 𝑥3𝑙−1 = 𝑥𝑙−1 + (𝐼𝑙 − 𝐼𝑙−1 ) + (𝐼2𝑙 − 𝐼2𝑙−1 ),
⎪
⎪
⎪
𝑥3𝑙 = 𝐼1 − 𝑥1 − · · · − 𝑥𝑙−1 + (𝐼𝑙+1 − 𝐼𝑙 ) + (𝐼2𝑙+1 − 𝐼2𝑙 ),
⎪
⎪
⎪
.
⎪
..
⎪
⎪
⎪
⎪
⎪
⎪
𝑥(𝑛−1)𝑙+1 = 𝑥1 + (𝐼2 − 𝐼1 ) + (𝐼𝑙+2 − 𝐼𝑙+1 ) + · · · + (𝐼(𝑛−2)𝑙+2 − 𝐼(𝑛−2)𝑙+1 ),
⎪
⎪
⎪
⎪
(𝑛−1)𝑙+2 = 𝑥2 + (𝐼3 − 𝐼2 ) + (𝐼𝑙+3 − 𝐼𝑙+2 ) + · · · + (𝐼(𝑛−2)𝑙+3 − 𝐼(𝑛−2)𝑙+2 ),
⎪ 𝑥
⎪
⎪
.
⎪
..
⎪
⎪
⎪
⎪
⎪
⎪
𝑥 = 𝑥𝑙−1 + (𝐼𝑙 − 𝐼𝑙−1 ) + (𝐼2𝑙 − 𝐼2𝑙−1 ) + · · · + (𝐼(𝑛−1)𝑙 − 𝐼(𝑛−1)𝑙−1 ),
⎪
⎪
⎩ 𝑛𝑙−1
⎪
⎪
𝑥𝑛𝑙 = 𝐼1 −𝑥1 −. . .−𝑥𝑙−1 +(𝐼𝑙+1 − 𝐼𝑙 )+(𝐼2𝑙+1 − 𝐼2𝑙 )+. . .+(𝐼(𝑛−1)𝑙+1 − 𝐼(𝑛−1)𝑙 ).
Substituting (4) into (2) and equating to zero the partial derivatives of the resulting expression
with respect to the variables 𝑥1 , 𝑥2 , . . . , 𝑥𝑙−1 , we obtain a set of (𝑙 − 1) relations
𝑛𝑥1 + 𝑛𝑥2 + · · · + 2𝑛𝑥𝑖 + · · · + 𝑛𝑥𝑙−1 =
[︀ ]︀
= 𝑛𝐼 − (𝑛−1)(𝐼𝑖+1 −𝐼𝑖 )+(𝑛−2)(𝐼𝑙+𝑖+1 −𝐼𝑙+𝑖 )+. . .+1 × (𝐼(𝑛−2)𝑙+𝑖+1 −𝐼(𝑛−2)𝑙+𝑖 ) +
[︀ ]︀ (5)
+ (𝑛−1)(𝐼𝑙+1 −𝐼𝑙 )+(𝑛−2)(𝐼2𝑙+1 −𝐼2𝑙 )+. . .+1 × (𝐼(𝑛−1)𝑙+1 −𝐼(𝑛−1)𝑙 ) ,
𝑖 = 1, . . . , 𝑙 − 1
and, after simple transformations,
𝑛𝑥𝑖 = 𝑛𝑥1 −{(𝑛−1) [(𝐼𝑖+1 −𝐼𝑖 )−(𝐼2 −𝐼1 )]+(𝑛−2) [(𝐼𝑙+𝑖+1 −𝐼𝑙+𝑖 ) − (𝐼𝑙+2 −𝐼𝑙+1 )]+
[︀ ]︀}︀ (6)
+. . .+ 1× (𝐼(𝑛−2)𝑙+𝑖+1 −𝐼(𝑛−2)𝑙+𝑖 )−(𝐼(𝑛−2)𝑙+2 −𝐼(𝑛−2)𝑙+1 ) , 𝑖 = 2, . . . , 𝑙−1.
Successively applying the last relation (6) to the variables 𝑥2 , . . . , 𝑥𝑙−1 and then substituting
the obtained expressions into the first of the equations of system (5), we obtain a solution for
the element 𝑥1 :
[︂ ]︂ [︂ ]︂ [︂ ]︂
2𝑛 − 1 + (𝑙 − 2)(𝑛 − 1) 𝑛−1 1
𝑥1 = 𝐼1 + − 𝐼2 + 𝐼 +
𝑛𝑙 𝑛 𝑛𝑙 (𝑛−1)𝑙+1
𝑛−2
∑︁ {︂[︂ (𝑛 − 𝑖 − 1)𝑙 + 1 ]︂ [︂ ]︂ }︂ (7)
𝑛−𝑖−1
+ 𝐼𝑖𝑙+1 + − 𝐼𝑖𝑙+2 .
𝑛𝑙 𝑛
𝑖=1
241
Aleksandr L. Reznik et al. CEUR Workshop Proceedings 238–245
Further, from (6) we find a solution for the elements 𝑥2 , . . . , 𝑥𝑙−1 :
𝑛−2
1 ∑︁
𝑥𝑖 = 𝑥1 − (𝑛 − 𝑠 − 1) [(𝐼𝑠𝑙+𝑖+1 − 𝐼𝑠𝑙+𝑖 ) − (𝐼𝑠𝑙+2 − 𝐼𝑠𝑙+1 )] , 𝑖 = 2, . . . , 𝑙 − 1. (8)
𝑛
𝑠=0
And, finally, we successively obtain a solution for all other elements of the photomatrix:
𝑗−1
∑︁
𝑥𝑗 = 𝐼𝑗−𝑙+1 − 𝑥𝑞 , 𝑗 = 𝑙, . . . , 𝑁. (9)
𝑞=𝑗−𝑙+1
2.2. Improving the spatial resolution of images — two-dimensional case
The problem of reconstructing a two-dimensional field with a minimum dispersion from a set
of low-resolution images shifted relative to each other is formulated as follows (Figure 2).
We need to construct an estimate for image elements 𝑋 = (𝑥𝑚𝑛 ), which would satisfy the
conditions:
∑︁ 𝑗+𝑙−1
𝑖+𝑙−1 ∑︁
1) 𝑥𝑝𝑞 = 𝐼𝑖𝑗 , 𝑖 = 1, . . . , 𝑀 − 𝑙 + 1, 𝑗 = 1, . . . , 𝑀 − 𝑙 + 1,
𝑝=𝑖 𝑞=𝑗
𝑀 ∑︁𝑁 𝑀 𝑁
(10)
∑︁ 2 1 ∑︁ ∑︁
2) (𝑥𝑝𝑞 − ⟨𝑥⟩) ⇒ min, where ⟨𝑥⟩ = 𝑥𝑝𝑞 .
𝑀𝑁
𝑝=1 𝑞=1 𝑝=1 𝑞=1
In the discrete case to restore the two-dimensional field 𝑋 = (𝑥𝑚𝑛 ), 𝑚 = 1, . . . , 𝑀 , 𝑛 =
1, . . . , 𝑁 according to the observed data, which is a matrix 𝐼 = (𝐼𝑖𝑗 ), 𝑖 = 1, . . . , 𝑀 − 𝑙 + 1,
𝑗 = 1, . . . , 𝑁 − 𝑙 + 1, the solution with the minimum variance is found by analogy with the one-
dimensional case with the only difference that the variance of the reconstructed field is expressed
in terms of its bordering elements (i.e., elements 𝑥𝑖𝑗 for which 𝑖 ≤ 𝑙 − 1 or 𝑗 ≤ 𝑙 − 1). Thus, the
dimension of the problem being solved decreases from 𝑀 × 𝑁 to (𝑙 − 1) × (𝑀 + 𝑁 − 𝑙 + 1), and
Figure 2: Scheme of registration of an image consisting of 𝑀 × 𝑁 pixels to be restored. The integrating
aperture has the dimensions of 𝑙 × 𝑙 elements. 𝐼𝑖𝑗 — measured results.
242
Aleksandr L. Reznik et al. CEUR Workshop Proceedings 238–245
the filter matrix, which is responsible for the formation of the restored field and does not depend
on the observation results, is calculated in advance only once. If the field size is a multiple of
the linear size 𝑙 of the integrating aperture, i.e. when 𝑀 = 𝑚 × 𝑙 and 𝑁 = 𝑛 × 𝑙, the solution is
found using factorization by two-fold sequential application of the one-dimensional procedure
described above. For this, new vectors are formed 𝑦𝑗 = (𝑦𝑗1 , 𝑦𝑗1 , . . . , 𝑦𝑗𝑚 ), 𝑗 = 1, . . . , 𝑁 −𝑙+1:
𝑙
∑︁
𝑦𝑗𝑘 = 𝑥𝑘,𝑗+𝑟−1 , 𝑘 = 1, . . . , 𝑀 (11)
𝑟=1
and then, for each of 𝑗 = 1, . . . , 𝑁 − 𝑙 + 1, a one-dimensional problem is solved (according to
the already described algorithm):
𝑙
∑︁
1) 𝑦𝑗,𝑘+𝑖−1 = 𝐼𝑘𝑗 , 𝑘 = 1, . . . , 𝑀 − 𝑙 + 1,
𝑖=1
𝑀 𝑀 𝑀 (12)
∑︁ 2 1 ∑︁ ∑︁
2) (𝑦𝑖𝑗 − ⟨𝑦𝑗 ⟩) ⇒ min, where ⟨𝑦𝑗 ⟩ = 𝑦𝑗𝑖 .
𝑀
𝑖=1 𝑖=1 𝑖=1
In conclusion, when all the elements of the solution (12) are found by the “columns”, i.e. all
variables (𝑗 = 1, . . . , 𝑁 − 𝑙 + 1, 𝑘 = 1, . . . , 𝑀 ) are defined, the initial field is restored as a
result of the independent solution of 𝑀 one-dimensional problems for each 𝑘 = 1, . . . , 𝑀 :
𝑙
∑︁
1) 𝑥𝑘,𝑗+𝑟−𝑙 = 𝑦𝑗𝑘 , 𝑗 = 1, . . . , 𝑁 − 𝑙 + 1,
𝑟=1
𝑁 𝑁 𝑁 (13)
∑︁ 2 1 ∑︁ ∑︁
2) (𝑥𝑘𝑞 − ⟨𝑥𝑘 ⟩) ⇒ min, where ⟨𝑥𝑘 ⟩ = 𝑥𝑘𝑞 .
𝑁
𝑞=1 𝑞=1 𝑖=1
A great advantage of the developed algorithms is that the filter matrix for specific parameters
of restoration can be calculated in advance, which significantly increases the speed of restoration
and, in fact (using modern computing equipment), allows the restoration of images in “real
time”.
2.3. Results
The proposed methods have been tested on a large number of real and artificially generated
control digital images. In the overwhelming majority of cases, a high quality of recovery has
been demonstrated [6, 7]. Examples of image reconstruction with minimal variance are shown
in Figure 3.
3. Improving the resolution of video sequences
To restore the spatial resolution of video sequences, the same algorithms are used that are
described in Section 2 with the only difference that each of the frames of the video sequence
(we use standard video with 24 frames per second) must be processed independently, which
243
Aleksandr L. Reznik et al. CEUR Workshop Proceedings 238–245
a b c
Figure 3: Examples of images with minimum variance obtained at different sizes of the integrating
aperture (2 × 2, 7 × 7): a — original image; b — “bad-resolution” image obtained using the original image
and the integrating aperture; c — reconstructed image (from b) with minimum variance.
leads to a significantly longer calculation time. This time can be reduced by the preliminary
calculation of the filter matrix for fixed video parameters, however, the whole video processing
process can be quite long for high-resolution video.
When restoring video sequences, the key problem is not the speed of calculations, but the
precise positioning of video cameras (it is necessary to achieve the same conditions for subpixel
shift, as in the case of processing two-dimensional digital images, as shown in Figure 2).
Simulation on test video sequences showed that with correct positioning, these algorithms can
be successfully used for video sequences with obtaining video files of higher spatial resolution.
Figure 4 shows examples of image restoration with 9 cameras installed in a 3 × 3 array.
a b
Figure 4: An example of an image from one of 9 low-resolution video cameras (a) and reconstructed
image based on data from 9 video cameras (b).
244
Aleksandr L. Reznik et al. CEUR Workshop Proceedings 238–245
4. Conclusion
The most important criterion in evaluating the developed algorithms is to determine how
the images with the minimum variance correspond to real images. Studies have shown that
most real images are quite “smooth”, which makes the developed algorithms very effective in
improving the spatial resolution of such images.
An important distinctive feature of the developed methods is that when performing calcula-
tions on special computers, the matrix templates necessary for carrying out all the required
mathematical operations can be calculated in advance, which significantly reduces the compu-
tational capacity of the algorithms.
The proposed methods have been tested on a large number of real and artificially gener-
ated control digital images and video sequences. The high quality of restoration has been
demonstrated in the overwhelming majority of cases.
Acknowledgments
This work was partially supported by the Russian Foundation for Basic Research (project No. 19-
01-00128), and Ministry of Science and Higher Education of the Russian Federation (project
No. 121022000116-0).
References
[1] Cressler J.D. Silicon Earth: Introduction to microelectronics and nanotechnology, second
edition. CRC Press, 2015. 617 p.
[2] Williams J.B. The electronics revolution: Inventing the future, first edition. Springer, 2017.
296 p.
[3] Lawson C.L., Hanson R.J. Solving least squares problems. Society for Industrial and Applied
Mathematics, New Ed edition, 1987. 350 p.
[4] Bates R., McDonnell M. Image restoration and reconstruction. Oxford University Press,
1986. 320 p.
[5] Pratt W.K. Digital image processing: 4th edition. Wiley-Interscience, 2007. 812 p.
[6] Reznik A.L., Efimov V.M., Vas’kov S.T. Optimal image reconstruction by results of two-
coordinate subpixel scanning // Pattern Recognition and Image Analysis. 2007. Vol. 17.
No. 2. P. 211–216.
[7] Efimov V.M., Reznik A.L., Torgov A.V. Application of increase spatial sampling for im-
proving the resolution of images obtained from photodetector arrays // Optoelectronics,
Instrumentation and Data Processing. 2009. Vol. 45. No. 5. P. 399–402.
245