-In this paper, we consider super-resolution image reconstruction using the method of projections onto convex sets. We explore an influence of input image set size on the result of super-resolution reconstruction. We propose an indicator value as a ratio between the number of images in the set and the square of upscale factor of reconstruction. The method of convex projections was implemented using the Python programming language. The experiments were conducted on the Standard test images from TESTIMAGES project set. The results and future plan for improving the POCS method for super-resolution reconstruction are discussed in the final part of the paper.
Super-resolution (SR) of an image provides a high pixel
density and, therefore, more details about the object can be
captured. The super-resolution problem is raised in computer
vision in regard to pattern recognition and image analysis [1]
[2], in the task of medical imaging [3] and the Earth remote
sensing [4]. CNN-based super-resolution algorithms were
successfully applied to image super-resolution problem [5],
[6]. These algorithms learn representations from large
training databases of high- and low-resolution image pairs or
exploit self-similarities within an image [7]. Super-resolution
imaging devices are expensive, and their usage is not always
possible due to sensor limitations and optical technology
(e.g., thermal imaging systems [8]). Image processing
algorithms partially solve these problems by simplifying the
system for obtaining images due to the greater computational
load. Existing methods for improving image resolution fall
into two large categories: linear [9] and adaptive [
easy to implement but do not allow us to completely extract information from source images. The use of adaptive methods provides a better result. Among the technologies for improving image resolution from the set of images, superresolution technology is the most effective.
images require multiple low-resolution images of the same
scene, which are aligned with sub-pixel accuracy [
(Fig. 1). In matrix form this observation model image is written as follows: [ ⋮ ] = [ ∙ ∙
⋮ ∙ ∙
] + [ ⋮ ] = [ ⋮ ] + [ ⋮ ]
where ( = ̅1̅̅,̅̅) are the low-resolution images with the
size of × pixels, is a subsampling matrix with the
size of 2 × 2 pixels; is a blurring matrix with the size
of 2 × 2 (the matrix is evaluated from the point spread
function (PSF) [
the POCS method. The operator of the corresponding convex set of constraints projects points from the solution space onto the nearest point on the surface of this convex set. After a finite number of iterations, a solution to the set of intersections comes to a convex set of constraints. following steps.
Calculation of the displacement (motion compensation) of pixels on each low-resolution.
The correspondence between high and low resolution images is given as ( 1, 2, ) = ∑ ( 1, 2)ℎ( 1, 2; 1′, 2′ ) where ( 1, 2) is a point of the interpolated low-resolution a corresponding point into
we evaluate the ℎ( 1, 2; 1′, 2′ ) parameter which is a value of point spread function according to the pixel position.
The obtained low-resolution image ( 1, 2, ) can be constrained by a convex set С 1, 2, . С 1, 2, = { ( 1, 2, ): | ( )( 1, 2, ) ≤ 0( 1, 2, )|} 0 ≤ 1, 2 ≤
− 1, = 1, … , arbitrary point ( 1, 2, ) can be represented as: Projection ( 1, 2, ) [ 1, 2, ] onto ( 1, 2, ) in ( 1, 2, ) + ℎ( 1, 2; 1 , 2 ) ( 1, 2, ) [ 1, 2, ] = ( )( 1, 2, )− 0( 1, 2, ) ∑∑ℎ2( 1, 2; 1′
, 2′ ) ( )( 1, 2, ) > 0( 1, 2, )
( 1, 2, ) − 0( 1, 2, ) < ( )( 1, 2, ) < 0( 1, 2, ) ( 1, 2, ) + ( )( 1, 2, )+ 0( 1, 2, ) ∑∑ℎ2( 1, 2; 1′ , 2′ ) ℎ( 1, 2; 1 , 2 ) {
( )( 1, 2, ) < − 0( 1, 2, )
We estimate a residual between the test image and the reconstructed using the described algorithm. The residual formula can be written as: ( )( 1, 2, ) = ( 1, 2, ) −
∑ ( 1, 2, ) · ℎ( 1, 2; 1′, 2′ ) where ℎ( 1, 2; 1′, 2′ ) is an impulse response coefficient, 0 is a confidence level for the observed results. These parameters define high-resolution images that correspond to low-resolution images within a confidence interval.
condition is met.
With the use of a projection operator, the estimated value ( 1, 2, ) of the high-resolution image can be found using all low-resolution images by performing some iterations: ̂( +1)( 1, 2, ) = ̃ [ ̂( )( 1, 2, )] = 0,1, …, where ̃ is a combination of all projection operators 0( 1, 2, ) is obtained by bilinear interpolation. associated with ( 1, 2, ) . The initial approximation
reconstruction using the method of POCS. An influence the parameters of image set formation and parameters of an above algorithm to the super-resolution reconstruction was investigated.
In this paper, an experimental study of the influence of the number of input images on the result of image reconstruction was carried out for different image scaling parameters.
Images from the TESTIMAGES project set [
programming language with libraries OpenCV [
The value of the relative "information completeness" has been proposed as a universal measure of the number of images in a set. This indicator was calculated as = { { 2 This value allows us to assess the degree of "completeness" of information. Evidently, in a randomly generated
set, an indicator value equal to 1 does not guarantee sufficient information to restore an absolutely accurate original image. However, it will be further shown that this indicator is quite meaningful.
Also, to assess the effect of the matching stage on the result, a test reconstruction was carried out using the same algorithm (POCS), but under the assumption that the image matching parameters are known (these parameters were stored at the low-resolution image generation stage). Fig. 2 shows dependence of the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM values on propose for different scale values for both experiments.
The experiment showed that in a perfect scenario (when the matching parameters are known) it is best to set the resolution upscaling parameter S so that the size of the set of images N satisfies the following constraints: 0.4 ≤ 2 .
are less than in the perfect scenario.
Moreover, adding images above the = 0.85 impair the result further. This is due to the fact that the matching itself does not always provide quite accurate result and POCS algorithm is not robust and requires additional procedures to 1.2 1,2 11 00.8,8 0.6 0,6 0,4 0.4 0,2 0.2 0 0 1.2 1 0.8 restriction: or, equivalently, 4400 3355 3300 2
We plan to investigate POCS robustness later in our further research. 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1
parameters of image set formation and the parameters of an algorithm on reconstruction. We have developed the recommendations for the super-resolution reconstruction problem using the method of POCS.
The research was carried out within the state assignment theme #0777-2020-0017, was partly financially supported by the RFBR under grant #17-29-03112 and #18-07-01390, and by the Ministry of Science and Higher Education.