We propose the technology of image correction in mobile devices based on the use of a parametric FIR filter. The filter is constructed using a frequency response specified in the form of a parametric family of centrally symmetric frequency characteristics. A model of a one-dimensional frequency response is used in the form of segments of three functions: parabola, constant, and exponential function. Within the framework of the proposed filter model, a mobile application has been created that implements two tuning schemes: identification of the filter parameters from the reference image and adjustment without a reference. Besides, it is possible to vary the frequency response parameter characterizing the mid-range interval, which provides additional possibilities for adjusting the quality of the recovery. An example of processing test images is given.
Statistics of recent years show that the volume of traffic of mobile devices on the Internet is growing rapidly. A mobile application is a program that works on tablet PCs and smartphones. The number of smartphone owners around the world is growing every day. Music, working with photos, social networks the most popular types of applications among smartphone owners [1]. One of the most popular functions of mobile devices is image registration[2-3]. Mass use of this function is associated with the ability to quickly register images in unexpected and unique situations. In this case, often you have to deal with problems due to defocusing and/or blurring (when the object being recorded is rapidly moving relative to the camera) [4]. Distortions such as blurring and defocusing are usually called dynamic. The task of correcting dynamic distortions in images recorded by mobile devices using the digital image processing algorithms implemented directly in the mobile device itself is extremely relevant.
Known classical approaches to the construction of filters for image processing are inverse filtering and Wiener filtering. When constructing an inverse filter, one often has to face the fact that the inverse operator does not exist or the corresponding transfer function has poles close to zero [5]. At the same time, noise is emphasized on the restored image. In the Wiener filter, this problem is overcome by taking into account in the filter transfer function the frequency characteristics of noise [6]. However, in practice these characteristics are often absent, and the synthesis of the optimal Wiener filter becomes a serious problem. Unfortunately, other known methods of constructing filters, in one form or another, also face these problems [5], [6]. Therefore, often filters for the correction of dynamic distortions are built by parametric identification in the class of filters with finite impulse response (FIR filters). In [7], such a technology was considered, based on the use of a parametric model of the frequency response in the form of segments of quadratic and exponential functions. In this work, two variants of the technology were considered: identifying the parameters of the impulse response by use cases and setting the filter by indirect characteristics without a reference image.
In [8], the model of the frequency response from segments of quadratic and exponential functions was generalized, in particular, the region of medium frequencies was expanded due to the adding of a frequency response constant interval. On model examples it was shown that this modification improves the quality of image reconstruction in comparison with the previous realization of the quadratic exponential FIR filter. However, the implementation of a filter with an extended frequency response region requires, albeit insignificantly, an increase in computational resources. In this paper, we present the results of work on developing a mobile application that implements both of the above options.
The work is structured as follows. Section 1 describes the methods and algorithms for filter adjustment, both for the reference image and for the absence of a reference. In Section 2, the libraries used for constructing the mobile application, pseudo- code and user interface are explained. Section 3 provides examples of processing test images in a mobile device.
An FIR filter with a radially symmetric real frequency response [5, 6] is constructed with a supporting region D in the form × - a square with the center at the point k1 = 0, k2 = 0. Assuming that the central sample of the impulse response h(0,0) of the reference region D is at point 1, 2 readings of the restored image ( 1, 2) [5] can be represented as:
( 1, 2) = 2), −1 −1 ∑ 2 ∑ 2
1=− 2−1 2=− 2−1 ℎ[ ( 1, 2)] ( 1 − 1, 2 − where ( 1, 2) = √ 12 + 22, and ℎ[ ( 1, 2)] – counts of the one-dimensional impulse response defined on a set of circles with radii = ( 1, 2), 1, 2 ∈ .
The one-dimensional frequency
function is used for all values 0 ≤
and exponential function:
(
0 ≤ < 1,
= 12, 1 ≤
− , ≥ 2,
≤ 2,
(
Filter, corresponding to the described frequency response, hereinafter referred to as a generalized square-law exponential filter (Generalized Square-Exponential) or briefly a GSE-filter. The graph of this function is shown in figure 1.
The impulse response corresponding to this spectral characteristic, by virtue of the radial symmetry property, is obtained as a function of the spatial parameter r inverse Fourier transform: h r 1 Re S e jr d 1 0
1 2
Re a2e jr d Ae jr d eсe jr d
0 1 2
(
r
sin 1r
2 cos 1r
(we excluded and 2, taking into account (
It is easy to see that when = 1 impulse response (
The counts of the two-dimensional impulse response, as in [8], are determined by discretizing the
continuous function (
ℎ(0,0) = ∑ ℎ( 1, 2), ∀ 1, 2 ∈ , 1, 2 ≠ 0. (
Then, all the samples in the reference area are normalized so that the requirement of maintaining the average brightness level of the processed image is fulfilled:
∑ ℎ( 1, 2) = 1, ∀ 1, 2 ∈ . (
3. Processing of the distorted image and calculation of the quality criterion ( ̂1, ̂, ̂). 4. If the ( ̂1, ̂, ̂) > −1( ̂1, ̂, ̂), estimates ̂1, ̂, ̂ save, otherwise, according to some rule, a new variant of estimates is formed and the transition to step 2 is performed. If all estimates from the range of admissible values are "viewed" is an output.
Based on the described technology, it is possible to adjust the filter parameters both from the reference image and without the reference. When setting up a benchmark as a criterion ( ̂1, ̂, ̂) proximity of the restored image to the standard is used indicator:
MAX 2 PSNR 10log10
MSE , (
In the filter adjustment technology without a reference image, an MSE is calculated between the
recovered and the original distorted images. In this case, in contrast to the setting according to the etalon,
the improvement in the quality of the restored image is accompanied by a decrease in the PSNR (
Thus, checking the fulfillment of the conditions on the k-th iteration ( ̂1, ̂, ̂) > −1( ̂1, ̂, ̂) on step 4 of the described technology, it reduces to verifying that the following conditions are met:
PSNR ˆk , cˆk , ˆ k ) PSNR ˆk1, cˆk1, ˆ k1 )
D ˆ , cˆk , ˆ D ˆ , cˆk , ˆ
k k k k , PSNR ˆk , cˆk , ˆ PSNRдоп .
k
(
Here ( ̂ , ̂, ̂ ) value indicator (
The development of any mobile image processing application is divided into 2 parts: the development of a filter and the development of the main part. For the first part - the implementation of the image processing algorithm - the OpenCV library was used, a special version for the Android platform. OpenCV is a crossplatform open source library that is free for commercial and academic use. OpenCV contains algorithms for interpreting images, determining optical distortions, determining the similarity, analyzing the movement of an object, determining the shape of an object, and much more, written in C / C ++ and using parallelism and multi-core for the tasks performed.
To use the OpenCV library on the Android platform there is also a ready-made Android set NDK - tool that allows to realize part of the Android app compiled programming languages such as C and C ++ and contains a library to manage activities and access to various physical components of the device. Android NDK is integrated with tools from a set of components for software development (Android SDK), as well as with the integrated development environment of Android Studio [9-11].
Thus, there are 2 ways to use the OpenCV library on the Android platform: 1. OpenCV Java API + Android SDK: functionality is written in the Java language; 2. OpenCV native interface + Android NDK: functionality is written in C ++.
We will use the first way.
The main library modules used are: module Core - contains basic operations, including arithmetic; module Imgproc - responsible for image processing; module Utils - contains helper methods, e.g., conversion Bitmap image format into a format Mat
OpenCV and back and Mat download from a resource identifier for the resource.
Since we are working with a limited amount of memory, it is advisable to load an image with a lower resolution into the memory. The version with the reduced resolution should correspond to the size of the component of the user interface that displays it. A high-resolution image takes up a large amount of memory and does not provide a visible advantage when displayed. The choice of the option can be made by the user taking into account the characteristics of a particular mobile device. Figure 2 shows the program's pseudo code, which implements the algorithm described in Section 2, written with regard to the features of programming on mobile devices.
To work with bitmaps, we use the type Bitmap, which allows you to maintain the responsiveness of the user interface and reduce the amount of memory [12,13]. Next, for image processing, the Bitmap type is converted to the Mat type. Mat is a class for storing images that can be interpreted in C ++.
Using the BitmapFactory class, you can learn the resolution and type of graphics data before creating a Bitmap object and before allocating memory to that object. When you receive information about the resolution and size of the image, you can decide whether to load into memory the full-sized version of the image or its reduced version. We list some factors that need to be taken into account. Image Processing and Earth Remote Sensing
Start start function1 if r > 0,5 then 2 ( cos 1 − sin 1 )) end start function2
2+ 2 end
end end
end end
end keep values of w/с/α keep value of PSNR end for w ∈ (w_In, w_End)/с ∈ (с_In, с_End)/α ∈ (α _In, α_End) with Step_w/Step_c/Step_ α do ℎ( , ) = exp(− 1)(sin 1 + sin 1 + 2 cos 1 − 2 sin 21 + (sin 1 −sin 1 ) + 2 1 3 1 for i ∈ (1, N) with step = 1 do for j ∈ (1, N) with step = 1 do ( , ) = ( , )+ ( − + , −
+ )ℎ( , ) +1 2 +1 2
New value of PSNR = psnr(XR, X, 1) if calculated value of PSNR < previous value of PSNR and dispersion of processed image(XR) > dispersion of distorted image (X) then for i ∈ (1, N) with step = 1 do
for j ∈ (1, N) with step = 1 do = |√( − +1)2 + ( − +1)2| 2
End
Output: Processed image. 1. Estimated memory consumption when downloading the full version.
2. The amount of memory that can be spent on an image, considering the total memory consumption of the application.
3. Resolution of the component that will display the downloaded data. 4. The size and density of the screen points on the current device.
Methods of the BitmapFactory class should not be executed in the main thread of the user interface, so as not to reduce system performance. It is impossible to predict how long it will take to download data and processing them, it depends on various factors (speed of reading from the disk, the size of pattern to be displayed, processing power, and e.g.). If such a task will block the main UI thread, the system will mark the application as hovering, and you can get out of it, and even if removed.
If you want to perform operations that take some time, they must necessarily be performed in separate threads, called "background" or "worker" threads. To implement this task, we use the class AsyncTask, which offers a simple and convenient mechanism for moving laborious operations into the background thread.
The method of the AsyncTask class allows you to perform asynchronous operation in the user interface. About nor can perform minor image loading operations, file operations, database operations etc. in the workflow, and then publish the results in the main thread of the user interface without having to independently process the threads and/or handlers.
Figure 3 shows an application mockup built using an online tool for prototyping applications marvelapp.com.
When you enter the application, the user enters the main screen, where he can choose how to load the image: choose the finished one from the "gallery" or take a photo using the camera. And for newer versions of Android, starting at 6, you need to allow access to the application to the camera. In the next step, the user can see the image he uploaded in the user interface component, make a decision, process whether it is an image by the standard, and click on the button that will start the process. After processing, the user goes to the screen where he can see the result. The processed image is saved in a special folder of the application, accessible through the built-in "gallery" application.
To change filter parameters for better results during processing, the user can go to the "settings" section of the application main menu.
To get information about the application, the user needs to go to the "About the Program" section.
The diagram of the main application classes that implement the described functionality is shown in Figure 4.
Figure 4 shows the result of the filter with the configured parameters , с и a. Figure 6 (a) shows an example of a distorted halftone image by modeling a low-pass Gaussian filter with a blur: = 3. Figure 6 (b) shows the reconstructed image using a filter configured without a reference image, with a reference area dimension of 7 × 7. The result achieved during the restoration: PSNR = 25.13.
To quantify the achievable quality of recovery, we used test images (undistorted and distorted). We emphasize that the original undistorted image was not used to adjust the filter and restore it. We carried out a "blind" configuration.
(а) (b)
Figure 6. Images of "Lena": (a) - distorted with a blur = 3, (b) – processed.
This algorithm for image restoration without using a reference image can also be used to restore color images. In this case, the processing is performed using the same algorithms for each component.
The method of correction of dynamic distortions is based on the "blind" identification of the parameters of the restoring filter, which does not require the reference image. A program is developed on the Android platform, designed for image processing in mobile devices. An example of processing test image obtained by modeling the lowpass Gaussian filter is given. The result is achieved when reconstructing the image with a blur: = 3 by PSNR = 25, 13. The results obtained may be of interest to users of tablets and smartphones. Further plans of the authors are in order to make it accessible to the wide range of users.
The work was supported by the Ministry of Education and Science of the Russian Federation and the Russian Foundation for Basic Research, projects No. 17-29-03112, No. 16-07-00729-a.