1. Introduction
Image quality enhancing is a fundamental problem in image processing that has received great
attention over several decades. A high-quality image is always expected in different tasks of vision,
and degradations like low-resolution, blur, and noise, are required to be removed. While the classical
techniques for this task have achieved great progress, the recent top performer, deep techniques
approaches, can significantly and substantially boost performance compared with classical ones. The
advantages of deep learning techniques which enable it to realize such success are its high
representational capacity and the strong nonlinearity of the approaches [
1
].
Image enhancement is adjusting process of digital images so that the results are more
appropriate for display or additional image analysis [
2-5
]. Image enhancement methods divide to two
main categories: “spatial domain” approaches and “frequency domain” approaches [
1
]. The term
spatial domain in image processing indicates to an image plane itself, and methods in this category are
depending to direct processing of image pixels. Spatial domain enhancement methods also divide to
two categories: the spatial domain filtering category and transformation category. The former is
depending on neighborhoods and the latter is depending on individual pixels. The spatial domain
transformation approaches in common use are basic on histogram processing and gray level
transformation. The frequency domain processing approaches are depending on modifying an image
Fourier transforms. Let ( , ) be the original image and its Fourier transform ( , ), while ℎ( , )
be a filter and ( , ) its Fourier transform, and then ( , ) is transformed to ( , ) after
convolving with ℎ( , ) and ( , ) is Fourier transform of ( , ).
The procedure can be written in the spatial domain as:
The frequency domain as:
( , ) = ( , ) ∗ ℎ( , )
( , ) = ( , ) ∙ ( , )
Where “∗”is convolution operator and “∙”means multiplication operator. It is clear from Eqs.
(1,2) that if ℎ( , ) is selected correctly, the image ( , ) will then be influentially enhanced.
The rest of this paper will explore the development of advanced deep approaches for image
enhancement by researching several fundamental issues with various motivations.
2. Deep Learning Applications on Image Enhancement
In image denoising, the basic concept of learning in image denoising is to use training data to
refine a module of the designed model. Dictionary learning is an example of this; image patches can
be interpreted using a coefficients sequence on a collection of bases, resulting in a redundant
dictionary. The dictionary is supposed to express the general structures of natural images in such a
way that clean image patches are well estimated on it for the denoising task. As a result, as in KSVD
[
6
], LSSC [
7
], and CSR [
8
], the dictionary is an essential item that is learned from a collection of
highquality training data or the to-be-processed degraded image. Deep learning is another example, in
which the goal is to learn a discriminative restoration function. The first attempts to use (CNNs) and
stacked auto-encoders to denoising natural images [
9
] are promising, demonstrating that these deep
models can work like or better than traditional wavelet or Markov random field-based denoising
approaches.
Agostinelli et al. [
10
] investigated for building multi-column stacked auto encoders to deal
with different forms of noise. DnCNN [
11
] used the residual learning technique to train deep CNN
models and achieved state-of-the-art results. These studies show that deep architecture, strong
learning, and high representational ability can significantly improve image denoising efficiency.
Example of image denoising result using deep learning is shown in figure 1.
(1)
(2)
In image deblurring, the most critical issue is ill-posedness. The observed fuzzy images do not
stably and uniquely evaluate as sharp images in the non-blind case, owing to the blur operator's
illconditioned existence [
11
]. Researchers have been concentrating on creating new strategies and
prototypes, as well as improving the efficiency of optimization techniques, to deal with ill-posedness.
For the task of deblurring, learning-based approaches are proposed. The motives in this case can be
divided into two categories. In the first group, the proposed methods [
11-13
] aim to learn a subspace
in which the sharp picture can be found. A subspace can be created by extracting local patterns from
multiple sharp images, where the sharp images and target images share similar information, allowing
for an accurate representation of details in the target image. In the second group, the aim is to find a
restoring mechanism that can transform a fuzzy target image into a sharp target image [
13
]. Multiple
sharp images with their fuzzy equivalents are widely used to train the restoring function parameters in
this scenario. The “Regression Tree Field” (RTF) was used by “Schmidt” et al. [
14
] to model a
nonlinear regressor that defines the parameters of local deblurring. Example of image deblurring result
using deep learning is shown in figure 2.
In Image Super-resolution, the aim of super-resolution is to create a high-resolution image
from one or more low-resolution images, recapturing high-frequency information that were lost during
the imaging process. A belief network is a type of learning technique that can be expressed in terms of
a Markov network. The images are analyzed with patch representation using a Markov network [
16
].
An observation function connects the low-resolution patches and their corresponding high-resolution
patches, defining how well one fits the other. A transition function connects the neighbor patches in
the reconstructed image. The model uses the belief propagation algorithm to restore the
highresolution image after the parameters of the functions have been well trained.
High-resolution images form a manifold with similar local geometry to that provided by
lowresolution images, according to manifold learning-based approaches [
17
]. In order to do restoration,
the relationship between points on the low-resolution manifold can be directly applied to the
highresolution manifold. However, the assumption of identical manifolds is overly restrictive, and it cannot
be fulfilled in many situations. To address this issue, it is proposed that two explicit mapping functions
be learned to find a common manifold for low and high-resolution images [
18
]. The idea behind the
sparse coding-based approaches [
19-20
] is that a signal can be interpreted by a sparse code on an
overcomplete dictionary. The linear relationship between the sparse codes of low and high-resolution
images can be recovered in the super-resolution task, resulting in encouraging restoration results. A
high-resolution image can be obtained from low-resolution patches by dividing images into patches.
The mapping function types like simple functions, support vector regression, and anchored
neighborhood regression [
21
] are indeed shallow types with limited representational capacity. It has
been shown that stacking shallow types into a deep one can significantly improve the performance of
super-resolution, which is related to deep learning. Kim et al. conducted research into the use of very
deep convolutional networks for SISR [
22
], who suggested residual learning to train a 20-layer CNN
model and achieved high efficiency. They also presented a profoundly recursive convolutional
network [
23
] that uses a limited number of model parameters to allow for long-range pixel
dependencies. In the SR mission, use self-examples on different scales to fine-tune a pre-trained
convolutional auto-encoder [
24
]. Shi et al. and Dong et al. [
25-26
], constructed networks in which the
majority of the computation occurs in the LR space and the up-scaling process occurs only in the last
layer of the networks to speed up the SR computation. Example of image super-resolution result using
deep learning is shown in figure 3.
Besides the above CNN-related works, the studies on other feed-forward neural networks were
conducted, such auto-encoders [
27-28
] and sparse coding-based networks [
29-30
]. An LR image patch
can be generated from various HR image patches which reside in a low-dimensional natural image
ramified. The MSE-based solution is a pixel-wise average of the possible HR patches on the ramified,
thus exhibiting blurry or over-smoothing effects, and lacking high-frequency details. To address this
task, in [
31
] presented a perceptual loss which measures the MSE between the VGG feature maps of
the SR result and the ground truth. In [
32
], by incorporating this loss and an adversarial loss, Ledig et
al. developed a generative adversarial network which can recover photo-realistic textures. Using
similar ideas, inference models can be applied to the statistics of CNN feature maps, such that the
statistics of SR solutions and that of natural images are as close as possible [
32
].