In their daily lives, people frequently need to obtain images with a high dynamic range and resolution. Due to technological equipment limitations, high dynamic range images are produced by multi-exposure fusion (MEF) of low dynamic range images, while high resolution images are frequently obtained by super-resolution (SR) of low resolution images. MEF and SR are often analyzed separately. This research examines existing approaches and proposes a coupled feedback network attention network and its method to address the issue that current models cannot achieve high dynamic range and high resolution simultaneously.
High dynamic range (HDR) images contain a broader dynamic range and richer texture features compared to typical low dynamic range (LDR) images and low resolution (LR) images, and high resolution (HR) images can enhance object detection accuracy. Technical methods to obtain HDR images and HR images, respectively, include single image super resolution (SISR) and multi-exposure image fusion (MEF).
By fusing two LDR images, the extreme exposure image fusion method creates an HDR image. Ma et al. [8] provided a quick approach for fusing multiple exposure images that improved the initial weights using a guided filter. Later, Xu et al [7] proposed a unified unsupervised fusion method that overcomes the fusion barrier of most images by constraining the similarity between the fused image and the original image.
With the continuous development of deep neural networks, many CNN-based methods have been proposed in the field of SISR. RCAN [4] introduces an attention mechanism to further improve the reconstruction quality. SRFBN [2] introduces a feedback structure to optimize shallow features through iteration to produce deeper features.
The above MEF and SISR methods are used to solve the LDR and LR problems, respectively, but in real life, people often need to see HDR and HR images on cell phones or TVs, so the joint MEF and SR methods are necessary. This paper proposes a coupled feedback attention network-based image exposure fusion and super-resolution method, which can effectively suppress the superposition of redundant information in cyclic iterations, improve the quality of parameter sharing as well as exposure feature propagation.
In order to solve the propagation of redundant features and enhance the propagation of useful features in the coupled feedback network, this paper combines the coupled attention mechanism and feedback mechanism, and proposes an image exposure fusion and image super-resolution method based on the coupled feedback attention network.
The structure of the coupled feedback network is shown in Fig. 1. The shallow features 𝐹 and 𝐹 go through T rounds of iteration by the coupled feedback attention module in the upper and lower network, respectively. The feedback features in each iteration combine the feedback features in the other network and the shallow features in this network, together as the input of the next iteration, to achieve the refinement fused features. The coupled-feedback attention layer contains multiple coupled-feedback blocks and an attention module.
The extraction process of shallow features 𝐹 and 𝐹 of LR images can be expressed as
where 𝑓 contains two convolutional layers Conv (3,4×m) and Conv(1,m), which are used to extract LR features and compress LR features, respectively. The extracted shallow features are first passed through SRB to obtain the deep features 𝐺 and 𝐺 , which can be expressed as
where 𝑓 is the super-resolution module (SRB) operation. Next, the deep exposure features of the two sub-networks are deeply fused after several iterations. At each iteration, the feedback features of the previous iteration are coupled and the shallow features 𝐹 and 𝐹 of the respective networks are together as the input of this iteration, and the feedback features 𝐶 and 𝐶 of the t-th iteration can be expressed as
This section specifically describe the specific iterative process of the coupled feedback block and channel attention module.
As shown in Fig. 2, the coupled feedback attention structure mainly contains iterative convolutional and deconvolutional layers constituting the CFB, and channel attention gates.
According to 3.1, in the upper sub-network, the inputs of the coupled feedback attention module are 𝐺 , 𝐺 , 𝐹 . firstly, the channel compression is performed through the convolutional layer Conv(1,m) to obtain the input 𝐿 (0) of the coupled feedback attention module.
Next, multiple working groups consisting of convolutional and deconvolutional layers, the HR feature 𝐻 (𝑛) of the n-th working group in the t-th iteration can be expressed as
where 𝑓 is the deconvolution layer Deconv (3,m). The HR features are generated by upsampling the LR features jointly from the first n-1 workgroups. Similarly, LR features 𝐿 (𝑛) can be expressed as
where 𝑓 is the convolutional layer Conv(3,m). The output of the final N-th working group is generated by the joint LR features of the previous N working groups passing through the convolution layer Conv(1,m) as follows.
The above describes the iterative process of the extreme high exposure branch, and the iterative process of the extreme low exposure branch is the same.
The feedback features 𝐺 and 𝐺 are output from each iteration, go through the channel attention module CA for feature optimization. The CA in this paper consists of three steps, which are global information compression, scaling and excitation, and recalibration.
In order to obtain the global information of each channel, this paper represents the feature values of each channel by global averaging pooling:
where 𝐺 (𝑖, 𝑗) and 𝐺 (𝑖, 𝑗) are the values at each position in the output extreme exposure feature, and compresses the multiple channels into a one-dimensional feature tensor.
2) Squeeze and excitation In order to more fully explore the dependencies between individual channels, the paper introduces a gate mechanism for learning the nonlinear mapping between each channel and uses a sigmoid activation function to avoid the formation of adversarial relationships between channels, which can be expressed as
Where 𝑊 and 𝑊 are the convolutional layer weights.
The original input features 𝐺 individual channels are scaled by the channel attention weight matrix just learned, thus enhancing useful features and suppressing useless features:
Figure 2 Coupled feedback attention structure
The method in this paper mainly achieves image super-resolution and image multi-exposure fusion, so the model in this paper uses a hierarchical loss function for optimization, and the loss function is expressed as
Where 𝐼 and 𝐼 are the HR standard images with extreme exposure, and 𝐼 is the HDR, HR standard image, which is the target to be achieved in the final fusion image. 𝜆 , 𝜆 , {𝜆 } are the weight coefficients of each loss part. In this paper, we set 𝜆 = 𝜆 = {𝜆 } = 1.
In this paper, the training model was trained on GeForce GTX 1070Ti.The experiments in this paper mainly use SICE [5] dataset, which contains 589 high-quality reference images and their corresponding image sequences, and only extremely exposure are used in this paper.
2
The network model proposed in this paper achieves both image super-resolution and image exposure fusion, we combine the current image super-resolution method and the image exposure fusion method as a comparison method. The image super-resolution methods are DBPN [3] , RCAN [4] , SRFBN [2] , and SwinIR [9] , and the main image exposure fusion methods are MGFF [10] , FAST SPD-MEF [6] , MEF-Net [8] , and U2Fusion [7] . We combined SR methods and MEF methods, and changed the order of SR methods and MEF methods, i.e., SR+MEF or MEF+SR, to generate 32 comparison methods. The CF-Net [1] was also selected for comparison.
In order to verify the effectiveness of the method in this paper under magnification factor of 2, we use the SICE dataset and compare it with other advanced methods. These comparison methods are combined by SR method and MEF method. Table 1 shows the results of our method with the comparison methods for magnification factor of 2 under three metrics.
In Table 1, highlighting the first value of the fusion quality index in bold and the second ranked value in underline. From Table 1, we can see that the method of this paper has the best fusion effect, ranking first among 34 methods in metrics. PSNR index is improved by 0.25 dB, SSIM by 0.0028, and MEF-SSIM by 0.0005 compared to the second place CF-Net method.
Fig. 3 visually depicts the fused images produced by this paper and other advanced methods at magnification of factor 2. From the experimental results, it can be seen that compared with SR+MEF and MEF+SR methods, the method in this paper has a great improvement in details, and compared with the coupled feedback network, this paper alleviates the phenomenon that there is redundant information in the image due to the coupled feedback mechanism.
Based on the powerful image reconstruction property of feedback mechanism and the property that channel attention mechanism can distinguish the importance of features. In this paper, a coupled feedback attention network is proposed to solve the image super-resolution problem and image exposure fusion problem simultaneously. The experimental results show that the algorithm in this paper retains
the detailed information of edges, region boundaries and textures of the original image sequence.