Gibbs-ringing artifact is a common artifact in MRI image processing. As MRI raw data is taken in a frequency domain, 2D inverse discrete Fourier transform is applied to visualize data. Inability to take inverse Fourier transform of full spectrum (full k-space) leads to the insufficient sampling of the high frequency data and results in a well-known Gibbs phenomenon. It is worth to notice that truncation of high frequency information generates a significant blur, thus some techniques from other image restoration problems (for example, image deblur task) can be successfully used. We propose attention-based convolutional neural network for Gibbs-ringing reduction which is the extension of recently proposed GAS-CNN (Gibbs-ringing Artifact Suppression Convolutional Neural Network). Proposed method includes simplified non-linear mapping, amended by LRNN (Layer Recurrent Neural Network) refinement block with feature attention module, controlling the correlation between input and output tensors of the refinement unit. The research shows that the proposed post-processing refinement construction considerably simplifies the non-linear mapping.
Gibbs-ringing artifact reduction is an image restoration problem, that can be solved by mathematical methods of image processing.
Gibbs oscillations (Gibbs phenomenon) often occur near high-frequency image features, for example, edges. Artifacts can be observed while mapping image on a finer grid, during contrast enhancement, video compression and MRI data visualizing. Slight image distortions can be left invisible, while severe Gibbs artifacts may even create obstacles in patients diagnosing, if we refer to Gibbs oscillations caused by k-space (Fourier space) truncation of MRI frequency domain (see Fig. 1). ? The reported study was funded by RFBR, CNPq and MOST according to the research project 19-57-80014 (BRICS2019-394).
Simple finite real-valued periodic function can be observed to disclose mathematical reasons for Gibbs phenomenon: (t) = (a; if t 2 [ =2; =2] 0; if t 2 [ T =2; T =2] n [ =2; =2]
; (t) = (t + T ); where (t) is a basic model of a contrast edge, a is the amplitude of edge and T is the period of the model function.
Assuming Fourier transform in a complex form and T = 2 , (1) can be rewritten in a form: +1
X k= 1 (t) =
dkei!kt;
T where dk = T1 R 2T (t)e i!ktdt, !k = 2 k, = 2 =T . (t) = 2 a X+1( 1)k 2 k=0
1 (2k + 1) cos (2k + 1) where = 2 =T .
In practice it is impossible to include all terms in Fourier series (3), so Gibbs oscillations occur (see Fig. 2). The amplitude of Gibbs oscillations is constant for a given signal and doesn’t depend on a chosen cut-off frequency.
In this paper we propose a new CNN architecture for MRI Gibbs-ringing
suppression. It differs from recently introduced GAS-CNN [
The remainder of this paper is organized as follows. In Section 2 we observe some known methods for MRI Gibbs-ringing suppression. In Section 3 we describe MRI dataset generation, give a detailed overview of the proposed architecture and show (1) (2) (3)
Attention-based CNN for MRI Gibbs-ringing Artifact Suppression 3 profit of involving our modifications to the architecture. In Section 4 the results and comparisons are presented. The work is concluded in Section 5. 2
Gibbs-ringing reduction task has been solved by many methods so far. For example, the problem can be tackled as variational, and the solution can be searched as a function, which minimizes the stated functional in some functional space (L2 or L1, for example): 1 J (u) = 2 ku u0k2 +
Z
j ru(x) j dx ! mu2iUn;
(4)
where u0 is an input Gibbs-corrupted image, u is a searched Gibbs-free image from the
chosen functional space U , is the image’s area and is the regularization parameter.
The parameter can depend on the distance from the nearest image edge [
Another recently introduced method is based on a search of optimal subpixels shifts [
Deep learning methods have acquired great popularity in computer vision and image processing nowadays. Convolutional neural networks map input images in high dimensional feature spaces, implement filtering with a convolutional set and produce output images using final image reconstruction net.
GAS-CNN [
The following distinct model’s features were presented by authors:
– external U-Net [
We chose this recently proposed model as a baseline and decided to conduct a
research on the ways of non-linear mapping simplification with the maintenance of
generalization ability. Despite of making quite extensive analysis of GAS-CNN, for
example, showing the advantages in utilizing external skip connections and residual
learning and making comparisons with other methods (sinc filtering, bilateral filtering [
So, in this article we demonstrate the way to shrink the number of convolutions in the non-linear mapping by two times and save (even improve) model’s generalization ability, utilizing the proposed attention LRNN refinement module. 3
The proposed architecture is shown in Fig. 3. We call it GAS14-ACNN (Gibbs-ringing
Artifact Supression Attention-based Convolutional Neural Network). It comprises of
the following structural blocks: representing input corrupted samples into high
dimensional feature space of depth 64 with the first convolution; performing non-linear
mapping with 14 RCAN blocks [
Training, validating and testing synthetic sets were generated from ground truth MRI dataset IXI1 using the following pipeline: – apply Fourier transform to ground truth image 256 256 from IXI dataset; – crop frequency spectrum: central 19 part of frequency domain is saved; – implement zero-padding, so that Gibbs-corrupted image fits the shape of ground truth image; – apply inverse Fourier transform to get Gibbs-corrupted image.
Attention-based CNN for MRI Gibbs-ringing Artifact Suppression 5
N create image pairs fIgt; IGibbsgi=1 of the same spatial size. Zero-padding is often used before iFFT to project image on a finer grid, and zero-padding is often passed to project image on a coarse grid by inverse Fourier transform.
IXI dataset contains 581 T1, 578 T2 and 578 PD volumes. Firstly, the intersection of
these volumes was taking, producing 577 volumes, which have all three modalities: T1,
T2 and PD. Then, first 400 volumes were utilized to synthesize training set, next 100
1 http://brain-development.org/ixi-dataset/
volumes to create testing set and the rest of data was taken to generate validating set.
25 slices at both ends were discarded and every tenth slice was obtained to produce pair
(Igt; IGibbs)i [
Imin = min(IGT ); Imax = max(IGT );
IGibbs normed =
IGT normed =
IGibbs Imax IGT Imax
Imin ;
Imin Imin ; Imin (5) (6) (7) where IGibbs is Gibbs-corrupted image, IGT is ground truth image, IGibbs normed is normed Gibbs-corrupted image and IGT normed is normed ground truth image. 3.2
Images with Gibbs-ringing artifact obtain a significant blur also, as Gibbs-corrupted
images are generated by high frequencies truncation. Noticed that RCAN structural
module was successfully used in the recently published deep CNN architecture for
image deblur [
To the best of our knowledge the authors of GAS-CNN didn’t provide code and weights, so to make fair comparisons we trained all presented here models ourselves, utilizing the same training procedure (refer to Section 4 for details) and the same synthetic generated dataset.
GAS-CNN and RCAN-GAS-CNN were trained to evaluate RCAN performance. RCAN-GAS-CNN precisely matches GAS-CNN architecture with the only difference that RCAN module is used in non-linear mapping instead of an ordinary ResBlock. The performance growth is shown in Table 1 and in Fig. 6
Attention-based CNN for MRI Gibbs-ringing Artifact Suppression 7
LRNN refinement block was introduced in [
We utilized LRNN approach in solving Gibbs-ringing problem and, moreover,
extended it by attention mechanism, controlling correlation between LRNN input and
output features. Such attention module aims to force LRNN to be a refinement block.
Authors [
LRNN has two input tensors: feature tensor (non-linear mapping result), to be recursively processed, and weights tensor. We acquire weights by the auxiliary RNN weights generation net (see Fig. 3), which is trained end-to-end with the whole neural network. LRNN implements 4 recursive updates: left-to-right, right-to-left, top-to-down and down-to-top, applying the rule:
Ht+1 := (1 !)
Ht+1 + !
Ht; (8) where Ht is the current processing row, if we refer to top-to-down or down-to-up operations. Subsequent concatenation and convolution fuse recursively processed tensors and conclude LRNN operation.
LRNN can be viewed as an alternative (hybrid) way to enlarge receptive field and to accumulate global spatial information within the layer. Despite of the locality of Gibbs oscillations, mentioned above, accounting overall information within the layer occurred to be very helpful and remarkably raised generalization ability of the architecture.
We trained two extra CNNs to reveal LRNN advantages: – RCAN8-GAS-CNN – model accurately matches with original GAS-CNN, but the number of blocks in non-linear mapping is heavily decreased by 4 times; – RCAN8-GAS-CNN+LRNN 2 – previously stated model, extended by proposed LRNN refinement;
The obvious positive LRNN impact can be observed in Table 2 and in Fig. 7. RCAN8-GAS-CNN+LRNN 2 has the comparable performance with the original 4 times deeper GAS-CNN, whereas repealing of LRNN post-processing leads to algorithm’s degradation.
Attention module is shown in Fig. 8. It gets three tensors as inputs: x1 2 IRC H W – non-linear mapping output (LRNN input), x2 2 IRH W C – LRNN output, x3 2 IRC H W – transposed copy of x2. Attention block performs weighted regrouping of x3 features in a manner to uplift feature at position i, mostly correlated with ith feature of LRNN’s input tensor x1.
Attention-based CNN for MRI Gibbs-ringing Artifact Suppression 9
Assume f1; :::; fC to be x3 features. Introduce following latent variables A = fa1; :::; aC g. Value ai 2 f1; :::; Cg defines x3 feature, mostly correlated with ith feature of LRNN’s input tensor x1. Weighted regrouping is executed via computed correlation tensor C 2 IRC C (probability distribution map over the latent variables values).
It deserves mentioning that there is an analogy with word aligning task in classical machine learning (for example, IBM Model 1). Proposed attention LRNN unit performs like feature aligner of LRNN input and output tensors, forcing LRNN to be a refinement block.
Finally, we get the overall proposed architecture: RCAN14-GAS-CNN+Attention LRNN 2 (see Fig. 3). We call it GAS14-ACNN. Table 3 shows the increase of performance, caused by the attention unit. Proposed attention-based convolutional neural network for Gibbs-ringing reduction was implemented in Python 3 with the use of deep learning framework Tensorflow 1.14. We provide implementation of our code at2.
Models were trained with Adam optimizer [
We used the following values of these parameters: lr0 = 10 4, lr1 = 0, Ne = 1000, bs = 20, p = 0:3.
We applied L1 loss function with l2 weights regularization ( = 10 4) to prevent models’ overfit.
We utilized augmentation by rotations and flips for patches of shape (48 48) during training. 10 random patches were cropped from each training image before an augmentation. Validation and testing were performed on full-size images.
All convolutions’ kernels have spatial size 3 3, except for one projection convolution just before LRNN refinement unit: it has kernel of spatial size 1 1, and it projects features on some trainable manifold of the less dimension.
GAS-CNN, chosen baseline model, and the proposed GAS14-ACNN can be compared in Fig. 9 and Fig. 10. It takes approximately 1.03 sec. and 0.05 sec. to process one image (256 256) for GAS-CNN on CPU and GPU respectively3. And it takes approximately 1.13 sec. and 0.08 sec. to process one image (256 256) for our GAS14-ACNN on the same CPU and GPU respectively3.
Attention-based CNN for MRI Gibbs-ringing Artifact Suppression 11 We proposed the new attention-based convolutional architecture GAS14-ACNN for MRI Gibbs-ringing suppression. This architecture is the extension of recently proposed GAS-CNN model with significantly simplified non-linear mapping, followed by attention LRNN unit to save generalization ability. The presented attention mechanism acts as auxiliary constraint for LRNN post-processing and as feature filtering module. The proposed GAS14-ACNN model outperforms baseline GAS-CNN on the generated synthetic testing set.