This paper considers an application map method usage together with the Nose Suppression Filter (NSF) in-loop filter for video compression. The application map method is described in details and simulations results of basic NSF are provided as well as simulation results of NSF powered with the application map method. It is demonstrated that the application map method allows to significantly improve objective performance results of basic NSF and additionally decrease decoder average complexity. As a result, an average bd-rate saving of NSF with the application map reaches 2.6% for luma component in Low Delay P coding configuration in comparing to Versatile Video Coding reference implementation (VTM) version 1.0.
At this moment digital video industry is keep extensively growing. Modern consumer
devices already support Ultra High Definition (UHD) resolution and High Dynamic
Range (HDR) video content. First 5G smartphones already available in the market and
it is expected that 5G technology will shortly substitute the current one. New augmented
and virtual reality (AR/VR) solution have been being demonstrated very frequently
over all popular market shows. All such factors actively push video industry forward
and it is expected that total amount of video traffic in the Network will exceed 80%
already in 2020 [
Another in-loop filter from the first category is a Noise Suppression Filter (NSF)
that was firstly proposed in VVC Call for Proposals (CfP) responses [
This paper is organized as follows. The NSF algorithm is described in section 2, where section 2.1 briefly summarizes a general NSF design and the application map approach detailed description is given section 2.2. Simulation results are provided in section 3. The conclusion of the paper is given in section 4. 2
The Noise Suppression Filter (NSF) is a non-local collaborative filter, which can be used in video compression lossy systems as in-loop filter in order to improve reconstructed video frame quality. NSF is carried out after reconstruction process and based on a block matching procedure targeted to estimate and reduce quantization noise caused by quantization process of the hybrid video coding scheme. Basically, NSF
Application Map in the Noise Suppression Filter for Video Compression 3 comprises of a filter core part, which is shared by encoder and decoder, and the application map method, which checks whether NSF filtering is beneficial on a frame or block level at the encoder and transmit this information to the decoder. Such approach allows to amplify the filter benefits in terms of its complexity and performance by utilizing additional information available on the encoder side and remove complex filtering operations, which do not contribute to performance improvement at the decoder side. Fig. 1 gives a high-level NSF processing encoder side flowchart. As can be seen from Fig. 1, NSF usage at the encoder side requires several additional operations besides NSF core part. It should be also noted, the NSF core part is applied twice at the encoder: first time without considerations of application map, assuming that whole frame is a subject to filtering; and second time considering actually calculated application map. Such design decision is determined by the fact that in video codecs encoder and decoder have to process data in the same way in order to guarantee that decoded results will be the same for any standard compliant decoders. Due to this reason and considering that in real systems decoder complexity is typically more critical than encoder one, it was decided to intentionally make encoder side more complex in order to simplify decoder.
Fig. 2 gives a high-level NSF processing flowchart for the decoder. As can be seen from Fig. 2, the NSF usage at the decoder side is much simpler than the encoder one. The only complex operation of NSF core part is performed under the application map condition and even if the condition is satisfied, NSF core part considers actual values of application map inside the process. 2.1
The NSF core part is a main filtering procedure inside NSF. It takes whole reconstructed frame and the application map as inputs, performs filtering and outputs the filtered frame. The filtering procedure comprises of the following main steps. 1. Dividing the frame into a set of blocks; 2. Performing blocks classification; 3. Finding a set of similar blocks (stack) for each block by using a block matching procedure; 4. Performing a Frequency Domain Filtering (FDF) procedure within each stack; 5. Performing a pixel domain averaging of every blocks corresponding to a certain location.
Application Map in the Noise Suppression Filter for Video Compression 5 It should be noted that the application map is widely used during NSF core part methods as an early skip mechanism. At the first encoder call of NSF core part, the application map is considered to specify whole frame to be filtered and at the second call of NSF core part instant values of the calculated application map are being used.
A general flowchart of NSF core part is demonstrated in Fig. 3. At the first step of the algorithm, whole frame is divided onto the set of regular square 8×8 blocks. In order to increase efficiency an overlapping of blocks can be used at this stage.
Then for each block two conditions are checked. First, whether the block is marked “to be filtered” by the application map. In negative case the block is simply bypassed, and algorithm moves to the next block. If the block is marked “to be filtered”, the second condition checks whether it is flat or not. Insofar as NSF demonstrates the biggest efficiency being applied among objects edges, blocks classification allows to avoid all further NSF processing for the blocks where filtering would not be efficient.
The third step of the algorithm performs block matching procedure to collect a stack
of blocks similar to the current within a search range also considering the
application map limitations by removing picture areas from the search region, where NSF is
not applicable. Specific block matching method, which is used at this step is described
in detail in [
At the fourth step of the algorithm the collected stack is being filtered by a Frequency Domain Filtering (FDF) procedure. In general, FDF procedure performs a collaborative filtering of each sample within stack , where filter strength is determined by the only additional parameter that, in turn, is a function of codec quantization parameter (QP). Thus, FDF process can be summarized as follows (1): : × → ̂, were ̂ is a filtered stack. (1) 00 is a 0 − ℎ sample of 0 − ℎ block, 01 is a 0 − ℎ More specifically for a given stack of blocks, where each block includes 2 samples, FDF process first makes 2 lines of elements, where each element of the line is an − ℎ samples of − ℎ block inside . E.g. 0 = { 00, 01, … , 0 −1}, where of 1 − block, etc.
Thus, stack whole can be represented as 2 lines of elements. Next FDF performs Hadamard transform of each line and then for each frequency element from first AC it calculates a gain value by the following formula (2): staring
In the end of the process, FDF performs all required backward operations and computes filtered stack ̂ .
It should be noted that since = (
), and QP value is already presented in the bitstream, the whole FDF process does not require any additional information to be signaled in the bitstream. In other words, most of FDF parameters are implicitly derived from the video signal itself.
At the fifth step all filtered blocks within different stacks are being collected back to the frame into their original locations.
Finally, FDF filters each component in a following way (3): = ( 2 ) 2 ) + 2
. ( (2) ̂ =
× .
(3)
Application Map in the Noise Suppression Filter for Video Compression 7 2.2
One of the main features of NSF core part is an absence of additional filter parameters. Indeed, the only external parameters for the core part of NSF is value, which is function of QP, so it does not bring any additional overhead for parameters transmission. However, on the other hand that also means that NSF core part does not have any mechanisms to utilize information from the original picture, which is available at the encoder so the core part always relies only on reconstructed frame that would inevitably lead over filtered and under filtered regions in the frame.
In order to increase NSF filter flexibility and minimize a number of filtering errors the application map method was introduced. This method allows to estimate how beneficial is NSF core part filtering process on a frame or block level at the encoder side based on the original picture and transmit this information to the decoder side. More specifically, after NSF core part the encoder checks several following options of NSF applicability: 1. Apply NSF for the whole frame; 2. Do not apply NSF for the whole frame; 3. Apply NSF on a block-based level: a. With the block size equal to CTU; b. With the block size equal to half of CTU; c. With the block size equal to quarter of CTU; d. With the block size equal to one eighth of CTU.
In options 1 and 2 encoder simply choose whether or not to use NSF filter for the whole frame. This encoder decision should be signaled in the bitstream, however signaling overhead is only 2 bits per picture, which normally does not affect whole codec performance. In a) – d) options under category 3 encoder introduces the application map with different block size, where each item of the application map determines whether the NSF filter is used for the current block. Fig. 4 gives an example of an application map for the picture, where green blocks demonstrate picture areas to be filtered by NSF and red block demonstrate picture areas to be kept without filtering.
Blocks to be filtered by NSF
Blocks to be directly kept from the reconstruction
process Obviously, the application map usage and especially option 3 d) will always lead better performance comparing to other options. However, the application map information requires additional 1 bit for each block, where block size depending on CTU size. In modern video codecs CTU is normally square blocks of 128 samples, so option 3 d) will require additional signaling for each 16 × 16 block, which can visibly affect total bitrate. Thus, in order to find an optimal solution of NSF application map option, encoder should consider all possibilities taking into account both parameters: filtered picture distortion and additional bit budget required to transmit the decision.
Since the application map is signaled in the bitstream, decoder can similarly read and apply it within the decoding process. In addition, the application map approach also allows to reduce NSF decoder complexity. Indeed, since decoder already has the information about blocks that are not filtered, these blocks can be removed from NSF core part and thus reduce the number of filtering operations at the decoder. 3
The Noise Suppression filter was implemented in VVC Test Model (VTM) version 1.0
and simulation results were carried out within JVET Common Test Condition (CTC)
[
According to CTC, the simulations should be performed in 4 different test configurations corresponding to 4 following application scenarios: 1) All intra (AI) configuration that allows only Intra frames usage within the coded sequence; 2) Random Access (RA) configuration that allows both Intra and Inter frames and also allows frames reordering; 3) Low Delay B (LDB) configuration that allows both Intra and Inter frames, but disallows frames reordering; and 4) Low Delay P (LDP) configuration, which is similar to LDB, but allows only unidirectional prediction (P) for Inter frames.
CTC simulations are performed on a CTC video test sequences data set, which consists of 7 following test subsets of YCbCr video sequences: class A1, class A2, class B, class C, class D, class E and class F. Classes from A1 to D include natural camera video content with different resolution, where classes A1 and A2 include UHD content (3840 × 2160), class B – FHD content (1920 × 1080), class C – 832 × 480 resolution content, and class D – 416 × 240 resolution content. Class E includes HD resolution (1280 × 720) teleconferences content and class F includes screen content sequences of a different resolution. CTC defines some of the classes as optional for some configurations. Namely, for RA configuration, class E is not mandatory as well as classes A1 and A2 for both low delay configurations.
CTC evaluation methodology assumes encoding all required sequences by reference
(anchor) and test encoders with 4 predefines quantization parameters (QP) equal to 22,
27, 32 and 37. Then for both anchor and test materials rate-distortion (RD) curve is
being built for each of video signal components with the piece-wise cubic interpolation
and BD-rate metrics [
Table 1 - 4 demonstrate CTC simulation results for a general NSF without the application map method. More specifically, they correspond to encoder flowchart (Fig. 1)
Application Map in the Noise Suppression Filter for Video Compression 9 modification, where both application map conditions are always forced to decide whole frame to be filtered by NSF.
Positive numbers in the result tables represent bitrate increase with the same video quality between anchor and test and mean codec performance degradation. Similarly, negative numbers represent bitrate reduction with the same video quality between anchor and test and mean codec performance improvement.
U
Y
As can be seen from Table 1 - 4, a straightforward usage of NSF without the application map is certainly only beneficial for All Intra configuration, however in all other configurations there are many places where NS filtering actually contributes to codec performance degradation.
Table 5 - 8 demonstrate CTC simulation results for NSF method with the application map.
U The Noise Suppression filter is a powerful in-loop that can be used in lossy video systems in order to improve reconstructed video quality without any additional filter parameters signaling. However, in all coding scenarios utilizing Inter frames redundancy removal, Nose Suppression Filter cannot be directly used due to a negative impact caused by over filtering that in many cases is more significant than the filter benefits. In order to eliminate such drawback and amplify NSF benefits for AI configuration the application map approach was introduced as an additional step for general NSF design. Together with the application map method, NSF demonstrates clear beneficial effect on top of VVC video codec for all CTC configurations. Namely, comparing simulation results for All Intra, where NSF improves quality even without the application map, it can be concluded that application map makes NSF performance roughly 2 times more efficient. For the other 3 Inter configuration the application map method allows to completely eliminate over filtering for all test classes. As a summarized result of NSF on top of VTM it demonstrated from 0.61% to 2.61% of luma bd-rate saving on average from 4 CTC test scenarios. In addition, the application map usage allows to significantly reduce decoder complexity especially such complexity reduction is observed on RA and both LD configurations.
Thus, the application map approach is an essential part of Noise Suppression Filter,
which makes this filter practically applicable for typical video compression scenarios.
In this paper NSF was considered as a normative part of VVC codec because filtered
frame was used for further temporal prediction. However, NSF can be also used as a
pure post filter, where filtered results will be only applied for output and unfiltered
frame will be used for further temporal prediction. It is expected that compression
efficiency of such approach will be less than one presented in this paper, on the other hand
it will not require a standardization and such design can be used in wider video systems
as a non-normative part. This exploration can become one of the further working
directions towards NSF investigations. It should be also noted that introducing application
map to the NSF method allows situation where some blocks of the current picture are
filtered by NSF but some other blocks within the same picture are directly kept from
the reconstruction process, which may theoretically lead visual blocking effects.
Although several examples of NSF subjective results given in [