Video quality measurement takes an important role in many applications. Full-reference quality metrics which are usually used in video codecs comparisons are expected to reflect any changes in videos. In this article, we consider different color corrections of compressed videos which increase the values of full-reference metric VMAF and almost don't decrease other widely-used metric SSIM. The proposed video contrast enhancement approach shows the metric in-applicability in some cases for video codecs comparisons, as it may be used for cheating in the comparisons via tuning to improve this metric values.
At the moment, video content takes a significant
part of worldwide network traffic and its share is
expected to grow up to 71% by 2021 [
Methods for evaluating encoded videos quality
can be divided into 3 categories [
One of the widely-used full-reference metrics which is
gaining popularity in the area of video quality
assessment is Video Multimethod Fusion Approach
(VMAF)[
Fig. 1. The scheme of VMAF algorithm.
Despite increasing attention to this metric, many
video quality analysis projects, such as Moscow State
University’s (MSU) Annual Video Codec Comparison
[
The main goal of our investigation was to prove the no-universality of the current version of VMAF algorithm. In this paper, we describe video color and contrast transformations which increase VMAFscore with keeping SSIM score the same or better. The possibility to improve full-reference metric score after adding any transformations to distorted im-age means that the metric can be cheated in some cases. Such transformations may allow competitors, for example, to cheat in video codecs comparisons, if they “tune” their codecs for increasing VMAF qual-ity scores. Types of video distortions that we were looking for change the visual quality of the video, which should lead to a decrease in the value of any full-reference metric. The fact that they lead to an increase in the value of VMAF, is a significant obstacle to using VMAF for all types of videos as the main quality indicator and proves the need of modification of the original VMAF algorithm.
During testing of VMAF algorithm forvideo codecs comparisons, we noticed that it reacts on contrast distortions, so we chose colorand contrast adjustments as basic types of the searched video transformations. Two famous and common approaches for color adjustments were tested to find the beststrat-egy for VMAF scores increasing. Two cases of transformations application to the video were tested: applying transformation before and after video encoding. In general, there was no significant difference between these options, because the compression step can be omitted for increasing VMAF with color enhance-ment. Therefore, further we will describe only the first case with adjustment before compression, and we leave the compression step because in our work VMAF tuning is considered in case of video-codec compar-isons.
We chose 4 videos which represent different
spatial and temporal complexity [
Three versions of VMAF were tested: 0.6.1, 0.6.2,
0.6.3. The implementations of all three metric
versions from MSU Video Quality Measurement Tool [
For color and brightness adjustment, two known
and widespread image processing algorithms were
chosen: unsharp mask and histogram equalization. We
used the implementations of these algorithms which
are available in open-source scikit-image [13] library.
In this library, unsharp mask has two parameters
which influence image levels: radius (the radius of
Gaussian blur) and amount (how much contrast was
added at the edges). For histogram equalization, a
parameter of clipping limit was analyzed. In order to
find optimal configurations of equalization parame-ters, a
multi-objective optimization algorithm NSGA-II [
SSIM and VMAF scores were calculated for each video processed with the considered color enhancement algorithms with diferent parameters. As it was mentioned before, after color correction the videos were compressed with medium preset of x264 encoder on 3 Mbps. Then, the diference between metric scores of processed videos and original video were calculated to compare, how color corrections influenced quality scores. Fig. 2 shows this diference for SSIM metric of Bay time-lapse video sequence for diferent parameter values of unsharp mask algorithm. The similarity scores for VMAF quality metric are presented in Fig. 3.
Fig. 2. SSIM scores for Fig. 3. VMAF scores for different parameters of different parameters of unsharp mask on Bay unsharp mask on Bay time-lapse video sequence. time-lapse video sequence.
On these plots, higher values mean that the objective quality of the color-adjusted video was better according to the metric. VMAF shows better scores for high radius and a medium amount of unsharp mask, and SSIM becomes worse for high radius and high amount. The optimal values of the algorithm parameters can be estimated on the difference in these plots. For another color adjustment algorithm (histogram equalization), one parameter was optimized and the results are presented on Fig. 4 together with the results of unsharp mask.
74 )72 3 . 6 . 0 (F70 A M V -Y68 66 0.82
Histogram equalization Unsharp mask Without correction 0.83 0.84 0.85 0.86
Y-SSIM 0.87 0.88 0.89
Fig. 4. Comparison of VMAF and SSIM scores for different configurations of unsharp mask and histogram equalization on Bay time-lapse video sequence. The results in the second quadrant, where SSIM values weren't changed and VMAF values increased, are interesting for us.
According to these results, for some configurations of histogram equalization VMAF become significantly better (from 68 to 74) and SSIM doesn’t change a lot (decrease from 0.88 to 0.86). The results slightly dif-fer for other videos. On Crowd run video sequence, VMAF was not increased by unsharp mask (Fig. 5a) and was increased a little by histogram equalization. For Red kayak and Speed bag videos, unsharp mask could significantly increase VMAF and justslightly decrease SSIM (Fig. 5b and Fig. 5c)
The following examples of frames from the testing videos demonstrate colorcorrections which increased VMAF and almost did not influence the values of SSIM. Unsharp maswkith radius = 2.843 and amount = 0.179 increased VMAF without significant decrease of SSIM for Bay time-lapse (Fig. 6a and Fig. 6b). The images before and after masking look equivalent (a comparison in a checkerboard view is in Fig. 7) and have similar SSIM score, while VMAF score is better after thetransformation. 54 )52 3 . 6 . 0 (F50 A M V -Y48 46 44 )43 3 .642 . 0 ( F41 A M -V40 Y 39 38 99 )398 . 6 . 0 (F97 A M V -96 Y 95
Histogram equalization Unsharp mask
Without correction
For Crowd run sequence, histogram equalization with kernelsize = 8 and cliplimit = 0.00419 also increased VMAF (Fig. 8a and Fig. 8b). The video is more contrasted, so the decrease in SSIM was more significant. However, tho images also look similar (Fig. 9) and have similar SSIM score, while VMAF showed better score after contrast transformation.
Red kayak looked better according to VMAF after unsharp mask with radius = 9.436, amount = 0:045.
For Speed bag, the following parameters of unsharp mask allowed to increase VMAF greatly without influencing SSIM: radius = 9.429, amount = 0.114. Fig. 8. Frame 1 from Crowd run video sequence and its histogram with and without color correction. Two images and their histograms look almost similar.
Fig. 9. Checkerboard comparison of frame 1 from Crowd run video sequence before and after distortions.
Video quality reference metrics are used to show the difference between original and distorted streams and are expected to take worse values when any transformations were applied to the originalvideo. However, sometimes it is possible to deceive objective metrics. In our article, we described the way to increase the values of popular full-reference metric VMAF. If the video is not contrasted, VMAF can be increased by color adjustments without influencing SSIM. In another case, contrasted video can also be tuned for VMAF but with little SSIMworsening.
Although VMAF has become popular and important, particularly for video codec developers and customers, there are still a number of issues in its application. This is why SSIM is used in many competitions, as well as in MSU Video-Codec Comparisons, as a main objective quality metric.
We wanted to pay attention to this problem and hope to see the progress in thisare, which is likely to happen since the metric is beingactively developed. Our further research will involve a subjective comparison of the proposed color adjustments to the original videos and the development of novel approaches for metric tuning.
This work was partially supported by the Russian Foundation for Basic Research under Grant 19-01-00785a.
[12] H. R. Sheikh and A. C. Bovik, “Image informa-tion and visual quality,” in IEEE International Conference on Acoustics, Speech, and Signal Pro-cessing, 2004, . 3. .– iii-709. [13] S. van der Walt, J. L. Schonberger, J. NunezIglesias, F. Boulogne, J. D. Warner, N. Yager, E. Gouillart, T. Yu, and the scikit-image contributors. scikit-image: Image processing in Python. PeerJ 2:e453, 2014.