Quality assessment is essential to creating and comparing video compression algorithms. Despite the development of many new quality-assessment methods, well-known and generally accepted codecs comparisons mainly employ classical methods such as PSNR, SSIM, and VMAF. These methods have diferent variations: temporal pooling techniques, color-component summations and versions. In this paper, we present comparison results for generally accepted video-quality metrics to determine which ones are most relevant to video codecs comparisons. For evaluation we used videos compressed by codecs of diferent standards at three bitrates, and subjective scores were collected for these videos. Evaluation dataset consists of 789 encoded streams and 320294 subjective scores. VMAF calculated for all Y, U, V color spaced showed the best correlation with subjective quality, and we also showed that the usage of smaller weighting coeficients for U and V components leads to a better correlation with subjective quality.
A Cisco forecast [
All of the above methods are common ways to increase quality control; for example, owing
to its high computation speed, PSNR serves in early development stages, in which thousands of
configurations require testing. Intermediate stages employ SSIM and VMAF, since they take
longer to compute. Generally accepted comparisons typically use all these metrics—examples
include the work of Jan Ozer [
Many implementations of modern compression standards have special modes to increase
their output’s score on popular metrics. For example, x264 and x265 have configuration modes
for PSNR and SSIM. The libaom encoder has a VMAF tuning mode. That tuning information
is visible, but many commercial solutions contain hidden settings to increase their scores on
generally accepted metrics, potentially reducing visual quality. A subjective comparison at
Moscow State University showed that the “–tune ssim” setting improves visual quality in
addition to SSIM scores. But the video-preprocessing techniques in libaom’s “–tune vmaf”
setting can substantially reduce visual quality, as [
For this paper, we analyzed the correspondence to visual quality of various PSNR, SSIM, and
VMAF configurations. We collected a special data set for our analysis, as open data sets with
visual-quality ratings contain distortions from just one or two codecs (usually H.264 and H.265).
Therefore, our study paid special attention to assembling a set of videos encoded by various
implementations of multiple standards. In this way, we obtained videos with representative
encoding distortions. This task was under way from 2018 to 2021 through the annual subjective
video-codec comparisons of Moscow State University [
Multiple studies have compared video-quality metrics. At the same time, each metric’s
correlation with subjective estimates can vary greatly depending on the data set and the distortion
type. For example, the goal of [
Thus, finding a suitable quality metric that maximally correlates with visual assessment is essential. The existing works on the correlation of video-quality metrics examine only a few video codecs (mainly open implementations of H.264 and H.265). Therefore, evaluating a metric’s relevance to a wide variety of videos containing many types of distortions remains important. Numerous data sets are useful for comparing algorithm performance, the most popular being Live-VQA [13] and Live-VQC [14]. Their biggest drawback when attempting to identify the best compression-quality metrics is that they contain few compression artifacts. Also, many new metrics that employ machine learning have been trained on these data sets. These factors call into question the applicability of such data sets to objective metric comparisons and make beneficial a study that uses an independent data set with a representative spectrum of compression artifacts.
In addition to the variety of metrics, there are several ways to calculate them: using RGB or YUV color models, using only the luminance component, or using all color panes with diferent summation coeficients. The YUV color space is a common choice for image- and video-quality measurement [15]. It enables quality measurement using only the Y (luminance) space, which provides more visual information about an image. The U and V spaces have less impact; for some metrics they improve overall correlation but require additional computation time. In this paper we show the eficiency of summing diferent Y, U, and V components for diferent metrics.
To analyze the relevance of quality metrics to codec comparisons, we collected a special data
set that includes video sequences and subjective scores. The subjective comparisons were
performed independently using diferent videos and a diferent encoders sets. Each one assessed
FullHD videos with diferent spatial and temporal complexities, which afect compression
quality and performance. We made our selection from a pool of more than 18,000 open-source
clips with high bitrate after analyzing more than five million source videos from the Vimeo
website. Our choice employed clustering in terms of space-time complexity. A description of the
video-selection method appears in [16]. The resulting video data sets for subjective assessments
are called CC-2018, CC-2019, CC-2020, and UGC-2020; the numbers indicate the year each one
was created. Each data set is available by request from its associated codec-comparison project
page [
We obtained a representative set of coding artifacts using diferent video-codec standards: 11 Data sets for analyzing video-quality metrics. CC-2018, CC-2019, CC-2020 and UGC-2020 are video sets used for subjective evaluation in MSU Codecs Comparisons in 2018, 2019 and 2020.
dataset CC-2018 CC-2019 CC-2020 UGC-2020
test videos
10 11 11 7 39 5 5 8 10 28 150 165 264 210 789 22542 25784 236736 35232 320294 H.265/HEVC encoders, five AV1 encoders, two H.264/AVC encoders, and four encoders based on other standards (VVC, VP9, SIF, and xvc). We compressed each video at three target bitrates: 1,000 Kbps, 2,000 Kbps, and 4,000 Kbps. The choice of this range simplifies the subjectivecomparison procedure, since the video quality is more dificult to distinguish visually at higher bitrates.
The subjective assessment involved pairwise comparisons using the Subjectify.us platform, which employs a Bradley-Terry model to transform the results of pairwise voting into a score for each video. A detailed description of the method appears on the website. To increase the relevance of the results, each pair of videos received at least 10 responses from participants. The number of subjective ratings per pair depended on the confidence intervals: more responses were received for complex videos as well as videos that were hard to be compared.
PSNRavg. (, ˆ ) = 10 log10 1 ∑︀
2 =1 ︁( (), ˆ () ︁) • PSNR average log – when aggregating frame-by-frame scores for the entire video, we calculate the PSNR for each frame and then the arithmetic mean for all frames.
PSNR. log(, ˆ ) = =1 1 ∑︁ 10 log10
2 ︁( (), ˆ () ︁) In addition, we considered diferent versions of the VMAF metric: • VMAF 0.6.1, VMAF 0.6.2, and VMAF 0.6.3. • VMAF 0.6.1 NEG (“no enhancement gain”), which is less prone to artificial increases through preprocessing. • The “Phone” variant of the above four, as well as a variant that handles 4K video using the VMAF 0.6.1 model.
For each reference method, our analysis considered the following options for calculating the color components: we evaluated Y metrics only for the luminance channel, we evaluated YUV4:1:1 metrics independently for the three components and averaged the result as 4*Y+U+V, and we applied similar methods for YUV6:1:1, YUV8:1:1, YUV10:1:1, and the rarely used YUV1:1:1 as well as YUV2:1:1. Our calculations of the metric values employed the MSU VQMT version 12.1 [17].
Because we conducted a separate subjective comparison of the videos for each year, we had to obtain for every metric an overall correlation across the entire data set. For each metric, our approach applied the Fisher z-transform to all correlations coeficients calculated on individual video sequences and used the resulting values to calculate the weighted mean and confidence interval, with the weights proportional to the number of distortions. We determined the final correlations using the inverse transformation [18].
Fig. 1 and Fig. 2 show the Spearman and Pearson correlations. The colors indicate diferent groups of metrics. The graphs reveal that VMAF variations have the highest correlation with subjective-quality scores, and the diferences between them are insignificant. Next are VMAF NEG and MS-SSIM, also with nearly equal correlation. PSNR variants have the lowest correlation among full-reference metrics.
In all cases, YUV1:1:1 and YUV2:1:1 show worse results than the other channel-summing methods. Diferent versions of some metrics (for example, MS-SSIM) exhibit nearly identical results, but no one YUV summation is best for all metrics.
Table 2 shows the best options for summing the components of diferent metrics. Some options have almost identical correlation; the table separates them by commas. For VMAF, the best YUV-summation coeficients are 6:1:1, 8:1:1, and 10:1:1 for PSNR, and 6:1:1 and 4:1:1 for SSIM. For MS-SSIM, nearly all methods have the same accuracy.
Fig. 3a and Fig. 3b show overall ranking of metrics with best summations only.
VMAF NEG showed a lesser correlation with visual quality than VMAF did (Fig. 4). Its developer, Netflix, recommends using VMAF NEG when comparing video codecs, as it helps prevent cheating and artificial metric increases through video preprocessing. When comparing diferent versions of the same algorithm, however, as well as when algorithms are incapable of artificially increasing VMAF, the classic VMAF yields a more accurate visual-quality estimate. (c) VMAF v0.6.2 (d) VMAF v0.6.3 (e) SSIM and MS-SSIM
MS-SSIM correlates better with visual quality compared with classic SSIM. Diferent PSNR variants correlate worse with visual quality than SSIM and MS-SSIM do, but PSNR average MSE is slightly better than PSNR average log (Fig. 2e and Fig. 2b). (c) VMAF v0.6.2 (d) VMAF v0.6.3 (e) SSIM and MS-SSIM
The results for AV1-encoded streams difer from those for streams encoded using other standards. Fig. 4 shows the diference in metric correlations between videos encoded using AV1 and those encoded using H.265. The correlation between PSNR and visual score is much less than that No significant diference between summing methods
Y (only luma component), 10:1:1, 6:1:1, 8:1:1 (a) Pearson correlation (b) Spearman correlation for other standards. This result may be a consequence of recent encoders employing neural networks, which restore object boundaries and thus cause pixel-by-pixel similarity violations—a characteristic that PSNR penalizes. Multiscale metrics (VMAF and MS-SSIM) yielded the best results for such videos. The most stable values are for metrics that evaluate H.264/AVC-encoded streams. The correlations for all metrics, when applied to these videos, exceed 0.94.
Fig. 6 shows the space-time complexity distribution and the correlation between PSNR and VMAF for diferent videos. For some videos that exhibit low spatial and temporal complexity, PSNR and SSIM difer greatly from VMAF. This diference may owe to other factors, so tracking several metrics when measuring the performance on individual videos is better.
This article describes the results of comparing diferent versions of popular objective methods with subjective quality rankings. Our analysis used numerous compression algorithms and revealed the best variants for video-codec comparisons. We used a large-scale data set containing 789 encoded videos distorted by 39 versions of H.264, H.265, AV1, VP9, and other codecs, as well (a) PSNR and SSIM (b) VMAF and VMAF NEG as three bitrates. We conducted a visual analysis of the resulting sequences using Subjectify.us; several hundred individuals participated. We analyzed many metric versions and modifications (diferent methods of averaging the values between frames, accounting for color and brightness, and so on).
Analysis of the results led us to the following conclusions:
1. VMAF and its variants exhibited higher correlation with visual quality than other metrics
did. Recent research, however, showed that if videos are specially prepared (preprocessed)
for this metric [
• For PSNR (average log), 4:1:1. 3. MS-SSIM showed better results than SSIM. 4. Modifications of PSNR (average log and average MSE) yielded no significant diferences. 5. When analyzing AV1 codecs or AV1-encoded videos, no PSNR modifications are justified. 6. Some metrics that have similar average correlations may yield lower-quality results for some videos. Therefore, when comparing the quality of video-encoding or videoprocessing algorithms, it makes sense to employ several metrics while taking into account their potentially sharp fluctuations for individual outputs.
The above results prove that comparing video-coding algorithms using objective quality metrics is a complex process with many issues and peculiarities. Ignoring these characteristics may lead to results that are unwarranted or, sometimes, that contradict the results of subjective visual analysis. For that reason, codec-industry professionals recognize only well-known codec comparisons, which should be carried out in laboratories by experienced teams in collaboration with codec developers and other industry experts. These comparisons employ a correct methodological basis and empirical confirmation of their various facets (e.g., choice of objective quality metrics, modifications to those metrics, and methods of averaging color components). Otherwise, given a certain selection of video data, metrics, and metric parameters, a careless or unsuspecting researcher can easily obtain the desired comparison result rather than an objective one.
This work is partially supported by the Russian Foundation for Basic Research under Grant 19-01-00785a. Anastasia Antsiferova was supported by the Fellowship from Non-commercial Foundation for the Advancement of Science and Education INTELLECT. Special thanks go to the Graphics and Media Lab at Moscow State University for providing valuable advice and support for our projects. //www.sciencedirect.com/science/article/pii/S0923596503000766. doi:https://doi.org/ 10.1016/S0923-5965(03)00076-6. [11] M. Vranješ, S. Rimac-Drlje, K. Grgić, Review of objective video quality metrics and performance comparison using diferent databases, Signal Processing: Image Communication 28 (2013) 1–19. URL: https://www.sciencedirect.com/science/article/pii/S0923596512001919. doi:https://doi.org/10.1016/j.image.2012.10.003. [12] D. Nandakumar, Y. Wu, H. Wei, A. Ten-Ami, On the accuracy of video quality measurement techniques, in: 2019 IEEE 21st International Workshop on Multimedia Signal Processing (MMSP), 2019, pp. 1–6. doi:10.1109/MMSP.2019.8901796. [13] K. Seshadrinathan, R. Soundararajan, A. C. Bovik, L. K. Cormack, Study of subjective and objective quality assessment of video, IEEE Transactions on Image Processing 19 (2010) 1427–1441. doi:10.1109/TIP.2010.2042111. [14] Z. Sinno, A. C. Bovik, Large-scale study of perceptual video quality, IEEE Transactions on
Image Processing 28 (2019) 612–627. doi:10.1109/TIP.2018.2869673. [15] M. Podpora, G. P. Korbas, A. Kawala-Janik, Yuv vs rgb-choosing a color space for humanmachine interaction., in: FedCSIS (Position Papers), 2014, pp. 29–34. [16] A. V. Zvezdakova, D. L. Kulikov, S. V. Zvezdakov, D. S. Vatolin, Bsq-rate: a new approach for video-codec performance comparison and drawbacks of current solutions, Programming and computer software 46 (2020) 183–194. [17] Msu quality measurement tool, n.d. URL: http://compression.ru/video/quality_measure/ vqmt_download.html. [18] D. M. Corey, W. P. Dunlap, M. J. Burke, Averaging correlations: Expected values and bias in combined pearson rs and fisher’s z transformations, The Journal of General Psychology 125 (1998) 245–261. URL: https://doi.org/10.1080/00221309809595548. doi:10.1080/ 00221309809595548. arXiv:https://doi.org/10.1080/00221309809595548. [19] A. Zvezdakova, S. Zvezdakov, D. Kulikov, D. Vatolin, Hacking vmaf with video color and contrast distortion, in: CEUR Workshop Proceedings, 2019, pp. 53–57.