Correction of Color Saturation for Tone Mapping Operator E.D. Birukov1, M.S. Kopylov1, A.A. Khlupina1 birukov@gin.keldysh.ru|kopylov@gin.keldysh.ru|nastyak@gin.keldysh.ru 1 Keldysh Institute of Applied Mathematics of RAS, Moscow, Russia This article reviews the problems of color saturation correction during dynamic range compression of images. Special attention is drawn to emulation of human vision and photo cameras effects during processing of the images which have very bright areas (with brightness more than an order of magnitude greater than average brightness of the rest of the image). A method of desaturation of the brightest parts of images is suggested. This method provides lowering of the color saturation of each pixel depending on the pixel brightness. The improved method lowers saturation only of the brightest parts of an image. The size of the desaturated part of the image should not exceed 25% of the size of an entire image. Keywords: color saturation, tone mapping operator, high dynamic range images, desaturation. 1. Introduction Basic formula of compression of separate color channels is as follows: Dynamic range of an image is ratio between the pixel 𝐢 πΆπ‘œπ‘’π‘‘ = 𝑖𝑛 πΏπ‘œπ‘’π‘‘ (1) brightness with maximal and minimal value. More broadly, 𝐿𝑖𝑛 dynamic range is ratio between the maximal and minimal where Cin - is one of the color channels of the original image (red, brightness which can be obtained from a photo or video camera, green or blue), simulated with a computer graphics system or displayed at some Cout - is one of the color channels of the transformed image, device. The problem of range compression of images is very Lin - is the pixel brightness of the original image, relevant in present time. Mostly this task is required during Lout - is the pixel brightness of the transformed image conversion of physical values such as luminance and illuminance obtained using some tone mapping (brightness to pixel brightness at the image display devices such as computer compression) operator. monitor or projector. Range of luminance and illuminance values This formula doesn’t correct color saturation, so output image which were obtained during a light propagation simulations in often becomes oversaturated. There are different methods of physically accurate computer graphics systems may be several correcting color saturation. Most of them allow tuning only the orders of magnitude higher than dynamic range of mass-market entire color/saturation balance over the entire dynamic range, computer monitors. Even many photo cameras already have high without taking into account human vision specifics described dynamic range [1]. above. A problem of color saturation often appears during solving this Tumblin and Turk [9] proposed the following formula of task. For example, if only brightness is compressed with keeping dynamic range compression for color correction: 𝐢 𝑠 original color values, then image will be significantly πΆπ‘œπ‘’π‘‘ = ( 𝑖𝑛 ) πΏπ‘œπ‘’π‘‘ (2) 𝐿𝑖𝑛 oversaturated. Otherwise, if each color channel (in RGB model) is where s - parameter in this formula controls color saturation. being compressed separately then most often it will seem that The main problem of this formula is that it may significantly saturation of the image is very low, so that the entire image modify image brightness in case if s is not equal to 1 and colors becomes nearer to grayscale. During setting color balance and are different from grayscale. Taking this into account Mantiuk et saturation of images it is also reasonable to take specifics of al [4] proposed their own formula for compressing separate color human vision into account [6]. For example, for the darker parts channels: of an image color saturation should be decreased to some extent because human color perception works worse in the darkness. But 𝐢𝑖𝑛 πΆπ‘œπ‘’π‘‘ = (( βˆ’ 1) 𝑠 + 1) πΏπ‘œπ‘’π‘‘. for the brightest highlights saturation also should be decreased 𝐿𝑖𝑛 because when eyes are adapted to low brightness then small bright This formula keeps the brightness and tunes only the linear part of the observed space will seem just white, without interpolation between chromatic and the corresponding distinguishing of color hues. Typical example of such saturation achromatic colors. But its side effect is shift of color hues, lowering is a dark room with a small window leading out, while especially for red and blue channels. So Mantiuk et al also outside there is a sunny weather with bright sky. suggested an alternative variant: usage of the single formula of dynamic range compression which is applied to all color channels. 2. Overview of the existing methods of color It should be equivalent to the formula (2) in case if color correction correction coefficient s is equal to dynamic range compression coefficient c and dynamic range compression function looks as In this article we describe compression of separate color follows: channels for RGB color model because it is the most popular πΏπ‘œπ‘’π‘‘ = (𝐿𝑖𝑛 𝑏)𝑐 color model in computer graphics. It should be noticed that pixel where b - is the brightness tuning coefficient which normalizes brightness value which is used for calculations should be obtained maximal value of the output brightness so that it will be equal to either from some external source (for example, during the lighting 1. simulation in the realistic computer graphics system), or using If the tone-curve is an arbitrary function, applying the same some conversion from RGB color model, for example, with tone-curve to all color channels is not equivalent to (2) under s = SRGB conversion: c, but the results are very close. In case of local tone mapping Y = 0.2126R + 0.7152G + 0.0722B operators, the three color channels usually cannot be modified Copyright Β© 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). simultaneously. If c is less than s, it will lead to decreasing of having very bright highlights to look more realistic after saturation. compressing dynamic range. With the help of this method one can simulate the effects described earlier for artificially rendered 3. Usage of saturation lowering in case of high images. The method suggests lowering of color saturation with brightness increasing of brightness. The first implementation of the method had the following Methods described above are used for tuning color balance and algorithm: saturation over the entire dynamic range of an image. Such tuning 1. Find the average value of the color channels brightness in the is indented for preserving color distinguishing of an original given pixel. We’ll call it 𝐢𝑖𝑛 ; image after dynamic range compression. But under certain 2. Find the difference between the color channel value and the circumstances also additional lowering of color saturation is average value; needed for the brightest parts of an image (in particular, for 3. Divide the pixel brightness by the maximal brightness value highlights). Many images which initially had a high dynamic of the image. It will be the individual coefficient of range look unnaturally after compression because they contain saturation lowering for this particular pixel; oversaturated colors in the brightest zones which should seem just 4. Finally, difference between the brightness value of each white. First of all it is caused by specifics of human vision. color channel of the pixel and the average value should be Besides, such β€œburnout” effect for brightest parts of an image multiplied by the coefficient described above. Then it should occurs in the photographic equipment, including the modern be subtracted from the color channel value itself. Due to this, digital devices. Usually it can appear if exposure time during values of the color channels will be nearer to each other and taking photo is too great. But it should be noticed that in some so color saturation of this pixel will be decreased. cases it is necessary to set a long enough exposure time So, formula (1) is being transformed to the following new intentionally for providing sufficient brightness of the key parts of formula: an image. And for all that some bright parts of the image inevitably will be overlighted. There are methods which decrease 𝐿𝑖𝑛 πΏπ‘œπ‘’π‘‘ maximal brightness in such cases providing maximal distinction πΆπ‘œπ‘’π‘‘ = (𝐢𝑖𝑛 βˆ’ ((𝐢𝑖𝑛 βˆ’ 𝐢𝑖𝑛 ) βˆ— )) βˆ— 𝐿𝑖𝑛 π‘šπ‘Žπ‘₯ 𝐿𝑖𝑛 of all details, for example, [3, 7]. But contrary actions are often required for obtaining realistic looking simulated images. It An example of the results of the tone mapping operator proposed causes simulated images to look as photo images of the real world in [1] and the same algorithm with saturation lowering proposed taken with a usual photo camera. here is shown on fig. 1. In Keldysh Institute of Applied Mathematics we have developed the following original method which makes images Initial tone mapping algorithm The same with additional saturation lowering Fig. 1. Saturation lowering without additional parameters. Attention should be drawn to the lamps and table surfaces. equal high enough brightness over most part of the image it They are shown at figs. 2 and 3. One can see that at the original should be decreased. image they are colored with yellow while after color correction they become almost white. At the same time other parts of the image have near the same color saturation. Fig. 4. Above is the original image, below is desaturated Fig. 2. Fragment of the image (table top). Above is the image. original tone mapping, below the desaturation is added. But forcing user to tune the desaturation coefficient value manually is not a user-friendly solution. Some universal algorithm is required. It should allow automatic tuning of desaturation depending on the specifics of each particular image. 4. Methods of automatic parameters choosing Desaturation should be applied only to the brightest parts of an image with brightness several times greater than the rest of it. So, the most optimal method which showed good results in most cases was setting some threshold value. Desaturation will not be applied at all for values below this value. Individual desaturation coefficient in this case increases linearly from 0 to 1 for Fig. 3. Fragment of the image (lamp). Above is the original brightness value from threshold to maximum correspondingly. tone mapping, below the desaturation is added. Empirically it was established that half of the maximal brightness It should be noticed that desaturation itself in this algorithm is value of the image as the threshold value will provide acceptable done in RGB color model by adding (subtracting) some value to result for most images. This is because dynamic range is usually each color channel in order to make the value of this channel so great that most part of the image has brightness several times nearer to the average one. In our opinion it is much easier than lower than the maximal one. preliminary conversion of the image to a color model with the But setting threshold value by the statistical parameters of the special saturation parameter. For example, IPT color model [2] is image revealed best results, including cases where dynamic range used in the method of Pouli el al [5]. Saturation parameter of the of the image doesn’t require desaturation of any fragments. This transformed image (which, in its turn is calculated for HCL color method is based on the preposition that threshold value should be model) in this method is multiplied by relation of the original significantly higher than brightness of the most part of the image. image saturation to the compressed one, and the saturation is Otherwise the desaturation just makes no sense because such being calculated in the IPT color model. images can be found neither while observing real world objects Unfortunately, using the same formula for desaturation in all with human eyes, nor at the photos with a little bit correct cases makes no sense. Under these circumstances images which exposure parameters. For example, some kind of average or have many pixels with high brightness will have inadequately low matrix metering (in several points) is generally used for exposure color saturation over most of the image. Example of such image is setting in modern digital photo cameras instead of a single shown at fig. 4. At this image the apple is unnecessary lighting measurement in some point [8]. It allows choosing such desaturated. exposure parameters that most part of the image will have normal Taking this into account, there should be found some ways of saturation. After several experiments there was elaborated the tuning saturation lowering. The easiest way of such tuning is following method for calculating threshold value. Such brightness adding the desaturation coefficient which should be set manually value should be used that certain number of image pixels would by a user. This coefficient may have values from 0 to 1. Individual have brightness below this value. In our algorithm the pixel desaturation coefficient of each particular pixel (which was quantity had been defined as 75% from the total. Then this value obtained at the step 3 of the described algorithm) should be is doubled and the result is used as the threshold. If the threshold multiplied by this additional coefficient. For images with very value is higher than the maximum then desaturation is not applied bright highlights and the low-brightness background the value of at all. this coefficient should be increased. For images which have near After applying this method to the first image one can see 6. Acknoledgements almost no difference from the initial desaturation method. Only the table top became a bit more saturated at the edge which is far This work was supported by RFBR, grants No 17-01-00363 from the lamp. So is looks even more realistic than after applying and 19-01-00435. the original algorithm, as it is seen at fig. 5: 7. References [1] Barladian, B.K., Voloboi, A.G., Galaktionov, V.A. and Kopylov, E.A., 2004. An effective tone mapping operator for high dynamic range images. Programming and Computer Software, 30(5), pp.266-272. [2] F. Ebner and M. D. Fairchild, 1998. Development and testing ofa color space (IPT) with improved hue uniformity. InSixthColor Imaging Conference: Color Science, Systems and Ap-plications, pages 8–13. Fig. 5. Desaturation with threshold based on statistical [3] D. Guo, Y. Cheng, S. Zhuo, T. Sim, 2010. Correcting parameters of the image. over-exposure in photographs. Correcting over-exposure In fig. 6 there are shown the results of the applying the in photographs. IEEE Computer Society, 2010 algorithm to another image. The tray and the teapot have [4] Mantiuk, R., Mantiuk, R., Tomaszewska, A. and desaturated highlights. Heidrich, W., 2009, April. Color correction for tone mapping. In Computer Graphics Forum (Vol. 28, No. 2, pp. 193-202). Oxford, UK: Blackwell Publishing Ltd. [5] Tania Pouli, Alessandro Artusi, Francesco Banterle, Ahmet Oguz Akyuz, Hans-Peter Seidel and Erik Reinhard, 2013. Color Correction for Tone Reproduction. Color and Imaging Conference, 21st Color and Imaging Conference Final Program and Proceedings, pp. 215-220(6) [6] Reinhard, E., Stark, M., Shirley, P. and Ferwerda, J., 2002, July. Photographic tone reproduction for digital images. In ACM transactions on graphics (TOG) (Vol. 21, No. 3, pp. 267-276). ACM. [7] Alessandro Rizzi, Carlo Gatta, Daniele Marini, 2003. A new algorithm for unsupervised global and local color correction. Pattern Recognition Letters, Volume 24, Issue 11, July 2003, Pages 1663-1677 [8] Sean T. McHugh, 2018. Understanding Photography. Cambridge University press. [9] Tumblin, J., 2000. Three methods of detail-preserving contrast reduction for displayed images (Doctoral dissertation, PhD thesis, Georgia Institute of Technology, 1999. http://www. cc. gatech. edu/gvu/people/jack. tumblin). Fig. 6. Desaturated highlights on the tray and teapot. As for images where original algorithm caused superfluous desaturation, like the apple from fig. 4, most often threshold becomes greater than the maximal brightness value, so desaturation is not applied at all. During testing there was added one more improvement of the method. Only those pixels which have nonzero brightness are taken into account during calculating of the threshold value by the pixel. This is due to the fact that absolutely zero brightness at the images usually means that the image represent itself simulated rendering results and the background is black. So, this background contains no significant information and in fact is not related to the image itself. 5. Conclusion Newly developed algorithm allows processing of high dynamic range images in so way that color saturation of the brightest highlights will be lowered as it is present in real life with taking into account specifics of human vision and photo technics. At the same time most images where such highlights are absent also looks realistic after processing with this algorithm.