-

On Measures of Visual Contrast and Their Use in Image Processing

0 L2TI, Institut Galilee, Universite Sorbonne Paris Nord , France

Visual contrast is one of the most studied notions in the eld of Visual Neuroscience and Psychophysics. It is a measure associated with a psycho-physical sensation that is not easy to de ne in an objective and unique way. Indeed, the contrast as de ned through Weber's famous experiment is associated with the subjective notion of just noticeable di erence between a stimulus observed against an uniform background. In this paper a critical review of di erent representative measures of visual contrast and their uses in various applications is presented. This study provides also some insights on how to de ne the contrast and how to chose the most appropriate measure for developing contrast based methods for visual information processing and analysis. Perspectives and challenges that remain to be addressed are also discussed in light of new trends in visual information processing. Through this study, it becomes clear that the concept of contrast and its use are highly application-dependent and that there is no universal contrast measure. It is also shown that, given the large number of psycho-physical parameters involved, it is not easy to de ne a contrast measure that is easy to use in the various methods of image processing and analysis. Simplifying contrast models without neglecting the most fundamental aspects seems to be the most pragmatic and practicable solution.

Contrast Measures Just Noticeable Di erence (JND) Visual Contrast Image Processing and Analysis Perceptual Image compression Perceptual watermarking

Visual contrast is one of the most studied notions in the eld of Psychophysics and Visual Neuroscience [ 13, 20 ]. It is a measure associated with a psychophysical sensation that is not easy to de ne objectively and in a unique way. Indeed, contrast as de ned through Weber's famous experiment is associated with the subjective notion of Just Noticeable Di erence (JND) between a stimulus observed against an uniformed background. From the study conducted by Hecht [ 25 ] it seems that Bouguer (1760) was the rst to study the di erential sensibility of the Human Visual System (HVS). The idea of measuring the just noticeable increment needed to discern one stimulus from another was later studied by Weber and Fechner [ 13 ]. Indeed, the idea of analysing and measuring the minimum increment of intensity between two stimuli in order to discern them is at the basis of the very de nition of what is commonly known as the WeberFechner contrast [ 17 ]. In fact it was Fechner, a few years later, who formalized Weber's results in an experimental and theoretical framework. Weber-Fechner law has led to many experiments and sophisticated models for understanding the notion of perceptual contrast [ 25,33 ]. Since these pioneer works, many visual contrast measures have been proposed in the literature [ 16, 24, 31, 35, 39, 41, 42 ].

The taking into account of psycho-physical factors such as perceptual contrast in a visual information processing and transmission system is essentially linked to the fact that the observer is the key element in any chain of acquisition, processing and transmission of visual information. Indeed, the human observer is often the supreme judge in the evaluation of the di erent stages of the processing and transmission chain. It is therefore quite natural to think of developing methods inspired by the mechanisms of the human visual system to incorporate perceptual criteria that meet the requirements of the observer. It is worth noticing that among the perceptual aspects of the HVS, visual contrast is one of the widely investigated psycho-visual aspects in vision research [ 13, 24, 25, 48 ]. According to this principle and reasoning several methods of image processing and analysis based on contrast measure have been developed [ 3, 48 ]. Indeed, contrast plays a prominent role in important applications such as medical imaging [ 19 ], image quality enhancement (IQE) [ 6 ] and image quality assessment (IQA) [ 14, 15 ], image fusion [ 9, 44 ], and other applications such as image quantization and compression [ 12, 28 ]. In this article we will review some of these applications and discuss the most suitable contrast measures in each case. This is not an exhaustive study of all known methods, but we will limit ourselves to a few representative studies. The main objectives of this contribution are as follows: { to discuss the fundamental criteria and factors that de ne visual contrast and present a critical review of the most representative contrast measures and associated models, { provide a brief description and discussion on some applications in the eld of visual information processing based on perceptual contrast measure { provide some insights on how to use and chose the appropriate contrast measure in some selected visual information processing and analysis applications, The paper is organized as follows: rst some historical contrast models are discussed in Section 2 followed by a classi cation discussion of some representative contrast measures in Sections 3, 4, and 5. Section 6 is dedicated to some selected contrast based applications. Finally the paper ends by concluding remarks, challenges and some future directions of research in Section 7.

Perceptual Contrast: History and Basic Notions This section presents a brief historical review of the research on contrast measures and associated models developed by the scienti c research community in psycho-physics, optics, neuroscience and digital visual information processing. The notion of visual contrast has been introduced in a clear and well de ned theoretical and experimental framework for the rst time by Fechner [ 17 ]. Since this pioneering work based on Weber experiments and model, several studies have been carried out to enrich existing models with advances on both the theoretical and experimental levels [ 24, 25, 33, 35 ]. The introduction of the frequency sensitivity aspect of contrast has been established through psycho-physical experiments [ 13 ]. The directional selectivity of HVS has been clearly demonstrated by Hubel and Wiesel [ 26 ], the two Nobel Prize laureates. Other aspects related to the distance or viewing angle parameter have been introduced explicitly according to optical models [ 33 ] or implicitly through a multi-resolution representation of visual contrast [ 14, 44 ]. However, the colour aspect was neglected for a long time in the rst experiments. This is due to the fact that the notion of contrast is much more related to the detection of details and more particularly the contours of objects which is traditionally regarded as an achromatic process [ 13 ]. However, it has been shown that the colour aspect also plays a major role in contrast sensitivity [ 22 ]. Other spatio-temporal and 3D aspects could be incorporated in the de nition of the spatio-temporal contrast. It is worth noticing that to the best of the author's knowledge there is no measure of contrast in an analytical form that integrates all these psycho-physical and geometrical parameters. Rather simpli ed and mathematically tractable expressions are often used to de ne an objective measure of visual contrast. The essential criteria and HVS properties that could be taken into account in the visual contrast measure are given below.

{ sensitivity to the relative change of luminance { luminance adaptation phenomena { frequency selectivity { directional sensitivity { multiscale/multiresolution aspects { color aspects { viewing distance (or viewing angle) { temporal aspects (in the case of spatio-temporal visual signals) Note that it is not easy to de ne a contrast measure that integrates all these properties and criteria. Simpli cations are often used, keeping only a few properties and aspects to establish a measure of visual contrast. There are two ways to de ne contrast, depending on whether we associate a single value to the entire image or a value for each pixel or group of pixels. In the rst case it is global contrast, while in the second case it is local contrast. The contrast measure could be. computed in the spatial domain, frequency domain, and even multi-resolution or multi-scale representations.

It should be noted that given the di erent forms of representation of the visual signal, the large number of contrast measures proposed in the literature and the di erent contexts and elds of application, it is not easy to classify all the contrast measures. In the following we classify contrast measures into three categories that is psycho-physic and neuroscience based contrasts, local structure based contrasts and statistical information based contrasts. Here we limit ourselves to a few representative contrast measures from each category. 3

Psycho-physic and neuroscience based contrasts In this rst category are some representative contrast measures based on psychophysical experiments or models from theoretical and experimental studies in neuroscience. Figure 1 illustrates the simultaneous contrast phenomena. It also illustrates one of the most relevant parameters that should be taken into account in the contrast de nition, that is the in uence of the surround and background luminance in the visual perception of stimuli. This gure represents the foveal image model used in several psycho-physical experiments such as the Moon and Spencer experiment [ 33 ].

What should be observed in Figure 1 is the e ect of the background on the appearance of the small dark disc and the ring around it. Indeed, as can be seen, despite having the same luminance and the same gradient in the ve con gurations the disc and the ring appear di erent according to the position in the non-uniform background. It can be concluded from this example that the gradient alone cannot account for the visual appearance of the visual signals. The background against which the object is observed is important. This observation leads us to question the many contrast measures based solely on the luminance di erence between stimuli, i.e. gradient. This is one of the reasons why it is important to de ne contrast as a measure of relative variation in luminance, i.e. a relative ratio. This is the case with the contrast measure proposed by Weber and Fechnner [ 17 ] described below.

Weber-Fechner contrast

Weber-Fechner contrast is one of the simplest contrast measures [ 17 ]. However, it only applies to simple scenarios in which a uniform luminance background L contains an object with an incremental luminance L. The aim of WeberFechner (W-F) contrast is to determine the value of L, referred to as the JND, which makes the object (target) just visible. The W-F contrast measure is de ned by:

CW = L (1) One of the most important results of the experiments conducted by Weber and Fechner is that this ratio remains constant over a fairly wide range of luminance values. This value is called the Just Noticeable Contrast (JNC) and is of the order of .02. It should be noted that many sophisticated contrast measures, used in visual information processing and coding, are in one form or another based on the Weber-Fechner de nition [ 2, 14, 35, 42, 44, 50 ]. 3.2

Michelson contrast

The Michelson contrast was rst introduced in a purely physical context and concerns the measurement of the visibility of interference fringes produced by thin lms [ 32 ]. However, it has been widely used in psycho-physical experiments and more particularly the study of the frequency sensitivity of the HVS to perceptual contrast [ 13, 45 ]. The Michelson contrast is de ned as follows: CM =

Lmax Lmin ; Lmax + Lmin (2) where Lmin and Lmax correspond to the minimum and maximum luminance values in the optical image, respectively. Although this contrast has been used extensively by the scienti c community of vision research, it has several limitations. Indeed, its use in the case of natural images can lead to over- or underestimation of contrast and in particular in the case of images contaminated by impulse-type noise. It is also the case of images containing some singularities, or isolated points even if they are perceptually invisible. Moreover, it does not take into account the in uence of the background and in particular the lumninance adaptation phenomenon. It also does not consider the frequency aspect although the stimulus signal shape is designed from a sinsusoidal function. It is nevertheless surprising that such a simple contrast model has been widely used by the vision research community for almost a century.. 3.3

Moon-Spencer contrast

The main idea behind Moon and Spencer's [ 33 ] model is to apply Holladay's principle [ 33 ] that any non-uniform background may be replaced by another uniform luminance that produces the same perceptual e ect. This leads to the de nition Cmin = 8 CW < LS

CW : LS

A + p

LA A + q LLSA2 2 2 if LA

LS if LA < LS

: C = log

LO LB Where CW corresponds to the Weber-Fechner JNC contrast, and A is a constant determined experimentally from psycho-visual tests and Hecht's law [ 25 ] which is equal to 0:808. Note that this contrast is more interesting in the sense that it corresponds more or less to realistic con gurations. It has been successfully used and adapted to digital images in various applications [ 6, 27, 28, 34 ]. 3.4

Lillesaeter contrast

Noting the asymmetry in Weber's de nition of contrast, Lillesaeter proposed two measures of contrast [ 30 ]. In the rst one only luminance is taken into account while in the second de nition he proposes to include the geometry and shapes of the objects observed in the image. Indeed, it is observed that the negative and positive Weber-Fechner contrasts with the same absolute increment are not perceived equally. The Lillesaeter contrast is then de ned as of luminance adaptation that can be calculated according to a simpli ed model. Based on this principle, Moon and Spencer proposed a simple enough model to express the adaptive luminance, which is given below.

LA =

S LS +

BLB; Where LS and LB are the luminance of the surround (immediate neighborhood) and that of the background (or far surrounding), respectively. The two weighting parameters, S and B are set experimentally to the values :923 and :077, respectively. Moon and Spencer de ne the minimum perceptible contrast as: where LO and LB correspond to the average luminance of the object and the background, respectively. Note that when LO and LB are very close to each other the Lillesaeter contrast is equivalent to that of the Weber-Fechner contrast measure

C =

LB LB log LO log LB (if jCj << 1): The second de nition of Lillesaeter contrast incorporates the object contour geometry as perceived by human. The idea of taking into account the geometry of perceived objects is relevant but impractical in the evaluation of contrast in digital images. Indeed, it leads to the computation of curvilinear integral which requires the exact knowledge of the object contours. This leads inevitably to an ill-posed problem which is image segmentation. (3) (4) (5) (6)

DOG based contrast

Based on models describing the retinal ganglion cells and Lateral Geniculate Nucleus (LGN) responses [ 29 ] to visual stimuli on the other, Tadmor and Tolhurst [ 42 ] proposed three measures of local contrast. The principle is based on the use of linear ltering by two isotropic Gaussian Impulse Responses (IR) of di erent sized kernels. The bandpass behaviour of the HVS is then modelled through the Di erence Of Gaussian (DOG) model [ 29 ]. The two Gaussian IRs are associated with two regions, a center zone c and a surround zone s, to mimic the receptive elds of the ON and OFF cells [ 20, 29 ].

The two responses are given by: (7) (8) (9) (10) (11) and

I c (i; j) = (I

h c )(i; j); I s (i; j) = (I h s )(i; j) where I is the input image signal, h c and h c are the two Gaussian IR associated with the center and surround zones .

The convolutions are performed in the sliding windows c(i; j) and s(i; j) of odd size [ 3 c; +3 c]x[ 3 c; +3 c] and [ 3 s; +3 s]x[ 3 s; +3 s], respectively.

Three local contrasts are then de ned as follows:

C1DOG(i; j) = C2DOG(i; j) = C3DOG(i; j) =

I c (i; j)

I s (i; j)

I c (i; j) I c (i; j)

I s (i; j)

I s (i; j) I c (i; j)

I s (i; j) I c (i; j) + I s (i; j) ; ; ; The global contrast is derived by averaging the local contrasts. It can be noticed that these contrasts do not take into account the directional and frequency selectivity nor the colorfulness aspects. Note that this contrast expressed as a ratio between a di erential signal component and a low-pass component is somehow inspired by Weber-Fechner's simple model and corresponds well to the notion of visual contrast. 4

Local structure based contrast measures In this section, we introduce and discuss some representative contrast measures that could be used in digital image processing and analysis applications. These are essentially measures that explicitly or implicitly incorporate some features of the SVH. 4.1

Edginess based contrast measure

It is well established that one of the primitives of the image signal that is most related to contrast and therefore to the visibility of detail is contour information. Indeed, the most representative contours of the image signal correspond to the spatial frequencies where the CSF reaches its maximum values [ 13, 29, 45 ]. Inspired by the contrast de ned by Gordon and Rangayan in their contrast enhancement method [ 19 ], Beghdadi and Le Negrate introduced a new measure of contrast incorporating edginess information [ 4 ]. This local contrast measure is computed using a sliding window of odd size . For each Wij window, the mean edge gray-level at the center pixel (i; j), is computed as Note that this contrast measure does not take into account the frequency selectivity nor the directional selectivity aspects. Furthermore, when used in CE it may introduce halo e ects around the edges which is the result of overenhancement [ 5, 38 ] In (12), f (k; l) corresponds to the gray level at the pixel (k; l) and ( kl) is an increasing monotonic function of the gradient operator at (k; l). A simple function would be knl, with n > 0. The local contrast is expressed as: Toet was the rst to propose a contrast measures taking into account the multiresolution aspect for image fusion [ 44 ]. It is based on the Burt and Adelson pyramid decomposition scheme [ 10 ]. The contrast is expressed as Where the components gk(i; j) and gk 1(i; j) are the gray-level of pixel (i; j) in the Gaussian pyramid at the kth and (k 1)th levels, respectively. Note that this expression of local contrast is also inspired by Weber-Frechner's intuitive de nition. Indeed, the numerator is nothing more than a di erential signal (intensity increment) and the denominator is the signal against which the increment is measured (reference signal). Here is also,the directional sensitivity and the colorfulness aspects are note taken into account. This contrast measure has inspired several works and led to various interesting applications such as image fusion [ 44 ], contrast enhancement [ 40 ] and image distortion prediction [ 14 ]. (12) (13) (14)

Bandlimited contrast

By exploiting the results of psycho-physical experiments on the frequency sensitivity of the HVS to contrast and in particular its behaviour as a bandpass lter, Peli introduced the notion of band-limited contrast [ 35 ]. The image signal is analyzed using a bank of cos-log type isotropic band-pass lters to extract the di erent components describing the signal at di erent frequency bands. The image component captured by the kth channel is given by Where hk, is the impulse response corresponding to the kth band-pass lter and gk the associated ltered component. For each pixel (i; j) in the kth component, the contrast is expressed as: gk(i; j) = (f

hk)(i; j): where the bk is given by:

Here, too, it can be assumed that this measure is somehow inspired by the intuitive idea of Weber and Fechner's model. Indeed, the numerator is the di erential signal , i.e. pass-band signal, and the denominator contains the sum of all the lower frequency components, i.e. baseband signal. This ratio does measure a relative change in signal amplitude just as in the Weber-Fechner contrast model.

Like Toet's contrast, Peli's contrast can be used for complex natural images, unlike other contrasts such as Weber or Michelson. However, the lack of directional selectivity and colorfulness aspects of Peli's contrast limits its use in some real-world applications where these aspects play prominent roles. 4.4

Daly contrast

The contrast model proposed by Daly [ 14 ] is essentially based on the cortex transform introduced by Watson [ 1 ]. It should be noted that in this contrast model both frequency selectivity and directional selectivity are taken into account by means of two families of linear lters. The input signal f is rst analysed by means of two cascades of isotropic band-pass linear lters called "dom" and "fan" corresponding to the frequency selectivity for the former and the directional selectivity for the latter [ 14 ]. The ltered versions of the input signal f are given by: gkl(i; j) = (hkl f )(i; j) (18) where k and l are the dom and fan lter indices as de ned in [ 1 ]. Daly contrast is then de ned by

Ckl(i; j) = gkl(i; j)

gkl(i; j) gkl(i; j) :

Where gkl is the mean of the band (k; l). Note that this measure is unstable because of the denominator, which tends to zero. Daly proposes two solutions to overcome this problem. He then introduces two contrast measures where the denominator is replaced in one case by the baseband signal mean and in the other by the baseband signal calculated at each pixel. These two modi ed contrast measures are given by: and

Ckl(i; j) = Ckl(i; j) = gkl(i; j)

gkl(i; j) gK gkl(i; j) gkl(i; j) gK (i; j) : : (19) (20) (21) (22) Note that Daly contrast model does not incorporate the colorfulness aspect. This contrast measure has been used successfully in the design of image distortion prediction models [ 14 ]. 4.5

Isotropic contrast

The consideration of the multi-scale aspect in the visual contrast measure is to some extent related to the characteristics of the HVS [ 26 ]. One of the rst contrast measures based on wavelet analysis was introduced by Winkler and Vandergheynst [ 50 ]. It is surprising that most of the contrast measures proposed so far did not explicitly take into account the multiscale aspect. However, it should be noted that Toet's proposed measure is somehow quite close in that it introduces the multi-resolution aspect. The main idea of the measure proposed by Winkler and Vandergheynst was to overcome the limitations of the contrast proposed by Peli. They proposed a contrast measure using a directional wavelet decomposition based on a translation invariant multiresolution representation using 2-D analytical lters. By combining the di erent analytic oriented lter responses they derived the isotropic contrast expressed as follows. where gkl(i; j) is the gray-level at pixel (i; j) in the band-limited directional ltered image obtained by ltering the input signal f by the directional wavelet at resolution k and direction l. Similarly to Peli's contrast, the denominator corresponds to the baseband signal, i.e. the ltered signal with the scaling function at scale k. It has been demonstrated that in contrast to the Peli's model, this new contrast gives a at response to sinusoidal patterns [ 50 ]. However, it is important to note that although in its design this contrast uses directional lters, it provides an isotropic contrast measure. This could be bene cial for certain applications where directionality is not important, such as in the case of digital watermarking [ 46 ]. 4.6

Directional bandlimited contrast

Based on the work of Peli and Daly, Dauphin et al. [ 16 ] proposed a contrast where directional selectivity is taken into account in the nal contrast measure. Indeed, other contrasts such as those of Daly and Winkler-Vandergheynst integrate directional selectivity in the analysis of signal components but the nal contrast is rather isotropic. Whereas, in the model de ned in [ 16 ] the nal contrast measure is anisotropic in the sense that the nal response emphasizes the most directional salient signal components in the signal. A non-linear operation of type max is thus used for the calculation of the local contrast. The image signal is analysed using a multichannel Gabor decomposition. The local directional bandlimited contrast is computed as Where gk(i; j) is the gray-level associated to the frequency sub-band k and to one of the four directions (0; =4; =2; 3 =4) represented by l. The normalization term gk(i; j) represents the total energy of the background below the band (k) which is obtained by ltering the original image by a Gaussian lter with a standard deviation This local bandlimited directional contrast does not incorporate the colorfulness aspect. It has been compared to Peli's contrast and has been proven more e cient and less complex than the wavelet-based contrast proposed in [ 50 ]. The RAMMG contrast measure proposed by Rizzi et al. [ 39 ] is one of the few measures of contrast that incorporates both multi-resolution and colour aspects. The image signal is decomposed using a pyramidal scheme in the CIELAB colour space. At each level of resolution each pixel is associated with a local contrast de ned as the response of a pseudo-Laplacian computed by convolving the input signal I by the mask S given by:

S = Ck(i; j) = (Dk The local contrast at the kth level of the pyramid is computed as follows. (23) (24) (25) where Dk(i; j) is the absolute di erence of the luminance between the current pixel (i; j) and the central pixel at the kth level of resolution. The RAMMG global contrast is obtained by averaging all local contrasts across all the di erent levels of the pyramidal decomposition.

CRAMMG =

Wl 1 Hl

K Wl Hl X X X Ck(i; j): k=1 i=1 j=1 (27) where K is the total number of decomposition levels and Wk and Hk represent the width and height of the image at the kth level respectively. A very similar global contrast measure called Gobal Contrast Factor (GCF) has been proposed in [ 31 ]. GCF contrast has been found somehow consistent with subjective ranking of a relatively wide range of natural images with varying contrast. However, it should be noted that both contrast RAMMG et GFC are not relative measures of the variation in the energy of the image signal and therefore cannot be included in the family of contrasts conforming to the notion of contrast as de ned by Weber-Fechner. 5

Statistical features based contrasts

Very few contrasts measures based on statistical information have been introduced in the literature. These measures are often related to the pixel values distribution, such as the grey-level histogram or the 2D distribution computed from the grey-level cooccurrence matrix (GLCM). Here we limit ourselves to three contrast metrics based on some simple statistical features of pixel values. 5.1

Texture contrast measure

Haralick was the rst who introduced the idea of using some statistical invariant features for texture analysis. The set of these spatial descriptors introduced in [ 23 ] are based on the GLCM computed from the digital image. Among the Haralick's texture descriptors a global contrast is de ned. It is computed as follows.

K 1 K 1 CH = X X )(i i=0 j=0 j)2pij (28) Where i and j are the grey-levels of adjacent pixels in a de ned neighbourhood and pij is the joint mass probability function computed from the GLCM. Although this measure has always been identi ed as a contrast, it does not meet the basic criteria to be truly considered as a measure of contrast in the usual sense and in line with psycho-physical experiences and the notion of contrast that has been well established since the 19th century.

Mutual information based contrast measure

Another way to exploit inter-pixel correlation, directly related to image contrast, is to consider the measure of mutual information extracted from the GLCM. A new global contrast measure based on mutual information has thus been introduced for the rst time to quantify the side e ects such as saturation or halo e ect that could result from contrast enhancement [ 38 ]. This contrast measure is de ned by:

CMI = KXi=01 KXj=01 pxy(i; j) log2 ppxx(iy)(pi;yj(j)) !; (29) where pxy is the joint probability mass function of the gray-level, whereas px and py represent the marginal probabilities computed from the GLCM. While this contrast is simple to compute, however, it does not provide information directly related to visual contrast as it is purely based on statistical analysis of the signal values distribution. 5.3

Root Mean Square contrast

The Root Mean Square (RMS) of the luminance in natural images has been considered by Bex and Makhous [ 7 ] as a potential contrast measure in their study on human observer sensitivity to contrast. As noticed by these authors, this measure when divided by the average mean luminance of the image is a good predictor of the relative contrast.

Another version of this RMS based contrast has been proposed by Frasor and Geisler [ 18 ] to make it more suitable to natural images . Local contrast is measured in di erent randomly selected patches in the image. The contrast associated with a given patch is calculated as follows: (30) (31) (32) (33) where N is the total number of pixels in the patch, Li is the luminance of the ith pixel, L the patch luminance and wi is a windowed isotropic weighting function given by: where, ri = p(xi xc)2 + (yi yc)2 , p is the radius of the patches , (xi; yi) is the position of the ith pixel within the patch, and (xc; yc) is the center of the patch. The patch luminance is given by

It could be noticed that none of these statistical information based contrast measures take into account the spatial frequency and directional content of the visual signal. Nor do they incorporate other important aspects such as the luminance in uence of near and far surrounds, the viewing distance and chromatic aspect. Table 1 summarizes the key features of these representative local and global contrast measures.

Concluding remarks From this brief review, we conclude that it is uneasy to nd a simple contrast measure that incorporates all the relevant perceptual aspects related to the contrast notion. Therefore, most often simple measures derived from psycho-physical experiments are used and adapted to digital images to solve a number of real problems such as image and video processing, analysis and compression. Despite the enormous amount of work dedicated to visual contrast, it is still not easy to evaluate and compare the di erent contrast measures objectively. However, a few studies limited to subjective evaluation have been carried out in the literature [ 41 ].

Moreover, through this critical analysis on the existing contrast measures, it becomes important to answer some relevant questions. One important question is: what are the most relevant visual signal characteristics that the contrast measure should capture? To the best of author's knowledge none of the published work addressed this issue properly. However, some interesting studies have been dedicated to address this critical question [ 36 ]. Peli conducted a thorough experimental study in 1997 on how to de ne the contrast measure [ 36 ]. To be consistent with the ndings on the human visual perception the study provides guidelines con rming that computational contrast metrics should take into account multiscale aspects [ 26 ].

It is also worth noticing that there is no clear and objective criteria nor ground truth data to exploit in comparing the proposed contrast measures. Although, some attempts have been made in the study of di erent measures of contrast in digital images [ 41 ]. 6

A Brief Overview of the use of Contrast in VIPA An important issue to consider is how the contrast measure is de ned and used in developing various methods for Visual Information Processing and Analysis (VIPA). We limit ourselves here to a few applications where the notion of contrast plays a predominant role. It should be noted that the choice of contrast measure depends on the application. It is not always easy to make an appropriate choice and pragmatic solutions are often used, based mainly on feedback. 6.1

Image Quality Assessment and Enhancement

Most objective HVS-based image quality methods explicitly or implicitly incorporate contrast measure [ 11 ]. Peli's contrast and its variants have been successfully incorporated in the design of IQA measures [ 15 ] and distortion predictor MI based contrast [ 38 ] 1992 1993 1999 2003 2004 2006 2015

Key features De ned on simple images (an object of uniform intensity in a uniform background). Could not be applied to natural and complex images used for sinusoidal signals, not adapted for natural images. Does not integrate any HVS properties. considers object embedded in a non-uniform luminance background. Can be applied to digital images of natural scenes with a few adjustments and adaptations. capture the average local variations and spatial dependence of the pixels computed from gray-level co-occurrence matrix uses local edginess information in an image and quanti es local sharpness of the contour by measuring the visibility of the salient features by using a sliding window. a spatial multi-resolution contrast based on Gaussian pyramid decomposition.

An image is decomposed into several channels using a bank of cosine-log bandpass lters, well suited for complex images and often used in the computation of the quality of encoded images.

It is band-limited contrast based on the cortex transform and used by Daly in the Visible Di erence Predictor (VDP) model [ 48 ]. not easy to use in practice since it requires beforehand knowledge of the object contours and computation of the curvilinear integral along the boundaries of the objects contained in the observed image. uses non-seperable directional lters. It has the advantage of giving a at response instead of an oscillating response to sinusoid gratings.

Based on Lateral Geniculate Nucleus responses model [ 20 ]. It uses Di erence Of Gaussian (DOG) to model the bandlimited responses.

It is local directional bandlimited contrast computed from multichannel Gabor decomposition and a nonlinear operation.

Multilevel analysis (pyramid representation), 8neighborhood pyramid subsampling of the image to various levels in the CIELAB. a based on the RMS contrast introduced by Bex and Makous [ 7 ] using a windowed isotropic weighting function. It is a single scale contrast based on only pixel values. based on the mutual information computed from the gray-level co-occurrence matrix. models [ 14 ]. Another application where the concept of contrast is important concerns the improvement of image quality and in particular Contrast Enhancement (CE). The CE methods could be roughly classi ed into two categories, namely direct methods and indirect methods [ 4 ]. The main idea of the direct methods is to estimate the local contrast and to amplify it by means of a monotonic transformation and to deduce the intensity of the pixel corresponding to this new contrast. The contrast de ned by Beghdadi and Le Negrate [ 4 ] and its variants have been successfully used in many CE methods [ 3 ]. The edginess based contrast measure has been also extended to stereo images by incorporating the depth information [ 21 ]. Another direct method based on the bandlimited contrast [ 35 ] proposed by Tan et al. [ 43 ] which operates in the discrete cosine transform domain. Unfortunately most CE methods su er from some side e ects and there is no uni ed framework to control these e ects. Indeed, this pre-processing that aims at amplifying the visibility of details by increasing gradient and sharpness may introduce some artefacts such noise ampli cation, saturation and halo e ect. It is therefore useful to quantify these undesirable side e ects. Many objective measures for Contrast Enhancement Evaluation (CEE) based on the local or global contrast have been studied in [ 37 ]. A critical study of CEE metrics has revealed that the mutual information contrast measure is the most promising in terms of simplicity and e cacy [ 5 ]. However, this metric has limitations and it is in our interest to develop other measures that integrate the multi-scale and multi-directional aspects. From this point of view the three contrast measures de ned in [ 8, 16, 50 ] are good candidates. 6.2

Visual Data Protection and Compression

The protection and coding of visual data are two classic problems of very active research. Here, it is more precisely about image watermarking using perceptual approaches [ 3 ]. The aim is to insert a watermark that is both invisible and resistant to various attacks. This is then a very di cult problem where one tries to achieve the best compromise between two antagonistic criteria that are robustness and transparency. Indeed, robustness requires putting more energy into the watermark, which inevitably makes it visible and therefore breaks the transparency. The visibility of the watermark is very much linked to the notion of JND de ned in the contrast measure. In this type of application where one seeks to insert in a robust and transparent way the watermark the multi-scale or pyramidal approach through JND measures related to the contrasts de ned in [ 44,50 ] is the most promising solution as shown by the studies in [ 34, 46 ]. The other application where contrast plays an important role is image compression with quality control. This involves using contrast as a measure of the visibility of distortion and artifacts inherent in lossy compression methods. The JND measure, again very much related to contrast, and the visual masking phenomenon are the most important parameters to be considered in the quantization and coe cient selection scheme in the transformed domain. The idea of exploiting contrast in the quanti zation of the image signal dates back to the work of Kretz [ 28 ] based on the contrast of Moon and Spencer [ 33 ]. Since this pioneering work other more sophisticated contrast measurements have been successfully introduced into image compression and video coding models [ 12, 47 ]. Another interesting perceptual coding scheme proposed in [ 51 ] based on the Watson-Solomon Contrast Gain Control (CGC) [ 49 ] model has been proposed for High E ciency Video Coding (HEVC). It is worth noticing that most of visually lossless coding methods in the literature exploit the contrast measure in an explicit or implicit manner [ 3, 48 ]. The only contrast model for perceptual coding and quantization that seems to be complete and e cient is the one introduced recently in [ 51 ]. Therefore, we recommend to use this model for perceptual coding. 6.3

Image fusion

Image fusion is becoming an active eld of research and especially with the reviving of arti cial intelligence based approaches. The use of perceptual information, and particularly contrast, in visual data fusion schemes seems to be the most promising approach in various applications [ 8, 40 ]. There are several ways to merge information, depending on the application and the data analysis method used. A rst approach is to exploit the multi-scale representation of visual information in the development of the fusion scheme. Perceptual contrast is one of the signal features that could be used in the design of the fusion scheme. The idea behind the use of perceptual contrast is to exploit the most relevant perceptual information that is contained in the contrast map. The strategy consists then in using the contrast measure in the weighting function used in the fusion scheme. One of the attractive image fusion approach based on this idea is to use the directional wavelet based contrast as done in [ 8 ]. This strategy could be used not only in multimodal medical imaging but also in various applications such multi-modal video-surveillance, multi-focus based computing photography and hyperspectral imaging, to name a few. It is also the case of the contrast enhancement method based on a perceptual fusion scheme proposed in [ 40 ]. 7

Conclusions and challenges

Through this panoramic and chronological study on visual contrast, and the underlying models and experimental studies, it becomes clear that it is not easy to express a contrast measure where all the relevant psycho-physical factors and parameters related to the notion of visual contrast are taken into account. Note also that the notion of contrast and its use are very application-dependent and the best way to exploit a visual contrast model to solve real problems is to simplify it while keeping the most fundamental aspects.

Furthermore, it is di cult to classify the existing de nitions and measures related to the notion of contrast. This is mainly due to the various forms and representations of visual and optical signals. Indeed, development of imaging technologies has led to various image modalities. Therefore, it is now necessary to rethink and de ne the concept of contrast according to the modality of the visual signal under consideration.

It should also be noted that the absence of a universal de nition of visual contrast has opened the door to formulations and extensions of this concept to other contexts where one seeks only to quantify the di erence between two stimuli or elements of the signal. This has led, for example, to the consideration of signal gradient measure as contrast in several studies. A unifying framework with clear criteria for de ning the contrast will certainly help in avoiding such confusions and mistakes and allows to progress properly in exploiting vision research results in developing e cient contrast based VIPA methods.

We can also see through this study that the measure of perceptual contrast in the case of colour images is not su ciently studied. According to the author's knowledge at the present time there is no well established de nition or measure of chromatic contrast that is recognized by the scienti c community in the eld of vision research or digital image processing.

The temporal aspect is also important to introduce in the contrast measure. Another aspect that could be taken into account is the inter-channel interactions in the de nition of multi-scale contrast. With the renewed interest in arti cial intelligence approaches for solving complex problems and in particular featurebased learning approaches, the contrast may play a key role in the design of perceptual loss functions in convolutional neural network architectures.

1. Andrew

, W.: The cortex transform: Rapid computation of simulated neural images . Computer Vision Graphics and Image Processing 39 ( 3 ), 311 { 327 ( 1987 )

2. Beghdadi , A. , Dauphin , G. , Bouzerdoum , A. : Image analysis using local band directional contrast . In: Proc of the International Symposium on Intelligent Multimedia, Video and Speech Processing , ISIMP' 04 ( 2004 )

3. Beghdadi , A. , Larabi , M.C. , Bouzerdoum , A. , Iftekharuddin , K.M.: A survey of perceptual image processing methods . Signal Processing: Image Communication 28 ( 8 ), 811 { 831 ( 2013 )

4. Beghdadi , A. , Le Negrate , A. : Contrast enhancement technique based on sc detection of edges . Computer Vision , Graphics, and Image Processing 46 ( 2 ), 162 { 174 ( 1989 )

5. Beghdadi , A. , Qureshi , M.A. , Deriche , M.: A critical look to some contrast enhancement evaluation measures . In: 2015 Colour and Visual Computing Symposium (CVCS) . pp. 1 { 6 . IEEE ( 2015 )

6. Belkacem-Boussaid , K. , Beghdadi , A.: A new image smoothing method based on a simple model of spatial processing in the early stages of human vision . IEEE Transactions on Image Processing 9 ( 2 ), 220 { 226 ( 2000 ). https://doi.org/10.1109/83.821735

7. Bex , P.J. , Makous , W. : Spatial frequency, phase, and the contrast of natural images . JOSA A 19 ( 6 ), 1096 { 1106 ( 2002 )

8. Bhatnagar , G. , Raman , B. : A new image fusion technique based on directive contrast. ELCVIA: electronic letters on computer vision and image analysis 8(2 ), 18 { 38 ( 2009 )

9. Bhatnagar , G. , Wu , Q.J. , Liu , Z. : Directive contrast based multimodal medical image fusion in nsct domain . IEEE transactions on multimedia 15(5) , 1014 { 1024 ( 2013 )

10. Burt , P. , Adelson , E. : The laplacian pyramid as a compact image code . IEEE Transactions on communications 31(4) , 532 { 540 ( 1983 )

11. Chandler , D.M.: Seven challenges in image quality assessment: past, present, and future research . ISRN Signal Processing 2013 ( 2013 )

12. Chandler , D.M. , Hemami , S.S.: Dynamic contrast-based quantization for lossy wavelet image compression . IEEE Transactions on Image Processing 14 ( 4 ), 397 { 410 ( 2005 )

13. Cornsweet , T. : Visual Perception. Academic Press ( 1970 )

14. Daly , S.J.: Visible di erences predictor: an algorithm for the assessment of image delity . In: Human Vision , Visual Processing, and Digital Display III . vol. 1666 , pp. 2 { 16 . International Society for Optics and Photonics ( 1992 )

15. Damera-Venkata , N. , Kite , T.D., Geisler , W.S. , Evans , B.L. , Bovik , A.C. : Image quality assessment based on a degradation model . IEEE transactions on image processing 9 ( 4 ), 636 { 650 ( 2000 )

16. Dauphin , G. , Beghdadi , A. , de Lesegno, P.V. : A local directional bandlimited contrast . In: Seventh International Symposium on Signal Processing and Its Applications . vol. 2 , pp. 197 { 200 . IEEE ( 2003 )

17. Fechner , G.: Elemente der psychophysik (leipzig: Breitkopf & hartel) . In: English translation of Vol. 1 by HE Adler 1966 . Holt, Rinehart, and Winston New York ( 1860 )

18. Frazor , R.A. , Geisler , W.S.: Local luminance and contrast in natural images . Vision research 46 ( 10 ), 1585 { 1598 ( 2006 )

19. Gordon , R. , Rangayyan , R.M. : Feature enhancement of lm mammograms using xed and adaptive neighborhoods . Applied optics 23(4) , 560 { 564 ( 1984 )

20. Grossberg , S. , Mingolla , E. , Todorovic , D.: A neural network architecture for preattentive vision . IEEE Transactions on Biomedical Engineering 36 ( 1 ), 65 { 84 ( 1989 )

21. Hachicha , W. , Beghdadi , A. , Cheikh , F.A. : Combining depth information and local edge detection for stereo image enhancement . In: 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO) . pp. 250 { 254 . IEEE ( 2012 )

22. Hansen , T. , Gegenfurtner , K.R. : Color contributes to object-contour perception in natural scenes . Journal of Vision 17 ( 3 ), 14 { 14 ( 2017 )

23. Haralick , R.M. , Shanmugam , K. , Dinstein , I.H. : Textural features for image classication . IEEE Transactions on systems, man, and cybernetics (6) , 610 { 621 ( 1973 )

24. Haun , A.M. , Peli , E.: Measuring the perceived contrast of natural images . In: SID Symposium Digest of Technical Papers . vol. 42 , pp. 302 { 304 . Wiley Online Library ( 2011 )

25. Hecht , S.: The visual discrimination of intensity and the weber-fechner law . The Journal of general physiology 7 ( 2 ), 235 { 267 ( 1924 )

26. Hubel , D.H. , Wiesel , T.N. : Brain mechanisms of vision . WH Freeman ( 1979 )

27. Iordache , R. , Beghdadi , A. , de Lesegno, P.V.: Pyramidal perceptual ltering using moon and spencer contrast . In: Image Processing , 2001 . Proceedings. 2001 International Conference on. vol. 3 , pp. 146 { 149 . IEEE ( 2001 )

28. Kretz , F. : Subjectively optimal quantization of pictures . IEEE Transactions on Communications 23 ( 11 ), 1288 { 1292 ( 1975 )

29. Levine , M.D.: Vision in man and machine . McGraw-Hill College ( 1985 )

30. Lillesaeter , O. : Complex contrast, a de nition for structured targets and backgrounds . JOSA A 10 ( 12 ), 2453 { 2457 ( 1993 )

31. Matkovic , K. , Neumann , L. , Neumann , A. , Psik , T. , Purgathofer , W. : Global contrast factor-a new approach to image contrast . Computational Aesthetics 2005 , 159 { 168 ( 2005 )

32. Michelson , A. : Studies in Optics . The Univ. of Chicago Science Series , University Press ( 1927 ), https://books.google.fr/books?id=FXazQgAACAAJ

33. Moon , P. , Spencer , D.E. : Visual data applied to lighting design . JOSA 34 ( 10 ), 605 { 617 ( 1944 )

34. Nguyen , P.B. , Beghdadi , A. , Luong , M. : Perceptual watermarking using a new just-noticeable-di erence model . Signal Processing: Image Communication 28 ( 10 ), 1506 { 1525 ( 2013 )

35. Peli , E.: Contrast in complex images . JOSA A 7 ( 10 ), 2032 { 2040 ( 1990 )

36. Peli , E.: In search of a contrast metric: Matching the perceived contrast of gabor patches at di erent phases and bandwidths . Vision Research 37 ( 23 ), 3217 { 3224 ( 1997 )

37. Qureshi , M.A. , Beghdadi , A. , Deriche , M. : Towards the design of a consistent image contrast enhancement evaluation measure . Signal Processing: Image Communication 58 , 212 { 227 ( 2017 )

38. Qureshi , M.A. , Deriche , M.A. , Beghdadi , A. , Mohandes , M.: An information based framework for performance evaluation of image enhancement methods . In: 2015 International Conference on Image Processing Theory, Tools and Applications , IPTA 2015 , Orleans, France, November 10-13 , 2015 . pp. 519 { 523 ( 2015 )

39. Rizzi , A. , Algeri , T. , Medeghini , G. , Marini , D.: A proposal for contrast measure in digital images . In: Second European Conference on Color in Graphics, Imaging, and Vision (CGIV). pp. 187 { 192 . Society for Imaging Science and Technology, Aachen ( 2004 )

40. Saleem , A. , Beghdadi , A. , Boashash , B. : A distortion-free contrast enhancement technique based on a perceptual fusion scheme . Neurocomputing 226 , 161 { 167 ( 2017 )

41. Simone , G. , Pedersen , M. , Hardeberg , J.Y. : Measuring perceptual contrast in digital images . Journal of Visual Communication and Image Representation 23 ( 3 ), 491 { 506 ( 2012 )

42. Tadmor , Y. , Tolhurst , D. : Calculating the contrasts that retinal ganglion cells and lgn neurones encounter in natural scenes . Vision research 40 ( 22 ), 3145 { 3157 ( 2000 )

43. Tang , J. , Peli , E. , Acton , S. : Image enhancement using a contrast measure in the compressed domain . IEEE signal processing LETTERS 10 ( 10 ), 289 { 292 ( 2003 )

44. Toet , A. : Image fusion by a ration of low-pass pyramid . Pattern Recognition Letters 9 ( 4 ), 245 { 253 ( 1989 )

45. Triantaphillidou , S. , Jarvis , J. , Psarrou , A. , Gupta , G.: Contrast sensitivity in images of natural scenes . Signal Processing: Image Communication 75 , 64 { 75 ( 2019 )

46. Vandergheynst , P. , Kutter , M. , Winkler , S. : Wavelet-based contrast computation and application to digital image watermarking . In: Wavelet Applications in Signal and Image Processing VIII . vol. 4119 , pp. 82 { 93 . International Society for Optics and Photonics ( 2000 )

47. Watson , A.B. : Dct quantization matrices visually optimized for individual images . In: Human vision, visual processing, and digital display IV . vol. 1913 , pp. 202 { 216 . International Society for Optics and Photonics ( 1993 )

48. Watson , A.B. (ed.): Digital Images and Human Vision . MIT Press, Cambridge, MA, USA ( 1993 )

49. Watson , A.B. , Solomon , J.A. : Model of visual contrast gain control and pattern masking . J. Opt. Soc. Am. A 14 ( 9 ), 2379 {2391 (Sep 1997 )

50. Winkler , S. , Vandergheynst , P. : Computing isotropic local contrast from oriented pyramid decompositions . In: Image Processing , 1999 . ICIP 99. Proceedings . 1999 International Conference on. vol. 4 , pp. 420 { 424 . IEEE ( 1999 )

51. Zhang , Y. , Alam , M.M. , Chandler , D.M.: Visually lossless perceptual image coding based on natural-scene masking models . Recent Advances in Image and Video Coding p. 1 ( 2016 )