Two diferent environments were considered in this study to conduct the psychophysical experiment to find a relationship between the perceived and calculated colour diference in display colours. One of the experiments was in a controlled environment, which took place in the NTNU colour lab using PsychoPy, and the other was conducted online by hosting the same experiment on the Pavlovia website. The experiment was conducted by comparing 48 diferent colour patches having 6 diferent color centers. A number of colour diference measurement formulas using diferent mathematical and statistical methods are introduced on a regular basis by incorporating diferent concepts. To find the relationship between perceived and computed colour diference, this study used CIEDE2000, CIE94 and CIE76 to calculate the colour diference using formulas. And a category judgment experiment using greyscale anchor point having six categories used for perceived colour diference. There were 17 observers in a controlled environment and 25 observers in an uncontrolled environment experiment without having any colour deficiency problems. The Standard Residual Sum of Squares (STRESS) index was used to find the relationship between the perceived and computed colour diference. Alongside, the STRESS function is also used to test the observers inter and intra observer variation and repeatability. The results show that controlled environment observers achieved a more rational relationship according to the stress index in perceived and computed colour diference using all three formulas. According to the result of inter and intra observer variation, controlled environment observers were more consistent, which archived a lower stress value.
Colour diference formulas are widely used in diferent industries including image quality
measurement, printing technologies, textiles, dentistry which have a commendable impact.
By comparing two colour samples, a predicted visual diference can be found using colour
diference formulas, which can be referred as ΔE. MacAdam started as one of the very first
approaches using psychological experiments in color vision research by viewing two diferent
colours to the observers, one is fixed as test colour and the other was adjustable and asked
to adjust the second one as test colour [
To inspect these challenges inter and inter-observer variability was investigated.
The measurement of visual diference is considered a subjective experiment conducted on
our visual sensory system where this perceived diference between two diferent stimuli is
considered as ΔV. On the other hand, to get the calculated diference ΔE, a number of metrics
were introduced by the time. CIELAB colour diference formula, which is known as CIE76 as
well, was figured out based on perceptually uniform colour space LAB, which had a considerable
agreement with the visual judgement of colour diferences [
A significant number of approaches were utilized to conduct the colour diference experiment. But the more accepted ways are using the grey scale comparison method and the anchor pair method. In [10], using the anchor pair method, the study asked the observer to find whether the colour diference is larger or smaller, comparing the anchor pair and test pair. Grey scale pairs method was used in [11] to compare the sample colour pair and a number of grey scale anchor pairs where the observer was asked to choose the grey scale pair comparing the most relate-able grey scale pair and sample coloured pairs having diferent lightness.
PF/3 (performance factor as an average of three terms) used to show the relationship between
the perceived colour diference and computed colour diferences in mid 2000s [ 12, 13]. However,
to evaluate the coherence in between the perceived colour diference and the calculated colour
diference the Standardized Residual Sum of Squares (STRESS) has been broadly used in recent
studies [14, 15, 16]. The less the amount of index value shows, the better consistency of visual
data and quantitative value. Mean-variance of observer group in diferent color centers utilized
in [
STRESS: Following equations are used to calculate the STRESS index value, where ΔE takes the value of the calculated diference and ΔV contains the perceived response [11, 14]. = 100 ︃( ∑︀ (Δ − 1Δ)2 )︃1/2
∑︀ 12Δ2 ∑︀ Δ2 1 = ∑︀ ΔΔ √︁ Δ* =
(*2 − *1)2 + (*2 − *1)2 + (*2 − *1)2
CIE76: To find the computed colour diference, following formula used in CIE76 [
The psychophysical experiment in a controlled environment was conducted using Psychopy v2021.2 which was locally installed in the NTNU Colourlab, in a dimly lit room. For the uncontrolled environment, the same experiment was hosted online using Pavlovia.org [18] website, the experiment starts with general instruction, having an Ichihara test in the very beginning to test whether the observers group has a normal colour vision or not. There was a trial test taken before the experiment to get familiar with the experiment-taking procedure. Moreover, a repeat test was taken to check the repeatability of the observer, although it was not disclosed earlier to the observers to test real repeatability. Figure 1 shows the flow chart of the Psychopy experiment.
Figure 2 presents the experimental setup which has a colour patch in the middle surrounded
by 6 anchor grey scale patches, which are referred as the categories of colour diferences.
Observers have the radio button options at the bottom to choose the scale of visual colour
diference between 1 to 6, which is mentioned in the top left corner of the grey patches. The 6
grey patches (1 to 6) refer to the six diferent categories of colour diferences which are: no colour
diference, just a noticeable diference, very small diference, small colour diference, medium
colour diference and large colour diference. These categories were introduced in [
The experiment in a controlled environment was conducted on an AdobeRGB calibrated display having 80/2 luminance and a white point of 6500k which have a gamma value of 2.2. There is a chin rest placed to maintain a distance of 50 cm from the display. There was a 15-minute warm-up time reserved after starting the monitor before starting the experiment, which ensures the highest luminance of the display. The tristimulus values of all the samples were measured using the Tele-spectroradiometer Konica Minolta CS2000, keeping a distance of 50 cm between the Tele-spectroradiometer and the display. Then using the display white point and utilizing [19], the tristimulus values were converted into CIELAB values. The display used for the experiment was an EIZO ColorEdge CG246 and the room lighting condition was dim. Due to the pandemic situation, the mouse, keyboard, and the chin rest used for the experiment were sanitized every time after completing the experiment by each observer.
Pavlovia.org [18] provides the facility to host the experiment online using Psychopy. The website is a subscription-based platform that requires credits to run the experiment online. The setup experiment went through the piloting to make sure it was working fine online with the same appearance as in the lab. In an uncontrolled environment, it is a bit challenging to impose restrictions on the observers to maintain certain criteria, e.g., being in a certain viewing condition, managing a particular viewing angle, distance and consent about chin rest. Additionally, the calibration state and display type could be diferent for the uncontrolled environment observers. But observers were instructed to use the Google Chrome browser to get a better experiment experience and to eradicate the problems related to the appearance of the experiment setup and patches.
For the controlled environment, observers were collected using social media among the COSI master’s degree students and other sources. There were 17 observers in total, and all the observers had normal colour vision. And the uncontrolled environment observers were invited to take the experiment by sending the link through the emails, and 25 observers in total conducted the experiment. An Ichihara test took place at the beginning of the experiment in both cases to acknowledge there was no colour deficiency problem in the observer group.
As all the observers had normal colour vision, the diference between the colour patches was chosen as a small colour diference according to the guideline of [20]. For this experiment, six colour centers were considered, and eight diferent variations of each colour center were created, and the colour center was also included in these variants. Variants were created considering the display gamut. A total of 60 patches were shown in two steps for the experiment to compare, 48 of them were diferent, and 12 patches were shown repeatedly to inspect variations and repeatability of the observer. Table 1 listed all the 6 colour centers used for the experiment and their corresponding CIELAB values. And the background used for the experiment has CILAB values L*=54, b*=0, c*=0. Figure 3 shows the colour centers listed in Table 1 and shows that all the centers were in the gamut of the controlled environments experiment setup display. Figure 3 includes the display white point as well in the middle of the 6 centers. Figure 4 shows that all the variants of diferent colour samples are in the gamut of the display as well. This display gamut will not be the same for the observers who took the experiment online. That varies from display to display used by the observers online. The measured size of the controlled environment colour patches was 3.6cm*3.6cm for each patch, according to the size of the object and the distance from the display, the observer angle became 4 degrees, so observers can be considered as 4-degree observers for the controlled environment.
All the observers for controlled and uncontrolled environments went through the Ichihara test, and all the observers found the correct element shown in the picture, which concludes that all observers have normal colour vision. The visual scores of the observers for the uncontrolled environment group are higher than those of the controlled environment group. Table 2 shows the range of selected categories by averaging all the observer responses according to the colours and the overall average score for the colour. According to the table uncontrolled environment observer chosen the higher categories most which provides higher average scores in each category with an exception in Green.
The tristimulus values of 48 colour patches, including the display white point, were measured using a Tele-spectroradiometer on a controlled environment experiment’s display. Then these XYZ tristimulus values were converted to the CIELAB colour space using the white point of the display. Afterwards, the computed colour diferences for CIEDE2000, CIE94 and CIE76 were calculated using MATLAB R2021a using these CIELAB values. The computed range of the colour diferences from just a noticeable diference to a large diference scale considered for the experiment is summarized in Table 3, according to the colour centers and formulas used for the result comparison.
One of the methods utilized in earlier studies to visualize the computed data and perceived response data is plotting ΔE and ΔV which was used in [11]. For the visual response, the average of all the responses was taken and CIEDE2000, CIE94 and CIE76 were utilized to get the computed colour diference data. There was a total of 6 categories in the experiment, average categories plotted in Y axis of figure 5, figure 6 and figure 7. On the other hand, the computed data was plotted on the X-axis of the figures. There are a total of 48 blue points in all the figures to show the relationship between ΔE and ΔV. In all these three figures 5,6 and 7, the left side graph shows the relationship of controlled environment experiment and the right graph shows the relationship of uncontrolled environment experiments. The graphs also show a similar reflection as in Table 2 that uncontrolled environment observers perceive larger categories than the controlled environment observers.
Standard Residual Sum of Squares(STRESS) is the most recommended approach to finding the relationship between perceived and computed colour diferences. Table 4 represents the STRESS results for both controlled and uncontrolled environments in between the CIEDE2000 colour diference formula and average perceived category response. A lower index of stress shows the better relationship, hence here the controlled environment observer achieved a lower STRESS index which represents the controlled environment observers perceived better category assumption with the formula provided colour diference. And in between the colours, green has considerably gained a lower index value, which means controlled environment observers perceive green more rationally according to the comparison with CIEDE2000. However, both environmental observers achieve a higher index for Red, which reflects that red was less rationally perceived by the observers.
According to the Table 5, it shows the STRESS indexes found by using average ΔV and computed ΔE (using CIE94). Here in the controlled environment again, green achieved a better relationship according to the stress value. However, in an uncontrolled environment, blue achieved a lower index value, which provides a better relationship between the formula of CIE94 and the average perceived diference in the blue colour center.
Furthermore, Table 6 shows a diferent relationship than the previous two formulas as CIE76 provided large numbers in computed colour diferences. Here blue colour was perceived more Colour
Blue Green Red
Cyan Magenta Yellow rationally in controlled and uncontrolled environments, which reflects STRESS value that was comparatively lower than others.
However in between the colours, cyan provides a higher STRESS index in both environment experiments with all the formulas used which is present in all three tables 4, 5 and 6. Comparing these 3 tables, CIE94 achieved a better relationship with the perceived colour diferences, having lower STRESS index values most of the time, with some exceptions in some colour centers.
Inter-observer variants refer to the variability between one and other observers, which indicates variations between observers. But the intra-observer variability was measured by the observer’s own variation which can be found by repeating the same task twice and observing the repeatability of the observer.
The inter-observer variability according to each color center was calculated using stress in table 7. Firstly, the individual observer’s variability was calculated for each colour center variant using STRESS considering each individual observer’s response and the average observer’s response. And then the mean stress of all observers was reported here in the table. The same method was applied to find the mean stress for both controlled and uncontrolled environment experiment. Whereas Table 8 shows the intra-observer variability which was calculated using repeated patch’s first attempt response and second attempt response. Primarily individual observer STRESS index was calculated according to the colour centers, afterwards average stress of all observers was reported in the table. Comparing the results controlled environment observers were consistent in most of the colours according to both inter and inter-observer variation with some exceptions for green and red in intra-observer variation. Observing the stress value of inter and intra-observer variability in both controlled and uncontrolled environments, the stress value diference found was less than 1 to 5 stress values. In most of the colours centers, it was between less than 1 to 2.
Inter observer variability Intra observer variability Controlled environment Uncontrolled environment
Zhongning et al[21]
Melgosa et al[17] centers may need to choose considering smaller gamut of the display.
This study shows the coherence between the perceived colour diference and computed colour diference using controlled and uncontrolled environment for the experiment. The observers in the controlled environment have more rational results according to the STRESS index rather than the uncontrolled environment’s observer in comparison with colour diference formulas and inter-intra variation, although the uncontrolled environment achieved a considerable STRESS index as well. The experiments create a new dataset for further research on the experiment conducted online which can be utilized to compare with others to find a solution on how to move the experiment online. To conduct the psychophysical experiment on large-scale observers, which will give more insightful analysis, online uncontrolled environment experiments can be promoted considering the challenges in doing the experiment online. The uncontrolled environment benefits by allowing the observer from any place, which helps to find the results from anywhere in the world, which can be useful to find the region-wise perceived result for many particular applications which can be investigated in future.
The authors are thankful to NTNU colour lab members for helping with all the instruments required for the experiments and especially to Gregory High, Phd candidate for his support by sharing credits to host the experiment online. The observers sacrifice their crucial time to conduct the experiment, authors are grateful to them. Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 34 (2009) 367–374. [7] CIE, Technical report: Improvement to industrial colordiference evaluation, CIE Publication 142 (2001). [8] Q. Pan, S. Westland, Comparative evaluation of color diferences between color palettes, in: Color and Imaging Conference, volume 2018, Society for Imaging Science and Technology, 2018, pp. 110–115. [9] S. Zufi, P. Scala, C. Brambilla, G. Beretta, Web-based versus controlled environment psychophysics experiments, in: Image Quality and System Performance IV, volume 6494, SPIE, 2007, pp. 56–63. [10] R. S. Berns, D. H. Alman, L. Renif, G. D. Snyder, M. R. Balonon-Rosen, Visual determination of suprathreshold color-diference tolerances using probit analysis, Color Research & Application 16 (1991) 297–316. [11] A. K. Kvitle, H. Oddløkken, P. Green, P. Nussbaum, Methods for psychophysical assessment of colour diference by observers with a colour vision deficiency, in: 2018 Colour and Visual Computing Symposium (CVCS), IEEE, 2018, pp. 1–6. [12] H. Xu, H. Yaguchi, Visual evaluation at scale of threshold to suprathreshold color diference, Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 30 (2005) 198–208. [13] M. R. Luo, G. Cui, C. Li, Uniform colour spaces based on ciecam02 colour appearance model, Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 31 (2006) 320–330. [14] M. Melgosa, A. Trémeau, G. Cui, Colour diference evaluation, in: Advanced color image processing and analysis, Springer, 2013, pp. 59–79. [15] J. Liang, M. Georgoula, N. Zou, G. Cui, M. R. Luo, Colour diference evaluation using display colours, Lighting Research & Technology 50 (2018) 1257–1269. [16] M. Melgosa, R. Huertas, R. S. Berns, Performance of recent advanced color-diference formulas using the standardized residual sum of squares index, JOSA A 25 (2008) 1828–1834. [17] M. Melgosa, P. A. García, L. Gómez-Robledo, R. Shamey, D. Hinks, G. Cui, M. R. Luo, Notes on the application of the standardized residual sum of squares index for the assessment of intra-and inter-observer variability in color-diference experiments, JOSA A 28 (2011) 949–953. [18] Pavlovia, 2022. URL: https://pavlovia.org/. [19] P. J. Green, A colour engineering toolbox, 2003. [20] CIE, Recommended method for evaluating the performance of colour-diference formulae,
International Commission on Illumination (2016). [21] Z. Huang, H. Xu, M. R. Luo, G. Cui, H. Feng, Assessing total diferences for efective samples having variations in color, coarseness, and glint, Chinese Optics Letters 8 (2010) 717–720.