Colour Difference Assessment in Controlled and Uncontrolled Environments Mohammad Jaber Hossain1,* , Phil Green1 1 Department of Computer Science, Norwegian University of Science and Technology, Gjovik, Norway Abstract Two different environments were considered in this study to conduct the psychophysical experiment to find a relationship between the perceived and calculated colour difference in display colours. One of the experiments was in a controlled environment, which took place in the NTNU colour lab using PsychoPy, and the other was conducted online by hosting the same experiment on the Pavlovia website. The experiment was conducted by comparing 48 different colour patches having 6 different color centers. A number of colour difference measurement formulas using different mathematical and statistical methods are introduced on a regular basis by incorporating different concepts. To find the relationship between perceived and computed colour difference, this study used CIEDE2000, CIE94 and CIE76 to calculate the colour difference using formulas. And a category judgment experiment using greyscale anchor point having six categories used for perceived colour difference. There were 17 observers in a controlled environment and 25 observers in an uncontrolled environment experiment without having any colour deficiency problems. The Standard Residual Sum of Squares (STRESS) index was used to find the relationship between the perceived and computed colour difference. Alongside, the STRESS function is also used to test the observers inter and intra observer variation and repeatability. The results show that controlled environment observers achieved a more rational relationship according to the stress index in perceived and computed colour difference using all three formulas. According to the result of inter and intra observer variation, controlled environment observers were more consistent, which archived a lower stress value. Keywords Visual colour difference, psychological experiment, controlled environment, uncontrolled environment, colour vision. 1. Introduction Colour difference formulas are widely used in different industries including image quality measurement, printing technologies, textiles, dentistry which have a commendable impact. By comparing two colour samples, a predicted visual difference can be found using colour difference formulas, which can be referred as ΔE. MacAdam started as one of the very first approaches using psychological experiments in color vision research by viewing two different colours to the observers, one is fixed as test colour and the other was adjustable and asked to adjust the second one as test colour [1]. Psychological experiments allow measuring the perceived colour difference, which can be referred as visual difference (ΔV). CIE76,CIE94, The 11th Colour and Visual Computing Symposium, September 08-09, 2022, Gjøvik, Norway * Corresponding author. $ mohammjh@stud.ntnu.no (M. J. Hossain); philip.green@ntnu.no (P. Green)  0000-0002-0611-4020 (M. J. Hossain) © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) CIEDE2000 are widely used colour difference formulas to get the computed colour difference [2, 3, 4]. According to the study in [5], these CIE recommended colour difference formulas are developed to find the relationship with visual colour differences more accurately, which are applicable on CIELAB colour spaces for small colour differences. In [6], the study claimed that a considerable amount of variability was found in colour difference experiment in between the observers, which was acknowledged by inter-observer and intra-observer variability. This variability finding approaches also referred as repeatability and respectively in some cases, was not addressed in the experiments done in earlier ages of this type of experiment alike MacAdam’s experiment [1]. For this study, we conducted the experiment in a controlled environment which took place in the NTNU colour Lab and in an uncontrolled environment, the experiment was conducted online. There are some uncertainties in an uncontrolled environment. It is difficult to make sure all variant colour is in the gamut of the display of the observer or not in an uncontrolled environment. For this reason, the visual difference in an uncontrolled environment may show some inconsistency. The main objective of the study is enlisted as follows: • The study focused on finding out the uncertainty of visual difference that can occur in psychophysical experiments conducted online without taking any control over observers and comparing that with experiment conducted in a controlled environment. • Additionally, the study also examined the relationship between quantitative formulas measurement (CIEDE2000, CIE94, CIE76) and perceived visual difference in different environments. • Alongside, one of the other challenges of doing psychophysical experiments for visual colour differences can be cross-individual differences and repeatability of the observers. To inspect these challenges inter and inter-observer variability was investigated. 2. Related work The measurement of visual difference is considered a subjective experiment conducted on our visual sensory system where this perceived difference between two different stimuli is considered as ΔV. On the other hand, to get the calculated difference ΔE, a number of metrics were introduced by the time. CIELAB colour difference formula, which is known as CIE76 as well, was figured out based on perceptually uniform colour space LAB, which had a considerable agreement with the visual judgement of colour differences [2]. Subsequently, CIE94 was introduced in 1995 by simply correcting CIE76, working on perceptual colour difference’s dependency on chroma variation[4]. Afterwards, CIE recommended one of the most recent colour difference formula CIEDE2000 which was proposed in 2001 and formulated by reliable experimental data using large-scale combined dataset[3, 7]. In [5] the study proposed, that during the selection of colour difference formulas for a different range, CIEDE2000 should be considered because of this formula’s reasonable performance in small and large range of experiments. All these formulas CIEDE200, CIE94 and CIE76 are dependent on CIELAB colour spaces, which forces the necessity of getting CIELAB values from measured tristimulus values. The study [8] worked on display colour patches where the measured CIE tristimulus values XYZ converted to CIELAB colour space using the display white point for the further calculation on CIE76 colour difference formula. In [9] the study showed a comparison between controlled and uncontrolled environment psychophysical experiment where the study discarded 35 percent of data which were miss-classified where the study complement of web based uncontrolled experiments for visual studies. A significant number of approaches were utilized to conduct the colour difference experiment. But the more accepted ways are using the grey scale comparison method and the anchor pair method. In [10], using the anchor pair method, the study asked the observer to find whether the colour difference is larger or smaller, comparing the anchor pair and test pair. Grey scale pairs method was used in [11] to compare the sample colour pair and a number of grey scale anchor pairs where the observer was asked to choose the grey scale pair comparing the most relate-able grey scale pair and sample coloured pairs having different lightness. PF/3 (performance factor as an average of three terms) used to show the relationship between the perceived colour difference and computed colour differences in mid 2000s [12, 13]. However, to evaluate the coherence in between the perceived colour difference and the calculated colour difference the Standardized Residual Sum of Squares (STRESS) has been broadly used in recent studies [14, 15, 16]. The less the amount of index value shows, the better consistency of visual data and quantitative value. Mean-variance of observer group in different color centers utilized in [2] for the Inter and intra-observer variation analysis. But to find the inter-observer and intra-observer consistency and repeatability, the most recommended method to use is STRESS as well [17]. The range of the STRESS value is between 0 to 100, a small number of STRESS value represent a good agreement between the relationship factors. STRESS: Following equations are used to calculate the STRESS index value, where ΔE takes the value of the calculated difference and ΔV contains the perceived response [11, 14]. (︃ ∑︀ )︃1/2 (Δ𝐸𝑖 − 𝐹1 Δ𝑉𝑖 )2 𝑆𝑇 𝑅𝐸𝑆𝑆 = 100 ∑︀ 2 (1) 𝐹1 Δ𝑉𝑖2 Δ𝐸𝑖2 ∑︀ 𝐹1 = ∑︀ (2) Δ𝐸𝑖 Δ𝑉𝑖 CIE76: To find the computed colour difference, following formula used in CIE76 [2]: √︁ Δ𝐸𝑎𝑏 = (𝐿*2 − 𝐿*1 )2 + (𝑎*2 − 𝑎*1 )2 + (𝑏*2 − 𝑏*1 )2 * (3) 3. Methodology The psychophysical experiment in a controlled environment was conducted using Psychopy v2021.2 which was locally installed in the NTNU Colourlab, in a dimly lit room. For the uncontrolled environment, the same experiment was hosted online using Pavlovia.org [18] website, the experiment starts with general instruction, having an Ichihara test in the very beginning to test whether the observers group has a normal colour vision or not. There was a trial test taken before the experiment to get familiar with the experiment-taking procedure. Moreover, a repeat test was taken to check the repeatability of the observer, although it was not disclosed earlier to the observers to test real repeatability. Figure 1 shows the flow chart of the Psychopy experiment. Figure 1: Flowchart of the PsychoPy experiment. Figure 2 presents the experimental setup which has a colour patch in the middle surrounded by 6 anchor grey scale patches, which are referred as the categories of colour differences. Observers have the radio button options at the bottom to choose the scale of visual colour difference between 1 to 6, which is mentioned in the top left corner of the grey patches. The 6 grey patches (1 to 6) refer to the six different categories of colour differences which are: no colour difference, just a noticeable difference, very small difference, small colour difference, medium colour difference and large colour difference. These categories were introduced in [5] for another type of colour difference experiment named Two-alternative force method. After selecting the number of grey patches according to the perceived colour difference, observers need to press the space button to go to the next patch to perceive, a pause with a blank background for 500 milliseconds before moving to the next patch. Observers have the opportunity to change the chosen radio before pressing space to go forward for the next patch, but once observers move forward with the space, there is no way to get back and change or check the selected category. Besides, the observer can not move forward without selecting the patch number, which means skipping a particular sample measurement was not allowed. There will be 48 different patches arrived randomly to choose the difference, 12 of them repeated again to check the intra-observer variability, total of 60 patches compared by the observers. Figure 2: The arrangement of experimental patches. 3.1. Experiment setup for controlled environment The experiment in a controlled environment was conducted on an AdobeRGB calibrated display having 80𝑐𝑑/𝑚2 luminance and a white point of 6500k which have a gamma value of 2.2. There is a chin rest placed to maintain a distance of 50 cm from the display. There was a 15-minute warm-up time reserved after starting the monitor before starting the experiment, which ensures the highest luminance of the display. The tristimulus values of all the samples were measured using the Tele-spectroradiometer Konica Minolta CS2000, keeping a distance of 50 cm between the Tele-spectroradiometer and the display. Then using the display white point and utilizing [19], the tristimulus values were converted into CIELAB values. The display used for the experiment was an EIZO ColorEdge CG246 and the room lighting condition was dim. Due to the pandemic situation, the mouse, keyboard, and the chin rest used for the experiment were sanitized every time after completing the experiment by each observer. 3.2. Experiment setup for uncontrolled environment Pavlovia.org [18] provides the facility to host the experiment online using Psychopy. The website is a subscription-based platform that requires credits to run the experiment online. The setup experiment went through the piloting to make sure it was working fine online with the same appearance as in the lab. In an uncontrolled environment, it is a bit challenging to impose restrictions on the observers to maintain certain criteria, e.g., being in a certain viewing condition, managing a particular viewing angle, distance and consent about chin rest. Additionally, the calibration state and display type could be different for the uncontrolled environment observers. But observers were instructed to use the Google Chrome browser to get a better experiment experience and to eradicate the problems related to the appearance of the experiment setup and patches. 3.3. Observer group For the controlled environment, observers were collected using social media among the COSI master’s degree students and other sources. There were 17 observers in total, and all the observers had normal colour vision. And the uncontrolled environment observers were invited to take the experiment by sending the link through the emails, and 25 observers in total conducted the experiment. An Ichihara test took place at the beginning of the experiment in both cases to acknowledge there was no colour deficiency problem in the observer group. 3.4. Sample creation As all the observers had normal colour vision, the difference between the colour patches was chosen as a small colour difference according to the guideline of [20]. For this experiment, six colour centers were considered, and eight different variations of each colour center were created, and the colour center was also included in these variants. Variants were created considering the display gamut. A total of 60 patches were shown in two steps for the experiment to compare, 48 of them were different, and 12 patches were shown repeatedly to inspect variations and repeatability of the observer. Table 1 listed all the 6 colour centers used for the experiment and their corresponding CIELAB values. And the background used for the experiment has CILAB values L*=54, b*=0, c*=0. Figure 3 shows the colour centers listed in Table 1 and shows that all the centers were in the gamut of the controlled environments experiment setup display. Figure 3 includes the display white point as well in the middle of the 6 centers. Figure 4 shows that all the variants of different colour samples are in the gamut of the display as well. This display gamut will not be the same for the observers who took the experiment online. That varies from display to display used by the observers online. The measured size of the controlled environment colour patches was 3.6cm*3.6cm for each patch, according to the size of the object and the distance from the display, the observer angle became 4 degrees, so observers can be considered as 4-degree observers for the controlled environment. Table 1 CIELAB values for the chosen colour centers. Colour Variants L* a* b* Red 26.02 41.14 18.1 Blue 42.73 44.45 -75.23 Green 55.54 -56.49 35.21 Yellow 76.78 -8.24 44.95 Cyan 83.4 -46.79 -13.93 Magenta 56.17 90.63 -54.8 Figure 3: Colour centers (circles) and the gamut of the display profile (triangle). 4. Results and Discussion All the observers for controlled and uncontrolled environments went through the Ichihara test, and all the observers found the correct element shown in the picture, which concludes that all observers have normal colour vision. The visual scores of the observers for the uncontrolled environment group are higher than those of the controlled environment group. Table 2 shows the range of selected categories by averaging all the observer responses according to the colours and the overall average score for the colour. According to the table uncontrolled environment observer chosen the higher categories most which provides higher average scores in each category with an exception in Green. The tristimulus values of 48 colour patches, including the display white point, were measured using a Tele-spectroradiometer on a controlled environment experiment’s display. Then these XYZ tristimulus values were converted to the CIELAB colour space using the white point of the display. Afterwards, the computed colour differences for CIEDE2000, CIE94 and CIE76 were calculated using MATLAB R2021a using these CIELAB values. The computed range of the colour differences from just a noticeable difference to a large difference scale considered for the experiment is summarized in Table 3, according to the colour centers and formulas used for the result comparison. Figure 4: All the sample variations used for the experiment (circles) and the gamut of the display profile (triangle). Table 2 Average selected category and range of selected categories in different environment experiments. Controlled Uncontrolled environment environment observers observers Colour Variants Average Range Average Range Blue 3.49 1.2-5.6 3.65 1.04-5.60 Green 2.78 1.13-5.8 3.69 1.24-5.64 Red 3.38 1.06-5.73 3.84 1.12-5.68 Cyan 2.78 1.13-5.93 3.01 1-5.28 Magenta 2.2 1.06-4.26 2.99 1.08-4.76 Yellow 2.39 1.06-5.73 2.65 1.04-4.8 4.1. Coherence between calculated and visual colour difference using Plotting One of the methods utilized in earlier studies to visualize the computed data and perceived response data is plotting ΔE and ΔV which was used in [11]. For the visual response, the average of all the responses was taken and CIEDE2000, CIE94 and CIE76 were utilized to get the computed colour difference data. There was a total of 6 categories in the experiment, average categories plotted in Y axis of figure 5, figure 6 and figure 7. On the other hand, the computed data was plotted on the X-axis of the figures. There are a total of 48 blue points in all the figures to show the relationship between ΔE and ΔV. In all these three figures 5,6 and 7, the left side Table 3 The ranges of computed colour differences according to the colour centers ranges between just noticeable colour difference and the large colour difference. Colour Variants CIEDE2000 CIE94 CIE76 Red 1.33-8.49 1.49-7.92 2.44-13.41 Green 0.66-8.49 0.74-9.67 2.72-11.49 Blue 0.71- 8.68 0.78-8.58 2.90-15.08 Cyan 0.91-8.83 1.21-10.11 1.57-16.30 Magenta 0.30-3.49 0.33-3.74 0.84-6.39 Yellow 0.42-7.29 0.39-6.96 0.64-11.49 graph shows the relationship of controlled environment experiment and the right graph shows the relationship of uncontrolled environment experiments. The graphs also show a similar reflection as in Table 2 that uncontrolled environment observers perceive larger categories than the controlled environment observers. Figure 5: Relationship between perceived and calculated (CIE76) colour difference in controlled (Left) and uncontrolled (right) environment. Figure 6: Relationship between perceived and calculated (CIE94) colour difference in controlled (Left) and uncontrolled (right) environment. Figure 7: Relationship between perceived and calculated (CIEDE2000) colour difference in controlled (Left) and uncontrolled (right) environment. 4.2. Coherence between calculated and visual colour difference using STRESS Standard Residual Sum of Squares(STRESS) is the most recommended approach to finding the relationship between perceived and computed colour differences. Table 4 represents the STRESS results for both controlled and uncontrolled environments in between the CIEDE2000 colour difference formula and average perceived category response. A lower index of stress shows the better relationship, hence here the controlled environment observer achieved a lower STRESS index which represents the controlled environment observers perceived better category assumption with the formula provided colour difference. And in between the colours, green has considerably gained a lower index value, which means controlled environment observers perceive green more rationally according to the comparison with CIEDE2000. However, both environmental observers achieve a higher index for Red, which reflects that red was less rationally perceived by the observers. Table 4 Stress results of ΔE (CIEDE2000) and ΔV. Colour Controlled environment Uncontrolled environment Blue 29.65 30.95 Green 17.54 38.38 Red 37.54 43.68 Cyan 27.75 39.41 Magenta 23.21 31.03 Yellow 21.46 32.94 According to the Table 5, it shows the STRESS indexes found by using average ΔV and computed ΔE (using CIE94). Here in the controlled environment again, green achieved a better relationship according to the stress value. However, in an uncontrolled environment, blue achieved a lower index value, which provides a better relationship between the formula of CIE94 and the average perceived difference in the blue colour center. Furthermore, Table 6 shows a different relationship than the previous two formulas as CIE76 provided large numbers in computed colour differences. Here blue colour was perceived more Table 5 Stress results of ΔE (CIE94) and ΔV. Colour Controlled environment Uncontrolled environment Blue 26.5 28.85 Green 19.77 38.74 Red 28.41 35.04 Cyan 26.55 37.36 Magenta 23.21 30.54 Yellow 20.55 30.22 rationally in controlled and uncontrolled environments, which reflects STRESS value that was comparatively lower than others. Table 6 Stress results of ΔE (CIE76) and ΔV. Colour Controlled environment Uncontrolled environment Blue 18.97 24.31 Green 20.55 34.71 Red 19.81 34.01 Cyan 29.54 41.69 Magenta 30.4 32.91 Yellow 22.04 34.55 However in between the colours, cyan provides a higher STRESS index in both environment experiments with all the formulas used which is present in all three tables 4, 5 and 6. Comparing these 3 tables, CIE94 achieved a better relationship with the perceived colour differences, having lower STRESS index values most of the time, with some exceptions in some colour centers. 4.3. Inter and Intra observer variation using STRESS Inter-observer variants refer to the variability between one and other observers, which indi- cates variations between observers. But the intra-observer variability was measured by the observer’s own variation which can be found by repeating the same task twice and observing the repeatability of the observer. The inter-observer variability according to each color center was calculated using stress in table 7. Firstly, the individual observer’s variability was calculated for each colour center variant using STRESS considering each individual observer’s response and the average observer’s response. And then the mean stress of all observers was reported here in the table. The same method was applied to find the mean stress for both controlled and uncontrolled environment experiment. Whereas Table 8 shows the intra-observer variability which was calculated using repeated patch’s first attempt response and second attempt response. Primarily individual ob- server STRESS index was calculated according to the colour centers, afterwards average stress of all observers was reported in the table. Comparing the results controlled environment observers Table 7 Inter observer variation according to the color centers using STRESS. Colour Controlled environment Uncontrolled environment Cyan 15.51 17.38 Magenta 16.27 19.25 Yellow 15.56 20.47 Blue 14.88 16.60 Green 14.05 15.00 Red 14.78 15.27 Average 15.18 17.32 Table 8 Intra observer variation according to the color centers using STRESS. Colour Controlled environment Uncontrolled environment Cyan 10.07 13.65 Magenta 12.02 13.42 Yellow 8.67 11.01 Blue 5.18 6.13 Green 9.28 8.5 Red 9.8 9.2 Average 9.17 10.32 were consistent in most of the colours according to both inter and inter-observer variation with some exceptions for green and red in intra-observer variation. Observing the stress value of inter and intra-observer variability in both controlled and uncontrolled environments, the stress value difference found was less than 1 to 5 stress values. In most of the colours centers, it was between less than 1 to 2. Table 9 Overall inter and intra observer variation using STRESS. Experiment type Inter observer variability Intra observer variability Controlled environment 16.64 19.77 Uncontrolled environment 19.57 20.16 Zhongning et al[21] 32 31 Melgosa et al[17] 35.4 - Table 9 presents the overall inter and intra-observer variability. Individual observer’s STRESS values were calculated first for both intra and inter-observer variation then summarized in the table by averaging all observer’s STRESS values. In between the controlled and uncontrolled environments, controlled environments observers were more consistent according to the STRESS value. As for some colour centers (e.g. Green, Cyan) stress value have a larger difference in between controlled and uncontrolled environment experiment, the variations of such colour centers may need to choose considering smaller gamut of the display. 5. Conclusion This study shows the coherence between the perceived colour difference and computed colour difference using controlled and uncontrolled environment for the experiment. The observers in the controlled environment have more rational results according to the STRESS index rather than the uncontrolled environment’s observer in comparison with colour difference formulas and inter-intra variation, although the uncontrolled environment achieved a considerable STRESS index as well. The experiments create a new dataset for further research on the experiment conducted online which can be utilized to compare with others to find a solution on how to move the experiment online. To conduct the psychophysical experiment on large-scale observers, which will give more insightful analysis, online uncontrolled environment experiments can be promoted considering the challenges in doing the experiment online. The uncontrolled environment benefits by allowing the observer from any place, which helps to find the results from anywhere in the world, which can be useful to find the region-wise perceived result for many particular applications which can be investigated in future. Acknowledgments The authors are thankful to NTNU colour lab members for helping with all the instruments required for the experiments and especially to Gregory High, Phd candidate for his support by sharing credits to host the experiment online. The observers sacrifice their crucial time to conduct the experiment, authors are grateful to them. References [1] D. L. MacAdam, Visual sensitivities to color differences in daylight, Josa 32 (1942) 247–274. [2] A. R. Robertson, The cie 1976 color-difference formulae, Color Research & Application 2 (1977) 7–11. [3] M. R. Luo, G. Cui, B. Rigg, The development of the cie 2000 colour-difference formula: Ciede2000, Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 26 (2001) 340–350. [4] CIE Publication 116, Industrial colour-difference evaluation, CIE Central Bureau,Vienna (1995). [5] H. Wang, G. Cui, M. R. Luo, H. Xu, Evaluation of colour-difference formulae for different colour-difference magnitudes, Color Research & Application 37 (2012) 316–325. [6] R. G. Kuehni, Variability in estimation of suprathreshold small color differences, Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 34 (2009) 367–374. [7] CIE, Technical report: Improvement to industrial colordifference evaluation, CIE Publica- tion 142 (2001). [8] Q. Pan, S. Westland, Comparative evaluation of color differences between color palettes, in: Color and Imaging Conference, volume 2018, Society for Imaging Science and Technology, 2018, pp. 110–115. [9] S. Zuffi, P. Scala, C. Brambilla, G. Beretta, Web-based versus controlled environment psychophysics experiments, in: Image Quality and System Performance IV, volume 6494, SPIE, 2007, pp. 56–63. [10] R. S. Berns, D. H. Alman, L. Reniff, G. D. Snyder, M. R. Balonon-Rosen, Visual determination of suprathreshold color-difference tolerances using probit analysis, Color Research & Application 16 (1991) 297–316. [11] A. K. Kvitle, H. Oddløkken, P. Green, P. Nussbaum, Methods for psychophysical assessment of colour difference by observers with a colour vision deficiency, in: 2018 Colour and Visual Computing Symposium (CVCS), IEEE, 2018, pp. 1–6. [12] H. Xu, H. Yaguchi, Visual evaluation at scale of threshold to suprathreshold color difference, Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 30 (2005) 198–208. [13] M. R. Luo, G. Cui, C. Li, Uniform colour spaces based on ciecam02 colour appearance model, Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur 31 (2006) 320–330. [14] M. Melgosa, A. Trémeau, G. Cui, Colour difference evaluation, in: Advanced color image processing and analysis, Springer, 2013, pp. 59–79. [15] J. Liang, M. Georgoula, N. Zou, G. Cui, M. R. Luo, Colour difference evaluation using display colours, Lighting Research & Technology 50 (2018) 1257–1269. [16] M. Melgosa, R. Huertas, R. S. Berns, Performance of recent advanced color-difference formulas using the standardized residual sum of squares index, JOSA A 25 (2008) 1828–1834. [17] M. Melgosa, P. A. García, L. Gómez-Robledo, R. Shamey, D. Hinks, G. Cui, M. R. Luo, Notes on the application of the standardized residual sum of squares index for the assessment of intra-and inter-observer variability in color-difference experiments, JOSA A 28 (2011) 949–953. [18] Pavlovia, 2022. URL: https://pavlovia.org/. [19] P. J. Green, A colour engineering toolbox, 2003. [20] CIE, Recommended method for evaluating the performance of colour-difference formulae, International Commission on Illumination (2016). [21] Z. Huang, H. Xu, M. R. Luo, G. Cui, H. Feng, Assessing total differences for effective samples having variations in color, coarseness, and glint, Chinese Optics Letters 8 (2010) 717–720.