When studying human color perception using displays, accurate and reproducible presentation of color stimuli is paramount, which requires color characterization of the display. Often simple (non-linear) models, such as the Gain-Ofset-Gamma (GOG) model, or low-resolution look-up tables (LUT), both based on a limited number of optical measurements, work well enough. However, some displays show a much more complex relationship between RGB input and color output, requiring more complex models or high-resolution LUTs based on a large number of measurements. In this paper, as the first step in a larger study, the feasibility and performance of neural networks (NN) for color characterization of a simulation display following a simple GOG model are explored and compared to a LUT-based method. Statistical analysis showed that the NN-method performed significantly better than the LUT model in terms of predicting the required RGB device input values that generate a set of target (device output) XYZ tristimulus values. In fact, to achieve the same accuracy, the number of training points can be substantially reduced for the NN-method compared to the LUT-method. Furthermore, the neural network is also more than 20 times faster than the look-up table in performing display characterization.
Color characterization is the process of converting a device-dependent color space to a
deviceindependent space like the XYZ or LMS. The necessity of characterization arises from the
primaries that a device uses to produce colors. As the primaries are diferent in each device,
these intrinsic diferences give rise to device-dependent color spaces [
The problem of characterization can be treated as finding the relation between two sets of points in R3. A function is to be determined from XYZ color space to RGB color space so that one can determine which RGB values should be sent to the headset to display the correct XYZ output. For instance, to achieve a particular output (1 = 19.2, 1 = 21, 1 = 73.2), the fitted function would predict (1, 1, 1) = (19.2, 21, 73.2). This triplet (1, 1, 1) can then be sent to the device to obtain the output as (1, 1, 1).
This function can be determined in a variety of ways, for instance, using a Gain-Ofset-Gamma
(GOG) model or look-up tables [
The Gain-Ofset-Gamma (GOG) model is one of the several ways of characterizing displays.
For an overview of other methods, one can refer to Brainard et. al. [
The first stage, which is also a forward transformation from digital to linearized RGB values,
is represented by a simple gamma transform [
After linearization, it is now possible to represent the device-independent (X,Y,Z) tristimulus values as a linear combination of the device-dependent linearized (R,G,B) values using a 3 x 3 matrix multiplication:
where T is the transformation matrix from linear RGB to tristimulus XYZ values. If one measures the tristimulus values for maximum red, maximum green, and maximum blue, the equation for T is given by equation (5).
= = = ⎡⎤ ⎡⎤ ⎣ ⎦ = ⎣⎦
Although it is typically more accurate to determine the 9 matrix coeficients based on a least-squares minimization algorithm using a suficiently large number of measured (RGB, XYZ) pairs.
If we also consider the minimal ambient light, the full equation becomes: ⎡⎤ ⎡⎤ ⎣ ⎦ − ⎣ ⎦
⎡⎤ = ⎣⎦
It represents the fact that even for R=G=B=0 (no display output), still some non-negative XYZ values might be measured due to the ambient light. This is also sometimes referred to as display flare.
The whole characterization process is explained in Figure 1.
A look-up table (LUT) is a dictionary that allows one to search for a value corresponding to a key.
Look-up tables have been used historically since the time of Babylonian mathematics when they
used them to calculate logarithmic tables [
When converting from device (, , ) to (, , ) using a fully populated LUT (i.e. one which has a pair for every possible r,g,b combination), the conversion process is reduced to a simple look-up. However, given that there would be 16777216(= 2563) entries loading the LUT into memory, and especially the look-up process might be quite time consuming. The actual measurement time required to generate the LUT would also be huge. Look-up tables are therefore usually determined by measuring pairs for only certain (, , ) combinations, whereby intermediate values are interpolated. Interpolation would almost certainly also be required for the inverse conversion, i.e. when going from device-independent (, , ) to device-dependent (, , ), as its highly unlikely that the desired (, , ) is one of the values in the LUT!
This paper presents a neural networks based method to address the problem of accurate color
characterization for displays. Cheung et. al. used neural networks and polynomial functions for
camera characterization and both methods were found capable of producing similar results [
Neural networks were used by Climent et. al. to test the accuracy of LCD displays [
Statistical techniques like Linear Color Correction (LCC) [14], polynomial regression [14] and geometric methods like 3D thin-plate spline [15] are some other methods used for colorimetric characterization of displays but methods like LCC can map linearized RGB to XYZ with high errors [14]. This paper used a feedforward network to perform display color characterization. In a feedforward neural network, the propagation of information is only in the forward direction and there are no cycles or loops. An instance of a feedforward network is shown in Figure 3.
A multilayer feedforward network can approximate any continuous function with just one
hidden layer. This is referred to as the universal approximation theorem[
For the purpose of this research, an Alienware m15 laptop with 16GB RAM, 64 bit Windows operating system, and Intel(R) Core(TM) i7-8750H CPU at 2.20 GHz was used. The simulation was done in Python using the Luxpy library [16].
A virtual display was simulated using a Gain Ofset Gamma model to generate ≡ pairs. It has been assumed that the display is perfectly characterized by the gain ofset gamma model (no measurement errors). Several data sets, composed of (, , ) − (, , ) pairs, were simulated to train the neural network, generate the look-up table, and test both methods. Both methods were always trained (for the LUT-based method this means generation of the LUT) and tested using the same data sets to ensure a fair comparison of their performance. 3.1.1. Sampling of RGB space The first step of data generation for the training set included the sampling of RGB space into equal voxels while ensuring that the axes of the RGB color space, and particularly the maximum value (255), were also sampled. The sampled (, , ) data points are represented by the set 3: = { : ∈ R & |} ∪ {255}, 6 ≤ ≤ 20 (7)
The red, green and blue axes are sampled uniformly due to the fact that equal sampling is the baseline or the simplest approach and it helps in representing non-additivity better. If any of the red, green or, blue channels contribute to the non-additivity, then the equal sampling makes sure an equal impact of all the channels is present. An example of the sampling with = 20 is shown in Figure 4. Here, n is called the LUT increment, and the data set composed of the sampled RGB space is henceforth referred to as the RGB cube.
RGB cubes for neural network training and LUT generation were created for LUT
increments of: [
The (r,g,b) values for the test set were generated using Numpy’s random generator function. The same test set was used in combination with all training sets and was composed of 10000 points. To complete the training and test sets and generate (r,g,b)-(X,Y,Z) pairs, their (r,g,b) values were converted into (X,Y,Z) tristimulus values using the GOG model.
The ≡ training data generated using the simulated algorithm are fed to a multilayer perceptron (coded with the Perceptron function from scikit-learn sklearn.linear_model class [17]) with the following parameters: • hidden layer = [number of hidden layers] • relu activation function ( () = (0, )) • maximum iteration = 100000 • adaptive learning rate: adam optimizer
The number of hidden layers is varied from the set [
The points in RGB cubes for each LUT increments of: [
For the LUT, predicting the value of input involved a few steps. First for finding the value
of the new element, one needs to find the nearest neighbors and assign a value based on the
values of the nearest neighbor from the table. This requires querying the LUT for the nearest
neighbors. The number of neighbors considered for diferent experiments ranged in the set
[
The predicted value for the input is calculated using a weighted average of the values of the nearest neighbors. The weights are calculated using the inverse squared distances of the query input point from the nearest neighbors. So, if a key is closer in distance to the input query point, it contributes more to determining the output value.
The predictions using the same test set are carried out using both the neural network and LUT and the results are presented in the next section.
A GOG model was used as the perfect model to generate the ground truth (RGB, XYZ) pairs. The forward model to go from display (, , ) to linear (, , ) is a simple Gamma function.
The transformation matrix given in Eq. (5) and (6) allows conversion from RGB to XYZ and vice versa. The forward transformation matrix (M) to convert linear RGB to XYZ and the reverse transformation matrix (N) to convert XYZ back to linear RGB are calculated and given below.
Error maps are generated both for the Neural Network method and the LUT-based method to compare their performances. For both methods, the LUT increments were kept consistent. Errors were calculated for the Neural Network method at various numbers of hidden layers and the LUT-based method at various numbers of nearest neighbors. The error maps are shown in Figure 6. The neural network performs extremely well with LUT increments of less than 16 and at a number of hidden layers more than 100. Even at finer resolution with a lower LUT increment, the performance of the LUT-based method is worse compared to the Neural Network based method.
Considering the error graph in Figure 6., there are recommendations for the number of hidden layers of the neural network where it performs substantially better than the LUT-based method.
In this paper, a comparison of the performance of display color characterization methods based on a neural network and a look-up table was carried out. The analysis showed that on average the neural network was substantially better than the look-up table. Prediction time was also better for the neural network which provided over 22 times speed-up compared to the look-up table method.
In the future, we propose to test the neural network and look-up-table methods on real measured displays, some of which that have more complex relationships between the RGB input values and the measured XYZ tristimulus output, due to, for example, additivity failure. Using a look-up table for high resolution images becomes computationally expensive and a neural network might significantly help in obtaining better real-time predictions, while also ofering the possibility of reducing the number of characterization measurements for the same accuracy. [13] D. H. Brainard, Calibration of a computer controlled color monitor, Color Research &
Application 14 (1989) 23–34. [14] G. D. Finlayson, M. Mackiewicz, A. Hurlbert, Color correction using root-polynomial regression, IEEE Transactions on Image Processing 24 (2015) 1460–1470. [15] P. Menesatti, C. Angelini, F. Pallottino, F. Antonucci, J. Aguzzi, C. Costa, Rgb color calibration for quantitative image analysis: The “3d thin-plate spline” warping approach, Sensors 12 (2012) 7063–7079. [16] K. A. Smet, Tutorial: The luxpy python toolbox for lighting and color science, Leukos 16 (2020) 179–201. [17] Sklearn.linear_model.perceptron, https://scikit-learn.org/stable/modules/generated/ sklearn.linear_model.Perceptron.html, 2022. Accessed: 2022-10-13. [18] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014). [19] scipy.spatial.ckdtree, https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.
cKDTree.html, 2022. Accessed: 2022-10-13. [20] J. Zhang, Y. Meuret, X. Wang, K. A. Smet, Improved and robust spectral reflectance estimation, Leukos 17 (2021) 359–379.