Unifying Optimization Methods for Color Filter Design Graham Finlayson? and Yuteng Zhu* (co-first authors) University of East Anglia, Norwich NR4 7TJ, UK yuteng.zhu@uea.ac.uk, g.finlayson@uea.ac.uk Abstract Through optimization we can solve for a filter that when the camera views the world through this filter, it is more colorimetric. Previous work solved for the filter that best satisfied the Luther condition: the camera spectral sensitivities after filtering were approximately a linear transform from the CIE XYZ color matching functions. A more recent method optimized for the filter that maximized the Vora-Value (a measure which relates to the closeness of the vector spaces spanned by the camera sensors and human vision sensors). The optimized Luther- and Vora-filters are different from one another. In this paper we begin by observing that the function defining the Vora- Value is equivalent to the Luther-condition optimization if we use the orthonormal basis of the XYZ color matching functions, i.e. we linearly transform the XYZ sensitivities to a set of orthonormal basis. In this for- mulation, the Luther-optimization algorithm is shown to almost optimize the Vora-Value. Moreover, experiments demonstrate that the modified orthonormal Luther-method finds the same color filter compared to the Vora-Value filter optimization. Significantly, our modified algorithm is sim- pler in formulation and also converges faster than the direct Vora-Value method. Keywords: Color filters · Design optimization · Image sensors. 1 Introduction A digital camera sees the world through Red, Green and Blue sensors. However, for practical considerations (including manufacturability and the need to have low image noise [13]), the RGB sensors are not linearly related to the human vision sensitivity functions [11]. A camera is said to be colorimetric if it satisfies the so-called Luther condition: its spectral sensitivities are a linear combination of the CIE XYZ color matching functions (CMFs) [9]. ? co-first authors Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Colour and Visual Computing Symposium 2020, Gjøvik, Norway, September 16-17, 2020. 2 G. Finlayson and Y. Zhu Figure 1. The upper panel illustrates the general idea of placing a filter in front of a digital camera. The bottom panel shows the optimized filters solved from two filter optimization methods for a Canon 5D Mark II DSLR camera. These two optimized filters have different spectral transmittance and consequently have noticeable performance in ∗ terms of the averaged CIELAB color error ∆Eab and the Vora-Value score. While the Luther condition is never met exactly, the closer the spectral responses of a camera are to being linearly related to the CMFs, the better we can correct the camera colors to XYZs or to a display RGB space such as sRGB [3]. The devil is in the detail, specifically in what we mean by ‘closer’. We need to quantitatively measure ‘closeness’ in some sense. Neugebauer proposed the first sensor quality factor referred to as the ‘Q-factor’ [10] but this is limited to the evaluation of a single sensor. Later more quality measures have been proposed to generalize to multi-sensor devices like cameras and scanners [1, 12, 16]. However, many have the weakness that they do not incorporate linear transforms into the measure. This is a serious omission as we always correct the measured camera RGBs - apply a linear transform - to make an image suitable for display. There is an exception: the Vora-Value proposed by Vora and Trussell [14]. The Vora-Value is a carefully crafted formula that both admits the idea of linear transform and also is designed to quantify sensor performance given all theoretically possible light stimuli. The Vora-Value is the measure of ‘goodness’ we adopt in this paper. Related to this paper, prior work has shown how the Vora-Value can, at least in principle, be used as a criterion with respect to which we might design the spectral sensitivities in scanners [16]. However, this is a theoretical result: the optimized filters cannot be easily manufactured. Rather than trying to make new spectral sensitivities, in this paper, we propose to make an off-the-shelf camera more colorimetric by placing a filter in front of the camera (see Fig. 1). The filter is designed so that the filtered RGBs are approximately linearly related to the reference XYZs. Previous work by Finlayson et al. [5] optimized for the filter to meet the Luther condition [9]. Unifying Optimization Methods for Color Filter Design 3 Specifically, they solve for the spectral transmittance of the color filter that when multiplied by the camera sensitivity functions is almost a linear transform from the CIE color matching functions. The best filter is found via a simple and intuitive alternating least-squares (ALS) algorithm where the filter and the mapping transformation solutions are solved in turn until convergence. The resulting Luther-filter for a Canon 5D Mark II DLSR camera is shown at the bottom left of Fig. 1. Arguably, a weakness of the method is that it is tied to a single fixed set of color matching functions. Indeed, if we solve for the best filter for the CIE XYZ and the sRGB matching functions (for Rec. 709 primaries [7]) then we obtain different filters. Recent work solved for - using a gradient ascent algorithm - the filter that maximized the Vora-Value criterion [19]. By formulation, the Vora-Value is based on the idea of the ‘vector space’ which can be spanned by any set of basis functions in the space. All basis functions in the same vector space are a linear transform apart from each other. Hence, for a given camera and the reference XYZ sensitivities (regarded as basis functions), the Vora-Value calculates the same score irrespective of any linear transform of the XYZ sensitivities. The Vora-Value designed filter is shown at the bottom right of Fig. 1. Interested readers are referred to [19] for more detail about the optimization. Depending on the optimization criterion at hand, we recover different fil- ters. Unsurprisingly, ‘scored’ for the closeness to XYZ CMFs in the Luther- optimization, the Luther-filter achieves a better fit to XYZ CMFs than the Vora-Value filter. Conversely, the Vora-Value optimized filter achieves a higher Vora-Value score compared to the Luther-filter. The the Vora-Value optimization is solved by using a Gradient Ascent approach. Not only is the form of the gradient function complex but the gradient search is slow to converge. In contrast, the Luther-condition method is simple and can be solved by the ALS algorithm with fast convergence speed. This said, the algorithm for the Luther-condition approach is much simpler than the Vora-Value approach. An additional advan- tage of the ALS approach is that it is straightforward to extend to incorporate measured lights and reflectances [4, 20]. In this paper, we modify the simpler Luther-optimization so it can be used to optimize the Vora-Value. We revisit the formulation of the Vora-Value and show that it is more simply expressed when we map the XYZ CMFs to the corresponding orthonormal basis. The ALS method can be used to find the filter so that the filtered camera sensitivities that are linearly related to a linear combination of XYZ CMFs (which is orthonormal). By solving the modified Luther condition, we can show we must also be optimizing the Vora-Value. Experiments demonstrate two important results. First, we find the same filter using the modified Luther optimization and the original gradient-ascent Vora- Value optimization. Second, the convergence speed is significantly faster using our new modified Luther-condition method. The rest of the paper is organized as follows. Section 2 reviews the definition of the Luther condition and Vora-Value, and also introduces the alternating least-squares algorithm. In Section 3, we present the modified Luther-condition 4 G. Finlayson and Y. Zhu optimization and prove its equivalence to the Vora-Value optimization. Later we show how the ALS algorithm can be used to find the optimal filter. Experimental results are presented in Section 4. The paper concludes in Section 5. 2 Background 2.1 The Luther Condition The Luther condition states that the spectral sensitivity curves of the camera sensors are a linear combination of the CIE XYZ color matching functions (CMFs). Let Q = [r, g, b] and X = [x, y, z] denote respectively the spectral sensitivities of the camera and the CMFs of the human visual sensors. The columns of matrices Q and X represent the spectral sensitivity for each sensor channel and the rows represent the sensor responses at a sampled wavelength. Both matrices are in the size of n × 3, where n is the number of sampling wavelengths across the visible spectrum (typically, n = 31 when the visible spectrum running from 400 nm to 700 nm is sampled every 10 nm). Mathematically, the Luther condition is written as X = QM (1) where M is a 3 × 3 matrix. Equivalently we write: XT1 = QT2 M0 (2) 0 where T1 , T2 and M are full rank 3 × 3 matrices. Clearly, given Eq. (1), then M0 = [T2 ]−1 MT1 . That is, the Luther condition does not depend on the particular basis used (we can map camera sensitivities to XYZs, cone functions or any linear combination thereof). If a camera satisfies the Luther condition, for any two color signals, f and g that produce the same values by the camera sensors, they should also make the same XYZ stimulus values. That is, M QT f = QT g =⇒ XT f = XT g (3) and thus are indistinguishable to the human observer. Readers are referred to [6] for more detail. 2.2 The Vora-Value The Vora-Value is often used to measure how similarly a camera samples the spectral world compared to the human visual system. The Vora-Value is a number between 0 and 1 where 1 means the camera is fully colorimetric such that RGBs are precisely a linear transform from XYZ tristimulus values. Given a camera sensor set Q and the human visual sensors X, the Vora-Value is defined [14] as 1 ν(Q, X) = trace(P{Q}P{X}) (4) 3 Unifying Optimization Methods for Color Filter Design 5 where P{Q} and P{X} denote the projection matrices of the camera spectral responses and the human visual sensitivities, respectively and trace() is the trace of a square matrix that sums up the elements along the diagonal of a matrix. The projector of a matrix - such as Q - is equal to P{Q} = Q(QT Q)−1 QT (5) where the superscripts T and −1 denote respectively the matrix transpose and inverse. 2.3 Orthonormal Basis Let U and V respectively denote linear combinations of Q and X that are orthogonal, i.e. U = QT1 and V = XT2 where UT U = I3 = VT V (I3 is the 3 × 3 identity matrix). By simple substitution into the matrix projector, we obtain P{X} = P{V} = VVT . (6) By using Eq. (6) for the Vora-Value definition, we have ν(Q, X) = ν(Q, V). We will make use of the orthonormal bases in the next section. 2.4 Least-squares Correction and Filtering First, let us begin with the simple least-squares (LS) regression case. Given two matrices, e.g. X and Q (in the size of m × n with m ≥ n), the best predictor of X from Q can be found in the closed-form by M = (QT Q)−1 QT X (7) where X ≈ QM gives the least sum-of-squared errors. Physically, the effect of placing a transmissive filter f (a 31 × 1 vector) in front of a camera is written as diag(f )Q where diag() is a function that turns a vector into a diagonal matrix. To ease notation by F = diag(f ), so the filter multiplied by camera is written as FQ. For a given Q, we find the best F row-by-row and in closed form: Qi · X i Fii = (8) Qi · Qi where the subscript i denotes the ith row of a matrix and ‘·’ is the vector dot-product. In the Luther-condition optimization - recapitulated below - we need to simultaneously solve for the best M and F that together map the camera sensitivities as close as possible to X as arg min k FQM − X k22 . (9) F,M 6 G. Finlayson and Y. Zhu 3 The Modified Luther-condition Optimization We propose a simple modification to the Luther-condition optimization. We simply substitute the target color matching sensitivities X with a special linear transform V = XT where V is chosen to be an orthogonal matrix. Now we optimize: arg min k FQM − V k22 . (10) F, M This modified Luther-condition optimization aims for the best filter matrix F and the 3 × 3 linear transform M that return the least errors between the two spectral sensitivities sets. Given the filter matrix F, in the least-squares sense, the best M is obtained M = ((FQ)T FQ)−1 (FQ)T V (11) Here we see that the best linear mapping M in the Luther-condition optimization of Eq. (10) is essentially a function of F. By multiplying with FQ, we have FQM = FQ((FQ)T FQ)−1 (FQ)T V = P{FQ}V (12) Substituting into Eq. (10), the optimization can be rewritten as arg min k (P{FQ} − I)V k22 (13) F where I is the identity matrix. Theorem 1. By minimizing arg min k (P{FQ} − I)V k22 , we maximize v(FQ, X). F Proof: We will use the following axioms: 1. trace(AT A) = ||A||22 (remember, trace is the sum of the diagonal of a matrix) 2. trace(AB) = trace(BA) 3. trace(A + B) = trace(A) + trace(B) 4. P{A}P{A} = P{A}, idempotency of projectors 5. P{A} = (P{A})T , the projector is symmetric. Using axiom 1, we rewrite the formula in Eq. (13) as arg min k (P{FQ} − I)V k22 = arg min trace(VT (P{FQ} − I)T (P{FQ} − I)V) F F (14) Now, we carry out some algebraic manipulation of the argument in Eq. (14). Using axiom 2 we rewrite k (P{FQ} − I)V k22 = trace((P{FQ} − I)T (P{FQ} − I)VVT ) (15) Unifying Optimization Methods for Color Filter Design 7 From Eq. (6), P{X} = VVT , so it follows k (P{FQ} − I)V k22 = trace((P{FQ} − I)(P{FQ} − I)P{X}) (16) Manipulating this expression using axioms 3 through 5, we obtain k (P{FQ} − I)V k22 = −trace(P{FQ}P{X}) + trace(P{X}) (17) Clearly, in Eq. (17), trace(P{X}) is a positive constant as the projector of a known matrix is a constant equal to the rank of the matrix. Therefore, we minimize the derived expression by maximizing trace(P{FQ}P{X}). Thus, we can write: arg min k (P{FQ} − I)V k22 ≡ arg max trace(P{FQ}P{X}) (18) F F From the definition of Vora-Value ν(FQ, V) ∝ trace(P{FQ}P{X}). We conclude: arg min k (P{FQ} − I)V k22 ≡ arg max ν(FQ, V) (19) F F  In summary, we have shown that a least-squares procedure that finds a filter that - in combination with a linear least-squares mapping (see Eqs. (11) through (13)) - that best fits an orthogonal basis of the color matching functions (i.e. minimizes the fitting error) must simultaneously maximize the Vora-Value. Equivalently, maximizing the Vora-Value is the same problem of minimizing a least-squares fit, for an orthogonal basis. 3.1 Alternating Least-Squares Algorithm To minimize Eq. (10), or via Theorem 1 to maximize the Vora-Value, we adopt an alternating least-squares algorithm: Algorithm 1 ALS algorithm solving for the filter optimization 1: i = 0, F 0 = diag(f initial ), Q0 = F 0 Q 2: repeat 3: i=i+1 4: min k F i−1 QM i − V k22 Mi 5: min k F i QM i − V k22 Fi 6: Qi = F i Q 7: until ν(Qi , V ) − ν(Qi−1 , V ) <  8: F = F i and M = M i Specifically, starting from an initial filter solution F0 , we first solve for the matrix M by holding the filter F fixed (see step 4) and alternatively using the 8 G. Finlayson and Y. Zhu 100 ALS Gradient Ascent % Spectral transmittance 80 60 40 20 0 400 450 500 550 600 650 700 Wavelength (nm) Figure 2. Spectral transmittance of the optimized filters for Canon 5D Mark II camera. The optimized filter in solid red line is solved by ALS algorithm while the filter in dotted green line is solved by the gradient ascent algorithm. newly calculated M to solve for the filter matrix F (see step 5) and the process will continue updating both matrices in turn until it converges to a stopping criterion (see step 7). The ALS method is guaranteed to converge (although not necessarily to the global optimum) [18]. Steps 4 and 5 - where we find the linear transform and the filter - are solved using simple, closed-form least-squares estimation, see Eqs. (8) and (9). In [19], the Vora-Value minimization was cast as a gradient ascent optimization. To implement that approach, it was found that the gradient step is complex in formulation and the gradient ascent converged slowly. It is important to note that the ALS and gradient ascent algorithm have the same starting optimization statement (that is what we have proved in Theorem 1). Moreover, both approaches are guaranteed to converge to a fixed point. However, this does not mean that these two minimizations must converge to the same filter solution. 4 Experiments and Results The optimal filter derived from the Luther-condition based optimization - where the target filter set is V = XT, where the columns of V are orthonormal vectors - for a Canon 5D Mark II camera [8] using Algorithm 1 is shown in Fig. 2, see the solid red line. We also plot the optimal filter of the Vora-Value formulated optimization solved by the gradient ascent algorithm developed in [19], see the dotted green line. From the figure, we can see that these two algorithms give almost the same filter solutions (except at the end of the visible spectrum). In Table 1, we evaluate the derived filters in terms of Vora-Value. The two filter design methods are termed as Luther-ALS and Vora-GA by its optimization formulation and the corresponding algorithm. We also include the results of the native (without a filter) camera sensor as baseline results (denoted Baseline). Unifying Optimization Methods for Color Filter Design 9 Table 1. Error statistics for Canon 5D Mark II camera ∗ Vora- color errors ∆Eab Method Value mean median 95% 99% max Baseline 0.9342 1.416 0.836 4.262 11.786 29.392 Luther-ALS 0.9952 0.298 0.147 0.983 2.472 10.127 Vora-GA 0.9952 0.300 0.149 0.986 2.472 10.027 From the table, we can see that Luther-ALS and Vora-GA deliver almost the same performance (the difference is very small and improve significantly from 0.9342 of the native camera sensor set to 0.9952 of the filtered camera sensors. As a higher Vora-Value indicates greater similarity of the subspaces spanned by the sensitivities of a camera and human visual system, therefore, generally relates to more accurate color measurement [14]. Now let us evaluate the derived filters with respect to a color measurement experiment. For a collection of 102 illuminants and 1995 refletance spectra [2], we calculate the RGBs (for the native camera and the camera sensitivities after filtering) and ground-truth XYZs. The corresponding CIELAB color difference ∗ metric ∆Eab statistics [17] are shown in the columns 3-7 of Table 1. We can see that the two algorithms also give very close color errors (which suggests that the difference caused by the filter transmittance difference at the spectrum end is negligible in terms of color errors). Compared to the baseline results, we can conclude that by using such optimized filters, we can effectively reduce the color errors by two thirds to three quarters. Convergence: An important practical issue in assessing algorithm performance is the convergence speed. In the experiment, we evaluate the filter refinement in each iteration in terms of Vora-Value and averaged mean color error. In Figure 3, we show how the two algorithms converge in terms of iterations. In red, we show the results solved by ALS algorithm while the dotted green for that solved by gradient ascent. We see here that both algorithms converges quickly just about 20 iterations for Vora-Value (about 50 iterations for color error) and converge to the same optimal target. Comparatively, ALS algorithm converges much quicker than the gradient ascent algorithm in terms of both metrics. 5 Conclusion Previous work has shown that by the addition of a specially designed transmit- tance filter, a camera can become significantly more colorimetric either by better satisfying the Luther condition [5] or by optimizing the Vora-Value score [19]. For a given camera, however, these two methods find different filters that have different performances. 10 G. Finlayson and Y. Zhu 1 0.98 ALS Gradient Ascent Vora-Value 0.96 0.94 0.92 0.9 0 5 10 15 20 iterations (a) Vora-Values ALS Gradient Ascent 1 E mean 0.5 0 0 10 20 30 40 50 iterations ∗ (b) mean ∆Eab ∗ Figure 3. Algorithm convergence in terms of Vora-Values and mean ∆Eab with respect to iterations. Solid red line represents the ALS algorithm and the dotted green line represents the gradient ascent algorithm. In this paper, we unify the filter design method by making a simple modifica- tion to the prior art of the Luther-condition filter design. We propose to find the filter that together with a linear transform best maps the camera sensitivities to an orthonormalized set of the color matching functions. Most importantly, we prove that by using the orthonormal basis, the Luther-condition based opti- mization becomes equivalent to the Vora-Value filter optimization. The optimal filter is solved by using the simple alternating least-squares (ALS) algorithm. Significantly, the ALS approach converges more quickly compared to the gradient ascent algorithm previously used for the Vora-filter. Experiments validate the proposed method solved by a simpler algorithm delivering almost the same results compared to the prior art of the Vora-Value optimization. The color filters solved from the filter design optimizations will not be easy to fabricate exactly. Future work involves improving the filter smoothness and as well as overall transmissivity which are important restrictions required by the fabrication process [15, 20]. Unifying Optimization Methods for Color Filter Design 11 Acknowledgment Supported by EPSRC under Grant EP/S028730 and Apple Inc. References 1. Alsam, A., Hardeberg, J.Y.: Metamerset based measures of goodness for colour cameras. In: International Conference on Computer Vision and Graphics, ICCVG 2004, Warsaw, Poland, September 2004, Proceedings. Computational Imaging and Vision, vol. 32, pp. 252–258. Springer (2004) 2. Barnard, K., Martin, L., Funt, B., Coath, A.: A data set for color research. Color Research & Application 27(3), 147–151 (2002) 3. Berns, R.S.: Billmeyer and Saltzman’s principles of color technology. John Wiley & Sons, 4 edn. (2019) 4. Finlayson, G.D., Zhu, Y.: Finding a colour filter to make a camera colorimetric by optimisation. In: International Workshop on Computational Color Imaging. pp. 53–62. Springer (2019) 5. Finlayson, G.D., Zhu, Y., Gong, H.: Using a simple colour pre-filter to make cameras more colorimetric. In: Color and Imaging Conference. pp. 182–186. Society for Imaging Science and Technology (2018) 6. Horn, B.K.P.: Exact reproduction of colored images. Computer Vision, Graphics, and Image Processing 26(2), 135 – 167 (1984) 7. ITU, R.: Parameter values for the hdtv standards for production and international programme exchange. Recommendation ITU-R BT. 709-5 (2002) 8. Jiang, J., Liu, D., Gu, J., Süsstrunk, S.: What is the space of spectral sensitivity functions for digital color cameras? In: 2013 IEEE Workshop on Applications of Computer Vision (WACV). pp. 168–179. IEEE (2013) 9. Luther, R.: Aus dem gebiet der farbreizmetrik. Zeitschrift Technische Physik 8, 540–558 (1927) 10. Neugebauer, H.: Quality factor for filters whose spectral transmittances are different from color mixture curves, and its application to color photography. JOSA 46(10), 821–824 (1956) 11. Schanda, J.: Colorimetry: understanding the CIE system. John Wiley & Sons (2007) 12. Sharma, G., Trussell, H.J.: Figures of merit for color scanners. IEEE Transactions on Image Processing 6(7), 990–1001 (1997) 13. Vazquez-Corral, J., Connah, D., Bertalmı́o, M.: Perceptual color characterization of cameras. Sensors 14(12), 23205–23229 (2014) 14. Vora, P.L., Trussell, H.J.: Measure of goodness of a set of color-scanning filters. Journal of the Optical Society of America A 10(7), 1499–1508 (1993) 15. Vora, P.L., Trussell, H.J.: Mathematical methods for the analysis of color scanning filters. IEEE Transactions on Image Processing 6(2), 321–327 (1997) 16. Vora, P.L., Trussell, H.J.: Mathematical methods for the design of color scanning filters. IEEE Transactions on Image Processing 6(2), 312–320 (1997) 17. Wyszecki, G., Stiles, W.S.: Color science: Concepts and Methods, Quantitative Data and Formulae. Wiley New York, 2 edn. (1982) 18. Zhang, T., Golub, G.H.: Rank-one approximation to high order tensors. SIAM Journal on Matrix Analysis and Applications 23(2), 534–550 (2001) 19. Zhu, Y., Finlayson, G.D.: Designing a color filter via optimization of vora-value for making a camera more colorimetric. arXiv:2005.06421 [cs.CV] (2020) 12 G. Finlayson and Y. Zhu 20. Zhu, Y., Finlayson, G.D.: An improved optimization method for finding a color filter to make a camera more colorimetric. In: Electronic Imaging 2020. Society for Imaging Science and Technology (2020)