Unifying Optimization Methods for Color Filter Design

Abstract

Through optimization we can solve for a lter that when the camera views the world through this lter, it is more colorimetric. Previous work solved for the lter that best satis ed the Luther condition: the camera spectral sensitivities after ltering were approximately a linear transform from the CIE XYZ color matching functions. A more recent method optimized for the lter that maximized the Vora-Value (a measure which relates to the closeness of the vector spaces spanned by the camera sensors and human vision sensors). The optimized Luther- and Vora- lters are di erent from one another. In this paper we begin by observing that the function de ning the VoraValue is equivalent to the Luther-condition optimization if we use the orthonormal basis of the XYZ color matching functions, i.e. we linearly transform the XYZ sensitivities to a set of orthonormal basis. In this formulation, the Luther-optimization algorithm is shown to almost optimize the Vora-Value. Moreover, experiments demonstrate that the modi ed orthonormal Luther-method nds the same color lter compared to the Vora-Value lter optimization. Signi cantly, our modi ed algorithm is simpler in formulation and also converges faster than the direct Vora-Value method.

Keywords

• Color lters Design optimization Image sensors

-

A digital camera sees the world through Red, Green and Blue sensors. However, for practical considerations (including manufacturability and the need to have low image noise [ 13 ]), the RGB sensors are not linearly related to the human vision sensitivity functions [ 11 ]. A camera is said to be colorimetric if it satis es the so-called Luther condition: its spectral sensitivities are a linear combination of the CIE XYZ color matching functions (CMFs) [ 9 ].

While the Luther condition is never met exactly, the closer the spectral responses of a camera are to being linearly related to the CMFs, the better we can correct the camera colors to XYZs or to a display RGB space such as sRGB [ 3 ].

The devil is in the detail, speci cally in what we mean by `closer'. We need to quantitatively measure `closeness' in some sense. Neugebauer proposed the rst sensor quality factor referred to as the `Q-factor' [ 10 ] but this is limited to the evaluation of a single sensor. Later more quality measures have been proposed to generalize to multi-sensor devices like cameras and scanners [ 1, 12, 16 ]. However, many have the weakness that they do not incorporate linear transforms into the measure. This is a serious omission as we always correct the measured camera RGBs - apply a linear transform - to make an image suitable for display. There is an exception: the Vora-Value proposed by Vora and Trussell [ 14 ]. The Vora-Value is a carefully crafted formula that both admits the idea of linear transform and also is designed to quantify sensor performance given all theoretically possible light stimuli. The Vora-Value is the measure of `goodness' we adopt in this paper.

Related to this paper, prior work has shown how the Vora-Value can, at least in principle, be used as a criterion with respect to which we might design the spectral sensitivities in scanners [ 16 ]. However, this is a theoretical result: the optimized lters cannot be easily manufactured.

Rather than trying to make new spectral sensitivities, in this paper, we propose to make an o -the-shelf camera more colorimetric by placing a lter in front of the camera (see Fig. 1). The lter is designed so that the ltered RGBs are approximately linearly related to the reference XYZs. Previous work by Finlayson et al. [ 5 ] optimized for the lter to meet the Luther condition [ 9 ]. Speci cally, they solve for the spectral transmittance of the color lter that when multiplied by the camera sensitivity functions is almost a linear transform from the CIE color matching functions. The best lter is found via a simple and intuitive alternating least-squares (ALS) algorithm where the lter and the mapping transformation solutions are solved in turn until convergence. The resulting Luther- lter for a Canon 5D Mark II DLSR camera is shown at the bottom left of Fig. 1. Arguably, a weakness of the method is that it is tied to a single xed set of color matching functions. Indeed, if we solve for the best lter for the CIE XYZ and the sRGB matching functions (for Rec. 709 primaries [ 7 ]) then we obtain di erent lters.

Recent work solved for - using a gradient ascent algorithm - the lter that maximized the Vora-Value criterion [ 19 ]. By formulation, the Vora-Value is based on the idea of the `vector space' which can be spanned by any set of basis functions in the space. All basis functions in the same vector space are a linear transform apart from each other. Hence, for a given camera and the reference XYZ sensitivities (regarded as basis functions), the Vora-Value calculates the same score irrespective of any linear transform of the XYZ sensitivities. The Vora-Value designed lter is shown at the bottom right of Fig. 1. Interested readers are referred to [ 19 ] for more detail about the optimization.

Depending on the optimization criterion at hand, we recover di erent lters. Unsurprisingly, `scored' for the closeness to XYZ CMFs in the Lutheroptimization, the Luther- lter achieves a better t to XYZ CMFs than the Vora-Value lter. Conversely, the Vora-Value optimized lter achieves a higher Vora-Value score compared to the Luther- lter. The the Vora-Value optimization is solved by using a Gradient Ascent approach. Not only is the form of the gradient function complex but the gradient search is slow to converge. In contrast, the Luther-condition method is simple and can be solved by the ALS algorithm with fast convergence speed. This said, the algorithm for the Luther-condition approach is much simpler than the Vora-Value approach. An additional advantage of the ALS approach is that it is straightforward to extend to incorporate measured lights and re ectances [ 4, 20 ].

In this paper, we modify the simpler Luther-optimization so it can be used to optimize the Vora-Value. We revisit the formulation of the Vora-Value and show that it is more simply expressed when we map the XYZ CMFs to the corresponding orthonormal basis. The ALS method can be used to nd the lter so that the ltered camera sensitivities that are linearly related to a linear combination of XYZ CMFs (which is orthonormal). By solving the modi ed Luther condition, we can show we must also be optimizing the Vora-Value. Experiments demonstrate two important results. First, we nd the same lter using the modi ed Luther optimization and the original gradient-ascent VoraValue optimization. Second, the convergence speed is signi cantly faster using our new modi ed Luther-condition method.

The rest of the paper is organized as follows. Section 2 reviews the de nition of the Luther condition and Vora-Value, and also introduces the alternating least-squares algorithm. In Section 3, we present the modi ed Luther-condition optimization and prove its equivalence to the Vora-Value optimization. Later we show how the ALS algorithm can be used to nd the optimal lter. Experimental results are presented in Section 4. The paper concludes in Section 5. 2 2.1

Background

The Luther Condition The Luther condition states that the spectral sensitivity curves of the camera sensors are a linear combination of the CIE XYZ color matching functions (CMFs). Let Q = [r; g; b] and X = [x; y; z] denote respectively the spectral sensitivities of the camera and the CMFs of the human visual sensors. The columns of matrices Q and X represent the spectral sensitivity for each sensor channel and the rows represent the sensor responses at a sampled wavelength. Both matrices are in the size of n 3, where n is the number of sampling wavelengths across the visible spectrum (typically, n = 31 when the visible spectrum running from 400 nm to 700 nm is sampled every 10 nm).

Mathematically, the Luther condition is written as (1) (2) (3) (4) where M is a 3 3 matrix.

Equivalently we write:

X = QM

XT1 = QT2M0 where T1, T2 and M0 are full rank 3 3 matrices. Clearly, given Eq. (1), then M0 = [T2] 1MT1. That is, the Luther condition does not depend on the particular basis used (we can map camera sensitivities to XYZs, cone functions or any linear combination thereof).

If a camera satis es the Luther condition, for any two color signals, f and g that produce the same values by the camera sensors, they should also make the same XYZ stimulus values. That is,

QTf = QTg =M) XTf = XTg and thus are indistinguishable to the human observer. Readers are referred to [ 6 ] for more detail. 2.2

The Vora-Value The Vora-Value is often used to measure how similarly a camera samples the spectral world compared to the human visual system. The Vora-Value is a number between 0 and 1 where 1 means the camera is fully colorimetric such that RGBs are precisely a linear transform from XYZ tristimulus values.

Given a camera sensor set Q and the human visual sensors X, the Vora-Value is de ned [ 14 ] as (Q; X) = 1 3 trace(PfQgPfXg) where the superscripts T and inverse. 2.3

Orthonormal Basis

1 denote respectively the matrix transpose and where PfQg and PfXg denote the projection matrices of the camera spectral responses and the human visual sensitivities, respectively and trace() is the trace of a square matrix that sums up the elements along the diagonal of a matrix. The projector of a matrix - such as Q - is equal to

PfQg = Q(QTQ) 1QT (5) Let U and V respectively denote linear combinations of Q and X that are orthogonal, i.e. U = QT1 and V = XT2 where UT U = I3 = VT V (I3 is the 3 3 identity matrix).

By simple substitution into the matrix projector, we obtain

PfXg = PfVg = VVT: By using Eq. (6) for the Vora-Value de nition, we have (Q; X) = (Q; V). We will make use of the orthonormal bases in the next section. 2.4

Least-squares Correction and Filtering First, let us begin with the simple least-squares (LS) regression case. Given two matrices, e.g. X and Q (in the size of m n with m n), the best predictor of X from Q can be found in the closed-form by

M = (QTQ) 1QTX where X QM gives the least sum-of-squared errors.

Physically, the e ect of placing a transmissive lter f (a 31 1 vector) in front of a camera is written as diag(f )Q where diag() is a function that turns a vector into a diagonal matrix. To ease notation by F = diag(f ), so the lter multiplied by camera is written as FQ. For a given Q, we nd the best F row-by-row and in closed form:

Fii =

Qi Xi

Qi Qi where the subscript i denotes the ith row of a matrix and ` ' is the vector dot-product.

In the Luther-condition optimization - recapitulated below - we need to simultaneously solve for the best M and F that together map the camera sensitivities as close as possible to X as arg min k FQM

F;M

X k22 : (6) (7) (8) (9)

The Modi ed Luther-condition Optimization

We propose a simple modi cation to the Luther-condition optimization. We simply substitute the target color matching sensitivities X with a special linear transform V = XT where V is chosen to be an orthogonal matrix. Now we optimize: arg min k FQM

F; M

V k22 :

This modi ed Luther-condition optimization aims for the best lter matrix F and the 3 3 linear transform M that return the least errors between the two spectral sensitivities sets.

Given the lter matrix F, in the least-squares sense, the best M is obtained

M = ((FQ)TFQ) 1(FQ)TV Here we see that the best linear mapping M in the Luther-condition optimization of Eq. (10) is essentially a function of F.

By multiplying with FQ, we have

FQM = FQ((FQ)TFQ) 1(FQ)TV = PfFQgV Substituting into Eq. (10), the optimization can be rewritten as (10) (11) (12) (13)

I)V k22, we maximize v(FQ; X). 5. PfAg = (PfAg)T , the projector is symmetric.

Using axiom 1, we rewrite the formula in Eq. (13) as 1. trace(ATA) = jjAjj22 (remember, trace is the sum of the diagonal of a matrix) From Eq. (6), PfXg = VVT, so it follows Clearly, in Eq. (17), trace(PfXg) is a positive constant as the projector of a known matrix is a constant equal to the rank of the matrix. Therefore, we minimize the derived expression by maximizing trace(PfFQgPfXg). Thus, we can write: From the de nition of Vora-Value (FQ; V) / trace(PfFQgPfXg). We conclude: In summary, we have shown that a least-squares procedure that nds a lter that - in combination with a linear least-squares mapping (see Eqs. (11) through (13)) - that best ts an orthogonal basis of the color matching functions (i.e. minimizes the tting error) must simultaneously maximize the Vora-Value. Equivalently, maximizing the Vora-Value is the same problem of minimizing a least-squares t, for an orthogonal basis. To minimize Eq. (10), or via Theorem 1 to maximize the Vora-Value, we adopt an alternating least-squares algorithm: Algorithm 1 ALS algorithm solving for the lter optimization 1: i = 0; F 0 = diag(f initial); Q0 = F 0Q 2: repeat 3: i = i + 1 4: mMiin k F i 1QM i V k22 5: mFiin k F iQM i

V k22 6: Qi = F iQ 7: until (Qi; V ) 8: F = F i and

(Qi 1; V ) <

M = M i

Speci cally, starting from an initial lter solution F0, we rst solve for the matrix M by holding the lter F xed (see step 4) and alternatively using the ALS Gradient Ascent 0 400 450 500

550 Wavelength (nm) 600 650 700 newly calculated M to solve for the lter matrix F (see step 5) and the process will continue updating both matrices in turn until it converges to a stopping criterion (see step 7). The ALS method is guaranteed to converge (although not necessarily to the global optimum) [ 18 ]. Steps 4 and 5 - where we nd the linear transform and the lter - are solved using simple, closed-form least-squares estimation, see Eqs. (8) and (9).

In [ 19 ], the Vora-Value minimization was cast as a gradient ascent optimization. To implement that approach, it was found that the gradient step is complex in formulation and the gradient ascent converged slowly. It is important to note that the ALS and gradient ascent algorithm have the same starting optimization statement (that is what we have proved in Theorem 1). Moreover, both approaches are guaranteed to converge to a xed point. However, this does not mean that these two minimizations must converge to the same lter solution. 4

Experiments and Results

The optimal lter derived from the Luther-condition based optimization - where the target lter set is V = XT, where the columns of V are orthonormal vectors - for a Canon 5D Mark II camera [ 8 ] using Algorithm 1 is shown in Fig. 2, see the solid red line. We also plot the optimal lter of the Vora-Value formulated optimization solved by the gradient ascent algorithm developed in [ 19 ], see the dotted green line. From the gure, we can see that these two algorithms give almost the same lter solutions (except at the end of the visible spectrum).

In Table 1, we evaluate the derived lters in terms of Vora-Value. The two lter design methods are termed as Luther-ALS and Vora-GA by its optimization formulation and the corresponding algorithm. We also include the results of the native (without a lter) camera sensor as baseline results (denoted Baseline). From the table, we can see that Luther-ALS and Vora-GA deliver almost the same performance (the di erence is very small and improve signi cantly from 0.9342 of the native camera sensor set to 0.9952 of the ltered camera sensors. As a higher Vora-Value indicates greater similarity of the subspaces spanned by the sensitivities of a camera and human visual system, therefore, generally relates to more accurate color measurement [ 14 ].

Now let us evaluate the derived lters with respect to a color measurement experiment. For a collection of 102 illuminants and 1995 re etance spectra [ 2 ], we calculate the RGBs (for the native camera and the camera sensitivities after ltering) and ground-truth XYZs. The corresponding CIELAB color di erence metric Eab statistics [ 17 ] are shown in the columns 3-7 of Table 1. We can see that the two algorithms also give very close color errors (which suggests that the di erence caused by the lter transmittance di erence at the spectrum end is negligible in terms of color errors). Compared to the baseline results, we can conclude that by using such optimized lters, we can e ectively reduce the color errors by two thirds to three quarters.

Convergence: An important practical issue in assessing algorithm performance is the convergence speed. In the experiment, we evaluate the lter re nement in each iteration in terms of Vora-Value and averaged mean color error. In Figure 3, we show how the two algorithms converge in terms of iterations. In red, we show the results solved by ALS algorithm while the dotted green for that solved by gradient ascent. We see here that both algorithms converges quickly just about 20 iterations for Vora-Value (about 50 iterations for color error) and converge to the same optimal target. Comparatively, ALS algorithm converges much quicker than the gradient ascent algorithm in terms of both metrics. 5

Conclusion

Previous work has shown that by the addition of a specially designed transmittance lter, a camera can become signi cantly more colorimetric either by better satisfying the Luther condition [ 5 ] or by optimizing the Vora-Value score [ 19 ]. For a given camera, however, these two methods nd di erent lters that have di erent performances. 0.98

In this paper, we unify the lter design method by making a simple modi cation to the prior art of the Luther-condition lter design. We propose to nd the lter that together with a linear transform best maps the camera sensitivities to an orthonormalized set of the color matching functions. Most importantly, we prove that by using the orthonormal basis, the Luther-condition based optimization becomes equivalent to the Vora-Value lter optimization. The optimal lter is solved by using the simple alternating least-squares (ALS) algorithm. Signi cantly, the ALS approach converges more quickly compared to the gradient ascent algorithm previously used for the Vora- lter. Experiments validate the proposed method solved by a simpler algorithm delivering almost the same results compared to the prior art of the Vora-Value optimization.

The color lters solved from the lter design optimizations will not be easy to fabricate exactly. Future work involves improving the lter smoothness and as well as overall transmissivity which are important restrictions required by the fabrication process [ 15, 20 ].

Acknowledgment

Supported by EPSRC under Grant EP/S028730 and Apple Inc.

References

1. Alsam , A. , Hardeberg , J.Y.: Metamerset based measures of goodness for colour cameras . In: International Conference on Computer Vision and Graphics, ICCVG 2004 , Warsaw, Poland, September 2004 , Proceedings . Computational Imaging and Vision , vol. 32 , pp. 252 { 258 . Springer ( 2004 )

2. Barnard , K. , Martin , L. , Funt , B. , Coath , A. : A data set for color research . Color Research & Application 27 ( 3 ), 147 { 151 ( 2002 )

3. Berns , R.S.: Billmeyer and Saltzman's principles of color technology . John Wiley & Sons, 4 edn . ( 2019 )

4. Finlayson , G.D. , Zhu , Y. : Finding a colour lter to make a camera colorimetric by optimisation . In: International Workshop on Computational Color Imaging . pp. 53 { 62 . Springer ( 2019 )

5. Finlayson , G.D. , Zhu , Y. , Gong , H.: Using a simple colour pre- lter to make cameras more colorimetric . In: Color and Imaging Conference . pp. 182 { 186 . Society for Imaging Science and Technology ( 2018 )

6. Horn , B.K.P. : Exact reproduction of colored images . Computer Vision , Graphics, and Image Processing 26 ( 2 ), 135 { 167 ( 1984 )

7. ITU , R.: Parameter values for the hdtv standards for production and international programme exchange . Recommendation ITU-R BT. 709-5 ( 2002 )

8. Jiang , J. , Liu , D. , Gu , J. , Susstrunk, S.: What is the space of spectral sensitivity functions for digital color cameras? In: 2013 IEEE Workshop on Applications of Computer Vision (WACV) . pp. 168 { 179 . IEEE ( 2013 )

9. Luther , R.: Aus dem gebiet der farbreizmetrik . Zeitschrift Technische Physik 8 , 540 { 558 ( 1927 )

10. Neugebauer , H.: Quality factor for lters whose spectral transmittances are di erent from color mixture curves, and its application to color photography . JOSA 46 ( 10 ), 821 { 824 ( 1956 )

11. Schanda , J.: Colorimetry: understanding the CIE system . John Wiley & Sons ( 2007 )

12. Sharma , G. , Trussell , H.J.: Figures of merit for color scanners . IEEE Transactions on Image Processing 6 ( 7 ), 990 { 1001 ( 1997 )

13. Vazquez-Corral , J. , Connah , D. , Bertalm o, M.: Perceptual color characterization of cameras . Sensors 14 ( 12 ), 23205 { 23229 ( 2014 )

14. Vora , P.L. , Trussell , H.J.: Measure of goodness of a set of color-scanning lters . Journal of the Optical Society of America A 10 ( 7 ), 1499 { 1508 ( 1993 )

15. Vora , P.L. , Trussell , H.J. : Mathematical methods for the analysis of color scanning lters . IEEE Transactions on Image Processing 6 ( 2 ), 321 { 327 ( 1997 )

16. Vora , P.L. , Trussell , H.J. : Mathematical methods for the design of color scanning lters . IEEE Transactions on Image Processing 6 ( 2 ), 312 { 320 ( 1997 )

17. Wyszecki , G. , Stiles , W.S.: Color science: Concepts and Methods, Quantitative Data and Formulae . Wiley New York, 2 edn . ( 1982 )

18. Zhang , T. , Golub , G.H. : Rank-one approximation to high order tensors . SIAM Journal on Matrix Analysis and Applications 23 ( 2 ), 534 { 550 ( 2001 )

19. Zhu , Y. , Finlayson , G.D.: Designing a color lter via optimization of vora-value for making a camera more colorimetric . arXiv: 2005 . 06421 [cs .CV] ( 2020 )

20. Zhu , Y. , Finlayson , G.D.: An improved optimization method for nding a color lter to make a camera more colorimetric . In: Electronic Imaging 2020 . Society for Imaging Science and Technology ( 2020 )