How to Look at Spectral Images? A Tentative Use of Metameric Black for Spectral Image Visualisation Jean-Baptiste Thomas and Jon Yngve Hardeberg The Norwegian Colour and Visual Computing Laboratory, Norwegian University of Science and Technology, Gjøvik, Norway jean.b.thomas@ntnu.no, jon.hardeberg@ntnu.no Abstract. The number of bands of a spectral image makes its visual- isation as a traditional colour image a challenge. Several directions are investigated in the literature. The state-of-the-art solutions are all lim- ited, either due to the reduced quantity of information displayed, or to such a severe reduction in naturalness or image quality that it is hard to analyse visually. This article surveys the different attempts and investi- gates a direction that uses a pair of images rather than a single image. We use the principle of metameric black to provide a dual image for visualisation. One image is then a colorimetric image that encompasses the fundamental metamer information, the other one is based on the metameric black and contains extra information related to the spectral nature of the signal. We show that in the case of metameric samples, this visualisation is useful to provide additional information. Keywords: Spectral image visualisation · Spectral imaging · Metameric black · LabPQR. 1 Introduction Spectral imaging is more and more used in several image related fields, from remote sensing to close range imaging, and its use has led to improvements in several applications such as medicine or precision agriculture. The spectral im- agers vary in spectral resolution, but an accepted standard format of the related data is as a normalised spectral radiance or spectral reflectance image. Much effort has been put in image acquisition, but with the development of single- shot imaging systems, e.g. [20], and the definition of a fully integrated imaging pipeline, e.g. [19], having convenient and efficient ways to visualise spectral im- ages has become of major importance. Attempts to interact with spectral images are originally oriented to a pixel manipulation, e.g. visualisation of a spectrum for a specific pixel, and band- based, information-based or target-based image processing, such as [17]. Copyright c 2020 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). Colour and Visual Com- puting Symposium 2020, Gjøvik, Norway, September 16-17, 2020. 2 J.-B. Thomas and J. Y. Hardeberg One traditional way to visualise spectral images in the visible range is to use standardized colorimetry to compute or estimate a colorimetric value for each of the acquired spectra. This process results in a colour image that can follow the traditional colour imaging pipeline for visualisation. This can be done in real time by the use of GPGPU [9], and more easily today thanks to the use of web technologies [7]. There are several limitations to this approach: – Information reduction, much information disappears from the visualisation by the reduction of dimensionality, e.g. phenomenon of metamerism. – Impact of the media characteristics, as in the traditional problem of colour management and visual rendering of displayed or printed content that will require the accurate modelling of the devices [8, 10, 29]. – Impact of the illumination for real-time aspect visualisation, the colorimetric computation of colour data is dependent on the illumination, thus a white balance process is required, referred to as spectral constancy in the literature [16, 15]. Another approach is the reduction of the spectral dimensionality to three bands based on information criteria, and then a visualisation in false colours of the three bands containing the most relevant information. One can then perform band selection [21, 28, 3, 24] and image enhancement [25], trying to maximise the visibility in the resulting image. Techniques to maximise information content on three false colour bands include PCA [14] or Wavelet [27] decomposition. It is also possible to implement the fusion of several bands or information channels until convergence to a colour or a panchromatic image [18]. More re- cently, techniques that map the spectral information space to colour space re- specting the expectation of human observers were developed: manifold align- ment [23], moving least squares [22], etc. Limitations of the approaches mentioned above are the assumption of a given dimensionality to visualise the data, and the limitation in how intuitive the combination of those data is in a given colour space. For example, labelling concentrations of potatoes in blue and concentrations of cabbage in red will result in some grades of purple, yielding the difficult question of what does purple mean in terms of cabbage and potatoes? Furthermore, other information visualisation strategies for spectral images as a volume [26] or within a colour space [11] tried to escape from the image format and to orient the problem toward general information visualisation, using graphs or other hierarchical structures. In this article we propose to use a pair of images rather than one single im- age as a support to spectral image visualisation. One of these images is a colour image, based on standard colour principles, that provides an intuitive represen- tation of the scene. The other image is a false colour image, which conceptually could be computed by any of the methods above. In this specific communication, we chose to propose, as an example, to emphasise the complimentarity to colori- metric information, and developed the use of an image based on the metameric black concept. How to look at spectral images? 3 2 Metameric black Fig. 1. Illustration of the decomposition of the spectral signal (yellow plain line) into its fundamental metamer (dashed blue line) and its metameric black (semi-dashed dark line) parts. We observe that the fundamental metamer is a positive signal, metameric to the original signal, while the metameric black is having negative parts. We argue that this last component would contain interesting information to help the visualisation of spectral content. The metamerism principle is that different radiant spectral power distribu- tions will look alike to an observer under standard colorimetric conditions, since the spectra will provide the same tristimulus values. In the case of spectral imaging, its fundamental interest and advantage over conventional colour imaging comes from the additional information contained in the spectrum beyond that which can be described by colorimetry. Wyszecki proposed that a set of metamers could be described by one fundamental spectral distribution that provides the same tristimulus value, the fundamental metamer and a rest, unique to each of the metamers, having a tristimulus value of (0, 0, 0), the metameric black, i.e. lying in a space orthogonal to the space of the colour mixture functions. There are an infinity of ways to compute those components. Cohen and Kap- pauf [6] proposed a method to decompose the spectral radiance into those two components. We recall the result of the method, following their notations so it is easier to relate to their article, by setting up N as the radiant spectral power distribution, A a transformation matrix from spectral radiant power distribution to tristimulus values and At its transpose, and T the tristimulus, such as: At N = T. (1) 4 J.-B. Thomas and J. Y. Hardeberg If we refer to N∗ the fundamental metamer and B, the metameric black, we write N − N∗ = B (2) and RN = N∗ , (3) t −1 t where R = A(A A) A , an orthogonal projector. This method is referred to as Matrix R in the colour science literature [5]. Figure 1 depicts a spectrum and its decomposition in fundamental metamer and metameric black. Note that the metameric black curve is having negative values, that permit a tristimulus value of (0, 0, 0). From one spectral image of radiance, we can then compute, for every pixel, a fundamental metamer image and a metameric black image. Metameric blacks were used in numerous fields, including several colour imaging applications, such as camera calibration [2], experimental physiology of vision [32], spectroradiom- etry [30]. In spectral imaging, this is found in the literature for the specific ap- plication of dimensionality reduction, and compression of spectral image data. Of particular interest, we note the LabPQR proposal by Derhak and Rosen [12] used in both spectral colour reproduction [31], and as a colorimetric-friendly compression scheme for spectral image representation [4]. In this space, Lab is the tristimulus computation from the fundamental metamer or the spectrum, and PQR is the metameric black encoded generally as the three first compo- nents of a Principal Component Analysis on the metameric black vector. The way to compute the Eigenvectors that define PQR varies in the literature and there is no international standard of PQR to our knowledge. 3 Method and experiment We demonstrate our proposal for the problem of detecting different material components that are metameric under one specific illumination. That means that two different components will have the same colorimetric values and thus will be undifferentiated in the colour image version. In general, an application only interested in the detection of metameric patches can be easily solved by consecutive measurement under different light sources, however in the context of real-time computer vision applications, the two metameric materials can be due to a diversity of reasons and be captured by only one frame by single- shot imaging. In our scenario, we hypothesise that we have a spectral image, whose associated true-colour image exhibits some metameric objects, and thus a visualisation of the colorimetric version of this single image does not allow the viewer to distinguish between the two materials. We use the Metacows data [13] as an example. The Metacows is a set of computer rendered cows, as shown in Figure 2, for which, each cow is composed of a pair of materials, metameric under illuminant D65. The provided data are spectral reflectance data in the shape of a spectral image. We use the sub-spatial resolution data, mini-cows, for the demonstration. All data were resampled based How to look at spectral images? 5 Fig. 2. Colorimetric rendering of the cows under D65 illumination (or equivalently of the fundamental metamer under the same illuminant). The observer is the 2 degrees standard observer from the CIE. The tristimulus image is converted into an sRGB image. on linear interpolation in the spectral direction to generate 100 data points between 390 and 760 nm (steps of 3.7374 nm) so that it complies with the other data used from diverse sources (Munsell, illumination, CMFs, meta-Cows). This diversity of sources and the resampling have generated slight changes in the way the meta-cows are rendered, so they are not perfectly metameric in the end as can be seen in some cows of Figure 2. Nevertheless, they are very similar and this data are suitable for our demonstration. We compute the fundamental metamers for each pixel as described in Sec- tion 2. The result is rendered for a colorimetric rendering as shown in Figure 2. Then we compute the metameric black part, and we address the question on how to visualise this data? The question is not trivial due to the presence of negative values and on the abstract interpretation of the metameric black. One tentative answer is to use the LabPQR proposal. We computed the transform from the black metamer to PQR as the three first components of a Principal Component Analysis on a set of 1600 glossy Munsell patches spectral measurements from University of Eastern Finland in Joensuu [1]. This is one of the possibilities studied by Derhak and Rosen [12] for colorimetric applications. This is a reasonable choice, rather than for instance computing the PCA on the meta-cow data itself, that will guarantee some stability and reproducibility of our experiment, and it is also much faster. Then we projected the metameric blacks onto those components. The colour rendering is based on normalised false colour in Figure 3. Because the PQR space is not a visual encoding of RGB, we tentatively applied a gamma correction of 1.8 to improve the visibility, as shown 6 J.-B. Thomas and J. Y. Hardeberg Fig. 3. Rendering based on a linear PQR space precomputed on the 1600 Munsell Glossy samples measurements. The metameric blacks are projected on the PQR space and visualised as false colour in RGB. Fig. 4. Rendering based on PQR space with a gamma of 1.8 precomputed on the 1600 Munsell Glossy samples measurements. The metameric blacks are projected on the PQR space and visualised as false colour in RGB corrected by an ad-hoc gamma value. This helps to improve the visibility. How to look at spectral images? 7 in Figure 4. Note that the choice of 1.8 for the gamma correction is based on an ad-hoc image-enhancement considerations based on visual investigation. We also tried different white balancing or image enhancement approaches to remove the yellow colour-cast but the results were not solving the problem and there is still some work to be done to figure out a good encoding of PQR to generate a pleasant visual representation. Fig. 5. Rendering based on a positive quadratic version of the black metamers before to be processed as radiant spectra. The observer is the 2 degrees standard observer from the CIE. The black-tristimulus resulting image is converted into an sRGB image. We also propose to make the metameric blacks positive, and then to process them as normal (positive) spectra for colorimetric rendering. For that, we simply squared the values. This is also a straightforward benchmark choice. Other pos- sibilities would also be interesting to investigate, e.g. to compute the absolute value. The result is shown in Figure 5, where we can observe a good distinction between the different metameric materials. Future work should be conducted to investigate how to optimally visualise the metameric black parts. 4 Analysis and Discussion Generally, the proposed visualisation strategy is exhibiting clearly the presence of metameric materials. If this image is paired with the colorimetric image, then we do not lose the semantic natural content of the image. Figure 6 shows a typical case where the difference in materials is highlighted clearly by both the proposed methods. 8 J.-B. Thomas and J. Y. Hardeberg (a) Colour (b) PQR (c) Quadratic Fig. 6. Close example on Cow 3.4: On the left the colour image, in the middle the image based on linear PQR, and on the right the quadratic metameric black as black- tristimulus. (a) Colour (b) PQR (c) Quadratic Fig. 7. Close-up example of Cow 1.1: On the left the colour image, in the middle the image based on linear PQR, and on the right the quadratic metameric black as black-tristimulus. Figures 7 and 8 show cases where the two materials are less clearly separated. Note that for one of those PQR images, the false colour versions are different than from the large cow panels, because the normalisation of the image rendering was conducted locally. This emphasises the need for a standard transform into PQR and a specific definition of encoding colour in this case for consistent analysis. 5 Conclusion We have suggested the use of a dual image to visualise spectral images. One of this image is a natural colour image, the other one is an information based image. In our demonstration we used the metameric black to generate the sec- ond image, but any other strategy could be considered. The use of metameric black to visualise metameric material distinction is shown to be efficient, but the visualisation strategy needs to be further developed and optimised for good How to look at spectral images? 9 (a) Colour (b) PQR (c) Quadratic Fig. 8. Close-up example of Cow 3.2: On the left the colour image, in the middle the image based on linear PQR, and on the right the quadratic metameric black as black-tristimulus. performance. One future direction is to define a colour space based on PQR, where the colour difference correlates with a metamerism index. References 1. Database – Munsell Colors Glossy (Spec). http://cs.joensuu.fi/~spectral/ databases/download/munsell_spec_glossy_all.htm, accessed: 12/08/2020 2. Alsam, A., Lenz, R.: Calibrating color cameras using metameric blacks. Journal of the Optical Society of America A 24(1), 11–17 (Jan 2007) 3. Amankwah, A., Aldrich, C.: A spatial information measure method for hyper- spectral image visualization. In: 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS). pp. 4542–4545 (2015) 4. Chen, Q., Wang, L.J., Westland, S.: A study of metameric blacks for the represen- tation of spectral images. In: Recent Trends in Materials and Mechanical Engineer- ing Materials, Mechatronics and Automation. Applied Mechanics and Materials, vol. 55, pp. 1116–1121. Trans Tech Publications Ltd (6 2011) 5. Cohen, J.B., Friden, T.P.: Euclidean color space and its invariants. Proceedings of the Technical Association of the Graphic Arts pp. 411–429 (1976) 6. Cohen, J.B., Kappauf, W.E.: Metameric color stimuli, fundamental metamers, and Wyszecki’s metameric blacks. The American Journal of Psychology 95(4), 537–564 (1982) 7. Colantoni, P., Thomas, J., Hebert, M., Trémeau, A.: An online tool for displaying and processing spectral reflectance images. In: 2019 15th International Conference on Signal-Image Technology Internet-Based Systems (SITIS). pp. 725–731 (2019) 8. Colantoni, P., Thomas, J., Trémeau, A., Hardeberg, J.Y.: Web technologies enable agile color management. In: 2019 15th International Conference on Signal-Image Technology Internet-Based Systems (SITIS). pp. 303–310 (2019) 9. Colantoni, P., Thomas, J.B.: A color management process for real time color re- construction of multispectral images. In: Salberg, A.B., Hardeberg, J.Y., Jenssen, R. (eds.) Image Analysis. pp. 128–137. Springer, Berlin, Heidelberg (2009) 10 J.-B. Thomas and J. Y. Hardeberg 10. Colantoni, P., Thomas, J.B., Hardeberg, J.Y.: High-end colorimetric display char- acterization using an adaptive training set. Journal of the Society for Information Display 19(8), 520–530 (2011). https://doi.org/10.1889/JSID19.8.520 11. Colantoni, P., Thomas, J.B., Pillay, R.: Graph-based 3d visualization of color content in paintings. In: Artusi, A., Joly, M., Lucet, G., Pitzalis, D., Ribes, A. (eds.) VAST: International Symposium on Virtual Reality, Archaeology and Intel- ligent Cultural Heritage - Short and Project Papers. The Eurographics Association (2010). https://doi.org/10.2312/PE/VAST/VAST10S/025-030 12. Derhak, M., Rosen, M.: Spectral colorimetry using LabPQR: An interim connection space. Journal of Imaging Science and Technology 50(1), 53–63 (2006) 13. Fairchild, M.D., Johnson, G.M.: METACOW: A public-domain, high-resolution, fully-digital, noise-free, metameric, extended-dynamic-range, spectral test target for imaging system analysis and simulation. In: The Twelfth Color Imaging Con- ference: Color Science and Engineering Systems, Technologies, Applications, CIC 2004. pp. 239–245. IS&T - The Society for Imaging Science and Technology (2004) 14. Kang, X., Duan, P., Li, S.: Hyperspectral image visualization with edge-preserving filtering and principal component analysis. Information Fusion 57, 130–143 (2020) 15. Khan, H.A., Thomas, J.B., Hardeberg, J.Y., Laligant, O.: Spectral adaptation transform for multispectral constancy. Journal of Imaging Science and Technology 62(2), 20504–1–12 (2018) 16. Khan, H.A., Thomas, J.B., Hardeberg, J.Y., Laligant, O.: Multispectral camera as spatio-spectrophotometer under uncontrolled illumination. Optics Express 27(2), 1051–1070 (2019). https://doi.org/10.1364/OE.27.001051 17. Kim, S.J., Zhuo, S., Deng, F., Fu, C., Brown, M.: Interactive visualization of hy- perspectral images of historical documents. IEEE Transactions on Visualization and Computer Graphics 16(6), 1441–1448 (2010) 18. Kotwal, K., Chaudhuri, S.: Visualization of hyperspectral images using bilateral filtering. IEEE Transactions on Geoscience and Remote Sensing 48(5), 2308–2316 (2010) 19. Lapray, P.J., Thomas, J.B., Gouton, P.: High dynamic range spectral imag- ing pipeline for multispectral filter array cameras. Sensors 17(6), 1281 (2017). https://doi.org/10.3390/s17061281 20. Lapray, P.J., Wang, X., Thomas, J.B., Gouton, P.: Multispectral filter arrays: Re- cent advances and practical implementation. Sensors 14(11), 21626–21659 (2014) 21. Le Moan, S., Mansouri, A., Voisin, Y., Hardeberg, J.Y.: A constrained band selec- tion method based on information measures for spectral image color visualization. IEEE Transactions on Geoscience and Remote Sensing 49(12), 5104–5115 (2011) 22. Liao, D., Chen, S., Qian, Y.: Visualization of hyperspectral images using mov- ing least squares. In: 2018 24th International Conference on Pattern Recognition (ICPR). pp. 2851–2856 (2018) 23. Liao, D., Qian, Y., Zhou, J.: Visualization of hyperspectral imaging data based on manifold alignment. In: 2014 22nd International Conference on Pattern Recogni- tion. pp. 70–75 (2014) 24. Liu, D., Wang, L., Benediktsson, J.A.: Interactive multi-image colour visualization for hyperspectral imagery. International Journal of Remote Sensing 38(4), 1062– 1082 (2017). https://doi.org/10.1080/01431161.2016.1277041 25. Pardo, A., Gutiérrez-Gutiérrez, J.A., López-Higuera, J.M., Conde, O.M.: Context- free hyperspectral image enhancement for wide-field optical biomarker visualiza- tion. Biomed. Opt. Express 11(1), 133–148 (Jan 2020) How to look at spectral images? 11 26. Polder, G., van der Heijden, G.W.: Visualization of spectral images. In: Censor, Y., Ding, M. (eds.) Visualization and Optimization Techniques. vol. 4553, pp. 132 – 137. International Society for Optics and Photonics, SPIE (2001) 27. Schockling, M., Bonce, R., Gutierrez, A., Robila, S.A.: Visualization of hyperspec- tral images. In: Shen, S.S., Lewis, P.E. (eds.) Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XV. vol. 7334, pp. 715 – 726. SPIE (2009) 28. Su, H., Du, Q., Du, P.: Hyperspectral image visualization using band selection. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sens- ing 7(6), 2647–2658 (2014) 29. Thomas, J.B., Hardeberg, J.Y., Foucherot, I., Gouton, P.: The PLVC display color characterization model revisited. Color Research & Application 33(6), 449–460 (2008). https://doi.org/10.1002/col.20447 30. van Trigt, C.: Metameric blacks and estimating reflectance. Journal of the Optical Society of America A 11(3), 1003–1024 (Mar 1994) 31. Tsutsumi, S., Rosen, M.R., Berns, R.S.: Spectral color management using interim connection spaces based on spectral decomposition. Color Research & Application 33(4), 282–299 (2008) 32. Viénot, F., Brettel, H.: The Verriest Lecture: Visual properties of metameric blacks beyond cone vision. Journal of the Optical Society of America A 31(4), A38–A46 (Apr 2014)