How to Look at Spectral Images? A Tentative Use of Metameric Black for Spectral Image Visualisation

Abstract

The number of bands of a spectral image makes its visualisation as a traditional colour image a challenge. Several directions are investigated in the literature. The state-of-the-art solutions are all limited, either due to the reduced quantity of information displayed, or to such a severe reduction in naturalness or image quality that it is hard to analyse visually. This article surveys the di erent attempts and investigates a direction that uses a pair of images rather than a single image. We use the principle of metameric black to provide a dual image for visualisation. One image is then a colorimetric image that encompasses the fundamental metamer information, the other one is based on the metameric black and contains extra information related to the spectral nature of the signal. We show that in the case of metameric samples, this visualisation is useful to provide additional information.

Keywords

• Spectral image visualisation Spectral imaging Metameric black LabPQR

-

Spectral imaging is more and more used in several image related elds, from remote sensing to close range imaging, and its use has led to improvements in several applications such as medicine or precision agriculture. The spectral imagers vary in spectral resolution, but an accepted standard format of the related data is as a normalised spectral radiance or spectral re ectance image. Much e ort has been put in image acquisition, but with the development of singleshot imaging systems, e.g. [ 20 ], and the de nition of a fully integrated imaging pipeline, e.g. [ 19 ], having convenient and e cient ways to visualise spectral images has become of major importance.

Attempts to interact with spectral images are originally oriented to a pixel manipulation, e.g. visualisation of a spectrum for a speci c pixel, and bandbased, information-based or target-based image processing, such as [ 17 ].

One traditional way to visualise spectral images in the visible range is to use standardized colorimetry to compute or estimate a colorimetric value for each of the acquired spectra. This process results in a colour image that can follow the traditional colour imaging pipeline for visualisation. This can be done in real time by the use of GPGPU [ 9 ], and more easily today thanks to the use of web technologies [ 7 ]. There are several limitations to this approach: { Information reduction, much information disappears from the visualisation by the reduction of dimensionality, e.g. phenomenon of metamerism. { Impact of the media characteristics, as in the traditional problem of colour management and visual rendering of displayed or printed content that will require the accurate modelling of the devices [ 8, 10, 29 ]. { Impact of the illumination for real-time aspect visualisation, the colorimetric computation of colour data is dependent on the illumination, thus a white balance process is required, referred to as spectral constancy in the literature [ 16, 15 ].

Another approach is the reduction of the spectral dimensionality to three bands based on information criteria, and then a visualisation in false colours of the three bands containing the most relevant information. One can then perform band selection [ 21, 28, 3, 24 ] and image enhancement [ 25 ], trying to maximise the visibility in the resulting image. Techniques to maximise information content on three false colour bands include PCA [ 14 ] or Wavelet [ 27 ] decomposition.

It is also possible to implement the fusion of several bands or information channels until convergence to a colour or a panchromatic image [ 18 ]. More recently, techniques that map the spectral information space to colour space respecting the expectation of human observers were developed: manifold alignment [ 23 ], moving least squares [ 22 ], etc.

Limitations of the approaches mentioned above are the assumption of a given dimensionality to visualise the data, and the limitation in how intuitive the combination of those data is in a given colour space. For example, labelling concentrations of potatoes in blue and concentrations of cabbage in red will result in some grades of purple, yielding the di cult question of what does purple mean in terms of cabbage and potatoes?

Furthermore, other information visualisation strategies for spectral images as a volume [ 26 ] or within a colour space [ 11 ] tried to escape from the image format and to orient the problem toward general information visualisation, using graphs or other hierarchical structures.

In this article we propose to use a pair of images rather than one single image as a support to spectral image visualisation. One of these images is a colour image, based on standard colour principles, that provides an intuitive representation of the scene. The other image is a false colour image, which conceptually could be computed by any of the methods above. In this speci c communication, we chose to propose, as an example, to emphasise the complimentarity to colorimetric information, and developed the use of an image based on the metameric black concept. 2

Metameric black

The metamerism principle is that di erent radiant spectral power distributions will look alike to an observer under standard colorimetric conditions, since the spectra will provide the same tristimulus values.

In the case of spectral imaging, its fundamental interest and advantage over conventional colour imaging comes from the additional information contained in the spectrum beyond that which can be described by colorimetry. Wyszecki proposed that a set of metamers could be described by one fundamental spectral distribution that provides the same tristimulus value, the fundamental metamer and a rest, unique to each of the metamers, having a tristimulus value of (0; 0; 0), the metameric black, i.e. lying in a space orthogonal to the space of the colour mixture functions.

There are an in nity of ways to compute those components. Cohen and Kappauf [ 6 ] proposed a method to decompose the spectral radiance into those two components. We recall the result of the method, following their notations so it is easier to relate to their article, by setting up N as the radiant spectral power distribution, A a transformation matrix from spectral radiant power distribution to tristimulus values and At its transpose, and T the tristimulus, such as:

If we refer to N the fundamental metamer and B, the metameric black, we write

N

N

= B RN = N ; (2) (3) and where R = A(AtA) 1At, an orthogonal projector. This method is referred to as Matrix R in the colour science literature [ 5 ]. Figure 1 depicts a spectrum and its decomposition in fundamental metamer and metameric black. Note that the metameric black curve is having negative values, that permit a tristimulus value of (0; 0; 0).

From one spectral image of radiance, we can then compute, for every pixel, a fundamental metamer image and a metameric black image. Metameric blacks were used in numerous elds, including several colour imaging applications, such as camera calibration [ 2 ], experimental physiology of vision [ 32 ], spectroradiometry [ 30 ]. In spectral imaging, this is found in the literature for the speci c application of dimensionality reduction, and compression of spectral image data. Of particular interest, we note the LabPQR proposal by Derhak and Rosen [ 12 ] used in both spectral colour reproduction [ 31 ], and as a colorimetric-friendly compression scheme for spectral image representation [ 4 ]. In this space, Lab is the tristimulus computation from the fundamental metamer or the spectrum, and PQR is the metameric black encoded generally as the three rst components of a Principal Component Analysis on the metameric black vector. The way to compute the Eigenvectors that de ne PQR varies in the literature and there is no international standard of PQR to our knowledge. 3

Method and experiment

We demonstrate our proposal for the problem of detecting di erent material components that are metameric under one speci c illumination. That means that two di erent components will have the same colorimetric values and thus will be undi erentiated in the colour image version. In general, an application only interested in the detection of metameric patches can be easily solved by consecutive measurement under di erent light sources, however in the context of real-time computer vision applications, the two metameric materials can be due to a diversity of reasons and be captured by only one frame by singleshot imaging. In our scenario, we hypothesise that we have a spectral image, whose associated true-colour image exhibits some metameric objects, and thus a visualisation of the colorimetric version of this single image does not allow the viewer to distinguish between the two materials.

We use the Metacows data [ 13 ] as an example. The Metacows is a set of computer rendered cows, as shown in Figure 2, for which, each cow is composed of a pair of materials, metameric under illuminant D65. The provided data are spectral re ectance data in the shape of a spectral image. We use the sub-spatial resolution data, mini-cows, for the demonstration. All data were resampled based on linear interpolation in the spectral direction to generate 100 data points between 390 and 760 nm (steps of 3.7374 nm) so that it complies with the other data used from diverse sources (Munsell, illumination, CMFs, meta-Cows). This diversity of sources and the resampling have generated slight changes in the way the meta-cows are rendered, so they are not perfectly metameric in the end as can be seen in some cows of Figure 2. Nevertheless, they are very similar and this data are suitable for our demonstration.

We compute the fundamental metamers for each pixel as described in Section 2. The result is rendered for a colorimetric rendering as shown in Figure 2. Then we compute the metameric black part, and we address the question on how to visualise this data? The question is not trivial due to the presence of negative values and on the abstract interpretation of the metameric black.

One tentative answer is to use the LabPQR proposal. We computed the transform from the black metamer to PQR as the three rst components of a Principal Component Analysis on a set of 1600 glossy Munsell patches spectral measurements from University of Eastern Finland in Joensuu [ 1 ]. This is one of the possibilities studied by Derhak and Rosen [ 12 ] for colorimetric applications. This is a reasonable choice, rather than for instance computing the PCA on the meta-cow data itself, that will guarantee some stability and reproducibility of our experiment, and it is also much faster. Then we projected the metameric blacks onto those components. The colour rendering is based on normalised false colour in Figure 3. Because the PQR space is not a visual encoding of RGB, we tentatively applied a gamma correction of 1.8 to improve the visibility, as shown in Figure 4. Note that the choice of 1.8 for the gamma correction is based on an ad-hoc image-enhancement considerations based on visual investigation. We also tried di erent white balancing or image enhancement approaches to remove the yellow colour-cast but the results were not solving the problem and there is still some work to be done to gure out a good encoding of PQR to generate a pleasant visual representation.

We also propose to make the metameric blacks positive, and then to process them as normal (positive) spectra for colorimetric rendering. For that, we simply squared the values. This is also a straightforward benchmark choice. Other possibilities would also be interesting to investigate, e.g. to compute the absolute value. The result is shown in Figure 5, where we can observe a good distinction between the di erent metameric materials. Future work should be conducted to investigate how to optimally visualise the metameric black parts. 4

Analysis and Discussion

Generally, the proposed visualisation strategy is exhibiting clearly the presence of metameric materials. If this image is paired with the colorimetric image, then we do not lose the semantic natural content of the image.

Figure 6 shows a typical case where the di erence in materials is highlighted clearly by both the proposed methods. (a) Colour We have suggested the use of a dual image to visualise spectral images. One of this image is a natural colour image, the other one is an information based image. In our demonstration we used the metameric black to generate the second image, but any other strategy could be considered. The use of metameric black to visualise metameric material distinction is shown to be e cient, but the visualisation strategy needs to be further developed and optimised for good (a) Colour performance. One future direction is to de ne a colour space based on PQR, where the colour di erence correlates with a metamerism index.

References

1. Database { Munsell Colors Glossy (Spec) . http://cs.joensuu.fi/~spectral/ databases/download/munsell_spec_glossy_all.htm, accessed: 12 /08/2020

2. Alsam , A. , Lenz , R.: Calibrating color cameras using metameric blacks . Journal of the Optical Society of America A 24 ( 1 ), 11 {17 (Jan 2007 )

3. Amankwah , A. , Aldrich , C. : A spatial information measure method for hyperspectral image visualization . In: 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) . pp. 4542 { 4545 ( 2015 )

4. Chen , Q. , Wang , L.J. , Westland , S.: A study of metameric blacks for the representation of spectral images . In: Recent Trends in Materials and Mechanical Engineering Materials, Mechatronics and Automation. Applied Mechanics and Materials , vol. 55 , pp. 1116 { 1121 . Trans Tech Publications Ltd (6 2011 )

5. Cohen , J.B. , Friden , T.P. : Euclidean color space and its invariants . Proceedings of the Technical Association of the Graphic Arts pp. 411 { 429 ( 1976 )

6. Cohen , J.B. , Kappauf , W.E.: Metameric color stimuli, fundamental metamers, and Wyszecki's metameric blacks . The American Journal of Psychology 95 ( 4 ), 537 { 564 ( 1982 )

7. Colantoni , P. , Thomas , J. , Hebert , M. , Tremeau , A. : An online tool for displaying and processing spectral re ectance images . In: 2019 15th International Conference on Signal-Image Technology Internet-Based Systems (SITIS) . pp. 725 { 731 ( 2019 )

8. Colantoni , P. , Thomas , J. , Tremeau , A. , Hardeberg , J.Y.: Web technologies enable agile color management . In: 2019 15th International Conference on Signal-Image Technology Internet-Based Systems (SITIS) . pp. 303 { 310 ( 2019 )

9. Colantoni , P. , Thomas , J.B. : A color management process for real time color reconstruction of multispectral images . In: Salberg, A.B. , Hardeberg , J.Y. , Jenssen , R . (eds.) Image Analysis . pp. 128 { 137 . Springer, Berlin, Heidelberg ( 2009 )

10. Colantoni , P. , Thomas , J.B. , Hardeberg , J.Y. : High-end colorimetric display characterization using an adaptive training set . Journal of the Society for Information Display 19 ( 8 ), 520 { 530 ( 2011 ). https://doi.org/10.1889/JSID19.8. 520

11. Colantoni , P. , Thomas , J.B. , Pillay , R.: Graph-based 3d visualization of color content in paintings . In: Artusi, A. , Joly , M. , Lucet , G. , Pitzalis , D. , Ribes , A . (eds.) VAST: International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage - Short and Project Papers. The Eurographics Association ( 2010 ). https://doi.org/10.2312/PE/VAST/VAST10S/025- 030

12. Derhak , M. , Rosen , M. : Spectral colorimetry using LabPQR: An interim connection space . Journal of Imaging Science and Technology 50 ( 1 ), 53 { 63 ( 2006 )

13. Fairchild , M.D. , Johnson, G.M.: METACOW: A public-domain, high-resolution, fully-digital, noise-free, metameric, extended-dynamic-range, spectral test target for imaging system analysis and simulation . In: The Twelfth Color Imaging Conference: Color Science and Engineering Systems , Technologies, Applications, CIC 2004 . pp. 239 { 245 . IS& T - The Society for Imaging Science and Technology ( 2004 )

14. Kang , X. , Duan , P. , Li , S. : Hyperspectral image visualization with edge-preserving ltering and principal component analysis . Information Fusion 57 , 130 { 143 ( 2020 )

15. Khan , H.A. , Thomas , J.B. , Hardeberg , J.Y. , Laligant , O. : Spectral adaptation transform for multispectral constancy . Journal of Imaging Science and Technology 62 ( 2 ), 20504 {1{ 12 ( 2018 )

16. Khan , H.A. , Thomas , J.B. , Hardeberg , J.Y. , Laligant , O. : Multispectral camera as spatio-spectrophotometer under uncontrolled illumination . Optics Express 27 ( 2 ), 1051 { 1070 ( 2019 ). https://doi.org/10.1364/OE.27.001051

17. Kim , S.J. , Zhuo , S. , Deng , F. , Fu , C. , Brown , M.: Interactive visualization of hyperspectral images of historical documents . IEEE Transactions on Visualization and Computer Graphics 16 ( 6 ), 1441 { 1448 ( 2010 )

18. Kotwal , K. , Chaudhuri , S. : Visualization of hyperspectral images using bilateral ltering . IEEE Transactions on Geoscience and Remote Sensing 48 ( 5 ), 2308 { 2316 ( 2010 )

19. Lapray , P.J., Thomas , J.B. , Gouton , P. : High dynamic range spectral imaging pipeline for multispectral lter array cameras . Sensors 17 ( 6 ), 1281 ( 2017 ). https://doi.org/10.3390/s17061281

20. Lapray , P.J. , Wang , X. , Thomas , J.B. , Gouton , P. : Multispectral lter arrays: Recent advances and practical implementation . Sensors 14 ( 11 ), 21626 { 21659 ( 2014 )

21. Le Moan , S. , Mansouri , A. , Voisin , Y. , Hardeberg , J.Y.: A constrained band selection method based on information measures for spectral image color visualization . IEEE Transactions on Geoscience and Remote Sensing 49 ( 12 ), 5104 { 5115 ( 2011 )

22. Liao , D. , Chen , S. , Qian , Y. : Visualization of hyperspectral images using moving least squares . In: 2018 24th International Conference on Pattern Recognition (ICPR) . pp. 2851 { 2856 ( 2018 )

23. Liao , D. , Qian , Y. , Zhou , J.: Visualization of hyperspectral imaging data based on manifold alignment . In: 2014 22nd International Conference on Pattern Recognition . pp. 70 { 75 ( 2014 )

24. Liu , D. , Wang , L. , Benediktsson , J.A. : Interactive multi-image colour visualization for hyperspectral imagery . International Journal of Remote Sensing 38 ( 4 ), 1062 { 1082 ( 2017 ). https://doi.org/10.1080/01431161. 2016 .1277041

25. Pardo , A. , Gutierrez-Gutierrez , J.A. , Lopez-Higuera , J.M. , Conde , O.M.: Contextfree hyperspectral image enhancement for wide- eld optical biomarker visualization . Biomed. Opt. Express 11 ( 1 ), 133 {148 (Jan 2020 )

26. Polder , G., van der Heijden, G.W.: Visualization of spectral images . In: Censor, Y. , Ding , M. (eds.) Visualization and Optimization Techniques . vol. 4553 , pp. 132 { 137 . International Society for Optics and Photonics, SPIE ( 2001 )

27. Schockling , M. , Bonce , R. , Gutierrez , A. , Robila , S.A. : Visualization of hyperspectral images . In: Shen, S.S. , Lewis , P.E. (eds.) Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XV . vol. 7334 , pp. 715 { 726 . SPIE ( 2009 )

28. Su , H. , Du , Q. , Du , P. : Hyperspectral image visualization using band selection . IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 7 ( 6 ), 2647 { 2658 ( 2014 )

29. Thomas , J.B. , Hardeberg , J.Y. , Foucherot , I. , Gouton , P. : The PLVC display color characterization model revisited . Color Research & Application 33 ( 6 ), 449 { 460 ( 2008 ). https://doi.org/10.1002/col.20447

30. van Trigt, C. : Metameric blacks and estimating re ectance . Journal of the Optical Society of America A 11 ( 3 ), 1003 {1024 (Mar 1994 )

31. Tsutsumi , S. , Rosen , M.R. , Berns , R.S.: Spectral color management using interim connection spaces based on spectral decomposition . Color Research & Application 33 ( 4 ), 282 { 299 ( 2008 )

32. Vienot , F. , Brettel , H.: The Verriest Lecture: Visual properties of metameric blacks beyond cone vision . Journal of the Optical Society of America A 31 ( 4 ), A38{A46 (Apr 2014 )