Exploring multispectral reconstruction based on camera response prediction

Abstract

Spectral reconstruction based on digital imaging has become an important way for obtaining spectral images with high spatial-resolution. Current research has made great achievements in the laboratory; however, dealing with rapidly changing light sources, illumination, and imaging parameters in an open environment presents significant challenges for spectral reconstruction. This is because a spectral reconstruction model established under one set of imaging conditions is not suitable for use under different imaging conditions. To deal with the challenges, in this study, we explored the feasibility of spectral reconstruction based on camera raw response prediction. In the proposed method, the camera raw response of the training dataset under specific imaging conditions is first predicted via the camera imaging model, then the spectral reconstruction algorithm is applied to spectrally characterize the digital camera based on the training dataset, at last, the spectral reflectance of testing target is reconstructed from the captured image under the same imaging condition. The performance of the proposed method (M2) is tested and compared with the traditional way (M1) which uses captured training dataset to reconstruct capture target. Results show the performance of M2 is slightly inferior to M1 but it still in a relatively good reconstruction accuracy. In addition, we find the proposed method is sensitive to the spectral reconstruction algorithms used in it, and different algorithms have different performances in spectral and chromaticity aspects.

Keywords

• spectral reconstruction
• digital Camera
• imaging model
• raw response prediction 1

1. Introduction

Spectral reflectance is not only the 'fingerprint' of color but also an important feature to describe the physical properties of object, therefore, it has been widely used in the filed of highfidelity color reproduction and material analysis. Multispectral reconstruction is one of the important techniques to acquire spectral images with high-spatial resolutions [ 1, 2, 3, 4 ], it can overcome the application limitation of spectrophotometers that can only perform single-point measurements. Also, compared with multispectral cameras, it can further improve the spatial resolution of spectral images and reduce the cost of hardware systems.

Multispectral reconstruction has made significant progress in the laboratory. However, due to the sensitivity of the spectral reconstruction model to changes in imaging conditions [ 1 ], it still faces many challenges when applied in open environments. As shown in Figure 1, when a spectral reconstruction matrix developed under the CIED65 is applied under the CIEA, the reconstructed spectral root-mean-square error (RMSE) increased from 2.87% to 41.32%, indicating that spectral reconstruction is sensitive to changes in light sources. In addition, changes in the illuminance and imaging parameters, such as exposure time and ISO, can cause variations in imaging conditions. Therefore, when dealing with an open environment where imaging conditions are constantly changing, it is crucial to develop a new spectral reconstruction method.

In recent years, researchers have been exploring ways to address the challenges of applying multispectral reconstruction in open environments. Shrestha et al. used a binocular camera and estimated the spectral power distribution of the light source by adding a broadband filter in front of one of the lenses. They then used this information to characterize the camera and reconstruct the spectral reflectance of the target [ 5 ]. Inspired by the color constancy theory, Khan et al. proposed the spectral adaptation transformation (SAT) for multispectral constancy [ 6, 7 ]. Finlayson and Liang et al. found that existing spectral reconstruction algorithms are sensitive to exposure level changes, leading to deviations of reconstructed spectral curve shape from the ground truth, and proposed the solutions [ 1, 8, 9, 10, 11 ]. Although the above research has achieved preliminary results, there is still a certain gap between theory study and practical applications in open environments.

In this study, we explored the feasibility of spectral reconstruction based on camera raw response prediction. In the proposed method, the camera raw response of the training dataset under specific imaging conditions is first predicted using the camera imaging model [ 12, 13, 14 ], and then the spectral reconstruction algorithm is applied to spectral characterize the digital camera based on the training dataset. At last, the spectral reflectance of the target object is reconstructed from the captured image under the same imaging condition. Theoretically speaking, the proposed method has the potential to achieve real-time spectral characterization of the camera and facilitate the practical applications of multispectral reconstruction technology in diverse open environments. Research results indicate that the proposed method (M2) can achieve satisfactory accuracy across different exposure and ISO settings, yet there remains an improvement space when compared to the result that uses captured training samples to reconstruct the captured target (M1).

2. Camera raw response prediction

For digital cameras, it is a complex workflow from radiant energy from scene to final visual pleased images [ 12, 13 ]. In this study, we research on camera raw response prediction-based multispectral reconstruction, as the raw response is linearized data and easier to predict than the post-processed digital values[ 14 ]. The prediction model for linearized camera raw response is described below.

First, the radiation spectrum of the light source is irradiated on the surface of the object. After absorption and reflection by the object, the radiation spectrum image of the object is obtained, as shown in Equation (1):

where Ω represents the spectral wavelength range, which is taken from 400 to 700nm in this study, is the camera spectral sensitivity function, ( , ) represents the photoelectrons conversion efficiency at position ( , ). Therefore, the total photo-electric conversion ( , ) by the camera sensor over a given exposure time can be defined as: 1 + 4 /# 2(1 + )2 , , , where /# is the f-number of the lens, is the magnification of the lens, is the transmittance of the lens, ( , , ) is the spectral radiance at each pixel, and , , denotes the irradiance at each pixel. Furthermore, the scene irradiance image enters the sensor, which undergoes photoelectric conversion and analog-to-digital conversion to obtain a mosaic image in the Bayer pattern. The demosaicing algorithm is then applied to get the raw format image of the scene captured by the camera. These specific steps are shown in Equations (3) to (6). ( , , ) =

( , , ) ( ) , where is the wavelength, ( , , ) is the spectral reflectance of the pixels in the image, ( ) denotes the relative spectral radiance of the light source, and ( , , ) denotes the spectral radiance at each pixel. Subsequently, the radiation spectrum from the scene, after passing through the camera lens, forms an optical irradiance image before it enters the sensor [ 13 ]. This can be expressed as shown in Equation (2): (1) (2) (3) (4) (5) , = ( , ) =

( , ) , where represents the exposure time, is the sensitivity function, which can usually be obtained by dividing the ISO value by 100. When the spectral radiance and the sensor's quantum efficiency remain constant during a single exposure, it can be further simplified to the product form as shown in Equation (5):

According to Equation (3) and Equation (5), the raw response value at any pixel in the image can be expressed in the form shown in Equation (6):

Finally, considering the nonlinear relationship between the irradiance of the camera sensor and the camera's readout response [ 14 ], the model further introduces three constants, 1, 2, and to characterize the non-equivalence between exposure and ISO, thus yielding the final expression for the raw response ( , ) of a pixel. + 1) + 2],

This study uses the camera imaging model shown in Equation (7) to predict the color card raw response under set imaging conditions to support the multispectral reconstruction research based on camera raw response prediction.

3. Experimental

In the experiment, we used the ColorChecker SG140 (CCSG) and ColorChecker 24 (CC) color charts as training samples and testing target respectively. The color charts were captured in a closed and nearly uniform illuminated lightbox with a Nikon D7200 digital camera, keeping the color charts located at the center of the camera's field of view (as shown in Figure 2). The color chart plane was approximately one meter away from the camera sensor plane, and the focal length was set as 35 mm. Five different combinations of exposure and ISO were set to capture the color charts. The settings of five exposure and ISO are shown in Table 1. For each setting, the white patches in the color charts were not overexposed, ensuring the validity of the data. (6) (7)

The camera sensitivity functions of Nikon D7200 were estimated by Jiang’s method and CC color chart[ 15, 16 ], where the CC color chart is used to implement the camera sensitivity estimation method of Jiang et al [ 15 ], and the result is plotted in Figure 3(a). The spectral power distribution of the light source was measured using the EVERFINE spectroradiometer of SPIC-300AW, and the spectral power distribution of the light source in the lightbox is plotted in Figure 3(b). The spectral reflectance of the color charts was measured using the spectrophotometer of X-rite i1-Pro3. With the spectral reflectance, the spectral sensitivity function, the spectral power distribution, and the imaging setting of each group, we can predict the camera raw response of CCSG using the prediction model described in section 2. Furthermore, the nonlinear parameters were estimated using the proposed curve fitting methods in reference [ 14 ], as shown in Equation (8):

where 1 denotes the measured spectral reflectance using the spectrophotometer, 2 represents the reconstructed spectral reflectance, is the transpose operator, and is the number of sampled wavelengths, which is 31 in this study.

In the spectral reconstruction stage, two types and four spectral algorithms of Liang, OLS, Cao, and Kernel are selected in the experiment [ 17, 18, 19, 20 ], where the former two methods are regression-based and the latter two are interpolation-based. We use ‘M1’ to represent the situation of using captured training dataset to reconstruct the captured testing target, and use ‘M2’ to represent the situation of using predicted training dataset to reconstruct the captured testing target. The training dataset in this study is the CCSG color chart, and the testing target is the CC color chart.

In addition, we use CIEDE2000 color difference (∆E00) to evaluate the accuracy of camera raw response prediction, and use the root-mean-square error (RMSE) and CIEDE2000 color difference (∆E00) to evaluate the accuracy of spectral reconstruction. The method to calculate the RMSE is in Equation (9):

RMSE = 1 1 − 2

1 − 2 , = 0 + 1

+ 2, where represents the captured camera raw response of the color chart, 0 represents the predicted camera raw response using the prediction model, and is the sensitivity function as described in Equation (3). The captured camera raw response is extracted with the help of Dcraw and a self-developed GUI interface in Matlab. (8) (9)

4. Result and analysis

4.1. Camera raw response prediction

Under the setting imaging conditions, we first check the camera raw response prediction results for each group of settings. The color difference of CIEDE2000 (∆E00) between the predicted and captured camera raw response for CCSG and CC are calculated and summarized in Table 2, respectively. It should be noted that since there is a certain degree of amplitude deviation between the predicted raw response of the color chart and the captured raw response, the max-min normalization is performed on the predicted raw response and the captured raw response before calculating the color difference ∆E00.

It can be seen from Table 2 that for the color chart CCSG, the average color difference ∆E00 for camera raw response prediction under five groups of settings is 1.81, and except for Group 5, the prediction ∆E00 for each group of settings is around the average color difference. The predicted difference between Group 5 and the other 4 groups may caused by the nonequivalence caused by the combination of exposure and ISO. For the color chart CC, the predicted color difference is bigger than the CCSG, which is around the mean ∆E00 of 3.22 of the five groups. The reason behind this may be the parameters estimated in the prediction model are carried out with the CCSG color chart. In order to more intuitively compare the predicted raw response and the captured raw response, the distribution of the predicted and captured raw responses are plotted in Figure 4.

(a) color chart CCSG (b) color chart CC

The results from Figure 5 indicate that the distribution of the predicted camera raw responses for both color charts of CCSG and CC closely overlaps with the distribution of the captured raw responses with minor deviations observed in certain individual colors, which means relatively good prediction results of the proposed method..

4.2. Response prediction-based spectral reconstruction

In this section, we present the results of camera raw response prediction-based spectral reconstruction. Both the camera raw response of CCSG and CC color charts have been predicted under different imaging settings to check the prediction accuracy. But for prediction-based spectral reconstruction, using the CCSG as a training dataset and the CC as a testing target, we compared the spectral reconstruction accuracy in two different situations by four and two types of different spectral reconstruction algorithms. In the first situation (M1), we use the captured raw response of CCSG to reconstruct the spectral reflectance of captured raw CC. In the second situation (M2), we use the predicted raw response of CCSG to reconstruct the spectral reflectance of captured raw CC. The selected spectral reconstruction methods are Liang, OLS, Cao, and Kernel[ 17, 18, 19, 20 ]. The spectra reconstruction accuracy is evaluated by RMSE and ∆E00, respectively. The results of M1 and M2 under five different groups of imaging settings are summarized in Table 3 and Table 4, respectively. And the box plots of spectral reconstruction errors in Table 3 and Table 4 are plotted in Figure 5.

From Table 3, Table 4, and Figure 5, it can be observed that for using the captured training dataset to reconstruct the captured testing target (M1), the spectral reconstruction errors are relatively good according to the experience. However, there are certain differences in spectral reconstruction errors between different algorithms. Among them, Liang's method generally shows the best spectral reconstruction for both RMSE and ∆E00, and Cao’s method shows the worst colorimetric accuracy. For using the predicted training dataset to reconstruct the captured testing target (M2), Liang’s method shows the highest RMSE of 5.73%, while Cao’s method shows the lowest RMSE of 4.02%, but for color difference, the results of the two methods are completely opposite, where Liang’s method shows the lowest ∆E00 of 3.53, while Cao’s method shows the highest ∆E00 of 5.30.

The results in Table 3, Table 4, and Figure 5 illustrated that the performance of M2 is slightly inferior to M1, but it still in a relatively good reconstruction accuracy. In addition, it easy to infer that the proposed camera raw response prediction based method is sensitive to the spectral reconstruction algorithms used in it, and different algorithms have different performances in spectral and chromaticity aspects. To further inspection the spectral reconstructed results, several spectral reflectance reconstructed by Cao’s method under M1 and M2 are compared with the measured results in Figure 6.

The results in Figure 6 show that the spectral reflectance reconstructed by Cao’s method under M1 and M2 overlaps and closely matches the measured data. This demonstrates the potential of the proposed method to address the dependency of spectral reconstruction techniques on imaging conditions. Overall, the experimental results in this study highlight the potential of the proposed method to achieve real-time spectral characterization of the camera and facilitate practical applications of multispectral reconstruction technology in open environments.

5. Conclusions

This study introduces a novel approach to spectral reconstruction based on camera raw response prediction to address the challenges posed by rapidly changing light sources, illumination, and imaging parameters in an open environment. The method involves predicting the camera raw response of the training dataset based on the digital imaging model, characterizing the camera spectrally, and reconstructing the spectral reflectance of the testing target using the predicted response and spectral reflectance of the training dataset. The proposed method differs from existing research by correcting imaging conditions from an open environment to a laboratory via illumination estimation. This method is more flexible for practical applications. The experimental results have proven the effectiveness of this method. In future research, the validation of the proposed method under different open environments will be conducted to further investigate the feasibility of its performance. In addition, we will conduct further optimization research on the proposed method and develop an integrated application system.

Acknowledgements

This work was supported by National Natural Science Foundation of China (62305255), Hubei Provincial Natural Science Foundation General Project (No.2022CFB537), Hubei Provincial Department of Education Science and Technology Research Program Youth Talent (No.Q20221706), and China Scholarship Council (202308420128). [19] Cao B, Liao N, Cheng H, Spectral reflectance reconstruction from RGB images based on weighting smaller color difference group, J. Color Research & Application 42.3 (2017) 327332. [20] Xu P, Xu H, Diao C, et al, Self-training-based spectral image reconstruction for art paintings with multispectral imaging, J. Applied optics 56.30 (2017) 8461-8470.

References

[1] Liang J , Xin L , Zuo Z , et al, Research on the deep learning-based exposure invariant spectral reconstruction method , J. Frontiers in Neuroscience 16 ( 2022 ): 1031546 .

[2] Zhao Y , Po , L M , Yan Q , et al. Hierarchical regression network for spectral reconstruction from RGB images , in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , 2020 , pp. 422 - 423 .

[3] Cai Y , Lin J , Lin Z , et al. Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction , in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022 , pp: 745 - 755 .

[4] Lin , Yi-Tun , and Graham D. Finlayson , On the optimization of regression-based spectral reconstruction . J. Sensors 21 .16 ( 2021 ): 5586 .

[5] Shrestha R , Hardeberg J Y , Spectrogenic imaging: A novel approach to multispectral imaging in an uncontrolled environment , J. Optics Express 22.8 ( 2014 ) 9123 - 9133 .

[6] Khan H A , Thomas J B , Hardeberg J Y , et al, Illuminant estimation in multispectral imaging , J. JOSA 34.7 ( 2017 ) 1085 - 1098 .

[7] Khan H A , Thomas J B , Hardeberg J Y , et al, Multispectral camera as spatiospectrophotometer under uncontrolled illumination , J. Optics Express 27.2 ( 2019 ) 1051 - 1070 .

[8] Lin Y T , Finlayson G D . Exposure invariance in spectral reconstruction from rgb images , in: Color and Imaging Conference. Society for Imaging Science and Technology , 2019 , pp. 284 - 289 .

[9] Lin Y T , Finlayson G D . Physically plausible spectral reconstruction from RGB images , in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , 2020 , pp. 532 - 533 .

[10] Ibrahim A , Tominaga S , Horiuchi T , A spectral invariant representation of spectral reflectance , J. Optical Review 18.2 ( 2011 ) 231 - 236 .

[11] Liang J , Xin L , Cheng J , et al, Adaptive Weighted Spectral Reconstruction Method Against Exposure Variation, J. Spectroscopy and spectral analysis 43.11 ( 2023 ) 3330 - 3338 .

[12] Ramanath R , Snyder W E , Yoo Y , et al, Color image processing pipeline , J. IEEE Signal Processing Magazine 22.1 ( 2005 ) 34 - 43 .

[13] Farrell J E , Catrysse P B , Wandell B A , Digital camera simulation , J. Applied Optics 51.4 ( 2012 ) A80 - A90 .

[14] Qiu J , Xu H , Camera response prediction for various capture settings using the spectral sensitivity and crosstalk model , J. Applied Optics 55 .25 ( 2016 ) 6989 - 6999 .

[15] Jiang J , Liu D , Gu J , et al, What is the space of spectral sensitivity functions for digital color cameras? . In 2013 IEEE Workshop on Applications of Computer Vision (WACV), ( 2013 ) pp. 168 - 179 .

[16] Tominaga , S , Shogo N , and Ryo O. Measurement and estimation of spectral sensitivity functions for mobile phone cameras . Sensors 21 .15 ( 2021 ): 4985 .

[17] Liang J , Wan X , Optimized method for spectral reflectance reconstruction from camera responses , J. Optics Express 25 .23 ( 2017 ) 28273 - 28287 .

[18] Connah D R , Hardeberg J Y. Spectral recovery using polynomial models, in: Color Imaging X: Processing, Hardcopy, and Applications. SPIE , 2005 , 5667 , pp. 65 - 75 .