Spectral reflectance is not only the 'fingerprint' of color but also an important feature to describe the physical properties of object, therefore, it has been widely used in the filed of highfidelity color reproduction and material analysis. Multispectral reconstruction is one of the important techniques to acquire spectral images with high-spatial resolutions [1,2,3,4], it can overcome the application limitation of spectrophotometers that can only perform single-point measurements. Also, compared with multispectral cameras, it can further improve the spatial resolution of spectral images and reduce the cost of hardware systems.
Multispectral reconstruction has made significant progress in the laboratory. However, due to the sensitivity of the spectral reconstruction model to changes in imaging conditions [1], it still faces many challenges when applied in open environments. As shown in Figure 1, when a spectral reconstruction matrix developed under the CIED65 is applied under the CIEA, the reconstructed spectral root-mean-square error (RMSE) increased from 2.87% to 41.32%, indicating that spectral reconstruction is sensitive to changes in light sources. In addition, changes in the illuminance and imaging parameters, such as exposure time and ISO, can cause variations in imaging conditions. Therefore, when dealing with an open environment where imaging conditions are constantly changing, it is crucial to develop a new spectral reconstruction method. In this study, we explored the feasibility of spectral reconstruction based on camera raw response prediction. In the proposed method, the camera raw response of the training dataset under specific imaging conditions is first predicted using the camera imaging model [12,13,14], and then the spectral reconstruction algorithm is applied to spectral characterize the digital camera based on the training dataset. At last, the spectral reflectance of the target object is reconstructed from the captured image under the same imaging condition. Theoretically speaking, the proposed method has the potential to achieve real-time spectral characterization of the camera and facilitate the practical applications of multispectral reconstruction technology in diverse open environments. Research results indicate that the proposed method (M2) can achieve satisfactory accuracy across different exposure and ISO settings, yet there remains an improvement space when compared to the result that uses captured training samples to reconstruct the captured target (M1).
For digital cameras, it is a complex workflow from radiant energy from scene to final visual pleased images [12,13]. In this study, we research on camera raw response prediction-based multispectral reconstruction, as the raw response is linearized data and easier to predict than the post-processed digital values [14]. The prediction model for linearized camera raw response is described below.
First, the radiation spectrum of the light source is irradiated on the surface of the object. After absorption and reflection by the object, the radiation spectrum image of the object is obtained, as shown in Equation ( 1):
where is the wavelength, (, , ) is the spectral reflectance of the pixels in the image, () denotes the relative spectral radiance of the light source, and (, , ) denotes the spectral radiance at each pixel. Subsequently, the radiation spectrum from the scene, after passing through the camera lens, forms an optical irradiance image before it enters the sensor [13]. This can be expressed as shown in Equation ( 2):
where /# is the f-number of the lens, is the magnification of the lens, is the transmittance of the lens, (, , ) is the spectral radiance at each pixel, and , , denotes the irradiance at each pixel. Furthermore, the scene irradiance image enters the sensor, which undergoes photoelectric conversion and analog-to-digital conversion to obtain a mosaic image in the Bayer pattern. The demosaicing algorithm is then applied to get the raw format image of the scene captured by the camera. These specific steps are shown in Equations ( 3) to (6).
where Ω represents the spectral wavelength range, which is taken from 400 to 700nm in this study, is the camera spectral sensitivity function, (, ) represents the photoelectrons conversion efficiency at position (, ). Therefore, the total photo-electric conversion (, ) by the camera sensor over a given exposure time can be defined as:
where represents the exposure time, is the sensitivity function, which can usually be obtained by dividing the ISO value by 100. When the spectral radiance and the sensor's quantum efficiency remain constant during a single exposure, it can be further simplified to the product form as shown in Equation ( 5):
According to Equation (3) and Equation ( 5), the raw response value at any pixel in the image can be expressed in the form shown in Equation ( 6):
Finally, considering the nonlinear relationship between the irradiance of the camera sensor and the camera's readout response [14], the model further introduces three constants, 1 , 2 , and to characterize the non-equivalence between exposure and ISO, thus yielding the final expression for the raw response (, ) of a pixel.
This study uses the camera imaging model shown in Equation (7) to predict the color card raw response under set imaging conditions to support the multispectral reconstruction research based on camera raw response prediction.
In the experiment, we used the ColorChecker SG140 (CCSG) and ColorChecker 24 (CC) color charts as training samples and testing target respectively. The color charts were captured in a closed and nearly uniform illuminated lightbox with a Nikon D7200 digital camera, keeping the color charts located at the center of the camera's field of view (as shown in Figure 2). The color chart plane was approximately one meter away from the camera sensor plane, and the focal length was set as 35 mm. Five different combinations of exposure and ISO were set to capture the color charts. The settings of five exposure and ISO are shown in Table 1. For each setting, the white patches in the color charts were not overexposed, ensuring the validity of the data. The camera sensitivity functions of Nikon D7200 were estimated by Jiang's method and CC color chart [15,16], where the CC color chart is used to implement the camera sensitivity estimation method of Jiang et al [15], and the result is plotted in Figure 3(a). The spectral power distribution of the light source was measured using the EVERFINE spectroradiometer of SPIC-300AW, and the spectral power distribution of the light source in the lightbox is plotted in Figure 3(b). The spectral reflectance of the color charts was measured using the spectrophotometer of X-rite i1-Pro3. With the spectral reflectance, the spectral sensitivity function, the spectral power distribution, and the imaging setting of each group, we can predict the camera raw response of CCSG using the prediction model described in section 2. Furthermore, the nonlinear parameters were estimated using the proposed curve fitting methods in reference [14], as shown in Equation ( 8):
where represents the captured camera raw response of the color chart, 0 represents the predicted camera raw response using the prediction model, and is the sensitivity function as described in Equation ( 3). The captured camera raw response is extracted with the help of Dcraw and a self-developed GUI interface in Matlab. In the spectral reconstruction stage, two types and four spectral algorithms of Liang, OLS, Cao, and Kernel are selected in the experiment [17,18,19,20], where the former two methods are regression-based and the latter two are interpolation-based. We use 'M1' to represent the situation of using captured training dataset to reconstruct the captured testing target, and use 'M2' to represent the situation of using predicted training dataset to reconstruct the captured testing target. The training dataset in this study is the CCSG color chart, and the testing target is the CC color chart.
In addition, we use CIEDE2000 color difference (∆E00) to evaluate the accuracy of camera raw response prediction, and use the root-mean-square error (RMSE) and CIEDE2000 color difference (∆E00) to evaluate the accuracy of spectral reconstruction. The method to calculate the RMSE is in Equation ( 9):
where 1 denotes the measured spectral reflectance using the spectrophotometer, 2 represents the reconstructed spectral reflectance, is the transpose operator, and is the number of sampled wavelengths, which is 31 in this study.
Under the setting imaging conditions, we first check the camera raw response prediction results for each group of settings. The color difference of CIEDE2000 (∆E00) between the predicted and captured camera raw response for CCSG and CC are calculated and summarized in Table 2, respectively. It should be noted that since there is a certain degree of amplitude deviation between the predicted raw response of the color chart and the captured raw response, the max-min normalization is performed on the predicted raw response and the captured raw response before calculating the color difference ∆E00.
The color difference of CIEDE2000 between the predicted and captured camera raw response for CCSG and CC under five group of settings.
CCSG It can be seen from Table 2 that for the color chart CCSG, the average color difference ∆E00 for camera raw response prediction under five groups of settings is 1.81, and except for Group 5, the prediction ∆E00 for each group of settings is around the average color difference. The predicted difference between Group 5 and the other 4 groups may caused by the nonequivalence caused by the combination of exposure and ISO. For the color chart CC, the predicted color difference is bigger than the CCSG, which is around the mean ∆E00 of 3.22 of the five groups. The reason behind this may be the parameters estimated in the prediction model are carried out with the CCSG color chart. In order to more intuitively compare the predicted raw response and the captured raw response, the distribution of the predicted and captured raw responses are plotted in Figure 4. The results from Figure 5 indicate that the distribution of the predicted camera raw responses for both color charts of CCSG and CC closely overlaps with the distribution of the captured raw responses with minor deviations observed in certain individual colors, which means relatively good prediction results of the proposed method..
In this section, we present the results of camera raw response prediction-based spectral reconstruction. Both the camera raw response of CCSG and CC color charts have been predicted under different imaging settings to check the prediction accuracy. But for prediction-based spectral reconstruction, using the CCSG as a training dataset and the CC as a testing target, we compared the spectral reconstruction accuracy in two different situations by four and two types of different spectral reconstruction algorithms. In the first situation (M1), we use the captured raw response of CCSG to reconstruct the spectral reflectance of captured raw CC. In the second situation (M2), we use the predicted raw response of CCSG to reconstruct the spectral reflectance of captured raw CC. The selected spectral reconstruction methods are Liang, OLS, Cao, and Kernel [17,18,19,20]. The spectra reconstruction accuracy is evaluated by RMSE and ∆E00, respectively. The results of M1 and M2 under five different groups of imaging settings are summarized in Table 3 and Table 4, respectively. And the box plots of spectral reconstruction errors in Table 3 and Table 4 are plotted in Figure 5.
The average spectral reconstruction errors of M1 with four algorithms under five different groups of imaging settings. From Table 3, Table 4, and Figure 5, it can be observed that for using the captured training dataset to reconstruct the captured testing target (M1), the spectral reconstruction errors are relatively good according to the experience. However, there are certain differences in spectral reconstruction errors between different algorithms. Among them, Liang's method generally shows the best spectral reconstruction for both RMSE and ∆E00, and Cao's method shows the worst colorimetric accuracy. For using the predicted training dataset to reconstruct the captured testing target (M2), Liang's method shows the highest RMSE of 5.73%, while Cao's method shows the lowest RMSE of 4.02%, but for color difference, the results of the two methods are completely opposite, where Liang's method shows the lowest ∆E00 of 3.53, while Cao's method shows the highest ∆E00 of 5.30.
The results in Table 3, Table 4, and Figure 5 illustrated that the performance of M2 is slightly inferior to M1, but it still in a relatively good reconstruction accuracy. In addition, it easy to infer that the proposed camera raw response prediction based method is sensitive to the spectral reconstruction algorithms used in it, and different algorithms have different performances in spectral and chromaticity aspects. To further inspection the spectral reconstructed results, several spectral reflectance reconstructed by Cao's method under M1 and M2 are compared with the measured results in Figure 6. The results in Figure 6 show that the spectral reflectance reconstructed by Cao's method under M1 and M2 overlaps and closely matches the measured data. This demonstrates the potential of the proposed method to address the dependency of spectral reconstruction techniques on imaging conditions. Overall, the experimental results in this study highlight the potential of the proposed method to achieve real-time spectral characterization of the camera and facilitate practical applications of multispectral reconstruction technology in open environments.
This study introduces a novel approach to spectral reconstruction based on camera raw response prediction to address the challenges posed by rapidly changing light sources, illumination, and imaging parameters in an open environment. The method involves predicting the camera raw response of the training dataset based on the digital imaging model, characterizing the camera spectrally, and reconstructing the spectral reflectance of the testing target using the predicted response and spectral reflectance of the training dataset. The proposed method differs from existing research by correcting imaging conditions from an open environment to a laboratory via illumination estimation. This method is more flexible for practical applications. The experimental results have proven the effectiveness of this method. In future research, the validation of the proposed method under different open environments will be conducted to further investigate the feasibility of its performance. In addition, we will conduct further optimization research on the proposed method and develop an integrated application system.
This work was supported by National Natural Science Foundation of China (62305255), Hubei Provincial Natural Science Foundation General Project (No.2022CFB537), Hubei Provincial Department of Education Science and Technology Research Program Youth Talent (No.Q20221706), and China Scholarship Council (202308420128).