=Paper=
{{Paper
|id=Vol-3766/CVCS2024_2_Paper_Liang_etal
|storemode=property
|title=Exploring multispectral reconstruction based on camera response prediction
|pdfUrl=https://ceur-ws.org/Vol-3766/CVCS2024_2_Paper_Liang_etal.pdf
|volume=Vol-3766
|authors=Jinxing Liang,Xin Hu,Zhuan Zuo,Xiao Liu,Yifan Li,Wensen Zhou,Hang Luo,Xinrong Hu,Kaida Xiao
|dblpUrl=https://dblp.org/rec/conf/cvcs/LiangHZLLZ0HX24
}}
==Exploring multispectral reconstruction based on camera response prediction==
https://ceur-ws.org/Vol-3766/CVCS2024_2_Paper_Liang_etal.pdf
Exploring multispectral reconstruction based on camera
response prediction
Jinxing Liang1,2,4, Xin Hu1, Zhuan Zuo1, Xiao Liu3, Yifan Li1, Wensen Zhou1, Hang Luo1,2,
Xinrong Hu1,2,* and Kaida Xiao4,*
1
School of Computer Science and Artificial Intelligence, Wuhan Textile University, Hubei Wuhan, China, 430200
2
Engineering Research Center of Hubei Province for Clothing Information, Hubei Wuhan, China, 430200
3
Wuhan Geomatics Institute, Hubei Wuhan, China, 430022
4
School of Design, University of Leeds, Leeds, UK, LS2 9JT
Abstract
Spectral reconstruction based on digital imaging has become an important way for obtaining spectral
images with high spatial-resolution. Current research has made great achievements in the laboratory;
however, dealing with rapidly changing light sources, illumination, and imaging parameters in an
open environment presents significant challenges for spectral reconstruction. This is because a
spectral reconstruction model established under one set of imaging conditions is not suitable for use
under different imaging conditions. To deal with the challenges, in this study, we explored the
feasibility of spectral reconstruction based on camera raw response prediction. In the proposed
method, the camera raw response of the training dataset under specific imaging conditions is first
predicted via the camera imaging model, then the spectral reconstruction algorithm is applied to
spectrally characterize the digital camera based on the training dataset, at last, the spectral
reflectance of testing target is reconstructed from the captured image under the same imaging
condition. The performance of the proposed method (M2) is tested and compared with the traditional
way (M1) which uses captured training dataset to reconstruct capture target. Results show the
performance of M2 is slightly inferior to M1 but it still in a relatively good reconstruction accuracy. In
addition, we find the proposed method is sensitive to the spectral reconstruction algorithms used in it,
and different algorithms have different performances in spectral and chromaticity aspects.
Keywords
spectral reconstruction, digital Camera, imaging model, raw response prediction 1
1. Introduction
Spectral reflectance is not only the 'fingerprint' of color but also an important feature to
describe the physical properties of object, therefore, it has been widely used in the filed of high-
fidelity color reproduction and material analysis. Multispectral reconstruction is one of the
important techniques to acquire spectral images with high-spatial resolutions [1, 2, 3, 4], it can
overcome the application limitation of spectrophotometers that can only perform single-point
measurements. Also, compared with multispectral cameras, it can further improve the spatial
resolution of spectral images and reduce the cost of hardware systems.
Multispectral reconstruction has made significant progress in the laboratory. However, due
to the sensitivity of the spectral reconstruction model to changes in imaging conditions [1], it
still faces many challenges when applied in open environments. As shown in Figure 1, when a
spectral reconstruction matrix developed under the CIED65 is applied under the CIEA, the
reconstructed spectral root-mean-square error (RMSE) increased from 2.87% to 41.32%,
indicating that spectral reconstruction is sensitive to changes in light sources. In addition,
CVCS2024: the 12th Colour and Visual Computing Symposium, September 5–6, 2024, Gjøvik, Norway
∗
Corresponding author.
jxliang@wtu.edu.cn (J. Liang); hxr@wtu.edu.cn (X. Hu); K.Xiao1@leeds.ac.uk (K. Xiao)
0000-0002-7570-1827 (J. Liang); 0000-0002-0402-8276 (K. Xiao)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
changes in the illuminance and imaging parameters, such as exposure time and ISO, can cause
variations in imaging conditions. Therefore, when dealing with an open environment where
imaging conditions are constantly changing, it is crucial to develop a new spectral
reconstruction method.
Figure 1: An example of spectral reconstruction being sensitive to changes in imaging
conditions of light source.
In recent years, researchers have been exploring ways to address the challenges of applying
multispectral reconstruction in open environments. Shrestha et al. used a binocular camera
and estimated the spectral power distribution of the light source by adding a broadband filter
in front of one of the lenses. They then used this information to characterize the camera and
reconstruct the spectral reflectance of the target [5]. Inspired by the color constancy theory,
Khan et al. proposed the spectral adaptation transformation (SAT) for multispectral constancy
[6, 7]. Finlayson and Liang et al. found that existing spectral reconstruction algorithms are
sensitive to exposure level changes, leading to deviations of reconstructed spectral curve shape
from the ground truth, and proposed the solutions [1, 8, 9, 10, 11]. Although the above research
has achieved preliminary results, there is still a certain gap between theory study and practical
applications in open environments.
In this study, we explored the feasibility of spectral reconstruction based on camera raw
response prediction. In the proposed method, the camera raw response of the training dataset
under specific imaging conditions is first predicted using the camera imaging model [12, 13,
14], and then the spectral reconstruction algorithm is applied to spectral characterize the
digital camera based on the training dataset. At last, the spectral reflectance of the target object
is reconstructed from the captured image under the same imaging condition. Theoretically
speaking, the proposed method has the potential to achieve real-time spectral characterization
of the camera and facilitate the practical applications of multispectral reconstruction
technology in diverse open environments. Research results indicate that the proposed
method (M2) can achieve satisfactory accuracy across different exposure and ISO settings, yet
there remains an improvement space when compared to the result that uses captured training
samples to reconstruct the captured target (M1).
2. Camera raw response prediction
For digital cameras, it is a complex workflow from radiant energy from scene to final visual
pleased images [12, 13]. In this study, we research on camera raw response prediction-based
multispectral reconstruction, as the raw response is linearized data and easier to predict than
the post-processed digital values[14]. The prediction model for linearized camera raw
response is described below.
First, the radiation spectrum of the light source is irradiated on the surface of the object.
After absorption and reflection by the object, the radiation spectrum image of the object is
obtained, as shown in Equation (1):
�(�, �, �) = �(�, �, �)�(�) ��, (1)
where � is the wavelength, �(�, �, �) is the spectral reflectance of the pixels in the image, �(�)
denotes the relative spectral radiance of the light source, and �(�, �, �) denotes the spectral
radiance at each pixel. Subsequently, the radiation spectrum from the scene, after passing
through the camera lens, forms an optical irradiance image before it enters the sensor [13].
This can be expressed as shown in Equation (2):
�� � � �
� �, �, � ≅ 2 2 � , ,� , (2)
1 + 4 �/# (1 + �) � �
where �/# is the f-number of the lens, � is the magnification of the lens, � � is the
transmittance of the lens, �(�, �, �) is the spectral radiance at each pixel, and � �, �, � denotes
the irradiance at each pixel. Furthermore, the scene irradiance image enters the sensor, which
undergoes photoelectric conversion and analog-to-digital conversion to obtain a mosaic image
in the Bayer pattern. The demosaicing algorithm is then applied to get the raw format image of
the scene captured by the camera. These specific steps are shown in Equations (3) to (6).
� �, � = � �, �, � �� � ��, (3)
�
where Ω represents the spectral wavelength range, which is taken from 400 to 700nm in
this study, �� � is the camera spectral sensitivity function, �(�, �) represents the photo-
electrons conversion efficiency at position (�, �). Therefore, the total photo-electric conversion
�(�, �) by the camera sensor over a given exposure time � can be defined as:
�(�, �) = �� �(�, �)�� , (4)
�
where � represents the exposure time, �� is the sensitivity function, which can usually be
obtained by dividing the ISO value by 100. When the spectral radiance and the sensor's
quantum efficiency remain constant during a single exposure, it can be further simplified to the
product form as shown in Equation (5):
�(�, �) = �� ��(�, �), (5)
According to Equation (3) and Equation (5), the raw response value at any pixel in the
image can be expressed in the form shown in Equation (6):
�� � � �
�(�, �) = �� � 2 2 � � , � , � �� � ��, (6)
� 1 + 4 �/# (1 + �)
Finally, considering the nonlinear relationship between the irradiance of the camera
sensor and the camera's readout response [14], the model further introduces three constants,
�1 , �2 , and � to characterize the non-equivalence between exposure and ISO, thus yielding the
final expression for the raw response �(�, �) of a pixel.
�� � � �
�(�, �) = �� [(� 2 2
� , , � �� � �� + �1 )� + �2 ], (7)
� 1 + 4 �/# (1 + �) � �
This study uses the camera imaging model shown in Equation (7) to predict the color card
raw response under set imaging conditions to support the multispectral reconstruction
research based on camera raw response prediction.
3. Experimental
In the experiment, we used the ColorChecker SG140 (CCSG) and ColorChecker 24 (CC) color
charts as training samples and testing target respectively. The color charts were captured in a
closed and nearly uniform illuminated lightbox with a Nikon D7200 digital camera, keeping the
color charts located at the center of the camera's field of view (as shown in Figure 2). The color
chart plane was approximately one meter away from the camera sensor plane, and the focal
length was set as 35 mm. Five different combinations of exposure and ISO were set to capture
the color charts. The settings of five exposure and ISO are shown in Table 1. For each setting,
the white patches in the color charts were not overexposed, ensuring the validity of the data.
Figure 2: Captured images of the ColorChecker SG140 color chart (left) and ColorChecker 24
color chart (right) under setting exposure 1/25s and ISO 100.
Table 1
Five groups of exposure settings in the experiment
Setting Exposure/s ISO
Group 1 1/25 100
Group 2 1/30 200
Group 3 1/60 400
Group 4 1/125 800
Group 5 1/250 1600
The camera sensitivity functions of Nikon D7200 were estimated by Jiang’s method and CC
color chart[15, 16], where the CC color chart is used to implement the camera sensitivity
estimation method of Jiang et al [15], and the result is plotted in Figure 3(a). The spectral
power distribution of the light source was measured using the EVERFINE spectroradiometer of
SPIC-300AW, and the spectral power distribution of the light source in the lightbox is plotted in
Figure 3(b). The spectral reflectance of the color charts was measured using the
spectrophotometer of X-rite i1-Pro3. With the spectral reflectance, the spectral sensitivity
function, the spectral power distribution, and the imaging setting of each group, we can predict
the camera raw response of CCSG using the prediction model described in section 2.
Furthermore, the nonlinear parameters were estimated using the proposed curve fitting
methods in reference [14], as shown in Equation (8):
�
� �0
= + �1 + �2 , (8)
�� ��
where � represents the captured camera raw response of the color chart, �0 represents the
predicted camera raw response using the prediction model, and �� is the sensitivity function as
described in Equation (3). The captured camera raw response is extracted with the help of
Dcraw and a self-developed GUI interface in Matlab.
Figure 3: (a) The spectral sensitivity function distribution of Nikon D7200, and (b) the relative
spectral power distribution of light source in lightbox.
In the spectral reconstruction stage, two types and four spectral algorithms of Liang, OLS,
Cao, and Kernel are selected in the experiment [17, 18, 19, 20], where the former two methods
are regression-based and the latter two are interpolation-based. We use ‘M1’ to represent the
situation of using captured training dataset to reconstruct the captured testing target, and use
‘M2’ to represent the situation of using predicted training dataset to reconstruct the captured
testing target. The training dataset in this study is the CCSG color chart, and the testing target is
the CC color chart.
In addition, we use CIEDE2000 color difference (∆E00) to evaluate the accuracy of camera
raw response prediction, and use the root-mean-square error (RMSE) and CIEDE2000 color
difference (∆E00) to evaluate the accuracy of spectral reconstruction. The method to calculate
the RMSE is in Equation (9):
1
RMSE = � − �2 � �1 − �2 , (9)
� 1
where �1 denotes the measured spectral reflectance using the spectrophotometer,
�2 represents the reconstructed spectral reflectance, � is the transpose operator, and � is the
number of sampled wavelengths, which is 31 in this study.
4. Result and analysis
4.1. Camera raw response prediction
Under the setting imaging conditions, we first check the camera raw response prediction
results for each group of settings. The color difference of CIEDE2000 (∆E00) between the
predicted and captured camera raw response for CCSG and CC are calculated and summarized
in Table 2, respectively. It should be noted that since there is a certain degree of amplitude
deviation between the predicted raw response of the color chart and the captured raw
response, the max-min normalization is performed on the predicted raw response and the
captured raw response before calculating the color difference ∆E00.
Table 2
The color difference of CIEDE2000 between the predicted and captured camera raw response
for CCSG and CC under five group of settings.
∆E00 CCSG CC
Group 1 1.71 3.19
Group 2 1.72 3.32
Group 3 1.68 3.11
Group 4 1.81 3.71
Group 5 2.12 2.79
Average 1.81 3.22
It can be seen from Table 2 that for the color chart CCSG, the average color difference
∆E00 for camera raw response prediction under five groups of settings is 1.81, and except for
Group 5, the prediction ∆E00 for each group of settings is around the average color difference.
The predicted difference between Group 5 and the other 4 groups may caused by the non-
equivalence caused by the combination of exposure and ISO. For the color chart CC, the
predicted color difference is bigger than the CCSG, which is around the mean ∆E00 of 3.22 of
the five groups. The reason behind this may be the parameters estimated in the prediction
model are carried out with the CCSG color chart. In order to more intuitively compare the
predicted raw response and the captured raw response, the distribution of the predicted and
captured raw responses are plotted in Figure 4.
(a) color chart CCSG (b) color chart CC
Figure 4: The distribution of the predicted and captured raw responses, (a) the predicted and
capture raw responses of CCSG, (b) the predicted and capture raw responses of CC.
The results from Figure 5 indicate that the distribution of the predicted camera raw
responses for both color charts of CCSG and CC closely overlaps with the distribution of the
captured raw responses with minor deviations observed in certain individual colors, which
means relatively good prediction results of the proposed method..
4.2. Response prediction-based spectral reconstruction
In this section, we present the results of camera raw response prediction-based spectral
reconstruction. Both the camera raw response of CCSG and CC color charts have been predicted
under different imaging settings to check the prediction accuracy. But for prediction-based
spectral reconstruction, using the CCSG as a training dataset and the CC as a testing target, we
compared the spectral reconstruction accuracy in two different situations by four and two
types of different spectral reconstruction algorithms. In the first situation (M1), we use the
captured raw response of CCSG to reconstruct the spectral reflectance of captured raw CC. In
the second situation (M2), we use the predicted raw response of CCSG to reconstruct the
spectral reflectance of captured raw CC. The selected spectral reconstruction methods are
Liang, OLS, Cao, and Kernel[17, 18, 19, 20]. The spectra reconstruction accuracy is evaluated by
RMSE and ∆E00, respectively. The results of M1 and M2 under five different groups of imaging
settings are summarized in Table 3 and Table 4, respectively. And the box plots of spectral
reconstruction errors in Table 3 and Table 4 are plotted in Figure 5.
Table 3
The average spectral reconstruction errors of M1 with four algorithms under five different
groups of imaging settings.
Liang OLS Cao Kernel
RMSE(%) ∆E00 RMSE(%) ∆E00 RMSE(%) ∆E00 RMSE(%) ∆E00
Group 1 2.30 2.17 3.04 2.27 2.97 4.39 3.23 2.50
Group 2 2.47 2.49 3.19 2.52 2.93 4.46 3.30 2.67
Group 3 2.38 2.20 3.01 2.29 2.63 3.78 3.14 2.46
Group 4 2.75 2.76 3.54 2.84 3.06 4.59 3.68 3.00
Group 5 3.24 3.18 3.84 3.2 3.51 4.93 4.04 3.34
Average 2.63 2.56 3.32 2.62 3.02 4.43 3.48 2.79
Table 4
The average spectral reconstruction errors of M2 with four algorithms under five different
groups of imaging settings.
Liang OLS Cao Kernel
RMSE(%) ∆E00 RMSE(%) ∆E00 RMSE(%) ∆E00 RMSE(%) ∆E00
Group 1 5.44 3.35 5.26 3.54 3.83 5.07 4.73 3.39
Group 2 5.68 3.46 5.41 3.65 3.97 5.40 4.88 3.49
Group 3 5.53 3.36 5.24 3.55 3.83 5.15 4.70 3.39
Group 4 6.71 4.21 6.54 4.38 5.03 6.44 5.96 4.17
Group 5 5.29 3.25 4.93 3.42 3.45 4.46 4.42 3.31
Average 5.73 3.53 5.48 3.71 4.02 5.30 4.94 3.55
Figure 5: Box plots of the spectral reconstruction errors, (a) RMSE of M1 with four algorithms,
(b) ∆E00 of M1 with four algorithms, (c) RMSE of M2 with four algorithms, (d) ∆E00 of M2 with
four algorithms.
From Table 3, Table 4, and Figure 5, it can be observed that for using the captured training
dataset to reconstruct the captured testing target (M1), the spectral reconstruction errors are
relatively good according to the experience. However, there are certain differences in spectral
reconstruction errors between different algorithms. Among them, Liang's method generally
shows the best spectral reconstruction for both RMSE and ∆E00, and Cao’s method shows the
worst colorimetric accuracy. For using the predicted training dataset to reconstruct the
captured testing target (M2), Liang’s method shows the highest RMSE of 5.73%, while Cao’s
method shows the lowest RMSE of 4.02%, but for color difference, the results of the two
methods are completely opposite, where Liang’s method shows the lowest ∆E00 of 3.53, while
Cao’s method shows the highest ∆E00 of 5.30.
The results in Table 3, Table 4, and Figure 5 illustrated that the performance of M2 is
slightly inferior to M1, but it still in a relatively good reconstruction accuracy. In addition, it
easy to infer that the proposed camera raw response prediction based method is sensitive to
the spectral reconstruction algorithms used in it, and different algorithms have different
performances in spectral and chromaticity aspects. To further inspection the spectral
reconstructed results, several spectral reflectance reconstructed by Cao’s method under M1
and M2 are compared with the measured results in Figure 6.
Figure 6: Compared the measured and several reconstructed spectral reflectance by Cao’s
method under M1 and M2: (a) 11th color patch in CC, (b) 16th color patch in CC, (c) 18th color
patch in CC, (d) 19th color patch in CC.
The results in Figure 6 show that the spectral reflectance reconstructed by Cao’s method under
M1 and M2 overlaps and closely matches the measured data. This demonstrates the potential
of the proposed method to address the dependency of spectral reconstruction techniques on
imaging conditions. Overall, the experimental results in this study highlight the potential of the
proposed method to achieve real-time spectral characterization of the camera and facilitate
practical applications of multispectral reconstruction technology in open environments.
5. Conclusions
This study introduces a novel approach to spectral reconstruction based on camera raw
response prediction to address the challenges posed by rapidly changing light sources,
illumination, and imaging parameters in an open environment. The method involves predicting
the camera raw response of the training dataset based on the digital imaging model,
characterizing the camera spectrally, and reconstructing the spectral reflectance of the testing
target using the predicted response and spectral reflectance of the training dataset. The
proposed method differs from existing research by correcting imaging conditions from an open
environment to a laboratory via illumination estimation. This method is more flexible for
practical applications. The experimental results have proven the effectiveness of this method.
In future research, the validation of the proposed method under different open environments
will be conducted to further investigate the feasibility of its performance. In addition, we will
conduct further optimization research on the proposed method and develop an integrated
application system.
Acknowledgements
This work was supported by National Natural Science Foundation of China (62305255),
Hubei Provincial Natural Science Foundation General Project (No.2022CFB537), Hubei
Provincial Department of Education Science and Technology Research Program Youth Talent
(No.Q20221706), and China Scholarship Council (202308420128).
References
[1] Liang J, Xin L, Zuo Z, et al, Research on the deep learning-based exposure invariant spectral
reconstruction method, J. Frontiers in Neuroscience 16 (2022): 1031546.
[2] Zhao Y, Po, L M, Yan Q, et al. Hierarchical regression network for spectral reconstruction
from RGB images, in: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition Workshops, 2020, pp. 422-423.
[3] Cai Y, Lin J, Lin Z, et al. Mst++: Multi-stage spectral-wise transformer for efficient spectral
reconstruction, in: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, 2022, pp: 745-755.
[4] Lin, Yi-Tun, and Graham D. Finlayson, On the optimization of regression-based spectral
reconstruction. J. Sensors 21.16 (2021): 5586.
[5] Shrestha R, Hardeberg J Y, Spectrogenic imaging: A novel approach to multispectral
imaging in an uncontrolled environment, J. Optics Express 22.8 (2014) 9123-9133.
[6] Khan H A, Thomas J B, Hardeberg J Y, et al, Illuminant estimation in multispectral imaging,
J. JOSA 34.7 (2017) 1085-1098.
[7] Khan H A, Thomas J B, Hardeberg J Y, et al, Multispectral camera as spatio-
spectrophotometer under uncontrolled illumination, J. Optics Express 27.2 (2019) 1051-
1070.
[8] Lin Y T, Finlayson G D. Exposure invariance in spectral reconstruction from rgb images, in:
Color and Imaging Conference. Society for Imaging Science and Technology, 2019, pp. 284-
289.
[9] Lin Y T, Finlayson G D. Physically plausible spectral reconstruction from RGB images, in:
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition
Workshops, 2020, pp. 532-533.
[10] Ibrahim A, Tominaga S, Horiuchi T, A spectral invariant representation of spectral
reflectance, J. Optical Review 18.2 (2011) 231-236.
[11] Liang J, Xin L, Cheng J, et al, Adaptive Weighted Spectral Reconstruction Method Against
Exposure Variation, J. Spectroscopy and spectral analysis 43.11 (2023) 3330-3338.
[12] Ramanath R, Snyder W E, Yoo Y, et al, Color image processing pipeline, J. IEEE Signal
Processing Magazine 22.1 (2005) 34-43.
[13] Farrell J E, Catrysse P B, Wandell B A, Digital camera simulation, J. Applied Optics 51.4
(2012) A80-A90.
[14] Qiu J, Xu H, Camera response prediction for various capture settings using the spectral
sensitivity and crosstalk model, J. Applied Optics 55.25 (2016) 6989-6999.
[15] Jiang J, Liu D, Gu J, et al, What is the space of spectral sensitivity functions for digital color
cameras?. In 2013 IEEE Workshop on Applications of Computer Vision (WACV), (2013) pp.
168-179.
[16] Tominaga, S, Shogo N, and Ryo O. Measurement and estimation of spectral sensitivity
functions for mobile phone cameras. Sensors 21.15 (2021): 4985.
[17] Liang J, Wan X, Optimized method for spectral reflectance reconstruction from camera
responses, J. Optics Express 25.23 (2017) 28273-28287.
[18] Connah D R, Hardeberg J Y. Spectral recovery using polynomial models, in: Color Imaging X:
Processing, Hardcopy, and Applications. SPIE, 2005, 5667, pp. 65-75.
[19] Cao B, Liao N, Cheng H, Spectral reflectance reconstruction from RGB images based on
weighting smaller color difference group, J. Color Research & Application 42.3 (2017) 327-
332.
[20] Xu P, Xu H, Diao C, et al, Self-training-based spectral image reconstruction for art paintings
with multispectral imaging, J. Applied optics 56.30 (2017) 8461-8470.