Identifying the pigments and their abundances in the mixtures of one artist's masterpiece is of fundamental importance for the preservation of the artifact. The re ectance spectrum of mixtures of pigments can be described by modeling the spectral signature of each component, following di erent rules and physical laws. We analyze and invert nine di erent mixing models, in order to perform Spectral Unmixing, using as targets two sets of mock-ups. Based on the results of the spectral reconstruction errors, we are able to point out that three models are best suited to describe the phenomenon: subtractive model, its derivation with extra parameters, and the linear model adapted with extra parameters.
The conservation of Cultural Heritage (CH) is a vital cornerstone on which
modern society is based upon, since it enables the current community to transmit
the inestimable value of knowledge, traditions, uses, and art to the next
generations [
This work focuses on Hyperspectral Imaging (HSI) in the Visible-Near
Infrared (VNIR) region of the electromagnetic spectrum. The mixture of pigments
in oil painting is studied. Historically, in oil painting, pigments in powder form
are blended together with a uid medium, called binder, usually linseed oil. The
mixture is then applied on a stretched canvas or wooden support, previously
primed, i.e. covered with a preparatory layer, usually made of gesso, to facilitate
the drying process [
HSI does not generally allow the examination in depth of an object in the
visible region [
We analyze nine mixing models, and we invert them to perform SU. The investigations of the models are conducted by creating mockups of mixtures of di erent pigments in approximated known proportions, which are captured in an HS set-up. We then can observe that the linear model and subtractive model, modi ed to include extra parameters, describe with a good degree of accuracy the mixture of pigments in oil painting.
The remainder of this paper is organized as follows: Sec.2 introduces the proposed mixing models, Sec.3 explains the strategies adopted in order to complete the task of spectral unmixing, Sec.4 describes the materials and methods applied, Sec.5 provides an overview of the results, and nally, Sec.6 gathers a few concluding remarks. 2
The radiant light incoming on a sensor is described by the Radiative Transfer Model, identifying two main components: an illuminant and an object. The radiation emitted by the light source, denominated Spectral Power Distribution (SPD), impinges on the surface of the object, which is assumed to be Lambertian. Part of the radiation is absorbed by the material, while the remaining is re ected according to the physical properties of the surface re ectance . / and captured by the sensor. The radiance E. / impinging on the sensor can be expressed with the formula:
E. / = L. / . / This quantity interacts with the sensor's response functions, which can be the Camera Sensitivity Function (CSF) in the case of a single-lens re ex camera, to output a matrix of pixel values. For equal illuminants and sensor responses, . / is the material-related quantity that allows the perception of di erent colours, and can be easily extracted in conditions of a controlled environment, by inverting the radiative transfer model.
Mixing is de ned when N or more endmembers are combined together in di erent concentrations . The concentration array C = ^ 1; 2; :::; N ` needs to comply with two constraints dictated by the physics: 1. Non-negativity Constraint (NC). Since negative concentrations don't have any physical meaning, all of the concentrations must be equal to or greater than 0:
Å i g 0; i = 1; :::; N 2. Sum-to-one Constraint (SC). The sum of the concentrations must be equal to one. If this condition is not met, there would be missing or exceeding matter in the mix.
N É i = 1 i=1 For some applications, it is possible to relax the constraints in order to introduce levels of error tolerance and have the algorithms work faster. However, in this research work, both NC and SC are treated as hard constraints. The output of the mixture is a new re ectance that carries information about the single endmembers and therefore should comply with the fundamental properties of spectra, such as being included in the interval [0; 1] and being a smooth curve.
It is well known that when the mixing of pigments and dyes is treated, it is
often referred to as a subtractive mixing. The choice of considering an additive
model might then sound incoherent. Although the linear mixing model cannot
(1)
(2)
(3)
accurately describe the intimate type of mixing that occurs at the pigments level
[
A purely subtractive model, labelled M2, treats images and media as
lters, accordingly to the transmittance model. The re ectance of the mixture is
then obtained as the consecutive products of the single endmembers' spectra,
implementing the concentrations in a geometric mean fashion [
The transmittance model seems appropriate when studying the mixture
taking place on a canvas, since pigments have well-known properties of
transmittance, especially in the NIR region. This is why for this study we decided to
adapt the models M1 and M2 to the rules dictated by the Logarithmic Image
Processing (LIP) framework [
The LIP framework, originally thought for 8-bit images with = 255, can be translated to re ectances with signi cant simpli cations, since in this new case = 1. The models M3 and M4 are derived from M1 and M2 and are therefore called LIP additive and LIP subtractive, respectively.
Purely additive and purely subtractive models are de ned as ideal, and a variety
of intermediate models between the two can be formulated. These new models
rely on a parameter that explains the balance between an additive and a
subtractive mixture. The mixing parameter can span between a value of 0, to
represent a purely subtractive model, to a value of 1, which indicates a purely
additive one. Three intermediate models are formulated in the literature:
{ M5 Yule-Nielsen model [
From a preliminary observation of the spectra we acquired, we noted a
property that is never achievable by the models proposed so far. One common trait
followed by these models is that the re ectance curve of a mixture is always
included within the boundaries of the endmembers' curves. In real-case scenarios,
this is generally the case (Fig.1a), but there exist instances in which this rule
is not followed (Fig.1b). For this reason, we propose 2 new models, labeled M8
1
0.8
0.6
)
(
0.4
and M9 which are equal to M1 and M2, except the fact that each component
now will present an extra factor that can allow the mixture's re ectance to go
out-of-bound. These new factors are assumed to be pigment-related constants
that can account for some degree of dominance in the mixture or some
extraabsorbance/scattering, similar to Kubelka-Munk theory [
q
x[b 1] = f .E[b q]; C[q 1]/ ; ÅC g 0 ; É C = 1
(4)
The notation b represents the number of spectral bands, while q is the number of
endmembers contained in the library. The function f will be in turn one of the
proposed mixing models, so the algorithm should be able to invert a constrained
non-linear function, in most of the instances. In this study, the algorithm that
inverts the objective function is the Nelder-Mead optimization [
PSNR = 20 log10 [max.x/] * 10 log10 MSE.x; y/
MSE.x; y/ = ³ib=1.xi * yi/2 b The assumption that we formulate hereby is that when the reconstructed spectrum approaches the target with a high value of PSNR, it means that the correct endmembers have been selected, in the appropriate relative concentrations as well. This assumption bears within some risks, and over tting is one of those. In this scenario, over tting can happen when the target spectrum is reconstructed with a good degree of accuracy, but analyzing the concentration array we discover that a large number of endmembers are included in signi cant proportions. Since we performed the mixtures on canvas, we know that only a limited number of pigments are composing the ground truth, thus high PSNR values can sometimes hide a signi cant classi cation error. In order to tackle this issue, each unmixing algorithm produces an estimated label of the target mixture, according to the estimated concentrations. If an endmember classi es with a concentration > 35~, its correspondent capital letter will be included in the estimation, while with a concentration > 5~ a lower case letter is provided. To be selected as reliable, one model needs to produce high PSNR values and a correct label, which are anyway two traits highly correlated in most of the cases. (5) (6) 4
For the investigation of the proposed models, two sets of mock-ups are created. Both sets are performed on pre-primed stretched canvases of size 27x22 cm. This article reports the preliminary results of a larger work that eventually involves pigments in powder form. However, for the moment, only pigments contained in commercially available pre-binded tubes are considered. The rst set of mockups includes the following primaries: Vermilion (R), Viridian green (G), Ultramarine Blue (B), Lemon Yellow (Y), and Titanium White (W). The exact chemical compositions of the tubes are not investigated, therefore the proportions of pigments and binding medium are unknown. The canvas was covered with a layer of white paint and after drying, 24 patches of dimensions 2; 5x2; 5 cm each are painted, including 4 patches for the pure R, G, B, and Y paints. The remaining patches are some of the possible combinations of the primaries in ratios approximately 1:1 when the label reports two capital letters, and approximately 2:1 when one capital and one lower-case letter are reported. Fig.2 reports the original set photographed in a non-controlled environment and its graphical representation, obtained computing the color from the re ectance spectrum under the standard illuminant D65. The second set is made up of
(a) (b) Fig. 2: First set of mock-ups. (a): photograph taken in an uncontrolled environment. (b): graphical representation of the mock-ups. Each spectrum is plotted in the range [418; 963 nm], while the colors on which the spectra are plotted are computed in the range [420; 780 nm] assuming a D65 standard illuminant. 2 canvases and 111 patches. This time a preparatory layer of white was not applied, and some of the tubes used are changed to include pigments with higher re ectance properties: Cerulean Blue (B), Titanium White (W), Scarlet Lake (R), Lemon Yellow (Y), Orange yellow (O), and Emerald Green (G). Mixtures of 3 pigments are included in a 1:1:1 ratio when all 3 letters are reported as capital, whereas an approximate 2:1:1 ratio is assumed when only one of the letter of the group is capital (Fig.3).
A push-broom hyperspectral camera HySpex VNIR-1800 produced by Norsko
Elektro Optikk has been used to capture all the HS images included in this
research work. This line scanner employs a di raction grating and results in
generating 186 images across the electromagnetic spectrum, from 400 nm to
1000 nm, at steps of approximately 3:19 nm. The acquisition distance is set to
30 cm, which translates into a eld of view of 10 cm and allows to reach an
optical resolution of approximately 0:06mm. The canvases are illuminated by a
halogen Smart Light 3900e produced by Illumination Technologies, guided on
the scene via optical bers, projecting lights at 45ý with respect to the camera.
At each acquisition, a Spectralon R calibration target with a known re ectance
factor, is included in the scene. The target will serve to estimate the illuminant's
spectrum and to compute the re ectance at the pixel level. The software enables
the user to select an option that performs radiometric correction, in order to
treat sensor and dark current errors. Flat eld correction is performed using the
same Spectralon reference target captured during the acquisition [
When the spectral cubes are processed, slices are selected manually at the locations of the patches and averaged at each wavelength band, in order to obtain a mean radiance spectrum. The same is done at the spatial location of the Spectralon target, to retrieve the SPD of the illuminant. The re ectance of each patch is then obtained inverting Eq.1 and considering the Spectralon's re ectance factor.
The two acquired sets of re ectances undergo the same work ow but are treated independently. Two tasks are performed in order to study the proposed optical models: 1. Prediction of the concentrations: the information contained in the label of each mixture is exploited to obtain the relative proportions of the endmembers involved. This task can be seen as a facilitated unmixing, since the algorithm is forced to use a limited set of endmembers. 2. Unmixing: the a priori knowledge is not used and any endmember can be picked out of the provided spectral library. 5
The two sets of mockups are analyzed independently, although they are subject to the same examination process. First, the information contained in the ground truth label is used to compose a reduced spectral library, containing only the endmembers involved in the mixture. This task is referred to as prediction of the concentrations. When all the primaries are included in the spectral library, the task is then a full unmixing. When unmixing is performed, we are both interested in the accuracy of the classi cation and of the reconstruction. The aim is then to produce an estimation of the label for every analyzed spectrum.
The rst set included 24 mock-ups and was realized on a single canvas. Fig.4 reports the graphical representation of the canvas for both tasks of prediction and unmixing. On each patch 3 spectra are plotted: measurement (black), prediction (dark gray), unmixing reconstruction (light gray).
It is observable that in most instances the latter two are generally faithful reconstructions of the ground truth, and almost overlap each other. This can be con rmed by looking at the RMSE and GFC values in Table 3. From this Table, it is observable that slightly better reconstructions are obtained with unmixing, indicating that the best models achieve more accurate reconstructions if they are allowed to select primaries originally not included in the ground truth. Each patch reports also the colors computed under illuminant D65, from left to right: prediction, ground truth, unmixing. It is noticeable that the errors in the color computation do not follow any pattern, as sometimes the ground truth is darker and some other it is brighter. Finally, the labels of each patch report the ground truth ( rst) and then the estimated label from the unmixing task. If we exclude the pure patches (R, G, B, Y), 15 instances detect all the primaries included, with only 3 of them matching the ground truth label perfectly, the remaining cases are able to select only 1 of the endmembers correctly (misclassifying the second one or not selecting it at all). A detailed overview of the results obtained with the rst set is reported in Tab. 4.
From this last Table, we can nally see which are the models that produced better results. In the prediction task, the linear extra model M8 resulted the most selected, followed by the purely subtractive model M2 and LIP subtractive M4. When M8 is considered, the extra constants seem to be pigment-dependent, especially for G and B, which present values of 3:58 , 0:26 and 1:54 , 0:17, respec
R G B Y
W PSNR
M4 M4 M8 M8 M8 M8 M8 M8 M8 M8 M2 M8 M4 M8 M8 M8 M8 M8 M8 M2 tively. In the unmixing task, the extra constants models M9 and M8 are mainly selected. The constants of M8 point out for speci city of pigments R and Y this time, with values 1:46 , 0:22 and 1:33 , 0:15. Concerning M9, this model sometimes is failed to be inverted, thus producing very poor results. However, when it is inverted correctly, the PSNR values reached are the best. The constants do not point out for material speci city, indeed the mean value of all constants is 1:23 , 0:17.
With 111 mock-ups, it is not possible to provide a graphical representation of the results. However, an indication of how the proposed models are selected and how they behave is presented in Fig.5 and Fig.6 for the tasks of prediction and unmixing, respectively. Once again, M8 resulted the most selected model when the primaries included in the mixture are known a priori, followed closely by M2. The PSNR values are plotted as Gaussian functions in the (b) side of the gures. For the prediction task, M9 is by far the worst model, having its PSNR distribution out of scale. The Gaussian curves of all models except M8 are clustered in a tight region of PSNR, indicating that their behavior is very similar. Little di erences in PSNR do not imply signi cant changes in the reconstructions, but with the work ow adopted, they can lead to di erent pigment classi cations. 60 50 40 N30 20 10 0 0.1 0.08 0.06 0.04 0.02 010
MMMMMMMM43526187
M9 PSNR 10 5 M1 M2 M3 M4 M5 M6 M7 M8 M9 25
PSNR 15 20 30 35 40 (a) (b) Fig. 5: Analysis of the prediction task on the second set of mock-ups. (a): number of times each model is selected with the highest PSNR. M8 and M2 produce clearly the best results, while the intermediate models and M9 are never selected. (b): PSNR of each model expressed as normal distributions.
In the unmixing task, the 3 most selected models are M2, M8 and M9. As stated before, M9 sometimes is failed to be inverted, thus is explained the at normal distribution in Fig.6b. However, it is somewhat surprising that it gets selected in almost 1_3 of the instances. As we can see from the plot, the PSNR values that this model reaches are not amongst the highest of the lot, meaning that it is able to describe mixtures that are complex for all the other models. The parameters of models M8 and M9 for the unmixing task do not suggest a
M1 M2 M3 M4 M5 M6 M7 M8 M9 010 15 20 25 30 PSNR 35 40 45 (a) (b) Fig. 6: Analysis of the unmixing task. (a): number of times each model is selected with the highest PSNR. M2, M8 and M9 are the most selected. All models except the intermediate M5 and M7 are selected at least once. (b): PSNR of each model expressed as normal distributions. material-speci city. As a matter of fact, M8 produces an overall 1:00 , 0:15 value for the constants, explaining the fact that the linear model describes fairly well the mixtures, and just needs little adjustments to reach better reconstructions. The same goes for model M9, which presents an overall value for the constants of 1:16 , 0:08. In the case of prediction of concentrations, the values for M8 and M9 are 0:97,1:00 and 0:2,0:65, respectively. The higher variance indicates instability in predicting the mixtures correctly. By observing Table 5 for the spectral metrics we can indeed notice that better reconstruction results are achieved in the task of unmixing. The better values of the spectral metrics, and the fact that M8 and M9 are unstable in the prediction and not in the unmixing, leads us to state that generally, more pigments than reported in the ground truth are needed, in order to obtain satisfactory results in terms of spectral reconstruction. In fact, we can observe that the estimated labels (not reported for readability) present more pigments than the ground truth in 24~ of the instances. 6
In this paper, we presented an investigation of optical mixing models in the speci c case of oil painting. Nine models are analyzed by performing the task of predicting the concentrations of the primaries using prior knowledge, as well as the task of unmixing, on two sets of mock-ups realized for the occasion. The models that describe best each mixture are retrieved, comparing the PSNR of each spectral reconstruction, while keeping an eye to the newly produced label. The subtractive model and the linear model adapted with extra parameters resulted the best in the task of prediction, whereas both, plus the subtractive model adapted with extra parameters, produced the best results in the unmixing task. The role of the parameters needs to be further evaluated, but from our results, it does not suggests a pigment-speci city for both the additive and subtractive models.
The aim for future works will regard the performing of mixtures with prior information on their concentrations of pigments in powder form, as well as the implementation of more complex models that can include multiple layers of a painting, and the application on hyperspectral images in pixel-based investigations.