Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting Federico Grillini, Jean-Baptiste Thomas, and Sony George The Norwegian Colour and Visual Computing Laboratory, Norwegian University of Science and Technology, GjΓΈvik, Norway federico.grillini@gmail.com, jean.b.thomas@ntnu.no, sony.george@ntnu.no Abstract. Identifying the pigments and their abundances in the mix- tures of one artist’s masterpiece is of fundamental importance for the preservation of the artifact. The reflectance spectrum of mixtures of pigments can be described by modeling the spectral signature of each component, following different rules and physical laws. We analyze and invert nine different mixing models, in order to perform Spectral Un- mixing, using as targets two sets of mock-ups. Based on the results of the spectral reconstruction errors, we are able to point out that three models are best suited to describe the phenomenon: subtractive model, its derivation with extra parameters, and the linear model adapted with extra parameters. Keywords: Optical mixing models Β· Spectral Unmixing Β· Pigment map- ping. 1 Introduction The conservation of Cultural Heritage (CH) is a vital cornerstone on which mod- ern society is based upon, since it enables the current community to transmit the inestimable value of knowledge, traditions, uses, and art to the next genera- tions [1]. During the Age of Post-Enlightenment in the 18π‘‘β„Ž century, CH started to be valued, as the first museums started their activity in England [2], and the methods of conservation slowly developed, using the resources coming from fields of the natural sciences. At the present times, a large body of research focuses on non-invasive and non-disruptive techniques, to investigate the arti- facts without making contact with them, and without the extraction of samples. Techniques such as Infra-Red photography [3], X-Ray Fluorescence [4], Particle Induced X-ray Emission [5], Fourier Transform Infra-Red spectroscopy [6], Opti- cal Coherence Tomography [7] and Raman Spectroscopy [8] have been deployed to study the physical and chemical properties of the artifacts. Some of them rely on a punctual response, whereas others are scanning methods that provide an Copyright c 2020 for this paper by its authors. Use permitted under Creative Com- mons License Attribution 4.0 International (CC BY 4.0). Colour and Visual Com- puting Symposium 2020, GjΓΈvik, Norway, September 16-17, 2020. 2 F. Grillini et al. image of the object, carrying information about the specific feature investigated. The main purpose for which these methodologies are applied is to get the precise composition of an object; this information can help the conservators to adopt the best suitable procedure for flawless preservation. These enlisted methods can carry out examinations and provide results with a high degree of accuracy, but the instrumentation required is often quite expensive, and the acquisition sessions complex and long-lasting. This work focuses on Hyperspectral Imaging (HSI) in the Visible-Near In- frared (VNIR) region of the electromagnetic spectrum. The mixture of pigments in oil painting is studied. Historically, in oil painting, pigments in powder form are blended together with a fluid medium, called binder, usually linseed oil. The mixture is then applied on a stretched canvas or wooden support, previously primed, i.e. covered with a preparatory layer, usually made of gesso, to facilitate the drying process [9]. HSI does not generally allow the examination in depth of an object in the visible region [10], thus only the radiant light of the outer layers of the artifact can be observed. The paints applied to create paintings are usually mixtures of different pigments. The optical properties of the mixture will carry information about the properties of each one of the components, denominated endmembers or primaries. Retrieving the endmembers present in a mixture in their rela- tive concentrations boils down to an inversion problem called Spectral Unmix- ing (SU). In order to solve this task, several procedures have been proposed, with the main classification made between methods that exploit the existence of a spectral library containing the possible endmembers, and methods that re- quire the implementation of endmember extracting algorithms [11]. The model that eventually is inverted needs to describe the physical properties of the phe- nomenon that takes place when different pigments are mixed together. That is why we strongly believe that the success of SU depends on the accuracy of the mixing model on which it is based upon. In remote sensing applications, unmix- ing is often addressed inverting a linear model [12], assuming that the mixture mainly takes place at the camera level and is the result of optical blending. How- ever, when pigments are mixed together, the nature of the mixture is intimate, meaning that the single components are not discernible even with sophisticated imaging technologies. In close-range applications, the optical resolution is high enough to safely discard the optical blending at the camera level and consider only the intimate mixture. We analyze nine mixing models, and we invert them to perform SU. The investigations of the models are conducted by creating mockups of mixtures of different pigments in approximated known proportions, which are captured in an HS set-up. We then can observe that the linear model and subtractive model, modified to include extra parameters, describe with a good degree of accuracy the mixture of pigments in oil painting. The remainder of this paper is organized as follows: Sec.2 introduces the pro- posed mixing models, Sec.3 explains the strategies adopted in order to complete the task of spectral unmixing, Sec.4 describes the materials and methods ap- Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting 3 plied, Sec.5 provides an overview of the results, and finally, Sec.6 gathers a few concluding remarks. 2 Optical Mixing Models The radiant light incoming on a sensor is described by the Radiative Transfer Model, identifying two main components: an illuminant and an object. The ra- diation emitted by the light source, denominated Spectral Power Distribution (SPD), impinges on the surface of the object, which is assumed to be Lam- bertian. Part of the radiation is absorbed by the material, while the remaining is reflected according to the physical properties of the surface reflectance 𝜌(πœ†) and captured by the sensor. The radiance 𝐸(πœ†) impinging on the sensor can be expressed with the formula: 𝐸(πœ†) = 𝐿(πœ†) β‹… 𝜌(πœ†) (1) This quantity interacts with the sensor’s response functions, which can be the Camera Sensitivity Function (CSF) in the case of a single-lens reflex camera, to output a matrix of pixel values. For equal illuminants and sensor responses, 𝜌(πœ†) is the material-related quantity that allows the perception of different colours, and can be easily extracted in conditions of a controlled environment, by invert- ing the radiative transfer model. Mixing is defined when 𝑁 or more endmembers are combined together in different concentrations 𝛼. The concentration array 𝐢 = {𝛼1 , 𝛼2 , ..., 𝛼𝑁 } needs to comply with two constraints dictated by the physics: 1. Non-negativity Constraint (NC). Since negative concentrations don’t have any physical meaning, all of the concentrations must be equal to or greater than 0: βˆ€ 𝛼𝑖 β‰₯ 0, 𝑖 = 1, ..., 𝑁 (2) 2. Sum-to-one Constraint (SC). The sum of the concentrations must be equal to one. If this condition is not met, there would be missing or exceeding matter in the mix. βˆ‘ 𝑁 𝛼𝑖 = 1 (3) 𝑖=1 For some applications, it is possible to relax the constraints in order to introduce levels of error tolerance and have the algorithms work faster. However, in this research work, both NC and SC are treated as hard constraints. The output of the mixture is a new reflectance that carries information about the single endmembers and therefore should comply with the fundamental properties of spectra, such as being included in the interval [0, 1] and being a smooth curve. It is well known that when the mixing of pigments and dyes is treated, it is often referred to as a subtractive mixing. The choice of considering an additive model might then sound incoherent. Although the linear mixing model cannot 4 F. Grillini et al. accurately describe the intimate type of mixing that occurs at the pigments level [13], it has been extensively used to solve tasks such as spectral unmixing and pigment mapping [14], which is why this model has been addressed and labelled with the code 𝑀1 . A purely subtractive model, labelled 𝑀2 , treats images and media as fil- ters, accordingly to the transmittance model. The reflectance of the mixture is then obtained as the consecutive products of the single endmembers’ spectra, implementing the concentrations in a geometric mean fashion [15]. The transmittance model seems appropriate when studying the mixture tak- ing place on a canvas, since pigments have well-known properties of transmit- tance, especially in the NIR region. This is why for this study we decided to adapt the models 𝑀1 and 𝑀2 to the rules dictated by the Logarithmic Image Processing (LIP) framework [16]. LIP has a solid physical and human-vision ba- sis, which allows the completion of several image processing-related tasks with a good degree of perceptual fidelity. The main contribution is the re-thinking of an image 𝑓 (π‘₯, 𝑦) as an intensity filter, therefore having an intrinsic transmis- sion function 𝑇𝑓 (π‘₯, 𝑦). The adoption of this framework allows to never exceed the intensity boundaries [0, πœ‡] dictated by the encoding system adopted. In LIP, the basic operators of addition, multiplication by a scalar, and multiplication between images [17] are revisited and applied to fulfill tasks such as high dy- namic range, feature extraction, and noise removal [18]. The new operators are reported in Tab.1. Table 1: Operators of the Logarithmic Image Processing framework. By adopting the modifications to standard operators, the maximum encoding value πœ‡ is never exceeded. Name Symbol Equation Image 𝑓 𝑓 (π‘₯, 𝑦) = πœ‡(1 βˆ’ 𝑇𝑓 (π‘₯, 𝑦)) Addition β¨Ή 𝑓 β¨Ή 𝑔 = 𝑓 + 𝑔 βˆ’ π‘“πœ‡π‘” ( )πœ† Multiplication by a scalar πœ† ⨻ πœ† ⨻ 𝑓 = πœ‡ βˆ’ πœ‡ 1 βˆ’ π‘“πœ‡ 𝑓 ⋆ 𝑔 = πœ‘βˆ’1 (πœ‘(𝑓 ) β‹… πœ‘(𝑔)) ( ) Multiplication between images ⋆ πœ‘(π‘₯) = βˆ’πœ‡ β‹… 𝑙𝑛 1 βˆ’ πœ‡π‘₯ ( ( )) πœ‘βˆ’1 (π‘₯) = πœ‡ 1 βˆ’ 𝑒π‘₯𝑝 βˆ’ πœ‡π‘₯ The LIP framework, originally thought for 8-bit images with πœ‡ = 255, can be translated to reflectances with significant simplifications, since in this new case Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting 5 πœ‡ = 1. The models 𝑀3 and 𝑀4 are derived from 𝑀1 and 𝑀2 and are therefore called LIP additive and LIP subtractive, respectively. Purely additive and purely subtractive models are defined as ideal, and a variety of intermediate models between the two can be formulated. These new models rely on a parameter that explains the balance between an additive and a sub- tractive mixture. The mixing parameter 𝜏 can span between a value of 0, to represent a purely subtractive model, to a value of 1, which indicates a purely additive one. Three intermediate models are formulated in the literature: – 𝑀5 Yule-Nielsen model [19]. It has been proposed in the study of half-toning, since neither subtractive nor additive models could explain such colors. It approximates the subtractive models when 𝜏 approaches 0 asymptotically (when 𝜏 is exactly zero, the function goes to the infinity), – 𝑀6 Additive-Subtractive model [20]. It privileges the additive mixture, even if 𝜏 takes on values ≀ 0.5, – 𝑀7 Subtractive-Additive model [20]. Conversely to the previous one, it priv- ileges the subtractive mixtures. From a preliminary observation of the spectra we acquired, we noted a prop- erty that is never achievable by the models proposed so far. One common trait followed by these models is that the reflectance curve of a mixture is always in- cluded within the boundaries of the endmembers’ curves. In real-case scenarios, this is generally the case (Fig.1a), but there exist instances in which this rule is not followed (Fig.1b). For this reason, we propose 2 new models, labeled 𝑀8 0.7 1 BY Rb 0.6 B R Y 0.8 B 0.5 0.6 0.4 ( ) ( ) 0.3 0.4 0.2 0.2 0.1 0 0 400 500 600 700 800 900 1000 400 500 600 700 800 900 1000 [nm] [nm] (a) (b) Fig. 1: Comparison of acquired spectra. The endmembers are reported with dashed lines, whereas their mixtures in an approximated 1:1 ratio are represented with solid lines. (a): the reflectance of the mixture is included between the two endmembers. (b): the mixture’s spectrum overshoots the boundaries of the endmembers and therefore presents unpredictable (for models 𝑀1 through 𝑀7 ) high values in the NIR region. and 𝑀9 which are equal to 𝑀1 and 𝑀2 , except the fact that each component now will present an extra factor that can allow the mixture’s reflectance to go out-of-bound. These new factors are assumed to be pigment-related constants 6 F. Grillini et al. that can account for some degree of dominance in the mixture or some extra- absorbance/scattering, similar to Kubelka-Munk theory [21]. The formulae of the models investigated in this research work are reported in Table 2 Table 2: Proposed optical mixing models. The classification is based according to the nature of the models: additive (A), subtractive (S), hybrid (H). The models 𝑀6 and 𝑀7 are indeed hybrid, but present strong additive and subtractive tendencies, respectively. Label Name Formula Category βˆ‘π‘ 𝑀1 Additive π‘Œ = 𝑖=1 πœŒπ‘– 𝛼𝑖 A βˆπ‘ 𝛼 𝑀2 Subtractive π‘Œ = πœŒπ‘– 𝑖=1 𝑖 S βˆπ‘ 𝑀3 LIP additive π‘Œ = 1 βˆ’ 𝑖=1 (1 βˆ’ πœŒπ‘– )𝛼𝑖 A [ ∏ [ ]𝛼 ] 𝑁 𝑀4 LIP subtractive π‘Œ = 1 βˆ’ 𝑒π‘₯𝑝 βˆ’ 𝑖=1 βˆ’π‘™π‘œπ‘”(1 βˆ’ πœŒπ‘– ) 𝑖 S (βˆ‘ )1 𝑁 𝜏 𝑀5 Yule-Nielsen π‘Œ = 𝛼 𝜌𝜏 𝑖=1 𝑖 𝑖 H βˆ‘π‘ βˆπ‘ 𝛼 𝑀6 Add-Sub π‘Œ = 𝜏 𝑖=1 𝛼𝑖 πœŒπ‘– + (1 βˆ’ 𝜏) πœŒπ‘– 𝑖=1 𝑖 H/A (βˆ‘ ) (∏ ) 𝑁 𝑁 𝛼 (1βˆ’πœ) 𝑀7 Sub-Add π‘Œ = 𝛼 𝜌𝜏 𝑖=1 𝑖 𝑖 πœŒπ‘– 𝑖=1 𝑖 H/S βˆ‘π‘ 𝑀8 Linear extra π‘Œ = 𝑖=1 πœŒπ‘– 𝛼𝑖 π‘˜π‘– A βˆπ‘ 𝛼 𝑀9 Subtractive extra π‘Œ = 𝑖=1 πœŒπ‘– 𝑖 π‘˜π‘– S 3 Spectral Unmixing Spectral Unmixing is a task whose goal is to invert the following problem defini- tion (Eq.4) for the array of concentrations 𝐢, knowing the target spectrum π‘₯(πœ†) and the spectral library of endmembers 𝐸. At the same time, the constraints of NC and SC must be respected. βˆ‘ π‘ž π‘₯[𝑏⋅1] = 𝑓 (𝐸[π‘β‹…π‘ž] , 𝐢[π‘žβ‹…1] ) , βˆ€πΆ β‰₯ 0 , 𝐢=1 (4) The notation 𝑏 represents the number of spectral bands, while π‘ž is the number of endmembers contained in the library. The function 𝑓 will be in turn one of the proposed mixing models, so the algorithm should be able to invert a constrained non-linear function, in most of the instances. In this study, the algorithm that inverts the objective function is the Nelder-Mead optimization [22]. In order to solve the optimization problem, the objective function is slightly modified in or- der to contain a cost function between the target spectrum and the reconstructed Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting 7 one. The new objective function takes thus in input the target reflectance π‘₯, the spectral library 𝐸, and the reconstructed spectrum 𝑦̃, retrieving 𝐢 by maximiz- ing the Peak Signal to Noise Ratio (PSNR) between π‘₯ and 𝑦̃, complying with NC and SC. [ ] 𝑃𝑆𝑁𝑅 = 20 π‘™π‘œπ‘”10 [π‘šπ‘Žπ‘₯(π‘₯)] βˆ’ 10 π‘™π‘œπ‘”10 𝑀𝑆𝐸(π‘₯, 𝑦̃) (5) βˆ‘π‘ ̃𝑖 )2 𝑖=1 (π‘₯𝑖 βˆ’ 𝑦 𝑀𝑆𝐸(π‘₯, 𝑦̃) = (6) 𝑏 The assumption that we formulate hereby is that when the reconstructed spec- trum approaches the target with a high value of PSNR, it means that the correct endmembers have been selected, in the appropriate relative concentrations as well. This assumption bears within some risks, and overfitting is one of those. In this scenario, overfitting can happen when the target spectrum is reconstructed with a good degree of accuracy, but analyzing the concentration array we discover that a large number of endmembers are included in significant proportions. Since we performed the mixtures on canvas, we know that only a limited number of pigments are composing the ground truth, thus high PSNR values can sometimes hide a significant classification error. In order to tackle this issue, each unmixing algorithm produces an estimated label of the target mixture, according to the es- timated concentrations. If an endmember classifies with a concentration > 35%, its correspondent capital letter will be included in the estimation, while with a concentration > 5% a lower case letter is provided. To be selected as reliable, one model needs to produce high PSNR values and a correct label, which are anyway two traits highly correlated in most of the cases. 4 Materials and Methods For the investigation of the proposed models, two sets of mock-ups are created. Both sets are performed on pre-primed stretched canvases of size 27π‘₯22 π‘π‘š. This article reports the preliminary results of a larger work that eventually involves pigments in powder form. However, for the moment, only pigments contained in commercially available pre-binded tubes are considered. The first set of mockups includes the following primaries: Vermilion (R), Viridian green (G), Ultramarine Blue (B), Lemon Yellow (Y), and Titanium White (W). The exact chemical compositions of the tubes are not investigated, therefore the proportions of pigments and binding medium are unknown. The canvas was covered with a layer of white paint and after drying, 24 patches of dimensions 2, 5π‘₯2, 5 π‘π‘š each are painted, including 4 patches for the pure R, G, B, and Y paints. The remaining patches are some of the possible combinations of the primaries in ratios approximately 1:1 when the label reports two capital letters, and approximately 2:1 when one capital and one lower-case letter are reported. Fig.2 reports the original set photographed in a non-controlled environment and its graphical representation, obtained computing the color from the reflectance spectrum under the standard illuminant 𝐷65 . The second set is made up of 8 F. Grillini et al. (a) (b) Fig. 2: First set of mock-ups. (a): photograph taken in an uncontrolled environment. (b): graphical representation of the mock-ups. Each spectrum is plotted in the range [418, 963 π‘›π‘š], while the colors on which the spectra are plotted are computed in the range [420, 780 π‘›π‘š] assuming a 𝐷65 standard illuminant. Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting 9 2 canvases and 111 patches. This time a preparatory layer of white was not applied, and some of the tubes used are changed to include pigments with higher reflectance properties: Cerulean Blue (B), Titanium White (W), Scarlet Lake (R), Lemon Yellow (Y), Orange yellow (O), and Emerald Green (G). Mixtures of 3 pigments are included in a 1:1:1 ratio when all 3 letters are reported as capital, whereas an approximate 2:1:1 ratio is assumed when only one of the letter of the group is capital (Fig.3). Fig. 3: Second set of mock-ups. All the possible combinations of 6 pigments, in 5 concentration levels [0, 0.33, 0.5, 0.67, 1], are performed, producing 111 patches. Each mixture is annotated with its correspondent label. A push-broom hyperspectral camera HySpex VNIR-1800 produced by Norsko Elektro Optikk has been used to capture all the HS images included in this research work. This line scanner employs a diffraction grating and results in generating 186 images across the electromagnetic spectrum, from 400 π‘›π‘š to 1000 π‘›π‘š, at steps of approximately 3.19 π‘›π‘š. The acquisition distance is set to 30 π‘π‘š, which translates into a field of view of 10 π‘π‘š and allows to reach an optical resolution of approximately 0.06π‘šπ‘š. The canvases are illuminated by a halogen Smart Light 3900e produced by Illumination Technologies, guided on the scene via optical fibers, projecting lights at 45β—¦ with respect to the camera. At each acquisition, a Spectralon R calibration target with a known reflectance factor, is included in the scene. The target will serve to estimate the illuminant’s spectrum and to compute the reflectance at the pixel level. The software enables the user to select an option that performs radiometric correction, in order to 10 F. Grillini et al. treat sensor and dark current errors. Flat field correction is performed using the same Spectralon reference target captured during the acquisition [23]. When the spectral cubes are processed, slices are selected manually at the locations of the patches and averaged at each wavelength band, in order to obtain a mean radiance spectrum. The same is done at the spatial location of the Spectralon target, to retrieve the SPD of the illuminant. The reflectance of each patch is then obtained inverting Eq.1 and considering the Spectralon’s reflectance factor. The two acquired sets of reflectances undergo the same workflow but are treated independently. Two tasks are performed in order to study the proposed optical models: 1. Prediction of the concentrations: the information contained in the label of each mixture is exploited to obtain the relative proportions of the endmem- bers involved. This task can be seen as a facilitated unmixing, since the algorithm is forced to use a limited set of endmembers. 2. Unmixing: the a priori knowledge is not used and any endmember can be picked out of the provided spectral library. 5 Results The two sets of mockups are analyzed independently, although they are subject to the same examination process. First, the information contained in the ground truth label is used to compose a reduced spectral library, containing only the endmembers involved in the mixture. This task is referred to as prediction of the concentrations. When all the primaries are included in the spectral library, the task is then a full unmixing. When unmixing is performed, we are both interested in the accuracy of the classification and of the reconstruction. The aim is then to produce an estimation of the label for every analyzed spectrum. First Set of Mock-ups The first set included 24 mock-ups and was realized on a single canvas. Fig.4 reports the graphical representation of the canvas for both tasks of prediction and unmixing. On each patch 3 spectra are plotted: measurement (black), prediction (dark gray), unmixing reconstruction (light gray). It is observable that in most instances the latter two are generally faithful reconstructions of the ground truth, and almost overlap each other. This can be confirmed by looking at the RMSE and GFC values in Table 3. From this Table, it is observable that slightly better reconstructions are obtained with unmixing, indicating that the best models achieve more accurate reconstructions if they are allowed to select primaries originally not included in the ground truth. Each patch reports also the colors computed under illuminant 𝐷65 , from left to right: prediction, ground truth, unmixing. It is noticeable that the errors in the color computation do not follow any pattern, as sometimes the ground truth is darker Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting 11 Fig. 4: Graphical representation of the first set of mock-ups undergoing the tasks of prediction and unmixing. Each patch plots in black the ground truth, in dark gray the predicted spectrum when the endmembers involved are known, and in light gray the unmixing reconstruction. Colors from left to right refer to prediction, ground truth, unmixing. The labels indicate the ground truth (left) and estimated after unmixing (right). and some other it is brighter. Finally, the labels of each patch report the ground truth (first) and then the estimated label from the unmixing task. If we exclude the pure patches (R, G, B, Y), 15 instances detect all the primaries included, with only 3 of them matching the ground truth label perfectly, the remaining cases are able to select only 1 of the endmembers correctly (misclassifying the second one or not selecting it at all). A detailed overview of the results obtained with the first set is reported in Tab. 4. From this last Table, we can finally see which are the models that produced better results. In the prediction task, the linear extra model 𝑀8 resulted the most selected, followed by the purely subtractive model 𝑀2 and LIP subtractive 𝑀4 . When 𝑀8 is considered, the extra constants seem to be pigment-dependent, 12 F. Grillini et al. Table 3: Spectral metrics related to the tasks of prediction and unmixing on the first set of mock-ups. Slightly better results in all categories are obtained with unmixing. SET 1 Metric Mean Median Min Max 𝐩%𝟏𝟎 𝐩%πŸ—πŸŽ RMSE 0.0551 0.0455 0.0240 0.0971 0.0300 0.0855 PRED GFC 0.9870 0.9925 0.9670 0.9981 0.9676 0.9974 PSNR 25.86 26.85 20.26 32.39 21.36 30.50 Δ𝐸2000 15.02 14.14 5.57 28.30 6.63 24.74 RMSE 0.0453 0.0444 0.0152 0.0886 0.0203 0.0733 UNMIX GFC 0.9903 0.9953 0.9688 0.9991 0.9715 0.9985 PSNR 27.90 27.05 21.05 36.34 22.70 33.85 Δ𝐸2000 13.13 12.47 2.35 30.62 5.44 20.87 Table 4: Results of prediction and unmixing on the first set of mock-ups. The con- centrations 𝛼 in the prediction section refer only to the endmembers reported in the annotated ground truth, whereas in the unmixing section they refer to all the pri- maries included in the spectral library. In several cases, 3 endmembers are selected by the unmixing algorithm, although only 2 are originally contained in the ground truth. PRED UNMIXING Label π›ΌπŸ π›ΌπŸ PSNR Model 𝛼𝐑 𝛼𝐆 𝛼𝐁 π›Όπ˜ 𝛼𝐖 PSNR Model Est label Rw 0.81 0.19 26.5 𝑀4 0.71 0 0 0.15 0.14 26.8 𝑀4 Ryw Gw 0.76 0.24 28.6 𝑀4 0 0.83 0 0 0.17 28.8 𝑀9 Gw Rb 0.57 0.43 28.2 𝑀8 0.61 0 0.20 0.19 0 33.5 𝑀9 Rby Ry 0.86 0.14 22.8 𝑀8 0.68 0 0 0.32 0 24.7 𝑀9 Ry By 0.83 0.17 27.2 𝑀8 0 0.08 0.54 0.38 0 32.8 𝑀9 gBY Yg 0.06 0.94 23.3 𝑀8 0 0.72 0 0.28 0 25.7 𝑀9 Gy RW 0.54 0.46 21.7 𝑀8 0.59 0 0 0.28 0.13 25.4 𝑀9 Ryw GW 0.94 0.06 22.0 𝑀8 0 0.65 0 0.02 0.33 22.8 𝑀9 Gw RB 0.44 0.56 28.4 𝑀8 0.45 0 0.54 0 0 28.5 𝑀8 RB RY 0.80 0.20 23.1 𝑀8 0.82 0 0 0.18 0 23.1 𝑀8 Ry BY 0.42 0.58 28.0 𝑀2 0 0.05 0.49 0.46 0 32.3 𝑀9 BY YG 0.49 0.51 28.5 𝑀8 0 0.80 0 0.20 0 30.4 𝑀9 Gy Wr 0.50 0.50 23.8 𝑀4 0.42 0 0 0.21 0.37 26.4 𝑀9 RyW gW 0.92 0.08 21.0 𝑀8 0 0.91 0.03 0 0.06 21.1 𝑀8 Gw rB 0.30 0.70 29.5 𝑀8 0.36 0.01 0.48 0.15 0 36.3 𝑀9 RBy rY 0.70 0.30 20.3 𝑀8 0.48 0 0 0.51 0.01 22.6 𝑀9 RY bY 0.78 0.22 23.2 𝑀8 0.09 0 0.73 0.18 0 23.4 𝑀8 rBy yG 0.50 0.50 32.4 𝑀8 0 0.86 0 0.14 0 34.2 𝑀9 Gy YW 0.73 0.27 27.3 𝑀8 0 0.04 0 0.75 0.21 27.3 𝑀8 Yw BW 0.40 0.60 31.5 𝑀2 0 0.06 0.43 0 0.51 32.0 𝑀4 gBW especially for G and B, which present values of 3.58Β±0.26 and 1.54Β±0.17, respec- Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting 13 tively. In the unmixing task, the extra constants models 𝑀9 and 𝑀8 are mainly selected. The constants of 𝑀8 point out for specificity of pigments R and Y this time, with values 1.46 Β± 0.22 and 1.33 Β± 0.15. Concerning 𝑀9 , this model some- times is failed to be inverted, thus producing very poor results. However, when it is inverted correctly, the PSNR values reached are the best. The constants do not point out for material specificity, indeed the mean value of all constants is 1.23 Β± 0.17. Second Set of Mock-ups With 111 mock-ups, it is not possible to provide a graphical representation of the results. However, an indication of how the proposed models are selected and how they behave is presented in Fig.5 and Fig.6 for the tasks of prediction and unmixing, respectively. Once again, 𝑀8 resulted the most selected model when the primaries included in the mixture are known a priori, followed closely by 𝑀2 . The PSNR values are plotted as Gaussian functions in the (b) side of the figures. For the prediction task, 𝑀9 is by far the worst model, having its PSNR distri- bution out of scale. The Gaussian curves of all models except 𝑀8 are clustered in a tight region of PSNR, indicating that their behavior is very similar. Little differences in PSNR do not imply significant changes in the reconstructions, but with the workflow adopted, they can lead to different pigment classifications. 60 0.1 M1 M9 50 M2 0.08 M3 M4 40 M5 0.06 M6 N 30 M7 0.04 M8 20 5 10 PSNR 0.02 10 0 0 M1 M2 M3 M4 M5 M6 M7 M8 M9 10 15 20 25 30 35 40 PSNR (a) (b) Fig. 5: Analysis of the prediction task on the second set of mock-ups. (a): number of times each model is selected with the highest PSNR. 𝑀8 and 𝑀2 produce clearly the best results, while the intermediate models and 𝑀9 are never selected. (b): PSNR of each model expressed as normal distributions. In the unmixing task, the 3 most selected models are 𝑀2 , 𝑀8 and 𝑀9 . As stated before, 𝑀9 sometimes is failed to be inverted, thus is explained the flat normal distribution in Fig.6b. However, it is somewhat surprising that it gets selected in almost 1βˆ•3 of the instances. As we can see from the plot, the PSNR values that this model reaches are not amongst the highest of the lot, meaning that it is able to describe mixtures that are complex for all the other models. The parameters of models 𝑀8 and 𝑀9 for the unmixing task do not suggest a 14 F. Grillini et al. 40 0.1 M1 M2 0.08 30 M3 M4 0.06 M5 M6 N 20 0.04 M7 M8 10 M9 0.02 0 0 10 15 20 25 30 35 40 45 M1 M2 M3 M4 M5 M6 M7 M8 M9 PSNR (a) (b) Fig. 6: Analysis of the unmixing task. (a): number of times each model is selected with the highest PSNR. 𝑀2 , 𝑀8 and 𝑀9 are the most selected. All models except the intermediate 𝑀5 and 𝑀7 are selected at least once. (b): PSNR of each model expressed as normal distributions. material-specificity. As a matter of fact, 𝑀8 produces an overall 1.00 Β± 0.15 value for the constants, explaining the fact that the linear model describes fairly well the mixtures, and just needs little adjustments to reach better reconstructions. The same goes for model 𝑀9 , which presents an overall value for the constants of 1.16 Β± 0.08. In the case of prediction of concentrations, the values for 𝑀8 and 𝑀9 are 0.97Β±1.00 and 0.2Β±0.65, respectively. The higher variance indicates instability in predicting the mixtures correctly. By observing Table 5 for the spectral metrics we can indeed notice that better reconstruction results are achieved in the task of unmixing. The better values of the spectral metrics, and the fact that 𝑀8 and 𝑀9 are unstable in the prediction and not in the unmixing, leads us to state that generally, more pigments than reported in the ground truth are needed, in order to obtain satisfactory results in terms of spectral reconstruction. In fact, we can observe that the estimated labels (not reported for readability) present more pigments than the ground truth in 24% of the instances. 6 Conclusion In this paper, we presented an investigation of optical mixing models in the specific case of oil painting. Nine models are analyzed by performing the task of predicting the concentrations of the primaries using prior knowledge, as well as the task of unmixing, on two sets of mock-ups realized for the occasion. The models that describe best each mixture are retrieved, comparing the PSNR of each spectral reconstruction, while keeping an eye to the newly produced la- bel. The subtractive model and the linear model adapted with extra parameters resulted the best in the task of prediction, whereas both, plus the subtractive model adapted with extra parameters, produced the best results in the unmix- ing task. The role of the parameters needs to be further evaluated, but from Linear, Subtractive and Logarithmic Optical Mixing Models in Oil Painting 15 Table 5: Spectral metrics related to the tasks of prediction and unmixing performed on the second set of mock-ups. The improvement of the metrics in the unmixing task is more significant than the one registered in the case of the first set, supporting the state- ment that the analysed models require generally more endmembers than the number reported in the annotated labels to obtain more accurate spectral reconstructions. SET 2 Metric Mean Median Min Max 𝐩%𝟏𝟎 𝐩%πŸ—πŸŽ RMSE 0.0492 0.0490 0.0076 0.1418 0.0171 0.0806 PRED GFC 0.9932 0.9950 0.9698 0.9999 0.9847 0.9995 PSNR 27.55 26.19 16.96 42.43 21.87 35.36 Δ𝐸2000 8.73 6.51 0.29 32.56 1.37 18.84 RMSE 0.0380 0.0352 0.0068 0.0942 0.0134 0.0687 UNMIX GFC 0.9956 0.9970 0.9834 0.9999 0.9882 0.9997 PSNR 29.76 29.07 20.52 43.29 23.26 37.47 Δ𝐸2000 7.20 5.70 0.24 28.80 1.10 17.30 our results, it does not suggests a pigment-specificity for both the additive and subtractive models. The aim for future works will regard the performing of mixtures with prior information on their concentrations of pigments in powder form, as well as the implementation of more complex models that can include multiple layers of a painting, and the application on hyperspectral images in pixel-based investiga- tions. References 1. Conference, U.G.: Draft Medium-term Plan, 1990-1995: General Conference, Twenty-fifth Session, Paris, 1989. Paris, France: Unesco (1989) 2. Abt, J.: The origins of the public museum. A companion to museum studies (2006) 115–134 3. Strojnik, M., Paez, G., Ortega, A.: Near IR diodes as illumination sources to remotely detect under-drawings on century-old paintings. In: 22nd Congress of the International Commission for Optics: Light for the Development of the World. Volume 8011., International Society for Optics and Photonics (2011) 801177 4. Mantler, M., Schreiner, M.: X-ray fluorescence spectrometry in art and archaeology. X-Ray Spectrometry: An International Journal 29 (2000) 3–17 5. Grassi, N., Migliori, A., MandoΜ€, P., Calvo Del Castillo, H.: Differential pixe measurements for the stratigraphic analysis of the painting madonna dei fusi by leonardo da vinci. X-Ray Spectrometry: An International Journal 34 (2005) 306– 309 6. Franquelo, M.L., Duran, A., Herrera, L.K., De Haro, M.J., Perez-Rodriguez, J.: Comparison between micro-raman and micro-ftir spectroscopy techniques for the characterization of pigments from southern spain cultural heritage. Journal of Molecular structure 924 (2009) 404–412 16 F. Grillini et al. 7. Targowski, P., Iwanicka, M.: Optical coherence tomography: its role in the non- invasive structural examination and conservation of cultural heritage objectsβ€”a review. Applied Physics A 106 (2012) 265–277 8. Garcı́a-Bucio, M.A., Casanova-González, E., Ruvalcaba-Sil, J.L., Arroyo-Lemus, E., Mitrani-Viggiano, A.: Spectroscopic characterization of sixteenth century panel painting references using raman, surface-enhanced raman spectroscopy and helium- raman system for in situ analysis of ibero-american colonial paintings. Philosoph- ical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374 (2016) 20160051 9. Wrapson, L., Rose, J., Miller, R., Bucklow, S.: In Artists’ Footsteps, the recon- struction of pigments and paintings. 1 edn. Archetype Publications (2012) 10. Cucci, C., Casini, A., Picollo, M., Stefani, L.: Extending hyperspectral imaging from vis to nir spectral regions: a novel scanner for the in-depth analysis of poly- chrome surfaces. In: Optics for Arts, Architecture, and Archaeology IV. Volume 8790., International Society for Optics and Photonics (2013) 879009 11. Bioucas-Dias, J.M., Plaza, A., Dobigeon, N., Parente, M., Du, Q., Gader, P., Chanussot, J.: Hyperspectral unmixing overview: Geometrical, statistical, and sparse regression-based approaches. IEEE journal of selected topics in applied earth observations and remote sensing 5 (2012) 354–379 12. Keshava, N., Mustard, J.F.: Spectral unmixing. IEEE signal processing magazine 19 (2002) 44–57 13. Zhao, Y.: Image segmentation and pigment mapping of cultural heritage based on spectral imaging. PhD thesis, Rochester Institute of Technology, http://scholarworks.rit.edu/theses (2008) 14. Deborah, H., George, S., Hardeberg, J.Y.: Pigment mapping of the scream (1893) based on hyperspectral imaging. In: International Conference on image and Signal processing, Springer (2014) 247–256 15. Burns, S.A.: Subtractive color mixture computation. arXiv preprint arXiv:1710.06364 (2017) 16. Jourlin, M., Pinoli, J.C.: Logarithmic image processing: the mathematical and physical framework for the representation and processing of transmitted images. In: Advances in imaging and electron physics. Volume 115. Elsevier (2001) 129–196 17. Panetta, K., Wharton, E., Agaian, S.: Parameterization of logarithmic image pro- cessing models. IEEE Tran. Systems, Man, and Cybernetics, Part A: Systems and Humans (2007) 1–12 18. Pinoli, J.C.: Modélisation et traitment des images logarithmiques: Théorie et ap- plications fondamentales. Report No.6 (1992) 19. Yule, J., Nielsen, W.: The penetration of light into paper and its effect on halftone reproduction. In: Proc. TAGA. Volume 3. (1951) 65–76 20. Simonot, L., Hébert, M.: Between additive and subtractive color mixings: inter- mediate mixing models. JOSA A 31 (2014) 58–66 21. Yang, L., Kruse, B.: Revised kubelka–munk theory. i. theory and application. JOSA A 21 (2004) 1933–1941 22. Nelder, J.A., Mead, R.: A simplex method for function minimization. The com- puter journal 7 (1965) 308–313 23. Pillay, R., Hardeberg, J.Y., George, S.: Hyperspectral imaging of art: Acquisition and calibration workflows. Journal of the American Institute for Conservation 58 (2019) 3–15