Introduction

Artworks are usually varnished for the purpose of protection. Although successful in their main purpose, with time, this application can change the visual qualities of artworks [1,2,3,4]. Therefore, physically removing the aged varnish in order to reestablish the original appearance of the artwork becomes of great importance [5,6]. There have been two major approaches to clean artwork, namely, physical and virtual cleaning. In the physical approach, the conservator physically removes the varnish layer using a solvent and gel system. These types of cleaning are very time-consuming and can also be detrimental to the artwork [7,8,9]. Virtual cleaning, on the other hand, refers to the outcome of the physical approach using simulation. Virtual cleaning could provide the conservator with the likely appearance of the cleaned artwork, helping them to see if the physical cleaning is necessary and potentially guide their work.

Most of the studies done in the area of virtual cleaning are based on first cleaning a small part of the painting physically. and they use an RGB image of the painting before and after cleaning. Using that small part, for which they have data belonging to both cleaned and unclean state, they attempt to virtually clean the entire painting producing a visualization of the cleaned work. They typically do that through fitting some type of regression to the data obtained from the small area before and after cleaning. They then apply the same regression model to the rest of the painting which leads to the artwork being virtually cleaned [10,11]. Papas and Pitas (2000) stated that the RGB color space of the camera does not work well and they proposed to use the CIELAB color space instead, asserting that CIELAB works better than RGB color space. Virtually cleaning the Mona Lisa was another breakthrough in the field of virtual cleaning [12]. Having access to the classical paints used in 16th century Italy, the authors were able to make a varnished and unvarnished color chart out of them. They were able to extract the relationship between the varnished and unvarnished color chart enabling them to estimate the unvarnished version of the Mona Lisa [12]. Palomero and Soriano (2011) developed the first neural network approach trying to virtually clean artworks [13]. They also first cleaned a part of the artwork and then trained a shallow network using that small part. They then used the same model to clean the rest of the artwork [13]. Trumpy, et al. (2015) developed the first physics-based model in order to virtually clean artworks [14] through making a few simplifying assumptions, such as that a dark site on the painting is a "perfect" black that absorbs all incident light (perfect meaning not grayish) and the varnish spectral reflectance is wavelength independent. Through first finding the darkest and lightest part of the painting and cleaning them, they were able to estimate the spectral transmittance of the varnish layer which would be used to estimate the cleaned spectral reflectance of the entire painting [14]. Kirchner, et al. (2018) used Kubelka-Munk trying to estimate the virtually cleaned artworks [15]. In order to do that, they first characterized the varnish layer through cleaning the artworks at a few spots that appeared white allowing them to compute the spectral transmittance of the varnish. Characterizing the varnish layer enabled them to estimate the cleaned version of the whole painting [15]. Linhares, et al. (2020) did a similar work as [15] through characterizing the varnish layer first. However, they characterized the varnish layer through removing the whole varnish and measuring the spectral reflectance of the painting before and after varnish removal [16]. The latest work in the area of virtual cleaning of artworks belong to Maali Amiri and Messinger (2021) [17]. They first developed a Convolutional Neural Network (CNN) model. The network was trained on images of natural scenes and humans that were artificially yellowed mimicking the impact that varnish has on the artwork visually. They were able to visualize the cleaned version of artworks using their proposed CNN model in a very acceptable manner [17]. The methods proposed until now suffer from a few limitations, namely, the requirement to specify the perfect black and white regions on the painting, the need to have access to spectral data, generalizability of the method to other works, and the need to have access to a large set of data for training.

In this work, we propose a Deep Generative Network (DGN) to virtually clean a partially cleaned artwork. The generative model we use herein has been used in the area of remote sensing for the purpose of denoising of the hyperspectral image and single image super-resolution [18,19]. The authors developed a convolutional generative network that was able to take in a noise cube and output a super-resolved remotely sensed image. The network is deep and symmetrical and has also borrowed the idea of skip connections from U-Net enabling it to use residual information as best as possible. In this work, we have modified the network to fit our purpose. Instead of feeding the network with a random noise image, we feed in the RGB image of the uncleaned artwork. To be more specific, we have information of a small area of the painting before and after cleaning. The RGB image of the artwork is first changed into CIELAB, and the a*b* channels are used to train the network. The loss function, on the other hand, is computed between the uncleaned and the corresponding cleaned area of the artwork (the small area for which we have access to both cleaned and uncleaned data). The model is tested on Macbeth ColorChecker and the Mona Lisa that are partially cleaned. The results show that our approach here has done a better job compared to the model proposed by [17] when it comes to the Mona Lisa but has done slightly worse when compared to [17] for the Macbeth ColorChecker. Overall, the method proposed herein is more applicable to a real situation where the conservator has no access to a large set of data with which to train the model. Comparing our model to that of [17] seems fair as in their paper they showed that their model had outperformed the only physics-based model proposed for artwork virtual cleaning [14,17].

This paper is laid out as follows: the next section will present the specifications of the data, while the method will be explained in a more detailed manner along with the evaluation metrics and experimental environment. After that, results are presented along with discussions in the next section. Finally the conclusions are presented.

Methodology

In this section, the data used are explained and the proposed algorithm is described in detail.

Data

One of the datasets used to test the proposed method is the Macbeth ColorChecker spectral reflectance data. The spectral reflectances were artificially yellowed using the same formula suggested by [17] in the spectral domain. The artificially yellowed spectral reflectance mimics the visual impact varnish has on the painting. So, due to the Macbeth ColorChecker having a wide range of colors along with neutral patches, we use it as an initial test for our approach. The Macbeth ColorChecker is simulated in a way that it is "varnished" with a layer of a particular spectral reflectance and transmittance (generally speaking, varnish is yellow, and its spectral reflectance and transmittance should represent that [14,15]), as explained by [17]. The yellowed spectral reflectances and the originals were converted into sRGB data afterwards. The Macbeth ColorChecker was primarily used to assess the feasibility of the proposed methods before application to a well-known work of art. Consequently, we apply the network to the Mona Lisa to further test the network. The varnished and cleaned versions of the Mona Lisa are taken from [12].

Deep Generative Network (Architecture and Application)

In this section, the Deep Generative Network (DGN) that has been developed in this work is described. This method requires only a small area of the artwork to be cleaned. Then using the data of both the cleaned and varnished conditions of the same area, the network learns how to map from the uncleaned condition to the clean one. It then applies the same map to the rest of the artwork resulting in a virtually cleaned artwork.

The idea behind a DGN is top learn the relationship 𝑥 = 𝑓 𝜃 (𝑧), which maps an image 𝑧 to another image 𝑥. This approach is used here to recover the virtually cleaned artwork from the unclean one in the RGB color domain. The goal here is to generate image 𝑋, which is the virtually cleaned image of the varnished artwork. Through feeding the varnished image 𝑍 into the generator, image 𝑋 with this characteristic will be attained. 𝑍 is the RGB image of the artwork before cleaning. As mentioned above, only a small area of the painting is cleaned and we have the RGB image of that area for both cleaned and uncleaned conditions. Let us call the area of the painting for which we have both the cleaned and uncleaned data 𝐴. The RGB image of this area that is physically cleaned is called 𝐴 𝑐 and the corresponding RGB image of this area that belongs to 𝑍 (that is unclean) is 𝐴 𝑢 . It makes sense that 𝐴 𝑢 belongs to 𝑍 as 𝑍 is the RGB image of the uncleaned artwork. When 𝑍 goes through the network, the part corresponding to 𝐴 𝑢 is taken out and the pixel-wise error between 𝐴 𝑢 and 𝐴 𝑐 is calculated to compute the loss, which is then back-propagated to the generator, through which the parameters 𝜃 of the mapping function are optimized. Fig. 1 shows the process described. It should be noted that there is no training in a traditional sense using this approach. The error computed between the 𝐴 𝑢 and 𝐴 𝑐 is back propagated to the generator and the generator will clean the whole image using this error coming from the loss function. This cleaning process is taken place step by step at each epochs, until the network reaches the maximum number of epochs.

Through trial and error we come to know that the network works better in CIELAB color space than in RGB. This improvement in the neural network performance by changing the color space to CIELAB has been reported in the literature as well [11]. Therefore, we first convert the RGB image, 𝑍, into the CIELAB color space. The L* channel is then set aside and the a*b* channels, as input, go through two main modules of the network, consisting of several blocks as follows:

1) The down-sampling block 𝑑 (𝑖) : Each 𝑑 (𝑖) is composed of convolutional layer 𝐶

𝑑 (𝑖) also performing the down-sampling operation through setting the stride 𝑆 = 2. After that, batch normalization and the LeakyReLU activation layer are performed. The output is then fed into the next convolutional layer 𝐶 (2) 𝑑 (𝑖) with the same stride. Similar to the first convolutional layer, this operation is followed by a batch normalization layer and the LeakyReLU activation function. 𝐶

𝑑 (𝑖), 𝑘(2)𝑑 (𝑖), 𝑛(1)

𝑑 (𝑖) and 𝑛

(2) 𝑑 (𝑖).

2) The up-sampling block 𝑢 (𝑖) : Each 𝑢 (𝑖) consists of a few stacked layers. Opposite to the down-sampling blocks, batch normalization is the first layer. Afterwards, the first convolutional layer 𝐶

𝐶𝑢 (𝑖), 𝑘(2)𝑢 (𝑖), 𝑛(1)

𝑢 (𝑖) and 𝑛

(2) 𝑢 (𝑖), respectively. The skip connection, shown as 𝑠 (𝑖) , is also utilized to connect the down-sampled data to the up-sampled data (the up-sampling and down-sampling blocks are symmetrical), so the residual information can be fully employed. 𝑜 (0) denotes the output block. It is indeed the up-sampling block that is modified so that the up-sampling layer is superseded with one convolutional layer which is followed by one Sigmoid activation layer.

The network has an hourglass architecture as shown in Fig. 2. Each down-sampling and up-sampling sections are comprised of 5 blocks and 5 skip connections. The filter size is 3 × 3 in the up-sampling and down-sampling blocks but it is 1 × 1 in the last convolutional layer. There are 128 filters in the convolutional layers in the down-sampling and up-sampling blocks and there are only 2 (to be equal to the a*b* channels) filters in the last convolutional layers. As it was mentioned, only a*b* channels of the image 𝑍 are input into the network. The output from the network is also the a*b* of the image 𝑋. This output will be combined with the L* channel of the image 𝑍 that was first set aside, constructing the CIELAB image of output 𝑋. The CIELAB image is then converted back into RGB image following standard formulae for sRGB.

As mentioned, the input to the network is the a*b* image of the uncleaned artwork 𝑍 and the generated image is 𝑋. The cost function is defined as the pixel-wise difference between 𝐴 𝑢 and 𝐴 𝑐 . 𝐴 𝑢 belongs to 𝑍 and therefore, it changes in each iteration. Consequently, the cost function is given as

𝑚𝑖𝑛‖𝐴𝑢 − 𝐴𝑐‖ 2(1)

It should be noted that the input to the model should be replaced with the output of the model after each iteration. The overall algorithm is shown in Algorithm 1.

Evaluation Metrics and Experimental Environment

Visualization of the results, per-pixel spectral Euclidean Distance (ED) and Spectral Angle (SA), between the original (cleaned) image and the virtually cleaned image, are the metrics used in this work for accuracy evaluation [20]. The color space used is RGB and each pixel is considered a vector in this space, with the vector tip located at a particular point in the color space according to the RGB values. The spectral Euclidean distance is obtained through calculating the Euclidean distance between two pixels in that color space. The spectral angle is calculated between two

𝑆𝐴 𝑘 = 𝑐𝑜𝑠 −1 (︂ t 𝑘 • r 𝑘 |t 𝑘 ||r k | )︂(2)

where 𝑘 denotes the 𝑘 𝑡ℎ pixel, t 𝑘 and r 𝑘 denote the two pixels belonging to the test and reference images, and 𝑆𝐴 𝑘 denotes the spectral angle between these two pixels. Python 3.9.7 |Anaconda, Inc. is used as a base coding environment for the DGN algorithm. More specifically, the DGN codes were written and run in the TensorFlow environment, which was installed onto the Anaconda. In terms of hardware, the programs are run on a GPU (NVIDIA GeForce MX350). The training of the DGN is performed using only one image and is consequently referred to as an unsupervised learning method [18]. As mentioned before, only a small area of the image is used to compute the loss function, and the same loss is then used for the whole image to virtually clean it. 1500 epochs are used to train the model. MATLAB R2022a, the package of mathematical software was also used for evaluation computations, making the Macbeth ColorChecker and yellowing it.

Results and Discussions

In this section, the results of applying DGN to virtually clean the Macbeth ColorChecker and the Mona Lisa are presented and examined. First, we consider the Macbeth ColorChecker.

The Macbeth ColorChecker was simulated as varnished and unvarnished and is used to test the approach, similarly to previous work by [17]. The Macbeth ColorChecker has 24 different color patches, including a range of neutral samples. As mentioned, the DGN needs only a small area of the painting to be physically cleaned and then, using that small part to learn the transfer function describing the varnish effect, the whole painting is virtually cleaned. Given that the Macbeth ColorChecker has different color patches, we empirically identified that the number of patches necessary to be physically cleaned is at least three. Therefore, we applied the method, using the following combination of three patches: a) red, green and blue, b) black and white and a neutral patch, and c) all of the neutral patches, i.e., six neutral patches that exist on the standard Macbeth ColorChecker. The combination in c obviously contains more than three patches, but is presented as an alternate approach to training the network for testing. The results are visually compared to the method proposed by [17], as shown in Fig. 3 and quantitatively compared in Table 1. We observe that the DGN has done an acceptable job compared to the CNN proposed by [17], even though the number of training samples required by the DGN is significantly smaller that that of t he CNN. To have a better understanding of the results, Table 1 shows the quantitative results in terms of the mean values of ED and SA for the whole ColorChecker. These metrics are computed between the virtually cleaned color chart and the original one. As it is observed from Table 1, the CNN model has done a slightly better job in terms of cleaning the Macbeth ColorChecker. This is not too concerning as the method proposed herein is more practical than the CNN proposed by [17]. The DGN proposed herein only needs a small area of the painting to be cleaned, while the CNN needs a significantly larger number of training samples to work. While the end goal of each approach is the same, a virtually cleaned work of art, the operational aspects of the two methods a significantly different.

Finally, we also applied the DGN to clean the Mona Lisa. The results are shown in Fig. 4. Fig. 4 (c) shows the area of the painting that was used to compute the loss; in other words, that area is used to train the network to go from the unclean to the clean version of the artwork. As shown in Fig. 4 (e), the DGN has again done a visually acceptable job of cleaning the artwork, considering that the area of the painting used to train the network is fairly small. The ED and SA are also computed between the original clean Mona Lisa and the virtually cleaned one. The results are both visualized (Fig. 5) and reported in terms of the mean values across the whole image (Table 2). The visualization of the ED and SA values show specific areas of the work that are not well cleaned (note that in Figure 5 all four results are normalized to 1). To better understand the absolute performance, the mean values of the ED and SA are also reported which clarifies which method has outperformed the other. As it is observed from Fig. 5, the CNN has not done a good job especially predicting the cleaned color of the sky, and overall the error is higher and more widespread in the CNN.

We see from Table 2, the proposed method here has surprisingly outperformed the CNN proposed by [17]. It is surprising as the CNN outperformed our proposed method when the Macbeth ColorChecker was the object of interest, but the results here are the opposite in the case of the Mona Lisa. This could be because of the richness of the colors and structural features that are present in the Mona Lisa, as opposed to the Macbeth ColorChecker, which is a simple color chart. This would also confirm that the method proposed herein is more practical than CNN, as asserted above. The method proposed herein has a potential of being applied to a wider type of artworks compared to the CNN, which requires a large set of training data with content similar to the artwork itself.

It is important to note that the small area chosen in the artwork should be a representative of all the features and material present on the painting. Looking at Figure 4 (c), one could see that the small area contains a small part of the sky, human eye and skin, and her dress. This will strengthen the performance of the DGN. To examine this point further, another experiment is performed in which, the small area varies from what has been chosen in Figure 4 (c). The new small area only comprises the person (part of her face, her dress, her hair and her skin) as shown in Fig. 6 As seen from the bottom row of Fig. 6, the sky and everything around the person, has not been cleaned as well as the top row where the area chosen is a better representative of everything in the image. It is also worthwhile noting that the DGN has not done terribly, however, with a more thorough area, it could lead to a better result.

Conclusions

In this work, we developed a Deep Generative Network (DGN) to tackle the problem of virtual cleaning of artwork for visualization. We compared our method to the latest method in this area which used a Convolutional Neural Network (CNN). We used the Macbeth ColorChecker and the Mona Lisa to test our method. We found that the proposed model did not outperform the CNN in the case of the Macbeth ColorChecker, but it did outperform the CNN in the case of the Mona Lisa. This shows the high potential of the work proposed herein to be applied in the real case and to a wider range of artworks. The method proposed herein could potentially help the conservators with seeing how a painting would look if it were to be physically cleaned, or aid them in choosing from different options they have for a physical cleaning and so forth.