It is well-known that the varnish applied to artwork yellows with time changing its appearance accordingly. Conservators are then sometimes prompted to physically clean the artwork in an attempt to recover the original look of the work. At times, the conservators only partially clean the artwork ifrst. They then try to virtually clean the rest of the artwork to visualize the result of the cleaning before physically cleaning the entire piece. There have been many diferent approaches that have been proposed to virtually clean a partially cleaned artwork. All of them have some limitations, the low accuracy of which is the main one. In this paper, a deep generative network is proposed to virtually clean a partially cleaned artwork in the RGB domain. The proposed generative model consists of several up-sampling and down-sampling convolution blocks and skip connections with a symmetric architecture. The loss function is calculated using the part of the artwork that has been physically cleaned for which we have access to both RGB images before and after cleaning. Therefore the network is able to clean the whole artwork using only a small area of it that has already been physically cleaned. A Macbeth ColorChecker and images of the Mona Lisa are used to test the approach and the results are compared with a recent approach available in the literature which uses a Convolutional Neural Network (CNN). The results are found to be acceptable given that the approach proposed herein has a potential to be applied in a real situation and there is no need for a large training dataset, on which the CNN method relied on.
Artworks are usually varnished for the purpose of protection. Although successful in their
main purpose, with time, this application can change the visual qualities of artworks [
Most of the studies done in the area of virtual cleaning are based on first cleaning a small
part of the painting physically. and they use an RGB image of the painting before and after
cleaning. Using that small part, for which they have data belonging to both cleaned and unclean
state, they attempt to virtually clean the entire painting producing a visualization of the cleaned
work. They typically do that through fitting some type of regression to the data obtained from
the small area before and after cleaning. They then apply the same regression model to the rest
of the painting which leads to the artwork being virtually cleaned [
In this work, we propose a Deep Generative Network (DGN) to virtually clean a partially cleaned artwork. The generative model we use herein has been used in the area of remote sensing for the purpose of denoising of the hyperspectral image and single image super-resolution [18, 19]. The authors developed a convolutional generative network that was able to take in a noise cube and output a super-resolved remotely sensed image. The network is deep and symmetrical and has also borrowed the idea of skip connections from U-Net enabling it to use residual information as best as possible. In this work, we have modified the network to ift our purpose. Instead of feeding the network with a random noise image, we feed in the RGB image of the uncleaned artwork. To be more specific, we have information of a small area of the painting before and after cleaning. The RGB image of the artwork is first changed into CIELAB, and the a*b* channels are used to train the network. The loss function, on the other hand, is computed between the uncleaned and the corresponding cleaned area of the artwork (the small area for which we have access to both cleaned and uncleaned data). The model is tested on Macbeth ColorChecker and the Mona Lisa that are partially cleaned. The results show that our approach here has done a better job compared to the model proposed by [17] when it comes to the Mona Lisa but has done slightly worse when compared to [17] for the Macbeth ColorChecker. Overall, the method proposed herein is more applicable to a real situation where the conservator has no access to a large set of data with which to train the model. Comparing our model to that of [17] seems fair as in their paper they showed that their model had outperformed the only physics-based model proposed for artwork virtual cleaning [14, 17].
This paper is laid out as follows: the next section will present the specifications of the data, while the method will be explained in a more detailed manner along with the evaluation metrics and experimental environment. After that, results are presented along with discussions in the next section. Finally the conclusions are presented.
In this section, the data used are explained and the proposed algorithm is described in detail.
2.1. Data
One of the datasets used to test the proposed method is the Macbeth ColorChecker spectral
reflectance data. The spectral reflectances were artificially yellowed using the same formula
suggested by [17] in the spectral domain. The artificially yellowed spectral reflectance mimics
the visual impact varnish has on the painting. So, due to the Macbeth ColorChecker having a
wide range of colors along with neutral patches, we use it as an initial test for our approach. The
Macbeth ColorChecker is simulated in a way that it is “varnished” with a layer of a particular
spectral reflectance and transmittance (generally speaking, varnish is yellow, and its spectral
reflectance and transmittance should represent that [ 14, 15]), as explained by [17]. The yellowed
spectral reflectances and the originals were converted into sRGB data afterwards. The Macbeth
ColorChecker was primarily used to assess the feasibility of the proposed methods before
application to a well-known work of art. Consequently, we apply the network to the Mona Lisa
to further test the network. The varnished and cleaned versions of the Mona Lisa are taken
from [
In this section, the Deep Generative Network (DGN) that has been developed in this work is described. This method requires only a small area of the artwork to be cleaned. Then using the data of both the cleaned and varnished conditions of the same area, the network learns how to map from the uncleaned condition to the clean one. It then applies the same map to the rest of the artwork resulting in a virtually cleaned artwork.
The idea behind a DGN is top learn the relationship = (), which maps an image to another image . This approach is used here to recover the virtually cleaned artwork from the unclean one in the RGB color domain. The goal here is to generate image , which is the virtually cleaned image of the varnished artwork. Through feeding the varnished image into the generator, image with this characteristic will be attained. is the RGB image of the artwork before cleaning. As mentioned above, only a small area of the painting is cleaned and we have the RGB image of that area for both cleaned and uncleaned conditions. Let us call the area of the painting for which we have both the cleaned and uncleaned data . The RGB image of this area that is physically cleaned is called and the corresponding RGB image of this area that belongs to (that is unclean) is . It makes sense that belongs to as is the RGB image of the uncleaned artwork. When goes through the network, the part corresponding to is taken out and the pixel-wise error between and is calculated to compute the loss, which is then back-propagated to the generator, through which the parameters of the mapping function are optimized. Fig. 1 shows the process described. It should be noted that there is no training in a traditional sense using this approach. The error computed between the and is back propagated to the generator and the generator will clean the whole image using this error coming from the loss function. This cleaning process is taken place step by step at each epochs, until the network reaches the maximum number of epochs.
Through trial and error we come to know that the network works better in CIELAB color
space than in RGB. This improvement in the neural network performance by changing the color
space to CIELAB has been reported in the literature as well [
2) The up-sampling block (): Each () consists of a few stacked layers. Opposite to the down-sampling blocks, batch normalization is the first layer. Afterwards, the first convolutional (1)() with S = 1 and a batch normalization and LeakyReLU activation function are layer (2)(). The output, after batch used. The output is then fed into the next convolutional layer normalization and non-linear activation, is input into the bilinear up-sampling layer with factor 2. (2)(), similar to the down-sampling block, can be set to diferent kernel sizes (1)() and and diferent numbers of filters shown as (1)(), (2)(), (1)() and (2)(), respectively.
The skip connection, shown as (), is also utilized to connect the down-sampled data to the up-sampled data (the up-sampling and down-sampling blocks are symmetrical), so the residual information can be fully employed. (0) denotes the output block. It is indeed the up-sampling block that is modified so that the up-sampling layer is superseded with one convolutional layer which is followed by one Sigmoid activation layer.
The network has an hourglass architecture as shown in Fig. 2. Each down-sampling and up-sampling sections are comprised of 5 blocks and 5 skip connections. The filter size is 3 × 3 in the up-sampling and down-sampling blocks but it is 1 × 1 in the last convolutional layer. There are 128 filters in the convolutional layers in the down-sampling and up-sampling blocks and there are only 2 (to be equal to the a*b* channels) filters in the last convolutional layers. As it was mentioned, only a*b* channels of the image are input into the network. The output from the network is also the a*b* of the image . This output will be combined with the L* channel of the image that was first set aside, constructing the CIELAB image of output . The CIELAB image is then converted back into RGB image following standard formulae for sRGB.
As mentioned, the input to the network is the a*b* image of the uncleaned artwork and the generated image is . The cost function is defined as the pixel-wise diference between and . belongs to and therefore, it changes in each iteration. Consequently, the cost function is given as ‖ − ‖2 (1) It should be noted that the input to the model should be replaced with the output of the model after each iteration. The overall algorithm is shown in Algorithm 1.
Visualization of the results, per-pixel spectral Euclidean Distance (ED) and Spectral Angle (SA), between the original (cleaned) image and the virtually cleaned image, are the metrics used in this work for accuracy evaluation[20]. The color space used is RGB and each pixel is considered a vector in this space, with the vector tip located at a particular point in the color space according to the RGB values. The spectral Euclidean distance is obtained through calculating the Euclidean distance between two pixels in that color space. The spectral angle is calculated between two Algorithm 1 Deep Generative Network Algorithm
Procedure: Virtual Cleaning () Input: a*b* image of the uncleaned artwork while epoch < max_epoch do = () (Model here stands for the deep generative model.) = (The part of the corresponding to is taken out) ‖ − ‖2 = (replace the input with the output of the model in each iteration) end while Return
End Procedure vectors and is reported in radians in the range [0, 3.142], defined as = − 1 ︂( t · r )︂ |t||rk| (2) where denotes the ℎ pixel, t and r denote the two pixels belonging to the test and reference images, and denotes the spectral angle between these two pixels.
Python 3.9.7 |Anaconda, Inc. is used as a base coding environment for the DGN algorithm. More specifically, the DGN codes were written and run in the TensorFlow environment, which was installed onto the Anaconda. In terms of hardware, the programs are run on a GPU (NVIDIA GeForce MX350). The training of the DGN is performed using only one image and is consequently referred to as an unsupervised learning method [18]. As mentioned before, only a small area of the image is used to compute the loss function, and the same loss is then used for the whole image to virtually clean it. 1500 epochs are used to train the model. MATLAB R2022a, the package of mathematical software was also used for evaluation computations, making the Macbeth ColorChecker and yellowing it.
In this section, the results of applying DGN to virtually clean the Macbeth ColorChecker and the Mona Lisa are presented and examined. First, we consider the Macbeth ColorChecker.
The Macbeth ColorChecker was simulated as varnished and unvarnished and is used to test the approach, similarly to previous work by [17]. The Macbeth ColorChecker has 24 diferent color patches, including a range of neutral samples. As mentioned, the DGN needs only a small area of the painting to be physically cleaned and then, using that small part to learn the transfer function describing the varnish efect, the whole painting is virtually cleaned. Given that the Macbeth ColorChecker has diferent color patches, we empirically identified that the number of patches necessary to be physically cleaned is at least three. Therefore, we applied the method, using the following combination of three patches: a) red, green and blue, b) black and white and a neutral patch, and c) all of the neutral patches, i.e., six neutral patches that exist on the standard Macbeth ColorChecker. The combination in c obviously contains more than three patches, but is presented as an alternate approach to training the network for testing. The results are visually compared to the method proposed by [17], as shown in Fig. 3 and quantitatively compared in Table 1.
We observe that the DGN has done an acceptable job compared to the CNN proposed by [17], even though the number of training samples required by the DGN is significantly smaller that that of t he CNN. To have a better understanding of the results, Table 1 shows the quantitative results in terms of the mean values of ED and SA for the whole ColorChecker. These metrics are computed between the virtually cleaned color chart and the original one.
All neutral patches Black, white and a neutral patch
Red, green and blue patches
CNN proposed by [17]
As it is observed from Table 1, the CNN model has done a slightly better job in terms of cleaning the Macbeth ColorChecker. This is not too concerning as the method proposed herein is more practical than the CNN proposed by [17]. The DGN proposed herein only needs a small area of the painting to be cleaned, while the CNN needs a significantly larger number of training samples to work. While the end goal of each approach is the same, a virtually cleaned work of art, the operational aspects of the two methods a significantly diferent.
Finally, we also applied the DGN to clean the Mona Lisa. The results are shown in Fig. 4.
Fig. 4 (c) shows the area of the painting that was used to compute the loss; in other words, that area is used to train the network to go from the unclean to the clean version of the artwork. As shown in Fig. 4 (e), the DGN has again done a visually acceptable job of cleaning the artwork, considering that the area of the painting used to train the network is fairly small. The ED and SA are also computed between the original clean Mona Lisa and the virtually cleaned one. The results are both visualized (Fig. 5) and reported in terms of the mean values across the whole image (Table 2). The visualization of the ED and SA values show specific areas of the work that are not well cleaned (note that in Figure 5 all four results are normalized to 1). To better understand the absolute performance, the mean values of the ED and SA are also reported which clarifies which method has outperformed the other. As it is observed from Fig. 5, the CNN has not done a good job especially predicting the cleaned color of the sky, and overall the error is higher and more widespread in the CNN.
We see from Table 2, the proposed method here has surprisingly outperformed the CNN proposed by [17]. It is surprising as the CNN outperformed our proposed method when the Macbeth ColorChecker was the object of interest, but the results here are the opposite in the case of the Mona Lisa. This could be because of the richness of the colors and structural features that are present in the Mona Lisa, as opposed to the Macbeth ColorChecker, which is a simple color chart. This would also confirm that the method proposed herein is more practical than CNN, as asserted above. The method proposed herein has a potential of being applied to a wider type of artworks compared to the CNN, which requires a large set of training data with content similar to the artwork itself.
It is important to note that the small area chosen in the artwork should be a representative of all the features and material present on the painting. Looking at Figure 4 (c), one could see that the small area contains a small part of the sky, human eye and skin, and her dress. This will strengthen the performance of the DGN. To examine this point further, another experiment is performed in which, the small area varies from what has been chosen in Figure 4 (c). The new small area only comprises the person (part of her face, her dress, her hair and her skin) as shown in Fig. 6. In this Figure, the top row shows the results of Figs 4 and 5 combined in the case of the DGN and the bottom row shows the results of the DGN when a diferent and smaller area is chosen.
As seen from the bottom row of Fig. 6, the sky and everything around the person, has not been cleaned as well as the top row where the area chosen is a better representative of everything in the image. It is also worthwhile noting that the DGN has not done terribly, however, with a more thorough area, it could lead to a better result.
In this work, we developed a Deep Generative Network (DGN) to tackle the problem of virtual cleaning of artwork for visualization. We compared our method to the latest method in this area which used a Convolutional Neural Network (CNN). We used the Macbeth ColorChecker and the Mona Lisa to test our method. We found that the proposed model did not outperform the CNN in the case of the Macbeth ColorChecker, but it did outperform the CNN in the case of the Mona Lisa. This shows the high potential of the work proposed herein to be applied in the real case and to a wider range of artworks. The method proposed herein could potentially help the conservators with seeing how a painting would look if it were to be physically cleaned, or aid them in choosing from diferent options they have for a physical cleaning and so forth.
This research was funded by the Xerox Chair in Imaging Science in the Chester F. Carson Center for Imaging Science at the Rochester Institute of Technology. [13] C. M. T. Palomero, M. N. Soriano, Digital cleaning and “dirt” layer visualization of an oil painting, Optics express 19 (2011) 21011–21017. [14] G. Trumpy, D. Conover, L. Simonot, M. Thoury, M. Picollo, J. K. Delaney, Experimental study on merits of virtual cleaning of paintings with aged varnish, Optics express 23 (2015) 33836–33848. [15] E. Kirchner, I. van der Lans, F. Ligterink, E. Hendriks, J. Delaney, Digitally reconstructing van gogh’s field with irises near arles. part 1: varnish, Color Research & Application 43 (2018) 150–157. [16] J. Linhares, L. Cardeira, A. Bailão, R. Pastilha, S. Nascimento, Chromatic changes in paintings of adriano de sousa lopes after the removal of aged varnish, Conservar Património 34 (2020) 50–64. [17] M. Maali Amiri, D. W. Messinger, Virtual cleaning of works of art using deep convolutional neural networks, Heritage Science 9 (2021) 1–19. [18] J. M. Haut, R. Fernandez-Beltran, M. E. Paoletti, J. Plaza, A. Plaza, F. Pla, A new deep generative network for unsupervised remote sensing single-image super-resolution, IEEE Transactions on Geoscience and Remote sensing 56 (2018) 6792–6810. [19] D. Ulyanov, A. Vedaldi, V. Lempitsky, Deep image prior, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9446–9454. [20] B. Park, W. Windham, K. Lawrence, D. Smith, Contaminant classification of poultry hyperspectral imagery using a spectral angle mapper algorithm, Biosystems Engineering 96 (2007) 323–333.