We propose a method to relight scenes in a single image while removing unwanted objects by the combination of 3D-aware inpainting and relighting for a new functionality in image editing. First, the proposed method estimates the depth image from an RGB image using single-view depth estimation. Next, the RGB and depth images are masked by the user by specifying unwanted objects. Then, the masked RGB and depth images are simultaneously inpainted by our proposed neural network. For relighiting, a 3D mesh model is first reconstructed from the inpainted depth image, and is then relit with a standard relighting pipeline. In this process, removing cast shadows and sky areas and albedo estimation are optionally performed to suppress the artifacts in outdoor scenes. Through these processes, various types of relighting can be achieved from a single photograph while excluding the colors and shapes of unwanted objects.
algorithms into difusion-based, patch-based, and learning-based approaches. 2.2.1. Difusion-based and Patch-based Methods to suppress the influence of existing light sources in outdoor scenes. Through these processes, various types of relighting can be achieved from a single photograph while excluding the colors and shapes of unwanted objects. Finally, the processed image can be applied to VR applications, image editing, and other applications using image synthesis.
Difusion-based methods smoothly transfer the effective information from the known region to the target region by difusing the pixels at the junction. The method of Bertalmio et al., who first 2. Related Work proposed the term "image inpainting" as an early study, belongs to this category [9]. In this category, 2.1. Relighting difusion is mathematically formulated in various manners. Mumford-Shah segmentation model was Image relighting is a technique to reproduce the adapted for image inpainting by introducing Euler’s shadows, brightness, and color of an object or scene Elastica [10]. Li et al. proposed difusion-based taken in a diferent lighting environment. It solves inpainting by analyzing the local variance of image the problem of reconstructing the light of a photo- Laplacian along the isophote direction [11]. graph. We classify the algorithms into geometry- Patch-based methods search for patches that can based and learning-based approaches. match the ROI in the source region of the image and then fill it with the patches. An early study [ 12] syn2.1.1. Geometry-based Methods thesizes patches into the ROI sequentially, followed Prior work on image relighting has generally relied by overall optimization of patch-based costs [13]. on scene geometry, light, and reflectance models us- Various improvements have been made since then. ing inverse rendering. The full use of geometry, ma- For example, the Markov random field (MRF) modterials, and lighting in scene representation allows eling method segments the image into blocks and for conventional rendering and shading techniques uses a prior to limit the efective matching context with promising results. Techniques such as semi- candidate patches of the source region [14]. Anautomatic vision-based geometry reconstruction [4] other method uses Gaussian-weighted nonlocal texor parameter estimation by viewing the same scene ture similarity measure to obtain multiple candidate under diferent lighting conditions [ 5] can simplify patches and nonlinear filtering ( -trimmed mean the capture process. In addition, advanced capture iflter) to the inpainting target region in pixel [15]. setups such as Light Stage have been used in film production [6]. 2.2.2. Learning-based Methods
2.1.2. Learning-based Methods image inpainting. We divide the mainstream apLearning-based methods have significantly improved proaches into convolutional neural networks (CNN) the performance of multiview relighting systems and generative adversarial networks (GAN) based for scene-scale relighting. A typical approach is on the network architecture. to use a single neural network to map the input CNN-based approaches can compensate for the image and a set of approximate guide maps, such lack of global information distortion. Networks with as depth maps or shadow maps, to new lighting the encoder-decoder structure are common in this conditions [7]. Some methods remove the original field. The context encoder in the encoder-decoder lighting influence and transform it into new lighting network [16] can efectively use the local information conditions, depending on the geometric property [8]. around the target region and the global information of the whole image to generate information. Zeng et al. proposed a pyramidal context encoding the 2.2. RGB Image Inpainting network PEN-Net based on the U-Net structure [17]. RGB image inpainting is a technique that fills in It can encode the contextual semantics from the fullmissing regions (ROI: Region of Interest) with im- resolution input and decode the learned semantic age texture consistency throughout the entire image. features for inpainting defective content. The techniques remove scratches and text masks in GAN-based approaches have become the active rephotographs, remove unwanted targets, and repro- search direction in image synthesis. The face feature duce image block contents due to network packet point generation network [18], an image inpainting loss during image transmission. We classify the method for human faces, consists of three branching networks of image segmentation networks, and a inpainted image is used as input for relighting so cooperative GAN based on CGAN was proposed. that the user is free to change the illumination to Two-branch network Pluralistic [19] based on the output the requested image.
CGAN architecture. One path is a reconstruction, The overview of our proposed method is illusand the other path is a generative path. trated in Fig. 1. First, the depth image is estimated from the input RGB image by using single-view 2.3. Depth Image Inpainting depth estimation (Fig. 1(a)) with MegaDepth [23]. The RGB and estimated depth images are inStudies have been conducted to inpaint missing re- painted together with our proposed inpainting gions not only in RGB images but also in depth network (Fig. 1(c)) after removing target regions images. As for the inpainting of depth images, some masked by the user (Fig. 1(b)). Then, optional research has been conducted to fill in missing re- processing is performed before relighting for the gions in a depth image using the corresponding outdoor environment (Fig. 1(d)). The processing RGB image as a guide because missing regions are includes shadow removal, sky removal for outdoor more likely to occur in depth images than in RGB scenes, and albedo map estimation. The 3D mesh images due to the diference in the measurement de- model is generated from the depth image, and is vice [20]. As a diferent application, a patch-based used for relighiting with a conventional rendering method has been proposed for removing unwanted pipeline (Fig. 1(e)). By feeding the image with ilobjects from two stereo RGB images while preserv- lumination mapping, we can obtain an image relit ing consistency and reconstructing the depths of the with new illumination that users can control freely. two images [21]. The method most relevant to this The depth of the RGB image is first estimated research has been proposed to remove unwanted to obtain the geometry of the RGB image by usobjects from RGB and depth images and simulta- ing the proposed depth estimation method. Next, neously inpaint the ROI in RGB and depth images the estimated depth map and the original image using a neural network [22]. are masked with the object mask image and are inpainted by the inpainting network. Relighting 2.4. Summary and Our Contribution runs the conventional rendering pipeline on the 3D model recovered from the depth map.
As mentioned above, relighting and inpainting have been studied separately. In other words, there is no study that has performed them simultaneously 3.2. Single-View Depth Dstimation for image synthesis. In this paper, we propose an In our method, the depth image estimation is imintegrated framework that takes advantage of the portant to realize relighting because the input is features of these studies. First, due to the excel- only a single RGB image. A depth image can be lent performance of deep learning on inpainting, obtained in various ways, such as a depth camera we take a cue from the literature [22] and use neu- or a depth map estimated using a depth estimaral networks to simultaneously inpaint RGB and tion algorithm. Since surface normals have a strong depth images. This two-stage structure can reduce guiding efect on illumination, the depth estimation the gap between the RGB map and the estimated with surface estimation is essential for relighting. depth map to some extent since we use the same In our implementation, we estimate a depth image intermediate output as a guide. Then, for now, from a single RGB image with MegaDepth [23]. learning-based relighting methods do not allow for free illumination changes or reasonable shadow gen- 3.3. Masking and RGBD Inpainting eration. We utilize the inpainted depths to achieve relighting in a geometry-based manner.
Both the estimated depth image and the input RGB image are masked with the object mask image generated by the user. Since our method requires inpainting both the RGB image and its depth image, we propose to inpaint a 4-channel RGBD image to increase their information agreement.
wanted objects from a photograph and change light- 3.3.1. Network Architecture ing conditions by considering 3D geometry. The method takes a single RGB image and an object mask that the user wants to remove as input. The
User input
Optional process
Color Depth estimation network D(a) epth (a) generators, referring to the two-stage GAN network 3.3.2. Loss Function architecture shown in Fig. 2. Both stages consist of a bundle of generators and discriminators. Let and
denote the generator and discriminator of the edge generator, and denote the generator and discriminator of the image inpainting network,
be an RGBD image, and and
be an edge image generated by an edge detector and a grayscale image of the RGB image, respectively. The edge generator takes the masked grayscale image ˜ corresponding edge image ˜ and the image mask = = ︀⨀ ¯ , the ︀⨀
¯ , as pre-condition (1 for missing regions and 0 for background), where ⨀︀ denotes the Hadamard product. Then, the generator predicts the edge image the masked regions.
by filling the edges in = (˜ , ˜
, ).
The image generators take a missing RGBD image ˜ image = ︀⨀
¯ and a composed edge , which is generated by compositing the
= ground-truth edges in the background region with the edges generated in the missing regions. That is, ︀⨀ ¯ + ︀⨀
. Finally, the image generator outputs a RGBD image with the same resolution as the input image and with missing regions inpainted as follows.
= (˜ , ).
increases the flexibility.
achieves the consistency between the overall structure of the missing region and the context.
loss 1_ℎ
because it corrects the zero-point non-smooth problem of L1 loss and is more robust against outliers than L2 loss. Also, adversarial Loss
In addition, we incorporate perceptual loss and style loss
with reference to [25]. These two losses are computed only for the three channels for RGB represented with a subsript . = =
1 ‖ ( | ( _ ) − ( _ ) | _ ) − ( _ )‖2 layers where is the activation map of the -th activation layer of the network with the × × map. In this equation, is the learned VGG-19 net feature work and corresponds to the activation maps from 1_1, 2_1 , 3_1, 4_1and
trix is a × matrix = is the matrix that resizes the matrix into / . × × .
怀amb + source + Residual Blocks ba ȴ a m怀 smoothing algorithm to smooth the model. The
lightmap is baked directly on Unity3D and used for the following iterative formula coordinates of the pixel points are computed di- the light between the two clusters is determined by rectly from a depth image. Since the input is a
We use the Laplace smoothing algorithm. The al- produces distinct cast shadows. If the cast shadows smoothing operation for each vertex is as follows. bine the Triple-cooperative Video Shadow Detecrendering. 3.4.1. Mesh Model Construction We use the basic conversion from a depth image to a 3D mesh model. In other words, the world ×
image only, camera parameters are single angle by not available. Therefore, we set a virtual viewing and calculates the virtual focal distance = 2 2 . In practice, because of this basic construction method, there is no change in the model UVs and the model UV mapping, which is useful for our relighting in the next step. gorithm directly shifts the vertex positions without destroying the UVs of the model. The triangle mesh is centered at a vertex
and its adjacent vertices 1.... −1 and all of its edges. More formally, the 1 ∑︁ =1 ( ) =
( ), where is the number of vertices adjacent to a vertex ,
( ) is the -th adjacent vertex, and ( ) is the new position of the vertex . 3.4.2. Light Map Generation
method is lightmaps.
mainly based on the generation of
When the lighting of the model is changed, the lightmap of the corresponding lighting is generated and displayed. We bake a lightmap directly on Unity3D. = + ∑︁
where is the final light at a point , is the light at the point itself, the bounce coeficient of , and is the light at a point , and denotes the material property. This is why Enlighten can support changing the light source while leaving the scene objects unchanged. 3.4.3. Optional Processing Light from strong sources, such as outdoor sunlight, in the original image are left as are, it may look strange due to inconsistencies with the cast shadows after relighting. Therefore, it is necessary to remove the shadows from the input image. We comtion (ViSha [26]) and Stacked Generative Adversarial Networks (STCGAN [27]) for shadow detection and removal.
ity, the shape of the sky may be reconstructed by depth estimation. Such sky shapes cause negative efects when relighting. Therefore, removing the sky is an additional necessary process in outdoor scenes.
semantic segmentation. Next, pixels labeled as sky are removed and relighting is performed withoug the influence of the sky. After that, the sky area is composited with either the original sky or, if necessary, a virtual sky generated by computer graphics.
Albedo maps primarily reflect the texture and color of the model and are often referred to as difuse reflectance maps. The albedo maps defines the color of the difuse light. Albedo maps are estimated to efectively remove the efect of light from the original image. For the estimation of albedo and original illumination, we use InverseRenderNet [29].
the performance of our RGBD inpainting network. Next, we show the results of the proposed method that combines image inpainting and relighting, using indoor and outdoor scenes constructed with computer graphics. Finally, we investigate the impact of optional processing: shadow removal, sky removal, and albedo estimation.
source_ m source_ 4 m _ m m _
text (COCO) dataset to train the network on the irregular mask dataset provided by Liu et al. [30].
Especially we used the 2015 release COCO dataset, w81h,i2c0h8cvoanltidaiantsioan tiomtaalgeosf,1a6n5d,48821,4tr3a4intiensgt iimmaaggeess,. RRRRG((G(Ga(aaGBa)(B))aBDO)DODrIIiirnmingiimpgmiapnuiagnatguegeet (b(((b)bb))MM)MaasaMskskekedaeddismikimmaeagadgegee (cR)(Rc(RG(c)GcIG)B)InBnIBIDnpnDpDpapiaaimiainimimintannetagtaedgetegddeeed ((d(Rdd)RR)(G)GGGdGGBrBBr)orDooDuDuunGinmindimdmdTatartgatrugrgeuuteehtthh Since the dataset does not contain depth images, we estimated depth images from RGB images by Figure 4: Result of RGBD image inpainting. MegaDepth [23] to obtain RGBD images and used them as the ground truth. For the mask dataset, 24,866 random datasets from the test dataset were and Fig. 4(c), we can clearly see that the lamp used for training. was removed and the area was successfully filled
Some of the results are shown in Fig. 3. The with the bookshelf and wall texture in the back. In results for the first and second row images are qual- the depth map, we can also see that the lamp was itatively good, while the edges are slightly blurred successfully removed. Note that the depth difers when compared to the ground truth. On the other significantly between the resulting image and the hand, for the image in the third row, there is a ground truth even outside of the target region berelatively large missing region around the boundary cause the depth in the input image was estimated where the two persons overlap, resulting in gener- using MegaDepth and that in the ground truth was ating an unnatural texture. Based on these results, created by computer graphics. the trained network can produce good results in For the inpainted result, we added a yellow point relatively simple cases where the background of the light source at the location of another lamp on the target to be removed is on the same object. How- left side of the image. We can see in the resulting ever, the performance became worse in scenes where image (Fig. 5(b)) that the shadow of the fireplace multiple objects overlap in the background of the is projected onto the wall on the right, and the target. lighting of the black sofa changes on the left of the image. Comparing the result with the ground truth 4.2. Inpainting and Relighting in Indoor (Fig. 5(c)), we can see that the shadow does not show the shape of the object well. This result is because
Scenes the shadow projection depends on the estimated Figure 4 shows the comparison of the RGBD- depth, but the accuracy of depth estimation from inpainted result image with the ground truth image a single image is not very high. However, the cast for an indoor scene. For the inpainting part, the shadow is adequately represented to the extent that lamp on the right of the image was deleted in the the added light source position can be seen from scene. By comparing the RGB images of Fig. 4(a) the image. (b) Relit image
(c) GT light source can be defined freely.
As noted in the indoor scene, the estimated depths and their ground truth are diferent. Thus, the positions of the additional light sources will be completely diferent from their positions in the ground truth scene. In the outdoor scene, as shown in Fig. 8, the positions are also diferent. Therefore, we added the light sources in diferent positions between the estimated scenes and the ground truth but used the same parameters for the corresponding light sources.
Figure 6 compares the RGBD-inpainted result im- Several optional processes can improve the quality age with the ground truth image for an outdoor of the lighting results. This section investigates the scene. For the inpainting part, the street lights impact of removing cast shadows and sky from the along the roadside were deleted in the outdoor scene. original image and estimating albedo and illuminaHere, we also marked some shadows as target re- tion. gions for inpainting. The result should be compared with the ground truth as shown in Fig. 6. By com- 4.4.1. Shadow Removal paring the RGB images of Fig. 6(a) and Fig. 6(c), we can clearly see that the street light was removed, First, we discuss the performance of shadow removal. and the area was successfully filled with the red Experiments show that STCGAN [27] performs relbuilding texture in the back. The depth map was atively well in removing shadows but has significant also naturally inpainted to match the building wall problems in shadow detection. On the other hand, shape. ViSha [26] is superior to the STCGAN method for
For relighting, we removed the sky area by the detecting a wide range of dark areas of an image sky mask and added a virtual sky to represent am- as shadows. Therefore, we used ViSha for shadow bient light in Fig. 7. The impact of sky removal detection and STCGAN for removal. is discussed in the next section. Because sunlight The removal results are shown in Fig. 9. From the can be seen as the only light source during the day ifgure, we can see that cast shadows are successfully in outdoor scenes, relighting is done primarily to removed from the photographed image. The results redirect sunlight. In Fig. 7(a), by the shadow of the show that the shadow removal algorithm could efred building on the right, we can see that the sun fectively remove shadows automatically. However, is shining from almost directly above. The first row the algorithm has some limitations. For example, of Fig. 7(c) shows the result when the position of the shadow removal algorithm used in this study, the sun was moved to the back of the red building. which combines ViSha and STCGAN, requires the We can see that the shadow of the red building has image to be resized to a 256 x 256 image before become longer. The cast shadow covers the whole feeding it into the network for calculation. As a road in front of the red building. The second row result, the final output image has a lower resolution of Fig. 7(c) shows the image in which the scene is than the original image. Another problem is that illuminated by dark ambient light and city lights the overall color of the image is slightly altered as such as street lamps and vending machines in the cast shadows are removed. evening and at night. The sky is the virtual one created by computer graphics. This way, various 4.4.2. Sky Removal light source settings can convert a scene from day Figure 10 shows the results of sky detection by to evening or night. PSPNet [28]. We can confirm that the sky areas are
Finally, we explain the addition of point lights. successfully detected from the results. Using one of When adding a light source, the light source is not the results, we examined the eefct of sky removal added directly to the image but to the generated on relighting. Figure 11 compares results relit with model. For example, if we want to illuminate a and without sky removal. Without sky removal, the street light, we find the position of the street light depth estimation generated shapes in the sky as in the model and insert a point light source at that well, resulting in casting shadows in various places, position. The characteristics and position of the (b) Masked (c) Inpainted (d) GT 4.4.3. Albedo Map Estimation Estimating the albedo map on the image after removing shadows is also efective in keeping the texture and color of the same material relatively constant, as shown in Fig. 12. Figure 13 shows a comparison of the results of rendering the scene (a) On the estimated model (b) On the GT model using the estimated albedo or original image under diferent light source conditions.
Figure 8: Comparison of relighting with addtional lights The results show that the relit image using the on the estimated model and the ground truth model estimated albedo map is blurred since the map has lost some texture detail. On the other hand, better results could be obtained by directly relighting the as shown in Fig. 11(b). On the other hand, the original image. This result is because the estimated unnatural shadow does not appear as in (b) when albedo map is not accurate enough. For example, the shadow is removed. It should be noted that a as shown in Fig. 12, the right side of a building virtual sky is needed instead of the real sky in the is always darker than the front side. Although it image. is expected that accurate albedo estimation would (a) Origin images (b) Shadow removed
Origin images Albedo maps Illumination
(a) With original shadow Shadow removal Albedo maps Illumination
(b) With shadow removal
In this work, image relighting is mainly based on a simple rendering pipeline process. Compared to neural rendering, it has a high degree of control, such that the lighting efects can be largely controlled by knowing the scene geometry. However, such conventional rendering requires a great deal of prior intelligence to guide the computation of realistic lighting. For example, conditions such as the smoothness of the material are not considered in this study. Instead, neural rendering focuses on generating and processing shadow maps to achieve lighting efects, which significantly reduces the amount of [2] C. Zhang, T. Chen, A survey on image-based rendering - representation, sampling and compression, Signal Processing: Image Communication 19 (2004) 1–28. [3] Y. Horry, K.-I. Anjyo, K. Arai, Tour into the picture: using a spidery mesh interface to make animation from a single image, in: SIGGRAPH, 1997, pp. 225–232.
4_ m 4_ 4 m _ m m _ [4] C. Loscos, G. Drettakis, L. Robert, Interactive Figure 14: Failure cases of our image inpainting network. virtual relighting of real scenes, IEEE Transactions on Visualization and Computer Graphics 6 (2000) 289–305. [5] E. Eisemann, F. Durand, Flash photography prior information input. enhancement via intrinsic relighting, ACM
Furthermore, since the estimated depth map is Transactions on Graphics 23 (2004) 673–678. directly converted to a model in this study, the [6] T. Sun, Z. Xu, X. Zhang, S. Fanello, C. Rhecomputation of shadows is inaccurate. For example, mann, P. Debevec, Y.-T. Tsai, J. T. Barron, shadows of the same thickness as the object are not R. Ramamoorthi, Light stage super-resolution: generated. However, this can be solved by using a continuous high-frequency relighting, ACM neural network for shadow map generation due to Transactions on Graphics 39 (2020) 1–12. its nature [31]. [7] M. El Helou, R. Zhou, S. Susstrunk, R. Timo
For inpainting, the network still has limitations. fte, Ntire 2021 depth guided image relighting In some results, complex textured areas cannot be challenge, in: CVPR, 2021, pp. 566–577. inpainted well. Also, the network cannot inpaint [8] A. Sanin, C. Sanderson, B. C. Lovell, Shadow well when large areas are lost as shown in Fig. 14. detection: A survey and comparative evaluation of recent methods, Pattern recognition 45 6. Conclusion (2012) 1684–1695. [9] M. Bertalmio, G. Sapiro, V. Caselles, We proposed a method to relight the scene in an C. Ballester, Tour into the picture: using image while removing unwanted objects. The input a spidery mesh interface to make animation RGB image and the mask image created by the user from a single image, in: SIGGRAPH, 2000, are the input to our system. First, the depth image pp. 417–424. is generated by using a neural network. Next, the [10] S. Esedoglu, J. Shen, Digital inpainting based RGBD is masked with the mask image and is in- on the mumford–shah–euler image model, Eupainted using our proposed neural network. A mesh ropean Journal of Applied Mathematics 13 model is reconstructed from the inpainted depth im- (2002) 353–370. age for the relighting process. In some scenes, cast [11] H. Li, W. Luo, J. Huang, Localization of shadow removal, sky region removal, and albedo difusion-based inpainting in digital images, estimation are selectively performed to suppress the IEEE transactions on information forensics and efects of existing light sources. Through these pro- security 12 (2017) 3050–3064. cesses, various types of relighting can be achieved [12] A. Criminisi, P. Pérez, K. Toyama, Region from a single photograph while excluding the color iflling and object removal by exemplar-based and shape of unwanted objects. Future work in- image inpainting, IEEE Transactions on Image cludes improving the performance of inpainting and Processing 13 (2004) 1200–1212. considering the use of neural network-based meth- [13] Y. Wexler, E. Shechtman, M. Irani, Spaceods for relighting. time completion of video, IEEE Transactions on Pattern Analysis and Machine Intelligence 29 (2007) 463–476.
References [14] T. Ruzic, A. Pizurica, Context-aware patchbased image inpainting using markov random [1] P. Milgram, F. Kishino, A taxonomy of mixed ifeld modeling, IEEE transactions on image reality visual displays, IEICE TRANSAC- processing 24 (2015) 444–456.
TIONS on Information and Systems 77 (1994) [15] D. Ding, S. Ram, J. J. Rodríguez, Image in1321–1329. painting using nonlocal texture matching and nonlinear filtering, IEEE Transactions on Image Processing 28 (2018) 1705–1719. ECCV, 2018, pp. 85–100. [16] D. Pathak, P. Krahenbuhl, J. Donahue, T. Dar- [31] D. Grifiths, T. Ritschel, J. Philip, Outcast: rell, A. A. Efros, Context encoders: Feature Outdoor single-image relighting with cast shadlearning by inpainting, in: CVPR, 2016, pp. ows, in: Computer Graphics Forum, volume 41, 2536–2544. Wiley Online Library, 2022, pp. 179–193. [17] Y. Zeng, J. Fu, H. Chao, B. Guo, Learning pyramid-context encoder network for highquality image inpainting, in: CVPR, 2019, pp.
1486–1494. [18] H. Liao, G. Funka-Lea, Y. Zheng, J. Luo,
S. Kevin Zhou, Face completion with semantic knowledge and collaborative adversarial learning, in: ACCV, Springer, 2018, pp. 382–397. [19] C. Zheng, T.-J. Cham, J. Cai, Pluralistic image
completion, in: CVPR, 2019, pp. 1438–1447. [20] W. Liu, X. Chen, J. Yang, Q. Wu, Robust color guided depth map restoration, IEEE Transactions on Image Processing 26 (2016) 315–327. [21] T.-J. Mu, J.-H. Wang, S.-P. Du, S.-M. Hu,
Stereoscopic image completion and depth recovery, The Visual Computer 30 (2014) 833– 843. [22] R. Fujii, R. Hachiuma, H. Saito, Joint inpainting of rgb and depth images by generative adversarial network with a late fusion approach, in: ISMAR Adjunct, 2019. [23] Z. Li, N. Snavely, Megadepth: Learning singleview depth prediction from internet photos, in:
CVPR, 2018, pp. 2041–2050. [24] K. Nazeri, E. Ng, T. Joseph, F. Z. Qureshi,
M. Ebrahimi, Edgeconnect: Generative image inpainting with adversarial edge learning, arXiv preprint arXiv:1901.00212 (2019). [25] J. Johnson, A. Alahi, L. Fei-Fei, Perceptual losses for real-time style transfer and superresolution, in: ECCV, Springer, 2016, pp.
694–711. [26] Z. Chen, L. Wan, L. Zhu, J. Shen, H. Fu,
W. Liu, J. Qin, Triple-cooperative video shadow detection, in: CVPR, 2021, pp. 2715– 2724. [27] J. Wang, X. Li, J. Yang, Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal, in: CVPR, 2018, pp. 1788–1797. [28] H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia, Pyramid scene parsing network, in: CVPR, 2017, pp. 2881–2890. [29] Y. Yu, W. A. Smith, Inverserendernet: Learning single image inverse rendering, in: CVPR, 2019, pp. 3155–3164. [30] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang,
A. Tao, B. Catanzaro, Image inpainting for irregular holes using partial convolutions, in: