This paper addresses the challenge of diferentiable rendering, focusing on a novel implementation designed to integrate 3D objects seamlessly into reconstructed 3D environments, thereby creating entirely new perspectives of the scene. Our methodology leverages Neural Radiance Field (NeRF) models to reconstruct the 3D environments with high fidelity, alongside monocular depth estimation algorithms for deriving the 3D characteristics of objects from single images. The main goal of our approach lies in harmonizing the depth map output from the NeRF model with the depth data of the inserted object. This synergy enables the accurate and space-coherent placement of the object within the scene, ensuring a natural integration that enhances the overall realism of the virtual environment.
cel in spatial object organization and camera positioning through the exploitation of visual and motion cues, oferThis work bridges two pivotal areas of computer vision: ing a cost-efective yet accurate 3D modeling approach view synthesis and 3D reconstruction, each targeting dis- by producing sparse point clouds from image correspontinct goals yet sharing a fundamental connection. View dences. synthesis, an image-based rendering technique, creates Despite these advantages, the advent of deep learnnew scenes from various images and perspectives. Con- ing has pivoted the focus towards neural network-based versely, 3D reconstruction aspires to model real-world solutions for 3D reconstruction, exemplified by the appliscenes in three dimensions, crafting geometric represen- cation of convolutional neural networks (CNNs) in stereo tations from visual data. reconstruction. These learning-based strategies integrate
In our approach, we amalgamate elements from both global semantic insights for improved matching accuracy
view synthesis and 3D reconstruction to fabricate entirely [
Recent years have underscored the significance of 3D location. This technique eschews convolutions for a reconstruction from 2D imagery within computer vision, deep, fully-connected network learning a regression propelled by its vast application potential and founda- from the 5D input to RGB color and volume density, tional ties to 3D perception—endeavoring to endow sys- facilitating rendering via traditional volume rendering tems with a nuanced understanding of scene composi- techniques. tions. Amidst various methodologies aimed at enhancing A notable aspect of NeRFs is their ability to deduce eficiency, structure-from-motion and multi-view stereo 3D geometric details, such as depth, from the model’s techniques have gained prominence. These methods ex- learned representation, positioning NeRF at the core of our proposed solution that adeptly leverages both view ICYRIME 2023: 8th International Conference of Yearly Reports on synthesis and 3D reconstruction.
Informatics, Mathematics, and Engineering. Naples, July 28-31, 2023 However, the NeRF model’s tendency to simply memo$ demagistris@diag.uniroma1.it (G. D. Magistris); rize scene radiance poses challenges for scene editing or rcondarpioglui@ezdgioamg.euzn@irdoimaga.1u.nitir(Com.Na1a.pito(lJi).D. Rodriguez); manipulation—core objectives of this paper. We propose © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License a novel solution integrating monocular depth estimaCPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) tion to render a 3D object representation from a single the more advanced monocular depth estimation method image, enabling precise object integration into the NeRF- introduced in [22] to facilitate the seamless embedding derived 3D scene map. This synthesis not only enhances of 3D objects into NeRF-generated scenes. the model’s editing capabilities but also marks a signifi- Combining View Synthesis and 3D Reconstruction: cant advancement in combining view synthesis with 3D While several studies have independently explored view reconstruction for dynamic scene generation. synthesis and 3D reconstruction, few have investigated their integration for enhanced scene rendering and object insertion. Our work is inspired by the pioneering eforts 2. Related work in both fields, aiming to create a cohesive framework that leverages the strengths of each to produce photorealistic and spatially coherent scene augmentations.
In summary, our proposed method stands at the intersection of multiple research domains, drawing from and contributing to a rich body of knowledge on diferentiable rendering, NeRFs, 3D reconstruction, and monocular depth estimation. By synthesizing these technologies, we aspire to advance the capabilities of computer vision systems in understanding and manipulating complex 3D environments.
in computer vision has been the subject of extensive research, driven by their potential to revolutionize how machines perceive and interact with their environments.
This section reviews the foundational works and recent advancements in these fields, setting the stage for our proposed methodology.
Diferentiable rendering has emerged as a critical bridge between 3D models and their 2D projections, allowing gradients of the image loss to be backpropagated through the rendering process to the model parameters. [8] and [9] have laid the groundwork in this domain, 3. Implementation proposing frameworks that enable optimization over mesh vertices and textures. However, the combinatorial The proposed approach was tested with two scenarios nature of meshes makes it dificult to directly optimize with diferent levels of illumination, to study how this the geometry from multi-view images. aspect afects the final representation of the environment.
Neural Radiance Fields (NeRFs), introduced in [10], The pictures for the first scenario were taken manually have significantly advanced view synthesis by learning a with a cellphone camera. In total, we collected 40 pictures continuous volumetric scene function from a sparse set with size 652x367 following the protocol described in of images. Subsequent research has expanded on NeRFs [23], where the maximum disparity between views was to improve training eficiency [ 11], inference speed [12] no more than about 64 pixels. The images for the second [13] and reducing constrains [14]. We extend the NeRF scenario instead are taken from [24], which provides 41 methodology to incorporate 3D objects into the scenes, images with size 504x378 pixels showcasing an ofice. A leveraging its depth inference capabilities for realistic sample from both scenarios is illustrated in Figure 1. scene reconstruction. While NeRF can be initialized with the poses extracted
The field of 3D reconstruction has evolved from from COLMAP [15] we opted for a fully data-driven geometry-based techniques to deep learning approaches. pipeline, removing the requirement of both poses and Early works on structure-from-motion [15] and Multi- camera parameters as shown in [25]. The images of the View Stereo [16] laid the foundation for understanding object to embed in the two scenarios are taken from [24], scene geometry from image sequences. More recently, which provided multiples views of the object as well as CNN-based methods have demonstrated superior per- segmentation masks. However the monocular depth esformance in extracting 3D information from 2D images, timation model [26] inputs a single image and returns with notable contributions from [17] and [18] in apply- the estimated depth map, hence the multi-view dataset ing deep learning for spatial understanding. Our ap- is used only to choose among diferent poses of the emproach synergizes with these advancements, utilizing bedding object. Figure 2 shows one of the views of the deep learning for enhanced depth estimation and scene object. reconstruction. Monocular depth estimation has seen rapid progress, transitioning from traditional methods 3.1. Model Pipeline reliant on hand-crafted features to learning-based approaches that utilize neural networks for depth prediction The model pipeline for this task is divided into two main from a single image. The authors of [19] initially explored components: NeRF model training for scenario representhe potential of using supervised learning for this task, tation and image embedding for adding 3D objects into while more recent eforts by [ 20] and [21] have intro- these scenarios. Initially, depth maps are generated to duced self-supervised and semi-supervised techniques, describe the scenes in 3D, a crucial step for seamlessly achieving remarkable accuracy. Our model integrates integrating new objects in a spatially coherent manner.
pipeline is fully diferentiable, allowing for the simultaneous optimization of NeRF and camera parameters.
In order to obtain fast training and comply with computational power limitations, it was decided to implement a tinyNeRF model which simplifies the original NeRF structure presented in [24], using just 4 layers within the multi-layer perceptron (MLP) architecture (instead of 8), also discarding the skip recurrent connection, due to the shallowness of the structure. The input and output of the model remained the same, a single continuous 5D coordinate (the spatial location (x, y, z) and viewing direction (theta, phi) as inputs, and the volume density and view-dependent emitted radiance at that spatial location as outputs.
The hyper-parameters used in the training process are presented in the table 1.
work, and the camera parameters (intrinsic and extrinsic) of the images of a scenario, by minimizing photometric Parameter Value Comments reconstruction errors [25]. # epochs 5000
In general terms, a 3D sampling process is done for Optimizer Adam each pixel, where camera rays are traced through the Learning rate 0.001 with scheduler scene to collect a set of samples at (x, d) locations (posi- Loss MSE tion and view direction). These tuples are then used at each sample as input to the NeRF model to generate a Initially, the number of epochs was set to a few huncontinuous function that outputs an RGB color and its dred, but the synthesized results indicated the need for a corresponding density. larger number of epochs. This was necessary to achieve
During the training process, a random selection of pix- not only improved output image quality, avoiding blurels is rendered per input image. The purpose is to mini- riness, but also a more accurate and well-defined depth mize the reconstruction loss by comparing the rendered map. Consequently, the final number of epochs used for colors of these pixels with the corresponding ground- the training process reached approximately 5k. truth colors. It is important to note that the complete
The embedding process started with the characterization of the object to integrate into the scenario. For this purpose, the image of the object (with no background) was used as input to the monocular depth estimation algorithm [22]. This technique estimated the distance to the camera for each pixel in the RGB image, giving as a result a depth map represented as a matrix. On the other hand the NeRF model does not provide directly a depth map, but a density field, i.e. a scalar density for each point in space. This information however can be used to approximate depth as follows: 1) First the surface is extracted as a level set of the density function for a ifxed threshold, i.e. the surface is represented implicitly as () = where is found empirically. 2) Then the Depth Map is computed through Ray Casting. For the given camera pose a ray r() = o + d is shot from each pixel and sampled. The first value such that r() ≥ is considered the depth value for the current pixel.
Once we have the two images with the corresponding depth maps we simply fuse the images according to the depth values, such that each pixel in the final image takes the value of the image with the smallest depth. To adjust the scale of the embedding object relative to the scene we can simply scale the width and height of the corresponding image and depth map. Moreover we can place the object at diferent distances simply scaling the depth map.
while the depth maps extracted with diferent values for the threshold are shown in Figure 4. the depth map achieve a realistic outcome for each of the novel view.
Figure 5 shows the 3D depth map of the embedding object, obtained from the monocular depth estimation algorithm. embedding object(see Figure 7 A). However, to mitigate this phenomenon, a simple Nearest Neighbour filter was applied to the final image (see Figure 7B).
This study presents a novel approach to scene manipulation using neural radiance fields, demonstrating promising results in integrating objects into scenes with realistic depth and color. Key to success is the precise depth characterization by the NeRF model and the streamlined optimization of camera parameters within the NeRF architecture, eliminating the need for external processing and enhancing novel view rendering. However, achieving truly realistic embeddings demands careful adjustment of object positioning and meticulous pre-processing to Table 3 ing vehicle sharing through behavioral analysis: A Parameters in novel views scenario A figure 9. solution to user clustering using recency-frequencymonetary and vehicle relocation based on neighParameter Value borhood splits, Information (Switzerland) 13 (2022).
View 1 doi:10.3390/info13110511.
Room img size 652x367 px [4] S. Pepe, S. Tedeschi, N. Brandizzi, S. Russo, L. IocImage size 220x220 px chi, C. Napoli, Human attention assessment usImage size 220x220 px ing a machine learning approach with gan-based Object x shift 240 px data augmentation technique trained using a cusObject y shift 120 px tom dataset, OBM Neurobiology 6 (2022). doi:10. Object closer depth 100 21926/obm.neurobiol.2204139.
View 2 [5] A. Alfarano, G. De Magistris, L. Mongelli, S. Russo, ObRjoeocmtiimmggssiizzee 360502xx330607 ppxx Jf.oSrmtaercrzbeawssekdi,aCrc.hNitaepcotlui,reAfonrovveiol lceonntvbmehixaevriotrradnes-Object x shift 10 px tection 14126 LNAI (2023) 3 – 16. doi:10.1007/ Object y shift 110 px 978-3-031-42508-0_1.
Object closer depth 90 [6] G. De Magistris, M. Romano, J. Starczewski, C. Napoli, A novel dwt-based encoder for human pose estimation, volume 3360, 2022, pp. 33 – 40. ensure seamless integration, highlighting the importance [7] M. Woźniak, D. Połap, M. Gabryel, R. K. Nowof coherent spatial logic and image adjustments for au- icki, C. Napoli, E. Tramontana, Can we prothentic synthesis. cess 2d images using artificial bee colony?, volume 9119, 2015, pp. 660 – 671. doi:10.1007/ 978-3-319-19324-3_59.
Acknowledgments [8] M. M. Loper, M. J. Black, Opendr: An approximate diferentiable renderer, in: Computer Vision–ECCV This work has been developed at is.Lab() Intelligent Sys- 2014: 13th European Conference, Zurich, Switzertems Laboratory at the Department of Computer, Control, land, September 6-12, 2014, Proceedings, Part VII and Management Engineering, Sapienza University of 13, Springer, 2014, pp. 154–169. Rome (https:// islab.diag.uniroma1.it). The work has also [9] S. Liu, T. Li, W. Chen, H. Li, Soft rasterizer: A difbeen partially supported from Italian Ministerial grant ferentiable renderer for image-based 3d reasoning, PRIN 2022 “ISIDE: Intelligent Systems for Infrastructural in: Proceedings of the IEEE/CVF international conDiagnosis in smart-concretE”, n. 2022S88WAY - CUP ference on computer vision, 2019, pp. 7708–7717. B53D2301318, and by the Age-It: Ageing Well in an age- [10] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Baring society project, task 9.4.1 work package 4 spoke 9, ron, R. Ramamoorthi, R. Ng, Nerf: Representing within topic 8 extended partnership 8, under the National scenes as neural radiance fields for view synthesis, Recovery and Resilience Plan (PNRR), Mission 4 Com- Communications of the ACM 65 (2021) 99–106. ponent 2 Investment 1.3—Call for tender No. 1557 of [11] T. Müller, A. Evans, C. Schied, A. Keller, Instant neu11/10/2022 of Italian Ministry of University and Research ral graphics primitives with a multiresolution hash funded by the European Union—NextGenerationEU, CUP encoding, ACM transactions on graphics (TOG) 41 B53C22004090006. (2022) 1–15. [12] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, References J. Valentin, Fastnerf: High-fidelity neural rendering at 200fps, in: Proceedings of the IEEE/CVF inter[1] G. De Magistris, E. Iacobelli, R. Brociek, C. Napoli, national conference on computer vision, 2021, pp.
An automatic cnn-based face mask detection algo- 14346–14355.
rithm tested during the covid-19 pandemics, vol- [13] C. Reiser, S. Peng, Y. Liao, A. Geiger, Kilonerf:
ume 3398, 2022, pp. 36 – 41. Speeding up neural radiance fields with thousands
[