Recovering 3D geometry from 2D observations is a fundamental challenge in computer vision, with applications in animation, virtual reality, and robotics. Recent advances in diferentiable rendering have enabled gradient-based optimization of 3D shapes using only image supervision. In this work, we propose a novel adversarial framework that enhances 3D mesh deformation by integrating a diferentiable renderer into a Generative Adversarial Network (GAN). The generator deforms an initial mesh and optimizes textures to match 2D supervision from target images, while the discriminator-featuring dense connections and self-attention-learns to distinguish between real and synthesized renderings. Our method improves upon baseline diferentiable renderers both quantitatively and qualitatively, achieving lower Chamfer distance and higher Intersection over Union (IoU) across a variety of object categories. The results demonstrate that adversarial training efectively guides mesh deformation, producing reconstructions that are more accurate and visually consistent with target images.
generalizable reconstruction pipelines.
Despite these advances, current diferentiable methods
Reconstructing 3D geometry from 2D images is a long- often produce over-smoothed or inaccurate shapes,
parstanding goal in computer vision, with applications in ticularly when only limited views are available [
Traditional 3D reconstruction methods include point cludes dense connections and self-attention to efectively
cloud processing, voxel grids, and mesh-based optimiza- capture fine-grained spatial features [
Recent advances in deep learning have enabled sig- • We integrate texture optimization and silhouette
nificant progress, particularly with the advent of difer- supervision to refine appearance and geometry
entiable rendering. By making the rendering process simultaneously.
diferentiable, neural networks can be trained end-to-end • We demonstrate quantitatively and qualitatively
to optimize 3D shape and appearance directly from 2D that our approach outperforms baseline
diferenimages. Frameworks such as Soft Rasterizer [
ing using sparse Cholesky factorization. While efective, such techniques are computationally intensive and remain sensitive to initialization and viewpoint ambiguity.
The problem of reconstructing 3D geometry from 2D observations has been studied extensively, with a range of techniques proposed over the years. These methods 2.4. Adversarial Training for 3D Shape can be broadly categorized into classical reconstruction pipelines, deep learning-based approaches, and diferen- Generation tiable rendering frameworks. Recent eforts have also Adversarial learning has recently been applied to 3D explored the integration of adversarial training to im- tasks to improve realism and detail preservation. For prove reconstruction fidelity. instance, GANs have been employed to refine volumetric reconstructions or to hallucinate missing geometry. 2.1. Classical 3D Reconstruction However, applying adversarial training in the context of diferentiable mesh deformation remains underexplored.
Our work addresses this gap by introducing a discriminator tailored to rendered images, combining dense blocks and self-attention mechanisms to provide fine-grained feedback to the generator during training.
Traditional methods rely on structured representations such as point clouds, voxel grids, or explicit meshes. Point cloud-based methods require dense and accurate data, which is often impractical to obtain without expensive scanning equipment. Voxel-based approaches discretize space into uniform grids [? ], but sufer from memory and resolution limitations. Mesh-based methods, while 2.5. Summary eficient in representing surfaces, often require manual In summary, while diferentiable rendering has signifiinitialization and lack robustness in unconstrained sce- cantly improved 3D reconstruction from 2D supervision, narios. challenges remain in achieving high-quality, generalizable mesh deformations. Adversarial training ofers a 2.2. Learning-Based Mesh Reconstruction promising solution by encouraging visual realism and better structural consistency. Our work builds upon this direction by integrating adversarial loss into a diferentiable mesh optimization pipeline, enabling the reconstruction of more detailed and accurate 3D shapes.
Deep learning has significantly advanced mesh-based
reconstruction. Pixel2Mesh [
Diferentiable renderers provide a powerful tool for
optimizing 3D representations directly from image-space
losses. Soft Rasterizer (SoftRas) [
Nicolet et al. [14] proposed improving the stability of gradient-based optimization in diferentiable render67–74
We propose an adversarial training pipeline for 3D mesh deformation, guided by diferentiable rendering and 2D supervision. The system consists of a generator that deforms a base mesh to match a target image, and a discriminator that evaluates the visual realism of rendered outputs. The generator is optimized using a combination of reconstruction and regularization losses, while the discriminator provides adversarial feedback based on rendered RGB images and silhouettes.
Given a target image of a 3D object, our goal is to deform a source mesh (initialized as a sphere) and optimize its texture to match the target. The mesh is rendered from multiple viewpoints using a diferentiable renderer, and the rendered images are compared to the ground truth using a composite loss function. The system alternates between optimizing the generator (mesh and texture) and training the discriminator to distinguish between real and synthetic renderings.
Meshes are normalized by translating them to the ori- The total generator loss is defined as: gin and scaling them to fit inside a unit sphere. This ensures consistent scale and positioning, which simplifies ℒ = RGBℒRGB + silℒsil + edgeℒedge+ optimization and stabilizes training. Meshes are loaded using PyTorch3D’s load_objs_as_meshes function, + normℒnorm + lapℒlap + advℒadv which constructs batched Meshes objects for down- where the weights are empirically set (see Section 4). stream processing. • Laplacian Smoothing Loss: ℒlap — Penalizes
large deviations from the mean vertex position. • Adversarial Loss: ℒadv — Binary cross-entropy loss from the discriminator, encouraging realism in rendered outputs.
3.3.1. Camera Configuration
3.5.1. Architecture We use multiple perspective cameras placed at uniformly sampled viewpoints around the ob- The network consists of an initial convolutional block ject. Camera transformations are computed using followed by two dense blocks with growth rate 32 and look_at_view_transform, and projection is per- intermediate channels of 64 and 192, respectively. Each formed using FoVPerspectiveCameras. dense block is followed by a self-attention layer with scaled dot-product attention. A final convolutional layer 3.3.2. Lighting and Shading reduces the feature map to a scalar output passed through a sigmoid activation function. Spectral normalization is applied to all convolutional layers to stabilize adversarial training.
Lighting is modeled with a single PointLights source positioned above and to the side of the object. For RGB rendering, we employ a SoftPhongShader, which models ambient, difuse, and specular reflections. For silhouette rendering, we use a SoftSilhouetteShader with a thresholded alpha channel to extract binary object contours. 3.3.3. Rasterization Settings
The generator is trained to minimize a composite loss comprising multiple terms: • RGB Loss: ℒRGB — L2 loss between rendered and target RGB images. • Silhouette Loss: ℒsil — L2 loss between rendered and target silhouettes. • Edge Loss: ℒedge — Encourages preservation of mesh edge lengths to prevent distortion. • Normal Consistency Loss: ℒnorm — Promotes smooth surfaces by enforcing normal alignment between adjacent faces.
Training proceeds in alternating steps: 1. Generator step: A batch of viewpoints is sampled. The generator deforms the mesh and optimizes texture to minimize the total loss ℒ. 2. Discriminator step: The discriminator receives real target images and generated renderings. It is trained using binary cross-entropy to maximize classification accuracy.
descent with momentum, while the discriminator uses the Adam optimizer. Training is performed for a fixed number of iterations, with periodic visualizations to track progress.
training conducted on Google Colab using an NVIDIA GPU runtime. All meshes are batched for eficient parallel processing. Code modules are structured for data loading, rendering, loss computation, and model optimization.
We evaluate our method on a diverse set of 3D objects and compare it against a baseline diferentiable rendering pipeline using PyTorch3D. Both quantitative metrics and qualitative visualizations are used to assess reconstruction accuracy, mesh quality, and generalization capability.
• Reconstruction Loss: The total loss defined in Section 3, combining RGB, silhouette, and regularization terms. • Chamfer Distance: Measures point-wise similarity between predicted and target meshes. • Intersection over Union (IoU): Measures volumetric overlap between the generated and ground truth meshes. • Visual Quality: Qualitative comparisons of mesh renderings across viewpoints.
reconstruction loss and geometric fidelity. Full tables and visualizations are provided in Appendix A. Notably, our method consistently outperforms the baseline in Chamfer distance and IoU with longer training, especially for highfrequency shapes such as hand and sword.
Model accurate deformations, especially in object regions with Union scores across multiple object categories. Qualitacomplex structure or fine detail (e.g., hand, sword). These tively, it produces more realistic deformations, especially improvements validate the benefit of adversarial super- in regions with fine-grained geometry such as limbs or vision for guiding mesh optimization under weak 2D object extremities. supervision. Compared to the baseline, which often fails This approach contributes to the broader goal of buildto preserve sharp boundaries or introduces artifacts in ing generalizable, high-fidelity 3D reconstruction sysregions with occlusion or high curvature, our approach tems that operate under weak supervision. Our design maintains geometric consistency and enhances fidelity remains simple and modular, leveraging widely available to the silhouette and inner contours observed in the tar- toolkits such as PyTorch3D and standard GAN compoget images. Notably, at 10,000 iterations, the refinement nents. introduced by our method leads to significant alignment Future work will explore extending this framework to not only in the external silhouette but also in internal dynamic or articulated objects, learning category-specific features such as joint articulation and surface topology, priors, and incorporating temporal consistency for videoconfirming the progressive advantage of adversarial cues based shape reconstruction. Additionally, improving texover traditional loss-only optimization strategies. ture fidelity and integrating semantic segmentation into
Furthermore, the visual quality improvements ob- the adversarial loss are promising directions. served in later iterations indicate that the adversarial discriminator plays a crucial role in discouraging unrealistic deformations and encouraging plausible mesh Declaration on Generative AI structures even when direct pixel supervision is limited. During the preparation of this work, the authors This qualitative evidence complements the quantitative used ChatGPT, Grammarly in order to: Grammar and results reported in Section ??, and supports the hypothe- spelling check, Paraphrase and reword. After using this sis that leveraging learned priors from adversarial train- tool/service, the authors reviewed and edited the content ing leads to more robust and semantically coherent re- as needed and take full responsibility for the publication’s constructions, particularly when only sparse or partial content. supervision is available.
deformation guided by diferentiable rendering and 2D image supervision. By integrating a mesh generator with a self-attention-based discriminator, our method improves the visual quality and geometric accuracy of reconstructed 3D meshes from sparse image inputs.
Our results demonstrate that adversarial training can enhance mesh fidelity over standard diferentiable rendering pipelines. Quantitatively, our method achieves lower Chamfer distances and higher Intersection over