Reconstruction of face 3D shape and its texture is a challenging task in the modern anthropology. While a skilled anthropologist could reconstruct an appearance of a prehistoric human from its skull, there are no automated methods to date for automatic anthropological face 3D reconstruction and texturing. We propose a deep learning framework for synthesis and visualization of photorealistic textures for 3D face reconstruction of prehistoric human. Our framework leverages a joint face-skull model based on generative adversarial networks. Specifically, we train two image-to-image translation models to separate 3D face reconstruction and texturing. The first model translates an input depth map of a human skull to a possible depth map of its face and its semantic parts labeling. The second model, performs a multimodal translation of the generated semantic labeling to multiple photorealistic textures. We generate a dataset consisting of 3D models of human faces and skulls to train our 3D reconstruction model. The dataset includes paired samples obtained from computed tomography and unpaired samples representing 3D models of skulls of prehistoric human. We train our texture synthesis model on the CelebAMask-HQ dataset. We evaluate our model qualitatively and quantitatively to demonstrate that it provides robust 3D face reconstruction of prehistoric human with multimodal photorealistic texturing.
Reconstruction of face 3D shape and its texture is a challenging task in the modern anthropology. While a skilled anthropologist could reconstruct an appearance of a prehistoric human from its skull, there are no automated methods to date for automatic anthropological face 3D reconstruction and texturing.
We propose a deep learning framework for synthesis and visualization of photorealistic textures for 3D face reconstruction of prehistoric human. Our framweork leverages a joint face-skull model based on generative adversarial networks. Specifically, we train two image-to-image translation models to separate 3D face reconstruction and texturing. The first model translates an input depth map of a human skull to a possible depth map of its face and its semantic parts labeling. The second model, performs a multimodal translation of the generated semantic labeling to multiple photorealistic textures.
We generate a dataset consisting of 3D models of human faces and skulls to train our 3D reconstruction model. The dataset includes paired samples obtained from computed tomography and unpaired samples representing 3D models of skulls of prehistoric human. We train our texture synthesis model on the CelebAMask-HQ dataset. We evaluate our model qualitatively and quantitatively to demonstrate that it provides robust 3D face reconstruction of prehistoric human with multimodal photorealistic texturing. 2
The development of modern technologies and the implementation of the new technologies in computer vision and deep learning have opened up wide opportunities for developing human face 3D reconstruction. 2.1
Human Face 3D Reconstruction
Manual facial approximation now is presented by the three main techniques:
anthropometrical (American) method, anatomical (Russian) method, combination (British)
method. The first one is based on soft tissue data and requires highly experienced stuff.
Russian method [
The using of the computer-aided techniques for digital data processing has opened
new possibilities for achieving realistic facial reconstruction. The facial approximation
can be carried out through a programmatic face modeling by a surface approximation
based on a skull 3D model and tissue thickness [
Recent possibilities for collecting and processing big amounts of digital
anthropological data allow to involve statistical and machine learning techniques for face
approximation problem. The applying statistical shape models representing the skull and
face morphology for the face approximation problem has been studied [
Generative Adversarial Networks
A new type of neural networks known as generative adversarial networks (GANs) [
One of the first goals solved using GANs was image synthesis. Image-to-image
translation problem was solved using conditional GAN termed pix2pix [
To overcome this difficulty a CycleGAN [
Our aim is training two deep generative adversarial model for joint 3D face
reconstruction and photorealistic texturing of prehistoric human. We use pix2pixHD [
skull2photo Framework Overview Our aim is 3D reconstruction and texture generation of a prehistoric human face from a single depth map of its skull. We consider four domains: the skull depth map domain A 2 RW H , the face depth map domain B 2 RW H , the face semantic labeling domain C 2 RW H 3, and the face texture domain D 2 RW H 3.
We train two generator models: depth map generator G1, and texture generator G2.
The aim of our depth map generator G1 is learning a mapping G : (A; N ) ! (B; C),
where N is a random vector drawn from a standard Gaussian distribution N (0; I)),
A 2 A is the input the skull depth map, B 2 B is the output face depth map, and C 2 C
is the semantic labeling of the face parts similar to [
The multimodal adversarial loss governs the training process of our texture generator G2
G ; E = arg minG;E maxD LVGAAEN(G; D; E) + +LGAN(G; D)
latent(G; E) + + latentL1
L1VAE(G; E)
KLLKL(E)
;
(1)
where E(D) is the latent code generated by an encoder network similar to [
LKL(E) = ED p(D) [DKL(E(D)kN (0; I))] ; and DKL(pkq) is an integral over a latent distribution encoded by E(D) DKL(pkq) =
Z p(z) log p(z) q(z) dz (2) (3) Overview of the proposed framework is presented in Figure 1. 3.2
Dataset Generation
For training the developed skull2photo framework a special crania-to-facial (C2F)
dataset was created [
Network Training
Our framework is contained two GAN networks. The first of them is the pix2pixHD
framework [
The second neural networks is the MaskGAN [
The network was trained and tested using the PyTorch library. It was trained using two NVIDIA RTX2080Ti captured GPU and was 200 epochs.This dataset was divided into independent training and test splits. The training of the generator G1 was completed in 27 hours and the generator G2 in 45 hours. 4.2
Qualitative Evaluation The trained model was tested on independent testing dataset to reconstruct unseen faces. Firstly, for the qualitative evaluation we reconstructed modern human faces. We used a small part of CelebAMask-HQ dataset. Secondly, we tried to reconstructed the ancient man’s face. This task is not easy because there are significant differences between modern man’s face and ancient man’s face.
Initially we generated face depth map images and semantic segmentation images using generator G1. Then, we used previous received images as input for generator G2 and reconstructed face photo texture. Finally, we selected several random style codes of face and synthesized several face samples. Examples were presented in Figure 2
Depth Map
Depth Map
1 l e d o M 2 l e d o M
Fig. 2. Examples of input data and the results of the neural network. 4.3
Quantitative Evaluation
We present quantitative results on the independent test split of our C2F dataset in in
Table 1. Depth maps predicted by the network are normalized to the range [0; 1], where 0
is the front clipping plane located at 0 mm from virtual camera, and 1 is the far clipping
plane located at 100 mm from camera. We use the L2 distance between the ground truth
face depth map and the reconstructed depth map. We compare our skull2photo
model to the skull2face [
The reported study was funded by Russian Foundation for Basic Research (RFBR) according to the research project 17-29-04509.