APHM: Learning Neural Parametric Head Models with 2D Adversarial Objectives

Tathagata Bandyopadhyay

Visual Computing Lab, Technical University of Munich
Code | MA Thesis PDF | Slides
Identity model using Tri-plane based 3D representation
Forward deformation field based expression model


Abstract

Modeling human head is a long standing problem in computer graphics and has got many practical applications in computer games, animation and film industry. Parametric head models try to approach this problem by providing low dimensional and ideally disentangled control parameters to change the identity and expression of a human face without requiring to explicitly model each face and expression combination. While a simple PCA based parametric model can be easily obtained directly from the 3D face scans, such model is restricted to fixed template mesh topology and thus limited in capacity to represent high frequency details of any real face due to inherent linearity of the model. Recent progress in neural networks and neural implicit surface representation has enabled the computers to learn more complex parametric head models with better generalization capability. However, they still struggle to fit sparse and possibly noisy point-clouds or depth maps of real faces due to under constrained latent space.

In this thesis, we propose to learn a neural parametric head model which utilizes 2D adversarial objective on multi-view differentiable renderings of 2D normal maps along side existing 3D point-cloud based objectives to properly constrain the latent spaces, thus enabling it to fit noisy point-clouds of real face scans. Additionally, we use tri-plane based hybrid neural surface representation to leverage its 3D aware rich features for representing high-frequency details and faster convergence. We empirically show that the proposed model not only out performs most of the existing models in reconstruction and fitting, but also is more robust to scanning noise. To our knowledge, we are the first to use tri-plane based hybrid surface representation in the context of neural parametric head models.

Qualitative Results


Identity reconstruction: proposed approach works better to reconstruct the hair regions.
Expression reconstruction as neural forward deformation field.
Identity interpolation: Identity changes smoothly keeping expression unchanged
Expression interpolation: expression changes smoothly keeping the identity unchanged


Ablation Study


Ablation study: as we can see both 2D and 3D objective functions complement each-other to produe more detailed face reconstruction