GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields

1Department of Computer Science, ETH Z├╝rich, 2Computer Vision Lab, Delft University of Technology, 3NVIDIA, 4School of Computer Science, University of Birmingham
*Equal contribution

Abstract

We propose GazeNeRF, a 3D-aware method for the task of gaze redirection. Existing gaze redirection methods operate on 2D images and struggle to generate 3D consistent results. Instead, we build on the intuition that the face region and eyeballs are separate 3D structures that move in a coordinated yet independent fashion. Our method leverages recent advancements in conditional image-based neural radiance fields and proposes a two-stream architecture that predicts volumetric features for the face and eye regions separately. Rigidly transforming the eye features via a 3D rotation matrix provides fine-grained control over the desired gaze angle. The final, redirected image is then attained via differentiable volume compositing. Our experiments show that this architecture outperforms naively conditioned NeRF baselines as well as previous state-of-the-art 2D gaze redirection methods in terms of redirection accuracy and identity preservation.

Video

Method

GazeNeRF trains a two-stream-MLP structure to learn the 3D-aware of the face without eyes feature and the two eyes feature separately via a NeRF-based model. To model the rigid rotation of two eyeballs, we explicitly multiply the two eyes feature with a gaze rotation matrix. We merge the two features of the face without eyes and two eyes to the feature of the whole face. All three features are used to render the face without eyes, the eyes, and the completed face image.

Results

The animation shown below is generated by GazeNeRF with three different settings, changing target gaze directions only, changing head poses only and changing targe gaze directions and head poses at the same time.

BibTeX

@inproceedings{ruzzi2023gazenerf,
  title={GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields},
  author={Ruzzi, Alessandro and Shi, Xiangwei and Wang, Xi and Li, Gengyan and De Mello, Shalini and Chang, Hyung Jin and Zhang, Xucong and Hilliges, Otmar},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={9676--9685},
  year={2023}
}