Abstract | ||
---|---|---|
Volumetric neural rendering methods, such as neural radiance fields (NeRFs), have enabled photo-realistic novel view synthesis. However, in their standard form, NeRFs do not support the editing of objects, such as a human head, within a scene. In this work, we propose RigNeRF, a system that goes beyond just novel view synthesis and enables full control of head pose and facial expressions learned from a single portrait video. We model changes in head pose and facial expressions using a deformation field that is guided by a 3D morphable face model (3DMM). The 3DMM effectively acts as a prior for RigNeRF that learns to predict only residuals to the 3DMM deformations and allows us to render novel (rigid) poses and (non-rigid) expressions that were not present in the input sequence. Using only a smartphone-captured short video of a subject for training, we demonstrate the effectiveness of our method on free view synthesis of a portrait scene with explicit head pose and expression controls. |
Year | DOI | Venue |
---|---|---|
2022 | 10.1109/CVPR52688.2022.01972 | IEEE Conference on Computer Vision and Pattern Recognition |
Keywords | DocType | Volume |
Face and gestures, 3D from multi-view and sensors, Scene analysis and understanding | Conference | 2022 |
Issue | Citations | PageRank |
1 | 0 | 0.34 |
References | Authors | |
0 | 5 |
Name | Order | Citations | PageRank |
---|---|---|---|
ShahRukh Athar | 1 | 0 | 0.34 |
Zexiang Xu | 2 | 101 | 10.17 |
Kalyan Sunkavalli | 3 | 500 | 31.75 |
Eli Shechtman | 4 | 4340 | 177.94 |
Zhixin Shu | 5 | 13 | 5.26 |