Lilly Tech Systems
Photo to 3D Avatar Conversion Intermediate
Converting 2D photographs into 3D avatar models is one of the most exciting applications of AI in computer graphics. This lesson covers the key techniques including 3D Morphable Models, Neural Radiance Fields, and single-image 3D reconstruction.
Reconstruction Approaches
| Approach | Input | Output | Quality |
|---|---|---|---|
| 3DMM Fitting | Single photo | Parameterized face mesh | Good (faces only) |
| NeRF | Multiple photos | Neural radiance field | Photorealistic |
| 3D Gaussian Splatting | Multiple photos | Point-based 3D scene | Photorealistic, fast |
| Single-Image 3D | Single photo | Textured mesh | Good |
| SMPL/SMPL-X Fitting | Photo or video | Body mesh with pose | Good (body shape) |
3D Morphable Models (3DMM)
3DMMs represent faces as a combination of shape and texture parameters learned from a dataset of 3D face scans. Given a 2D photo, optimization or regression finds the 3DMM parameters that best explain the image, producing a textured 3D face mesh.
Neural Radiance Fields (NeRF)
NeRF creates a 3D representation by training a neural network to predict color and density for any point in 3D space. Given multiple photos from different angles, NeRF learns a volumetric representation that can be rendered from any viewpoint. Recent advances like Instant NGP make this process take seconds rather than hours.
Single-Image Reconstruction
The most accessible approach uses a single photo to generate a 3D model. Models like PIFuHD, ECON, and SiTH predict 3D geometry from a single image by leveraging prior knowledge about human body structure learned during training.