• on: Dec. 20, 2023
• in: CVPR
• ✨Oral

pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction

• David Charatan
• Sizhe Li
• Andrea Tagliasacchi
• Vincent Sitzmann

• Project Page
• Paper
• PDF
• Code
• Twitter Thread

We introduce pixelSplat, a feed-forward model that learns to reconstruct 3D radiance fields parameterized by 3D Gaussian primitives from pairs of images. Our model features real-time and memory-efficient rendering for scalable training as well as fast 3D reconstruction at inference time. To overcome local minima inherent to sparse and locally supported representations, we predict a dense probability distribution over 3D and sample Gaussian means from that probability distribution. We make this sampling operation differentiable via a reparameterization trick, allowing us to back-propagate gradients through the Gaussian splatting representation. We benchmark our method on wide-baseline novel view synthesis on the real-world RealEstate10k and ACID datasets, where we outperform state-of-the-art light field transformers and accelerate rendering by 2.5 orders of magnitude while reconstructing an interpretable and editable 3D radiance field.

Citation

@inproceedings{charatan2023pixelsplat,
    title = { pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction },
    author = { Charatan, David and 
               Li, Sizhe and 
               Tagliasacchi, Andrea and 
               Sitzmann, Vincent },
    year = { 2024 },
    booktitle = { CVPR },
}