If you’re interested in computer vision and that’s a question you’ve been asking yourself, along with NeRFs (Neural Radiance Fields), it refers to real-time radiance field rendering of photorealistic scenes learned from (in this case small) samples of images.
Here’a a well-written explanation by Esri’s Arkadiusz (Arek) Szadkowski, on Linkedin.
In volume rendering, you deal with a 3D grid of samples (like the pixels in a 2D image, but in 3D space). For each volume sample, you project it onto the 2D viewing plane. Think of this as determining where on your screen this 3D point should appear. When you splat the projected point onto the 2D viewing plane, instead of simply putting a dot there, you “spread” its value using a Gaussian curve.
Normal distribution, also known as the Gaussian distribution, is a probability distribution that is symmetric about the mean, showing that data near the mean are more frequent in occurrence than data far from the mean.
The center of the splat (where the original projected point was) will have the highest value (the peak of the Gaussian), and the values will taper off as you move away from the center, following the Gaussian curve. As you project more and more points from the 3D volume onto the 2D viewing plane, some of their Gaussian splats will overlap.
When this happens, the splats are combined based on their weights (from the Gaussian function) to produce a smooth and blended result. As you project more and more points from the 3D volume onto the 2D viewing plane, some of their Gaussian splats will overlap. When this happens, the splats are combined based on their weights (from the Gaussian function) to produce a smooth and blended result.
What are the Pros of Gaussian splatting?
- it reconstructs very well reflecting surfaces and thin objects
- you can process and train the model in minutes when deployed in the cloud
- it’s interactive and extremely immersive
What are the Cons?
- it’s noisy, especially when observed from a different angle than the captured images
- it doesn’t seem to maintain scale (I haven’t used it yet with metric camera input)
- the orientation is relative in the local project. Haven’t tried yet add GCP
For a more in-depth explanation, with images, Dylan Ebert on Hugging Face is your man and here’s a link to the paper 3D Gaussian Splatting For Real-Time Radiance Field Rendering, also on Hugging Face.
Mondatum offers advice, guidance and support to organisations who wish to make more use of machine learning, computer vision and new cutting edge media technology to drive their businesses. For more information, get in touch – contact@mondatum.com.