Skip to content

Latest commit

 

History

History
203 lines (147 loc) · 8.65 KB

README.md

File metadata and controls

203 lines (147 loc) · 8.65 KB
Raw

This repository is forked from the repository, which is the source code for the best paper 3D Gaussian Splatting for Real-Time Radiance Field Rendering in SIGGRAPH 2023.

  • Followed up papers are updated here.

  • Math theory to understand what is 3D Gaussians.

  • 201 tips to understand the paper in details.

Installing and running

The installation can refer to a step-by-step Youtube tutorial.

Installation steps in Anaconda

Below packages may be needed

  • conda install -c conda-forge pillow
  • python3 -m pip install -U pip
  • python3 -m pip install pillow-heif
  • conda install -c conda-forge pcl
  • conda install -c open3d-admin open3d
  • conda install -c anaconda numpy

if error: Importing the numpy C-extensions failed.

  • pip install setuptools
  • pip install numpy
  • pip install cupy
  • pip install probreg
  • pip install pillow-heif
Running in Anaconda Prompt
  • open Anaconda Prompt
  • conda activate gaussian_splatting
  • cd C:\User<username>\gaussian-splatting
  • put images in the folder data/input
  • python convert.py -s data (wait within 5 mins)

    produced 4 new folders: images, sparse, distroted and stereo, where only the first two are important.

  • python train.py -s data (wait around 1h)
  • output GaussianSplatting ply appears in the folder output/
  • rename to out
  • cd viewers/bin
  • SIBR_gaussianViewer_app.exe -m C:\Users<username>\gaussian-splatting\output\out
  • the GUI pops up, if not, may the CUDA support problem.

Visualization

The default viewer is by SIBR. Other viewers are developing gradually. I test several viewers, including SIBR, Unity, Blender, PlayCanvas and viewer in DreamGaussian, and comment the comparsion between them as seen below:

1. SIBR Viewer
  • Source: default viewer from the paper
  • Pros.: focused view of the object when opening
  • Cons.: need keyboard to navigate, mouse control is so bad even freezes the GUI
beacon_cut_small.mp4
2. Unity Viewer
  • Source: Unity (free-to-use)
  • Pros.: can directly trim the Gaussian Splatting in the scene and export the ply; parameters are interactivly set
  • Cons.: global view of the whole scene; tilt basement; need mouse to zoom in-out; hard to control
beacon_unity_small.mp4
3. Blender Viewer
  • Source: Blender addon in Github
  • Pros.: presented as point-cloud; can directly edit(trim) in the scene and export; controlled size
  • Cons.: low resolution; only presented in render view by Cycles; slow process

File

4. PlayCanvas Viewer
  • Source: PlayCanvas online
  • Pros.: directly import .ply to show online; high resolution; easy navigation; suitable to share
  • Cons.: no editting(trim)

File

5. DreamGaussian Viewer
  • Source: DreamGaussian viewer
  • Pros.: import .ply to navigate; three modes: image, depth, alpha to show
  • Cons.: no editting(trim); can save a mesh with texture, but very bad quality

File

  • Note: DreamGaussian provides a way to get a textured mesh from one input image, but the mesh quality is very bad as seen here.
  • Open problem: how to get a high-quality mesh with texture from images of multiview.
  • Below is another idea to get a textured mesh from pointcloud (.ply) that is computed from 3D Gaussian Splatting data. The pipeline is below:
graph LR
    A(Gaussian Splatting) --> B(Clearning noise)
    B --> C(Pointcloud with color)
    C --> D(Mesh with texture)
Loading

Cleaning noisy splatters

There have been several ways to delete unwanted splatters by GUI, such as Unity, Blender Plugin, SuperSplat and so on.

Or running the code:

python 3dgsconverter.py -i input_3dgs.ply -o output_dr.ply -f cc --density_filter --remove_flyers

where input_3dgs.ply is the input path to the Gaussian Splatting .ply file and output_dr.ply is the output path to the trimmed one by filtering (density_filter) the region with higher point density and removing (remove_flyers) the lower density ones.

File

GS to pointcloud with colors

Running the code:

python 3dgsconverter.py -i input_3dgs.ply -o output_cc.ply -f cc --rgb

where input Gaussian Splatting input_3dgs.ply is produced to be pointcloud output_cc.ply with colors.

Together with the cleaning process, this process can be organized by one commond:

python 3dgsconverter.py -i input_3dgs.ply -o output_drcc.ply -f cc --density_filter --remove_flyers --rgb

Colored pointcloud output_drcc.ply can be visualized in MeshLab:

If one wants to turn the pointcloud back to Gaussian Splatting, run below:

python 3dgsconverter.py -i input_drcc.ply -o output_drcc_3dgs.ply -f 3dgs
  • Note: But if one continues to repeat the deleting process, it won't work (the produced _dr.ply file is the same).

Pointcloud to mesh with texture

It's still an open problem about reconstructing a good mesh with texture from pointcloud produced from 3D Gaussian Splattings. Below are some tryings.

Open3D

There are three ways Alpha Shapes [Edelsbrunner1983], Ball Pivoting [Bernardini1999], and Poisson Surface Reconstruction [Kazhdan2006] to reconstruct a mesh from pointcloud, however they all need the vertex normal information in advance.

MeshLab

MeshLab also has the famous Poisson Surface Reconstruction [Kazhdan2006] function to reconstruct a mesh, however the produced one is far away from the ideal one. It tries to make the mesh as smooth as possible but loses the shape features.

Function Ball Pivoting [Bernardini1999] can help to produce one. Even though the shape features looks OK, it is quite messy with some parts missing. File

Parametric Gauss Reconstruction

As introduced in the paper Surface Reconstruction from Point Clouds without Normals by Parametrizing the Gauss Formula (SIGGRAPH 2023), the method PGR can help to reconstruct a mesh from pointcloud without normals. However, the test one shows the result is not good.

File