OSIRIS & GSHARC

A comprehensive implementation lab guide to Neural Radiance Fields (NeRF c-o OSIRIS) and Gaussian Splatting (GS c-o GSHARC)

Project OSIRIS

Open-Source Immersed Radiant Imaging Survey

Project GSHARC

Gaussian Splatting for Heritage,  Archaeology, Reefs, and Conservation

Scuba.Tech Gaussian Splatting (Project ORISIS) and NeRF (Project GSHARC) Workflow Guide

We make our documentation Sailor-proof   —   Updated: 3/22/2024

Table of Contents

Hardware and Software

Hardware Minimums

Software Requirements

General Notes on Ingesting & Staging Site Images

Workflow 1 – GS w/ Automatic Alignment Using COLMAP

Overall Workflow

Downloading Sources

Building the Software

Step 1 – Ingest Images and Align

Step 2 – Train / Optimize Gaussian Splats

Step 3 – View Output

Workflow 2 – GS w/ External Alignment from Metashape

Downloading Sources

Commands

NeRF Workflow

Overall Workflow

Downloading Sources

Building the Software

Step 1 – Ingest Images and Align

Step 2 – Prepare for Training

Step 3 – Train NeRF

Step 4 – Launch nerfstudio Viewer

 

Hardware and Software

Hardware Minimums:


Software Requirements:


General Notes on Ingesting & Staging Site Images

If you have external alignment data you would like to use, proceed to Workflow 2. If not, proceed through Workflow 1.


GSHARC — Workflow 1 – GS w/ Automatic Alignment Using COLMAP

This process takes a set of photos from a shipwreck and automatically processes the dataset into a model using nerfstudio. The Gaussian Splatting method is used in this workflow.


Overall Workflow


Downloading Sources

The following commands are run in a command-line terminal. Yes, you can copy+paste! 

git clone https://github.com/nerfstudio-project/nerfstudio.git

git clone https://github.com/jonstephens85/nerfstudio_guassians.git

cd nerfstudio

 

Building the Software

Note that these commands will take a while to complete. 

Once in the nerfstudio directory, run the following in the terminal:

sudo docker build -t a .

cd ../nerfstudio_guassians


In the nerfstudio_gaussian directory, run:

sudo docker build -t b .

cd ../nerfstudio

 

NOTE: Make sure to copy your dataset into the nerfstudio folder BEFORE you run the next docker command!

sudo docker run -it --rm -v /mnt/Experiments_3/nerfstudio:/app/  --gpus all --network=host a

 

This command runs the docker container. The ‘/mnt/Experiments/nerfstudio:/app/’ section takes in a folder in the first part and sets that folder to the ‘app’ folder in the container. You will need this to take in whatever images you require or other data and work with them in the container.

Note: Please make your own output directory to put results into using the mkdir command. This is critical for the next step.

After the sudo docker command, please do the following to get to correct directory:

cd ../app/


Step 1 – Ingest Images and Align

This command processes data from images into COLMAP and image formats. Please run this in the ‘nerfstudio’ directory in the container. The proper format will be shown in the next step.

Base Command:

ns-process-data images --data [PATH TO IMAGES] --output-dir [PATH TO SAVE OUTPUT] --verbose


Scuba.Tech Example:

Note: The image directory is the ‘jpg’ directory.

Note: My output directory is defined as ‘outputs_Silver_Comet/’

Note: If using ‘videos’ you must point to the video file directly.

What our command looks like:

ns-process-data images --data ./jpg/ --output-dir outputs_Silver_Comet/ --verbose


Finished Output Example:

Please then use the ‘Exit’ command to leave the current container

Make sure to copy the output directory to the nerfstudio_guassians directory

Move over to the nerfstudio_guassians directory and run the following command:

sudo docker run -it --rm -v /mnt/Experiments_3/nerfstudio_guassians:/app/ --gpus all --network=host b

This command runs the docker container. The ‘/mnt/Experiments/nerfstudio_guassians:/app/’ section takes in a folder in the first part and sets that folder to the ‘app’ folder in the container. You will need this to take in whatever images you require or other data and work with them in the container. Make sure the last flag is set to ‘b’, as opposed ‘a’ here. This makes sure you jump into the correct docker container.

 

Step 2 – Train / Optimize Gaussian Splats

Run the following, in the nerfstudio_guassians directory, which should be the ‘app’ directory in the container, accessible by the following command:

cd ../app

You must make sure to properly point the --data argument to the directory containing the output of the previous container. All outputs that are possible to be worked in via NeRF or COLMAP usually require a similar setup to this output container. COLMAP format is defined as a folder labeled ‘colmap’ and the ‘images’ format is defined as a set of four folders with decreasing image sizes labeled images, images_2, images_4, and images_8.

Note: Make another results or output directory.

The following command trains on output data using the Gaussian Splatting method.

Base Command:

ns-train gaussian-splatting --data [OUTPUT OF PREVIOUS CONTAINER] --output-dir  [New Output Directory]

What our command looks like continuing the Scuba.Tech Example:

ns-train gaussian-splatting --data ./outputs_Silver_Comet/ --output-dir  ./results

 

Step 3 – View Output

From the current directory you trained in, run the ns-viewer command with the configs pointed to the exact config.yml file that will be created from the ns-train command in your output directory.

Sample Command:

ns-viewer --load-config [PATH TO THE YAML FILE]

What the Scuba.Tech Example command looks like:

ns-viewer --load-config ./results/outputs_Silver_Comet/gaussian-splatting/2024-02-23_210751/config.yml

 

This command must be run in the directory where both the processed output and training results folders are located. This can also be the same folder you ran your ns-train command in. The output should be a link that you can paste into your browser and play around with.

 

A note on browser compatibility:


OR, INSTEAD OF USING A BROWSER... 

Run the following command to output a .ply file:

ns-export gaussian-splat --load-config PATH --output-dir PATH

With your --load-config PATH pointing to the .yml file in your output directory and the –outpit-dir PATH being where you want the .ply file to go. This will give you a gaussian splat file.

 

GSHARC — Workflow 2 – GS w/ External Alignment from Metashape

NOTICE:  This section is under active development by the Scuba.Tech R&D team and is not yet fully implemented. As such, the code as-shown will not reliably export a COLMAP-formatted alignment database. We are including the working copy (1) to demonstrate progress thus far, and (2) to open the discussion for any contributions from your end! (If you have insights, please feel free to email us at hello (åt) scuba (døt) tech

Downloading Sources

Note: you want to make sure you are one directory above ‘workdir’, which is where you will be cloning your repos.

git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive

cd gaussian-splatting

nano Dockerfile

>ctrl+k to delete all lines in the dockerfile

>insert custom, provided dockerfile into the Dockerfile you’ve just created

>ctrl+O to write and ctrl+x to leave the file

Commands

sudo docker build -t exta .

Note: Make sure to remember that this is you docker image labeled ‘a’, and to change it appropriately when building different containers.

mkdir workdir

cd workdir

mkdir images

mkdir sparse

cd sparse

mkdir 0

>move all your external alignment data into directory ‘0’

>move all your pictures into ‘images’

cd ../../

 

sudo apt-get install graphicsmagick

>Run the image_rename.py script provided. Make sure it is run in the directory labeled ‘images’

git clone -b gaussian_splatting --recursive https://github.com/yzslab/nerfstudio.git

 

>insert custom, provided dockerfile into the Dockerfile you’ve just created

sudo docker build -t extb .

 

Note: Make sure to remember that this is you docker image labeled ‘a’, and to change it appropriately when building different containers.

sudo docker run -it --rm -v [PATH TO YOUR IMAGES FOLDER, EXCLUDING THE FOLDER ITSELF]:/app/  --gpus all --network=host a

 

python train.py -s ./workdir/ -r 4

Change ‘r’ to whatever image resolution you want in conjunction with the 1 2 4 or 8 folders

 

sudo docker run -it --rm -v /mnt/Experiments/Gaussian_splat_experiment/nerfstudio:/app/  --gpus all --network=host b

 

from within the docker container, run the following

pip install ./submodules/diff-gaussian-rasterization

pip install ./submodules/simple-knn



PROJECT OSIRIS — NeRF Workflow

In this process, we take a set of photos from a shipwreck and have the dataset automatically processed into a model using nerfstudio. The method used inside nerfstudio is SeaThru NeRF Lite.

 

Overall Workflow


Downloading Sources

The following commands are run in a command-line terminal. Yes, you can copy+paste! 

git clone https://github.com/nerfstudio-project/nerfstudio.git

cd nerfstudio


Building the Software

Note that these commands will take a while to complete.  

Once in the nerfstudio directory, run:

sudo docker build -t a .

 

NOTE: Make sure to copy your dataset into the nerfstudio folder BEFORE you run the next docker command!

sudo docker run -it --rm -v /mnt/Experiments_3/nerfstudio:/app/  --gpus all --network=host a

This command runs the docker container. The ‘/mnt/Experiments/nerfstudio:/app/’ section takes in a folder in the first part and sets that folder to the ‘app’ folder in the container. You will need this to take in whatever images you require or other data and work with them in the container.

 

Note: Please make your own output directory to put results into using the mkdir command. This is critical for the next step.

After the sudo docker command, please do the following to get to correct directory:

cd ../app/

Then, run this command to install seathru_nerf:

pip install git+https://github.com/AkerBP/seathru_nerf

Step 1 – Ingest Images and Align

This command processes data from images into COLMAP and image formats. Please run this in the ‘nerfstudio’ directory in the container. The proper format will be shown in the next step.

 

Base Command:

ns-process-data images --data [PATH TO IMAGES] --output-dir [PATH TO SAVE OUTPUT] --verbose

 

Scuba.Tech Example:

Note: The image directory is the ‘jpg’ directory.

Note: My output directory is defined as ‘outputs_Silver_Comet/’

Note: If using ‘videos’ you must point to the video file directly.

What our command looks like:

ns-process-data images --data ./jpg/ --output-dir outputs_Silver_Comet/ --verbose

 

 Finished Output Example : (upload in-progress)

  

Step 2 – Prepare for Training

Download seathru-nerf-lite.

Run the following command to make sure your method is prepared inside the Docker Container:

ns-train seathru-nerf-lite

 

This downloads the dependencies for Seathru-nerf, allowing you to run the next command. You must make sure you run this before you run the ns-train command

 

Step 3 – Train NeRF

Make a new directory to store your NeRF:

mkdir [New Output Directory]

 

Then, from the same location that you ran your ns-process command, run the ns-train command.  You must make sure to properly point the --data argument to the directory containing the output of the previous container. All outputs that are possible to be worked in via NeRF or COLMAP usually require a similar setup to this output container. COLMAP format is defined as a folder labeled ‘colmap’ and the ‘images’ format is defined as a set of four folders with decreasing images sizes labeled images, images_2, images_4, and images_8.

Example output from the previous container:

Base Command:

ns-train seathru-nerf-lite --data [OUTPUT OF PREVIOUS CONTAINER] --output-dir  [New Output Directory]

 

Command following the Scuba.Tech Example:

Note: New Output Directory was named Sea-thru-Output via mkdir Sea-thru-Output at the start of Step 3.


ns-train seathru-nerf-lite --data ./output --output-dir ./Sea-thru-Output

 

Step 4 – Launch nerfstudio Viewer

From the current directory you trained in, run the ns-viewer command with the configs pointed to the exact config.yml file that will be created from the ns-train command in your output directory.

Base Command:

ns-viewer --load-config [PATH TO THE YAML FILE]

 

What our command looks like with the Scuba.Tech Example:

ns-viewer --load-config ./results/outputs_Silver_Comet/gaussian-splatting/2024-02-23_210751/config.yml

 

This command must be run in the directory where both the processed output and training results folders are located. This can also be the same folder you ran your ns-train command in. The output should be a link that you can paste into your browser and play around with.


A note on browser compatibility: