OSIRIS & GSHARC
A comprehensive implementation lab guide to Neural Radiance Fields (NeRF c-o OSIRIS) and Gaussian Splatting (GS c-o GSHARC)
Embedded Files
Project OSIRIS
Open-Source Immersed Radiant Imaging Survey
Project GSHARC
Gaussian Splatting for Heritage, Archaeology, Reefs, and Conservation
Scuba.Tech Gaussian Splatting (Project OSIRIS) and NeRF (Project GSHARC) Workflow Guide
We make our documentation Sailor-proof — Updated: 3/22/2024
Table of Contents
Hardware and Software
Hardware Minimums
Software Requirements
General Notes on Ingesting & Staging Site Images
Workflow 1 – GS w/ Automatic Alignment Using COLMAP
Overall Workflow
Downloading Sources
Building the Software
Step 1 – Ingest Images and Align
Step 2 – Train / Optimize Gaussian Splats
Step 3 – View Output
Workflow 2 – GS w/ External Alignment from Metashape
Downloading Sources
Commands
NeRF Workflow
Overall Workflow
Downloading Sources
Building the Software
Step 1 – Ingest Images and Align
Step 2 – Prepare for Training
Step 3 – Train NeRF
Step 4 – Launch nerfstudio Viewer
Hardware and Software
Hardware Minimums:
8 GB of RAM
CPU supporting AVX instruction set (most CPUs around 2013 or newer)
An internet connection
NVIDIA graphics card
24 GB VRAM highly recommended for mapping small-medium vessels
20 GB VRAM useful for modeling discrete artifacts
8 GB VRAM minimum for training purposes
CUDA-ready GPU with Compute Capability 7.0+
Software Requirements:
We highly recommend using Ubuntu 22.04, which is where we have managed to get all dependencies to work
Docker + Docker NVIDIA runtime
Other dependencies installed with docker
General Notes on Ingesting & Staging Site Images
Ensure photoset has ≥70% overlap H+V axis, with no discontinuities
Working mountpoint should be on an SSD
If you have external alignment data you would like to use, proceed to Workflow 2. If not, proceed through Workflow 1.
GSHARC — Workflow 1 – GS w/ Automatic Alignment Using COLMAP
This process takes a set of photos from a shipwreck and automatically processes the dataset into a model using nerfstudio. The Gaussian Splatting method is used in this workflow.
Overall Workflow
Process in nerfstudio Docker container
Train in gaussian Docker container
View/render in gaussian Docker container
Downloading Sources
The following commands are run in a command-line terminal. Yes, you can copy+paste!
git clone https://github.com/nerfstudio-project/nerfstudio.git
git clone https://github.com/jonstephens85/nerfstudio_guassians.git
cd nerfstudio
Building the Software
Note that these commands will take a while to complete.
Once in the nerfstudio directory, run the following in the terminal:
sudo docker build -t a .
cd ../nerfstudio_guassians
In the nerfstudio_gaussian directory, run:
sudo docker build -t b .
cd ../nerfstudio
NOTE: Make sure to copy your dataset into the nerfstudio folder BEFORE you run the next docker command!
sudo docker run -it --rm -v /mnt/Experiments_3/nerfstudio:/app/ --gpus all --network=host a
This command runs the docker container. The ‘/mnt/Experiments/nerfstudio:/app/’ section takes in a folder in the first part and sets that folder to the ‘app’ folder in the container. You will need this to take in whatever images you require or other data and work with them in the container.
Note: Please make your own output directory to put results into using the mkdir command. This is critical for the next step.
After the sudo docker command, please do the following to get to correct directory:
cd ../app/
Step 1 – Ingest Images and Align
This command processes data from images into COLMAP and image formats. Please run this in the ‘nerfstudio’ directory in the container. The proper format will be shown in the next step.
Base Command:
ns-process-data images --data [PATH TO IMAGES] --output-dir [PATH TO SAVE OUTPUT] --verbose
Scuba.Tech Example:
Note: The image directory is the ‘jpg’ directory.
Note: My output directory is defined as ‘outputs_Silver_Comet/’
Note: If using ‘videos’ you must point to the video file directly.
What our command looks like:
ns-process-data images --data ./jpg/ --output-dir outputs_Silver_Comet/ --verbose
Finished Output Example:
Please then use the ‘Exit’ command to leave the current container
Make sure to copy the output directory to the nerfstudio_guassians directory
Move over to the nerfstudio_guassians directory and run the following command:
sudo docker run -it --rm -v /mnt/Experiments_3/nerfstudio_guassians:/app/ --gpus all --network=host b
This command runs the docker container. The ‘/mnt/Experiments/nerfstudio_guassians:/app/’ section takes in a folder in the first part and sets that folder to the ‘app’ folder in the container. You will need this to take in whatever images you require or other data and work with them in the container. Make sure the last flag is set to ‘b’, as opposed ‘a’ here. This makes sure you jump into the correct docker container.
Step 2 – Train / Optimize Gaussian Splats
Run the following, in the nerfstudio_guassians directory, which should be the ‘app’ directory in the container, accessible by the following command:
cd ../app
You must make sure to properly point the --data argument to the directory containing the output of the previous container. All outputs that are possible to be worked in via NeRF or COLMAP usually require a similar setup to this output container. COLMAP format is defined as a folder labeled ‘colmap’ and the ‘images’ format is defined as a set of four folders with decreasing image sizes labeled images, images_2, images_4, and images_8.
Note: Make another results or output directory.
The following command trains on output data using the Gaussian Splatting method.
Base Command:
ns-train gaussian-splatting --data [OUTPUT OF PREVIOUS CONTAINER] --output-dir [New Output Directory]
What our command looks like continuing the Scuba.Tech Example:
ns-train gaussian-splatting --data ./outputs_Silver_Comet/ --output-dir ./results
Step 3 – View Output
From the current directory you trained in, run the ns-viewer command with the configs pointed to the exact config.yml file that will be created from the ns-train command in your output directory.
Sample Command:
ns-viewer --load-config [PATH TO THE YAML FILE]
What the Scuba.Tech Example command looks like:
ns-viewer --load-config ./results/outputs_Silver_Comet/gaussian-splatting/2024-02-23_210751/config.yml
This command must be run in the directory where both the processed output and training results folders are located. This can also be the same folder you ran your ns-train command in. The output should be a link that you can paste into your browser and play around with.
A note on browser compatibility:
Firefox v. prod-latest circa. 2024-02-23 is supported
Chromium v. prod-latest circa. 2024-02-23 is supported
Safari iOS/iPadOS v. prod-latest 2024-02-23 is supported
Brave browser v. prod-latest circa. 2024-02-23 NOT supported
OR, INSTEAD OF USING A BROWSER...
Run the following command to output a .ply file:
ns-export gaussian-splat --load-config PATH --output-dir PATH
With your --load-config PATH pointing to the .yml file in your output directory and the –outpit-dir PATH being where you want the .ply file to go. This will give you a gaussian splat file.
GSHARC — Workflow 2 – GS w/ External Alignment from Metashape
NOTICE: This section is under active development by the Scuba.Tech R&D team and is not yet fully implemented. As such, the code as-shown will not reliably export a COLMAP-formatted alignment database. We are including the working copy (1) to demonstrate progress thus far, and (2) to open the discussion for any contributions from your end! (If you have insights, please feel free to email us at hello (åt) scuba (døt) tech
Downloading Sources
Note: you want to make sure you are one directory above ‘workdir’, which is where you will be cloning your repos.
git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive
cd gaussian-splatting
nano Dockerfile
>ctrl+k to delete all lines in the dockerfile
>insert custom, provided dockerfile into the Dockerfile you’ve just created
>ctrl+O to write and ctrl+x to leave the file
Commands
sudo docker build -t exta .
Note: Make sure to remember that this is you docker image labeled ‘a’, and to change it appropriately when building different containers.
mkdir workdir
cd workdir
mkdir images
mkdir sparse
cd sparse
mkdir 0
>move all your external alignment data into directory ‘0’
>move all your pictures into ‘images’
cd ../../
sudo apt-get install graphicsmagick
>Run the image_rename.py script provided. Make sure it is run in the directory labeled ‘images’
git clone -b gaussian_splatting --recursive https://github.com/yzslab/nerfstudio.git
>insert custom, provided dockerfile into the Dockerfile you’ve just created
sudo docker build -t extb .
Note: Make sure to remember that this is you docker image labeled ‘a’, and to change it appropriately when building different containers.
sudo docker run -it --rm -v [PATH TO YOUR IMAGES FOLDER, EXCLUDING THE FOLDER ITSELF]:/app/ --gpus all --network=host a
python train.py -s ./workdir/ -r 4
Change ‘r’ to whatever image resolution you want in conjunction with the 1 2 4 or 8 folders
sudo docker run -it --rm -v /mnt/Experiments/Gaussian_splat_experiment/nerfstudio:/app/ --gpus all --network=host b
from within the docker container, run the following
pip install ./submodules/diff-gaussian-rasterization
pip install ./submodules/simple-knn
PROJECT OSIRIS — NeRF Workflow
In this process, we take a set of photos from a shipwreck and have the dataset automatically processed into a model using nerfstudio. The method used inside nerfstudio is SeaThru NeRF Lite.
Overall Workflow
Process in nerfstudio Docker container
Download seathru-nerf-lite
Train in nerfstudio Docker container
View/render in Nerfstudio Docker container
Downloading Sources
The following commands are run in a command-line terminal. Yes, you can copy+paste!
git clone https://github.com/nerfstudio-project/nerfstudio.git
cd nerfstudio
Building the Software
Note that these commands will take a while to complete.
Once in the nerfstudio directory, run:
sudo docker build -t a .
NOTE: Make sure to copy your dataset into the nerfstudio folder BEFORE you run the next docker command!
sudo docker run -it --rm -v /mnt/Experiments_3/nerfstudio:/app/ --gpus all --network=host a
This command runs the docker container. The ‘/mnt/Experiments/nerfstudio:/app/’ section takes in a folder in the first part and sets that folder to the ‘app’ folder in the container. You will need this to take in whatever images you require or other data and work with them in the container.
Note: Please make your own output directory to put results into using the mkdir command. This is critical for the next step.
After the sudo docker command, please do the following to get to correct directory:
cd ../app/
Then, run this command to install seathru_nerf:
pip install git+https://github.com/AkerBP/seathru_nerf
Step 1 – Ingest Images and Align
This command processes data from images into COLMAP and image formats. Please run this in the ‘nerfstudio’ directory in the container. The proper format will be shown in the next step.
Base Command:
ns-process-data images --data [PATH TO IMAGES] --output-dir [PATH TO SAVE OUTPUT] --verbose
Scuba.Tech Example:
Note: The image directory is the ‘jpg’ directory.
Note: My output directory is defined as ‘outputs_Silver_Comet/’
Note: If using ‘videos’ you must point to the video file directly.
What our command looks like:
ns-process-data images --data ./jpg/ --output-dir outputs_Silver_Comet/ --verbose
Finished Output Example : (upload in-progress)
Step 2 – Prepare for Training
Download seathru-nerf-lite.
Run the following command to make sure your method is prepared inside the Docker Container:
ns-train seathru-nerf-lite
This downloads the dependencies for Seathru-nerf, allowing you to run the next command. You must make sure you run this before you run the ns-train command
Step 3 – Train NeRF
Make a new directory to store your NeRF:
mkdir [New Output Directory]
Then, from the same location that you ran your ns-process command, run the ns-train command. You must make sure to properly point the --data argument to the directory containing the output of the previous container. All outputs that are possible to be worked in via NeRF or COLMAP usually require a similar setup to this output container. COLMAP format is defined as a folder labeled ‘colmap’ and the ‘images’ format is defined as a set of four folders with decreasing images sizes labeled images, images_2, images_4, and images_8.
Example output from the previous container:
Base Command:
ns-train seathru-nerf-lite --data [OUTPUT OF PREVIOUS CONTAINER] --output-dir [New Output Directory]
Command following the Scuba.Tech Example:
Note: New Output Directory was named Sea-thru-Output via mkdir Sea-thru-Output at the start of Step 3.
ns-train seathru-nerf-lite --data ./output --output-dir ./Sea-thru-Output
Step 4 – Launch nerfstudio Viewer
From the current directory you trained in, run the ns-viewer command with the configs pointed to the exact config.yml file that will be created from the ns-train command in your output directory.
Base Command:
ns-viewer --load-config [PATH TO THE YAML FILE]
What our command looks like with the Scuba.Tech Example:
ns-viewer --load-config ./results/outputs_Silver_Comet/gaussian-splatting/2024-02-23_210751/config.yml
This command must be run in the directory where both the processed output and training results folders are located. This can also be the same folder you ran your ns-train command in. The output should be a link that you can paste into your browser and play around with.
A note on browser compatibility:
Firefox v. prod-latest circa. 2024-02-23 is supported
Chromium v. prod-latest circa. 2024-02-23 is supported
Safari iOS/iPadOS v. prod-latest 2024-02-23 is supported
Brave browser v. prod-latest circa. 2024-02-23 NOT supported
Page updated
Google Sites
Report abuse