Video to 4DGS

Convert videos to 4D Gaussian Splats for volumetric video playback.

Overview

4DGS (4D Gaussian Splatting) enables volumetric video by:

Extracting frames from video
Processing each frame into a Gaussian Splat
Playing back the sequence for 3D video effect

The result is a video that can be viewed from any angle.

Requirements

Hardware

GPU: NVIDIA GPU with 8GB+ VRAM (required)
RAM: 32GB+ recommended
Storage: 50GB+ free (videos generate many frames)

Software

Python 3.10+ (3.11 recommended)
NVIDIA drivers with CUDA support

Using the Desktop App

Step 1: Select Video

Open Video→4DGS tab
Drag and drop video file or click to browse
Supported formats: MP4, MOV, AVI, MKV

Step 2: Configure Settings

Setting	Default	Description
Output FPS	24	Frames per second to extract
Start Time	0:00	Where to start extraction
Duration	Full	Length to process
Quality	Balanced	Processing quality
Output Format	SOG	Frame file format

Step 3: Process

Click Process
Monitor progress:
Frame extraction
Per-frame Gaussian training
Format conversion
Manifest generation
Processing time varies by video length and quality

Processing Time

4DGS processing is compute-intensive. A 10-second video at 24fps = 240 frames, each requiring full Gaussian training. Expect several hours for longer videos.

Step 4: Export

Output includes:

output/
├── frames/
│   ├── frame_000.sog
│   ├── frame_001.sog
│   ├── frame_002.sog
│   └── ...
├── manifest.json
└── preview.mp4

Using the CLI

Basic Usage

# Convert video to 4DGS
splat-tools video2splat --input video.mp4 --output ./frames

# With specific FPS
splat-tools video2splat --input video.mp4 --output ./frames --fps 24

Advanced Options

splat-tools video2splat [options]

Options:
  --input, -i        Input video file
  --output, -o       Output directory
  --fps              Output frames per second (default: 24)
  --start            Start time (HH:MM:SS or seconds)
  --duration         Duration to process (HH:MM:SS or seconds)
  --format, -f       Output format (sog, ply)
  --quality, -q      Quality preset (fast, balanced, high)
  --gpu              GPU device ID
  --parallel         Number of parallel frame processes
  --manifest         Generate manifest.json (default: true)

Examples

# Process first 5 seconds at 30fps
splat-tools video2splat \
  --input dance.mp4 \
  --output ./dance-4dgs \
  --fps 30 \
  --duration 5

# High quality with parallel processing
splat-tools video2splat \
  --input performance.mov \
  --output ./performance-4dgs \
  --quality high \
  --parallel 2

# Extract specific segment
splat-tools video2splat \
  --input long-video.mp4 \
  --output ./clip-4dgs \
  --start 01:30 \
  --duration 10

Programmatic Usage

import { video2splat } from '@storysplat/splat-tools';

const result = await video2splat({
  input: 'path/to/video.mp4',
  outputDir: './frames',
  fps: 24,
  format: 'sog',
  quality: 'balanced',
  onProgress: (stage, frame, total, message) => {
    console.log(`${stage}: Frame ${frame}/${total} - ${message}`);
  },
});

console.log(`Created ${result.totalFrames} frames`);
console.log('Frame URLs:', result.frames);

Manifest File

The generated manifest.json contains:

{
  "version": "1.0",
  "fps": 24,
  "loop": true,
  "frameCount": 120,
  "duration": 5.0,
  "frames": [
    "frame_000.sog",
    "frame_001.sog",
    "frame_002.sog"
  ],
  "metadata": {
    "sourceVideo": "dance.mp4",
    "resolution": "1920x1080",
    "processedAt": "2024-01-15T10:30:00Z"
  }
}

Hosting 4DGS Content

After processing, upload frames to a CDN:

Upload to Cloud Storage

# AWS S3
aws s3 sync ./frames s3://my-bucket/4dgs/dance/

# Cloudflare R2
rclone sync ./frames r2:my-bucket/4dgs/dance/

# Google Cloud Storage
gsutil -m cp -r ./frames gs://my-bucket/4dgs/dance/

Using with StorySplat Viewer

import { createViewer } from 'storysplat-viewer';

// Load manifest
const response = await fetch('https://cdn.example.com/4dgs/dance/manifest.json');
const manifest = await response.json();

// Build frame URLs
const baseUrl = 'https://cdn.example.com/4dgs/dance/frames/';
const frameUrls = manifest.frames.map(f => baseUrl + f);

// Create viewer
const viewer = createViewer(container, {
  frameSequence: {
    frameUrls,
    fps: manifest.fps,
    loop: manifest.loop,
  },
});

viewer.play();

Video Capture Guidelines

For Best Results

Fixed camera position - Use tripod or stabilization
Consistent lighting - Avoid changing light conditions
Good coverage - Subject should be well-lit from multiple angles
Clean background - Simple backgrounds process better
Avoid motion blur - Use faster shutter speed

Multi-Camera Capture

For professional 4DGS:

        Camera 1
           │
           ▼
     ┌─────────────┐
     │   Subject   │
     └─────────────┘
    ▲               ▲
    │               │
Camera 2        Camera 3

Each camera position generates one viewpoint in the reconstruction.

Performance Optimization

Reduce Processing Time

Lower output FPS - 15fps instead of 24fps
Shorter duration - Process key segments only
Fast quality preset - For quick previews
Parallel processing - Use --parallel 2 (requires more VRAM)

Reduce Output Size

Use SOG format - 60-80% smaller than PLY
Lower quality - For mobile delivery
Reduce FPS - Fewer frames = smaller total size

Processing Time Estimates

Video Length	FPS	Frames	Est. Time (Balanced)
1 second	24	24	~12 min
5 seconds	24	120	~1 hour
10 seconds	24	240	~2 hours
30 seconds	24	720	~6 hours

GPU Dependent

Times vary significantly based on GPU. RTX 4090 is ~3x faster than RTX 3060.

Troubleshooting

"GPU out of memory"

Reduce quality setting
Close other GPU applications
Use --parallel 1 (sequential processing)
Reduce source video resolution

"Frames look inconsistent"

Check for lighting changes in source video
Use more stable camera (tripod)
Try --quality high for better temporal consistency

"Processing fails on certain frames"

Check source video for corruption
Try re-encoding source video
Skip problematic frames manually

"Output files too large"

Use SOG format instead of PLY
Reduce quality setting
Lower output FPS

Next Steps

Hosting 4DGS Content - CDN setup guide
4DGS Playback - Viewer integration
CLI Reference - Full command documentation

Setting	Recommendation
Resolution	1080p or 4K
Frame Rate	Match output FPS (24-30)
Codec	H.264 or ProRes
Bitrate	High (50+ Mbps)
Shutter	1/60 or faster