cosmic-sbs

Procedurally generates 3D side-by-side (SBS) sci-fi music videos for XR glasses (built for Viture Luma Ultra) on a RunPod GPU. A library of raymarched scenes is rendered from two offset eye-cameras, so the depth is geometrically real — no AI depth estimation, no inpainting artifacts. Feed it a single track or a whole album and it schedules a different scene per track/segment with crossfades, all driven by the soundtrack's beats and bands.

Scene library (shaders/scenes/): nebula, blackhole (gravitationally lensed accretion disk), wormhole, planet (atmosphere, rings, night-side city lights), asteroids, megastructure (alien Tron-grid city). Add your own by dropping a *.frag in that folder — it auto-registers.

Whole album

# put your tracks in a folder, then:
python -m src.album --album /workspace/album --out /workspace/out/album_sbs.mp4 \
    --eye-w 1920 --eye-h 1080 --fps 30 --segment 60
# --segment = seconds per scene; --scenes nebula,blackhole,planet to restrict

soundtrack ──► librosa beat/band analysis ──► per-frame envelopes
                                                     │
shaders/cosmic.frag  ──► render LEFT eye  + RIGHT eye  (parallax = real geometry)
                                                     │
                              hstack ──► ffmpeg ──► full-SBS .mp4 (+ audio)
                                                     │
                                   play on the Lumas, toggle 3D / SBS mode

Why true-stereo (not 2D→3D)

Each frame is rendered twice; the only difference between eyes is a sideways camera shift (u_ipd) plus a convergence shift that sets the zero-parallax plane. Because the scene is real 3D, near dust pops out and distant stars recede correctly. See the comment block at the top of shaders/cosmic.frag.

Run on RunPod

Option A — single Pod (simplest)

1. Build & push the image: docker build -t <you>/cosmic-sbs . && docker push ...
2. Launch a GPU Pod (a 4090 is plenty; A100 for 4K) from that image, attach a volume, drop a track at /workspace/track.mp3.
3. bash runpod/run_pod.sh /workspace/track.mp3 /workspace/out/cosmic_sbs.mp4

Option B — Serverless fan-out (long videos, fastest)

Deploy runpod/handler.py as a Serverless endpoint with a Network Volume attached. Submit N jobs, each rendering a frame slice (frame_start/frame_end) to the shared out_prefix; then run one CPU mux job (src/sbs.py) over all the frames. Linear speedup across workers.

Run locally (preview, no GPU cloud)

On macOS/Linux desktop it falls back to native GL so you can preview short clips:

pip install -r requirements.txt   # ffmpeg must be on PATH for the mux step
python -m src.pipeline --audio track.mp3 --out out/preview.mp4 \
    --eye-w 960 --eye-h 540 --fps 24 --seconds 10

Generate the soundtrack too (no upload needed)

src/music.py synthesizes a track from a text prompt, then the same pipeline turns it into the video. Default backend is MusicGen (public weights, no Hugging Face token):

python -m src.pipeline \
    --generate "slow cosmic ambient drone, deep space, evolving pads, no drums" \
    --seconds 30 --out out/cosmic_sbs.mp4
# higher quality: --music-model facebook/musicgen-medium  (or -large)

First run downloads the MusicGen weights (~1–3 min), then they're cached.

Optional Stable Audio Open backend (nicer ambient, but weights are gated): accept the license on Hugging Face, export HF_TOKEN=..., pip install stable-audio-tools torchaudio, then add --music-backend stable_audio.

Viture playback notes

• Output is full-SBS (frame = [left | right], double width). Copy to the glasses' host and play; switch the Lumas to 3D / SBS mode.
• If your player expects a 16:9 file, render with --half for half-SBS.
• Tuning depth: increase --ipd for a stronger pop-out (too high = eye strain); --convergence slides the zero-parallax plane (what sits "on the screen"). Start at --ipd 0.065 --convergence 0.9 and adjust to taste on-device.

Tuning the look

Edit shaders/cosmic.frag: density() (cloud shape/threshold), palette() (color), stars() (star density/twinkle), STEPS/stepSize (quality vs speed). Audio reactivity lives in the u_bass/u_mid/u_treble/u_pulse/u_energy uses.