[crazy-robotaxi] Live-edit abilities: skin swap, coins, spawned obstacles, weather (stacked on #463)

[wenqingw-nv]

Stacked on #463 (Crazy Robotaxi standalone) — only the top 2 commits are new; implements the plan from #489 using the mechanisms from #431/#458. cc @aidanfnv

Adds an optional, flag-gated live_edit/ ability package to Crazy Robotaxi (vanilla behavior unchanged with flags off; all keys work in both the native HUD and the browser/MJPEG path):

• K — world-skin cycle (base → arcade → comic → cyberpunk → pixel): multi-style LoRA + drift corrector attach at the session seam, prompt-swap at chunk boundaries (lands next chunk, ≈0.7–1.4 s), duty-cycled re-swap keeps the style crisp indefinitely. Attach is bit-clean at rest (validated against an unattached control at the rasterizer's reproducibility floor).
• C — coin pickups: course laid on the driving-lane centerlines, projected per frame through the FTheta camera model, pickup by ego proximity, HUD counter chip.
• O — obstacle events: a template-cloned vehicle crossing the lane ahead, materialized via box-axis guidance (s=2; crossing paths materialize fully opaque — pace-matched lead clones stay ghost, so events prefer crossing templates). Visual + HIT chip; physics integration left out deliberately.
• V — weather cycle (clear → rain → snow → clear) via guided prompt swap; composes with skins (cyberpunk + rain validated as one compositional prompt).

Validation: end-to-end headless MJPEG runs (65 chunks) with all abilities; 511 CPU tests pass (+62 new in tests/test_live_edit*.py) against the synthetic fixtures; ruff clean. Known limits documented in code/tests: corrector currently requires CUDA graphs + compile off (graph-safe corrector is the follow-up), snow strength is honest-partial at the pinned guidance, revert-to-clear can't undo accumulated scene change (history-carrying model).

Demo videos attached below.

robotaxi_liveedit_demo3_rain.mp4

robotaxi_liveedit_demo3_annotated.mp4

robotaxi_liveedit_demo3_cyberpunk.mp4

---

Update: real-time performance + weather refinements (head a13d4c74)

• Fused per-state drift corrector (rides the omnidreams fused mode from the live-edit PRs): pre-merged weight-sets per game state (base / skin / weather), swapped by torch._foreach_copy_ before each CUDA-graph replay — so the corrector, the style-LoRA delta, and the per-timestep gate all run with CUDA graphs on. This also makes the dispatch the sole writer of the self-attn projections, resolving the LoRA/corrector composition cleanly.
• Measured in-game: 206.6 ms per 8-frame chunk model-side (38.7 fps), 237.9 ms end-to-end (33.6 fps) — vs ~1.4 s/chunk on the previous unfused path (~6.8×). Dispatch VRAM: 7.0 GiB (3 states). --live-edit-corrector-mode unfused keeps the legacy path.
• Weather is now a base-world-only ability (V rejected while a skin is active; activating a skin clears weather) with per-state corrector gains; rain preset re-prompted for visible rain (guidance 3.0), storm preset added behind the same gating.
• Base world now runs the photoreal clean-forcing corrector by default (--live-edit-base-corrector), reducing long-session drift with no features active.
• Coin sprite is procedural by default; --live-edit-coin-sprite <path> accepts a custom RGBA sprite.

Test suites: 87 live_edit CPU tests green; ruff clean.

---

Final feature set (head 8ae7277d)

Controls: WASD drive · K skin power-up · C coins toggle · O obstacle/traffic burst · V weather cycle · plus in-world pickups (weather icons trigger their weather; a mystery box grants a random timed skin burst) — pickups and keys drive the same ability state machines.

• Skins (arcade / comic / cyberpunk / pixel via one prompt-selected multi-style LoRA): optional timed "power-up" mode (--live-edit-skin-duration-chunks, e.g. 11 ≈ 3 s) with HUD countdown and residue-free auto-revert — style bursts end before long-hold drift begins. Swap boundary ~48 ms (prompts pre-encoded at startup).
• Weather (rain / snow / storm / hurricane; base-world-only): land-then-release guidance (~1.5 s landing at ~2×, then base-cost hold), optional timed auto-clear with a guided clear-landing — accumulated effects fade realistically (roads dry, snow melts). Rain/snow reachable by pickup items; storm/hurricane key-only.
• Obstacles/traffic: template-cloned vehicles crossing ahead, materialized by CUDA-graph-safe box-axis guidance (two graph replays during events only); burst mode spawns N staggered cars (--live-edit-obstacle-count). Static roadblock mode exists behind default-off flags (requires continuous guidance from chunk 0; quality is limited — documented, not recommended).
• Items/coins: lane-course pickups with FTheta projection, spatial-hash culling, GPU compositing (~0.6 ms/frame), nearest-64 cap, HUD counter/flash chips. All sprites are path-configured (procedural placeholders by default; no bundled assets).
• Corrector policy (per A/B evidence): clean-forcing corrector active only during skin states (in-game A/B: decisive win — without it a held skin collapses by +60 chunks; base-state corrector measured as a wash in game driving and defaults off).
• Streaming: shared-token gate on every endpoint, drop-to-latest delivery for slow clients (bounded ~1 s staleness on constrained links), --stream-jpeg-quality / --stream-scale bandwidth knobs.

Performance (GB300, end-to-end): ~34 fps steady with features on; ~31 fps with obstacle guidance armed; transients only at swap boundaries (~48 ms) and during weather landings / obstacle events (~1.5–2× for a few seconds). Native-DIT: coins/keys compatible; skins/weather require it off (the executor lacks replace_text wiring and doesn't re-read LoRA weight toggles — error messages name the exact flags; upstream item).

Tests: 203 live-edit CPU tests (664 across the app suite) against the synthetic fixtures; ruff clean. Demo videos below/attached.

all.mp4

---

Update: pickup items + nitro (head 1f6727be)

The abilities are now also reachable as in-world pickups (same state machines as the keys; keys unchanged):

• Effect items on the driving lanes at configurable rarity (--live-edit-items, --live-edit-item-spacing): weather icons trigger their weather (rain/snow; storm/hurricane stay key-only), a mystery box grants a random timed skin burst (seeded RNG, per-activation duration override), and a nitro item gives an instant 1.6× speed+acceleration boost for 4 s (pure game physics, zero model cost; re-pickup resets the timer; boosted max speed capped at 16 m/s to stay on the world model's manifold — frames verified coherent at 13–15 m/s).
• Timed weather: any activation (key or pickup) can auto-revert after --live-edit-weather-duration-chunks via a guided clear-landing; accumulated effects fade realistically (roads dry, snow melts). Weather-during-skin pickups are ignored with a HUD hint (matches the key semantics).
• All item sprites are path-configured (--live-edit-item-{rain,snow,mystery,nitro}-sprite); defaults render procedural placeholder icons — no bundled assets.
• HUD: pickup flash chips, weather/skin/nitro countdown chips.

Corrector policy per in-game A/B evidence: active only during skin states (decisive win — an unaided held skin collapses by +60 chunks); base-state corrector measured as a wash in game driving and defaults off; weather corrector gain defaults 0.

Tests: 687 passing across the app CPU suite (56 new for items/nitro); ruff clean.

[greptile-apps]

Greptile Summary

The PR adds flag-gated live-edit abilities to Crazy Robotaxi and extends their model, runtime, HUD, browser-streaming, configuration, and test integrations.

• Adds skins, weather, coins, effect items, nitro, and obstacle events.
• Adds browser-stream authentication, scaling, JPEG-quality controls, and drop-to-latest delivery behavior.
• Extends Omnidreams conditioning, LoRA/corrector dispatch, pipeline, and transformer integration.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/crazy_robotaxi/crazy_robotaxi/app.py	Wires live-edit configuration, per-rollout abilities, pickup effects, obstacle actors, nitro physics, and presentation into the application lifecycle.
apps/crazy_robotaxi/crazy_robotaxi/live_edit/style_ability.py	Implements the skin and weather state coordination with model-session prompt and corrector integration.
apps/crazy_robotaxi/crazy_robotaxi/live_edit/coin_ability.py	Implements lane-based course generation, spatially culled pickup detection, and FTheta sprite projection.
apps/crazy_robotaxi/crazy_robotaxi/live_edit/obstacle_ability.py	Implements spawned traffic actors and optional model-side obstacle guidance.
apps/crazy_robotaxi/crazy_robotaxi/streaming_presenter.py	Adds authenticated endpoints, live-edit browser input, configurable frame encoding, and bounded-latency MJPEG delivery.
integrations/omnidreams/omnidreams/pipeline.py	Extends the Omnidreams pipeline to support runtime live-edit conditioning and weight-state transitions.
integrations/omnidreams/omnidreams/_drift_corrector.py	Adds corrector deployment and fused state-dispatch support for live model transitions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Input[Native or browser input] --> Requests[Live-edit requests]
  Pickups[In-world pickups] --> Abilities[Ability state machines]
  Requests --> Abilities
  Abilities --> Model[Omnidreams prompt and weight state]
  Abilities --> Runtime[Taxi runtime and physics]
  Model --> Frames[Generated frames]
  Runtime --> Frames
  Frames --> Presenter[HUD and sprite compositor]
  Presenter --> Native[Native window]
  Presenter --> MJPEG[Token-gated MJPEG stream]

Loading

_{Reviews (13): Last reviewed commit: "Add the nitro pickup item: an instant ti..." | Re-trigger Greptile}

[greptile-apps]

Browser hint omits controls

The changed browser control hint advertises K and C but omits the supported V weather-cycle and O obstacle-spawn bindings, leaving those two live-edit abilities undiscoverable to MJPEG users.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[aidanfnv]

Awesome!
The style transfers probably need a follow-up to clear up some issues with how they look in motion (see my message in Slack), but the weather effects look incredible, and the coins and obstacles work well. My new map format will break the obstacle ability, but that will of course be on me to fix.
Thank you for this!