Repository navigation
feat: add 3D world effect pack and multipass host - #9
Open
jmeistrich wants to merge 30 commits into
Open
jmeistrich wants to merge 30 commits into
jmeistrich wants to merge 30 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Use analytic sphere surfaces and radial emission for smooth round cores without tessellation cost. Refine local material lighting and rough mirror reflections while keeping the existing API. **Test plan** ✅ `bun test packages/slides-effects/tests/worldLook.test.ts packages/slides-effects/tests/world.test.ts packages/slides-effects/tests/worldRender.test.ts` ✅ Offline Chrome WebGPU compilation and smoke render with the full world graph.
Reject fully clipped and transparent sphere bounds in the vertex stage so hidden instances do not rasterize viewport quads. Preserve independent size and scale clamps for analytic intersections. Test plan: - 31 focused world tests passed. - Scoped renderer typecheck passed with zero diagnostics. - Nine Chrome WebGPU checks and offline 1080p/4K captures completed. - Fixed-workload ABBA renderer measurements completed at both resolutions.
The bloom chain filtered with an X-shaped five-tap kernel and a hard threshold, so tiny HDR points spread into box-shaped halos down the mip chain. Downsample with a 13-tap filter (Karis-averaged on the first level), upsample with a 3x3 tent, and threshold with a soft knee. Bloom props are unchanged.
Move the data-flow scenes behind short slide overlays and expand the tree to 53 nodes. Contrast the magenta cascade with a cyan pulse whose thinner, fainter curved trail leaves its spherical head distinct. Keep recycled-row focus, stage flights and voxel reduction on the shared playback clock, with stronger bloom, fog and lighting throughout. **Test plan** - ✅ Run the scoped World suites in isolated Bun processes, including the compiled deck's scene/uniform assertions. - ✅ Validate `data-flow.mdx` with the real deck compiler. - Native visual and performance verification follows separately.
Deck-local error boundaries evaluate to undefined when unlowered classes reach Hermes through Function evaluation, preventing World from rendering. Apply the native Babel preset with inline helpers while preserving host imports and captured loop bindings. Test plan: - Pass the focused compiler, effect-pack, World playback/render/resolution and GPU host suites (8 files). - Pass the scoped compiler typecheck. - Confirm World renders in the native macOS Debug audience window; full native presentation validation remains in progress.
A World with no instances still encodes an empty render pass to clear its target, and that pass drew zero instances. Dawn reports such draws through a device logging callback that react-native-wgpu forwards to console.warn on the JS runtime from the UI thread, which raced the JS thread and crashed Hermes when the next slide was prepared. Keep the pass (it still clears) but skip draws and per-instance dispatches whose count is zero. Test plan: - New render-graph test fails before and passes after; the focused compiler, effect-pack, World and GPU host suites pass (8 files, 94 tests); scoped typecheck is clean. - Native Debug macOS app: unfixed, traversing the World tour crashed entering slide 9 with the same native stack; fixed, it traversed every slide and step of all three 3D example decks without Dawn zero-count warnings or crashes.
Changed rows now hold their flash color with a steady glow for the rest of the slide while unchanged rows dim, so the settled state still shows what changed. The data flow example uses magenta heat and a low camera so depth of field separates the changed rows.
Changed rows went white-hot and bloomed magenta over their neighbours. The steady glow drops from +1.5 to +0.5 emissive and the entry flash peak from +8 to +4, so changed rows stay saturated magenta with a tight halo while the others dim. Test plan: - world.test.ts row highlight test fails before (entry 6.72, expected < 4) and passes after; world, worldRender, worldLook, worldDataFlow, worldPlayback and worldBackgroundPlayback suites: 47 pass. - Scoped TypeScript on scene.ts: 0 diagnostics. - Native Debug app, Data Flow deck: all 13 states rendered with no runtime errors; checked settled and 550 ms entry stills of the rows slide.
Each spray particle embedded every candidate source, and each source embedded the whole graph layout. The Data Flow title scene grew to 1.5M copied values from 4k unique objects. Every step re-prepares the scene and copies it into two UI mappers, so the UI heap climbed to 1.6 GB and the UI thread stalled for seconds per step. A particle only samples one source, so preparation now keeps just that one; rendering is unchanged. Test plan: - New world.test.ts spray test fails before (6 sources per particle, expected 1) and passes after. world, worldRender, worldLook, worldDataFlow, worldPlayback and worldBackgroundPlayback: 48 pass. - Scoped TypeScript on scene.ts: 0 diagnostics. - Native Debug app, fresh launch, Data Flow slide 0 steps 0-1-2-1-0 three times: UI heap peak 1640 MB before, 172 MB after; copied values per prepare 1.53M before, 118k after; max RSS 2.07 GB before, 0.68 GB after; no stalled UI samples and no runtime errors after.
On macOS, Worklets stopped its CVDisplayLink on every frame and restarted
it only after the UI runtime's callbacks finished. When a frame took more
than about 6 ms, the restart missed the next pulse, so UI animations ran
at 30 fps. Keeping the link running fixes that, but React Native macOS's
display link queues a main-thread tick for every pulse. After a main
thread stall, those queued ticks would run back to back.
Worklets now uses its own macOS display link in the existing patched
file. It keeps at most one tick queued or running, runs while frames are
requested, and stops on an idle tick (rechecked under the queue lock).
After invalidation it ignores Start requests and drops queued ticks. The
iOS path matches upstream again, and frame timestamps are read the same
way as before.
Test plan:
- `patches/tests/run-react-native-worklets-frame-queue.sh` compiles the
patched AnimationFrameQueue.mm with RCTPlatformDisplayLink and drives a
real CVDisplayLink (60 Hz). Old source: 4 failures (8 and 12 ms of work
per frame run at 30 fps; a queued tick runs after invalidate; a request
after invalidate restarts the link). New source: passes all 5 runs.
The one-hunk variant without coalescing runs 9 frames 0 ms apart after
a 133 ms stall; the new link runs one per pulse.
- Slides Debug rebuilt with the patched source (cold build into its
existing DerivedData path). The new debug dylib contains
WorkletsFrameLink; the old product's does not. UI runtime rAF probe on
data-flow, fullscreen 3420x2040 on a 60 Hz virtual display. These runs
were not under the fps lock (it was held by another job) and are
diagnostic only:
- old app: 0/4/8/12 ms of work per frame = 60/33/30/30 fps; voxel
slide 32 fps; rows slide 59 fps
- new app: 60/60/59.5/60 fps; voxel slide 60 fps; rows slide 60 fps
- Native smoke on the new app: world-tour (22 states), data-flow (13)
and cinematic-title (4) all ok, with no runtime errors and max RSS
1.15 GB.
- Not run: Release build, fps-locked present-commit series, iOS build.
React Compiler memoized the frame worklet on `renderScale?.value`, so every UIFrames render read the shared value on JS. A native World smoke logged 280 Reanimated "Reading from `value` during component render" warnings from this read. The worklet now reads it with `get()`, which the compiler does not hoist into a render-time dependency. Frames with zero particles or zero particle vertices no longer dispatch compute or draw particles. Fullscreen and composite draws, attachment clears, submit and present are unchanged. Frame data longer than its GPU buffer now throws before the upload, which stops the surface and reports through the existing frame error path. Test plan: - typeGPUShader.test.ts: new tests fail on the base and pass here: render under React Compiler + worklets reads `renderScale` 3 times in 3 renders (now 0); count 0 / vertices 0 matrix with compute and composite records the empty dispatch/draw (now skipped, with clears, ends, submit and present unchanged); oversize frame data uploads (now stops and reports once). 20/20 pass. - Debug app (Oct 28 product, task Metro): data-flow, world-tour and cinematic-title, 39 slide/step states, 0 render-read warnings, 0 errors. Two UIFrames mounted under World runtimes; served module uses `get()`.
The final React Native preset pass strips comments, which removed esbuild's PURE annotations from the executable deck output. The final transform now keeps comments; Hermes lowering, worklets and inline helpers are unchanged, and there is still one source map comment. Test plan: - compileDeck.test.ts: new pure-annotation test fails on the base and passes here; whole file 40/40, including the Hermes class/loop test. - Packaged compiler helper test (signs a helper bundle, compiles local worklets and Webview components) passes.
World sorted draw keys, which include custom WGSL, with localeCompare, so draw order depended on the engine's collation. It now compares UTF-16 code units. Test plan: - worldRender.test.ts: custom shaders differing only in letter case keep locale order on the base and code-unit order here, with matching draw offsets, instance counts and uniform size. 11/11 pass.
The World example decks were not compiled or executed by any check. New exampleShaders tests compile world-tour, data-flow and cinematic-title from their source files and: - check the compiled module has no class syntax, requires only host modules and has no Babel runtime or regenerator references; - on macOS, evaluate each module in React Native's Hermes binary with a strict host-module allowlist and expect a default export function; - render every slide, including Background content, and collect each World graph pass, covering custom geometry, SDF, custom post passes and shaft sample-count variants; - on macOS, validate every unique pass, plus all-effects graphs at low/medium/high/auto quality, with the existing Dawn shader module validator. A broken real World shader must fail it. This checks WGSL module compilation only, not pipeline layout or rendering. worldDataFlow now passes the deck source (so no effect lock is written) and asserts the named node and orb trail exist before reading them. A new workflow runs the portable tests on Linux and the Hermes tests on a hosted macOS runner. Dawn needs a Metal adapter; no runner for that is provisioned, so the Dawn test runs locally only. Test plan: - exampleShaders.test.ts under the heavy gate: 5/5 pass, including Dawn validation and the broken-shader failure. - Hermes rejects class syntax and unknown host modules in the same probe form (manual check). - worldDataFlow.test.tsx 4/4, compileDeck.test.ts -t "in Hermes" 1/1. - The workflow file parses; it has not run on GitHub.
Report insufficient frame delivery before indexing or slicing samples. Require a 60 Hz display for the fixed workloads and capture windows. **Test plan** ```sh patches/tests/run-react-native-worklets-frame-queue.sh "$patched_queue" ``` ✅ The real patched native queue completes the corrected harness without failures. A task-local zero-duration capture exits 1 with an insufficient-samples diagnostic instead of evaluating invalid ranges.
Run World checks when shared compiler inputs change and document the native frame harness's local display and source prerequisites. **Test plan** ✅ Parsed the workflow with Ruby YAML and verified all three compiler input paths select the portable and Hermes jobs. Reviewed the documented command against the existing runner; no runtime source changed.
The 3D example decks import slides:world, which is not published, so a
fresh checkout failed with 'Effect pack "world" is not in the catalog'
unless LEGEND_SLIDES_EFFECT_PACKS was set.
A pack at <deck folder>/effects/<id>/pack.json now resolves from source
before installed snapshots or the catalog, so no install prompt appears.
The lockfile records it as { "path": "effects/<id>" } instead of a
version and checksum, so edits neither fail verification nor rewrite
the lock. A lock path can point elsewhere in the repository; the 3D
example decks and the world example use one to reach
packages/slides-effects/packs/world. The pack's manifest is already a
compile dependency, so the app watches its folder like the deck's.
**Test plan**
```sh
bun test packages/presentation/src/__tests__/effectPacks.test.ts
```
✅ New tests cover local-before-installed resolution, the relative lock
entry staying byte-identical across pack edits, lock paths outside the
deck folder, a missing local path failing without an install request,
and installation preserving local lock entries without the catalog.
✅ All four World decks compile with LEGEND_SLIDES_EFFECT_PACKS unset and
no installed packs, tracking the pack's source files.
✅ Scoped typecheck of the changed compiler and lock consumers: 0 diagnostics.
Native WebGPU drawables were sized in points and ignored the slide's display scale, so background worlds rendered at half the Retina resolution, and auto quality could halve that again. Back every surface at device pixels, as the web host already did. The render graph gains optional 4x MSAA targets: colour resolves once in the last render pass that writes it, and depth is read as texture_depth_multisampled_2d. World uses it for the HDR scene, depth and the mirror reflection (except at low quality), renders reflections at full scale so thin edges no longer break up, and ends with FXAA after tonemapping. Rings use 128 segments. Shafts render at half resolution with interleaved-gradient jitter, and fog reads them through a 3x3 tent, so they no longer look like grain. Auto quality now lowers resolution only when cadence collapses below ~40 fps, and never below 0.75. Per-frame uniform uploads reuse staging arrays instead of allocating.
A `>` inside a JSX expression, such as `<Rows changed={step >= 1} />`,
ended the tag regex early, so the tag counted as an unclosed opening
tag and every later `---` stayed inside the first slide. Scan tags
with expression, brace and string state instead, skipping children
expressions whose lines expression depth already covers. Notes
scanning also stops resetting its tag state on `<` or `>` inside
braces, so attribute strings after a comparison stay literal.
A dashed cyan line ran along every pillar base in cinematic-title. The reflection pass kept faces lying exactly on the mirror plane, so each box's bottom face, seen at a grazing angle from the mirrored camera, picked up full rim light and showed through the floor as a broken line. Mesh reflections now discard at the floor plus .001, as SDFs already did. Opaque boxes also drop back faces, which removes the coplanar bottom face from the main pass and halves box fragment work. Cheaper post passes with the same output: - The fog's 3x3 shaft tent and the bloom upsample tent use four bilinear taps at half-texel offsets instead of nine. - The shaft march hoists the per-step segment length and replaces pow(x, 1.5) with x * sqrt(x). - The GPU frame callback writes its simulation epoch only when the slide, step or phase changes, not on every frame.
The 1000-voxel data-flow slide held 30 fps because every UI frame built a 32k-number array, sampled every voxel's position into an id-keyed record and recomputed all slots, then copied the array again for upload. World now samples into one Float32Array per prepared scene and uploads it directly. Instances whose tracks never animate are written once, voxels only when their collapse progress changes, positions are computed by id only for instances that edges or pulses read, and the counter source is indexed instead of searched. Test plan - world suites: 50/50, with a new test that reused sampling keeps one buffer and matches fresh sampling through a collapse (fails when voxel slots are treated as static) - scoped tsc on the changed sources and benchmark scripts: 0 diagnostics - native audience window, Debug build: data-flow 4 30.7 -> 59.9 fps on the first trace; stills unchanged
react-native-webgpu reports each createTexture size as Hermes external memory, and Hades counts external bytes toward its 3 GiB heap limit. World's device-resolution MSAA targets across the presenter and audience windows pushed the UI worklet runtime past it, aborting with an OOM. Textures created by the device now opt out, like the canvas texture.
On macOS the drawable stayed at the view's size in points, so a fullscreen Retina audience window rendered 3D slides at 1710x1020 and upscaled them. getNativeSurface() and getContext() return new wrappers on every call, so setting a size on one never reached the context that renders. Each canvas now keeps one context, which both sizes the drawable and renders, on every platform. Consecutive render passes that load the same attachments now share one encoder pass, so tiled GPUs keep multisampled color and depth on chip instead of storing and reloading them between draws.
With the native drawable now at device pixels, a fullscreen Retina audience window has 1.5 or more device pixels per slide unit. Auto quality drops MSAA there; FXAA and the analytic edges handle anti-aliasing. Thumbnails and explicit high quality keep 4x MSAA.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a small generic multipass WebGPU host and an installable
worldpack for deterministic 3D scenes on macOS and web. Scaled HDR/depth targets, mips and ordered compute/render/fullscreen passes share the existing shader preparation, preview sampling, UI-thread frames, diagnostics and deferred disposal. Resizing keeps compiled pipelines.The pack provides camera shots/orbit/follow/stage flights, instanced primitives, graph cascades, curved path pulses, sprays, recycled rows and voxel reduction with a counter. Its pipeline includes mirror reflections, materials/lights, height fog, volumetric shafts, DOF, bloom, chromatic aberration and tonemapping. Authors can reorder HDR stages and insert WGSL
Pass/Custom/SDFhooks. Examples include a 15-slide tour, five-slide data-flow deck, three-slide cinematic deck and gallery.World.renderScaleand callback-driven auto resolution control offscreen resolution; paused/outgoing frames retain their scale. The HDR post chain reuses two targets while preserving explicit custom reads. The pack is version 1.0.0 on host API 2; v1 packs remain compatible. Changes cover Slides, presentation, slides-effects and slides-web, with no new native dependency or native rebuild.Selected GPU backgrounds receive their renderer provider and sample slide-relative World time while retaining camera continuity by id. Canvas ownership transfers on the UI queue after retirement. Each topology generation owns its uniform mapper and GPU subtree; overlays, camera entry anchors and resolution history persist across scene-size changes. This fixes blank backgrounds, already-settled camera shots, cross-device screen attachments and stale uniform capacities on scene revisits.
Spheres, orbs and spray particles render as analytic round impostors: one six-vertex quad per instance with curved normals, surface depth, antialiased circular coverage and hot radial cores, and hidden spheres are rejected in the vertex stage. The mirror floor stays dark instead of picking up the object rim; reflections fade with roughness, distance, fresnel and finite edges. Point and spot lights use inverse-square range attenuation, GGX specular, Schlick fresnel and energy-conserving diffuse, with finite HDR headroom so smooth metals don't overflow bloom. Bloom now downsamples with a 13-tap filter (Karis-averaged on the first level), upsamples with a 3×3 tent and thresholds with a soft knee, so small bright objects get round halos instead of box-shaped ones. Existing props are unchanged.
Validation
Final head
db5caf4f(look merge plus bloom kernel), all throughscripts/heavy slides --:world.test.ts20,worldPlayback.test.tsx6,worldRender.test.ts10,worldResolution.test.ts5,worldLook.test.ts4,gpuRenderGraph.test.ts5,typeGPUShader.test.ts17 andpackages/slides-web/tests/export.test.mjs9 passed, 0 failed. Changed-source scoped typecheck: 16 files, zero diagnostics.file://exports of all three decks re-audited with networking blocked: cinematic 10 WebGPU + 4 fallback states, tour 22 + 22, data-flow 21 + 13, zero player/runtime/GPU/browser errors or external requests. Orbs, sprays and their halos are round in every deck.Earlier phase-3 runs (
5d6d70df), usingbun test --preload ./apps/slides/src/__tests__/nativeMock.ts <file>in separate processes:packages/slides-effects/tests/worldPlayback.test.tsx: 6 passed / 487 assertions, including compiled isolated worklets, uniform generations, camera entry, overlay identity, auto scale and resize.packages/slides-effects/tests/worldBackgroundPlayback.test.tsx: 1 passed / 23 assertions using real BackgroundHost registration and clocks.apps/slides/src/__tests__/deckRuntime.test.tsx: 15 passed / 109 assertions.gpuSetup.test.ts,typeGPUShader.test.ts,gpuRenderGraph.test.ts,gpuResourceLifecycle.test.tsx: 28 passed, including deferred web/macOS ownership, cancellation and disposal.scripts/typecheck-queue slides --: 15 source files, zero diagnostics; World, DeckRenderer and TypeGPUShader each renewed after their final edits with zero diagnostics. Installed TypeScript 5.9.3 /WebGPU 0.1.72 declarations; no whole-project typecheck.LEGEND_SLIDES_EFFECT_PACKS=packages/slides-effects/packs bun packages/slides-web/export.ts <deck> --out <local-output>passed for all three decks.file://audits with networking blocked: cinematic 10 WebGPU + 4 fallback states; tour 22 + 22; data-flow 21 + 13. All 92 states passed with zero player/runtime/GPU/browser errors or external requests, including direct/reverse entry, camera continuity, pulse/row flashes and voxel collapse to 102.Codex and focused Opus final source reviews found no remaining blockers. Apex completed a pinned Phase 2 review without confirmed bugs; Phase 3 was outside that scope; a final-head Apex review is pending. Native macOS visual/allocation smoke remains unverified. Screenshots exist locally and are not attached.
Measurements
Look and bloom changes (unpaced completed-submit renderer throughput, same harness and fixture as below, not displayed FPS). Round impostors, floor and lighting against their phase-2 base, 720 samples per side, ABBA: median change 1080p low/medium/high +10.3% (a separate confirmation run measured −3.2%; no consistent change) / 0.0% / +2.1%, 4K low/medium/high +0.6% / +0.8% / +0.6%. Bloom kernel against the merged head, two runs per side, ABBA: 1080p/high completed median 5.53→5.65 ms and GPU median 3.58→3.62 ms (+1.2%); 4K/high completed 15.15→15.45 ms and GPU median 12.09→12.09 ms.
Phase 3:
Apple M4, macOS 26.6.1, Chrome 156 headless/WebGPU Metal, DPR 1. The seeded 269-instance all-effects fixture uses exact matching viewport/canvas sizes. Baseline is immutable Phase 2
3d85dfffd5a95b010e70995beea67b9b72ca3221; measured candidate is5f1e5969bb5d32fa67acdd71f7dedf09510ce472; final head is5d6d70dfa2f5981e50e6b64a0d452666b2a79dd1. The final React lifetime fixes are outside the byte-identical measured renderer closure. PR baseeb6ef87eef7a44a8b272fdab6264d69f2c29797chas no equivalent 3D pipeline.The raw harness bundles production scene/render/encode/resolution helpers with Bun browser defaults, without React/app-shell overhead. Offline exports use the normal production minified browser player. Neither measures native Debug/Release FPS. Each configuration runs baseline/head/head/baseline, 30 warmup frames then 3×120 samples per run: 720 measured samples per side. The fps gate covers measurement. Completion includes uniforms/upload, encoding, submission and queue completion.
Timing changes are mixed; no general throughput improvement is established. Baseline rerun drift reached 10.3% for 1080p/medium. The structural gain is 12→10 offscreen targets: estimated 4K/high format bytes fall 433.70→307.14 MiB (−29.2%). These estimates exclude swapchains/driver overhead and are not measured process/GPU peak memory.
Separate paced high/auto runs at both exact resolutions observed about 60.00 callbacks/s with zero intervals above 25 ms over 720 samples per configuration. Head 4K/high completed median/p95 was 13.6/ 15.2 ms; auto 13.5/ 15.2 ms and remained at scale 1. Manual 4K/high scale 0.75 completed at 9.9/ 12.2 ms with about 60.00 callbacks/s, trading internal resolution for headroom.
A separate actual cinematic-export run used a 3840×2160 viewport with 3840×2109 GPU backing (controls consume 51 px), scale 1, one active queue, 1.2 s excluded warmup and a 6 s capture. It observed 360 submissions / 60.0104 submissions/s; submission spacing median 15.30 ms, p95 23.90 ms, max 45 ms, with 3 intervals above 25 ms. This is submission cadence for a different workload, without GPU-completion or physical-presentation measurement. Its p95 exceeds the 16.667 ms budget by 7.23 ms; stable displayed/native 4K60 remains unverified.
Per-pass Metal timestamp scopes overlap and are non-additive. Permitted paid work remained registered on the shared machine. Full samples, drift, per-pass scopes, environment and immutable source proofs are retained locally. All 37 manifest files match the clean final commit; manifest SHA-256 is
f8247c42915e54df76010d8e62a89a25573e2febd25041e97cb0f5bae6b8ee3c. Measured/fresh head HTML is byte-identical SHA-256a29470584f74153c27fefe64cf07ca37c09a991c94ff5f1982922c89970026d7.Auto uses callback cadence as an indirect pressure signal; CPU load, low-refresh displays or GPU backlog can mislead it. Uniform evaluation/upload scales with instance count, with graphs targeting 20–60 nodes. Quarter-resolution shafts trade bands for fixed spatial noise at low sample counts. Post-tonemap custom hooks make grain follow offscreen resolution.