🧪 HTML-in-Canvas desktop: live-texture window effects (experimental, not for merging) - #431
Draft
AllTerrainDeveloper wants to merge 12 commits into
Draft
🧪 HTML-in-Canvas desktop: live-texture window effects (experimental, not for merging)#431AllTerrainDeveloper wants to merge 12 commits into
AllTerrainDeveloper wants to merge 12 commits into
Conversation
Puts the whole shell inside a `<canvas layoutsubtree>` and mirrors it into a PixiJS texture via the experimental HTML-in-Canvas API, so fragment shaders can post-process every pixel of the desktop at once. Ships three effects — scanlines, a CRT tube, and pixel art — behind a new OS Settings → Experimental tab, switchable live. The shell is moved, not cloned. `layoutsubtree` keeps the canvas's direct children laid out, hit-tested and in the accessibility tree, so windows, iframes, text fields and links keep working untouched: the canvas is a display surface, never an input surface. Three constraints shaped the design: - The wrap must happen in JS. Content inside a `<canvas>` is fallback content, so emitting the wrapper server-side would blank the desktop for every browser without the API. - Moving the shell re-parents every iframe, which reloads it. The wrap therefore happens at boot before any window exists, and session restore is gated on it. At runtime the master toggle only wraps live when no iframe window is open; otherwise it offers a reload. - The stage proves it can upload before touching the shell. PixiJS uploads from inside the browser's paint event, where a throw is uncatchable and repeats every frame, so a failure there would leave a blank desktop. A pre-flight probe plus a runtime failsafe (unwrap after repeated upload errors) make that unreachable. Includes a shim for a real API mismatch: PixiJS 8.19 still calls the legacy 6-argument `texElementImage2D`, while Chromium 150+ implements the finalised 3-argument signature. Chrome's own PixiJS demo carries the same patch. It installs only when the browser has the new signature, and only rewrites legacy-shaped calls. Extensibility follows the house pattern: a shared-store registry, a `wp.desktop.stage.*` API, and `desktop_mode_register_screen_effect_script()` so a plugin's effect appears live on activation without an F5. Accessibility: effects honour `prefers-reduced-motion`, and the CRT's flicker rate is capped at 3 Hz (WCAG 2.3.1) — an earlier revision ran at a fixed 10 Hz, squarely in the photosensitivity range. Bumps pixi.js to 8.19.0 and vendors its `html-source` browser bundle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
Animates individual windows through their lifecycle — open, close, minimise, restore, maximise, unmaximise, focus, blur — as PixiJS objects on the canvas stage. Configured in OS Settings → Experimental under "Window animations". No CSS transitions are involved. A window becomes a Pixi object without being reparented into the canvas. `HTMLSource` requires its element to be a direct child, so moving windows there would reload every iframe and break the window manager's layout, z-order and snapping. Instead the stage's texture already contains every window, so the engine freezes the window's rectangle out of it with `renderer.generateTexture()`, hides the real element, and hands the effect a sprite of those frozen pixels — the pattern PixiJS documents for shatter-style effects. The engine owns capture, positioning, timing and cleanup; a def owns only the animation. Four built-ins ship through the same public API a plugin would use: scale & fade (all transitions), genie (minimise/restore), morph (maximise/unmaximise), and vanish — the dissolve, which slices the texture into a particle grid that drifts and fades in a sweep across the window. Adds the close gate this needs. `Window.close()` finalised on the CSS `transitionend` with a hardcoded 300 ms backstop, which truncates anything richer. It now fires `desktop-mode.window.close-animation` first; a handler returning a duration claims the close, the transitionend shortcut is skipped, and the backstop is sized to fit — capped at 3 s so a miscalculated duration cannot strand a window mid-teardown. `destroy()` bypasses the filter, so forced teardowns stay synchronous. Capture lands on the "before" state by design: the stage texture only re-uploads on browser repaint, so at the instant a lifecycle action fires it still holds the previous frame — the window as it looked before it minimised. That is the only way to animate transitions whose sole notification arrives once the element is already hidden. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
Two separate faults, which together produced a confusing pattern: close,
minimise and maximise animated on native windows, nothing animated on
iframe windows, and open/focus/blur did nothing anywhere.
Iframe closes skipped the effect. Closing an iframe window is
re-entrant — the first `close()` posts a beforeunload query to the iframe
and returns, and the response (or a 500 ms timeout) sets
`_suppressCloseFilter` and calls `close()` again. That flag exists so the
cancellable close filter is not re-run on the second pass; reusing it for
the animation gate meant an iframe window could never claim its own close
animation. Split into a separate `_suppressCloseAnimation`, set only by
`destroy()`, so forced teardowns stay synchronous while ordinary iframe
closes animate.
Open, focus and blur never ran for any window. Those arrive as document
CustomEvents carrying only `windowId`, so the engine looks the element
up — but the DOM id is `wp-window-<id>`, not the bare id, so the lookup
always returned null. The action-driven transitions receive `element` in
their payload, which is exactly why they were the ones that appeared to
work.
Also hardens the stage against a crash the effects exposed: a resize
rebuild destroyed the effects overlay with `{ children: true }`, killing
in-flight effect sprites without aborting their animations, and the next
ticker frame threw on a destroyed object — taking every other ticker
listener, including the stage's own repaint, down with it. The overlay
now survives rebuilds, the body-class observer only reacts to the
fullscreen class actually changing (minimising a window touches body
classes), and both built-in effects bail if their sprite has gone.
CRT curvature now defaults to 0 and is labelled as offsetting clicks.
`getElementTransform()` returns a DOMMatrix, so it can keep hit-testing
aligned for affine transforms — which is why the rotating PixiJS demos
stay clickable — but barrel distortion is not affine and no matrix can
express it. Adds a `bezel` parameter that applies the same barrel maths
to the glass EDGE only, giving the rounded-tube framing with the picture
left where the DOM thinks it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
Windows can now hang from their title bar like fabric while you drag them: a Verlet solver drives a MeshPlane's vertex grid, pinned to the live title-bar edge, so the sheet lags, swings and settles. `drag` joins the transition list as the first SUSTAINED one — it runs until drag-end aborts it rather than for a duration. Getting there turned up three separate bugs, each worth its own note. Only one drag ever worked. PixiJS's GlMeshAdaptor owns a single renderer-lifetime Shader, and BindGroup.setResource subscribes it to every texture source it binds. TextureSource.destroy() emits 'change' with `destroyed` already true, and BindGroup.onResourceChange responds by destroying ITSELF — so releasing a captured texture permanently nulled the shared mesh shader's bind group, and every mesh draw after that threw `Cannot read properties of null (reading '0')`, every frame, for the rest of the page's life. Sever 'change' before destroying; 'destroy' and 'unload' stay intact, so the GPU memory is still freed on schedule. Sustained effects had a 60-second ceiling. There is no honest number there — a long drag is ordinary, and any ceiling eventually fires mid-drag and re-renders the real window behind its own animation. The failsafe is now a fact rather than a clock: a pointerup or pointercancel anywhere in the document ends the effect. Momentary effects keep their 4-second watchdog, where a stuck effect really would leave a window nobody can click. The hand-off blinked. Two causes, and the second is the one that mattered. The stage draws a once-per-frame snapshot of the DOM, so restoring a window does not put it back on screen for another frame or two — the stand-in now outlives the restore by two frames, overlapping identical pixels instead of leaving a gap. And windows carry `opacity 0.2s ease` in their base transition list, so writing the property never hid or showed a window, it animated it. `--dragging` masked that at drag start but comes off at pointer-up, long before a sustained effect finishes settling. Both writes now happen with transitions pinned off — around the write only, so snap-drag's own 90 ms transition survives. Also drops the focus and blur transitions. An effect animates a COPY of the window, and focus fires mid-click: hide the original and it swallows the click that caused it, leave it visible and every click flashes a ghost. There is no third option, and unfocus effects already serve that job with CSS on the real element. 17 new tests across texture retirement, the hidden-window contract and the hand-off. Docs cover `drag`, Cloth, the hiding rules and the PixiJS mesh-shader trap, which catches any effect that destroys its own textures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
Dragging a window that sat behind another froze the OVERLAPPING window into the stand-in, so the cloth animated the wrong pixels for the whole drag. Not a stacking bug — the window is already on top by then. `focus()` restacks synchronously on the pointerdown that arms the drag. What lags is the picture: the stage draws a snapshot recorded in the browser's paint event and uploaded on the following render, and a drag begins on pointermove, frequently in the very same frame as the pointerdown that did the raising. So the DOM has the window on top while the snapshot still has it underneath. The obvious fix — wait for the snapshot before capturing — was worse than the bug on both counts, and both are worth recording. Waiting to capture means waiting to START, so the real window carried on being dragged, unaltered, for a beat before the animation appeared. A shorter wait would only have been a shorter ugly thing. And handing the callback back through `ticker.addOnce()` put `captureRegion()`, which runs a whole `renderer.render()`, inside PixiJS's own render loop. Re-entrant rendering, which PixiJS does not support, and the frame rate showed it. So nothing waits. The stand-in goes up immediately with whatever the snapshot holds, the real window stays visible underneath it — it has to be in the next snapshot for a clean capture to contain it — and `recaptureRegion()` then repaints the SAME texture from that snapshot before hiding the element. The texture object never changes, only its pixels, so effects that built a mesh or a thousand particle sprites around `ctx.texture` need no API to hear about it. The occluder survives for exactly one frame, behind an animation that is already moving. Only `drag` corrects. Minimise, maximise and close are announced AFTER the change, where the stale snapshot is the whole point: it is the only surviving record of what the window looked like before, and repainting it would capture the aftermath. Their stacking is safe anyway, since the pointerup that triggers them lands frames after the pointerdown that raised the window. `afterNextSnapshot()` waits on the texture source's own `update` event rather than counting frames — that event IS the paint having happened — then hops one task to get clear of both the paint step and the render loop before doing GPU work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
…hting it An opening window looked see-through: the effect animated the wallpaper that had been behind it, then the real window appeared. Correcting the capture from a later snapshot (the previous commit's machinery) was not enough on its own, because the window manager's own opening animation was still running underneath. `Window`'s constructor adds `desktop-mode-window--opening`, and `window-states.css` runs a 200 ms `opacity: 0 → 1` + `scale(0.92 → 1)` keyframe on it. That defeated the effect twice over. The corrected capture landed on a window still at roughly zero opacity, so the effect animated a ghost. And a running CSS animation outranks inline styles, so the engine's own `opacity` hide did not apply until the animation ended — the real window showing through its own stand-in for the duration. The engine now removes that class when an effect claims the open. It runs in the same synchronous block that mounted the window, so the animation is cancelled before its first frame paints, which is right: the effect the user chose replaces it. The stand-in is also held `visible = false` until its capture has been corrected. Its pixels are known wrong for that frame, and on an open they are not merely wrong but the wallpaper — showing them is the exact flash being fixed. The real window is the better thing to look at until the correction lands. Reconstruct is Vanish run backwards, and shares its shape: the capture is sliced into a grid of sub-textures over one source, and each tile starts scattered at its own angle and distance, spinning and transparent, then decelerates onto the square it belongs in. The per-column delay knits the window together from one edge rather than snapping it into place. The scatter is derived from the cell index rather than Math.random, so a given window always assembles the same way and a bug is reportable. Open only — a window that assembles itself on minimise and then flies to the dock is two ideas fighting. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
The stand-in was removed two `requestAnimationFrame`s after the real window was restored, with a 250 ms timer behind it. That was only ever the right order of magnitude: land a frame early and the gap is back — neither the copy nor the window drawn — and land late and the copy overstays. `stage.afterNextSnapshot()` already exists for the capture correction and answers exactly this question, so the hand-off uses it too. The copy now goes the moment the snapshot actually contains the restored window. No frame counting remains anywhere in the effect path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
A window capture comes from `getBoundingClientRect()` — the border box — and `box-shadow` paints outside it. So a frozen stand-in never carried a shadow, and the shadow snapped back into existence the instant an effect handed the window back. A real difference between the two images, which is why no amount of timing the swap could hide it. `stage.css` now zeroes the desktop's shadow tokens on the staged shell. A desktop with no drop shadows is consistent; one with almost-right shadows is not. Two details worth keeping. The tokens are set to a TRANSPARENT zero-size shadow rather than `none`, because several get composed into longer shadow lists — the linked-window halo is `0 0 18px 4px <glow>, var( --desktop-mode-window-shadow )` — and `none` is only valid alone, so substituting it there would invalidate the declaration and take the halo with it. And only the tokens are touched: there are ninety-odd hard-coded `box-shadow` declarations in the stylesheets, a dozen of them `inset`, and a blanket rule would have taken out focus indicators and the highlights that draw borders and pressed states. None of those extend past the box an effect captures. The PixiJS shadow this replaces stays as the fallback, because a desktop theme styling `.desktop-mode-window` directly beats a token override — the precedence a theme author would expect — and a shadow that comes back that way should be matched rather than left to pop. It reads the window's own computed `box-shadow`, draws every layer back to front (CSS paints the first-declared on top), and converts the CSS blur radius through both halves of the mismatch: CSS defines it as twice the standard deviation, and PixiJS's `strength` is a tap SPACING whose 5-tap kernel carries a deviation of ~1.292 taps of its own. Halving alone left every shadow about 30% too soft. `ctx.shadow` is handed to effects. The engine keeps it aligned to the stand-in sprite until an effect hides or fades that sprite, at which point the effect has replaced it and owns the shadow — the cloth pins it under the window rather than the swinging sheet, and the reconstruct ramps it up with the assembly, because a window that is not there yet casts no shadow. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
The stage asks the browser to re-record the shell every frame and
uploads the result every frame. The first half is not negotiable —
scrolling inside a window's iframe can be composited off the main
thread, so waiting for the browser's own invalidation would let the
desktop silently stop updating, a far worse failure than a wasted
upload.
The second half is. The `paint` event reports which of the canvas's
children changed, and PixiJS's `HTMLSource` only consults that list to
skip paints belonging to OTHER children; when it says nothing changed at
all, it uploads regardless. Those pixels are identical to the ones
already on the GPU. At a 2x device pixel ratio on a 1440p display,
re-sending them is roughly 850 MB/s of nothing.
`element-source.ts` subclasses `HTMLSource` and drops that upload.
Subclassed rather than patched because `HTMLSource` binds its paint
handler in its own constructor, so the bound copy resolves through the
prototype at construction and an instance patch would never be called.
Not `autoUpdate: false` with a listener of our own, which would have
read more cleanly: that flag also decides `_isReady`, which would then
be true before any paint had been recorded, and PixiJS would try to
upload an element with no cached paint record and throw on the stage's
very first render.
Two guards, because a wrong guess here is a frozen desktop rather than a
slow one. An ABSENT `changedElements` is not an empty one — absent means
the browser did not say, and inferring "nothing changed" from silence is
how a mirror stops mirroring. And an upload is forced after a bounded
run of idle paints regardless, so if the assumption is ever wrong the
cost is staleness measured in frames rather than forever.
`wp.desktop.stage.stats()` reports `{ paints, uploads, skipped }` so this
can be checked rather than trusted: `skipped` climbs on an idle desktop
and stalls the moment anything moves. If it stays at zero, this browser
always reports a change and the saving is a no-op here — worth knowing
either way, which is why the counters ship.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
The cloth drag froze the window's pixels at drag-start and animated the copy — a video stopped playing the moment you grabbed its window. The HTML-in-Canvas spec's own pattern (see Chrome's deformable-page demo) is to texture the element itself every paint, which needs the element to be a direct child of the canvas. So the engine now PROMOTES the window there for the duration of a drag or open effect, with Node.moveBefore() — atomic and state-preserving, so iframes do not reload — and gives it its own per-paint texture. A layoutsubtree child lays out at the canvas origin and only transforms still drive hit-testing, so the promotion mirrors the window manager's left/top into a translate every frame, self-calibrated against where layout actually put the element, and hands everything back (with the transform flushed before transitions return) when the effect resolves. Frozen captures remain for the after-the-fact transitions — minimise, maximise, close — where the stale snapshot is the only record of the "before" state, and as the fallback when promotion declines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
Each fleck now tints from its natural colour through scorched brown into smoky ash-grey while it drifts — fully charred by 60% of its life so the scorch lands on solid pixels, jittered ±15% per fleck so the burn is uneven, multiplicative so dark UI smoulders where a white page browns. A new Burn slider (default 0.85) dials it down; at 0 the dissolve is exactly the old untinted behaviour. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
The experimental notice and the master checkbox rendered flush against each other. wpd-section already ships the fix — the opt-in stack attribute that lays slotted children out with a consistent gap — so both variants of the pane (supported and unsupported-browser) now use it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015sKW1mu1SGg5k9oSHgxv5s
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
This PR is EXPERIMENTAL and exists for fun. It is a playground for the WICG HTML-in-Canvas proposal — an API that is still behind a flag (
chrome://flags/#canvas-draw-element, Chromium 152.0.7973.0 (Official Build) canary (arm64)) — and it is not meant to be merged. Opened as a draft to share, discuss, and marvel at windows that wobble like cloth.html-in-canvas.mov
What this is
The whole WordPress admin desktop rendered through a
<canvas layoutsubtree>via PixiJS, so the real, fully-interactive UI can be post-processed by shaders and deformed per-window:Node.moveBefore()(atomic, state-preserving, no iframe reload) and textured from the live element every paint, like Chrome's deformable-page demo. A video keeps playing while its window swings as cloth. Frozen snapshots remain only where they are semantically right (close/minimize — the stale frame is the only record of the "before") and as the fallback.Why it should not be merged
texElementImage2Darity; there's a shim).layoutsubtreelayout semantics (children lay out at the canvas origin; transforms drive hit-testing) required a per-frame geometry mirror — delightful, but not something to support in production yet.State of play
🤖 Generated with Claude Code