Repository navigation
Let an agent see a frame of the animation it wrote - #1501
Merged
Merged
Conversation
After publishing an animation an agent had no way to look at it short of driving the screen: session_get said only whether the script ran without an error. session_render shows one frame of an animation, or a clip at one marker, on the whiteboard open in Desktop and returns it as a PNG image, with the error the stage reported, if any; the PNG is also written beside the animation drafts and named in the result, for agents that read files rather than images. The animation instructions tell agents to render each frame they changed after publishing, instead of taking screenshots. Desktop captures the frame from its own canvas rather than a headless renderer drawing it: the code peeks and embedded blocks a frame shows are Desktop's own native editors and canvas components laid over the stage, so only Desktop can draw a frame as the reader sees it, and the headless server reaches it through the same relay session_open uses. The canvas asks the block to show the stop at rest (switching scenario when the frame needs one, pausing playback, waiting for the stage to settle, paint in and hold still in view) and reports where its figure is; the canvas part captures that rectangle with Electron's capturePage, now able to answer a PNG scaled to at most 1600 pixels on its longest edge. The board must be open in Desktop and its window on screen; the result says so when it is not, instead of waiting out the relay. Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683 Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b Agent-Session: b315a4f8-1333-461c-9be6-366ceb231b7f Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad Agent-Session: b5f9f841-35c5-4ec1-95a8-6485f64d254b Agent-Session: 32e129b9-586e-4fd2-8961-e7224c5978ad
…ted-solomon-90e5dd Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683 Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b Agent-Session: b315a4f8-1333-461c-9be6-366ceb231b7f Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad Agent-Session: b5f9f841-35c5-4ec1-95a8-6485f64d254b Agent-Session: 32e129b9-586e-4fd2-8961-e7224c5978ad # Conflicts: # packages/review/app/src/animation/animation-block.tsx
Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683 Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b Agent-Session: b315a4f8-1333-461c-9be6-366ceb231b7f Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad Agent-Session: b5f9f841-35c5-4ec1-95a8-6485f64d254b Agent-Session: 32e129b9-586e-4fd2-8961-e7224c5978ad
…ted-solomon-90e5dd
…d into mb/elated-solomon-90e5dd Agent-Session: 62100749-c77c-4d19-ad48-c6d4ba1e867b Agent-Session: cbacb871-2558-4fe4-9272-77524bb21683 Agent-Session: ffed49d6-ca86-4940-b93c-e8eb7e0de440 Agent-Session: 7bc797d9-b46c-48dc-b281-4fce85dc15ad # Conflicts: # packages/review/src/server/review-server-core.ts
thesiti92
approved these changes
Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1498 (the clip block and the reference animations live on that stack, so this is based on
mb/scene-card-forms).After publishing an animation an agent had no way to see what it drew:
session_getsays only whether the script ran, so agents fell back to driving the screen.session_render({ sessionId, targetId, frame })shows one frame of an animation (or a clip at one marker; omitted, the first) on the whiteboard open in Desktop and returns it as a PNG image, with the error the stage reported, if any. The PNG is also written to<home>/animation-renders/<session>/<block>/<key>.pngand named in the result, for agents that read files rather than images (whiteboard api session_renderprints the JSON without the base64). Step 5 of "Work in files" inanimation.mdtells agents to render each frame they changed after publishing, instead of taking screenshots.What the agent gets back, from a scene animation and a clip published through
whiteboard apiinto a dev Desktop:{"sessionId":"2d67401e-…","targetId":"block-1","type":"animation","frame":{"key":"query","title":"Postgres answers","position":2,"count":4},"clipped":false,"error":null,"file":"…/animation-renders/2d67401e-…/block-1/query.png"}Over MCP the same call answers an
imagecontent block (the PNG) ahead of that text.How a frame is rendered: Desktop captures its own canvas
Two ways were weighed. (A) The attached Desktop shows the frame on the open board and captures the block's rectangle with Electron's
capturePage. (B) The review server renders the stage headless. This PR does A:whiteboard server startwith an attached Desktop renders through the same relaysession_openuses; without one the route answers 409.session_open); the window must be on screen (a covered or minimized window paints nothing and throttles its timers, so the canvas fails fast with a message saying so rather than waiting out the relay); the reader sees the frame change; a figure taller than the window is captured as far as it fits, withclipped: true.The pieces
renderAnimation {reviewId, targetId, frame};ReviewCanvasHandle.renderAnimationandReviewRenderedFrame.animation/stage-renderer.tskeeps the stage blocks on a board by id.StageBlock.show(key)waits for the stage to boot and paint in, switches to a scenario that has the frame when the one showing does not, picks the stop with the newPickedBy"render"(playback stops; a clip pauses), waits forsettledor the clip'smarker, scrolls the figure into view, waits for it to hold still, and reports its rectangle in CSS px. The stage port now notes the frame or marker last asked for, and the block notes what the stage reports showing, so a render waits until they agree.reviewCanvasPart.tsanswers the verb for the board it shows (main window only), scales the rectangle by the window's zoom and callshostService.getScreenshot(rect, { format: "png", maxEdge: 1600 }). The native host'sgetScreenshotgained those two options; JPEG stays the default, and an empty capture answersundefined.review-api/animation-render.tsholds the query and result schemas andrenderTarget, which checks the frame against the block's frames or markers (naming them when it is wrong) and picks the first scenario's first frame when omitted.GET /:id/renderinhttp.ts;relayReviewCallbacks.renderinreview-server-core.ts. The/control/resultbound is 16 MiB so the PNG can ride back.session_checkoutbeside it);public-tools.tswrites the file;mcp.tsreturns the image block;agent-cli.tskeeps the base64 out of stdout.Verification
animation-render.test.ts(target resolution, the route with a stub Desktop, the tool listing, the file and the image split),render-relay.test.ts(the relay callback: result parsing, Desktop's refusal and a missing Desktop as 409s), plus the existing authoring, public-tools, agent-client and authoring-capabilities tests.stage-renderer.browser.test.tsxmounts the retry-backoff reference, rendersa3fail(only in its second scenario), checks the reported rectangle against the figure and that Playwright's capture of it is a non-empty PNG; lands a clip on a marker at rest; reports unknown stops and blocks. The clip, paint, blocks and references suites pass (100 tests).pnpm desktop:buildanddesktop:rununder an isolatedDEV_REVIEW_HOME, animations enabled, a scene animation and a clip published throughwhiteboard api.session_renderreturned the captures above, exact to the figure, for frames in either direction and for markers, and the errors for a frame the block lacks and for a hidden window. Not checked: dark theme, a zoomed window, secondary windows.pnpm lint,pnpm format, both typechecks and the fork'stypecheck-clientare clean.The screenshots live on
pr-assets/render-animation-frames; delete that branch with this PR.