Add agent whiteboard: per-agent tab, MCP drawing tools, live SSE sync#735
Open
lukebrevoort wants to merge 6 commits into
Open
Add agent whiteboard: per-agent tab, MCP drawing tools, live SSE sync#735lukebrevoort wants to merge 6 commits into
lukebrevoort wants to merge 6 commits into
Conversation
Excalidraw-based whiteboard as a third center-pane tab. Scenes persist to a new whiteboards table with optimistic versioning; the client exports a debounced PNG snapshot into the agent's media dir (no media row, so no unseen-badge churn). Agents see the board via a new whiteboard_get MCP tool: simplified elements plus the snapshot path for vision. Fonts are self-hosted so nothing fetches from esm.sh at runtime. Review round-1 fixes: empty-board hint communicating that the agent can see the board, and snapshot deletion when the board is cleared so whiteboard_get never points at an erased drawing. Also: resolveMediaDir now expands a leading "~" (pre-existing bug wrote media into a literal ./~ dir); tsx added so the dev seed runs; pnpm 11 config migrated into pnpm-workspace.yaml; PLAYWRIGHT_CHROMIUM_EXECUTABLE_PATH escape hatch for system-Chromium machines. Agent drawing (whiteboard_update), the Ask-agent button, and repo-scoped boards are Phases 3-4 (see PLAN.md). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agents answer visually now: a whiteboard_update MCP tool takes
simplified ops (add/update/delete over rect|ellipse|diamond|arrow|
line|text|frame with label/from/to/color) and a server-side builder
expands them into full Excalidraw elements — ids, seeds, bound label
text, arrow start/endBindings with back-references — merged via the
existing saveWhiteboard optimistic-version path (retried on races) and
published as whiteboard.changed {source:"agent"} so open canvases apply
them live. Agent-supplied readable ids let arrows bind to elements
created in the same call; the tool result echoes created ids plus the
simplified scene so agents iterate without a second whiteboard_get.
Spike A (the flagged D5 risk) passed: the editor accepts hand-built
elements, heals null fractional indices, and re-routes bindings on
drag. Excalidraw itself only re-routes arrows during in-editor drags,
so the builder re-routes bound arrows itself when an update op moves a
shape, and re-centers bound labels.
Product layer (P1-P4 in PLAN.md): agent strokes default to violet with
customData.author="agent" (soft attribution); drawing etiquette lives
in the tool description (point near, never cover; label on the shape;
make room before squeezing in); whiteboard_get now reports
snapshotStale when the PNG predates the scene, and a connected client
re-exports the snapshot after applying an agent edit. A violet dot on
the Whiteboard tab (jotai atomFamily set from SSE, cleared on view)
says "agent drew" while you're elsewhere. Tool is AGENT_TOOLS-only;
personas stay read-only.
Validated: 15 builder unit tests, e2e spec driving the real MCP route
(scoped route rejects the server bearer token — call it tokenless),
full suite green (typecheck, 2147+188 unit, finalize:web, 175 e2e), and
a live tmux Claude agent demo: sketch -> agent draws a message queue ->
"tidy it up" -> agent re-labels/spaces its own elements by id.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The unmount cleanup flushed the debounced scene save but silently dropped a pending snapshot export, so navigating away within the 4s debounce left agents a stale whiteboard.png. exportToBlob renders offscreen from scene data, so flushing during teardown is safe. Review round-1 fix (product-review selfcontained#20). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Scoped MCP routes 403 on the server bearer token (agent-scoped HMAC or tokenless only) — cost an e2e cycle to rediscover in Phase 3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uting - shared/whiteboard-store.ts owns persistence (was in routes/, imported by server/ — layering inversion) - server/mcp-whiteboard-handlers.ts extracts whiteboard MCP handlers (mirrors mcp-review-handlers.ts) - WhiteboardGetResult/WhiteboardUpdateResult defined once in shared/whiteboard.ts (was hand-written 3x) - routeLinear() collapses three near-identical arrow-routing blocks in whiteboard-builder.ts - EXCALIDRAW_CAPTURE_VERSION + pin-guard test: bumping @excalidraw/excalidraw fails loudly until mirrored math is re-verified - whiteboard_update description documents non-transactional op batches - WhiteboardPane component owns keep-mounted/drew-dot logic (out of agents-view.tsx) Includes in-flight builder/tool work that was uncommitted at review time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The violet signature stroke clashed on many board colors and lost expressiveness. Default to Excalidraw's default ink (#1e1e1e), which inverts to near-white in dark mode and reads cleanly on any theme. Agents can still pick violet (and every named color) explicitly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Adds a whiteboard to Dispatch: each agent gets a Whiteboard tab in the center pane where both the user and the agent can draw on a shared tldraw canvas, synced live over SSE.
whiteboard_updateMCP tool lets agents draw shapes, text, arrows, and diagrams on the board.Dispatch's Self Described Architecture
Agent
Intent: Give agents a visual communication channel (diagrams, sketches, annotations) alongside text, scoped per agent session.
Key pieces:
apps/server/src/shared/whiteboard-builder.ts— converts high-level shape ops from the MCP tool into tldraw store records (the bulk of the diff, plus ~1k lines of unit tests).apps/server/src/shared/mcp/whiteboard-tools.ts—whiteboard_updatetool definition; handlers inapps/server/src/server/mcp-whiteboard-handlers.ts.apps/server/src/routes/whiteboard.ts+whiteboard-store.ts— snapshot persistence and REST endpoints; UI sync via SSE events (ui-events.ts,use-sse.ts).apps/web/src/components/app/whiteboard-tab.tsx/whiteboard-pane.tsx— tldraw canvas, snapshot export flushed on unmount.Constraints / edge cases:
/api/mcp/:agentId, ...) 403 on the server bearer token — agent-scoped HMAC tokens only (documented in CLAUDE.md).use-agents-view-routing.ts/agent-routes.tsafter architecture review.Out of scope: multi-user cursors, job-run whiteboards, freehand-stroke input from agents.
Tests: unit (
whiteboard-builder.test.ts,whiteboard.test.ts) and E2E (e2e/whiteboard.spec.ts).🤖 Generated with Claude Code