Skip to content

feat(scrawl): capture the screen with voice, clicks and ink, inside T3 Code - #25

Merged
collissimo merged 4 commits into
collin/mainfrom
t3/native-scrawl-voice-input
Oct 31, 2026
Merged

collissimo merged 4 commits into
collin/mainfrom
t3/native-scrawl-voice-input

Conversation

@collissimo

Copy link
Copy Markdown
Owner

Showing an agent a problem needed the separate Scrawl app, and its text cut sentences apart at every click: on 32 real captures with measured word times, 59 of 127 lines broke mid-sentence, while the nearest pointing word lay up to 3 s either side of the click.

How it works now

  • Alt+Shift+S anywhere, or the new composer button, records the monitor under the mouse in the desktop app, also over other programs. Voice, clicks and Shift-drawn red ink are captured; text and contact sheet land in the composer.
  • Whole sentences own a keyframe. Click bursts fold into one, and clicks the screen did not answer read as pointed. The same 32 captures go to 1 cut of 127 lines, losing no word, click or mark.
  • Word times come from the existing dictation engine, on request and mapped back across the pauses it removes. Ordinary dictation does not ask and keeps its small reply.
  • Bundles keep Scrawl's format, so its skill and scrawl.exe inspect/zoom read them.

Checked

  • vp run check:changed origin/collin/main passes.
  • Dev app on Windows: captures through the real composer with the window in the background, with clicks, ink, video, frames, contact sheet.
  • Real speech through the app's dictation worker: 149 timed words. On a 76 s capture the times are within 46 ms of Scrawl's at p95.

Not checked or not done

  • A capture with live speech end to end; the dev runs were silent.
  • The global shortcut itself: the running Scrawl app holds it. T3 Code logs that and retries every 15 s.
  • The recording pill shows in the video: setContentProtection did not keep it out on Windows 10 22H2, so it sits in the corner.
  • Not ported from Scrawl yet: mark shape recognition, per-keyframe auto-crop (one box per capture here), a built-in zoom/export, scroll events, a rebindable shortcut, cursor and window logs.
  • No before/after image: the test recordings showed private browser content and were deleted.

Done with Claude Opus 5.5 in Claude Code.

🤖 Generated with Claude Code

collissimo and others added 4 commits October 31, 2026 17:15
A screen capture lines clicks up with what was said, so the engine can time words and maps them back across the pauses it cut. Dictation does not ask and keeps its small reply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…3 Code

Alt+Shift+S or the composer button records the monitor under the mouse, also over other programs. Sentences stay whole and own their keyframes, clicks the screen did not answer read as pointing, and the bundle keeps Scrawl's shape.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erer frees the overlays

The recording pill moves to the corner because it shows in the recording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XXL labels Oct 31, 2026
@collissimo
collissimo merged commit c2f508f into collin/main Oct 31, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant