Skip to content

chore: SCO-212 move AI call sites from Sonnet 4.6 to Sonnet 5.5 - #394

Merged
scottcarlton merged 1 commit into
devfrom
chore/sco-212-sonnet-5-5-migration
Oct 27, 2026
Merged

scottcarlton merged 1 commit into
devfrom
chore/sco-212-sonnet-5-5-migration

Conversation

@scottcarlton

Copy link
Copy Markdown
Owner

Summary

  • Model: all six Sonnet call sites move to claude-sonnet-5-5. That's the Stitch turn (STITCH_MODEL), both org agent executor calls, the SMS/WhatsApp agent, the inbound email parser and the line sheet parser.
  • Forced tool use removed: Sonnet 5.5 returns a 400 on a forced tool_choice. The email intake and line sheet parsers now use auto and ask for the tool call in the prompt. Both already fail cleanly when no tool call comes back (an error for email intake, a 422 for line sheets).
  • Effort set on every route: medium for the tool loops (Stitch turn and org agents), low for SMS chat and both extractors. The levels are recalibrated on 5.5, so nothing is left on the default.
  • Room for thinking: thinking is on by default and counts toward max_tokens. The Stitch turn and org agents go from 4096 to 16000, email intake from 4096 to 16000, and SMS from 1024 to 4096. The line sheet parser stays at 16384.
  • Refusals reported: logUsage flags stop_reason: "refusal" as a new model_refusal degraded kind with its stop_details.category. Every Claude call reports through logUsage, so this covers all endpoints.
  • Eval pricing: added Sonnet 5.5 ($2/$10) to the Stitch eval price table and updated the model-tied fixtures.
  • Security sweep doc: the model and max_tokens columns are current. The briefing row also gets corrected; it was already running on Haiku 4.5.

Why

Move from Sonnet 4.6 to Sonnet 5.5. It's cheaper per token ($2/$10 against $3/$15) and stronger at multistep tool use. The changes above are the Sonnet 5.5 breaking changes that apply to this codebase. The rest of the migration checklist doesn't apply here: no budget_tokens, no disabled thinking, no sampling parameters, no computer use, no advisor tool, and no history editing (the loops only append and pass assistant content back unchanged).

Verification

  • bun run lint passes
  • bun run check passes (0 errors)
  • bun run test:run passes (2032), including a new test for refusal reporting
  • Live calls to claude-sonnet-5-5 using the new request shapes:
    • Extraction with auto tool choice at effort low returned the tool call with the correct order data.
    • A streamed tool loop at effort medium, passing content back unchanged, completed the round trip and ended normally.
  • Each production route not exercised end to end in the app (Stitch chat, SMS, email intake, line sheet upload, org agents)
  • Stitch eval not re-run on 5.5, so the effort levels are the guide's starting points, not measured

Test plan

  • Ask Stitch a question that uses tools; the answer streams and the tools run
  • Upload a line sheet PDF; products parse
  • Run an org agent; output comes through
  • Watch Sentry for ai-degraded events of kind model_refusal or max_tokens after deploy

Follow-ups

  • Server-side refusal fallback (fallbacks: "default") needs an Anthropic SDK upgrade from 0.82
  • Stitch eval effort sweep on 5.5

Closes SCO-212

🤖 Generated with Claude Code

Sonnet 5.5 rejects forced tool_choice, thinks by default (counting toward
max_tokens), recalibrates effort, and can decline with stop_reason
"refusal". Extraction routes now ask for the tool call in the prompt, each
route sets its effort and a cap with room for thinking, and logUsage flags
refusals with their category.

SCO-212

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
threadline Ready Ready Preview Oct 27, 2026 5:19pm UTC

@scottcarlton
scottcarlton merged commit 50666d4 into dev Oct 27, 2026
8 checks passed
@scottcarlton
scottcarlton deleted the chore/sco-212-sonnet-5-5-migration branch October 27, 2026 17:21

This branch was successfully deployed

1 active deployment
Preview — 0329ec28 Deployed Oct 27, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant