Skip to content

feat(vercel_ai): add VAI-017 tool stdout diagnostics rule - #114

Open
joeymussalli wants to merge 1 commit into
trustabl:mainfrom
joeymussalli:feat/vercel-stdout-vai-017
Open

feat(vercel_ai): add VAI-017 tool stdout diagnostics rule#114
joeymussalli wants to merge 1 commit into
trustabl:mainfrom
joeymussalli:feat/vercel-stdout-vai-017

Conversation

@joeymussalli

@joeymussalli joeymussalli commented Aug 24, 2026

Copy link
Copy Markdown

Summary

  • Add vercel_ai/observability.yaml with VAI-017: Vercel AI execute() writes diagnostics to stdout (has_print_call: true).
  • Ports the OAI-010 observability pattern to the Vercel pack. stderr remediations (console.error, etc.) do not fire once the engine prints_stdout fact is present.

Depends on

Test plan

  • Pack validates against schema
  • Engine fire/silent cases for VAI-017 pass once #187 (or #180 + fixture) is available

Port the OAI-010 observability pattern to Vercel AI execute() handlers
via has_print_call (prints_stdout fact for console.log and friends).

Co-authored-by: Cursor <cursoragent@cursor.com>
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @joeymussalli. Schema-wise this is correct: vercel_ai_tool is valid at tool scope, has_print_call already exists in the engine, low with confidence 0.65 matches OAI-010 exactly, and VAI-017 is not claimed by any other open PR, so there is no renumbering risk here. The YAML matches the fixture in trustabl/trustabl#187 byte for byte, and thanks for opening the rulebook PR alongside rather than leaving the grounding to us.

One change before this ships. The explanation is ported from OAI-010, and both consequences it names are a stretch for this SDK. A Vercel AI tool usually runs in a Next.js route on a platform that captures stdout as its log stream, so the "silently disappears" claim mostly does not hold there, and ai package tools are not served over MCP stdio (the SDK consumes MCP as a client). Please lead instead with what does hold: execute() receives model-controlled arguments, so a stdout write puts untrusted input, and whatever the tool read from it, into shared logs, and none of it is addressable by the model. ADK-009 is a good in-repo model for that framing.

One note for me rather than for you: on a binary older than trustabl/trustabl#187 this rule loads fine and silently never fires, so I need to decide whether that warrants a schema_version bump.

Happy to take this once the explanation is reframed and the fixture is updated to match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants