Skip to content

feat(pydantic_ai): add PYD-013, ambiguous tool name - #107

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/pydantic-ai-ambiguous-tool-name
Open

feat(pydantic_ai): add PYD-013, ambiguous tool name#107
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/pydantic-ai-ambiguous-tool-name

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown

Ports the ambiguous-name check to Pydantic AI. Claude SDK (CSDK-007), OpenAI (OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the newer packs had none.

The tool name sits directly beside the description in what the model sees, so it is half the selection signal — and a generic verb like process or handle spends that half on nothing. Pydantic AI derives the tool name from the function name unless one is passed explicitly, so a generically named helper function silently becomes a generically named tool — the defect arrives without anyone deciding to name a tool process.

Same name_in list as CSDK-007, so behavior stays consistent across packs.

Verification — engine built at main:

$ trustabl rules validate .
OK: 85 rule pack(s), 211 rule(s) valid under rule schema version 14

Fire (tool named process): includes PYD-013
Silent (same tool renamed summarize_invoice): PYD-013 absent

A note on scope. I drafted a fifth rule in this batch for the Vercel AI SDK and dropped it after testing: ToolDef.Name is deliberately empty for vercel_ai_tool, because the SDK derives a tool's name from the key in the tools: { ... } record rather than from the tool({...}) call. The engine asserts this directly —

// internal/analysis/ts_vercel_tools_test.go
if tl.Name != "" {
    t.Errorf("Name: got %q, want empty (Vercel derives name from the tools-record key)", tl.Name)
}

so every name-based predicate (name_in, name_has_prefix) is structurally inert for that pack — it also blocks a Vercel idempotency rule, which is why the pack has none. Back-filling Name from the record key at discovery time would unblock both. Happy to open that engine PR if it'd be useful.

Paired engine PR follows, mirroring the rule into testdata/rules-fixture/ with fire/silent cases.

Ports the ambiguous-name check to pydantic_ai. Claude SDK (CSDK-007), OpenAI
(OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the five newer
packs had none.

The tool name sits directly beside the description in what the model sees,
so it is half the selection signal, and a generic verb like process or
handle spends that half on nothing.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a clean port. Checking PYD-013 against main: the ID is free, every required field is present, pydantic_ai_tool is a valid applies_to at tool scope, name_in already exists in the engine so no schema_version bump is needed here, and low with 0.9 matches both CSDK-007 and PYD-001 in this pack. PredNameIn compares the whole name case insensitively, so the list behaves the way your description says it does.

One question before this goes in. Discovery honors an explicit name= on the Tool(...) factory, but the decorator path in internal/analysis/discovery.go builds ToolDef.Name straight from the function name and never reads the decorator kwargs. So if @agent.tool takes a name= override, @agent.tool(name="summarize_invoice") over def process(...) would fire here even though the name the model actually sees is fine. Decorator kwargs do land in Config, so a not: tool_decorator_kwarg_present: [name] guard would close it. CSDK-007 has the same gap, so this is not something you introduced.

Small nit, the tail of the explanation runs as one long unwrapped line; please rewrap it to match the rest of the pack. Also worth knowing that #64 appends PYD-010 and PYD-011 to the end of this same file, so whichever lands first leaves the other needing a rebase. This will want a rationale doc in trustabl-rulebook before it ships, which we can handle or pair on. I have reviewed the engine half, trustabl/trustabl#179, alongside this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants