Skip to content

feat(autogen): add AG2-019, ambiguous tool name - #106

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/autogen-ambiguous-tool-name
Open

feat(autogen): add AG2-019, ambiguous tool name#106
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/autogen-ambiguous-tool-name

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown

Ports the ambiguous-name check to AutoGen. Claude SDK (CSDK-007), OpenAI (OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the newer packs had none.

The tool name sits directly beside the description in what the model sees, so it is half the selection signal — and a generic verb like process or handle spends that half on nothing. Each wrong pick costs a full round of the conversation — propose, execute, reply — so an ambiguous name spends the AG2-004 max_round budget rather than the chat reaching an answer.

Same name_in list as CSDK-007, so behavior stays consistent across packs.

Verification — engine built at main:

$ trustabl rules validate .
OK: 85 rule pack(s), 211 rule(s) valid under rule schema version 14

Fire (tool named process): includes AG2-019
Silent (same tool renamed summarize_invoice): AG2-019 absent

A note on scope. I drafted a fifth rule in this batch for the Vercel AI SDK and dropped it after testing: ToolDef.Name is deliberately empty for vercel_ai_tool, because the SDK derives a tool's name from the key in the tools: { ... } record rather than from the tool({...}) call. The engine asserts this directly —

// internal/analysis/ts_vercel_tools_test.go
if tl.Name != "" {
    t.Errorf("Name: got %q, want empty (Vercel derives name from the tools-record key)", tl.Name)
}

so every name-based predicate (name_in, name_has_prefix) is structurally inert for that pack — it also blocks a Vercel idempotency rule, which is why the pack has none. Back-filling Name from the record key at discovery time would unblock both. Happy to open that engine PR if it'd be useful.

Paired engine PR follows, mirroring the rule into testdata/rules-fixture/ with fire/silent cases.

Ports the ambiguous-name check to autogen. Claude SDK (CSDK-007), OpenAI
(OAI-007), ADK (ADK-007), and MCP (MCP-003) all ship it; the five newer
packs had none.

The tool name sits directly beside the description in what the model sees,
so it is half the selection signal, and a generic verb like process or
handle spends that half on nothing.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a clean port. I checked the name_in list against CSDK-007 on main and it matches item for item, and severity: low with confidence: 0.9 lines up with AG2-007 in the same file, so nothing here is an outlier. autogen_tool is valid at scope: tool, name_in already exists on main, and since no new predicate is involved there is correctly no schema_version bump.

What I like most is that the rule is genuinely accurate for AutoGen rather than just plausible. DiscoverAutoGenTools honors the name= override on both register_for_llm and register_function, so ToolDef.Name holds the name the model actually sees, and a def process registered as name="summarize_invoice" will not fire.

One nit: the closing sentence of explanation is a single long line while the rest of the pack wraps near 75 columns. Please rewrap it.

AG2-019 will also need a rationale block in the trustabl-rulebook doc for this pack before it ships. That is on us rather than on you, and we are happy to pair on it.

Your Vercel note checks out too. ts_vercel_tools.go never sets Name and the test asserts it stays empty, so an engine PR backfilling it from the tools-record key would be welcome as its own change.

This pairs with trustabl/trustabl#178, whose fixture copy is byte identical, so the two should land together.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants