Skip to content

feat(langchain): add LC-021, LC-022 TypeScript description quality rules - #93

Open
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ts-description-quality
Open

feat(langchain): add LC-021, LC-022 TypeScript description quality rules#93
bradAGI wants to merge 1 commit into
trustabl:mainfrom
bradAGI:feat/langchain-ts-description-quality

Conversation

@bradAGI

@bradAGI bradAGI commented Aug 24, 2026

Copy link
Copy Markdown

The TypeScript counterpart to LC-018/019 (#66). LC-010 only checks that a description exists, so a tool described "TODO: describe this tool." or "Gets data." passes today while giving the model no selection signal.

Three points, in the order they bite:

  • No docstring fallback. LangChain.js has nothing behind the description field, so it's the entire account of the tool the model sees — unlike the Python side, where a thin description at least sits next to readable source.
  • The Zod schema doesn't compensate. It constrains the shape of the arguments once the model has chosen this tool, never whether choosing it was right.
  • Mis-selection scales badly and costs the run. A LangChain agent is routinely handed a dozen or more tools at once, and each wrong pick spends an iteration against the maxIterations bound LC-111 checks for — so a run can exhaust its step budget and return nothing useful.

Verification — engine built at main:

$ trustabl rules validate .
OK: 85 rule pack(s), 208 rule(s) valid under rule schema version 14

Fire (one tool(...) described "TODO: describe this tool.", one "Gets data."): LC-021, LC-022, LC-201
Silent (full description naming when to prefer the neighboring tool): LC-201

(LC-201 is the pre-existing missing-AGENTS.md repo rule.)

LC-022 pairs description_length_lt: 40 with has_docstring: true so it doesn't double-report against LC-010 on an empty description, same as CSDK-018.

No new predicates, so no schema_version bump.

Merge note: this and #66 both append to langchain/tool_definition.yaml, so whichever lands second needs a trivial rebase — the rule blocks are independent and the resolution is "keep both." Happy to rebase on request.

The TypeScript counterpart to LC-018/019. LC-010 only checks that a
description exists, so a tool described "TODO: describe this tool." or
"Gets data." passes today while giving the model no selection signal.

LangChain.js has no docstring to fall back on, so the description field is
the entire account of the tool the model sees, and the Zod schema does not
compensate — it constrains the arguments once the model has chosen this
tool, never whether choosing it was right. Mis-selection scales badly here
because an agent is routinely handed a dozen or more tools at once, and
each wrong pick spends an iteration against the maxIterations bound
LC-111 checks for.
@jhumel-code

Copy link
Copy Markdown
Collaborator

Thanks @bradAGI, this is a clean one. Checking both rules against origin/main: has_description_text, has_docstring and description_length_lt all exist in internal/rules/schema.go, langchain_tool is a valid applies_to at scope: tool, and LC-021 / LC-022 are free on main and unclaimed by any other open PR. Severity and confidence line up with LC-001 and LC-010 next door, and pairing description_length_lt: 40 with has_docstring: true so that an absent description stays LC-010's finding rather than double reporting is exactly right, the same shape as CSDK-018. No new predicates, so you are correct that no schema_version bump is needed here.

The engine half in trustabl/trustabl#171 mirrors this file byte for byte (both sides land on the same blob hash), so rules-sync stays green as long as the two land together.

One thought rather than a blocker: todo is matched as a case-insensitive substring, so a genuine tool described "Add an item to the user's todo list." will fire LC-021. CSDK-017 already ships that needle, so I am not asking you to diverge from it here.

Before this ships it needs a rationale doc in trustabl-rulebook for the LangChain pack. That part is on us, and we are happy to pair on it. Looks good to me otherwise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants