Skip to content

refactor(router): clarify routing prompt within token budget - #232

Open
Light-code11 wants to merge 2 commits into
mainfrom
codex/router-prompt-budget
Open

refactor(router): clarify routing prompt within token budget#232
Light-code11 wants to merge 2 commits into
mainfrom
codex/router-prompt-budget

Conversation

@Light-code11

@Light-code11 Light-code11 commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • shorten the router catalog scaffolding in a dedicated prerequisite so feature PRs do not alter every skill indirectly
  • preserve strong Do NOT guidance for exclusions
  • distinguish action examples from alternative-tool counterexamples with Actions: and Other tools: headings
  • replace the previous vacuous assertion with direct checks for every rendered routing section and target namespace

Why

The Victor routing surface was already approximately 5,998/6,000 tokens before adding another skill. This prerequisite creates room for the Meeting Room skill without bundling a global router change into #225.

The compact prompt still tells the model what each value means:

  • Use: contains the skill's positive routing condition
  • Do NOT: contains the skill's exclusion condition
  • Actions: maps examples to actions within the current skill
  • Other tools: maps counterexamples to a different tool

Validation

  • router unit, catalog, and hermetic routing gates: 15 passed
  • CI-equivalent security suite: 493 passed
  • deterministic routing evaluation: 242 cases, 0 misroutes, no baseline regressions
  • catalog size: public 5,081.5 approximate tokens; Victor 5,731.5 approximate tokens
  • full repository: 842 passed, 20 failed; the same 20 unrelated failures reproduce on feat: add private meeting room booking skill #225
  • Python compilation and git diff --check passed

The live OpenAI-backed routing evaluation was not run locally because no real OPENAI_API_KEY is available. The deterministic gate and all local suites above were run; the live command remains python scripts/run_routing_eval.py --mode v2 --check when a key is available.

Companion feature: #225

@Light-code11 Light-code11 changed the title refactor(router): preserve routing semantics within token budget refactor(router): clarify routing prompt within token budget Aug 22, 2026
@Light-code11

Copy link
Copy Markdown
Collaborator Author

Updated the router prerequisite to address the prompt-semantics concern directly.

  • positive examples are labeled Actions: because their values are actions inside the current skill
  • counterexamples are labeled Other tools: because their values select a different tool
  • exclusion guidance remains the explicit Do NOT: directive
  • tests assert the rendered action and alternative-tool mappings, rather than only checking that headings exist

Validation: 15 focused router/catalog tests passed, the 493-test CI-equivalent suite passed, and 242 deterministic cases produced 0 misroutes with no baseline regression. Public and Victor catalogs are approximately 5,081.5 and 5,731.5 tokens. A live OpenAI-backed eval could not be run locally because no real API key is available.

@Light-code11
Light-code11 requested review from callumholt and removed request for callumholt August 22, 2026 00:56
@callumholt

Copy link
Copy Markdown
Collaborator

PR review

I reviewed the router prompt-format changes, generated tool schemas, catalog lint/budget gates, and routing evaluation behavior at commit 526f3f4591e77582f465bc4dc965cdc33e126c75.

Findings

No blocking correctness, security, or routing-contract issues found.

  • The compact format preserves each routing dimension: Use: for positive scope, Do NOT: for exclusions, Actions: for examples mapped to actions in the current skill, and Other tools: for counterexamples mapped to alternate tools.
  • Action enums, parameter schemas, required fields, channel filtering, event-only filtering, validation, and retry behavior are unchanged.
  • The revised tests directly assert all rendered sections and mappings, replacing the previous weak conditional assertion.
  • Generated catalog sizes are approximately 5,081.5 tokens public, 5,731.5 tokens Victor, and 1,911.2 tokens admin, all below the 6,000-token gate.

Validation

  • Focused router, catalog, and hermetic routing gates: 15 passed.
  • Deterministic routing evaluation: 242 cases, 0 misroutes, no baseline regressions.
  • GitHub security-checks: passed (493 tests).
  • Python compilation and git diff --check: passed.

The deterministic evaluation only exercises exact-match routing and therefore cannot measure how the live model interprets the changed descriptions. The live OpenAI-backed evaluation was not run because no authorized real API key was available. This is a documented residual prompt-quality risk, but I consider it non-blocking because the semantic mappings remain explicit, the schemas are unchanged, and the structural/budget gates pass.

Conclusion: Safe and ready to merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants