Skip to content

feat: subagent model routing with Jev (PoC) - #1390

Open
hary73 wants to merge 15 commits into
mainfrom
hary/subagent-auto
Open

hary73 wants to merge 15 commits into
mainfrom
hary/subagent-auto

Conversation

@hary73

@hary73 hary73 commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

When a subagent is spawned, one call to Jev (the TrueFoundry decision model) rates each of the caller's scored models for the task. The subagent runs on the highest-rated one. It's off by default and configured per tenant: shadow logs the pick and keeps the parent's model, and on runs the pick. This is a PoC for internal tenants in TrueFoundry mode.

Design: the LLD was reviewed on this branch and isn't merged. Last version: lld.md at 7c2a7d2.

Related:

  • AGE-2837 / feat(core): tag model calls with tfg.thread_id #1392: tfg.thread_id on model calls, which routing analysis joins on.
  • AGE-2841: a routed subagent must use its own model's default params and keep its model on resume. That fix isn't in this PR. This PR makes the subagent run on the routed model, but the saved THREAD_CREATED record and the inherited params are AGE-2841. It's needed before on mode, not for shadow.

Changes

Core (trueforge-core)

  • An optional routeModel hook on ITurnResourceResolver, accepted by TurnResourceResolver.
  • makeCreateDynamicSubAgentThread calls it when the create_sub_agent request has no model, and continues with the returned model through the existing request.model path.

Server: provider-neutral routing (src/modelRouting/)

  • types.ts: RoutingCandidate, DecisionClient and the modes.
  • modelRoutingScores.ts + scores.ts: the Artificial Analysis Intelligence Index v5 per slug, and the lookup by slug and effort. Variants match exactly: the exact effort, else the plain slug, else the nearest effort.
  • pick.ts: highest yes-probability, with the cheaper model winning a tie.
  • benchmarks.ts: the nine Intelligence Index evaluations Jev is asked about for resemblance (logged only), with descriptions quoted from their official pages.
  • router.ts: builds the questions, calls the decision client, picks and logs one line. It never throws, and on any failure the subagent runs on the parent's model.

Server: TrueFoundry mode

  • An optional listRoutingCandidates() on IModelProviderStore, implemented by the TrueFoundry store and forwarded by the inline store. mapEnabledModels reads the slug, default effort, price and image input. Each new field falls back to undefined on an unexpected shape, so model listing can't break.
  • jevDecisionClient.ts: POST {gateway}/v1/systemone with Jev's own gateway config (the store's by-name lookup now resolves decision-type models; listings stay chat-only) and the turn metadata, plus tfg.thread_id.
  • The TRUEFOUNDRY_MODEL_ROUTING config (timeout, candidate cap, and per tenant: mode and decision model), wired in apis/turns.ts.

Invariants to check

# Invariant Where to check
I1 One Jev call per subagent, before it starts. Never during the agent loop, and never on resume or rebuild. SessionHandle.makeCreateDynamicSubAgentThread
I2 Jev sees only the task and the candidate pool. One yes/no question per candidate (name, effort, score), plus the nine benchmark-resemblance questions. No parent model, prices or comparisons. modelRouting/router.ts (questions, state)
I3 The pick is the highest yes-probability, the cheaper model on a tie. No threshold, no price ceiling yet. modelRouting/pick.ts
I4 The fallback is the parent's model. It runs whenever routing can't produce a pick. modelRouting/router.ts
I5 Candidates are the caller's models that have a score. Nothing else filters them; price and image input are only for tie-breaks and the log line. router.ts (scoredCandidates), TrueFoundryModelProviderStore.listRoutingCandidates
I6 Scores are bundled data, keyed by Artificial Analysis slug. No Artificial Analysis call at request time. modelRouting/modelRoutingScores.ts, scores.ts
I7 Jev sees each candidate's score at its TrueFoundry default effort. The subagent only runs at that effort once AGE-2841 lands. scores.ts (lookupScore), mapEnabledModels.ts
I8 The router never throws. Errors and timeouts log a reason and keep the parent's model. router.ts (the try/catch); router test
I9 Shadow mode asks Jev and logs the pick, but never changes the model. router.ts; router test
I10 The routed model goes through the existing request.model path. Saving it on THREAD_CREATED for resume is AGE-2841, not this PR. SessionHandle.ts
I11 One optional core hook, routeModel. Routing logic lives in the server, split into provider-neutral code (modelRouting/) and TrueFoundry-specific code (truefoundry/). ITurnResourceResolver.ts
I12 Jev is never listed. The TrueFoundry store resolves decision models by name only; model listings stay chat-only. TrueFoundryModelProviderStore.#records; store test
I13 Model listing can't break on routing metadata. The new metadata fields fall back to undefined on an unexpected shape. mapEnabledModels.ts (.catch(undefined))
I14 Off by default, per tenant. Routing is wired only for tenants listed in TRUEFOUNDRY_MODEL_ROUTING with mode shadow or on, in TrueFoundry mode. apis/turns.ts (subagentModelRouter), config.ts
I15 Observability: gateway traces joined on tfg.thread_id, plus one info log line per decision. No new tables, schema or public API. jevDecisionClient.ts, router.ts

How was this tested?

  • Unit tests:

    • the router: on mode runs the candidate Jev prefers, even when it's the pricier one listed second; shadow mode asks Jev but keeps the parent's model; a failing decision model doesn't throw;
    • the TrueFoundry store resolves a decision model by name but never lists it.

    One existing assertion was relaxed to objectContaining.

  • trueforge-core and trueforge typecheck and the full unit suites pass. ESLint and Prettier are clean on the changed files.

  • A live check against internal, with a throwaway script that isn't committed:

    • the real model list gave 44 models and 18 scored candidates;
    • Jev answered in 0.4–0.7 s and picked Sonnet 5.5 for a research task with an Opus parent;
    • shadow mode returned no model, and on mode returned Sonnet.

Checklist

  • I have read the contributing guidelines
  • pnpm build, pnpm test, pnpm typecheck, pnpm lint:ci, and pnpm format:check pass locally (ran trueforge-core and trueforge typecheck and unit tests, plus lint and format on the changed files)
  • Tests added/updated where it makes sense
  • No hand-edits to generated code
  • .env.example updated for the new configuration

Note

Medium Risk
Touches turn execution and subagent LLM selection with external gateway calls; mitigated by default-off config, shadow mode, timeouts, and fail-open fallback to the parent model.

Overview
Adds optional per-tenant subagent model routing in TrueFoundry mode. Before a dynamic subagent starts (when no model is set on the request), an optional routeModel hook on ITurnResourceResolver can choose a model; core wires this in SessionHandle.makeCreateDynamicSubAgentThread and passes the result through the existing request.model path.

On the server, TRUEFOUNDRY_MODEL_ROUTING configures routing per tenant (off / shadow / on), timeout, candidate cap, and a gateway decision model (Jev). The router scores the caller’s models using bundled Artificial Analysis index data, asks Jev yes/no questions per candidate, picks the highest probability (cheaper on ties), and never throws—failures keep the parent’s model. shadow logs the pick without changing the model; on runs the pick.

Supporting pieces include listRoutingCandidates on the model provider store, TrueFoundry catalog metadata (slug, effort, price), a Jev client to /v1/systemone, and unit tests for the router and decision-model listing behavior.

Reviewed by Cursor Bugbot for commit 3adbb05. Bugbot is set up for automated code reviews on this repo. Configure here.

@changeset-bot

changeset-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 3adbb05

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@truefoundry/trueforge-core Patch
@truefoundry/trueforge Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

hary73 and others added 5 commits November 16, 2026 14:06
Co-authored-by: Cursor <cursoragent@cursor.com>
…ing, routing trace keys)

Co-authored-by: Cursor <cursoragent@cursor.com>
…ad_id

Co-authored-by: Cursor <cursoragent@cursor.com>
…hreshold

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@hary73
hary73 force-pushed the hary/subagent-auto branch from 42ed3b4 to 88b3379 Compare November 16, 2026 08:36
@hary73 hary73 changed the title docs: LLD for subagent auto model routing with Jev feat: subagent model routing with Jev (PoC, TrueFoundry mode) Nov 16, 2026
hary73 and others added 3 commits November 16, 2026 15:10
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@hary73 hary73 changed the title feat: subagent model routing with Jev (PoC, TrueFoundry mode) feat: subagent model routing with Jev (PoC) Nov 16, 2026
hary73 and others added 2 commits November 16, 2026 15:39
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@hary73
hary73 marked this pull request as ready for review November 16, 2026 10:19
@veria-ai

veria-ai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

PR overview

No security issues were found in this pull request.

Reviews

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

models: mapEnabledModels({
integrations,
...(input.filter !== undefined ? { modelTypes: ['chat', 'decision'] } : {}),
}),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision model blocked by allowlist

Medium Severity

Jev is resolved through getProvider, which runs filterEnvModels. A tenant with TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS that omits Jev's account (typesafe in the example) will fail every decision lookup. Routing then always falls back to the parent's model with no distinct signal that the decision model was filtered out.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 97a4952. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 2766586. Configure here.

return input.mode === 'on' ? { model: picked.candidate.model } : {};
} catch (error) {
return fallback(timeout.aborted ? 'decision_timeout' : 'decision_error', { err: error });
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cancel during routing still spawns

Medium Severity

When the turn abort signal fires during routing, the router catches it and returns a fallback instead of propagating cancellation. makeCreateDynamicSubAgentThread then always continues into resolveAgentDefinition and builds the child thread, so a cancelled turn can still spawn a subagent.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 2766586. Configure here.

hary73 and others added 2 commits November 16, 2026 17:32
…bilities

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant