From 7de4ad215f4d52cc5201f49839b4916229e35bb6 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Fri, 13 Nov 2026 18:16:16 +0530 Subject: [PATCH 01/14] docs: add LLD for subagent auto model routing with Jev Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 343 ++++++++++++++++++++++++++++++ 1 file changed, 343 insertions(+) create mode 100644 docs/subagent-auto-routing/lld.md diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md new file mode 100644 index 000000000..6a4064123 --- /dev/null +++ b/docs/subagent-auto-routing/lld.md @@ -0,0 +1,343 @@ +# Model routing with Jev: low-level design (draft for review) + +Status: ready for review. Replaces the selection rule in `lld.md` (tier question plus same-creator price rule). The routing is built as one generic router with two call sites: +- **Subagent spawn,** implemented now. +- **The root agent's turn,** a planned extension that uses the same router, tables, Jev call and tracing. + +From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. + +## Decisions at a glance + +**Fixed by this design.** Changing any of these means a new design review. + +| # | Invariant | Where | +|---|---|---| +| I1 | **One Jev call per routing decision, made before the thread starts.** Never during the agent loop. A subagent is routed once, and the root once per turn. | §1, §8 | +| I2 | **The Jev payload shape:** `state` = `{ scope, name, task, context, tools, reference_model { name, score } }`; `questions` = one yes/no `can_` per candidate (model name, effort and score in the text) plus one yes/no `resembles_` per benchmark. | §3 | +| I3 | **The pick rule shape:** candidates in price order, cheapest first; take the first whose yes-probability is at or above the threshold; otherwise keep the reference model. | §4 | +| I4 | **Never pick a model that costs more than the reference.** A cheaper model with a higher score is allowed. | §4 | +| I5 | **Candidates come only from the caller's own model list** (the same `provider-integrations` fetch as `listModels`, no extra account filter), and only models in the score table. | §2 | +| I6 | **Scores are bundled data,** generated offline from the Artificial Analysis API. There's no call to Artificial Analysis while a request runs. | §2 | +| I7 | **Scores are read at the effort the thread will actually run at.** A routed thread runs at its own model's default params, never the reference's. | §2, §4 | +| I8 | **If a model is already set on the thread, routing is skipped.** | §4 | +| I9 | **The router never throws.** Any error, timeout or unusable pick means the reference model runs, as today. | §4 | +| I10 | **Only the chosen model is persisted** (`agent_info.model` for subagents). Resume and rebuild never call the router. | §1 | +| I11 | **One generic core hook,** `routeModel(RouteModelInput) → RouteModelResult`, with a `scope`. All routing logic lives in the server. | §1, §9 | +| I12 | **Observability comes from gateway traces,** with `tfg.thread_id` on every model call and on the Jev call, plus one info log line. No new tables or public API for the subagent scope. | §6, §7 | +| I13 | **Mode per tenant and per scope:** off, shadow or on. Off is the default. | §5 | + +**Tunable without a design change,** as config or data: +- **Config:** the threshold (0.7), the Jev timeout (2 s), `max_candidates` (60), the decision model ID, and the mode for each tenant and scope. +- **Data:** the score table's contents and which benchmark it's based on (today the Intelligence Index v5), and the benchmark-resemblance list (today five benchmarks). + +**Needs a reviewer's decision:** see §11. + +## What it does + +Before a thread runs, the router asks Jev, the decision model, two things about the work. The thread is a subagent now, and later the root agent. + +1. **Routing:** for each candidate model, "can this model complete the task?". Candidates are models the caller can use that are in the model table (§2) and cheaper than the **reference model**: the model the thread would use without routing. + - **For a subagent,** that's the parent's model. + - **For the root,** it's the agent spec's model. + + Each question carries the candidate's score. The thread runs on the cheapest candidate Jev says yes to, or otherwise on the reference model. It never picks a model that costs more than the reference. A cheaper model with a higher score is allowed. +2. **Benchmark resemblance:** for each benchmark on a hard-coded list, "does this task resemble the benchmark?". This is recorded only. Over real traffic, it tells us which benchmark our work actually resembles, which is the one to use for evaluation later. + +Every model call carries `tfg.thread_id` in `x-tfy-metadata`, and so does the Jev call. The gateway traces then show which thread each call belongs to and how it was routed. + +```mermaid +flowchart TD + S1["Subagent spawn
(makeCreateDynamicSubAgentThread)"] --> R + S2["Root turn start (later)
(root thread build in SessionHandle)"] -.-> R + R{"resolver.routeModel?
(wired if the scope's mode is shadow/on)"} + R -- not wired / explicit model --> K["keep reference model"] + R -- wired --> J["Jev: POST {gateway}/v1/systemone
candidate + benchmark questions
x-tfy-metadata: tfg.thread_id, tfg.routing_scope"] + J --> P["pick: cheapest candidate below the reference with P(yes) >= threshold, else reference"] + P --> L["one info log line (scope, thread_id, pick, reason)"] + L --> A["apply at the call site:
subagent → agent_info.model · root → turn model override"] + A --> T["every model call of the thread: tfg.thread_id (+ tfg.parent_thread_id)"] +``` + +## 1. The router contract (generic) + +The router knows nothing about subagents or roots. Each call site builds one `RouteModelInput`, and applies the `RouteModelResult` its own way. + +```ts +// trueforge-core: agent-session/ITurnResourceResolver.ts +type RoutingScope = 'subagent' | 'root'; + +interface RouteModelInput { + scope: RoutingScope; + threadId: string; // the thread being routed (child thread id, or root thread id) + parentThreadId?: string; // subagent only + referenceModel: string; // model the thread would use without routing (account/model) + work: { // what Jev classifies + name?: string; // subagent name; root: agent name + text: string; // subagent: delegated task; root: latest user message + context?: string; // subagent: the user's request; root: short recent-conversation excerpt + tools: string[]; // MCP server names, web_search, sandbox + }; + spec: AgentSpec; + signal: AbortSignal; + tracing: AgentTracing; +} + +interface RouteModelResult { + model?: string; // set only in "on" mode when a cheaper model was picked +} + +interface ITurnResourceResolver { + // ...existing methods + routeModel?(input: RouteModelInput): Promise; // never throws; undefined hook = today +} +``` + +**Call sites:** + +| Scope | Where | Builds `work` from | Applies the result | When | +|---|---|---|---|---| +| `subagent` | `SessionHandle.makeCreateDynamicSubAgentThread` | `create_sub_agent` name and input, the turn's user message, the subagent's tools | `request.model = result.model`, so `resolveAgentDefinition` picks it up. It's persisted on `agent_info.model` through `THREAD_CREATED`. | Now | +| `root` | Where `SessionHandle` builds the root thread's definition for a turn (`resolveAgentDefinition` without `agent_info`) | The turn's user message, a clipped excerpt of recent conversation, the agent's tools | Pass as a model override into the root `resolveAgentDefinition`, and persist it with the turn so approval or user-input resumes keep the same model (§8) | Later | + +Resume and rebuild never call the router. A subagent is rebuilt from its saved `agent_info`. A resumed root turn reuses the model stored for that turn. + +## 2. Hard-coded tables + +**The score is the Artificial Analysis Intelligence Index, v5,** on a 0–100 scale. It's the closest single score to agentic work: seven of its nine evaluations use tools, all run in Artificial Analysis's open-source agent harness, Stirrup. It has four equal categories: +- **Knowledge Work:** AA-Briefcase, AutomationBench-AA, GDP.pdf. +- **Coding:** FrontierCode-AA, Terminal-Bench 4.0. +- **Science:** Terminal-Bench-Science, CritPt. +- **General:** AA-Omniscience, HLE. + +The General quarter isn't tool use, and that's accepted for now. + +**The model table** is keyed by `canonical_model`, so one row covers every account serving the model. +- **Source:** the Artificial Analysis API (`/api/v2/language/models/free`), matched by the `external_ids.artificial_analysis_slug` that TrueFoundry already stores on each model. +- **Storage:** a JSON file checked into the server package, `truefoundry/modelRoutingScores.json`. It's written by an offline refresh script and reviewed like any code change. The server loads it once at startup into an in-memory `Map`, and validates it with zod, so a bad file fails at startup, not mid-request. Refreshing means re-running the script, committing and deploying. +- **Shape:** nested by effort level, keeping every score Artificial Analysis reports for the model. Non-reasoning variants go under `none`, and a variant with no effort label under `default`: + +```json +{ + "source": "Artificial Analysis Intelligence Index", "index_version": "5", "retrieved_at": "2026-11-13", + "models": { + "anthropic/claude-haiku-5-5": { "aa_slug": "claude-haiku-5-5", + "efforts": { "low": 18.6, "medium": 24.6, "high": 29.9, "xhigh": 34.6, "max": 36.5 } }, + "anthropic/claude-opus-5-5": { "aa_slug": "claude-opus-5-5", + "efforts": { "low": 34.8, "medium": 48.2, "high": 51.8, "xhigh": 57.4, "max": 59.1 } }, + "google/gemini-3-5-flash-lite": { "aa_slug": "gemini-3-5-flash-lite", "efforts": { "default": 11.7 } } + } +} +``` + +- **Lookup at request time** uses the effort the thread will actually run at: + - **For a candidate,** that's its default effort from TrueFoundry's metadata. + - **For the reference,** it's `spec.model.params.reasoning_effort` if set, otherwise its default. So an Opus root running at high is compared at 51.8, not 48.2. + - **The fallback order** is the exact effort, then `default`, then the nearest effort with a score, with ties going to the higher one. + +Scores looked up at each model's default effort, with the variant used and the TrueFoundry default: + +| `canonical_model` | Intelligence Index | Variant used (TrueFoundry default) | +|---|---|---| +| `openai/gpt-6-astra` | 51.6 | medium (medium) | +| `google/gemini-4-argon` | 50.4 | high (medium) | +| `anthropic/claude-opus-5-5` | 48.2 | medium (medium) | +| `anthropic/claude-fable-5-1` | 47.2 | high (high) | +| `openai/gpt-6-1-sol` | 46.3 | medium (medium) | +| `openai/gpt-6-sol` | 44.2 | max (medium) | +| `anthropic/claude-opus-5` | 44.2 | max (high) | +| `anthropic/claude-sonnet-5-5` | 41.5 | high (high) | +| `xai/grok-4-7` | 35.5 | xhigh (not set) | +| `openai/gpt-5-6-terra` | 32.3 | max (medium) | +| `xai/grok-4-6` | 31.0 | xhigh (low) | +| `google/gemini-3-8-flash` | 30.1 | medium (medium) | +| `anthropic/claude-haiku-5-5` | 24.6 | medium (medium) | +| `openai/gpt-5-6-luna` | 23.8 | max (medium) | +| `openai/gpt-6-luna` | 18.5 | medium (medium) | +| `kimi/kimi-k3` | 17.4 | low (max) | +| `google/gemini-3-5-flash-lite` | 11.7 | unlabelled (minimal) | + +That's 35 of devtest's 465 distinct models, and 17 of the 49 pool models. Models used on devtest that have no v5 score in any variant include Sonnet 4.6 (the most common devtest root), the Opus 4.x models, GLM 5.2 and Gemini 3.6 and 3.7 Flash. A model not in the table is never a candidate, and a reference model not in the table isn't routed (§5). + +**Candidates are built per call from the caller's model list,** through an internal `listRoutingCandidates()` on `TrueFoundryModelProviderStore`. +- **It shares `listModels`' fetch and mapping:** the same `provider-integrations` call, with the caller's token and the tenant's account allow-list. +- **Only the internal `TrueFoundryEnabledModel` type is widened,** with `canonical_model`, price, features, status, retirement date and default effort. +- **`listModels` and the public `AvailableModel` schema are unchanged.** +- **It's called once per turn and cached.** + +A model is a candidate when it's: +- an active chat model with tool calling, not retiring within 30 days, +- reasoning-capable, if the reference reasons, +- in the model table, +- and cheaper than the reference, using the blended price of 0.8 × input plus 0.2 × output. + +If several accounts serve the same model, it's asked about once, preferring the reference's account, then its provider, then the first by name. There's no fixed limit. On devtest, Jev took 0.9, 0.9, 1.2 and 1.2 seconds for 6, 20, 40 and 60 candidate questions plus 7 benchmark questions. + +**Benchmarks for the resemblance questions.** These are only benchmarks that `model-benchmarks` has data for, and that cover all the models we route between today (Opus 5.5, Sonnet 5.5, Haiku 5.5, GPT-6.1 Sol, GPT-6 Sol, GPT-6 Luna, Gemini 4, Gemini 3.8 Flash and Grok 4.7). So whichever one our tasks resemble, we can switch the score table to it straight away. The descriptions and examples are our own paraphrases. + +| Key | Benchmark (`model-benchmarks` metric) | Pool coverage | Description given to Jev | Example task given to Jev | +|---|---|---|---|---| +| `automation` | AutomationBench-AA (`automationbench_partial_score`) | 34/49 | Multi-step workflows across apps through API tools: look things up, combine results, act on them | "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner" | +| `terminal_bench` | Terminal-Bench 4.0 (`terminalbench_v4_0`) | 33/49 | Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something | "Find why the service fails to start from its logs and fix the config" | +| `gdpval` | GDPval-AA (`gdpval_aa_elo`) | 37/49 | Knowledge-work deliverables (reports, analyses, plans) produced with tools | "From these project documents, write a one-page status report with risks and next steps" | +| `lcr` | AA-LCR (`lcr`) | 48/49 | Answer questions that need reasoning across long documents or large outputs | "From these three long reports, which risk do all of them mention and how does each address it?" | +| `hle` | Humanity's Last Exam (`hle`) | 49/49 | Hard expert-level knowledge and reasoning questions with one correct answer | "A difficult graduate-level question in a technical field" | + +These were left out because the repo covers them too thinly, or not for our key models: EnterpriseOps-Gym, tau2 and IFBench (none of the key models), ITBench, FrontierCode and AnalystAgent (sparse in the pool). + +## 3. The Jev request + +One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway URL and token borrowed from the reference model's provider config, as in `lld.md`. A reference model not in the table skips the call. + +```json +{ + "model": "archit-typesafe/jev-1.13.0", + "state": { + "scope": "subagent", + "name": "", + "task": "", + "context": "", + "tools": ["", "web_search", "sandbox"], + "reference_model": { "name": "Claude Opus 5.5", "intelligence_index": 48.2 }, + "benchmark_note": "Artificial Analysis Intelligence Index (0-100, higher is better) averages agentic knowledge work, coding, science and general reasoning evaluations." + }, + "questions": { + "can_0": { + "type": "noul", + "instructions": "Can Claude Haiku 5.5 at medium effort (Intelligence Index 24.6) complete this task fully and correctly, about as well as the reference model (Intelligence Index 48.2)?", + "criteria": { + "true": "It would produce a result that can be used as-is: complete, accurate, and with the needed tool use done.", + "false": "It would likely miss parts, make errors, or stop before the task is done." + } + }, + "can_1": { "type": "noul", "instructions": "Can Claude Sonnet 5.5 (Intelligence Index 41.5) complete ...", "criteria": { "...": "same as above" } }, + "resembles_automation": { + "type": "noul", + "instructions": "Does this task resemble the benchmark AutomationBench? Multi-step workflows across apps through API tools: look things up, combine results, act on them. Example: 'When a support ticket is tagged urgent, look up the customer's plan and notify the account owner.'", + "criteria": { "true": "Same kind of work as the benchmark.", "false": "A different kind of work." } + } + } +} +``` + +- **The wording is scope-neutral,** with "task" and "reference model", so the root call site sends the same shape with `scope: "root"`. +- **Candidate questions** are keyed `can_`, and the router maps them back to models. +- **All five benchmark questions** go in every call. +- **One call,** with a 2-second timeout and `max_candidates` (default 60) as a safety cap. + +## 4. Picking the model + +1. **Order:** candidates sorted by price, cheapest first. On a price tie, the higher score goes first. +2. **Pick** the first candidate with `noul >= threshold`. The default threshold is 0.7. +3. **Fall back to the reference model:** + - when no candidate reaches the threshold (`no_candidate_confident`), + - when the reference isn't in the table (`reference_not_in_table`), + - when Jev fails or times out (`decision_error` / `decision_timeout`), + - or when the pick can't be resolved for the caller (`model_unavailable`, using the existing `getModel` check). +4. **Never pick a more expensive model.** The limit is price, not quality: a candidate must be cheaper than the reference, but it may score higher. For example, Gemini 4 (50.4) is a candidate for an Opus 5.5 reference (48.2 at medium). Moving to a pricier model, even a better one, isn't done, because the goal is saving cost and the reference's price is what the user signed up for. +5. **If a model is already set, skip routing.** + - **For a subagent,** that's `agent_info.model`. It can only be set today if the agent spec defines a model set for `create_sub_agent` (`modelSetConfig`), which nothing does yet. + - **For the root,** it's a model the user picked explicitly (§8). + + A deliberate choice is never overridden. +6. **Routed threads use their own model's default params,** not the reference's (the fix from `lld.md` §5). + +**Example:** a subagent whose reference is Opus 5.5 (48.2). In price order, Jev answers Haiku 5.5 0.42, GPT-6 Luna 0.35, Gemini 3.8 Flash 0.30, …, Grok 4.7 0.81, …, Sonnet 5.5 0.88. The first answer at or above 0.7 is Grok 4.7 ($2.80), so the subagent runs on Grok 4.7. + +## 5. Modes and config + +Modes are set per tenant **and per scope**, so root routing can be turned on separately later. + +```json +TRUEFOUNDRY_MODEL_ROUTING = { + "decision_model": "archit-typesafe/jev-1.13.0", + "timeout_ms": 2000, + "threshold": 0.7, + "max_candidates": 60, + "tenants": { + "": { "subagent": "shadow", "root": "off" } + } +} +``` + +- **off** (the default, or when unset): no call. The hook isn't wired for that scope. +- **shadow:** ask Jev and log the decision, but run on the reference model. +- **on:** run on the pick. + +## 6. Tracing: `tfg.thread_id` + +TrueForge's own `AgentTracing` is a no-op by default (`createTracing()` returns `NOOP_AGENT_TRACING`). The traces we can see are the gateway's request traces, which record each call's `x-tfy-metadata`. The turn-level keys are there (`tfg.session_id`, `tfg.turn_id`, `tfg.agent_id`, `tfg.agent_name`), but nothing identifies the thread. +- **Every thread's model calls:** in `AgentThread`, the model call `modelClient.create(requestBody)` gets `{ metadata: { 'tfg.thread_id': this.threadId, ...(parent ? { 'tfg.parent_thread_id': parent.thread_id } : {}) } }`. + - `buildOutboundRequestHeaders` already folds this into `x-tfy-metadata`. + - It covers the root thread and subagents alike, so root routing needs nothing extra here. + - Context compaction merges the same keys alongside `tfg.compaction`. +- **The Jev call** carries the turn metadata, plus `tfg.thread_id` (the thread being routed), `tfg.routing_scope` and `tfg.model_routing: "true"`. +- **Joining on `tfg.thread_id`** links the Jev request and response with every model call the thread made (§7). + +## 7. What's observable, and the one log line + +Most of the decision is already in the gateway traces, joined by `tfg.thread_id`: +- **The Jev request:** every candidate, with its score, and the benchmark questions. +- **The Jev response:** the probability for each one, including the benchmark-resemblance answers. +- **Every model call the thread made:** the served model, tokens, cost and latency. + +Two things never reach a trace: +- **Decisions where Jev isn't called:** the reference isn't in the table, the model was explicit, or the scope is off. +- **The router's own result:** which candidate it picked, why it fell back, and in shadow mode the pick it didn't apply. + +For those, the router writes one plain `logger.info('model routing', { scope, thread_id, mode, reference_model, picked_model, chosen_model, reason })`. It's for our own understanding, not a schema or a contract, and never includes task text. + +User feedback is joined by `thread_id`. The benchmark-resemblance counts are read from the Jev responses in the traces. + +## 8. Extending to the root agent (later) + +The router, tables, Jev request, pick rule, config and tracing are all shared. What the root needs on top: + +- **A call site:** where `SessionHandle` builds the root thread's definition for a turn. It builds `RouteModelInput` with `scope: 'root'` and the spec's model as the reference, and passes the result as a model override into the root `resolveAgentDefinition`. That's an optional `modelOverride` input, alongside `agent_info`. +- **Granularity: once per turn,** keeping the model for the whole turn. Re-deciding mid-turn would change model during an agent loop, which the Artemis baseline showed hurts. A session-level sticky option (keep the first turn's choice) can come later, if prompt caching matters more than per-turn savings. +- **Persistence:** store the routed model with the turn, as a turn-snapshot field, so a resumed turn (after an approval or user input) keeps the same model and never re-classifies. It's the only new stored field, and it's needed only for root. +- **Opt-in:** route the root only when the agent or tenant opts in. For example, a spec setting like `model: { name: , auto: true }`, or the per-scope tenant mode. A model the user picked explicitly is never overridden. +- **State:** the latest user message plus a clipped excerpt of recent conversation, instead of a delegated task. + +## 9. Code changes (subagent scope now) + +**Core (`packages/trueforge-core`):** + +| File | Change | +|---|---| +| `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult`, and the optional hook `routeModel?(input)` (§1). | +| `agent-session/TurnResourceResolver.ts` | Forward the hook from deps, and pass `isModelAvailable` (built on `getModel`). Plus the params fix from `lld.md` §5. | +| `agent-session/SessionHandle.ts` `makeCreateDynamicSubAgentThread` | Build `RouteModelInput` (`scope: 'subagent'`) and call `routeModel`. If `model` comes back, use `{ ...request, model }` from then on. | +| `core/runtime/AgentThreadOrchestrator.ts` (`AGENT_CREATE_SUBAGENT`) | Build `THREAD_CREATED.agent_info` from `subAgentThread.agentInfo`, so the routed model is persisted. | +| `core/runtime/AgentThread.ts` | Pass `tfg.thread_id` and `tfg.parent_thread_id` metadata on every model call (§6). | + +**Server (`packages/trueforge`):** + +| File | Change | +|---|---| +| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING`, with per-tenant, per-scope modes (§5). | +| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with `canonical_model`, price, features, status, retirement date and default effort. `listModels` keeps returning the public `AvailableModel`, unchanged. | +| `truefoundry/TrueFoundryModelProviderStore.ts` | The internal `listRoutingCandidates()`, using the same fetch and mapping as `listModels` (§2). | +| **New** `truefoundry/modelRoutingScores.json` | The score table, nested by effort level, with source, version and date (§2). It's generated by the refresh script. | +| **New** `truefoundry/modelRoutingTables.ts` | Loads and validates the JSON at startup into an in-memory map, does the lookup by effort with its fallback order, and holds the five benchmark descriptions (§2). | +| **New** `benchmark/refresh_model_scores.py` (offline) | Calls the Artificial Analysis API, matches models by slug, and writes `modelRoutingScores.json`. | +| **New** `truefoundry/modelRouter.ts` | `createModelRouter()`, which returns `routeModel`. It's scope-agnostic: candidates (§2), the Jev request (§3), the pick (§4), one info log line (§7). It never throws. | +| `apis/turns.ts` | Wire `routeModel` when any scope for the tenant isn't off. The router checks the per-scope mode itself. | + +The subagent scope needs no database schema or public API changes, since `agent_info.model` already exists. The root scope adds one turn-snapshot field (§8). + +## 10. Rollout + +1. Merge with the config unset, so nothing changes. +2. Run subagent scope in shadow mode for two or three devtest tenants. Check latency, the fallback reasons, the pick distribution and `top_benchmark`. +3. Switch subagent scope to "on" for those tenants. Measure with user feedback, joined by `thread_id`. +4. Add the root call site (§8), and run root scope in shadow mode first. +5. Use the resemblance data to choose the evaluation benchmark, and replace the score table if a narrower one fits better. + +## 11. Open questions for review + +1. **Unscored models in use:** Sonnet 4.6, the Opus 4.x models and GLM 5.2 have no v5 score. Should they get a hand-assigned tier so they can be candidates or references, or stay unrouted? +2. **Pick rule:** the cheapest candidate with P(yes) ≥ 0.7, or the highest P(yes) among cheaper candidates? +3. **Model names in the questions:** name them, as in option 1 of the picture, or say only "a model with Intelligence Index S" to avoid Jev's priors about names? +4. **Benchmark list:** are these five right? Should there also be a single "closest benchmark" choice question? +5. **Root scope:** routing once per turn or once per session, and how a user or agent opts in to an automatically selected model. +6. **Traces:** is the gateway request trace with `x-tfy-metadata` enough, or should a real TrueForge tracer be wired up as well? From 7aeebe699c4df28b32407379f3f47de63a101986 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 09:49:47 +0530 Subject: [PATCH 02/14] docs: update subagent routing LLD (no reference in Jev, no price ceiling, routing trace keys) Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 144 +++++++++++++++++------------- 1 file changed, 80 insertions(+), 64 deletions(-) diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md index 6a4064123..d8b467822 100644 --- a/docs/subagent-auto-routing/lld.md +++ b/docs/subagent-auto-routing/lld.md @@ -13,21 +13,20 @@ From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. | # | Invariant | Where | |---|---|---| | I1 | **One Jev call per routing decision, made before the thread starts.** Never during the agent loop. A subagent is routed once, and the root once per turn. | §1, §8 | -| I2 | **The Jev payload shape:** `state` = `{ scope, name, task, context, tools, reference_model { name, score } }`; `questions` = one yes/no `can_` per candidate (model name, effort and score in the text) plus one yes/no `resembles_` per benchmark. | §3 | -| I3 | **The pick rule shape:** candidates in price order, cheapest first; take the first whose yes-probability is at or above the threshold; otherwise keep the reference model. | §4 | -| I4 | **Never pick a model that costs more than the reference.** A cheaper model with a higher score is allowed. | §4 | +| I2 | **The Jev payload shape:** `state` = `{ scope, name, task, context, tools }`; `questions` = one yes/no `can_` per candidate (model name, effort and score in the text) plus one yes/no `resembles_` per benchmark. **Jev sees only the task and a pool of candidates.** It's never told about the fallback model, prices, or any comparison. | §3 | +| I3 | **The pick rule shape:** candidates in price order, cheapest first; take the first whose yes-probability is at or above the threshold; otherwise run the fallback model. | §4 | +| I4 | **The fallback model is the model the thread would use without routing:** the parent's model for a subagent, the spec's model for the root. It's used only when routing doesn't pick. It doesn't limit which candidates are asked about: **there's no price ceiling,** so a routed thread can run on a model that costs more than its parent. | §4 | | I5 | **Candidates come only from the caller's own model list** (the same `provider-integrations` fetch as `listModels`, no extra account filter), and only models in the score table. | §2 | | I6 | **Scores are bundled data,** generated offline from the Artificial Analysis API. There's no call to Artificial Analysis while a request runs. | §2 | -| I7 | **Scores are read at the effort the thread will actually run at.** A routed thread runs at its own model's default params, never the reference's. | §2, §4 | -| I8 | **If a model is already set on the thread, routing is skipped.** | §4 | -| I9 | **The router never throws.** Any error, timeout or unusable pick means the reference model runs, as today. | §4 | -| I10 | **Only the chosen model is persisted** (`agent_info.model` for subagents). Resume and rebuild never call the router. | §1 | -| I11 | **One generic core hook,** `routeModel(RouteModelInput) → RouteModelResult`, with a `scope`. All routing logic lives in the server. | §1, §9 | -| I12 | **Observability comes from gateway traces,** with `tfg.thread_id` on every model call and on the Jev call, plus one info log line. No new tables or public API for the subagent scope. | §6, §7 | -| I13 | **Mode per tenant and per scope:** off, shadow or on. Off is the default. | §5 | +| I7 | **The score Jev sees matches how the model will actually run.** Artificial Analysis scores each effort level separately, so Jev gets the score at the candidate's default effort on TrueFoundry. A routed thread then runs at its own model's default params, not copied from the parent, so the score stays accurate. | §2, §4 | +| I8 | **The router never throws.** Any error, timeout or unusable pick means the fallback model runs, as today. | §4 | +| I9 | **The routed model is applied through the existing `request.model` path** (`create_sub_agent` request, then `agent_info.model`, then `resolveAgentDefinition`). Only the chosen model is persisted, on `agent_info.model`. Resume and rebuild never call the router. | §1 | +| I10 | **One generic core hook,** `routeModel(RouteModelInput) → RouteModelResult`, with a `scope`. All routing logic lives in the server. | §1, §9 | +| I11 | **Observability is in the gateway traces.** `tfg.thread_id` is on every model call and on the Jev call. The routing result (`tfg.routing_*`) is on every model call of the routed thread. One info log line repeats the decision server-side. No new tables or public API for the subagent scope. | §6, §7 | +| I12 | **Mode per tenant and per scope:** off, shadow or on. Off is the default. | §5 | **Tunable without a design change,** as config or data: -- **Config:** the threshold (0.7), the Jev timeout (2 s), `max_candidates` (60), the decision model ID, and the mode for each tenant and scope. +- **Config:** the threshold (0.4), the Jev timeout (2 s), `max_candidates` (60), the decision model ID, and the mode for each tenant and scope. - **Data:** the score table's contents and which benchmark it's based on (today the Intelligence Index v5), and the benchmark-resemblance list (today five benchmarks). **Needs a reviewer's decision:** see §11. @@ -36,11 +35,11 @@ From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. Before a thread runs, the router asks Jev, the decision model, two things about the work. The thread is a subagent now, and later the root agent. -1. **Routing:** for each candidate model, "can this model complete the task?". Candidates are models the caller can use that are in the model table (§2) and cheaper than the **reference model**: the model the thread would use without routing. +1. **Routing:** for each candidate model, "can this model complete the task?". Candidates are models the caller can use that are in the model table (§2). Each question carries the candidate's score. The thread runs on the cheapest candidate Jev says yes to. Otherwise it runs on the **fallback model**, the model it would use without routing: - **For a subagent,** that's the parent's model. - **For the root,** it's the agent spec's model. - Each question carries the candidate's score. The thread runs on the cheapest candidate Jev says yes to, or otherwise on the reference model. It never picks a model that costs more than the reference. A cheaper model with a higher score is allowed. + There's no price ceiling. A subagent of a cheap parent can be routed to a stronger, pricier model, and a subagent of an expensive parent to a cheaper one. This is deliberate for the first version: both directions give us data on how the router behaves. A ceiling can be added later as config (§11). 2. **Benchmark resemblance:** for each benchmark on a hard-coded list, "does this task resemble the benchmark?". This is recorded only. Over real traffic, it tells us which benchmark our work actually resembles, which is the one to use for evaluation later. Every model call carries `tfg.thread_id` in `x-tfy-metadata`, and so does the Jev call. The gateway traces then show which thread each call belongs to and how it was routed. @@ -50,10 +49,10 @@ flowchart TD S1["Subagent spawn
(makeCreateDynamicSubAgentThread)"] --> R S2["Root turn start (later)
(root thread build in SessionHandle)"] -.-> R R{"resolver.routeModel?
(wired if the scope's mode is shadow/on)"} - R -- not wired / explicit model --> K["keep reference model"] + R -- not wired --> K["keep the model the thread would use today"] R -- wired --> J["Jev: POST {gateway}/v1/systemone
candidate + benchmark questions
x-tfy-metadata: tfg.thread_id, tfg.routing_scope"] - J --> P["pick: cheapest candidate below the reference with P(yes) >= threshold, else reference"] - P --> L["one info log line (scope, thread_id, pick, reason)"] + J --> P["pick: cheapest candidate with P(yes) >= threshold, else fallback model"] + P --> L["tfg.routing_* trace keys + one info log line"] L --> A["apply at the call site:
subagent → agent_info.model · root → turn model override"] A --> T["every model call of the thread: tfg.thread_id (+ tfg.parent_thread_id)"] ``` @@ -70,7 +69,7 @@ interface RouteModelInput { scope: RoutingScope; threadId: string; // the thread being routed (child thread id, or root thread id) parentThreadId?: string; // subagent only - referenceModel: string; // model the thread would use without routing (account/model) + fallbackModel: string; // model the thread would use without routing (account/model) work: { // what Jev classifies name?: string; // subagent name; root: agent name text: string; // subagent: delegated task; root: latest user message @@ -83,7 +82,8 @@ interface RouteModelInput { } interface RouteModelResult { - model?: string; // set only in "on" mode when a cheaper model was picked + model?: string; // set only in "on" mode when a candidate was picked + traceMetadata: Record; // tfg.routing_* keys (§6), stamped on every model call of the thread } interface ITurnResourceResolver { @@ -96,11 +96,17 @@ interface ITurnResourceResolver { | Scope | Where | Builds `work` from | Applies the result | When | |---|---|---|---|---| -| `subagent` | `SessionHandle.makeCreateDynamicSubAgentThread` | `create_sub_agent` name and input, the turn's user message, the subagent's tools | `request.model = result.model`, so `resolveAgentDefinition` picks it up. It's persisted on `agent_info.model` through `THREAD_CREATED`. | Now | +| `subagent` | `SessionHandle.makeCreateDynamicSubAgentThread` | `create_sub_agent` name and input, the turn's user message, the subagent's tools | `request.model = result.model`, so `resolveAgentDefinition` picks it up. It's persisted on `agent_info.model` through `THREAD_CREATED`. `result.traceMetadata` goes on the child's definition, for its model calls. | Now | | `root` | Where `SessionHandle` builds the root thread's definition for a turn (`resolveAgentDefinition` without `agent_info`) | The turn's user message, a clipped excerpt of recent conversation, the agent's tools | Pass as a model override into the root `resolveAgentDefinition`, and persist it with the turn so approval or user-input resumes keep the same model (§8) | Later | Resume and rebuild never call the router. A subagent is rebuilt from its saved `agent_info`. A resumed root turn reuses the model stored for that turn. +**Reusing the existing model-override path.** `create_sub_agent` already supports a per-subagent model. With `modelSetConfig`, the parent LLM picks from a list, and the pick travels as `request.model`, then `agent_info.model`, then `getModel()` in `resolveAgentDefinition`. Nothing wires `modelSetConfig` today, so routing reuses only the second half: the router fills `request.model`, and the existing path resolves and persists it. Two gaps in that path are fixed here, and they'd apply to `modelSetConfig` too: +- **Params:** `resolveAgentDefinition` always merges the parent spec's `model.params`, so a child runs with the parent's effort. Even `ModelSetEntry.model_params` is ignored today. This is the params fix from `lld.md` §5. +- **Persistence:** `THREAD_CREATED.agent_info` is rebuilt from the original tool call, so a model set after the call isn't saved (§9). + +If `request.model` is already set, routing doesn't run. + ## 2. Hard-coded tables **The score is the Artificial Analysis Intelligence Index, v5,** on a 0–100 scale. It's the closest single score to agentic work: seven of its nine evaluations use tools, all run in Artificial Analysis's open-source agent harness, Stirrup. It has four equal categories: @@ -131,8 +137,8 @@ The General quarter isn't tool use, and that's accepted for now. - **Lookup at request time** uses the effort the thread will actually run at: - **For a candidate,** that's its default effort from TrueFoundry's metadata. - - **For the reference,** it's `spec.model.params.reasoning_effort` if set, otherwise its default. So an Opus root running at high is compared at 51.8, not 48.2. - - **The fallback order** is the exact effort, then `default`, then the nearest effort with a score, with ties going to the higher one. + - **The fallback model needs no score.** It's never shown to Jev. + - **If that effort has no score,** use `default`, then the nearest effort with a score, with ties going to the higher one. Scores looked up at each model's default effort, with the variant used and the TrueFoundry default: @@ -156,21 +162,20 @@ Scores looked up at each model's default effort, with the variant used and the T | `kimi/kimi-k3` | 17.4 | low (max) | | `google/gemini-3-5-flash-lite` | 11.7 | unlabelled (minimal) | -That's 35 of devtest's 465 distinct models, and 17 of the 49 pool models. Models used on devtest that have no v5 score in any variant include Sonnet 4.6 (the most common devtest root), the Opus 4.x models, GLM 5.2 and Gemini 3.6 and 3.7 Flash. A model not in the table is never a candidate, and a reference model not in the table isn't routed (§5). +That's 35 of devtest's 465 distinct models, and 17 of the 49 pool models. Models used on devtest that have no v5 score in any variant include Sonnet 4.6 (the most common devtest root), the Opus 4.x models, GLM 5.2 and Gemini 3.6 and 3.7 Flash. A model not in the table is never a candidate. The fallback model doesn't need to be in the table, since it isn't shown to Jev, so a Sonnet 4.6 parent's subagents can still be routed to scored candidates. **Candidates are built per call from the caller's model list,** through an internal `listRoutingCandidates()` on `TrueFoundryModelProviderStore`. - **It shares `listModels`' fetch and mapping:** the same `provider-integrations` call, with the caller's token and the tenant's account allow-list. - **Only the internal `TrueFoundryEnabledModel` type is widened,** with `canonical_model`, price, features, status, retirement date and default effort. - **`listModels` and the public `AvailableModel` schema are unchanged.** -- **It's called once per turn and cached.** +- **It's called once per turn and cached for that turn only,** shared by all subagents in the turn. Nothing is cached across turns. +- **Revoked access:** the picked model is resolved through the existing `getModel` path before the thread runs. The first use of a model in a turn fetches its provider fresh, so access revoked before the pick fails that check and the thread runs on the fallback model (`model_unavailable`). Access revoked mid-turn, after resolution, behaves as it does today for any model: the gateway rejects the call. A model is a candidate when it's: - an active chat model with tool calling, not retiring within 30 days, -- reasoning-capable, if the reference reasons, -- in the model table, -- and cheaper than the reference, using the blended price of 0.8 × input plus 0.2 × output. +- and in the model table. -If several accounts serve the same model, it's asked about once, preferring the reference's account, then its provider, then the first by name. There's no fixed limit. On devtest, Jev took 0.9, 0.9, 1.2 and 1.2 seconds for 6, 20, 40 and 60 candidate questions plus 7 benchmark questions. +Candidates are ordered by the blended price of 0.8 × input plus 0.2 × output. If several accounts serve the same model, it's asked about once, preferring the fallback model's account, then its provider, then the first by name. There's no fixed limit. On devtest, Jev took 0.9, 0.9, 1.2 and 1.2 seconds for 6, 20, 40 and 60 candidate questions plus 7 benchmark questions. **Benchmarks for the resemblance questions.** These are only benchmarks that `model-benchmarks` has data for, and that cover all the models we route between today (Opus 5.5, Sonnet 5.5, Haiku 5.5, GPT-6.1 Sol, GPT-6 Sol, GPT-6 Luna, Gemini 4, Gemini 3.8 Flash and Grok 4.7). So whichever one our tasks resemble, we can switch the score table to it straight away. The descriptions and examples are our own paraphrases. @@ -186,7 +191,7 @@ These were left out because the repo covers them too thinly, or not for our key ## 3. The Jev request -One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway URL and token borrowed from the reference model's provider config, as in `lld.md`. A reference model not in the table skips the call. +One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway URL and token borrowed from the fallback model's provider config, as in `lld.md`. If no candidate qualifies, there's no call. ```json { @@ -197,19 +202,18 @@ One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway "task": "", "context": "", "tools": ["", "web_search", "sandbox"], - "reference_model": { "name": "Claude Opus 5.5", "intelligence_index": 48.2 }, "benchmark_note": "Artificial Analysis Intelligence Index (0-100, higher is better) averages agentic knowledge work, coding, science and general reasoning evaluations." }, "questions": { "can_0": { "type": "noul", - "instructions": "Can Claude Haiku 5.5 at medium effort (Intelligence Index 24.6) complete this task fully and correctly, about as well as the reference model (Intelligence Index 48.2)?", + "instructions": "Can Claude Haiku 5.5 at medium effort (Intelligence Index 24.6) complete this task fully and correctly?", "criteria": { "true": "It would produce a result that can be used as-is: complete, accurate, and with the needed tool use done.", "false": "It would likely miss parts, make errors, or stop before the task is done." } }, - "can_1": { "type": "noul", "instructions": "Can Claude Sonnet 5.5 (Intelligence Index 41.5) complete ...", "criteria": { "...": "same as above" } }, + "can_1": { "type": "noul", "instructions": "Can Claude Sonnet 5.5 at high effort (Intelligence Index 41.5) complete this task fully and correctly?", "criteria": { "...": "same as above" } }, "resembles_automation": { "type": "noul", "instructions": "Does this task resemble the benchmark AutomationBench? Multi-step workflows across apps through API tools: look things up, combine results, act on them. Example: 'When a support ticket is tagged urgent, look up the customer's plan and notify the account owner.'", @@ -219,7 +223,14 @@ One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway } ``` -- **The wording is scope-neutral,** with "task" and "reference model", so the root call site sends the same shape with `scope: "root"`. +- **Jev sees a task and a pool of candidates, nothing else.** There's no parent model, no price and no comparison in the request. Each candidate is judged on its own: "can this model do this task?". +- **The wording is scope-neutral,** so the root call site sends the same shape with `scope: "root"`. +- **Offline test** on the 49 PoC subagent tasks × 11 candidates (an offline script, not part of this change), checked against each model's real outcome on the task: + - **Jev doesn't say yes to everything.** It's cautious: average yes-probability per model was 0.17–0.42, and 1 answer in 539 reached 0.7. + - **It ranks models sensibly on average.** Models that pass more often get higher answers (rank correlation 0.53). + - **On a single task, it only weakly tells whether that model will do well:** 0.6, where 0.5 is a coin flip and 1.0 is perfect. + - **An earlier wording** showed Jev the parent model ("about as well as the reference, Intelligence Index 48.2"). Jev then mostly compared the two numbers: rank correlation with the score 0.93, and with how often models actually passed only 0.25. That's why the parent model isn't in the request. + - **The number of questions doesn't matter.** The same models asked alongside 3, 11 or 16 candidates changed by 0.01 on average, and every call took about 0.7 s. - **Candidate questions** are keyed `can_`, and the router maps them back to models. - **All five benchmark questions** go in every call. - **One call,** with a 2-second timeout and `max_candidates` (default 60) as a safety cap. @@ -227,21 +238,20 @@ One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway ## 4. Picking the model 1. **Order:** candidates sorted by price, cheapest first. On a price tie, the higher score goes first. -2. **Pick** the first candidate with `noul >= threshold`. The default threshold is 0.7. -3. **Fall back to the reference model:** +2. **Pick** the first candidate with `noul >= threshold`. The threshold is a config value, starting at 0.4. In the offline test, with an Opus 5.5 parent: + - **At 0.4,** 29 of 49 tasks went to cheaper models. The same number of tasks were done correctly as when everything stayed on Opus, at 74% of the cost. + - **At 0.7,** almost nothing was routed. + + Shadow mode and user feedback tune it from there. +3. **Fall back to the parent's model** (the fallback model): - when no candidate reaches the threshold (`no_candidate_confident`), - - when the reference isn't in the table (`reference_not_in_table`), + - when no candidate qualifies at all (`no_candidates`), - when Jev fails or times out (`decision_error` / `decision_timeout`), - or when the pick can't be resolved for the caller (`model_unavailable`, using the existing `getModel` check). -4. **Never pick a more expensive model.** The limit is price, not quality: a candidate must be cheaper than the reference, but it may score higher. For example, Gemini 4 (50.4) is a candidate for an Opus 5.5 reference (48.2 at medium). Moving to a pricier model, even a better one, isn't done, because the goal is saving cost and the reference's price is what the user signed up for. -5. **If a model is already set, skip routing.** - - **For a subagent,** that's `agent_info.model`. It can only be set today if the agent spec defines a model set for `create_sub_agent` (`modelSetConfig`), which nothing does yet. - - **For the root,** it's a model the user picked explicitly (§8). +4. **No price ceiling.** The pick can cost more than the parent's model. Because the rule takes the cheapest confident candidate, a pricier model is picked only when Jev isn't confident about anything cheaper. The log line records whether each pick went up or down in price (§7), so shadow mode shows how often each happens. +5. **Routed threads use their own model's default params,** not the parent's (the fix from `lld.md` §5). - A deliberate choice is never overridden. -6. **Routed threads use their own model's default params,** not the reference's (the fix from `lld.md` §5). - -**Example:** a subagent whose reference is Opus 5.5 (48.2). In price order, Jev answers Haiku 5.5 0.42, GPT-6 Luna 0.35, Gemini 3.8 Flash 0.30, …, Grok 4.7 0.81, …, Sonnet 5.5 0.88. The first answer at or above 0.7 is Grok 4.7 ($2.80), so the subagent runs on Grok 4.7. +**Example,** with illustrative answers in the range Jev gave in the offline test: a subagent of a Haiku 5.5 parent. Every scored model the caller can use is asked about. In price order, Jev answers Kimi K3 0.17, GPT-6 Luna 0.30, Gemini 3.8 Flash 0.32, Haiku 5.5 0.35, Grok 4.6 0.33, Grok 4.7 0.38, …, Sonnet 5.5 0.46, …. The first answer at or above 0.4 is Sonnet 5.5, so the subagent runs on Sonnet 5.5, a pricier model than its parent. If no answer had reached 0.4, it would run on Haiku 5.5. ## 5. Modes and config @@ -251,7 +261,7 @@ Modes are set per tenant **and per scope**, so root routing can be turned on sep TRUEFOUNDRY_MODEL_ROUTING = { "decision_model": "archit-typesafe/jev-1.13.0", "timeout_ms": 2000, - "threshold": 0.7, + "threshold": 0.4, "max_candidates": 60, "tenants": { "": { "subagent": "shadow", "root": "off" } @@ -260,10 +270,10 @@ TRUEFOUNDRY_MODEL_ROUTING = { ``` - **off** (the default, or when unset): no call. The hook isn't wired for that scope. -- **shadow:** ask Jev and log the decision, but run on the reference model. +- **shadow:** ask Jev and log the decision, but run on the fallback model. - **on:** run on the pick. -## 6. Tracing: `tfg.thread_id` +## 6. Tracing: `tfg.thread_id` and the routing result TrueForge's own `AgentTracing` is a no-op by default (`createTracing()` returns `NOOP_AGENT_TRACING`). The traces we can see are the gateway's request traces, which record each call's `x-tfy-metadata`. The turn-level keys are there (`tfg.session_id`, `tfg.turn_id`, `tfg.agent_id`, `tfg.agent_name`), but nothing identifies the thread. - **Every thread's model calls:** in `AgentThread`, the model call `modelClient.create(requestBody)` gets `{ metadata: { 'tfg.thread_id': this.threadId, ...(parent ? { 'tfg.parent_thread_id': parent.thread_id } : {}) } }`. @@ -271,20 +281,24 @@ TrueForge's own `AgentTracing` is a no-op by default (`createTracing()` returns - It covers the root thread and subagents alike, so root routing needs nothing extra here. - Context compaction merges the same keys alongside `tfg.compaction`. - **The Jev call** carries the turn metadata, plus `tfg.thread_id` (the thread being routed), `tfg.routing_scope` and `tfg.model_routing: "true"`. +- **The routing result on the routed thread's model calls.** The router returns `traceMetadata`, which goes on the child's definition. `AgentThread` merges it into the same `metadata` as `tfg.thread_id`, on every model call of that thread: + - `tfg.routing_mode`: `shadow` or `on`. + - `tfg.routing_reason`: `picked`, `no_candidate_confident`, `no_candidates`, `decision_error`, `decision_timeout` or `model_unavailable`. + - `tfg.routing_pick`: the model the router chose. In shadow mode, that's the one it would have used. + - `tfg.routing_price`: `up`, `down` or `same`, the pick's price against the fallback model's. + + These are on every call, not just the first, so any single trace row says how its thread was routed. A thread rebuilt after a resume doesn't call the router, so its later calls carry `tfg.thread_id` but not the routing keys. The earlier calls still have them. - **Joining on `tfg.thread_id`** links the Jev request and response with every model call the thread made (§7). ## 7. What's observable, and the one log line -Most of the decision is already in the gateway traces, joined by `tfg.thread_id`: +Everything about a routing decision is in the gateway traces, joined by `tfg.thread_id`: - **The Jev request:** every candidate, with its score, and the benchmark questions. - **The Jev response:** the probability for each one, including the benchmark-resemblance answers. +- **The router's result:** the `tfg.routing_*` keys on the thread's model calls. They also cover decisions where Jev wasn't called (`no_candidates`), and shadow-mode picks that weren't applied. - **Every model call the thread made:** the served model, tokens, cost and latency. -Two things never reach a trace: -- **Decisions where Jev isn't called:** the reference isn't in the table, the model was explicit, or the scope is off. -- **The router's own result:** which candidate it picked, why it fell back, and in shadow mode the pick it didn't apply. - -For those, the router writes one plain `logger.info('model routing', { scope, thread_id, mode, reference_model, picked_model, chosen_model, reason })`. It's for our own understanding, not a schema or a contract, and never includes task text. +The router also writes one plain `logger.info('model routing', { scope, thread_id, mode, fallback_model, picked_model, chosen_model, price_direction, reason })`, with the same values as the trace keys. It's for reading server logs while debugging. The traces are the place to analyse routing. It's not a schema or a contract, and never includes task text. User feedback is joined by `thread_id`. The benchmark-resemblance counts are read from the Jev responses in the traces. @@ -292,10 +306,10 @@ User feedback is joined by `thread_id`. The benchmark-resemblance counts are rea The router, tables, Jev request, pick rule, config and tracing are all shared. What the root needs on top: -- **A call site:** where `SessionHandle` builds the root thread's definition for a turn. It builds `RouteModelInput` with `scope: 'root'` and the spec's model as the reference, and passes the result as a model override into the root `resolveAgentDefinition`. That's an optional `modelOverride` input, alongside `agent_info`. +- **A call site:** where `SessionHandle` builds the root thread's definition for a turn. It builds `RouteModelInput` with `scope: 'root'` and the spec's model as the fallback, and passes the result as a model override into the root `resolveAgentDefinition`. That's an optional `modelOverride` input, alongside `agent_info`. - **Granularity: once per turn,** keeping the model for the whole turn. Re-deciding mid-turn would change model during an agent loop, which the Artemis baseline showed hurts. A session-level sticky option (keep the first turn's choice) can come later, if prompt caching matters more than per-turn savings. - **Persistence:** store the routed model with the turn, as a turn-snapshot field, so a resumed turn (after an approval or user input) keeps the same model and never re-classifies. It's the only new stored field, and it's needed only for root. -- **Opt-in:** route the root only when the agent or tenant opts in. For example, a spec setting like `model: { name: , auto: true }`, or the per-scope tenant mode. A model the user picked explicitly is never overridden. +- **Opt-in:** route the root only when the agent or tenant opts in. For example, a spec setting like `model: { name: , auto: true }`, or the per-scope tenant mode. A model the user picked explicitly is never overridden. - **State:** the latest user message plus a clipped excerpt of recent conversation, instead of a delegated task. ## 9. Code changes (subagent scope now) @@ -306,9 +320,10 @@ The router, tables, Jev request, pick rule, config and tracing are all shared. W |---|---| | `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult`, and the optional hook `routeModel?(input)` (§1). | | `agent-session/TurnResourceResolver.ts` | Forward the hook from deps, and pass `isModelAvailable` (built on `getModel`). Plus the params fix from `lld.md` §5. | -| `agent-session/SessionHandle.ts` `makeCreateDynamicSubAgentThread` | Build `RouteModelInput` (`scope: 'subagent'`) and call `routeModel`. If `model` comes back, use `{ ...request, model }` from then on. | +| `agent-session/SessionHandle.ts` `makeCreateDynamicSubAgentThread` | If `request.model` is unset, build `RouteModelInput` (`scope: 'subagent'`) and call `routeModel`. If `model` comes back, use `{ ...request, model }` from then on, through the existing override path (§1). Put `traceMetadata` on the child definition. | +| `core/runtime/AgentDefinition.ts` | An optional `callMetadata: Record`, merged into every model call's `metadata`. | | `core/runtime/AgentThreadOrchestrator.ts` (`AGENT_CREATE_SUBAGENT`) | Build `THREAD_CREATED.agent_info` from `subAgentThread.agentInfo`, so the routed model is persisted. | -| `core/runtime/AgentThread.ts` | Pass `tfg.thread_id` and `tfg.parent_thread_id` metadata on every model call (§6). | +| `core/runtime/AgentThread.ts` | Pass `tfg.thread_id`, `tfg.parent_thread_id` and the definition's `callMetadata` on every model call (§6). | **Server (`packages/trueforge`):** @@ -320,7 +335,7 @@ The router, tables, Jev request, pick rule, config and tracing are all shared. W | **New** `truefoundry/modelRoutingScores.json` | The score table, nested by effort level, with source, version and date (§2). It's generated by the refresh script. | | **New** `truefoundry/modelRoutingTables.ts` | Loads and validates the JSON at startup into an in-memory map, does the lookup by effort with its fallback order, and holds the five benchmark descriptions (§2). | | **New** `benchmark/refresh_model_scores.py` (offline) | Calls the Artificial Analysis API, matches models by slug, and writes `modelRoutingScores.json`. | -| **New** `truefoundry/modelRouter.ts` | `createModelRouter()`, which returns `routeModel`. It's scope-agnostic: candidates (§2), the Jev request (§3), the pick (§4), one info log line (§7). It never throws. | +| **New** `truefoundry/modelRouter.ts` | `createModelRouter()`, which returns `routeModel`. It's scope-agnostic: candidates (§2), the Jev request (§3), the pick (§4), the `tfg.routing_*` trace keys (§6) and one info log line (§7). It never throws. | | `apis/turns.ts` | Wire `routeModel` when any scope for the tenant isn't off. The router checks the per-scope mode itself. | The subagent scope needs no database schema or public API changes, since `agent_info.model` already exists. The root scope adds one turn-snapshot field (§8). @@ -328,16 +343,17 @@ The subagent scope needs no database schema or public API changes, since `agent_ ## 10. Rollout 1. Merge with the config unset, so nothing changes. -2. Run subagent scope in shadow mode for two or three devtest tenants. Check latency, the fallback reasons, the pick distribution and `top_benchmark`. +2. Run subagent scope in shadow mode for two or three devtest tenants. Check latency, the fallback reasons, the pick distribution, how often picks go up or down in price, and `top_benchmark`. 3. Switch subagent scope to "on" for those tenants. Measure with user feedback, joined by `thread_id`. 4. Add the root call site (§8), and run root scope in shadow mode first. 5. Use the resemblance data to choose the evaluation benchmark, and replace the score table if a narrower one fits better. ## 11. Open questions for review -1. **Unscored models in use:** Sonnet 4.6, the Opus 4.x models and GLM 5.2 have no v5 score. Should they get a hand-assigned tier so they can be candidates or references, or stay unrouted? -2. **Pick rule:** the cheapest candidate with P(yes) ≥ 0.7, or the highest P(yes) among cheaper candidates? -3. **Model names in the questions:** name them, as in option 1 of the picture, or say only "a model with Intelligence Index S" to avoid Jev's priors about names? -4. **Benchmark list:** are these five right? Should there also be a single "closest benchmark" choice question? -5. **Root scope:** routing once per turn or once per session, and how a user or agent opts in to an automatically selected model. -6. **Traces:** is the gateway request trace with `x-tfy-metadata` enough, or should a real TrueForge tracer be wired up as well? +1. **Unscored models in use:** Sonnet 4.6, the Opus 4.x models and GLM 5.2 have no v5 score. As fallback models they're fine, since they aren't shown to Jev. As candidates they're never picked. Should they get a hand-assigned tier so they can be candidates, or stay out of the candidate pool? +2. **Pick rule:** the cheapest candidate with P(yes) ≥ 0.4, or the highest P(yes) overall? +3. **Price ceiling, later:** none in the first version. If shadow or "on" data shows routing up in price costs too much for too little, add a per-tenant cap as a multiple of the fallback model's price. +4. **Model names in the questions:** name them, as in option 1 of the picture, or say only "a model with Intelligence Index S" to avoid Jev's priors about names? +5. **Benchmark list:** are these five right? Should there also be a single "closest benchmark" choice question? +6. **Root scope:** routing once per turn or once per session, and how a user or agent opts in to an automatically selected model. +7. **Traces:** is the gateway request trace with `x-tfy-metadata` enough, or should a real TrueForge tracer be wired up as well? From 909e4039e8ac6bbe5078e17a1f1263f6200a03b8 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 09:51:50 +0530 Subject: [PATCH 03/14] docs: keep routing result in the log line; traces carry only tfg.thread_id Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 32 +++++++++++++------------------ 1 file changed, 13 insertions(+), 19 deletions(-) diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md index d8b467822..db863878f 100644 --- a/docs/subagent-auto-routing/lld.md +++ b/docs/subagent-auto-routing/lld.md @@ -22,7 +22,7 @@ From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. | I8 | **The router never throws.** Any error, timeout or unusable pick means the fallback model runs, as today. | §4 | | I9 | **The routed model is applied through the existing `request.model` path** (`create_sub_agent` request, then `agent_info.model`, then `resolveAgentDefinition`). Only the chosen model is persisted, on `agent_info.model`. Resume and rebuild never call the router. | §1 | | I10 | **One generic core hook,** `routeModel(RouteModelInput) → RouteModelResult`, with a `scope`. All routing logic lives in the server. | §1, §9 | -| I11 | **Observability is in the gateway traces.** `tfg.thread_id` is on every model call and on the Jev call. The routing result (`tfg.routing_*`) is on every model call of the routed thread. One info log line repeats the decision server-side. No new tables or public API for the subagent scope. | §6, §7 | +| I11 | **Observability comes from gateway traces,** with `tfg.thread_id` on every model call and on the Jev call, plus one info log line. No new tables or public API for the subagent scope. | §6, §7 | | I12 | **Mode per tenant and per scope:** off, shadow or on. Off is the default. | §5 | **Tunable without a design change,** as config or data: @@ -52,7 +52,7 @@ flowchart TD R -- not wired --> K["keep the model the thread would use today"] R -- wired --> J["Jev: POST {gateway}/v1/systemone
candidate + benchmark questions
x-tfy-metadata: tfg.thread_id, tfg.routing_scope"] J --> P["pick: cheapest candidate with P(yes) >= threshold, else fallback model"] - P --> L["tfg.routing_* trace keys + one info log line"] + P --> L["one info log line (scope, thread_id, pick, reason)"] L --> A["apply at the call site:
subagent → agent_info.model · root → turn model override"] A --> T["every model call of the thread: tfg.thread_id (+ tfg.parent_thread_id)"] ``` @@ -83,7 +83,6 @@ interface RouteModelInput { interface RouteModelResult { model?: string; // set only in "on" mode when a candidate was picked - traceMetadata: Record; // tfg.routing_* keys (§6), stamped on every model call of the thread } interface ITurnResourceResolver { @@ -96,7 +95,7 @@ interface ITurnResourceResolver { | Scope | Where | Builds `work` from | Applies the result | When | |---|---|---|---|---| -| `subagent` | `SessionHandle.makeCreateDynamicSubAgentThread` | `create_sub_agent` name and input, the turn's user message, the subagent's tools | `request.model = result.model`, so `resolveAgentDefinition` picks it up. It's persisted on `agent_info.model` through `THREAD_CREATED`. `result.traceMetadata` goes on the child's definition, for its model calls. | Now | +| `subagent` | `SessionHandle.makeCreateDynamicSubAgentThread` | `create_sub_agent` name and input, the turn's user message, the subagent's tools | `request.model = result.model`, so `resolveAgentDefinition` picks it up. It's persisted on `agent_info.model` through `THREAD_CREATED`. | Now | | `root` | Where `SessionHandle` builds the root thread's definition for a turn (`resolveAgentDefinition` without `agent_info`) | The turn's user message, a clipped excerpt of recent conversation, the agent's tools | Pass as a model override into the root `resolveAgentDefinition`, and persist it with the turn so approval or user-input resumes keep the same model (§8) | Later | Resume and rebuild never call the router. A subagent is rebuilt from its saved `agent_info`. A resumed root turn reuses the model stored for that turn. @@ -273,7 +272,7 @@ TRUEFOUNDRY_MODEL_ROUTING = { - **shadow:** ask Jev and log the decision, but run on the fallback model. - **on:** run on the pick. -## 6. Tracing: `tfg.thread_id` and the routing result +## 6. Tracing: `tfg.thread_id` TrueForge's own `AgentTracing` is a no-op by default (`createTracing()` returns `NOOP_AGENT_TRACING`). The traces we can see are the gateway's request traces, which record each call's `x-tfy-metadata`. The turn-level keys are there (`tfg.session_id`, `tfg.turn_id`, `tfg.agent_id`, `tfg.agent_name`), but nothing identifies the thread. - **Every thread's model calls:** in `AgentThread`, the model call `modelClient.create(requestBody)` gets `{ metadata: { 'tfg.thread_id': this.threadId, ...(parent ? { 'tfg.parent_thread_id': parent.thread_id } : {}) } }`. @@ -281,24 +280,20 @@ TrueForge's own `AgentTracing` is a no-op by default (`createTracing()` returns - It covers the root thread and subagents alike, so root routing needs nothing extra here. - Context compaction merges the same keys alongside `tfg.compaction`. - **The Jev call** carries the turn metadata, plus `tfg.thread_id` (the thread being routed), `tfg.routing_scope` and `tfg.model_routing: "true"`. -- **The routing result on the routed thread's model calls.** The router returns `traceMetadata`, which goes on the child's definition. `AgentThread` merges it into the same `metadata` as `tfg.thread_id`, on every model call of that thread: - - `tfg.routing_mode`: `shadow` or `on`. - - `tfg.routing_reason`: `picked`, `no_candidate_confident`, `no_candidates`, `decision_error`, `decision_timeout` or `model_unavailable`. - - `tfg.routing_pick`: the model the router chose. In shadow mode, that's the one it would have used. - - `tfg.routing_price`: `up`, `down` or `same`, the pick's price against the fallback model's. - - These are on every call, not just the first, so any single trace row says how its thread was routed. A thread rebuilt after a resume doesn't call the router, so its later calls carry `tfg.thread_id` but not the routing keys. The earlier calls still have them. - **Joining on `tfg.thread_id`** links the Jev request and response with every model call the thread made (§7). ## 7. What's observable, and the one log line -Everything about a routing decision is in the gateway traces, joined by `tfg.thread_id`: +Most of the decision is already in the gateway traces, joined by `tfg.thread_id`: - **The Jev request:** every candidate, with its score, and the benchmark questions. - **The Jev response:** the probability for each one, including the benchmark-resemblance answers. -- **The router's result:** the `tfg.routing_*` keys on the thread's model calls. They also cover decisions where Jev wasn't called (`no_candidates`), and shadow-mode picks that weren't applied. - **Every model call the thread made:** the served model, tokens, cost and latency. -The router also writes one plain `logger.info('model routing', { scope, thread_id, mode, fallback_model, picked_model, chosen_model, price_direction, reason })`, with the same values as the trace keys. It's for reading server logs while debugging. The traces are the place to analyse routing. It's not a schema or a contract, and never includes task text. +Two things never reach a trace: +- **Decisions where Jev isn't called:** no candidate qualifies, or the scope is off. +- **The router's own result:** which candidate it picked, why it fell back, and in shadow mode the pick it didn't apply. + +For those, the router writes one plain `logger.info('model routing', { scope, thread_id, mode, fallback_model, picked_model, chosen_model, price_direction, reason })`. Here `price_direction` is `up`, `down` or `same`: the pick's price against the fallback model's. It's for our own understanding, not a schema or a contract, and never includes task text. User feedback is joined by `thread_id`. The benchmark-resemblance counts are read from the Jev responses in the traces. @@ -320,10 +315,9 @@ The router, tables, Jev request, pick rule, config and tracing are all shared. W |---|---| | `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult`, and the optional hook `routeModel?(input)` (§1). | | `agent-session/TurnResourceResolver.ts` | Forward the hook from deps, and pass `isModelAvailable` (built on `getModel`). Plus the params fix from `lld.md` §5. | -| `agent-session/SessionHandle.ts` `makeCreateDynamicSubAgentThread` | If `request.model` is unset, build `RouteModelInput` (`scope: 'subagent'`) and call `routeModel`. If `model` comes back, use `{ ...request, model }` from then on, through the existing override path (§1). Put `traceMetadata` on the child definition. | -| `core/runtime/AgentDefinition.ts` | An optional `callMetadata: Record`, merged into every model call's `metadata`. | +| `agent-session/SessionHandle.ts` `makeCreateDynamicSubAgentThread` | If `request.model` is unset, build `RouteModelInput` (`scope: 'subagent'`) and call `routeModel`. If `model` comes back, use `{ ...request, model }` from then on, through the existing override path (§1). | | `core/runtime/AgentThreadOrchestrator.ts` (`AGENT_CREATE_SUBAGENT`) | Build `THREAD_CREATED.agent_info` from `subAgentThread.agentInfo`, so the routed model is persisted. | -| `core/runtime/AgentThread.ts` | Pass `tfg.thread_id`, `tfg.parent_thread_id` and the definition's `callMetadata` on every model call (§6). | +| `core/runtime/AgentThread.ts` | Pass `tfg.thread_id` and `tfg.parent_thread_id` metadata on every model call (§6). | **Server (`packages/trueforge`):** @@ -335,7 +329,7 @@ The router, tables, Jev request, pick rule, config and tracing are all shared. W | **New** `truefoundry/modelRoutingScores.json` | The score table, nested by effort level, with source, version and date (§2). It's generated by the refresh script. | | **New** `truefoundry/modelRoutingTables.ts` | Loads and validates the JSON at startup into an in-memory map, does the lookup by effort with its fallback order, and holds the five benchmark descriptions (§2). | | **New** `benchmark/refresh_model_scores.py` (offline) | Calls the Artificial Analysis API, matches models by slug, and writes `modelRoutingScores.json`. | -| **New** `truefoundry/modelRouter.ts` | `createModelRouter()`, which returns `routeModel`. It's scope-agnostic: candidates (§2), the Jev request (§3), the pick (§4), the `tfg.routing_*` trace keys (§6) and one info log line (§7). It never throws. | +| **New** `truefoundry/modelRouter.ts` | `createModelRouter()`, which returns `routeModel`. It's scope-agnostic: candidates (§2), the Jev request (§3), the pick (§4), one info log line (§7). It never throws. | | `apis/turns.ts` | Wire `routeModel` when any scope for the tenant isn't off. The router checks the per-scope mode itself. | The subagent scope needs no database schema or public API changes, since `agent_info.model` already exists. The root scope adds one turn-snapshot field (§8). From febca1075015c60a7d2addce798a8e9364fbe041 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 10:39:25 +0530 Subject: [PATCH 04/14] docs: pick the candidate with the highest yes-probability; drop the threshold Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 45 +++++++++++++++---------------- 1 file changed, 22 insertions(+), 23 deletions(-) diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md index db863878f..a8c53fa44 100644 --- a/docs/subagent-auto-routing/lld.md +++ b/docs/subagent-auto-routing/lld.md @@ -14,7 +14,7 @@ From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. |---|---|---| | I1 | **One Jev call per routing decision, made before the thread starts.** Never during the agent loop. A subagent is routed once, and the root once per turn. | §1, §8 | | I2 | **The Jev payload shape:** `state` = `{ scope, name, task, context, tools }`; `questions` = one yes/no `can_` per candidate (model name, effort and score in the text) plus one yes/no `resembles_` per benchmark. **Jev sees only the task and a pool of candidates.** It's never told about the fallback model, prices, or any comparison. | §3 | -| I3 | **The pick rule shape:** candidates in price order, cheapest first; take the first whose yes-probability is at or above the threshold; otherwise run the fallback model. | §4 | +| I3 | **The pick rule:** the candidate with the highest yes-probability. On a tie, the cheaper one. There's no threshold. The fallback model runs only when routing can't produce a pick (no candidates, a Jev error or timeout, or an unusable pick). | §4 | | I4 | **The fallback model is the model the thread would use without routing:** the parent's model for a subagent, the spec's model for the root. It's used only when routing doesn't pick. It doesn't limit which candidates are asked about: **there's no price ceiling,** so a routed thread can run on a model that costs more than its parent. | §4 | | I5 | **Candidates come only from the caller's own model list** (the same `provider-integrations` fetch as `listModels`, no extra account filter), and only models in the score table. | §2 | | I6 | **Scores are bundled data,** generated offline from the Artificial Analysis API. There's no call to Artificial Analysis while a request runs. | §2 | @@ -26,7 +26,7 @@ From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. | I12 | **Mode per tenant and per scope:** off, shadow or on. Off is the default. | §5 | **Tunable without a design change,** as config or data: -- **Config:** the threshold (0.4), the Jev timeout (2 s), `max_candidates` (60), the decision model ID, and the mode for each tenant and scope. +- **Config:** the Jev timeout (2 s), `max_candidates` (60), the decision model ID, and the mode for each tenant and scope. - **Data:** the score table's contents and which benchmark it's based on (today the Intelligence Index v5), and the benchmark-resemblance list (today five benchmarks). **Needs a reviewer's decision:** see §11. @@ -35,7 +35,7 @@ From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. Before a thread runs, the router asks Jev, the decision model, two things about the work. The thread is a subagent now, and later the root agent. -1. **Routing:** for each candidate model, "can this model complete the task?". Candidates are models the caller can use that are in the model table (§2). Each question carries the candidate's score. The thread runs on the cheapest candidate Jev says yes to. Otherwise it runs on the **fallback model**, the model it would use without routing: +1. **Routing:** for each candidate model, "can this model complete the task?". Candidates are models the caller can use that are in the model table (§2). Each question carries the candidate's score. The thread runs on the candidate with the highest yes-probability. If routing can't produce a pick, it runs on the **fallback model**, the model it would use without routing: - **For a subagent,** that's the parent's model. - **For the root,** it's the agent spec's model. @@ -51,7 +51,7 @@ flowchart TD R{"resolver.routeModel?
(wired if the scope's mode is shadow/on)"} R -- not wired --> K["keep the model the thread would use today"] R -- wired --> J["Jev: POST {gateway}/v1/systemone
candidate + benchmark questions
x-tfy-metadata: tfg.thread_id, tfg.routing_scope"] - J --> P["pick: cheapest candidate with P(yes) >= threshold, else fallback model"] + J --> P["pick: candidate with the highest P(yes), cheaper on a tie"] P --> L["one info log line (scope, thread_id, pick, reason)"] L --> A["apply at the call site:
subagent → agent_info.model · root → turn model override"] A --> T["every model call of the thread: tfg.thread_id (+ tfg.parent_thread_id)"] @@ -174,7 +174,7 @@ A model is a candidate when it's: - an active chat model with tool calling, not retiring within 30 days, - and in the model table. -Candidates are ordered by the blended price of 0.8 × input plus 0.2 × output. If several accounts serve the same model, it's asked about once, preferring the fallback model's account, then its provider, then the first by name. There's no fixed limit. On devtest, Jev took 0.9, 0.9, 1.2 and 1.2 seconds for 6, 20, 40 and 60 candidate questions plus 7 benchmark questions. +Price, the blended 0.8 × input plus 0.2 × output, is used only to break ties and for the log line's price direction. If several accounts serve the same model, it's asked about once, preferring the fallback model's account, then its provider, then the first by name. There's no fixed limit. On devtest, Jev took 0.9, 0.9, 1.2 and 1.2 seconds for 6, 20, 40 and 60 candidate questions plus 7 benchmark questions. **Benchmarks for the resemblance questions.** These are only benchmarks that `model-benchmarks` has data for, and that cover all the models we route between today (Opus 5.5, Sonnet 5.5, Haiku 5.5, GPT-6.1 Sol, GPT-6 Sol, GPT-6 Luna, Gemini 4, Gemini 3.8 Flash and Grok 4.7). So whichever one our tasks resemble, we can switch the score table to it straight away. The descriptions and examples are our own paraphrases. @@ -236,21 +236,22 @@ One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway ## 4. Picking the model -1. **Order:** candidates sorted by price, cheapest first. On a price tie, the higher score goes first. -2. **Pick** the first candidate with `noul >= threshold`. The threshold is a config value, starting at 0.4. In the offline test, with an Opus 5.5 parent: - - **At 0.4,** 29 of 49 tasks went to cheaper models. The same number of tasks were done correctly as when everything stayed on Opus, at 74% of the cost. - - **At 0.7,** almost nothing was routed. - - Shadow mode and user feedback tune it from there. -3. **Fall back to the parent's model** (the fallback model): - - when no candidate reaches the threshold (`no_candidate_confident`), +1. **Pick** the candidate with the highest `noul`. On a tie, the cheaper one wins, using the blended price (§2). There's no threshold. Only the order of Jev's answers matters, not their level, so there's nothing to tune. +2. **Fall back to the parent's model** (the fallback model) only when there's no usable pick: - when no candidate qualifies at all (`no_candidates`), - when Jev fails or times out (`decision_error` / `decision_timeout`), - or when the pick can't be resolved for the caller (`model_unavailable`, using the existing `getModel` check). -4. **No price ceiling.** The pick can cost more than the parent's model. Because the rule takes the cheapest confident candidate, a pricier model is picked only when Jev isn't confident about anything cheaper. The log line records whether each pick went up or down in price (§7), so shadow mode shows how often each happens. -5. **Routed threads use their own model's default params,** not the parent's (the fix from `lld.md` §5). +3. **No price ceiling.** The pick can cost more than the parent's model. The log line records whether each pick went up or down in price (§7), so shadow mode shows how often each happens. +4. **Routed threads use their own model's default params,** not the parent's (the fix from `lld.md` §5). + +**In the offline test** (49 tasks, the 11 candidates cheaper than Opus 5.5, compared against Opus): +- **Sonnet 5.5 was picked for 47 of 49 tasks.** Jev's highest answer was nearly always the same model, so on this pool the rule behaves almost like a fixed pick. +- **As good as Opus on 76% of tasks, at 53% of Opus's cost.** +- **The gap between the top two answers was small:** a median of 0.05, and under 0.02 on 20% of tasks. + +Opus and the models priced above it weren't candidates in that test. With no ceiling they will be. Since Jev's answers rise with the score, the top-scoring model the caller can use may be picked for most tasks. Shadow mode should check the pick distribution and the price direction before switching to "on". -**Example,** with illustrative answers in the range Jev gave in the offline test: a subagent of a Haiku 5.5 parent. Every scored model the caller can use is asked about. In price order, Jev answers Kimi K3 0.17, GPT-6 Luna 0.30, Gemini 3.8 Flash 0.32, Haiku 5.5 0.35, Grok 4.6 0.33, Grok 4.7 0.38, …, Sonnet 5.5 0.46, …. The first answer at or above 0.4 is Sonnet 5.5, so the subagent runs on Sonnet 5.5, a pricier model than its parent. If no answer had reached 0.4, it would run on Haiku 5.5. +**Example,** with illustrative answers: a subagent of a Haiku 5.5 parent. Every scored model the caller can use is asked about. Jev answers Kimi K3 0.17, GPT-6 Luna 0.30, Haiku 5.5 0.35, Grok 4.7 0.38, GPT-6 Sol 0.41, Sonnet 5.5 0.46. The highest is Sonnet 5.5, so the subagent runs on Sonnet 5.5, a pricier model than its parent. ## 5. Modes and config @@ -260,7 +261,6 @@ Modes are set per tenant **and per scope**, so root routing can be turned on sep TRUEFOUNDRY_MODEL_ROUTING = { "decision_model": "archit-typesafe/jev-1.13.0", "timeout_ms": 2000, - "threshold": 0.4, "max_candidates": 60, "tenants": { "": { "subagent": "shadow", "root": "off" } @@ -345,9 +345,8 @@ The subagent scope needs no database schema or public API changes, since `agent_ ## 11. Open questions for review 1. **Unscored models in use:** Sonnet 4.6, the Opus 4.x models and GLM 5.2 have no v5 score. As fallback models they're fine, since they aren't shown to Jev. As candidates they're never picked. Should they get a hand-assigned tier so they can be candidates, or stay out of the candidate pool? -2. **Pick rule:** the cheapest candidate with P(yes) ≥ 0.4, or the highest P(yes) overall? -3. **Price ceiling, later:** none in the first version. If shadow or "on" data shows routing up in price costs too much for too little, add a per-tenant cap as a multiple of the fallback model's price. -4. **Model names in the questions:** name them, as in option 1 of the picture, or say only "a model with Intelligence Index S" to avoid Jev's priors about names? -5. **Benchmark list:** are these five right? Should there also be a single "closest benchmark" choice question? -6. **Root scope:** routing once per turn or once per session, and how a user or agent opts in to an automatically selected model. -7. **Traces:** is the gateway request trace with `x-tfy-metadata` enough, or should a real TrueForge tracer be wired up as well? +2. **Price ceiling, later:** none in the first version. If shadow or "on" data shows the highest-probability pick goes up in price too often for too little, add a per-tenant cap as a multiple of the fallback model's price. +3. **Model names in the questions:** name them, as in option 1 of the picture, or say only "a model with Intelligence Index S" to avoid Jev's priors about names? +4. **Benchmark list:** are these five right? Should there also be a single "closest benchmark" choice question? +5. **Root scope:** routing once per turn or once per session, and how a user or agent opts in to an automatically selected model. +6. **Traces:** is the gateway request trace with `x-tfy-metadata` enough, or should a real TrueForge tracer be wired up as well? From 88b33790be969512066200c1616ee526ef3dfb55 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 14:06:07 +0530 Subject: [PATCH 05/14] docs: update subagent routing LLD for the internal PoC Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 419 +++++++++++++----------------- 1 file changed, 183 insertions(+), 236 deletions(-) diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md index a8c53fa44..de7c50178 100644 --- a/docs/subagent-auto-routing/lld.md +++ b/docs/subagent-auto-routing/lld.md @@ -1,10 +1,12 @@ -# Model routing with Jev: low-level design (draft for review) +# Subagent model routing with Jev: low-level design -Status: ready for review. Replaces the selection rule in `lld.md` (tier question plus same-creator price rule). The routing is built as one generic router with two call sites: -- **Subagent spawn,** implemented now. -- **The root agent's turn,** a planned extension that uses the same router, tables, Jev call and tracing. +Status: ready for review. Scope: a PoC on internal TrueFoundry tenants, in TrueFoundry mode only. -From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. +When a subagent is spawned, one Jev call picks the model it runs on. The router is generic (it takes a `scope`), so the root agent can use it later (§8). Its modules are split so that model routing can later become a capability that also works outside TrueFoundry mode (§9). Only the subagent scope is built now. + +**Related work, tracked separately:** +- **`tfg.thread_id` on every model call** (AGE-2837, PR #1392). Routing analysis joins on it. +- **Subagent model overrides** (AGE-2841). A subagent on a different model must use that model's own default params, not the parent's, and `THREAD_CREATED` must save the model the subagent actually runs on. Shadow mode doesn't need this. "On" mode does. ## Decisions at a glance @@ -12,54 +14,43 @@ From `lld.md`, the model-params fix and the `THREAD_CREATED` fix still apply. | # | Invariant | Where | |---|---|---| -| I1 | **One Jev call per routing decision, made before the thread starts.** Never during the agent loop. A subagent is routed once, and the root once per turn. | §1, §8 | -| I2 | **The Jev payload shape:** `state` = `{ scope, name, task, context, tools }`; `questions` = one yes/no `can_` per candidate (model name, effort and score in the text) plus one yes/no `resembles_` per benchmark. **Jev sees only the task and a pool of candidates.** It's never told about the fallback model, prices, or any comparison. | §3 | -| I3 | **The pick rule:** the candidate with the highest yes-probability. On a tie, the cheaper one. There's no threshold. The fallback model runs only when routing can't produce a pick (no candidates, a Jev error or timeout, or an unusable pick). | §4 | -| I4 | **The fallback model is the model the thread would use without routing:** the parent's model for a subagent, the spec's model for the root. It's used only when routing doesn't pick. It doesn't limit which candidates are asked about: **there's no price ceiling,** so a routed thread can run on a model that costs more than its parent. | §4 | -| I5 | **Candidates come only from the caller's own model list** (the same `provider-integrations` fetch as `listModels`, no extra account filter), and only models in the score table. | §2 | -| I6 | **Scores are bundled data,** generated offline from the Artificial Analysis API. There's no call to Artificial Analysis while a request runs. | §2 | -| I7 | **The score Jev sees matches how the model will actually run.** Artificial Analysis scores each effort level separately, so Jev gets the score at the candidate's default effort on TrueFoundry. A routed thread then runs at its own model's default params, not copied from the parent, so the score stays accurate. | §2, §4 | -| I8 | **The router never throws.** Any error, timeout or unusable pick means the fallback model runs, as today. | §4 | -| I9 | **The routed model is applied through the existing `request.model` path** (`create_sub_agent` request, then `agent_info.model`, then `resolveAgentDefinition`). Only the chosen model is persisted, on `agent_info.model`. Resume and rebuild never call the router. | §1 | -| I10 | **One generic core hook,** `routeModel(RouteModelInput) → RouteModelResult`, with a `scope`. All routing logic lives in the server. | §1, §9 | -| I11 | **Observability comes from gateway traces,** with `tfg.thread_id` on every model call and on the Jev call, plus one info log line. No new tables or public API for the subagent scope. | §6, §7 | -| I12 | **Mode per tenant and per scope:** off, shadow or on. Off is the default. | §5 | - -**Tunable without a design change,** as config or data: -- **Config:** the Jev timeout (2 s), `max_candidates` (60), the decision model ID, and the mode for each tenant and scope. -- **Data:** the score table's contents and which benchmark it's based on (today the Intelligence Index v5), and the benchmark-resemblance list (today five benchmarks). - -**Needs a reviewer's decision:** see §11. +| I1 | **One Jev call per subagent, made before it starts.** Never during the agent loop. | §1 | +| I2 | **Jev sees only the task and the candidate pool.** One yes/no question per candidate (name, effort, score), plus one per benchmark. No parent model, prices or comparisons. | §3 | +| I3 | **The pick is the candidate with the highest yes-probability,** the cheaper one on a tie. There's no threshold and no price ceiling yet. | §4 | +| I4 | **The fallback is the parent's model.** It runs only when routing can't produce a pick. | §4 | +| I5 | **Candidates are the caller's models that have a score.** No other metadata filters candidates. Other fields are read only for the log line. | §2 | +| I6 | **Scores are bundled data,** keyed by Artificial Analysis slug. No Artificial Analysis call while a request runs. | §2 | +| I7 | **Jev sees each candidate's score at the effort it will actually run at:** its default effort on TrueFoundry. | §2 | +| I8 | **The router never throws.** Any error, timeout or unusable pick means the parent's model runs, as today. | §4 | +| I9 | **The routed model is applied through the existing `request.model` path** and saved on `agent_info.model`. Resume and rebuild never call the router. | §1 | +| I10 | **One optional core hook,** `routeModel`. The router's logic lives in the server, split into provider-neutral and TrueFoundry-specific modules. | §1, §9 | +| I11 | **Observability:** gateway traces joined on `tfg.thread_id`, plus one info log line per decision. No new tables or public API. | §6 | +| I12 | **Mode per tenant and per scope:** off, shadow or on. Off by default. | §5 | + +**Tunable without a design change:** +- **Config:** the decision model, the Jev timeout (2 s), `max_candidates` (60), and each tenant's mode. +- **Data:** the score table, and the benchmark-resemblance list. ## What it does -Before a thread runs, the router asks Jev, the decision model, two things about the work. The thread is a subagent now, and later the root agent. - -1. **Routing:** for each candidate model, "can this model complete the task?". Candidates are models the caller can use that are in the model table (§2). Each question carries the candidate's score. The thread runs on the candidate with the highest yes-probability. If routing can't produce a pick, it runs on the **fallback model**, the model it would use without routing: - - **For a subagent,** that's the parent's model. - - **For the root,** it's the agent spec's model. - - There's no price ceiling. A subagent of a cheap parent can be routed to a stronger, pricier model, and a subagent of an expensive parent to a cheaper one. This is deliberate for the first version: both directions give us data on how the router behaves. A ceiling can be added later as config (§11). -2. **Benchmark resemblance:** for each benchmark on a hard-coded list, "does this task resemble the benchmark?". This is recorded only. Over real traffic, it tells us which benchmark our work actually resembles, which is the one to use for evaluation later. - -Every model call carries `tfg.thread_id` in `x-tfy-metadata`, and so does the Jev call. The gateway traces then show which thread each call belongs to and how it was routed. +1. **The parent calls `create_sub_agent`** with a name and a task. +2. **The router builds the candidates:** the caller's models that have a score (§2). +3. **One Jev call** asks, for each candidate, "can this model complete this task fully and correctly?" (§3). It also asks whether the task resembles each of five public benchmarks. Those answers are only recorded, to learn later which benchmark our work looks like. +4. **The router picks** the candidate with the highest yes-probability (§4). In shadow mode it logs the pick and runs the parent's model. In "on" mode the subagent runs on the pick. +5. **If anything fails,** the subagent runs on the parent's model. ```mermaid flowchart TD - S1["Subagent spawn
(makeCreateDynamicSubAgentThread)"] --> R - S2["Root turn start (later)
(root thread build in SessionHandle)"] -.-> R - R{"resolver.routeModel?
(wired if the scope's mode is shadow/on)"} - R -- not wired --> K["keep the model the thread would use today"] - R -- wired --> J["Jev: POST {gateway}/v1/systemone
candidate + benchmark questions
x-tfy-metadata: tfg.thread_id, tfg.routing_scope"] - J --> P["pick: candidate with the highest P(yes), cheaper on a tie"] - P --> L["one info log line (scope, thread_id, pick, reason)"] - L --> A["apply at the call site:
subagent → agent_info.model · root → turn model override"] - A --> T["every model call of the thread: tfg.thread_id (+ tfg.parent_thread_id)"] + S["create_sub_agent call
(makeCreateDynamicSubAgentThread)"] --> R{"routeModel wired?
(tenant's subagent mode is shadow/on)"} + R -- no --> K["parent's model, as today"] + R -- yes --> C["candidates: caller's models with a score"] + C --> J["Jev: POST {gateway}/v1/systemone
one question per candidate + 5 benchmark questions"] + J --> P["pick: highest P(yes), cheaper on a tie"] + P --> L["one info log line"] + L --> A["on: request.model = pick · shadow: parent's model"] ``` -## 1. The router contract (generic) - -The router knows nothing about subagents or roots. Each call site builds one `RouteModelInput`, and applies the `RouteModelResult` its own way. +## 1. The router contract ```ts // trueforge-core: agent-session/ITurnResourceResolver.ts @@ -67,139 +58,100 @@ type RoutingScope = 'subagent' | 'root'; interface RouteModelInput { scope: RoutingScope; - threadId: string; // the thread being routed (child thread id, or root thread id) - parentThreadId?: string; // subagent only - fallbackModel: string; // model the thread would use without routing (account/model) - work: { // what Jev classifies - name?: string; // subagent name; root: agent name - text: string; // subagent: delegated task; root: latest user message - context?: string; // subagent: the user's request; root: short recent-conversation excerpt - tools: string[]; // MCP server names, web_search, sandbox + threadId: string; // the thread being routed + fallbackModel: string; // the parent's model (account/model) + work: { + name?: string; // subagent name + text: string; // the delegated task + context?: string; // the turn's user message + tools: string[]; // MCP server names, web_search, sandbox }; - spec: AgentSpec; signal: AbortSignal; - tracing: AgentTracing; } interface RouteModelResult { - model?: string; // set only in "on" mode when a candidate was picked + model?: string; // set only in "on" mode when a candidate was picked } interface ITurnResourceResolver { // ...existing methods - routeModel?(input: RouteModelInput): Promise; // never throws; undefined hook = today + routeModel?(input: RouteModelInput): Promise; // never throws; undefined = today } ``` -**Call sites:** - -| Scope | Where | Builds `work` from | Applies the result | When | -|---|---|---|---|---| -| `subagent` | `SessionHandle.makeCreateDynamicSubAgentThread` | `create_sub_agent` name and input, the turn's user message, the subagent's tools | `request.model = result.model`, so `resolveAgentDefinition` picks it up. It's persisted on `agent_info.model` through `THREAD_CREATED`. | Now | -| `root` | Where `SessionHandle` builds the root thread's definition for a turn (`resolveAgentDefinition` without `agent_info`) | The turn's user message, a clipped excerpt of recent conversation, the agent's tools | Pass as a model override into the root `resolveAgentDefinition`, and persist it with the turn so approval or user-input resumes keep the same model (§8) | Later | +**The call site:** `SessionHandle.makeCreateDynamicSubAgentThread`. If `request.model` is unset, it builds the input and calls `routeModel`. If a model comes back, it continues with `{ ...request, model }`. From there, the existing override path does the rest: `resolveAgentDefinition` resolves `agent_info.model`, and `THREAD_CREATED` saves it (after AGE-2841). A rebuilt or resumed subagent uses its saved model and never calls the router. -Resume and rebuild never call the router. A subagent is rebuilt from its saved `agent_info`. A resumed root turn reuses the model stored for that turn. +## 2. Candidates and scores -**Reusing the existing model-override path.** `create_sub_agent` already supports a per-subagent model. With `modelSetConfig`, the parent LLM picks from a list, and the pick travels as `request.model`, then `agent_info.model`, then `getModel()` in `resolveAgentDefinition`. Nothing wires `modelSetConfig` today, so routing reuses only the second half: the router fills `request.model`, and the existing path resolves and persists it. Two gaps in that path are fixed here, and they'd apply to `modelSetConfig` too: -- **Params:** `resolveAgentDefinition` always merges the parent spec's `model.params`, so a child runs with the parent's effort. Even `ModelSetEntry.model_params` is ignored today. This is the params fix from `lld.md` §5. -- **Persistence:** `THREAD_CREATED.agent_info` is rebuilt from the original tool call, so a model set after the call isn't saved (§9). +**The score is the Artificial Analysis Intelligence Index, v5** (0–100). It's the closest single score to agentic work: seven of its nine evaluations use tools. It averages four equal categories: Knowledge Work, Coding, Science and General. -If `request.model` is already set, routing doesn't run. - -## 2. Hard-coded tables - -**The score is the Artificial Analysis Intelligence Index, v5,** on a 0–100 scale. It's the closest single score to agentic work: seven of its nine evaluations use tools, all run in Artificial Analysis's open-source agent harness, Stirrup. It has four equal categories: -- **Knowledge Work:** AA-Briefcase, AutomationBench-AA, GDP.pdf. -- **Coding:** FrontierCode-AA, Terminal-Bench 4.0. -- **Science:** Terminal-Bench-Science, CritPt. -- **General:** AA-Omniscience, HLE. - -The General quarter isn't tool use, and that's accepted for now. - -**The model table** is keyed by `canonical_model`, so one row covers every account serving the model. -- **Source:** the Artificial Analysis API (`/api/v2/language/models/free`), matched by the `external_ids.artificial_analysis_slug` that TrueFoundry already stores on each model. -- **Storage:** a JSON file checked into the server package, `truefoundry/modelRoutingScores.json`. It's written by an offline refresh script and reviewed like any code change. The server loads it once at startup into an in-memory `Map`, and validates it with zod, so a bad file fails at startup, not mid-request. Refreshing means re-running the script, committing and deploying. -- **Shape:** nested by effort level, keeping every score Artificial Analysis reports for the model. Non-reasoning variants go under `none`, and a variant with no effort label under `default`: +**The score table** is a JSON file checked into the server package, `modelRoutingScores.json`. It's keyed by Artificial Analysis slug, with one score per effort variant: ```json { "source": "Artificial Analysis Intelligence Index", "index_version": "5", "retrieved_at": "2026-11-13", "models": { - "anthropic/claude-haiku-5-5": { "aa_slug": "claude-haiku-5-5", - "efforts": { "low": 18.6, "medium": 24.6, "high": 29.9, "xhigh": 34.6, "max": 36.5 } }, - "anthropic/claude-opus-5-5": { "aa_slug": "claude-opus-5-5", - "efforts": { "low": 34.8, "medium": 48.2, "high": 51.8, "xhigh": 57.4, "max": 59.1 } }, - "google/gemini-3-5-flash-lite": { "aa_slug": "gemini-3-5-flash-lite", "efforts": { "default": 11.7 } } + "claude-haiku-5-5": { "default": 36.5, "low": 18.6, "medium": 24.6, "high": 29.9, "xhigh": 34.6 }, + "mimo-v2-6-pro": { "default": 29.9 } } } ``` -- **Lookup at request time** uses the effort the thread will actually run at: - - **For a candidate,** that's its default effort from TrueFoundry's metadata. - - **The fallback model needs no score.** It's never shown to Jev. - - **If that effort has no score,** use `default`, then the nearest effort with a score, with ties going to the higher one. +- **How it's produced:** generated once for the PoC from the Artificial Analysis free endpoint (`/api/v2/language/models/free`), and reviewed like code. An entry groups the plain slug (`default`) with its exact effort variants (`-low`, `-medium`, `-high`, `-xhigh`, `-max`, and `-non-reasoning` as `none`). Variants are matched exactly, never by prefix. A prefix match would merge `claude-sonnet-5` with `claude-sonnet-5-5-*`. A refresh script can come later. +- **How it's loaded:** once at startup, validated with zod, so a bad file fails at startup and not mid-request. -Scores looked up at each model's default effort, with the variant used and the TrueFoundry default: +**Candidates are built once per turn from the caller's model list.** An internal `listRoutingCandidates()` on `TrueFoundryModelProviderStore` shares `listModels`' `provider-integrations` fetch. It reads these fields from each model's metadata: -| `canonical_model` | Intelligence Index | Variant used (TrueFoundry default) | -|---|---|---| -| `openai/gpt-6-astra` | 51.6 | medium (medium) | -| `google/gemini-4-argon` | 50.4 | high (medium) | -| `anthropic/claude-opus-5-5` | 48.2 | medium (medium) | -| `anthropic/claude-fable-5-1` | 47.2 | high (high) | -| `openai/gpt-6-1-sol` | 46.3 | medium (medium) | -| `openai/gpt-6-sol` | 44.2 | max (medium) | -| `anthropic/claude-opus-5` | 44.2 | max (high) | -| `anthropic/claude-sonnet-5-5` | 41.5 | high (high) | -| `xai/grok-4-7` | 35.5 | xhigh (not set) | -| `openai/gpt-5-6-terra` | 32.3 | max (medium) | -| `xai/grok-4-6` | 31.0 | xhigh (low) | -| `google/gemini-3-8-flash` | 30.1 | medium (medium) | -| `anthropic/claude-haiku-5-5` | 24.6 | medium (medium) | -| `openai/gpt-5-6-luna` | 23.8 | max (medium) | -| `openai/gpt-6-luna` | 18.5 | medium (medium) | -| `kimi/kimi-k3` | 17.4 | low (max) | -| `google/gemini-3-5-flash-lite` | 11.7 | unlabelled (minimal) | - -That's 35 of devtest's 465 distinct models, and 17 of the 49 pool models. Models used on devtest that have no v5 score in any variant include Sonnet 4.6 (the most common devtest root), the Opus 4.x models, GLM 5.2 and Gemini 3.6 and 3.7 Flash. A model not in the table is never a candidate. The fallback model doesn't need to be in the table, since it isn't shown to Jev, so a Sonnet 4.6 parent's subagents can still be routed to scored candidates. - -**Candidates are built per call from the caller's model list,** through an internal `listRoutingCandidates()` on `TrueFoundryModelProviderStore`. -- **It shares `listModels`' fetch and mapping:** the same `provider-integrations` call, with the caller's token and the tenant's account allow-list. -- **Only the internal `TrueFoundryEnabledModel` type is widened,** with `canonical_model`, price, features, status, retirement date and default effort. -- **`listModels` and the public `AvailableModel` schema are unchanged.** -- **It's called once per turn and cached for that turn only,** shared by all subagents in the turn. Nothing is cached across turns. -- **Revoked access:** the picked model is resolved through the existing `getModel` path before the thread runs. The first use of a model in a turn fetches its provider fresh, so access revoked before the pick fails that check and the thread runs on the fallback model (`model_unavailable`). Access revoked mid-turn, after resolution, behaves as it does today for any model: the gateway rejects the call. - -A model is a candidate when it's: -- an active chat model with tool calling, not retiring within 30 days, -- and in the model table. - -Price, the blended 0.8 × input plus 0.2 × output, is used only to break ties and for the log line's price direction. If several accounts serve the same model, it's asked about once, preferring the fallback model's account, then its provider, then the first by name. There's no fixed limit. On devtest, Jev took 0.9, 0.9, 1.2 and 1.2 seconds for 6, 20, 40 and 60 candidate questions plus 7 benchmark questions. - -**Benchmarks for the resemblance questions.** These are only benchmarks that `model-benchmarks` has data for, and that cover all the models we route between today (Opus 5.5, Sonnet 5.5, Haiku 5.5, GPT-6.1 Sol, GPT-6 Sol, GPT-6 Luna, Gemini 4, Gemini 3.8 Flash and Grok 4.7). So whichever one our tasks resemble, we can switch the score table to it straight away. The descriptions and examples are our own paraphrases. - -| Key | Benchmark (`model-benchmarks` metric) | Pool coverage | Description given to Jev | Example task given to Jev | -|---|---|---|---|---| -| `automation` | AutomationBench-AA (`automationbench_partial_score`) | 34/49 | Multi-step workflows across apps through API tools: look things up, combine results, act on them | "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner" | -| `terminal_bench` | Terminal-Bench 4.0 (`terminalbench_v4_0`) | 33/49 | Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something | "Find why the service fails to start from its logs and fix the config" | -| `gdpval` | GDPval-AA (`gdpval_aa_elo`) | 37/49 | Knowledge-work deliverables (reports, analyses, plans) produced with tools | "From these project documents, write a one-page status report with risks and next steps" | -| `lcr` | AA-LCR (`lcr`) | 48/49 | Answer questions that need reasoning across long documents or large outputs | "From these three long reports, which risk do all of them mention and how does each address it?" | -| `hle` | Humanity's Last Exam (`hle`) | 49/49 | Hard expert-level knowledge and reasoning questions with one correct answer | "A difficult graduate-level question in a technical field" | +| Field | Used for | +|---|---| +| `external_ids.artificial_analysis_slug` | Score lookup. A model without a slug, or with one that isn't in the table, isn't a candidate. | +| `params[reasoning_effort].defaultValue` | Which effort's score Jev sees. | +| Price (`public_cost` or `costs`), blended 0.8 × input + 0.2 × output | Breaking ties, and the log line. It's also needed if a threshold or price ceiling is added. | +| `modalities.input`, `features` | The log line only. No filtering. | + +- **The internal type is widened:** `TrueFoundryEnabledModel` gains these fields. `listModels` and the public `AvailableModel` are unchanged. +- **How a score is looked up:** at the default effort. If that effort has no score, use `default`, then the nearest effort, with ties going to the higher one. +- **A model served by several accounts is asked about once** (same slug), preferring the parent model's account, then its provider, then the first by name. +- **The parent's model is a candidate like any other** if it has a score. If it doesn't, routing still runs among the scored candidates. +- **Revoked access:** the pick goes through the existing `getModel` check before the subagent runs. Access revoked before the pick means `model_unavailable`, and the parent's model runs. Access revoked mid-turn behaves as it does today for any model. -These were left out because the repo covers them too thinly, or not for our key models: EnterpriseOps-Gym, tau2 and IFBench (none of the key models), ITBench, FrontierCode and AnalystAgent (sparse in the pool). +**The internal pool today:** 44 chat models, 18 of them scored. + +| Model | Score | TrueFoundry default effort | Score taken from | $/M blended | +|---|---|---|---|---| +| GPT-6 Astra | 51.6 | medium | medium | 18.00 | +| Claude Opus 5.5 | 48.2 | medium | medium | 7.20 | +| GPT-6.1 Sol | 46.3 | medium | medium | 3.60 | +| Claude Opus 5 | 44.2 | high | unlabelled | 9.00 | +| Claude Sonnet 5.5 | 41.5 | high | high | 3.60 | +| Grok 4.7 | 35.2 | high | high | 2.80 | +| Qwen 3.8 Max | 33.7 | xhigh | unlabelled | 2.80 | +| GPT-5.6 Terra | 32.3 | medium | unlabelled | 4.00 | +| GLM 5.3 | 31.7 | max | unlabelled | 4.00 | +| MiMo V2.6 Pro | 29.9 | medium | unlabelled | 0.52 | +| GLM 5.3 Flash | 27.8 | max | unlabelled | 0.22 | +| MiMo V2.6 Flash | 26.7 | medium | unlabelled | 0.17 | +| Claude Haiku 5.5 | 24.6 | medium | medium | 0.18 | +| GPT-5.6 Luna | 23.8 | medium | unlabelled | 0.40 | +| MiniMax M3 | 18.9 | medium | unlabelled | 0.96 | +| GPT-6 Luna | 18.5 | medium | medium | 0.18 | +| Kimi K3 | 17.4 | max | low (nearest) | 5.40 | +| gpt-oss-120b | 3.8 | medium | unlabelled | 0.43 | + +"Unlabelled" means Artificial Analysis publishes a single score with no effort label. That score is usually the model's maximum effort: Haiku 5.5's is 36.5, above its xhigh 34.6. So models that only have an unlabelled score may be overrated at a lower default. "Nearest" means only other efforts are published. Kimi K3, for example, has only a `low` score, so it's likely underrated at its `max` default. ## 3. The Jev request -One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway URL and token borrowed from the fallback model's provider config, as in `lld.md`. If no candidate qualifies, there's no call. +One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, token and turn headers from the parent model's provider config. There's no call when there are no candidates. ```json { - "model": "archit-typesafe/jev-1.13.0", + "model": "typesafe/jev-1.13.0", "state": { "scope": "subagent", - "name": "", - "task": "", - "context": "", + "name": "", + "task": "", + "context": "", "tools": ["", "web_search", "sandbox"], "benchmark_note": "Artificial Analysis Intelligence Index (0-100, higher is better) averages agentic knowledge work, coding, science and general reasoning evaluations." }, @@ -213,140 +165,135 @@ One call per routing decision to `POST {gateway}/v1/systemone`, with the gateway } }, "can_1": { "type": "noul", "instructions": "Can Claude Sonnet 5.5 at high effort (Intelligence Index 41.5) complete this task fully and correctly?", "criteria": { "...": "same as above" } }, - "resembles_automation": { + "resembles_gdpval": { "type": "noul", - "instructions": "Does this task resemble the benchmark AutomationBench? Multi-step workflows across apps through API tools: look things up, combine results, act on them. Example: 'When a support ticket is tagged urgent, look up the customer's plan and notify the account owner.'", + "instructions": "Does this task resemble the benchmark GDPval? Knowledge-work deliverables (reports, analyses, plans) produced with tools. Example: 'From these project documents, write a one-page status report with risks and next steps.'", "criteria": { "true": "Same kind of work as the benchmark.", "false": "A different kind of work." } } } } ``` -- **Jev sees a task and a pool of candidates, nothing else.** There's no parent model, no price and no comparison in the request. Each candidate is judged on its own: "can this model do this task?". -- **The wording is scope-neutral,** so the root call site sends the same shape with `scope: "root"`. -- **Offline test** on the 49 PoC subagent tasks × 11 candidates (an offline script, not part of this change), checked against each model's real outcome on the task: - - **Jev doesn't say yes to everything.** It's cautious: average yes-probability per model was 0.17–0.42, and 1 answer in 539 reached 0.7. - - **It ranks models sensibly on average.** Models that pass more often get higher answers (rank correlation 0.53). - - **On a single task, it only weakly tells whether that model will do well:** 0.6, where 0.5 is a coin flip and 1.0 is perfect. - - **An earlier wording** showed Jev the parent model ("about as well as the reference, Intelligence Index 48.2"). Jev then mostly compared the two numbers: rank correlation with the score 0.93, and with how often models actually passed only 0.25. That's why the parent model isn't in the request. - - **The number of questions doesn't matter.** The same models asked alongside 3, 11 or 16 candidates changed by 0.01 on average, and every call took about 0.7 s. +- **The decision model comes from config,** set per environment: `typesafe/jev-1.13.0` on internal. If a tenant can't reach it, the call fails and the parent's model runs (`decision_error`). - **Candidate questions** are keyed `can_`, and the router maps them back to models. -- **All five benchmark questions** go in every call. -- **One call,** with a 2-second timeout and `max_candidates` (default 60) as a safety cap. +- **Limits:** one call, with a 2-second timeout and at most `max_candidates` (60) candidate questions. +- **Benchmark-resemblance questions:** five benchmarks, all of which our key models have data for in `model-benchmarks`. The answers are recorded only. + +| Key | Benchmark | Description given to Jev | Example task given to Jev | +|---|---|---|---| +| `automation` | AutomationBench-AA | Multi-step workflows across apps through API tools: look things up, combine results, act on them | "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner" | +| `terminal_bench` | Terminal-Bench 4.0 | Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something | "Find why the service fails to start from its logs and fix the config" | +| `gdpval` | GDPval-AA | Knowledge-work deliverables (reports, analyses, plans) produced with tools | "From these project documents, write a one-page status report with risks and next steps" | +| `lcr` | AA-LCR | Answer questions that need reasoning across long documents or large outputs | "From these three long reports, which risk do all of them mention and how does each address it?" | +| `hle` | Humanity's Last Exam | Hard expert-level knowledge and reasoning questions with one correct answer | "A difficult graduate-level question in a technical field" | + +**Offline test** on 49 real subagent tasks × 11 candidates, checked against how each model actually did: +- **Jev doesn't say yes to everything.** Average yes-probability per model was 0.17–0.42. +- **It ranks models sensibly on average** (rank correlation 0.53 with how often each model passed). On a single task, though, it only weakly tells whether a model will do well: 0.6, where 0.5 is a coin flip. +- **Showing Jev the parent model made it worse.** Jev then mostly compared the two scores. That's why the request has no parent model. +- **The number of questions doesn't matter.** Answers moved by about 0.01 between 3, 11 and 16 candidates. Calls took 0.4–1.2 s for up to 60 questions. ## 4. Picking the model -1. **Pick** the candidate with the highest `noul`. On a tie, the cheaper one wins, using the blended price (§2). There's no threshold. Only the order of Jev's answers matters, not their level, so there's nothing to tune. -2. **Fall back to the parent's model** (the fallback model) only when there's no usable pick: - - when no candidate qualifies at all (`no_candidates`), - - when Jev fails or times out (`decision_error` / `decision_timeout`), - - or when the pick can't be resolved for the caller (`model_unavailable`, using the existing `getModel` check). -3. **No price ceiling.** The pick can cost more than the parent's model. The log line records whether each pick went up or down in price (§7), so shadow mode shows how often each happens. -4. **Routed threads use their own model's default params,** not the parent's (the fix from `lld.md` §5). +1. **Pick** the candidate with the highest yes-probability. On a tie, the cheaper one wins. +2. **Fall back to the parent's model** when there's no usable pick. The reason is logged: + - `no_candidates`; + - `decision_error` or `decision_timeout`; + - `model_unavailable`: the pick fails the `getModel` check. +3. **Shadow mode** logs the pick but runs the parent's model. **On mode** runs the pick. -**In the offline test** (49 tasks, the 11 candidates cheaper than Opus 5.5, compared against Opus): -- **Sonnet 5.5 was picked for 47 of 49 tasks.** Jev's highest answer was nearly always the same model, so on this pool the rule behaves almost like a fixed pick. -- **As good as Opus on 76% of tasks, at 53% of Opus's cost.** -- **The gap between the top two answers was small:** a median of 0.05, and under 0.02 on 20% of tasks. +**In the offline test** (candidates cheaper than Opus 5.5, judged against Opus): Sonnet 5.5 was picked for 47 of 49 tasks. It was as good as Opus on 76% of tasks, at 53% of Opus's cost. -Opus and the models priced above it weren't candidates in that test. With no ceiling they will be. Since Jev's answers rise with the score, the top-scoring model the caller can use may be picked for most tasks. Shadow mode should check the pick distribution and the price direction before switching to "on". - -**Example,** with illustrative answers: a subagent of a Haiku 5.5 parent. Every scored model the caller can use is asked about. Jev answers Kimi K3 0.17, GPT-6 Luna 0.30, Haiku 5.5 0.35, Grok 4.7 0.38, GPT-6 Sol 0.41, Sonnet 5.5 0.46. The highest is Sonnet 5.5, so the subagent runs on Sonnet 5.5, a pricier model than its parent. +**Expected risk on internal:** with no price ceiling, the pool includes GPT-6 Astra, the highest score at $18/M, which is 2.5× Opus. Jev's answers rise with the score, so many subagents may be picked onto Astra. Shadow mode measures the pick distribution and how often the pick costs more than the parent's model. If picks go up in price too often, the next step is a threshold rule: the cheapest candidate at or above a yes-probability. Prices are already on each candidate, so that's a change to the pick function and config only. ## 5. Modes and config -Modes are set per tenant **and per scope**, so root routing can be turned on separately later. - ```json TRUEFOUNDRY_MODEL_ROUTING = { - "decision_model": "archit-typesafe/jev-1.13.0", + "decision_model": "typesafe/jev-1.13.0", "timeout_ms": 2000, "max_candidates": 60, "tenants": { - "": { "subagent": "shadow", "root": "off" } + "": { "subagent": "shadow" } } } ``` -- **off** (the default, or when unset): no call. The hook isn't wired for that scope. -- **shadow:** ask Jev and log the decision, but run on the fallback model. -- **on:** run on the pick. +- **off** (the default, or when unset): the hook isn't wired, and nothing changes. +- **shadow:** ask Jev and log the decision, but run the parent's model. +- **on:** run the pick. -## 6. Tracing: `tfg.thread_id` +## 6. Observability -TrueForge's own `AgentTracing` is a no-op by default (`createTracing()` returns `NOOP_AGENT_TRACING`). The traces we can see are the gateway's request traces, which record each call's `x-tfy-metadata`. The turn-level keys are there (`tfg.session_id`, `tfg.turn_id`, `tfg.agent_id`, `tfg.agent_name`), but nothing identifies the thread. -- **Every thread's model calls:** in `AgentThread`, the model call `modelClient.create(requestBody)` gets `{ metadata: { 'tfg.thread_id': this.threadId, ...(parent ? { 'tfg.parent_thread_id': parent.thread_id } : {}) } }`. - - `buildOutboundRequestHeaders` already folds this into `x-tfy-metadata`. - - It covers the root thread and subagents alike, so root routing needs nothing extra here. - - Context compaction merges the same keys alongside `tfg.compaction`. -- **The Jev call** carries the turn metadata, plus `tfg.thread_id` (the thread being routed), `tfg.routing_scope` and `tfg.model_routing: "true"`. -- **Joining on `tfg.thread_id`** links the Jev request and response with every model call the thread made (§7). +- **Gateway traces,** joined on `tfg.thread_id`: + - every model call a subagent makes (AGE-2837); + - the Jev call, which carries the turn metadata plus `tfg.thread_id` (the subagent being routed) and `tfg.model_routing: "true"`. -## 7. What's observable, and the one log line + The Jev request and response hold every candidate's question and probability, and the benchmark-resemblance answers. +- **One log line per decision:** -Most of the decision is already in the gateway traces, joined by `tfg.thread_id`: -- **The Jev request:** every candidate, with its score, and the benchmark questions. -- **The Jev response:** the probability for each one, including the benchmark-resemblance answers. -- **Every model call the thread made:** the served model, tokens, cost and latency. + ```ts + logger.info('model routing', { + scope, thread_id, mode, reason, fallback_model, picked_model, chosen_model, + picked_p_yes, picked_price, fallback_price, price_direction, picked_image_input, candidate_count, + }); + ``` -Two things never reach a trace: -- **Decisions where Jev isn't called:** no candidate qualifies, or the scope is off. -- **The router's own result:** which candidate it picked, why it fell back, and in shadow mode the pick it didn't apply. + - `price_direction` is `up`, `down` or `same`: the pick's price against the parent's model. + - `picked_image_input` shows whether the pick can read images. That's useful if routed subagents turn out to receive images. + - The line covers what the traces can't: decisions where Jev wasn't called, why routing fell back, and shadow-mode picks that weren't applied. + - It never includes task text. -For those, the router writes one plain `logger.info('model routing', { scope, thread_id, mode, fallback_model, picked_model, chosen_model, price_direction, reason })`. Here `price_direction` is `up`, `down` or `same`: the pick's price against the fallback model's. It's for our own understanding, not a schema or a contract, and never includes task text. +## 7. Rollout -User feedback is joined by `thread_id`. The benchmark-resemblance counts are read from the Jev responses in the traces. +1. **Merge** with the config unset, so nothing changes. +2. **Shadow mode** on a few internal tenants. Check latency, the pick distribution, how often the pick costs more or less than the parent's model, and the fallback reasons. +3. **Switch on** for those tenants once AGE-2841 has merged. Measure quality with user feedback (§10). -## 8. Extending to the root agent (later) +## 8. Root agent (later) -The router, tables, Jev request, pick rule, config and tracing are all shared. What the root needs on top: +The same router, score table, Jev request and pick rule. The root needs four things on top: +- **A call site** where the root thread's definition is built for a turn, with `scope: 'root'` and the spec's model as the fallback. +- **Once per turn,** never mid-turn. +- **Persistence:** save the routed model with the turn, so a resumed turn keeps it. +- **Opt-in:** a per-tenant or per-agent opt-in. A model the user picked explicitly is never overridden. -- **A call site:** where `SessionHandle` builds the root thread's definition for a turn. It builds `RouteModelInput` with `scope: 'root'` and the spec's model as the fallback, and passes the result as a model override into the root `resolveAgentDefinition`. That's an optional `modelOverride` input, alongside `agent_info`. -- **Granularity: once per turn,** keeping the model for the whole turn. Re-deciding mid-turn would change model during an agent loop, which the Artemis baseline showed hurts. A session-level sticky option (keep the first turn's choice) can come later, if prompt caching matters more than per-turn savings. -- **Persistence:** store the routed model with the turn, as a turn-snapshot field, so a resumed turn (after an approval or user input) keeps the same model and never re-classifies. It's the only new stored field, and it's needed only for root. -- **Opt-in:** route the root only when the agent or tenant opts in. For example, a spec setting like `model: { name: , auto: true }`, or the per-scope tenant mode. A model the user picked explicitly is never overridden. -- **State:** the latest user message plus a clipped excerpt of recent conversation, instead of a delegated task. +## 9. Code changes -## 9. Code changes (subagent scope now) +**Core (`packages/trueforge-core`):** only the hook and the call site. Core has no access to the model list, Jev or the scores. -**Core (`packages/trueforge-core`):** +| File | Change | +|---|---| +| `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult` and the optional `routeModel` hook (§1). | +| `agent-session/TurnResourceResolver.ts` | Accept `routeModel` in its deps and expose it. | +| `agent-session/SessionHandle.ts` | In `makeCreateDynamicSubAgentThread`, call `routeModel` when `request.model` is unset, and continue with `{ ...request, model }` if one comes back. | + +**Server: provider-neutral routing (`packages/trueforge/src/modelRouting/`).** None of this knows about TrueFoundry. It's the part a future routing capability would reuse or move into core. | File | Change | |---|---| -| `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult`, and the optional hook `routeModel?(input)` (§1). | -| `agent-session/TurnResourceResolver.ts` | Forward the hook from deps, and pass `isModelAvailable` (built on `getModel`). Plus the params fix from `lld.md` §5. | -| `agent-session/SessionHandle.ts` `makeCreateDynamicSubAgentThread` | If `request.model` is unset, build `RouteModelInput` (`scope: 'subagent'`) and call `routeModel`. If `model` comes back, use `{ ...request, model }` from then on, through the existing override path (§1). | -| `core/runtime/AgentThreadOrchestrator.ts` (`AGENT_CREATE_SUBAGENT`) | Build `THREAD_CREATED.agent_info` from `subAgentThread.agentInfo`, so the routed model is persisted. | -| `core/runtime/AgentThread.ts` | Pass `tfg.thread_id` and `tfg.parent_thread_id` metadata on every model call (§6). | +| **New** `types.ts` | The two interfaces a provider implements: `RoutingCandidateSource`, which lists candidates (`model`, `slug`, `defaultEffort`, `price`, plus metadata for the log line), and `DecisionClient`, which asks yes/no questions and returns probabilities. | +| **New** `scores.ts` + `modelRoutingScores.json` | Loads and validates the score table at startup, and looks up a score by slug and effort (§2). | +| **New** `pick.ts` | The pick rule, as a pure function from candidates and probabilities to a pick (§4). A threshold or price ceiling would change only this file and the config. | +| **New** `router.ts` | `createModelRouter({ candidates, decision, scores, config, logger })`. It builds the questions (§3), calls the decision client, picks, logs (§6), and returns `routeModel`. It never throws. | -**Server (`packages/trueforge`):** +**Server: TrueFoundry implementations.** | File | Change | |---|---| -| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING`, with per-tenant, per-scope modes (§5). | -| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with `canonical_model`, price, features, status, retirement date and default effort. `listModels` keeps returning the public `AvailableModel`, unchanged. | -| `truefoundry/TrueFoundryModelProviderStore.ts` | The internal `listRoutingCandidates()`, using the same fetch and mapping as `listModels` (§2). | -| **New** `truefoundry/modelRoutingScores.json` | The score table, nested by effort level, with source, version and date (§2). It's generated by the refresh script. | -| **New** `truefoundry/modelRoutingTables.ts` | Loads and validates the JSON at startup into an in-memory map, does the lookup by effort with its fallback order, and holds the five benchmark descriptions (§2). | -| **New** `benchmark/refresh_model_scores.py` (offline) | Calls the Artificial Analysis API, matches models by slug, and writes `modelRoutingScores.json`. | -| **New** `truefoundry/modelRouter.ts` | `createModelRouter()`, which returns `routeModel`. It's scope-agnostic: candidates (§2), the Jev request (§3), the pick (§4), one info log line (§7). It never throws. | -| `apis/turns.ts` | Wire `routeModel` when any scope for the tenant isn't off. The router checks the per-scope mode itself. | - -The subagent scope needs no database schema or public API changes, since `agent_info.model` already exists. The root scope adds one turn-snapshot field (§8). - -## 10. Rollout - -1. Merge with the config unset, so nothing changes. -2. Run subagent scope in shadow mode for two or three devtest tenants. Check latency, the fallback reasons, the pick distribution, how often picks go up or down in price, and `top_benchmark`. -3. Switch subagent scope to "on" for those tenants. Measure with user feedback, joined by `thread_id`. -4. Add the root call site (§8), and run root scope in shadow mode first. -5. Use the resemblance data to choose the evaluation benchmark, and replace the score table if a narrower one fits better. - -## 11. Open questions for review - -1. **Unscored models in use:** Sonnet 4.6, the Opus 4.x models and GLM 5.2 have no v5 score. As fallback models they're fine, since they aren't shown to Jev. As candidates they're never picked. Should they get a hand-assigned tier so they can be candidates, or stay out of the candidate pool? -2. **Price ceiling, later:** none in the first version. If shadow or "on" data shows the highest-probability pick goes up in price too often for too little, add a per-tenant cap as a multiple of the fallback model's price. -3. **Model names in the questions:** name them, as in option 1 of the picture, or say only "a model with Intelligence Index S" to avoid Jev's priors about names? -4. **Benchmark list:** are these five right? Should there also be a single "closest benchmark" choice question? -5. **Root scope:** routing once per turn or once per session, and how a user or agent opts in to an automatically selected model. -6. **Traces:** is the gateway request trace with `x-tfy-metadata` enough, or should a real TrueForge tracer be wired up as well? +| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with the fields in §2. | +| `truefoundry/TrueFoundryModelProviderStore.ts` | `listRoutingCandidates()`, the `RoutingCandidateSource` for TrueFoundry mode. | +| **New** `truefoundry/jevDecisionClient.ts` | The `DecisionClient` for Jev: `POST {gateway}/v1/systemone` with the parent model's gateway URL, token and turn headers. | +| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING` (§5). | +| `apis/turns.ts` | When the tenant's subagent mode isn't off, create the router per turn from the TrueFoundry candidate source and the Jev client, and pass `routeModel` into the resolver. | + +No database schema or public API changes. + +**Extending outside TrueFoundry mode later:** add a `RoutingCandidateSource` for configured providers (slug from the shipped catalogue), and a `DecisionClient` for whatever decision model is available. Then wire them where the resolver is built. The score table, pick rule, router and core hook stay the same. + +## 10. Open questions + +1. **How is quality measured?** The TrueForge backend doesn't store user feedback. Only the UI runtime passes a feedback adapter through. Feedback is also given per turn, on the root agent's answer, not per subagent. Before "on" mode we need to know where feedback lands on internal tenants, and how to attribute it to subagents. +2. **Model names in the questions:** keep them, or say only "a model with Intelligence Index S" to avoid Jev's preconceptions about names? +3. **Benchmark list:** are these five right? +4. **Threshold or price ceiling:** likely needed soon, if shadow data shows picks go up in price too often (§4). It's a change to `pick.ts` and config only. From be8179c4c88bdf55759c9a695d1999769bb5c21e Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 14:17:55 +0530 Subject: [PATCH 06/14] feat: subagent model routing with Jev (TrueFoundry mode, off by default) Co-authored-by: Cursor --- .changeset/subagent-model-routing.md | 6 + docs/subagent-auto-routing/lld.md | 203 +++++++++-------- .../agent-session/ITurnResourceResolver.ts | 26 +++ .../src/agent-session/SessionHandle.ts | 29 ++- .../src/agent-session/TurnResourceResolver.ts | 13 +- .../trueforge-core/src/agent-session/index.ts | 2 +- packages/trueforge/.env.example | 4 + packages/trueforge/src/apis/turns.ts | 47 +++- packages/trueforge/src/config.ts | 32 +++ .../trueforge/src/db/modelProviderStore.ts | 3 + .../src/modelRouting/modelRoutingScores.ts | 137 ++++++++++++ packages/trueforge/src/modelRouting/pick.ts | 20 ++ packages/trueforge/src/modelRouting/router.ts | 206 ++++++++++++++++++ packages/trueforge/src/modelRouting/scores.ts | 48 ++++ packages/trueforge/src/modelRouting/types.ts | 37 ++++ .../truefoundry/InlineModelProviderStore.ts | 6 + .../TrueFoundryModelProviderStore.ts | 11 + .../src/truefoundry/jevDecisionClient.ts | 52 +++++ .../src/truefoundry/mapEnabledModels.ts | 46 ++++ .../tests/unit/modelRouting/router.test.ts | 52 +++++ .../tests/unit/modelRouting/scores.test.ts | 22 ++ .../unit/truefoundry/mapEnabledModels.test.ts | 22 ++ .../truefoundry/trueFoundryNaming.test.ts | 4 +- 23 files changed, 927 insertions(+), 101 deletions(-) create mode 100644 .changeset/subagent-model-routing.md create mode 100644 packages/trueforge/src/modelRouting/modelRoutingScores.ts create mode 100644 packages/trueforge/src/modelRouting/pick.ts create mode 100644 packages/trueforge/src/modelRouting/router.ts create mode 100644 packages/trueforge/src/modelRouting/scores.ts create mode 100644 packages/trueforge/src/modelRouting/types.ts create mode 100644 packages/trueforge/src/truefoundry/jevDecisionClient.ts create mode 100644 packages/trueforge/tests/unit/modelRouting/router.test.ts create mode 100644 packages/trueforge/tests/unit/modelRouting/scores.test.ts create mode 100644 packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts diff --git a/.changeset/subagent-model-routing.md b/.changeset/subagent-model-routing.md new file mode 100644 index 000000000..0b4be525b --- /dev/null +++ b/.changeset/subagent-model-routing.md @@ -0,0 +1,6 @@ +--- +'@truefoundry/trueforge-core': patch +'@truefoundry/trueforge': patch +--- + +Add optional subagent model routing. In TrueFoundry mode, `TRUEFOUNDRY_MODEL_ROUTING` turns it on per tenant: before a subagent starts, a decision model rates each of the caller's scored models for the task, and the highest-rated one is picked. `shadow` logs the pick and keeps the parent's model; `on` runs the pick. Off by default. diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md index de7c50178..e1883bb31 100644 --- a/docs/subagent-auto-routing/lld.md +++ b/docs/subagent-auto-routing/lld.md @@ -5,6 +5,7 @@ Status: ready for review. Scope: a PoC on internal TrueFoundry tenants, in TrueF When a subagent is spawned, one Jev call picks the model it runs on. The router is generic (it takes a `scope`), so the root agent can use it later (§8). Its modules are split so that model routing can later become a capability that also works outside TrueFoundry mode (§9). Only the subagent scope is built now. **Related work, tracked separately:** + - **`tfg.thread_id` on every model call** (AGE-2837, PR #1392). Routing analysis joins on it. - **Subagent model overrides** (AGE-2841). A subagent on a different model must use that model's own default params, not the parent's, and `THREAD_CREATED` must save the model the subagent actually runs on. Shadow mode doesn't need this. "On" mode does. @@ -12,22 +13,23 @@ When a subagent is spawned, one Jev call picks the model it runs on. The router **Fixed by this design.** Changing any of these means a new design review. -| # | Invariant | Where | -|---|---|---| -| I1 | **One Jev call per subagent, made before it starts.** Never during the agent loop. | §1 | -| I2 | **Jev sees only the task and the candidate pool.** One yes/no question per candidate (name, effort, score), plus one per benchmark. No parent model, prices or comparisons. | §3 | -| I3 | **The pick is the candidate with the highest yes-probability,** the cheaper one on a tie. There's no threshold and no price ceiling yet. | §4 | -| I4 | **The fallback is the parent's model.** It runs only when routing can't produce a pick. | §4 | -| I5 | **Candidates are the caller's models that have a score.** No other metadata filters candidates. Other fields are read only for the log line. | §2 | -| I6 | **Scores are bundled data,** keyed by Artificial Analysis slug. No Artificial Analysis call while a request runs. | §2 | -| I7 | **Jev sees each candidate's score at the effort it will actually run at:** its default effort on TrueFoundry. | §2 | -| I8 | **The router never throws.** Any error, timeout or unusable pick means the parent's model runs, as today. | §4 | -| I9 | **The routed model is applied through the existing `request.model` path** and saved on `agent_info.model`. Resume and rebuild never call the router. | §1 | -| I10 | **One optional core hook,** `routeModel`. The router's logic lives in the server, split into provider-neutral and TrueFoundry-specific modules. | §1, §9 | -| I11 | **Observability:** gateway traces joined on `tfg.thread_id`, plus one info log line per decision. No new tables or public API. | §6 | -| I12 | **Mode per tenant and per scope:** off, shadow or on. Off by default. | §5 | +| # | Invariant | Where | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | +| I1 | **One Jev call per subagent, made before it starts.** Never during the agent loop. | §1 | +| I2 | **Jev sees only the task and the candidate pool.** One yes/no question per candidate (name, effort, score), plus one per benchmark. No parent model, prices or comparisons. | §3 | +| I3 | **The pick is the candidate with the highest yes-probability,** the cheaper one on a tie. There's no threshold and no price ceiling yet. | §4 | +| I4 | **The fallback is the parent's model.** It runs only when routing can't produce a pick. | §4 | +| I5 | **Candidates are the caller's models that have a score.** No other metadata filters candidates. Other fields are read only for the log line. | §2 | +| I6 | **Scores are bundled data,** keyed by Artificial Analysis slug. No Artificial Analysis call while a request runs. | §2 | +| I7 | **Jev sees each candidate's score at the effort it will actually run at:** its default effort on TrueFoundry. | §2 | +| I8 | **The router never throws.** Any error, timeout or unusable pick means the parent's model runs, as today. | §4 | +| I9 | **The routed model is applied through the existing `request.model` path** and saved on `agent_info.model`. Resume and rebuild never call the router. | §1 | +| I10 | **One optional core hook,** `routeModel`. The router's logic lives in the server, split into provider-neutral and TrueFoundry-specific modules. | §1, §9 | +| I11 | **Observability:** gateway traces joined on `tfg.thread_id`, plus one info log line per decision. No new tables or public API. | §6 | +| I12 | **Mode per tenant and per scope:** off, shadow or on. Off by default. | §5 | **Tunable without a design change:** + - **Config:** the decision model, the Jev timeout (2 s), `max_candidates` (60), and each tenant's mode. - **Data:** the score table, and the benchmark-resemblance list. @@ -58,24 +60,23 @@ type RoutingScope = 'subagent' | 'root'; interface RouteModelInput { scope: RoutingScope; - threadId: string; // the thread being routed - fallbackModel: string; // the parent's model (account/model) + threadId: string; // the thread being routed + fallbackModel: string; // the parent's model (account/model) work: { - name?: string; // subagent name - text: string; // the delegated task - context?: string; // the turn's user message - tools: string[]; // MCP server names, web_search, sandbox + name?: string; // subagent name + text: string; // the delegated task (self-contained by the create_sub_agent contract) + tools: string[]; // MCP server names, web_search, sandbox }; signal: AbortSignal; } interface RouteModelResult { - model?: string; // set only in "on" mode when a candidate was picked + model?: string; // set only in "on" mode when a candidate was picked } interface ITurnResourceResolver { // ...existing methods - routeModel?(input: RouteModelInput): Promise; // never throws; undefined = today + routeModel?(input: RouteModelInput): Promise; // never throws; undefined = today } ``` @@ -85,58 +86,55 @@ interface ITurnResourceResolver { **The score is the Artificial Analysis Intelligence Index, v5** (0–100). It's the closest single score to agentic work: seven of its nine evaluations use tools. It averages four equal categories: Knowledge Work, Coding, Science and General. -**The score table** is a JSON file checked into the server package, `modelRoutingScores.json`. It's keyed by Artificial Analysis slug, with one score per effort variant: +**The score table** is a typed TypeScript constant in the server package, `modelRouting/modelRoutingScores.ts`. It mirrors Artificial Analysis: one entry per Artificial Analysis slug, so each effort variant is its own entry: -```json -{ - "source": "Artificial Analysis Intelligence Index", "index_version": "5", "retrieved_at": "2026-11-13", - "models": { - "claude-haiku-5-5": { "default": 36.5, "low": 18.6, "medium": 24.6, "high": 29.9, "xhigh": 34.6 }, - "mimo-v2-6-pro": { "default": 29.9 } - } -} +```ts +'claude-haiku-5-5': { name: 'Claude Haiku 5.5', score: 36.5 }, // plain slug: Artificial Analysis labels it (Max) +'claude-haiku-5-5-medium': { name: 'Claude Haiku 5.5', score: 24.6 }, +'mimo-v2-6-pro': { name: 'MiMo-V2.6-Pro', score: 29.9 }, ``` -- **How it's produced:** generated once for the PoC from the Artificial Analysis free endpoint (`/api/v2/language/models/free`), and reviewed like code. An entry groups the plain slug (`default`) with its exact effort variants (`-low`, `-medium`, `-high`, `-xhigh`, `-max`, and `-non-reasoning` as `none`). Variants are matched exactly, never by prefix. A prefix match would merge `claude-sonnet-5` with `claude-sonnet-5-5-*`. A refresh script can come later. -- **How it's loaded:** once at startup, validated with zod, so a bad file fails at startup and not mid-request. +- **How it's produced:** generated once for the PoC from the Artificial Analysis free endpoint (`/api/v2/language/models/free`), and reviewed like code. A refresh script can come later. +- **Why a TypeScript constant:** it's bundled like the shipped catalogues, and a malformed entry fails at compile time. No loader is needed. +- **The name** is Artificial Analysis's model name without the effort label. It's what Jev sees in the question. **Candidates are built once per turn from the caller's model list.** An internal `listRoutingCandidates()` on `TrueFoundryModelProviderStore` shares `listModels`' `provider-integrations` fetch. It reads these fields from each model's metadata: -| Field | Used for | -|---|---| -| `external_ids.artificial_analysis_slug` | Score lookup. A model without a slug, or with one that isn't in the table, isn't a candidate. | -| `params[reasoning_effort].defaultValue` | Which effort's score Jev sees. | -| Price (`public_cost` or `costs`), blended 0.8 × input + 0.2 × output | Breaking ties, and the log line. It's also needed if a threshold or price ceiling is added. | -| `modalities.input`, `features` | The log line only. No filtering. | +| Field | Used for | +| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `external_ids.artificial_analysis_slug` | Score lookup. A model without a slug, or with one that isn't in the table, isn't a candidate. | +| `params[reasoning_effort].defaultValue` | Which effort's score Jev sees. | +| Price (`public_cost` or `costs`), blended 0.8 × input + 0.2 × output | Breaking ties, and the log line. It's also needed if a threshold or price ceiling is added. | +| `modalities.input`, `features` | The log line only. No filtering. | - **The internal type is widened:** `TrueFoundryEnabledModel` gains these fields. `listModels` and the public `AvailableModel` are unchanged. -- **How a score is looked up:** at the default effort. If that effort has no score, use `default`, then the nearest effort, with ties going to the higher one. +- **How a score is looked up:** the exact variant `-` (`-non-reasoning` for `none`). If there isn't one, use the plain slug, then the nearest effort variant, with ties going to the higher one. Variants match exactly, never by prefix. A prefix match would let `claude-sonnet-5` read `claude-sonnet-5-5-*`. - **A model served by several accounts is asked about once** (same slug), preferring the parent model's account, then its provider, then the first by name. - **The parent's model is a candidate like any other** if it has a score. If it doesn't, routing still runs among the scored candidates. -- **Revoked access:** the pick goes through the existing `getModel` check before the subagent runs. Access revoked before the pick means `model_unavailable`, and the parent's model runs. Access revoked mid-turn behaves as it does today for any model. +- **Revoked access:** candidates come from the caller's model list for this turn, so a pick is a model the caller could use when the turn started. Access revoked mid-turn behaves as it does today for any model: the gateway rejects the call. **The internal pool today:** 44 chat models, 18 of them scored. -| Model | Score | TrueFoundry default effort | Score taken from | $/M blended | -|---|---|---|---|---| -| GPT-6 Astra | 51.6 | medium | medium | 18.00 | -| Claude Opus 5.5 | 48.2 | medium | medium | 7.20 | -| GPT-6.1 Sol | 46.3 | medium | medium | 3.60 | -| Claude Opus 5 | 44.2 | high | unlabelled | 9.00 | -| Claude Sonnet 5.5 | 41.5 | high | high | 3.60 | -| Grok 4.7 | 35.2 | high | high | 2.80 | -| Qwen 3.8 Max | 33.7 | xhigh | unlabelled | 2.80 | -| GPT-5.6 Terra | 32.3 | medium | unlabelled | 4.00 | -| GLM 5.3 | 31.7 | max | unlabelled | 4.00 | -| MiMo V2.6 Pro | 29.9 | medium | unlabelled | 0.52 | -| GLM 5.3 Flash | 27.8 | max | unlabelled | 0.22 | -| MiMo V2.6 Flash | 26.7 | medium | unlabelled | 0.17 | -| Claude Haiku 5.5 | 24.6 | medium | medium | 0.18 | -| GPT-5.6 Luna | 23.8 | medium | unlabelled | 0.40 | -| MiniMax M3 | 18.9 | medium | unlabelled | 0.96 | -| GPT-6 Luna | 18.5 | medium | medium | 0.18 | -| Kimi K3 | 17.4 | max | low (nearest) | 5.40 | -| gpt-oss-120b | 3.8 | medium | unlabelled | 0.43 | +| Model | Score | TrueFoundry default effort | Score taken from | $/M blended | +| ----------------- | ----- | -------------------------- | ---------------- | ----------- | +| GPT-6 Astra | 51.6 | medium | medium | 18.00 | +| Claude Opus 5.5 | 48.2 | medium | medium | 7.20 | +| GPT-6.1 Sol | 46.3 | medium | medium | 3.60 | +| Claude Opus 5 | 44.2 | high | unlabelled | 9.00 | +| Claude Sonnet 5.5 | 41.5 | high | high | 3.60 | +| Grok 4.7 | 35.2 | high | high | 2.80 | +| Qwen 3.8 Max | 33.7 | xhigh | unlabelled | 2.80 | +| GPT-5.6 Terra | 32.3 | medium | unlabelled | 4.00 | +| GLM 5.3 | 31.7 | max | unlabelled | 4.00 | +| MiMo V2.6 Pro | 29.9 | medium | unlabelled | 0.52 | +| GLM 5.3 Flash | 27.8 | max | unlabelled | 0.22 | +| MiMo V2.6 Flash | 26.7 | medium | unlabelled | 0.17 | +| Claude Haiku 5.5 | 24.6 | medium | medium | 0.18 | +| GPT-5.6 Luna | 23.8 | medium | unlabelled | 0.40 | +| MiniMax M3 | 18.9 | medium | unlabelled | 0.96 | +| GPT-6 Luna | 18.5 | medium | medium | 0.18 | +| Kimi K3 | 17.4 | max | low (nearest) | 5.40 | +| gpt-oss-120b | 3.8 | medium | unlabelled | 0.43 | "Unlabelled" means Artificial Analysis publishes a single score with no effort label. That score is usually the model's maximum effort: Haiku 5.5's is 36.5, above its xhigh 34.6. So models that only have an unlabelled score may be overrated at a lower default. "Nearest" means only other efforts are published. Kimi K3, for example, has only a `low` score, so it's likely underrated at its `max` default. @@ -151,7 +149,6 @@ One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, t "scope": "subagent", "name": "", "task": "", - "context": "", "tools": ["", "web_search", "sandbox"], "benchmark_note": "Artificial Analysis Intelligence Index (0-100, higher is better) averages agentic knowledge work, coding, science and general reasoning evaluations." }, @@ -164,7 +161,11 @@ One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, t "false": "It would likely miss parts, make errors, or stop before the task is done." } }, - "can_1": { "type": "noul", "instructions": "Can Claude Sonnet 5.5 at high effort (Intelligence Index 41.5) complete this task fully and correctly?", "criteria": { "...": "same as above" } }, + "can_1": { + "type": "noul", + "instructions": "Can Claude Sonnet 5.5 at high effort (Intelligence Index 41.5) complete this task fully and correctly?", + "criteria": { "...": "same as above" } + }, "resembles_gdpval": { "type": "noul", "instructions": "Does this task resemble the benchmark GDPval? Knowledge-work deliverables (reports, analyses, plans) produced with tools. Example: 'From these project documents, write a one-page status report with risks and next steps.'", @@ -179,15 +180,16 @@ One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, t - **Limits:** one call, with a 2-second timeout and at most `max_candidates` (60) candidate questions. - **Benchmark-resemblance questions:** five benchmarks, all of which our key models have data for in `model-benchmarks`. The answers are recorded only. -| Key | Benchmark | Description given to Jev | Example task given to Jev | -|---|---|---|---| -| `automation` | AutomationBench-AA | Multi-step workflows across apps through API tools: look things up, combine results, act on them | "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner" | -| `terminal_bench` | Terminal-Bench 4.0 | Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something | "Find why the service fails to start from its logs and fix the config" | -| `gdpval` | GDPval-AA | Knowledge-work deliverables (reports, analyses, plans) produced with tools | "From these project documents, write a one-page status report with risks and next steps" | -| `lcr` | AA-LCR | Answer questions that need reasoning across long documents or large outputs | "From these three long reports, which risk do all of them mention and how does each address it?" | -| `hle` | Humanity's Last Exam | Hard expert-level knowledge and reasoning questions with one correct answer | "A difficult graduate-level question in a technical field" | +| Key | Benchmark | Description given to Jev | Example task given to Jev | +| ---------------- | -------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | +| `automation` | AutomationBench-AA | Multi-step workflows across apps through API tools: look things up, combine results, act on them | "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner" | +| `terminal_bench` | Terminal-Bench 4.0 | Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something | "Find why the service fails to start from its logs and fix the config" | +| `gdpval` | GDPval-AA | Knowledge-work deliverables (reports, analyses, plans) produced with tools | "From these project documents, write a one-page status report with risks and next steps" | +| `lcr` | AA-LCR | Answer questions that need reasoning across long documents or large outputs | "From these three long reports, which risk do all of them mention and how does each address it?" | +| `hle` | Humanity's Last Exam | Hard expert-level knowledge and reasoning questions with one correct answer | "A difficult graduate-level question in a technical field" | **Offline test** on 49 real subagent tasks × 11 candidates, checked against how each model actually did: + - **Jev doesn't say yes to everything.** Average yes-probability per model was 0.17–0.42. - **It ranks models sensibly on average** (rank correlation 0.53 with how often each model passed). On a single task, though, it only weakly tells whether a model will do well: 0.6, where 0.5 is a coin flip. - **Showing Jev the parent model made it worse.** Jev then mostly compared the two scores. That's why the request has no parent model. @@ -198,8 +200,8 @@ One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, t 1. **Pick** the candidate with the highest yes-probability. On a tie, the cheaper one wins. 2. **Fall back to the parent's model** when there's no usable pick. The reason is logged: - `no_candidates`; - - `decision_error` or `decision_timeout`; - - `model_unavailable`: the pick fails the `getModel` check. + - `decision_error`: the call failed, or Jev answered none of the candidate questions; + - `decision_timeout`. 3. **Shadow mode** logs the pick but runs the parent's model. **On mode** runs the pick. **In the offline test** (candidates cheaper than Opus 5.5, judged against Opus): Sonnet 5.5 was picked for 47 of 49 tasks. It was as good as Opus on 76% of tasks, at 53% of Opus's cost. @@ -230,12 +232,24 @@ TRUEFOUNDRY_MODEL_ROUTING = { - the Jev call, which carries the turn metadata plus `tfg.thread_id` (the subagent being routed) and `tfg.model_routing: "true"`. The Jev request and response hold every candidate's question and probability, and the benchmark-resemblance answers. + - **One log line per decision:** ```ts logger.info('model routing', { - scope, thread_id, mode, reason, fallback_model, picked_model, chosen_model, - picked_p_yes, picked_price, fallback_price, price_direction, picked_image_input, candidate_count, + scope, + thread_id, + mode, + reason, + fallback_model, + picked_model, + chosen_model, + picked_p_yes, + picked_price, + fallback_price, + price_direction, + picked_image_input, + candidate_count, }); ``` @@ -253,6 +267,7 @@ TRUEFOUNDRY_MODEL_ROUTING = { ## 8. Root agent (later) The same router, score table, Jev request and pick rule. The root needs four things on top: + - **A call site** where the root thread's definition is built for a turn, with `scope: 'root'` and the spec's model as the fallback. - **Once per turn,** never mid-turn. - **Persistence:** save the routed model with the turn, so a resumed turn keeps it. @@ -262,34 +277,36 @@ The same router, score table, Jev request and pick rule. The root needs four thi **Core (`packages/trueforge-core`):** only the hook and the call site. Core has no access to the model list, Jev or the scores. -| File | Change | -|---|---| -| `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult` and the optional `routeModel` hook (§1). | -| `agent-session/TurnResourceResolver.ts` | Accept `routeModel` in its deps and expose it. | -| `agent-session/SessionHandle.ts` | In `makeCreateDynamicSubAgentThread`, call `routeModel` when `request.model` is unset, and continue with `{ ...request, model }` if one comes back. | +| File | Change | +| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult` and the optional `routeModel` hook (§1). | +| `agent-session/TurnResourceResolver.ts` | Accept `routeModel` in its deps and expose it. | +| `agent-session/SessionHandle.ts` | In `makeCreateDynamicSubAgentThread`, call `routeModel` when `request.model` is unset, and continue with `{ ...request, model }` if one comes back. | **Server: provider-neutral routing (`packages/trueforge/src/modelRouting/`).** None of this knows about TrueFoundry. It's the part a future routing capability would reuse or move into core. -| File | Change | -|---|---| -| **New** `types.ts` | The two interfaces a provider implements: `RoutingCandidateSource`, which lists candidates (`model`, `slug`, `defaultEffort`, `price`, plus metadata for the log line), and `DecisionClient`, which asks yes/no questions and returns probabilities. | -| **New** `scores.ts` + `modelRoutingScores.json` | Loads and validates the score table at startup, and looks up a score by slug and effort (§2). | -| **New** `pick.ts` | The pick rule, as a pure function from candidates and probabilities to a pick (§4). A threshold or price ceiling would change only this file and the config. | -| **New** `router.ts` | `createModelRouter({ candidates, decision, scores, config, logger })`. It builds the questions (§3), calls the decision client, picks, logs (§6), and returns `routeModel`. It never throws. | +| File | Change | +| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **New** `types.ts` | What a provider supplies: `RoutingCandidate` (`model`, `slug`, `defaultEffort`, `price`, `imageInput`), a function that lists them, and `DecisionClient`, which asks yes/no questions and returns probabilities. Also the routing modes. | +| **New** `scores.ts` + `modelRoutingScores.ts` | The score table, and the lookup by slug and effort (§2). | +| **New** `pick.ts` | The pick rule, as a pure function from candidates and probabilities to a pick (§4). A threshold or price ceiling would change only this file and the config. | +| **New** `router.ts` | `createModelRouter({ candidates, decision, scores, config, logger })`. It builds the questions (§3), calls the decision client, picks, logs (§6), and returns `routeModel`. It never throws. | **Server: TrueFoundry implementations.** -| File | Change | -|---|---| -| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with the fields in §2. | -| `truefoundry/TrueFoundryModelProviderStore.ts` | `listRoutingCandidates()`, the `RoutingCandidateSource` for TrueFoundry mode. | -| **New** `truefoundry/jevDecisionClient.ts` | The `DecisionClient` for Jev: `POST {gateway}/v1/systemone` with the parent model's gateway URL, token and turn headers. | -| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING` (§5). | -| `apis/turns.ts` | When the tenant's subagent mode isn't off, create the router per turn from the TrueFoundry candidate source and the Jev client, and pass `routeModel` into the resolver. | +| File | Change | +| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `db/modelProviderStore.ts` | An optional `listRoutingCandidates()` on `IModelProviderStore`. Stores that can't supply catalogue metadata omit it, and routing isn't wired for them. | +| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with the fields in §2. Each new field falls back to undefined on an unexpected shape, so it can never break model listing. | +| `truefoundry/TrueFoundryModelProviderStore.ts` | `listRoutingCandidates()` for TrueFoundry mode, from the same `provider-integrations` fetch. | +| `truefoundry/InlineModelProviderStore.ts` | Forwards `listRoutingCandidates()` to the wrapped store. | +| **New** `truefoundry/jevDecisionClient.ts` | The `DecisionClient` for Jev: `POST {gateway}/v1/systemone` with the parent model's gateway URL, token and turn headers. | +| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING` (§5). | +| `apis/turns.ts` | When the tenant's subagent mode isn't off, create the router per turn from the TrueFoundry candidate source and the Jev client, and pass `routeModel` into the resolver. | No database schema or public API changes. -**Extending outside TrueFoundry mode later:** add a `RoutingCandidateSource` for configured providers (slug from the shipped catalogue), and a `DecisionClient` for whatever decision model is available. Then wire them where the resolver is built. The score table, pick rule, router and core hook stay the same. +**Extending outside TrueFoundry mode later:** implement `listRoutingCandidates()` on the configured-provider stores (slug from the shipped catalogue), and add a `DecisionClient` for whatever decision model is available. Then wire them where the resolver is built. The score table, pick rule, router and core hook stay the same. ## 10. Open questions diff --git a/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts b/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts index 63d5d56c2..39f77548f 100644 --- a/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts +++ b/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts @@ -14,6 +14,27 @@ export interface ResolvedAgentDefinition { extraCapabilities?: AgentCapability[] | undefined; } +export type RoutingScope = 'subagent' | 'root'; + +/** The thread about to start, as seen by {@link ITurnResourceResolver.routeModel}. */ +export interface RouteModelInput { + scope: RoutingScope; + threadId: string; + /** Model the thread runs on when routing doesn't pick one (`provider/model`). */ + fallbackModel: string; + work: { + name?: string | undefined; + text: string; + tools: string[]; + }; + signal: AbortSignal; +} + +export interface RouteModelResult { + /** Model to run instead of the fallback. Undefined keeps the fallback. */ + model?: string | undefined; +} + /** * Per-run wiring contract consumed by SessionHandle.createTurn(). Implementations hold * per-request state (secrets, providers, caches) freely. @@ -63,6 +84,11 @@ export interface ITurnResourceResolver; + /** + * Chooses the model for a thread before it starts. Omit to keep every thread on its default model. + * Must not throw: on any failure, resolve without a model. + */ + routeModel?(input: RouteModelInput): Promise; /** * Release per-run resources. Called by the harness at the end of the run: * by TurnHandle.stream() in its finally (AFTER the terminal state write), or by diff --git a/packages/trueforge-core/src/agent-session/SessionHandle.ts b/packages/trueforge-core/src/agent-session/SessionHandle.ts index e709a5846..e3c9979d3 100644 --- a/packages/trueforge-core/src/agent-session/SessionHandle.ts +++ b/packages/trueforge-core/src/agent-session/SessionHandle.ts @@ -600,9 +600,30 @@ export class SessionHandle< tracing: AgentTracing; }): CreateDynamicSubAgentThread { return async params => { + let request = params.request; + if (request.model === undefined && input.resolver.routeModel !== undefined) { + const routed = await input.resolver.routeModel({ + scope: 'subagent', + threadId: params.threadId, + fallbackModel: params.parentDefinition.modelName, + work: { + name: request.name, + text: request.input, + tools: [ + ...(input.spec.mcp_servers ?? []).map(server => server.name), + ...(input.spec.config.web_search.enabled && input.resolver.webSearchProvider ? ['web_search'] : []), + ...(input.sandbox ? ['sandbox'] : []), + ], + }, + signal: params.signal, + }); + if (routed.model !== undefined) { + request = { ...request, model: routed.model }; + } + } const { definition, extraCapabilities } = await input.resolver.resolveAgentDefinition({ spec: input.spec, - agent_info: params.request, + agent_info: request, previousTurn: input.previous, signal: params.signal, tracing: input.tracing, @@ -615,7 +636,7 @@ export class SessionHandle< // The delegated task goes in as the initial user message; the sub-agent // system prompt is SUB_AGENT_IDENTITY (added by AgentThread.buildInstruction). instruction: undefined, - messages: [{ role: 'user', content: params.request.input }], + messages: [{ role: 'user', content: request.input }], // Sub-agents should return free-form summaries to the parent, not the user-facing structured response. responseFormat: undefined, }; @@ -634,10 +655,10 @@ export class SessionHandle< return new AgentThread({ definition: childDefinition, threadId: params.threadId, - title: params.request.name, + title: request.name, sandbox: input.sandbox, parent: params.parent, - agentInfo: params.request, + agentInfo: request, capabilities, tracing: input.tracing, logger: input.resolver.logger, diff --git a/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts b/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts index 25bc1e3b9..1f426709c 100644 --- a/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts +++ b/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts @@ -14,7 +14,12 @@ import type { HarnessSandbox, SandboxInfo } from '../core/sandbox/Sandbox'; import type { AgentTracing } from '../core/tracing/AgentTracing'; import { NOOP_AGENT_TRACING } from '../core/tracing/NoopAgentTracing'; import type { IWebSearchProvider } from '../core/web-search/WebSearchProvider'; -import type { ITurnResourceResolver, ResolvedAgentDefinition } from './ITurnResourceResolver'; +import type { + ITurnResourceResolver, + ResolvedAgentDefinition, + RouteModelInput, + RouteModelResult, +} from './ITurnResourceResolver'; import type { TurnRecord } from './models/TurnRecord'; import type { AgentSpec } from './schemas/agentSpec'; @@ -80,6 +85,7 @@ export class TurnResourceResolver< readonly #models = new Map>(); #sandbox?: HarnessSandbox | undefined; readonly maxParallelSubAgents?: number; + readonly routeModel?: (input: RouteModelInput) => Promise; constructor( protected readonly deps: { @@ -110,6 +116,8 @@ export class TurnResourceResolver< * `resolveAgentDefinition` without calling `super` does not mount them. */ rootCapabilities?: (() => readonly AgentCapability[]) | undefined; + /** Model router for threads about to start; omit to disable routing. */ + routeModel?: ((input: RouteModelInput) => Promise) | undefined; /** Forwarded to RemoteMCP / AgentThread (required by their constructors). */ logger: Logger; }, @@ -117,6 +125,9 @@ export class TurnResourceResolver< if (deps.maxParallelSubAgents !== undefined) { this.maxParallelSubAgents = deps.maxParallelSubAgents; } + if (deps.routeModel !== undefined) { + this.routeModel = deps.routeModel; + } } get webSearchProvider(): IWebSearchProvider | undefined { diff --git a/packages/trueforge-core/src/agent-session/index.ts b/packages/trueforge-core/src/agent-session/index.ts index f420f70f0..74589f666 100644 --- a/packages/trueforge-core/src/agent-session/index.ts +++ b/packages/trueforge-core/src/agent-session/index.ts @@ -133,7 +133,7 @@ export { TurnOwnershipLostError, } from './store/SessionStoreErrors'; -export type { ITurnResourceResolver } from './ITurnResourceResolver'; +export type { ITurnResourceResolver, RouteModelInput, RouteModelResult, RoutingScope } from './ITurnResourceResolver'; export { TRUEFORGE_MCP_ID, TurnResourceResolver } from './TurnResourceResolver'; export type { TurnSandboxFactory } from './TurnResourceResolver'; diff --git a/packages/trueforge/.env.example b/packages/trueforge/.env.example index 87731cfdf..de52c70e9 100644 --- a/packages/trueforge/.env.example +++ b/packages/trueforge/.env.example @@ -307,6 +307,10 @@ POSTGRES_PORT=5432 ## Tenants omitted are unfiltered. Example: {"internal":["openai-main","anthropic-main"]} # TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS={"internal":["openai-main"]} +## Optional subagent model routing (JSON object). A decision model picks each subagent's model from the +## caller's scored models. Per-tenant mode: off (default) | shadow (log the pick only) | on. +# TRUEFOUNDRY_MODEL_ROUTING={"decision_model":"typesafe/jev-1.13.0","tenants":{"internal":{"subagent":"shadow"}}} + ## Internal mutual TLS for calls to the ServiceFoundry server. When true, trueforge presents its client ## certificate (and upgrades a mesh-direct http:// peer URL to https://). Only meaningful in-cluster, ## where TRUEFOUNDRY_SERVICEFOUNDRY_SERVER_URL points at the ServiceFoundry server Service directly. diff --git a/packages/trueforge/src/apis/turns.ts b/packages/trueforge/src/apis/turns.ts index 26c71d79d..17fb7a61b 100644 --- a/packages/trueforge/src/apis/turns.ts +++ b/packages/trueforge/src/apis/turns.ts @@ -75,7 +75,7 @@ import { type RequestContext, type ResolveRequestContext, } from '../auth/identity'; -import configuration from '../config'; +import configuration, { isTrueFoundryModeEnabled } from '../config'; import type { IAccessControlStore } from '../db/accessControlStore'; import type { AgentRecord, IAgentStore } from '../db/agentStore'; import { EmptyCapabilitiesStore, type ICapabilitiesStore } from '../db/capabilitiesStore'; @@ -91,6 +91,7 @@ import { currentLogContext, runWithLogContext, updateLogContext } from '../logge import { TRUEFORGE_SESSION_HEADER, X_TFG_TRUEFORGE_MCP } from '../mcp/server/trueforge/headers'; import type { TenantMemoryDeps } from '../memory/providers'; import { resolveSessionMemory } from '../memory/sessionMemory'; +import { createModelRouter } from '../modelRouting/router'; import { createAndExecuteTurnRoute, createInternalSessionTurnRoute, @@ -123,6 +124,7 @@ import { restoreTurnEventLog } from '../runtime/turnEventLogRestore'; import { MAX_SESSION_TITLE_LENGTH } from '../schemas/session'; import { assertGatewayMetadataRequestHeaders, X_TFY_METADATA } from '../truefoundry/gatewayMetadata'; import { X_TFG_MCP, X_TFG_MODELS, X_TFG_SKILLS, X_TFG_TOOLS } from '../truefoundry/inlineResources'; +import { createJevDecisionClient } from '../truefoundry/jevDecisionClient'; import { X_TFG_MCP_HEADERS } from '../truefoundry/perServerMcpHeaders'; import { newId } from '../utils/id'; import { resolveWebSearchProvider } from '../websearch/providers'; @@ -363,6 +365,47 @@ interface BeginTurnExecutionParams { deps: BeginTurnExecutionDeps; } +/** This turn's subagent model router, or undefined when routing is off for the tenant. */ +function subagentModelRouter(input: { + tenant_id: string; + modelProviderStore: IModelProviderStore; + turnMetadata: TurnMetadata; + logger: Logger; +}): ReturnType | undefined { + const { tenant_id, modelProviderStore, turnMetadata, logger } = input; + const routing = isTrueFoundryModeEnabled(configuration) ? configuration.TRUEFOUNDRY_MODEL_ROUTING : undefined; + const mode = routing?.tenants[tenant_id]?.subagent ?? 'off'; + const listRoutingCandidates = modelProviderStore.listRoutingCandidates?.bind(modelProviderStore); + if (routing === undefined || mode === 'off' || listRoutingCandidates === undefined) { + return undefined; + } + return createModelRouter({ + mode, + listCandidates: () => listRoutingCandidates({ tenant_id }), + decision: createJevDecisionClient({ + model: routing.decision_model, + resolveGateway: async model => { + const { providerConfig } = await getModelDetails({ + tenant_id, + name: model, + store: modelProviderStore, + turnMetadata, + }); + if (providerConfig.provider.type !== 'truefoundry' || providerConfig.baseUrl === undefined) { + throw new Error(`Model "${model}" is not served by the TrueFoundry gateway`); + } + return { + baseUrl: providerConfig.baseUrl, + headers: { Authorization: `Bearer ${providerConfig.apiKey}`, ...providerConfig.headers }, + }; + }, + }), + timeoutMs: routing.timeout_ms, + maxCandidates: routing.max_candidates, + logger, + }); +} + /** * Builds the per-turn resolver. Agent / MCP / sandbox / LLM lookups are wired * the same way: async factories over the corresponding stores. @@ -404,8 +447,10 @@ function createTurnResolver(deps: { } = deps; const tenant_id = session.tenant_id; const sessionId = session.session_id; + const routeModel = subagentModelRouter({ tenant_id, modelProviderStore, turnMetadata, logger }); return new TurnResourceResolver({ + ...(routeModel === undefined ? {} : { routeModel }), llm: async name => { const resolved = await getModelDetails({ tenant_id, diff --git a/packages/trueforge/src/config.ts b/packages/trueforge/src/config.ts index 18d893a6b..27eb33bf3 100644 --- a/packages/trueforge/src/config.ts +++ b/packages/trueforge/src/config.ts @@ -20,6 +20,7 @@ import { DEFAULT_MAX_PARALLEL_SUB_AGENTS } from '@truefoundry/trueforge-core/cor import envPaths from 'env-paths'; import { z } from 'zod'; +import { MODEL_ROUTING_MODES } from './modelRouting/types'; import { parseRedisSentinelNodes, type RedisConnection } from './runtime/redis'; export type { RedisConnection }; @@ -156,6 +157,30 @@ export function parseOidcAllowedEmails(raw: string | undefined): string[] { return parseCommaSeparatedEnvList(raw); } +const TrueFoundryModelRoutingSchema = z.object({ + decision_model: z.string().min(1), + timeout_ms: z.number().int().positive().default(2_000), + max_candidates: z.number().int().positive().default(60), + tenants: z.record(z.string(), z.object({ subagent: z.enum(MODEL_ROUTING_MODES).default('off') })).default({}), +}); + +export type TrueFoundryModelRouting = z.infer; + +/** Parses `TRUEFOUNDRY_MODEL_ROUTING` JSON. Empty / unset → undefined (routing off everywhere). */ +export function parseTrueFoundryModelRouting(raw: string | undefined): TrueFoundryModelRouting | undefined { + if (raw === undefined || raw.trim() === '') { + return undefined; + } + try { + return TrueFoundryModelRoutingSchema.parse(JSON.parse(raw)); + } catch (error) { + throw new Error( + 'Environment variable TRUEFOUNDRY_MODEL_ROUTING must be a JSON object with "decision_model" and per-tenant "tenants" modes (e.g. {"decision_model":"typesafe/jev-1.13.0","tenants":{"":{"subagent":"shadow"}}})', + { cause: error }, + ); + } +} + /** * Parses `TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS` JSON. * Empty / unset → `{}` (no filtering). @@ -1102,6 +1127,12 @@ export type DistributedServerConfiguration = SharedServerConfiguration & { * Env: `TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS`. */ TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS: Record; + /** + * Optional subagent model routing (TrueFoundry mode only). JSON object with the decision model, + * `timeout_ms`, `max_candidates`, and per-tenant modes (`off` | `shadow` | `on`). Unset → off for + * every tenant. Env: `TRUEFOUNDRY_MODEL_ROUTING`. + */ + TRUEFOUNDRY_MODEL_ROUTING: TrueFoundryModelRouting | undefined; /** * Optional built-in web search provider (TrueFoundry mode only). JSON object * `Record` with `name` (`parallel`) and `api_key`. @@ -1474,6 +1505,7 @@ const configuration: ServerConfiguration = standalone TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS: parseTenantIdToAllowedModelProviderAccounts( getEnv('TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS', { required: false }), ), + TRUEFOUNDRY_MODEL_ROUTING: parseTrueFoundryModelRouting(getEnv('TRUEFOUNDRY_MODEL_ROUTING', { required: false })), TRUEFOUNDRY_WEB_SEARCH_PROVIDER: parseTrueFoundryWebSearchProvider( getEnv('TRUEFOUNDRY_WEB_SEARCH_PROVIDER', { required: false }), ), diff --git a/packages/trueforge/src/db/modelProviderStore.ts b/packages/trueforge/src/db/modelProviderStore.ts index 314f99319..5bd7dc263 100644 --- a/packages/trueforge/src/db/modelProviderStore.ts +++ b/packages/trueforge/src/db/modelProviderStore.ts @@ -3,6 +3,7 @@ * `ModelProviderManifest` document. Implementations: Postgres, Sqlite, and * TrueFoundry (read-only ServiceFoundry server listing). */ +import type { RoutingCandidate } from '../modelRouting/types'; import type { ResourceName } from '../schemas/common'; import type { AvailableModel, ModelProperties, ModelProviderManifest } from '../schemas/modelProvider'; import type { TurnMetadata } from './turnMetadata'; @@ -85,6 +86,8 @@ export interface IModelProviderStore { record: ModelProviderRecord; turnMetadata?: TurnMetadata; }): Promise>; + /** The caller's models with the catalog fields model routing needs. Omitted by stores that can't route. */ + listRoutingCandidates?(input: ListModelProvidersInput): Promise; } export function modelPropertiesWarning(input: { properties: ModelProperties }): string | undefined { diff --git a/packages/trueforge/src/modelRouting/modelRoutingScores.ts b/packages/trueforge/src/modelRouting/modelRoutingScores.ts new file mode 100644 index 000000000..3501fd2e8 --- /dev/null +++ b/packages/trueforge/src/modelRouting/modelRoutingScores.ts @@ -0,0 +1,137 @@ +// Generated from the Artificial Analysis free API; regenerate rather than hand-editing. + +/** Intelligence Index v5 per Artificial Analysis slug. Effort variants are separate slugs (`-high`). */ +export const MODEL_ROUTING_SCORES: { + readonly retrievedAt: string; + readonly scores: Readonly>; +} = { + retrievedAt: '2026-11-13', + scores: { + 'agnes-2-5-pro-theta': { name: 'Agnes 2.5 Pro Theta', score: 24 }, + 'apodex-1-1': { name: 'Apodex 1.1', score: 13.7 }, + 'claude-4-5-haiku-reasoning': { name: 'Claude 4.5 Haiku', score: 7.4 }, + 'claude-fable-5-1': { name: 'Claude Fable 5.1', score: 49.2 }, + 'claude-fable-5-1-high': { name: 'Claude Fable 5.1', score: 47.2 }, + 'claude-fable-5-1-low': { name: 'Claude Fable 5.1', score: 38.7 }, + 'claude-fable-5-1-medium': { name: 'Claude Fable 5.1', score: 43.2 }, + 'claude-fable-5-1-xhigh': { name: 'Claude Fable 5.1', score: 50 }, + 'claude-fable-5-2': { name: 'Claude Fable 5.2', score: 64.6 }, + 'claude-fable-5-2-high': { name: 'Claude Fable 5.2', score: 61.8 }, + 'claude-fable-5-2-low': { name: 'Claude Fable 5.2', score: 53.9 }, + 'claude-fable-5-2-medium': { name: 'Claude Fable 5.2', score: 56.4 }, + 'claude-fable-5-2-xhigh': { name: 'Claude Fable 5.2', score: 62.2 }, + 'claude-haiku-5-5': { name: 'Claude Haiku 5.5', score: 36.5 }, + 'claude-haiku-5-5-high': { name: 'Claude Haiku 5.5', score: 29.9 }, + 'claude-haiku-5-5-low': { name: 'Claude Haiku 5.5', score: 18.6 }, + 'claude-haiku-5-5-medium': { name: 'Claude Haiku 5.5', score: 24.6 }, + 'claude-haiku-5-5-xhigh': { name: 'Claude Haiku 5.5', score: 34.6 }, + 'claude-opus-5': { name: 'Claude Opus 5', score: 44.2 }, + 'claude-opus-5-5': { name: 'Claude Opus 5.5', score: 59.1 }, + 'claude-opus-5-5-high': { name: 'Claude Opus 5.5', score: 51.8 }, + 'claude-opus-5-5-low': { name: 'Claude Opus 5.5', score: 34.8 }, + 'claude-opus-5-5-medium': { name: 'Claude Opus 5.5', score: 48.2 }, + 'claude-opus-5-5-xhigh': { name: 'Claude Opus 5.5', score: 57.4 }, + 'claude-sonnet-5-5': { name: 'Claude Sonnet 5.5', score: 54.5 }, + 'claude-sonnet-5-5-high': { name: 'Claude Sonnet 5.5', score: 41.5 }, + 'claude-sonnet-5-5-low': { name: 'Claude Sonnet 5.5', score: 25.6 }, + 'claude-sonnet-5-5-medium': { name: 'Claude Sonnet 5.5', score: 30.8 }, + 'claude-sonnet-5-5-xhigh': { name: 'Claude Sonnet 5.5', score: 49.9 }, + 'command-a-plus': { name: 'Command A+', score: 6.2 }, + 'deepseek-v4-1-flash-1017': { name: 'DeepSeek V4.1 Flash 1017', score: 32.6 }, + 'deepseek-v4-1-flash-1017-high': { name: 'DeepSeek V4.1 Flash 1017', score: 28.6 }, + 'deepseek-v4-1-flash-1017-non-reasoning': { name: 'DeepSeek V4.1 Flash 1017', score: 18.5 }, + 'deepseek-v4-pro': { name: 'DeepSeek V4 Pro 0813', score: 23.1 }, + 'deepseek-v4-pro-non-reasoning': { name: 'DeepSeek V4 Pro 0813', score: 12.6 }, + 'gemini-3-1-flash-lite-preview': { name: 'Gemini 3.1 Flash-Lite', score: 6.2 }, + 'gemini-3-5-flash-lite': { name: 'Gemini 3.5 Flash-Lite', score: 11.7 }, + 'gemini-3-8-flash': { name: 'Gemini 3.8 Flash', score: 30.8 }, + 'gemini-3-8-flash-low': { name: 'Gemini 3.8 Flash', score: 22.5 }, + 'gemini-3-8-flash-medium': { name: 'Gemini 3.8 Flash', score: 30.1 }, + 'gemini-4-argon': { name: 'Gemini 4 Argon', score: 50.4 }, + 'gemini-4-carbon': { name: 'Gemini 4 Carbon', score: 60 }, + 'gemini-4-carbon-low': { name: 'Gemini 4 Carbon', score: 54.2 }, + 'gemini-4-carbon-medium': { name: 'Gemini 4 Carbon', score: 58.3 }, + 'gemini-4-hydrogen': { name: 'Gemini 4 Hydrogen', score: 53.9 }, + 'gemini-4-hydrogen-high': { name: 'Gemini 4 Hydrogen', score: 50.8 }, + 'gemini-4-hydrogen-low': { name: 'Gemini 4 Hydrogen', score: 26.9 }, + 'gemini-4-hydrogen-medium': { name: 'Gemini 4 Hydrogen', score: 41.2 }, + 'gemma-4-31b': { name: 'Gemma 4 31B', score: 4.4 }, + 'glm-5-3': { name: 'GLM-5.3', score: 31.7 }, + 'glm-5-3-flash': { name: 'GLM-5.3-Flash', score: 27.8 }, + 'glm-5-3-low': { name: 'GLM-5.3', score: 23.1 }, + 'gpt-5-6-luna': { name: 'GPT-5.6 Luna', score: 23.8 }, + 'gpt-5-6-terra': { name: 'GPT-5.6 Terra', score: 32.3 }, + 'gpt-6-1-sol': { name: 'GPT-6.1 Sol', score: 54.6 }, + 'gpt-6-1-sol-high': { name: 'GPT-6.1 Sol', score: 50.3 }, + 'gpt-6-1-sol-low': { name: 'GPT-6.1 Sol', score: 34.8 }, + 'gpt-6-1-sol-medium': { name: 'GPT-6.1 Sol', score: 46.3 }, + 'gpt-6-1-sol-xhigh': { name: 'GPT-6.1 Sol', score: 52.5 }, + 'gpt-6-1-terra': { name: 'GPT-6.1 Terra', score: 51.3 }, + 'gpt-6-1-terra-high': { name: 'GPT-6.1 Terra', score: 42.4 }, + 'gpt-6-1-terra-low': { name: 'GPT-6.1 Terra', score: 29.4 }, + 'gpt-6-1-terra-medium': { name: 'GPT-6.1 Terra', score: 38.2 }, + 'gpt-6-1-terra-non-reasoning': { name: 'GPT-6.1 Terra', score: 30.5 }, + 'gpt-6-1-terra-xhigh': { name: 'GPT-6.1 Terra', score: 45.3 }, + 'gpt-6-astra': { name: 'GPT-6 Astra', score: 56.9 }, + 'gpt-6-astra-high': { name: 'GPT-6 Astra', score: 54.8 }, + 'gpt-6-astra-low': { name: 'GPT-6 Astra', score: 43.3 }, + 'gpt-6-astra-medium': { name: 'GPT-6 Astra', score: 51.6 }, + 'gpt-6-astra-xhigh': { name: 'GPT-6 Astra', score: 55.3 }, + 'gpt-6-luna': { name: 'GPT-6 Luna', score: 27.9 }, + 'gpt-6-luna-high': { name: 'GPT-6 Luna', score: 21.4 }, + 'gpt-6-luna-low': { name: 'GPT-6 Luna', score: 9.7 }, + 'gpt-6-luna-medium': { name: 'GPT-6 Luna', score: 18.5 }, + 'gpt-6-luna-non-reasoning': { name: 'GPT-6 Luna', score: 9.4 }, + 'gpt-6-luna-xhigh': { name: 'GPT-6 Luna', score: 23.5 }, + 'gpt-6-sol': { name: 'GPT-6 Sol', score: 44.2 }, + 'gpt-oss-120b': { name: 'gpt-oss-120b', score: 3.8 }, + 'granite-4-2-3b': { name: 'Granite 4.2 3B', score: 4.7 }, + 'granite-4-2-8b': { name: 'Granite 4.2 8B', score: 5.1 }, + 'grok-4-6-xhigh': { name: 'Grok 4.6', score: 31 }, + 'grok-4-7': { name: 'Grok 4.7', score: 35.5 }, + 'grok-4-7-high': { name: 'Grok 4.7', score: 35.2 }, + 'grok-4-7-low': { name: 'Grok 4.7', score: 31.1 }, + inkling: { name: 'Inkling', score: 12.5 }, + 'inkling-2-small': { name: 'Inkling 2 Small', score: 30.6 }, + 'inkling-small': { name: 'Inkling Small', score: 12.9 }, + 'jt-4-1-flash-236b-a21b': { name: 'JT-4.1 Flash 236B A21B', score: 19 }, + 'k-exaone-2-0': { name: 'K-EXAONE 2.0', score: 7.6 }, + 'k2-horizon-375b-a23b': { name: 'K2 Horizon 375B A23B', score: 17.8 }, + 'kimi-k3-1-pro': { name: 'Kimi K3.1 Pro', score: 46.5 }, + 'kimi-k3-low': { name: 'Kimi K3', score: 17.4 }, + 'ling-3-1-flash-1019': { name: 'Ling 3.1 Flash 1019', score: 29.6 }, + 'mimo-v2-6-flash': { name: 'MiMo-V2.6-Flash', score: 26.7 }, + 'mimo-v2-6-pro': { name: 'MiMo-V2.6-Pro', score: 29.9 }, + 'minimax-m3': { name: 'MiniMax-M3', score: 18.9 }, + 'minimax-m3-1-flash': { name: 'MiniMax-M3.1-Flash', score: 29.1 }, + 'mistral-large-4': { name: 'Mistral Large 4', score: 26.1 }, + 'mistral-large-4-medium': { name: 'Mistral Large 4', score: 23.1 }, + 'mistral-large-4-preview': { name: 'Mistral Large 4 Preview', score: 25.3 }, + 'mistral-medium-3-5': { name: 'Mistral Medium 3.5', score: 4.6 }, + 'mistral-small-4': { name: 'Mistral Small 4', score: 4.3 }, + 'muse-ember': { name: 'Muse Ember', score: 36 }, + 'muse-ember-xhigh': { name: 'Muse Ember', score: 34.1 }, + 'muse-glimmer': { name: 'Muse Glimmer', score: 6.3 }, + 'muse-spark-1-3': { name: 'Muse Spark 1.3', score: 36.6 }, + 'muse-spark-open': { name: 'Muse Spark Open', score: 24.9 }, + 'muse-wonder': { name: 'Muse Wonder', score: 43.6 }, + 'muse-wonder-full': { name: 'Muse Wonder Full', score: 50.1 }, + 'muse-wonder-high': { name: 'Muse Wonder', score: 40.5 }, + 'nanbeige4-3-33b': { name: 'Nanbeige4.3-33B', score: 20.4 }, + 'nemotron-3-5-lightning': { name: 'Nemotron 3.5 Lightning', score: 5.5 }, + 'nemotron-3-5-super-vl': { name: 'Nemotron 3.5 Super VL', score: 13.6 }, + 'north-mini-code': { name: 'North Mini Code', score: 3.4 }, + 'nvidia-nemotron-3-nano-4b': { name: 'NVIDIA Nemotron 3 Nano 4B', score: 1.8 }, + 'nvidia-nemotron-3-ultra-550b-a55b': { name: 'Nemotron 3 Ultra 550B A55B', score: 11.8 }, + 'qwen3-8-27b': { name: 'Qwen3.8 27B', score: 20.1 }, + 'qwen3-8-27b-low': { name: 'Qwen3.8 27B', score: 15.7 }, + 'qwen3-8-27b-medium': { name: 'Qwen3.8 27B', score: 17.5 }, + 'qwen3-8-27b-non-reasoning': { name: 'Qwen3.8 27B', score: 10 }, + 'qwen3-8-flash-next': { name: 'Qwen3.8-Flash-Next', score: 28.6 }, + 'qwen3-8-max': { name: 'Qwen3.8 Max', score: 33.7 }, + 'reflection-beam': { name: 'Beam 1.0', score: 17.7 }, + 'solar-pro4': { name: 'Solar Pro 4', score: 14.2 }, + 'step-5': { name: 'Step 5 Preview', score: 28.5 }, + 'tencent-hy4': { name: 'Tencent Hy4', score: 34.6 }, + }, +}; diff --git a/packages/trueforge/src/modelRouting/pick.ts b/packages/trueforge/src/modelRouting/pick.ts new file mode 100644 index 000000000..9264c4e3c --- /dev/null +++ b/packages/trueforge/src/modelRouting/pick.ts @@ -0,0 +1,20 @@ +export interface ScoredAnswer { + candidate: T; + pYes: number; + price?: number | undefined; +} + +/** Highest yes-probability wins; ties go to the cheaper model, then to the earlier candidate. */ +export function pickCandidate(answers: readonly ScoredAnswer[]): ScoredAnswer | undefined { + let best: ScoredAnswer | undefined; + for (const answer of answers) { + if ( + best === undefined || + answer.pYes > best.pYes || + (answer.pYes === best.pYes && (answer.price ?? Infinity) < (best.price ?? Infinity)) + ) { + best = answer; + } + } + return best; +} diff --git a/packages/trueforge/src/modelRouting/router.ts b/packages/trueforge/src/modelRouting/router.ts new file mode 100644 index 000000000..b4aaae9a6 --- /dev/null +++ b/packages/trueforge/src/modelRouting/router.ts @@ -0,0 +1,206 @@ +import type { RouteModelInput, RouteModelResult } from '@truefoundry/trueforge-core/agent-session'; +import type { Logger } from 'winston'; +import { pickCandidate } from './pick'; +import { lookupScore } from './scores'; +import type { DecisionClient, DecisionQuestion, ListRoutingCandidates, RoutingCandidate } from './types'; + +const TASK_CHARS = 8_000; + +const SCORE_NOTE = + 'Artificial Analysis Intelligence Index (0-100, higher is better) averages agentic knowledge work, coding, science and general reasoning evaluations.'; + +const CAN_COMPLETE: DecisionQuestion['criteria'] = { + true: 'It would produce a result that can be used as-is: complete, accurate, and with the needed tool use done.', + false: 'It would likely miss parts, make errors, or stop before the task is done.', +}; + +const RESEMBLES: DecisionQuestion['criteria'] = { + true: 'Same kind of work as the benchmark.', + false: 'A different kind of work.', +}; + +// Recorded only: tells us over time which public benchmark this work resembles. +const BENCHMARKS: Record = { + automation: { + name: 'AutomationBench', + description: 'Multi-step workflows across apps through API tools: look things up, combine results, act on them.', + example: "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner.", + }, + terminal_bench: { + name: 'Terminal-Bench', + description: + 'Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something.', + example: 'Find why the service fails to start from its logs and fix the config.', + }, + gdpval: { + name: 'GDPval', + description: 'Knowledge-work deliverables (reports, analyses, plans) produced with tools.', + example: 'From these project documents, write a one-page status report with risks and next steps.', + }, + lcr: { + name: 'AA-LCR', + description: 'Answer questions that need reasoning across long documents or large outputs.', + example: 'From these three long reports, which risk do all of them mention and how does each address it?', + }, + hle: { + name: "Humanity's Last Exam", + description: 'Hard expert-level knowledge and reasoning questions with one correct answer.', + example: 'A difficult graduate-level question in a technical field.', + }, +}; + +interface ScoredCandidate extends RoutingCandidate { + name: string; + score: number; +} + +type FallbackReason = 'no_candidates' | 'decision_error' | 'decision_timeout'; + +function clip(text: string): string { + if (text.length <= TASK_CHARS) { + return text; + } + const half = TASK_CHARS / 2; + return `${text.slice(0, half)}\n…\n${text.slice(-half)}`; +} + +function accountOf(model: string): string { + return model.split('/', 1)[0] ?? model; +} + +/** One candidate per scored slug, preferring the fallback model's account, capped by score. */ +function scoredCandidates(input: { + models: readonly RoutingCandidate[]; + fallbackModel: string; + maxCandidates: number; +}): ScoredCandidate[] { + const fallbackAccount = accountOf(input.fallbackModel); + const bySlug = new Map(); + const ordered = [...input.models].sort((a, b) => { + const preferred = Number(accountOf(b.model) === fallbackAccount) - Number(accountOf(a.model) === fallbackAccount); + return preferred !== 0 ? preferred : a.model.localeCompare(b.model); + }); + for (const model of ordered) { + if (model.slug === undefined || bySlug.has(model.slug)) { + continue; + } + const scored = lookupScore({ slug: model.slug, effort: model.defaultEffort }); + if (scored !== undefined) { + bySlug.set(model.slug, { ...model, ...scored }); + } + } + return [...bySlug.values()].sort((a, b) => b.score - a.score).slice(0, input.maxCandidates); +} + +function questions(candidates: readonly ScoredCandidate[]): Record { + const out: Record = {}; + candidates.forEach((candidate, index) => { + const effort = candidate.defaultEffort ?? 'its default'; + out[`can_${String(index)}`] = { + instructions: `Can ${candidate.name} at ${effort} effort (Intelligence Index ${String(candidate.score)}) complete this task fully and correctly?`, + criteria: CAN_COMPLETE, + }; + }); + for (const [key, benchmark] of Object.entries(BENCHMARKS)) { + out[`resembles_${key}`] = { + instructions: `Does this task resemble the benchmark ${benchmark.name}? ${benchmark.description} Example: '${benchmark.example}'`, + criteria: RESEMBLES, + }; + } + return out; +} + +function priceDirection(input: { picked?: number | undefined; fallback?: number | undefined }) { + if (input.picked === undefined || input.fallback === undefined) { + return undefined; + } + return input.picked > input.fallback ? 'up' : input.picked < input.fallback ? 'down' : 'same'; +} + +/** + * `routeModel` for one turn. Candidates are listed once and shared by the turn's subagents. + * Shadow mode logs the pick and keeps the fallback. Never throws. + */ +export function createModelRouter(input: { + mode: 'shadow' | 'on'; + listCandidates: ListRoutingCandidates; + decision: DecisionClient; + timeoutMs: number; + maxCandidates: number; + logger: Logger; +}): (route: RouteModelInput) => Promise { + let models: Promise | undefined; + + return async route => { + const startedMs = Date.now(); + const log = (fields: Record) => { + input.logger.info('model routing', { + scope: route.scope, + thread_id: route.threadId, + mode: input.mode, + fallback_model: route.fallbackModel, + latency_ms: Date.now() - startedMs, + ...fields, + }); + }; + const fallback = (reason: FallbackReason, extra: Record = {}): RouteModelResult => { + log({ reason, chosen_model: route.fallbackModel, ...extra }); + return {}; + }; + + const timeout = AbortSignal.timeout(input.timeoutMs); + try { + models ??= input.listCandidates().catch((error: unknown) => { + models = undefined; + throw error; + }); + const all = await models; + const candidates = scoredCandidates({ + models: all, + fallbackModel: route.fallbackModel, + maxCandidates: input.maxCandidates, + }); + if (candidates.length === 0) { + return fallback('no_candidates'); + } + const answers = await input.decision.ask({ + fallbackModel: route.fallbackModel, + threadId: route.threadId, + state: { + scope: route.scope, + name: route.work.name, + task: clip(route.work.text), + tools: route.work.tools, + benchmark_note: SCORE_NOTE, + }, + questions: questions(candidates), + signal: AbortSignal.any([route.signal, timeout]), + }); + const picked = pickCandidate( + candidates.flatMap((candidate, index) => { + const pYes = answers[`can_${String(index)}`]; + return pYes === undefined ? [] : [{ candidate, pYes, price: candidate.price }]; + }), + ); + if (picked === undefined) { + return fallback('decision_error', { candidate_count: candidates.length }); + } + const fallbackPrice = all.find(model => model.model === route.fallbackModel)?.price; + const chosen = input.mode === 'on' ? picked.candidate.model : route.fallbackModel; + log({ + reason: 'picked', + picked_model: picked.candidate.model, + chosen_model: chosen, + picked_p_yes: picked.pYes, + picked_price: picked.candidate.price, + fallback_price: fallbackPrice, + price_direction: priceDirection({ picked: picked.candidate.price, fallback: fallbackPrice }), + picked_image_input: picked.candidate.imageInput, + candidate_count: candidates.length, + }); + return input.mode === 'on' ? { model: picked.candidate.model } : {}; + } catch (error) { + return fallback(timeout.aborted ? 'decision_timeout' : 'decision_error', { err: error }); + } + }; +} diff --git a/packages/trueforge/src/modelRouting/scores.ts b/packages/trueforge/src/modelRouting/scores.ts new file mode 100644 index 000000000..1e2f23600 --- /dev/null +++ b/packages/trueforge/src/modelRouting/scores.ts @@ -0,0 +1,48 @@ +import { MODEL_ROUTING_SCORES } from './modelRoutingScores'; + +const EFFORTS = ['none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max']; + +type ScoreTable = Readonly>; + +function variantSlug({ slug, effort }: { slug: string; effort: string }): string { + return effort === 'none' ? `${slug}-non-reasoning` : `${slug}-${effort}`; +} + +/** + * Score for `slug` at `effort`: that exact variant, else the unlabelled slug, else the nearest + * effort (ties go higher). Variants match exactly, never by prefix, so `claude-sonnet-5` never + * reads `claude-sonnet-5-5-*`. + */ +export function lookupScore({ + slug, + effort, + scores = MODEL_ROUTING_SCORES.scores, +}: { + slug: string; + effort: string | undefined; + scores?: ScoreTable; +}): { name: string; score: number } | undefined { + const target = effort !== undefined && EFFORTS.includes(effort) ? effort : undefined; + if (target !== undefined) { + const exact = scores[variantSlug({ slug, effort: target })]; + if (exact !== undefined) { + return exact; + } + } + const unlabelled = scores[slug]; + if (unlabelled !== undefined) { + return unlabelled; + } + const from = EFFORTS.indexOf(target ?? 'medium'); + const byDistance = [...EFFORTS].sort((a, b) => { + const distance = Math.abs(EFFORTS.indexOf(a) - from) - Math.abs(EFFORTS.indexOf(b) - from); + return distance !== 0 ? distance : EFFORTS.indexOf(b) - EFFORTS.indexOf(a); + }); + for (const candidate of byDistance) { + const entry = scores[variantSlug({ slug, effort: candidate })]; + if (entry !== undefined) { + return entry; + } + } + return undefined; +} diff --git a/packages/trueforge/src/modelRouting/types.ts b/packages/trueforge/src/modelRouting/types.ts new file mode 100644 index 000000000..f80c115c9 --- /dev/null +++ b/packages/trueforge/src/modelRouting/types.ts @@ -0,0 +1,37 @@ +export const MODEL_ROUTING_MODES = ['off', 'shadow', 'on'] as const; + +export type ModelRoutingMode = (typeof MODEL_ROUTING_MODES)[number]; + +/** One of the caller's models. Only models whose slug has a score become routing candidates. */ +export interface RoutingCandidate { + /** `provider/model`, as the harness resolves it. */ + model: string; + /** Artificial Analysis slug; the score lookup key. */ + slug?: string | undefined; + /** Reasoning effort the model runs at when none is set. */ + defaultEffort?: string | undefined; + /** Blended USD per million tokens: 0.8 × input + 0.2 × output. */ + price?: number | undefined; + imageInput?: boolean | undefined; +} + +export type ListRoutingCandidates = () => Promise; + +export interface DecisionQuestion { + instructions: string; + criteria: { true: string; false: string }; +} + +export interface DecisionClient { + /** + * Yes-probability per question key. `fallbackModel` is the model whose provider config the call + * may borrow; keys the decision model didn't answer are omitted. + */ + ask(input: { + fallbackModel: string; + threadId: string; + state: Record; + questions: Record; + signal: AbortSignal; + }): Promise>; +} diff --git a/packages/trueforge/src/truefoundry/InlineModelProviderStore.ts b/packages/trueforge/src/truefoundry/InlineModelProviderStore.ts index f3d1181ac..ff6c5877d 100644 --- a/packages/trueforge/src/truefoundry/InlineModelProviderStore.ts +++ b/packages/trueforge/src/truefoundry/InlineModelProviderStore.ts @@ -9,6 +9,7 @@ import type { UpsertModelProviderInput, } from '../db/modelProviderStore'; import type { TurnMetadata } from '../db/turnMetadata'; +import type { RoutingCandidate } from '../modelRouting/types'; import type { AvailableModel } from '../schemas/modelProvider'; import { gatewayMetadataHeadersForTurn } from './gatewayMetadata'; import type { InlineModelProviders } from './inlineResources'; @@ -64,6 +65,11 @@ export class InlineModelProviderStore implements IModelPro return this.#inner.listModels(input, transaction); } + /** Request-scoped providers carry no catalog metadata, so only the inner store's models are routable. */ + listRoutingCandidates(input: ListModelProvidersInput): Promise { + return this.#inner.listRoutingCandidates?.(input) ?? Promise.resolve([]); + } + /** * The manifest carries its own `api_key`, which the adapter sends as the bearer, so no caller * token is resolved here. Mid-turn, still stamp `x-tfy-metadata` the same way registry diff --git a/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts b/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts index a7c18abe5..89e4a3734 100644 --- a/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts +++ b/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts @@ -13,6 +13,7 @@ import { type UpsertModelProviderInput, } from '../db/modelProviderStore'; import type { TurnMetadata } from '../db/turnMetadata'; +import type { RoutingCandidate } from '../modelRouting/types'; import type { AvailableModel, ModelProviderManifest } from '../schemas/modelProvider'; import { accessTokenForRequest, @@ -101,6 +102,16 @@ export class TrueFoundryModelProviderStore implements IMod return flattenProviderModels(await this.listProviders(input, transaction)); } + async listRoutingCandidates(input: ListModelProvidersInput): Promise { + const integrations = await this.#client.listProviderIntegrations({ + accessToken: await this.#resolveServiceFoundryAuthorization(), + }); + return filterEnvModels({ tenant_id: input.tenant_id, models: mapEnabledModels({ integrations }) }).map(model => ({ + model: `${model.accountName}/${model.modelName}`, + ...model.routing, + })); + } + async resolveInvokeHeaders(input: { record: ModelProviderRecord; turnMetadata?: TurnMetadata; diff --git a/packages/trueforge/src/truefoundry/jevDecisionClient.ts b/packages/trueforge/src/truefoundry/jevDecisionClient.ts new file mode 100644 index 000000000..b663d913a --- /dev/null +++ b/packages/trueforge/src/truefoundry/jevDecisionClient.ts @@ -0,0 +1,52 @@ +import { modelSsrfFetch } from '@truefoundry/trueforge-core/core'; +import { z } from 'zod'; +import type { DecisionClient } from '../modelRouting/types'; +import { gatewayMetadataHeaders, headerValue, parseGatewayMetadataHeader, X_TFY_METADATA } from './gatewayMetadata'; + +const SystemOneResponseSchema = z.object({ + answers: z.record(z.string(), z.object({ noul: z.number().nullish() })), +}); + +/** Gateway base URL and invoke headers (auth plus the turn's `x-tfy-metadata`) for a model. */ +export type ResolveGateway = (model: string) => Promise<{ baseUrl: string; headers: Record }>; + +/** Yes/no questions to a TrueFoundry decision model (Jev) through the gateway's `/v1/systemone`. */ +export function createJevDecisionClient(input: { model: string; resolveGateway: ResolveGateway }): DecisionClient { + return { + async ask({ fallbackModel, threadId, state, questions, signal }) { + const gateway = await input.resolveGateway(fallbackModel); + const turnMetadataHeader = headerValue(gateway.headers, X_TFY_METADATA); + const headers = { + ...Object.fromEntries(Object.entries(gateway.headers).filter(([key]) => key.toLowerCase() !== X_TFY_METADATA)), + ...gatewayMetadataHeaders({ + ...(turnMetadataHeader === undefined ? {} : parseGatewayMetadataHeader(turnMetadataHeader)), + 'tfg.thread_id': threadId, + 'tfg.model_routing': 'true', + }), + 'content-type': 'application/json', + }; + const root = gateway.baseUrl.replace(/\/+$/, '').replace(/\/v1$/, ''); + const response = await modelSsrfFetch(`${root}/v1/systemone`, { + method: 'POST', + headers, + body: JSON.stringify({ + model: input.model, + state, + questions: Object.fromEntries( + Object.entries(questions).map(([key, question]) => [key, { type: 'noul', ...question }]), + ), + }), + signal, + }); + if (!response.ok) { + throw new Error(`Decision model ${input.model} returned HTTP ${String(response.status)}`); + } + const { answers } = SystemOneResponseSchema.parse(await response.json()); + return Object.fromEntries( + Object.entries(answers).flatMap(([key, answer]) => + typeof answer.noul === 'number' ? [[key, answer.noul]] : [], + ), + ); + }, + }; +} diff --git a/packages/trueforge/src/truefoundry/mapEnabledModels.ts b/packages/trueforge/src/truefoundry/mapEnabledModels.ts index 32e0cfa2e..b49005b86 100644 --- a/packages/trueforge/src/truefoundry/mapEnabledModels.ts +++ b/packages/trueforge/src/truefoundry/mapEnabledModels.ts @@ -2,13 +2,21 @@ import { SUPPORTED_REASONING_EFFORTS } from '@truefoundry/trueforge-core/core'; import { HTTPException } from 'hono/http-exception'; import { z } from 'zod'; import configuration, { isTrueFoundryModeEnabled } from '../config'; +import type { RoutingCandidate } from '../modelRouting/types'; import type { ModelProperties, ReasoningEffort } from '../schemas/modelProvider'; const ModelParamSchema = z.object({ key: z.string(), supportedValues: z.array(z.string()).optional(), + defaultValue: z.unknown().optional(), }); +const TokenCostSchema = z.object({ + input_cost_per_token: z.number().nullish(), + output_cost_per_token: z.number().nullish(), +}); + +// Routing-only fields fall back to undefined on an unexpected shape, so they can never break listing. const MetadataSchema = z.object({ limits: z .object({ @@ -19,6 +27,16 @@ const MetadataSchema = z.object({ }) .optional(), params: z.array(ModelParamSchema).optional(), + external_ids: z.object({ artificial_analysis_slug: z.string().nullish() }).optional().catch(undefined), + public_cost: TokenCostSchema.optional().catch(undefined), + costs: z + .array(TokenCostSchema.extend({ region: z.string().nullish() })) + .optional() + .catch(undefined), + modalities: z + .object({ input: z.array(z.string()).optional() }) + .optional() + .catch(undefined), }); const IntegrationSchema = z.object({ @@ -61,10 +79,37 @@ function toProperties(metadata: IntegrationMetadata | undefined): ModelPropertie }; } +/** Blended USD per million tokens from the catalog price: the public price, else the global region, else the first priced one. */ +function toBlendedPrice(metadata: IntegrationMetadata | undefined): number | undefined { + const priced = [ + metadata?.public_cost, + ...(metadata?.costs?.filter(cost => cost.region === '*') ?? []), + ...(metadata?.costs ?? []), + ]; + for (const cost of priced) { + if (typeof cost?.input_cost_per_token === 'number' && typeof cost.output_cost_per_token === 'number') { + return (0.8 * cost.input_cost_per_token + 0.2 * cost.output_cost_per_token) * 1e6; + } + } + return undefined; +} + +function toRoutingMetadata(metadata: IntegrationMetadata | undefined): TrueFoundryEnabledModel['routing'] { + const defaultEffort = metadata?.params?.find(param => param.key === 'reasoning_effort')?.defaultValue; + return { + slug: metadata?.external_ids?.artificial_analysis_slug ?? undefined, + defaultEffort: typeof defaultEffort === 'string' ? defaultEffort : undefined, + price: toBlendedPrice(metadata), + imageInput: metadata?.modalities?.input?.includes('image'), + }; +} + export interface TrueFoundryEnabledModel { accountName: string; modelName: string; properties: ModelProperties; + /** Catalog fields the model router reads; never part of the public model listing. */ + routing: Omit; } export function mapEnabledModels(input: { integrations: readonly unknown[] }): TrueFoundryEnabledModel[] { @@ -78,6 +123,7 @@ export function mapEnabledModels(input: { integrations: readonly unknown[] }): T accountName: integration.providerAccount.name, modelName: integration.name, properties: toProperties(integration.metadata), + routing: toRoutingMetadata(integration.metadata), }); } return models; diff --git a/packages/trueforge/tests/unit/modelRouting/router.test.ts b/packages/trueforge/tests/unit/modelRouting/router.test.ts new file mode 100644 index 000000000..896050aed --- /dev/null +++ b/packages/trueforge/tests/unit/modelRouting/router.test.ts @@ -0,0 +1,52 @@ +import { createLogger } from 'winston'; +import { createModelRouter } from '../../../src/modelRouting/router'; +import type { DecisionClient } from '../../../src/modelRouting/types'; + +const logger = createLogger({ silent: true }); + +const candidates = [ + { model: 'acct/claude-haiku-5-5', slug: 'claude-haiku-5-5', defaultEffort: 'medium', price: 0.2 }, + { model: 'acct/claude-sonnet-5-5', slug: 'claude-sonnet-5-5', defaultEffort: 'high', price: 3.6 }, +]; + +function route(decision: DecisionClient, mode: 'shadow' | 'on') { + const routeModel = createModelRouter({ + mode, + listCandidates: () => Promise.resolve(candidates), + decision, + timeoutMs: 1_000, + maxCandidates: 60, + logger, + }); + return routeModel({ + scope: 'subagent', + threadId: 'child', + fallbackModel: 'acct/claude-opus-5-5', + work: { name: 'research', text: 'Compare pricing of three CI tools.', tools: [] }, + signal: new AbortController().signal, + }); +} + +const prefersHaiku: DecisionClient = { + ask: ({ questions }) => + Promise.resolve( + Object.fromEntries( + Object.entries(questions).map(([key, question]) => [key, question.instructions.includes('Haiku') ? 0.6 : 0.3]), + ), + ), +}; + +describe('createModelRouter', () => { + it('runs the highest yes-probability candidate in on mode', async () => { + await expect(route(prefersHaiku, 'on')).resolves.toEqual({ model: 'acct/claude-haiku-5-5' }); + }); + + it('keeps the fallback model in shadow mode', async () => { + await expect(route(prefersHaiku, 'shadow')).resolves.toEqual({}); + }); + + it('keeps the fallback model when the decision model fails', async () => { + const failing: DecisionClient = { ask: () => Promise.reject(new Error('HTTP 403')) }; + await expect(route(failing, 'on')).resolves.toEqual({}); + }); +}); diff --git a/packages/trueforge/tests/unit/modelRouting/scores.test.ts b/packages/trueforge/tests/unit/modelRouting/scores.test.ts new file mode 100644 index 000000000..b6b2f02c3 --- /dev/null +++ b/packages/trueforge/tests/unit/modelRouting/scores.test.ts @@ -0,0 +1,22 @@ +import { lookupScore } from '../../../src/modelRouting/scores'; + +const scores = { + 'model-a': { name: 'Model A', score: 40 }, + 'model-a-low': { name: 'Model A', score: 10 }, + 'model-a-medium': { name: 'Model A', score: 20 }, + 'model-b-low': { name: 'Model B', score: 5 }, + 'model-b-high': { name: 'Model B', score: 15 }, + 'model-a-5-high': { name: 'Model A 5', score: 99 }, +}; + +describe('lookupScore', () => { + it.each([ + { case: 'the exact effort variant', slug: 'model-a', effort: 'medium', score: 20 }, + { case: 'the unlabelled slug when the effort has no variant', slug: 'model-a', effort: 'max', score: 40 }, + { case: 'the nearest effort, ties going higher', slug: 'model-b', effort: 'medium', score: 15 }, + { case: 'its own score, never a longer slug that starts with it', slug: 'model-a', effort: 'high', score: 40 }, + { case: 'nothing for an unknown slug', slug: 'model', effort: 'medium', score: undefined }, + ])('returns $case', ({ slug, effort, score }) => { + expect(lookupScore({ slug, effort, scores })?.score).toBe(score); + }); +}); diff --git a/packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts b/packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts new file mode 100644 index 000000000..710ebe356 --- /dev/null +++ b/packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts @@ -0,0 +1,22 @@ +import { mapEnabledModels } from '../../../src/truefoundry/mapEnabledModels'; + +describe('mapEnabledModels', () => { + it('still lists a model whose routing metadata has an unexpected shape', () => { + const models = mapEnabledModels({ + integrations: [ + { + name: 'model-a', + manifest: { model_types: ['chat'] }, + providerAccount: { name: 'acct' }, + metadata: { + external_ids: 'not-an-object', + public_cost: 5, + costs: [{ input_cost_per_token: 'free' }], + modalities: null, + }, + }, + ], + }); + expect(models).toEqual([expect.objectContaining({ accountName: 'acct', modelName: 'model-a' })]); + }); +}); diff --git a/packages/trueforge/tests/unit/truefoundry/trueFoundryNaming.test.ts b/packages/trueforge/tests/unit/truefoundry/trueFoundryNaming.test.ts index 2b7400e19..b0cb1d486 100644 --- a/packages/trueforge/tests/unit/truefoundry/trueFoundryNaming.test.ts +++ b/packages/trueforge/tests/unit/truefoundry/trueFoundryNaming.test.ts @@ -313,7 +313,9 @@ describe('TrueFoundry naming vs NameSchema', () => { }, ], }); - expect(models).toEqual([{ accountName: PROVIDER_ACCOUNT, modelName: MODEL_NAME, properties: {} }]); + expect(models).toEqual([ + expect.objectContaining({ accountName: PROVIDER_ACCOUNT, modelName: MODEL_NAME, properties: {} }), + ]); const fqn = `${models[0]?.accountName}/${models[0]?.modelName}`; expect(fqn).toBe(`${PROVIDER_ACCOUNT}/${MODEL_NAME}`); expect(NameSchema.safeParse(fqn).success).toBe(false); From 2a2e0c4ddbd530d070ae140084d24a796d761fc8 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 15:10:03 +0530 Subject: [PATCH 07/14] refactor: address review on subagent model routing Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 69 +++++++++---------- .../agent-session/ITurnResourceResolver.ts | 15 ++-- .../src/agent-session/SessionHandle.ts | 5 +- .../src/agent-session/TurnResourceResolver.ts | 2 +- .../trueforge-core/src/agent-session/index.ts | 2 +- packages/trueforge/.env.example | 5 +- packages/trueforge/src/apis/turns.ts | 16 +++-- packages/trueforge/src/config.ts | 15 ++-- .../trueforge/src/db/modelProviderStore.ts | 5 +- .../trueforge/src/modelRouting/benchmarks.ts | 55 +++++++++++++++ packages/trueforge/src/modelRouting/pick.ts | 20 +++--- packages/trueforge/src/modelRouting/router.ts | 48 +++---------- packages/trueforge/src/modelRouting/scores.ts | 37 ++++++---- packages/trueforge/src/modelRouting/types.ts | 3 +- .../src/truefoundry/jevDecisionClient.ts | 1 - .../src/truefoundry/mapEnabledModels.ts | 15 ++-- .../tests/unit/modelRouting/router.test.ts | 3 +- 17 files changed, 176 insertions(+), 140 deletions(-) create mode 100644 packages/trueforge/src/modelRouting/benchmarks.ts diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md index e1883bb31..673b4761c 100644 --- a/docs/subagent-auto-routing/lld.md +++ b/docs/subagent-auto-routing/lld.md @@ -2,7 +2,7 @@ Status: ready for review. Scope: a PoC on internal TrueFoundry tenants, in TrueFoundry mode only. -When a subagent is spawned, one Jev call picks the model it runs on. The router is generic (it takes a `scope`), so the root agent can use it later (§8). Its modules are split so that model routing can later become a capability that also works outside TrueFoundry mode (§9). Only the subagent scope is built now. +When a subagent is spawned, one Jev call picks the model it runs on. Root-agent routing can reuse the same router later (§8). The modules are split so that model routing can later become a capability that also works outside TrueFoundry mode (§9). **Related work, tracked separately:** @@ -26,18 +26,18 @@ When a subagent is spawned, one Jev call picks the model it runs on. The router | I9 | **The routed model is applied through the existing `request.model` path** and saved on `agent_info.model`. Resume and rebuild never call the router. | §1 | | I10 | **One optional core hook,** `routeModel`. The router's logic lives in the server, split into provider-neutral and TrueFoundry-specific modules. | §1, §9 | | I11 | **Observability:** gateway traces joined on `tfg.thread_id`, plus one info log line per decision. No new tables or public API. | §6 | -| I12 | **Mode per tenant and per scope:** off, shadow or on. Off by default. | §5 | +| I12 | **Mode per tenant:** off, shadow or on. Off by default. | §5 | **Tunable without a design change:** -- **Config:** the decision model, the Jev timeout (2 s), `max_candidates` (60), and each tenant's mode. +- **Config:** the Jev timeout (2 s), `max_candidates` (60), and each tenant's mode and decision model. - **Data:** the score table, and the benchmark-resemblance list. ## What it does 1. **The parent calls `create_sub_agent`** with a name and a task. 2. **The router builds the candidates:** the caller's models that have a score (§2). -3. **One Jev call** asks, for each candidate, "can this model complete this task fully and correctly?" (§3). It also asks whether the task resembles each of five public benchmarks. Those answers are only recorded, to learn later which benchmark our work looks like. +3. **One Jev call** asks, for each candidate, "can this model complete this task fully and correctly?" (§3). It also asks whether the task resembles each of the nine benchmarks in the Intelligence Index. Those answers are only recorded, to learn later which kind of work our subagents do. 4. **The router picks** the candidate with the highest yes-probability (§4). In shadow mode it logs the pick and runs the parent's model. In "on" mode the subagent runs on the pick. 5. **If anything fails,** the subagent runs on the parent's model. @@ -46,7 +46,7 @@ flowchart TD S["create_sub_agent call
(makeCreateDynamicSubAgentThread)"] --> R{"routeModel wired?
(tenant's subagent mode is shadow/on)"} R -- no --> K["parent's model, as today"] R -- yes --> C["candidates: caller's models with a score"] - C --> J["Jev: POST {gateway}/v1/systemone
one question per candidate + 5 benchmark questions"] + C --> J["Jev: POST {gateway}/v1/systemone
one question per candidate + 9 benchmark questions"] J --> P["pick: highest P(yes), cheaper on a tie"] P --> L["one info log line"] L --> A["on: request.model = pick · shadow: parent's model"] @@ -56,18 +56,15 @@ flowchart TD ```ts // trueforge-core: agent-session/ITurnResourceResolver.ts -type RoutingScope = 'subagent' | 'root'; - interface RouteModelInput { - scope: RoutingScope; - threadId: string; // the thread being routed + threadId: string; // the subagent being routed fallbackModel: string; // the parent's model (account/model) - work: { + task: { name?: string; // subagent name - text: string; // the delegated task (self-contained by the create_sub_agent contract) + instructions: string; // the delegated task (self-contained by the create_sub_agent contract) tools: string[]; // MCP server names, web_search, sandbox }; - signal: AbortSignal; + signal: AbortSignal; // the subagent's abort signal: cancelling the turn cancels the Jev call } interface RouteModelResult { @@ -84,7 +81,7 @@ interface ITurnResourceResolver { ## 2. Candidates and scores -**The score is the Artificial Analysis Intelligence Index, v5** (0–100). It's the closest single score to agentic work: seven of its nine evaluations use tools. It averages four equal categories: Knowledge Work, Coding, Science and General. +**The score is the Artificial Analysis Intelligence Index, v5** (0–100). It's the closest single score to agentic work: six of its nine evaluations use tools. It averages four equal categories: Knowledge Work, Coding, Science and General. **The score table** is a typed TypeScript constant in the server package, `modelRouting/modelRoutingScores.ts`. It mirrors Artificial Analysis: one entry per Artificial Analysis slug, so each effort variant is its own entry: @@ -146,7 +143,6 @@ One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, t { "model": "typesafe/jev-1.13.0", "state": { - "scope": "subagent", "name": "", "task": "", "tools": ["", "web_search", "sandbox"], @@ -166,27 +162,31 @@ One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, t "instructions": "Can Claude Sonnet 5.5 at high effort (Intelligence Index 41.5) complete this task fully and correctly?", "criteria": { "...": "same as above" } }, - "resembles_gdpval": { + "resembles_automationbench": { "type": "noul", - "instructions": "Does this task resemble the benchmark GDPval? Knowledge-work deliverables (reports, analyses, plans) produced with tools. Example: 'From these project documents, write a one-page status report with risks and next steps.'", + "instructions": "Does this task resemble the benchmark AutomationBench? It tests whether models can complete realistic SaaS workflows that span multiple simulated business apps, using REST APIs as the tool interface.", "criteria": { "true": "Same kind of work as the benchmark.", "false": "A different kind of work." } } } } ``` -- **The decision model comes from config,** set per environment: `typesafe/jev-1.13.0` on internal. If a tenant can't reach it, the call fails and the parent's model runs (`decision_error`). +- **The decision model is configured per tenant,** because it's a model on the tenant's own gateway: `typesafe/jev-1.13.0` on internal. If the tenant can't reach it, the call fails and the parent's model runs (`decision_error`). - **Candidate questions** are keyed `can_`, and the router maps them back to models. - **Limits:** one call, with a 2-second timeout and at most `max_candidates` (60) candidate questions. -- **Benchmark-resemblance questions:** five benchmarks, all of which our key models have data for in `model-benchmarks`. The answers are recorded only. - -| Key | Benchmark | Description given to Jev | Example task given to Jev | -| ---------------- | -------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -| `automation` | AutomationBench-AA | Multi-step workflows across apps through API tools: look things up, combine results, act on them | "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner" | -| `terminal_bench` | Terminal-Bench 4.0 | Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something | "Find why the service fails to start from its logs and fix the config" | -| `gdpval` | GDPval-AA | Knowledge-work deliverables (reports, analyses, plans) produced with tools | "From these project documents, write a one-page status report with risks and next steps" | -| `lcr` | AA-LCR | Answer questions that need reasoning across long documents or large outputs | "From these three long reports, which risk do all of them mention and how does each address it?" | -| `hle` | Humanity's Last Exam | Hard expert-level knowledge and reasoning questions with one correct answer | "A difficult graduate-level question in a technical field" | +- **Benchmark-resemblance questions:** the nine evaluations in the Artificial Analysis Intelligence Index v5, the score routing uses, so the answers say which part of the index our subagent work resembles. The answers are recorded only. Each description is quoted verbatim from the benchmark's official page (`modelRouting/benchmarks.ts`): the [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking), and [lastexam.ai](https://lastexam.ai/) for HLE. + +| Category | Benchmark | +| -------------- | ---------------------- | +| Knowledge Work | AA-Briefcase | +| Knowledge Work | AutomationBench | +| Knowledge Work | GDP.pdf | +| Coding | FrontierCode | +| Coding | Terminal-Bench | +| Science | Terminal-Bench-Science | +| Science | CritPt | +| General | AA-Omniscience | +| General | Humanity's Last Exam | **Offline test** on 49 real subagent tasks × 11 candidates, checked against how each model actually did: @@ -212,16 +212,15 @@ One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, t ```json TRUEFOUNDRY_MODEL_ROUTING = { - "decision_model": "typesafe/jev-1.13.0", "timeout_ms": 2000, "max_candidates": 60, "tenants": { - "": { "subagent": "shadow" } + "": { "subagent": "shadow", "decision_model": "typesafe/jev-1.13.0" } } } ``` -- **off** (the default, or when unset): the hook isn't wired, and nothing changes. +- **off** (the default, or when the tenant isn't listed): the hook isn't wired, and nothing changes. - **shadow:** ask Jev and log the decision, but run the parent's model. - **on:** run the pick. @@ -229,7 +228,7 @@ TRUEFOUNDRY_MODEL_ROUTING = { - **Gateway traces,** joined on `tfg.thread_id`: - every model call a subagent makes (AGE-2837); - - the Jev call, which carries the turn metadata plus `tfg.thread_id` (the subagent being routed) and `tfg.model_routing: "true"`. + - the Jev call, which carries the turn metadata plus `tfg.thread_id` (the subagent being routed). Its model name identifies it as a routing call. The Jev request and response hold every candidate's question and probability, and the benchmark-resemblance answers. @@ -237,7 +236,6 @@ TRUEFOUNDRY_MODEL_ROUTING = { ```ts logger.info('model routing', { - scope, thread_id, mode, reason, @@ -268,7 +266,7 @@ TRUEFOUNDRY_MODEL_ROUTING = { The same router, score table, Jev request and pick rule. The root needs four things on top: -- **A call site** where the root thread's definition is built for a turn, with `scope: 'root'` and the spec's model as the fallback. +- **A call site** where the root thread's definition is built for a turn, with the spec's model as the fallback, and a `scope` field on the input to tell the two apart. - **Once per turn,** never mid-turn. - **Persistence:** save the routed model with the turn, so a resumed turn keeps it. - **Opt-in:** a per-tenant or per-agent opt-in. A model the user picked explicitly is never overridden. @@ -279,7 +277,7 @@ The same router, score table, Jev request and pick rule. The root needs four thi | File | Change | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `agent-session/ITurnResourceResolver.ts` | `RoutingScope`, `RouteModelInput`, `RouteModelResult` and the optional `routeModel` hook (§1). | +| `agent-session/ITurnResourceResolver.ts` | `RouteModelInput`, `RouteModelResult` and the optional `routeModel` hook (§1). | | `agent-session/TurnResourceResolver.ts` | Accept `routeModel` in its deps and expose it. | | `agent-session/SessionHandle.ts` | In `makeCreateDynamicSubAgentThread`, call `routeModel` when `request.model` is unset, and continue with `{ ...request, model }` if one comes back. | @@ -288,7 +286,7 @@ The same router, score table, Jev request and pick rule. The root needs four thi | File | Change | | --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **New** `types.ts` | What a provider supplies: `RoutingCandidate` (`model`, `slug`, `defaultEffort`, `price`, `imageInput`), a function that lists them, and `DecisionClient`, which asks yes/no questions and returns probabilities. Also the routing modes. | -| **New** `scores.ts` + `modelRoutingScores.ts` | The score table, and the lookup by slug and effort (§2). | +| **New** `scores.ts` + `modelRoutingScores.ts` | The score table, and the lookup by slug and effort (§2). `benchmarks.ts` holds the resemblance benchmarks (§3). | | **New** `pick.ts` | The pick rule, as a pure function from candidates and probabilities to a pick (§4). A threshold or price ceiling would change only this file and the config. | | **New** `router.ts` | `createModelRouter({ candidates, decision, scores, config, logger })`. It builds the questions (§3), calls the decision client, picks, logs (§6), and returns `routeModel`. It never throws. | @@ -312,5 +310,4 @@ No database schema or public API changes. 1. **How is quality measured?** The TrueForge backend doesn't store user feedback. Only the UI runtime passes a feedback adapter through. Feedback is also given per turn, on the root agent's answer, not per subagent. Before "on" mode we need to know where feedback lands on internal tenants, and how to attribute it to subagents. 2. **Model names in the questions:** keep them, or say only "a model with Intelligence Index S" to avoid Jev's preconceptions about names? -3. **Benchmark list:** are these five right? -4. **Threshold or price ceiling:** likely needed soon, if shadow data shows picks go up in price too often (§4). It's a change to `pick.ts` and config only. +3. **Threshold or price ceiling:** likely needed soon, if shadow data shows picks go up in price too often (§4). It's a change to `pick.ts` and config only. diff --git a/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts b/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts index 39f77548f..96330cabc 100644 --- a/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts +++ b/packages/trueforge-core/src/agent-session/ITurnResourceResolver.ts @@ -14,19 +14,18 @@ export interface ResolvedAgentDefinition { extraCapabilities?: AgentCapability[] | undefined; } -export type RoutingScope = 'subagent' | 'root'; - -/** The thread about to start, as seen by {@link ITurnResourceResolver.routeModel}. */ +/** Input to {@link ITurnResourceResolver.routeModel}: a subagent that's about to start, and the task it was given. */ export interface RouteModelInput { - scope: RoutingScope; threadId: string; - /** Model the thread runs on when routing doesn't pick one (`provider/model`). */ + /** Model the subagent runs on when routing doesn't pick one (`provider/model`): the parent's model. */ fallbackModel: string; - work: { + task: { name?: string | undefined; - text: string; + instructions: string; + /** MCP server names, plus `web_search` and `sandbox` when available. */ tools: string[]; }; + /** The subagent's abort signal; cancels routing when the turn is cancelled. */ signal: AbortSignal; } @@ -85,7 +84,7 @@ export interface ITurnResourceResolver; /** - * Chooses the model for a thread before it starts. Omit to keep every thread on its default model. + * Chooses the model for a subagent before it starts. Omit to keep every subagent on its parent's model. * Must not throw: on any failure, resolve without a model. */ routeModel?(input: RouteModelInput): Promise; diff --git a/packages/trueforge-core/src/agent-session/SessionHandle.ts b/packages/trueforge-core/src/agent-session/SessionHandle.ts index e3c9979d3..99a994c63 100644 --- a/packages/trueforge-core/src/agent-session/SessionHandle.ts +++ b/packages/trueforge-core/src/agent-session/SessionHandle.ts @@ -603,12 +603,11 @@ export class SessionHandle< let request = params.request; if (request.model === undefined && input.resolver.routeModel !== undefined) { const routed = await input.resolver.routeModel({ - scope: 'subagent', threadId: params.threadId, fallbackModel: params.parentDefinition.modelName, - work: { + task: { name: request.name, - text: request.input, + instructions: request.input, tools: [ ...(input.spec.mcp_servers ?? []).map(server => server.name), ...(input.spec.config.web_search.enabled && input.resolver.webSearchProvider ? ['web_search'] : []), diff --git a/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts b/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts index 1f426709c..359fa2a61 100644 --- a/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts +++ b/packages/trueforge-core/src/agent-session/TurnResourceResolver.ts @@ -116,7 +116,7 @@ export class TurnResourceResolver< * `resolveAgentDefinition` without calling `super` does not mount them. */ rootCapabilities?: (() => readonly AgentCapability[]) | undefined; - /** Model router for threads about to start; omit to disable routing. */ + /** Chooses a subagent's model before it starts; omit to keep subagents on their parent's model. */ routeModel?: ((input: RouteModelInput) => Promise) | undefined; /** Forwarded to RemoteMCP / AgentThread (required by their constructors). */ logger: Logger; diff --git a/packages/trueforge-core/src/agent-session/index.ts b/packages/trueforge-core/src/agent-session/index.ts index 74589f666..4d9550964 100644 --- a/packages/trueforge-core/src/agent-session/index.ts +++ b/packages/trueforge-core/src/agent-session/index.ts @@ -133,7 +133,7 @@ export { TurnOwnershipLostError, } from './store/SessionStoreErrors'; -export type { ITurnResourceResolver, RouteModelInput, RouteModelResult, RoutingScope } from './ITurnResourceResolver'; +export type { ITurnResourceResolver, RouteModelInput, RouteModelResult } from './ITurnResourceResolver'; export { TRUEFORGE_MCP_ID, TurnResourceResolver } from './TurnResourceResolver'; export type { TurnSandboxFactory } from './TurnResourceResolver'; diff --git a/packages/trueforge/.env.example b/packages/trueforge/.env.example index de52c70e9..d38b8fad2 100644 --- a/packages/trueforge/.env.example +++ b/packages/trueforge/.env.example @@ -308,8 +308,9 @@ POSTGRES_PORT=5432 # TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS={"internal":["openai-main"]} ## Optional subagent model routing (JSON object). A decision model picks each subagent's model from the -## caller's scored models. Per-tenant mode: off (default) | shadow (log the pick only) | on. -# TRUEFOUNDRY_MODEL_ROUTING={"decision_model":"typesafe/jev-1.13.0","tenants":{"internal":{"subagent":"shadow"}}} +## caller's scored models. Per tenant: mode off (default) | shadow (log the pick only) | on, and the tenant's +## own decision model on its gateway (account/model). +# TRUEFOUNDRY_MODEL_ROUTING={"tenants":{"internal":{"subagent":"shadow","decision_model":"typesafe/jev-1.13.0"}}} ## Internal mutual TLS for calls to the ServiceFoundry server. When true, trueforge presents its client ## certificate (and upgrades a mesh-direct http:// peer URL to https://). Only meaningful in-cluster, diff --git a/packages/trueforge/src/apis/turns.ts b/packages/trueforge/src/apis/turns.ts index 17fb7a61b..5a6c8f145 100644 --- a/packages/trueforge/src/apis/turns.ts +++ b/packages/trueforge/src/apis/turns.ts @@ -374,16 +374,20 @@ function subagentModelRouter(input: { }): ReturnType | undefined { const { tenant_id, modelProviderStore, turnMetadata, logger } = input; const routing = isTrueFoundryModeEnabled(configuration) ? configuration.TRUEFOUNDRY_MODEL_ROUTING : undefined; - const mode = routing?.tenants[tenant_id]?.subagent ?? 'off'; - const listRoutingCandidates = modelProviderStore.listRoutingCandidates?.bind(modelProviderStore); - if (routing === undefined || mode === 'off' || listRoutingCandidates === undefined) { + const tenantRouting = routing?.tenants[tenant_id]; + if ( + routing === undefined || + tenantRouting === undefined || + tenantRouting.subagent === 'off' || + modelProviderStore.listRoutingCandidates === undefined + ) { return undefined; } return createModelRouter({ - mode, - listCandidates: () => listRoutingCandidates({ tenant_id }), + mode: tenantRouting.subagent, + listCandidates: () => modelProviderStore.listRoutingCandidates?.({ tenant_id }) ?? Promise.resolve([]), decision: createJevDecisionClient({ - model: routing.decision_model, + model: tenantRouting.decision_model, resolveGateway: async model => { const { providerConfig } = await getModelDetails({ tenant_id, diff --git a/packages/trueforge/src/config.ts b/packages/trueforge/src/config.ts index 27eb33bf3..91c9107ac 100644 --- a/packages/trueforge/src/config.ts +++ b/packages/trueforge/src/config.ts @@ -158,10 +158,13 @@ export function parseOidcAllowedEmails(raw: string | undefined): string[] { } const TrueFoundryModelRoutingSchema = z.object({ - decision_model: z.string().min(1), timeout_ms: z.number().int().positive().default(2_000), max_candidates: z.number().int().positive().default(60), - tenants: z.record(z.string(), z.object({ subagent: z.enum(MODEL_ROUTING_MODES).default('off') })).default({}), + /** Per tenant: the decision model is the tenant's own gateway model (`account/model`). */ + tenants: z.record( + z.string(), + z.object({ subagent: z.enum(MODEL_ROUTING_MODES).default('off'), decision_model: z.string().min(1) }), + ), }); export type TrueFoundryModelRouting = z.infer; @@ -175,7 +178,7 @@ export function parseTrueFoundryModelRouting(raw: string | undefined): TrueFound return TrueFoundryModelRoutingSchema.parse(JSON.parse(raw)); } catch (error) { throw new Error( - 'Environment variable TRUEFOUNDRY_MODEL_ROUTING must be a JSON object with "decision_model" and per-tenant "tenants" modes (e.g. {"decision_model":"typesafe/jev-1.13.0","tenants":{"":{"subagent":"shadow"}}})', + 'Environment variable TRUEFOUNDRY_MODEL_ROUTING must be a JSON object with per-tenant "tenants" entries (e.g. {"tenants":{"":{"subagent":"shadow","decision_model":"typesafe/jev-1.13.0"}}})', { cause: error }, ); } @@ -1128,9 +1131,9 @@ export type DistributedServerConfiguration = SharedServerConfiguration & { */ TRUEFOUNDRY_TENANT_ID_TO_ALLOWED_MODEL_PROVIDER_ACCOUNTS: Record; /** - * Optional subagent model routing (TrueFoundry mode only). JSON object with the decision model, - * `timeout_ms`, `max_candidates`, and per-tenant modes (`off` | `shadow` | `on`). Unset → off for - * every tenant. Env: `TRUEFOUNDRY_MODEL_ROUTING`. + * Optional subagent model routing (TrueFoundry mode only). JSON object with `timeout_ms`, + * `max_candidates`, and per-tenant entries: mode (`off` | `shadow` | `on`) and the tenant's decision + * model. Unset, or a tenant not listed → off. Env: `TRUEFOUNDRY_MODEL_ROUTING`. */ TRUEFOUNDRY_MODEL_ROUTING: TrueFoundryModelRouting | undefined; /** diff --git a/packages/trueforge/src/db/modelProviderStore.ts b/packages/trueforge/src/db/modelProviderStore.ts index 5bd7dc263..0a6cea417 100644 --- a/packages/trueforge/src/db/modelProviderStore.ts +++ b/packages/trueforge/src/db/modelProviderStore.ts @@ -86,7 +86,10 @@ export interface IModelProviderStore { record: ModelProviderRecord; turnMetadata?: TurnMetadata; }): Promise>; - /** The caller's models with the catalog fields model routing needs. Omitted by stores that can't route. */ + /** + * The caller's models with the catalog fields model routing needs. Only the TrueFoundry-mode store + * implements it for now; stores that omit it never get routing. + */ listRoutingCandidates?(input: ListModelProvidersInput): Promise; } diff --git a/packages/trueforge/src/modelRouting/benchmarks.ts b/packages/trueforge/src/modelRouting/benchmarks.ts new file mode 100644 index 000000000..3d6300904 --- /dev/null +++ b/packages/trueforge/src/modelRouting/benchmarks.ts @@ -0,0 +1,55 @@ +// Descriptions are quoted verbatim, retrieved 2026-11-16; don't paraphrase them. Sources: +// https://artificialanalysis.ai/methodology/intelligence-benchmarking (all but HLE), https://lastexam.ai/ (HLE). + +/** + * The nine evaluations in the Artificial Analysis Intelligence Index v5, the score routing uses. + * Jev is asked which ones a task resembles; the answers are logged, never used to pick a model. + */ +export const RESEMBLANCE_BENCHMARKS: Readonly> = + { + briefcase: { + name: 'AA-Briefcase', + description: + 'AA-Briefcase is a new benchmark for testing models on realistic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files.', + }, + automationbench: { + name: 'AutomationBench', + description: + 'It tests whether models can complete realistic SaaS workflows that span multiple simulated business apps, using REST APIs as the tool interface.', + }, + gdp_pdf: { + name: 'GDP.pdf', + description: + 'A benchmark that tests whether models can reason over long, real-world professional documents and satisfy task-specific criteria.', + }, + frontiercode: { + name: 'FrontierCode', + description: + 'Each task asks an agent to make a change to a real open-source repository, and the benchmark measures whether the change is good enough for a maintainer to merge.', + }, + terminal_bench: { + name: 'Terminal-Bench', + description: + 'Covers software engineering, system administration, data processing, model training, and security, with each task graded by its own verification suite.', + }, + terminal_bench_science: { + name: 'Terminal-Bench-Science', + description: + 'Its research workflow tasks are drawn from real scientific practice, and experts curate and review each one. Each task has its own set of tests that the agent must pass by working in a terminal.', + }, + critpt: { + name: 'CritPt', + description: + 'Research-level physics reasoning benchmark with unpublished, frontier physics problems spanning a wide range of subfields.', + }, + omniscience: { + name: 'AA-Omniscience', + description: + 'AA-Omniscience is a knowledge and hallucination benchmark that measures factual reliability, rewards precise knowledge, and penalizes incorrect guesses or hallucinations.', + }, + hle: { + name: "Humanity's Last Exam", + description: + "Humanity's Last Exam, a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage.", + }, + }; diff --git a/packages/trueforge/src/modelRouting/pick.ts b/packages/trueforge/src/modelRouting/pick.ts index 9264c4e3c..8825f2748 100644 --- a/packages/trueforge/src/modelRouting/pick.ts +++ b/packages/trueforge/src/modelRouting/pick.ts @@ -4,17 +4,13 @@ export interface ScoredAnswer { price?: number | undefined; } -/** Highest yes-probability wins; ties go to the cheaper model, then to the earlier candidate. */ +/** Higher yes-probability first; among equal probabilities, cheaper first, and an unpriced model last. */ +function byPYesThenPrice(a: ScoredAnswer, b: ScoredAnswer): number { + const price = (answer: ScoredAnswer) => answer.price ?? Number.MAX_VALUE; + return b.pYes - a.pYes || price(a) - price(b); +} + +/** The answer with the highest yes-probability; ties go to the cheapest model. Undefined when there are no answers. */ export function pickCandidate(answers: readonly ScoredAnswer[]): ScoredAnswer | undefined { - let best: ScoredAnswer | undefined; - for (const answer of answers) { - if ( - best === undefined || - answer.pYes > best.pYes || - (answer.pYes === best.pYes && (answer.price ?? Infinity) < (best.price ?? Infinity)) - ) { - best = answer; - } - } - return best; + return [...answers].sort(byPYesThenPrice)[0]; } diff --git a/packages/trueforge/src/modelRouting/router.ts b/packages/trueforge/src/modelRouting/router.ts index b4aaae9a6..a65ba3fcf 100644 --- a/packages/trueforge/src/modelRouting/router.ts +++ b/packages/trueforge/src/modelRouting/router.ts @@ -1,8 +1,9 @@ import type { RouteModelInput, RouteModelResult } from '@truefoundry/trueforge-core/agent-session'; import type { Logger } from 'winston'; +import { RESEMBLANCE_BENCHMARKS } from './benchmarks'; import { pickCandidate } from './pick'; import { lookupScore } from './scores'; -import type { DecisionClient, DecisionQuestion, ListRoutingCandidates, RoutingCandidate } from './types'; +import type { DecisionClient, DecisionQuestion, RoutingCandidate } from './types'; const TASK_CHARS = 8_000; @@ -19,36 +20,6 @@ const RESEMBLES: DecisionQuestion['criteria'] = { false: 'A different kind of work.', }; -// Recorded only: tells us over time which public benchmark this work resembles. -const BENCHMARKS: Record = { - automation: { - name: 'AutomationBench', - description: 'Multi-step workflows across apps through API tools: look things up, combine results, act on them.', - example: "When a support ticket is tagged urgent, look up the customer's plan and notify the account owner.", - }, - terminal_bench: { - name: 'Terminal-Bench', - description: - 'Multi-step technical tasks in a shell: run commands, inspect files and logs, fix or configure something.', - example: 'Find why the service fails to start from its logs and fix the config.', - }, - gdpval: { - name: 'GDPval', - description: 'Knowledge-work deliverables (reports, analyses, plans) produced with tools.', - example: 'From these project documents, write a one-page status report with risks and next steps.', - }, - lcr: { - name: 'AA-LCR', - description: 'Answer questions that need reasoning across long documents or large outputs.', - example: 'From these three long reports, which risk do all of them mention and how does each address it?', - }, - hle: { - name: "Humanity's Last Exam", - description: 'Hard expert-level knowledge and reasoning questions with one correct answer.', - example: 'A difficult graduate-level question in a technical field.', - }, -}; - interface ScoredCandidate extends RoutingCandidate { name: string; score: number; @@ -101,9 +72,9 @@ function questions(candidates: readonly ScoredCandidate[]): Record Promise; decision: DecisionClient; timeoutMs: number; maxCandidates: number; @@ -135,7 +107,6 @@ export function createModelRouter(input: { const startedMs = Date.now(); const log = (fields: Record) => { input.logger.info('model routing', { - scope: route.scope, thread_id: route.threadId, mode: input.mode, fallback_model: route.fallbackModel, @@ -167,10 +138,9 @@ export function createModelRouter(input: { fallbackModel: route.fallbackModel, threadId: route.threadId, state: { - scope: route.scope, - name: route.work.name, - task: clip(route.work.text), - tools: route.work.tools, + name: route.task.name, + task: clip(route.task.instructions), + tools: route.task.tools, benchmark_note: SCORE_NOTE, }, questions: questions(candidates), diff --git a/packages/trueforge/src/modelRouting/scores.ts b/packages/trueforge/src/modelRouting/scores.ts index 1e2f23600..f5e643144 100644 --- a/packages/trueforge/src/modelRouting/scores.ts +++ b/packages/trueforge/src/modelRouting/scores.ts @@ -1,17 +1,33 @@ +import { SUPPORTED_REASONING_EFFORTS } from '@truefoundry/trueforge-core/core'; import { MODEL_ROUTING_SCORES } from './modelRoutingScores'; -const EFFORTS = ['none', 'minimal', 'low', 'medium', 'high', 'xhigh', 'max']; +type Effort = (typeof SUPPORTED_REASONING_EFFORTS)[number]; type ScoreTable = Readonly>; -function variantSlug({ slug, effort }: { slug: string; effort: string }): string { +const EFFORTS: readonly string[] = SUPPORTED_REASONING_EFFORTS; + +function isEffort(value: string | undefined): value is Effort { + return value !== undefined && EFFORTS.includes(value); +} + +/** Artificial Analysis publishes each effort as its own slug: `-high`, and `-non-reasoning` for `none`. */ +function variantSlug({ slug, effort }: { slug: string; effort: Effort }): string { return effort === 'none' ? `${slug}-non-reasoning` : `${slug}-${effort}`; } +/** Efforts ordered by closeness to `target`; of two equally close, the higher effort comes first. */ +function effortsNearest(target: Effort): Effort[] { + const rank = (effort: Effort) => SUPPORTED_REASONING_EFFORTS.indexOf(effort); + const distance = (effort: Effort) => Math.abs(rank(effort) - rank(target)); + return [...SUPPORTED_REASONING_EFFORTS].sort((a, b) => distance(a) - distance(b) || rank(b) - rank(a)); +} + /** - * Score for `slug` at `effort`: that exact variant, else the unlabelled slug, else the nearest - * effort (ties go higher). Variants match exactly, never by prefix, so `claude-sonnet-5` never - * reads `claude-sonnet-5-5-*`. + * The Intelligence Index for a model running at `effort`. Tries, in order: the exact effort variant + * (`-high`); the model's unlabelled score, which Artificial Analysis publishes at max effort; and the + * closest effort that has a score (from medium when `effort` is unknown). Slugs match exactly, so a model never + * picks up scores of a longer slug that starts with its own (`claude-sonnet-5` vs `claude-sonnet-5-5`). */ export function lookupScore({ slug, @@ -22,7 +38,7 @@ export function lookupScore({ effort: string | undefined; scores?: ScoreTable; }): { name: string; score: number } | undefined { - const target = effort !== undefined && EFFORTS.includes(effort) ? effort : undefined; + const target = isEffort(effort) ? effort : undefined; if (target !== undefined) { const exact = scores[variantSlug({ slug, effort: target })]; if (exact !== undefined) { @@ -33,13 +49,8 @@ export function lookupScore({ if (unlabelled !== undefined) { return unlabelled; } - const from = EFFORTS.indexOf(target ?? 'medium'); - const byDistance = [...EFFORTS].sort((a, b) => { - const distance = Math.abs(EFFORTS.indexOf(a) - from) - Math.abs(EFFORTS.indexOf(b) - from); - return distance !== 0 ? distance : EFFORTS.indexOf(b) - EFFORTS.indexOf(a); - }); - for (const candidate of byDistance) { - const entry = scores[variantSlug({ slug, effort: candidate })]; + for (const nearby of effortsNearest(target ?? 'medium')) { + const entry = scores[variantSlug({ slug, effort: nearby })]; if (entry !== undefined) { return entry; } diff --git a/packages/trueforge/src/modelRouting/types.ts b/packages/trueforge/src/modelRouting/types.ts index f80c115c9..0602b3a6c 100644 --- a/packages/trueforge/src/modelRouting/types.ts +++ b/packages/trueforge/src/modelRouting/types.ts @@ -12,11 +12,10 @@ export interface RoutingCandidate { defaultEffort?: string | undefined; /** Blended USD per million tokens: 0.8 × input + 0.2 × output. */ price?: number | undefined; + /** Logged with the pick only; never used to choose a model. */ imageInput?: boolean | undefined; } -export type ListRoutingCandidates = () => Promise; - export interface DecisionQuestion { instructions: string; criteria: { true: string; false: string }; diff --git a/packages/trueforge/src/truefoundry/jevDecisionClient.ts b/packages/trueforge/src/truefoundry/jevDecisionClient.ts index b663d913a..b9f3a534f 100644 --- a/packages/trueforge/src/truefoundry/jevDecisionClient.ts +++ b/packages/trueforge/src/truefoundry/jevDecisionClient.ts @@ -21,7 +21,6 @@ export function createJevDecisionClient(input: { model: string; resolveGateway: ...gatewayMetadataHeaders({ ...(turnMetadataHeader === undefined ? {} : parseGatewayMetadataHeader(turnMetadataHeader)), 'tfg.thread_id': threadId, - 'tfg.model_routing': 'true', }), 'content-type': 'application/json', }; diff --git a/packages/trueforge/src/truefoundry/mapEnabledModels.ts b/packages/trueforge/src/truefoundry/mapEnabledModels.ts index b49005b86..f075f18af 100644 --- a/packages/trueforge/src/truefoundry/mapEnabledModels.ts +++ b/packages/trueforge/src/truefoundry/mapEnabledModels.ts @@ -79,14 +79,15 @@ function toProperties(metadata: IntegrationMetadata | undefined): ModelPropertie }; } -/** Blended USD per million tokens from the catalog price: the public price, else the global region, else the first priced one. */ +/** + * Blended USD per million tokens, weighted 0.8 × input + 0.2 × output since agent loops read far more than + * they write. Uses the public price if set, else the global-region price, else the first region with a price. + */ function toBlendedPrice(metadata: IntegrationMetadata | undefined): number | undefined { - const priced = [ - metadata?.public_cost, - ...(metadata?.costs?.filter(cost => cost.region === '*') ?? []), - ...(metadata?.costs ?? []), - ]; - for (const cost of priced) { + const regionCosts = metadata?.costs ?? []; + const globalCosts = regionCosts.filter(cost => cost.region === '*'); + const inPreferenceOrder = [metadata?.public_cost, ...globalCosts, ...regionCosts]; + for (const cost of inPreferenceOrder) { if (typeof cost?.input_cost_per_token === 'number' && typeof cost.output_cost_per_token === 'number') { return (0.8 * cost.input_cost_per_token + 0.2 * cost.output_cost_per_token) * 1e6; } diff --git a/packages/trueforge/tests/unit/modelRouting/router.test.ts b/packages/trueforge/tests/unit/modelRouting/router.test.ts index 896050aed..a91d4494f 100644 --- a/packages/trueforge/tests/unit/modelRouting/router.test.ts +++ b/packages/trueforge/tests/unit/modelRouting/router.test.ts @@ -19,10 +19,9 @@ function route(decision: DecisionClient, mode: 'shadow' | 'on') { logger, }); return routeModel({ - scope: 'subagent', threadId: 'child', fallbackModel: 'acct/claude-opus-5-5', - work: { name: 'research', text: 'Compare pricing of three CI tools.', tools: [] }, + task: { name: 'research', instructions: 'Compare pricing of three CI tools.', tools: [] }, signal: new AbortController().signal, }); } From 56fab8f311f9715245538e01208c36b99e10fce4 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 15:17:02 +0530 Subject: [PATCH 08/14] refactor: resolve the decision model's own gateway config Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 20 ++++++------ packages/trueforge/src/modelRouting/router.ts | 1 - packages/trueforge/src/modelRouting/types.ts | 6 +--- .../TrueFoundryModelProviderStore.ts | 6 +++- .../src/truefoundry/jevDecisionClient.ts | 6 ++-- .../src/truefoundry/mapEnabledModels.ts | 9 ++++-- .../TrueFoundryModelProviderStore.test.ts | 32 +++++++++++++++++++ 7 files changed, 58 insertions(+), 22 deletions(-) diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md index 673b4761c..c50b7430c 100644 --- a/docs/subagent-auto-routing/lld.md +++ b/docs/subagent-auto-routing/lld.md @@ -137,7 +137,7 @@ interface ITurnResourceResolver { ## 3. The Jev request -One call per subagent to `POST {gateway}/v1/systemone`, using the gateway URL, token and turn headers from the parent model's provider config. There's no call when there are no candidates. +One call per subagent to `POST {gateway}/v1/systemone`, using Jev's own provider config: gateway URL, token and turn headers. The TrueFoundry store's by-name lookup resolves decision-type models as well as chat models, so `getModelDetails` works for Jev exactly as for any model. Model listings stay chat-only, so Jev never shows up in the model picker. There's no call when there are no candidates. ```json { @@ -292,15 +292,15 @@ The same router, score table, Jev request and pick rule. The root needs four thi **Server: TrueFoundry implementations.** -| File | Change | -| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `db/modelProviderStore.ts` | An optional `listRoutingCandidates()` on `IModelProviderStore`. Stores that can't supply catalogue metadata omit it, and routing isn't wired for them. | -| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with the fields in §2. Each new field falls back to undefined on an unexpected shape, so it can never break model listing. | -| `truefoundry/TrueFoundryModelProviderStore.ts` | `listRoutingCandidates()` for TrueFoundry mode, from the same `provider-integrations` fetch. | -| `truefoundry/InlineModelProviderStore.ts` | Forwards `listRoutingCandidates()` to the wrapped store. | -| **New** `truefoundry/jevDecisionClient.ts` | The `DecisionClient` for Jev: `POST {gateway}/v1/systemone` with the parent model's gateway URL, token and turn headers. | -| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING` (§5). | -| `apis/turns.ts` | When the tenant's subagent mode isn't off, create the router per turn from the TrueFoundry candidate source and the Jev client, and pass `routeModel` into the resolver. | +| File | Change | +| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `db/modelProviderStore.ts` | An optional `listRoutingCandidates()` on `IModelProviderStore`. Stores that can't supply catalogue metadata omit it, and routing isn't wired for them. | +| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with the fields in §2. Each new field falls back to undefined on an unexpected shape, so it can never break model listing. | +| `truefoundry/TrueFoundryModelProviderStore.ts` | `listRoutingCandidates()` for TrueFoundry mode, from the same `provider-integrations` fetch. The by-name lookup (`getProvider`) also resolves decision-type models, so Jev's config resolves like any model's; listings stay chat-only. | +| `truefoundry/InlineModelProviderStore.ts` | Forwards `listRoutingCandidates()` to the wrapped store. | +| **New** `truefoundry/jevDecisionClient.ts` | The `DecisionClient` for Jev: `POST {gateway}/v1/systemone` with Jev's own gateway URL, token and turn headers. | +| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING` (§5). | +| `apis/turns.ts` | When the tenant's subagent mode isn't off, create the router per turn from the TrueFoundry candidate source and the Jev client, and pass `routeModel` into the resolver. | No database schema or public API changes. diff --git a/packages/trueforge/src/modelRouting/router.ts b/packages/trueforge/src/modelRouting/router.ts index a65ba3fcf..f67febc46 100644 --- a/packages/trueforge/src/modelRouting/router.ts +++ b/packages/trueforge/src/modelRouting/router.ts @@ -135,7 +135,6 @@ export function createModelRouter(input: { return fallback('no_candidates'); } const answers = await input.decision.ask({ - fallbackModel: route.fallbackModel, threadId: route.threadId, state: { name: route.task.name, diff --git a/packages/trueforge/src/modelRouting/types.ts b/packages/trueforge/src/modelRouting/types.ts index 0602b3a6c..f2853abe4 100644 --- a/packages/trueforge/src/modelRouting/types.ts +++ b/packages/trueforge/src/modelRouting/types.ts @@ -22,12 +22,8 @@ export interface DecisionQuestion { } export interface DecisionClient { - /** - * Yes-probability per question key. `fallbackModel` is the model whose provider config the call - * may borrow; keys the decision model didn't answer are omitted. - */ + /** Yes-probability per question key; keys the decision model didn't answer are omitted. */ ask(input: { - fallbackModel: string; threadId: string; state: Record; questions: Record; diff --git a/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts b/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts index 89e4a3734..e274e87e2 100644 --- a/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts +++ b/packages/trueforge/src/truefoundry/TrueFoundryModelProviderStore.ts @@ -140,9 +140,13 @@ export class TrueFoundryModelProviderStore implements IMod this.#client.listGatewayInstallations(serviceFoundryAccessToken), ]); const gatewayUrl = resolveDefaultGatewayUrl(installations); + // A by-name lookup also resolves decision models (the model router's Jev); listings stay chat-only. const models = filterEnvModels({ tenant_id: input.tenant_id, - models: mapEnabledModels({ integrations }), + models: mapEnabledModels({ + integrations, + ...(input.filter !== undefined ? { modelTypes: ['chat', 'decision'] } : {}), + }), }); return toRecords({ tenant_id: input.tenant_id, diff --git a/packages/trueforge/src/truefoundry/jevDecisionClient.ts b/packages/trueforge/src/truefoundry/jevDecisionClient.ts index b9f3a534f..ad0d0db82 100644 --- a/packages/trueforge/src/truefoundry/jevDecisionClient.ts +++ b/packages/trueforge/src/truefoundry/jevDecisionClient.ts @@ -7,14 +7,14 @@ const SystemOneResponseSchema = z.object({ answers: z.record(z.string(), z.object({ noul: z.number().nullish() })), }); -/** Gateway base URL and invoke headers (auth plus the turn's `x-tfy-metadata`) for a model. */ +/** The model's gateway base URL and invoke headers (auth plus the turn's `x-tfy-metadata`). */ export type ResolveGateway = (model: string) => Promise<{ baseUrl: string; headers: Record }>; /** Yes/no questions to a TrueFoundry decision model (Jev) through the gateway's `/v1/systemone`. */ export function createJevDecisionClient(input: { model: string; resolveGateway: ResolveGateway }): DecisionClient { return { - async ask({ fallbackModel, threadId, state, questions, signal }) { - const gateway = await input.resolveGateway(fallbackModel); + async ask({ threadId, state, questions, signal }) { + const gateway = await input.resolveGateway(input.model); const turnMetadataHeader = headerValue(gateway.headers, X_TFY_METADATA); const headers = { ...Object.fromEntries(Object.entries(gateway.headers).filter(([key]) => key.toLowerCase() !== X_TFY_METADATA)), diff --git a/packages/trueforge/src/truefoundry/mapEnabledModels.ts b/packages/trueforge/src/truefoundry/mapEnabledModels.ts index f075f18af..eeb9c0167 100644 --- a/packages/trueforge/src/truefoundry/mapEnabledModels.ts +++ b/packages/trueforge/src/truefoundry/mapEnabledModels.ts @@ -113,11 +113,16 @@ export interface TrueFoundryEnabledModel { routing: Omit; } -export function mapEnabledModels(input: { integrations: readonly unknown[] }): TrueFoundryEnabledModel[] { +/** `modelTypes` defaults to chat only, so listings never show models agents can't run on. */ +export function mapEnabledModels(input: { + integrations: readonly unknown[]; + modelTypes?: readonly string[]; +}): TrueFoundryEnabledModel[] { + const modelTypes = input.modelTypes ?? ['chat']; const models: TrueFoundryEnabledModel[] = []; for (const row of input.integrations) { const integration = IntegrationSchema.parse(row); - if (!integration.manifest.model_types?.includes('chat')) { + if (!integration.manifest.model_types?.some(type => modelTypes.includes(type))) { continue; } models.push({ diff --git a/packages/trueforge/tests/unit/truefoundry/TrueFoundryModelProviderStore.test.ts b/packages/trueforge/tests/unit/truefoundry/TrueFoundryModelProviderStore.test.ts index 197db0ce8..83efec5b4 100644 --- a/packages/trueforge/tests/unit/truefoundry/TrueFoundryModelProviderStore.test.ts +++ b/packages/trueforge/tests/unit/truefoundry/TrueFoundryModelProviderStore.test.ts @@ -120,3 +120,35 @@ describe('TrueFoundryModelProviderStore authorization', () => { expect(client.vendToken).toHaveBeenCalledTimes(1); }); }); + +describe('TrueFoundryModelProviderStore decision models', () => { + it('resolves a decision model by name but never lists it', async () => { + const client = { + listProviderIntegrations: jest.fn().mockResolvedValue([ + { name: 'gpt-4', manifest: { model_types: ['chat'] }, providerAccount: { name: 'openai-main' } }, + { name: 'jev-1.13.0', manifest: { model_types: ['decision'] }, providerAccount: { name: 'typesafe' } }, + ]), + listGatewayInstallations: jest + .fn() + .mockResolvedValue([{ isDefault: true, manifest: { url: 'https://gateway.example' } }]), + vendToken: jest.fn().mockResolvedValue({ subjectToken: SUBJECT_TOKEN, actorToken: ACTOR_TOKEN }), + } as unknown as TrueFoundryServiceFoundryServerClient; + const store = new TrueFoundryModelProviderStore({ + client, + requestContext: createTrueFoundryRequestContext({ + tenant_id: TENANT, + subject: { id: 'user-1', type: 'user', display_name: 'User' }, + roles: [], + user_credential: CALLER_TOKEN, + }), + agent: undefined, + logger: createLogger({ silent: true }), + }); + + const listed = await store.listModels({ tenant_id: TENANT }); + expect(listed.map(model => model.name)).toEqual(['openai-main/gpt-4']); + await expect( + store.getProvider({ tenant_id: TENANT, name: 'typesafe', model_name: 'jev-1.13.0' }), + ).resolves.toMatchObject({ name: 'typesafe' }); + }); +}); From e3652ab639ce83a3a03ff21fbc7a1c3bc9157d08 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 15:26:08 +0530 Subject: [PATCH 09/14] docs: trim lookupScore description Co-authored-by: Cursor --- packages/trueforge/src/modelRouting/scores.ts | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/packages/trueforge/src/modelRouting/scores.ts b/packages/trueforge/src/modelRouting/scores.ts index f5e643144..afaa32675 100644 --- a/packages/trueforge/src/modelRouting/scores.ts +++ b/packages/trueforge/src/modelRouting/scores.ts @@ -26,8 +26,7 @@ function effortsNearest(target: Effort): Effort[] { /** * The Intelligence Index for a model running at `effort`. Tries, in order: the exact effort variant * (`-high`); the model's unlabelled score, which Artificial Analysis publishes at max effort; and the - * closest effort that has a score (from medium when `effort` is unknown). Slugs match exactly, so a model never - * picks up scores of a longer slug that starts with its own (`claude-sonnet-5` vs `claude-sonnet-5-5`). + * closest effort that has a score (from medium when `effort` is unknown). */ export function lookupScore({ slug, From 7c2a7d2f1f80b82597f15c8ba4602e40fad45c61 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 15:39:59 +0530 Subject: [PATCH 10/14] test: drop lookup and mapping tests; tighten router guardrails Co-authored-by: Cursor --- .../tests/unit/modelRouting/router.test.ts | 27 ++++++++++++------- .../tests/unit/modelRouting/scores.test.ts | 22 --------------- .../unit/truefoundry/mapEnabledModels.test.ts | 22 --------------- 3 files changed, 18 insertions(+), 53 deletions(-) delete mode 100644 packages/trueforge/tests/unit/modelRouting/scores.test.ts delete mode 100644 packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts diff --git a/packages/trueforge/tests/unit/modelRouting/router.test.ts b/packages/trueforge/tests/unit/modelRouting/router.test.ts index a91d4494f..64ba50745 100644 --- a/packages/trueforge/tests/unit/modelRouting/router.test.ts +++ b/packages/trueforge/tests/unit/modelRouting/router.test.ts @@ -4,6 +4,7 @@ import type { DecisionClient } from '../../../src/modelRouting/types'; const logger = createLogger({ silent: true }); +// Haiku is listed first and is cheaper, so a tie would pick it; only Jev's answer can pick Sonnet. const candidates = [ { model: 'acct/claude-haiku-5-5', slug: 'claude-haiku-5-5', defaultEffort: 'medium', price: 0.2 }, { model: 'acct/claude-sonnet-5-5', slug: 'claude-sonnet-5-5', defaultEffort: 'high', price: 3.6 }, @@ -26,22 +27,30 @@ function route(decision: DecisionClient, mode: 'shadow' | 'on') { }); } -const prefersHaiku: DecisionClient = { - ask: ({ questions }) => - Promise.resolve( - Object.fromEntries( - Object.entries(questions).map(([key, question]) => [key, question.instructions.includes('Haiku') ? 0.6 : 0.3]), +function prefersSonnet(): DecisionClient & { ask: jest.Mock } { + return { + ask: jest.fn(({ questions }: Parameters[0]) => + Promise.resolve( + Object.fromEntries( + Object.entries(questions).map(([key, question]) => [ + key, + question.instructions.includes('Sonnet') ? 0.6 : 0.3, + ]), + ), ), ), -}; + }; +} describe('createModelRouter', () => { it('runs the highest yes-probability candidate in on mode', async () => { - await expect(route(prefersHaiku, 'on')).resolves.toEqual({ model: 'acct/claude-haiku-5-5' }); + await expect(route(prefersSonnet(), 'on')).resolves.toEqual({ model: 'acct/claude-sonnet-5-5' }); }); - it('keeps the fallback model in shadow mode', async () => { - await expect(route(prefersHaiku, 'shadow')).resolves.toEqual({}); + it('asks the decision model but keeps the fallback model in shadow mode', async () => { + const decision = prefersSonnet(); + await expect(route(decision, 'shadow')).resolves.toEqual({}); + expect(decision.ask).toHaveBeenCalledTimes(1); }); it('keeps the fallback model when the decision model fails', async () => { diff --git a/packages/trueforge/tests/unit/modelRouting/scores.test.ts b/packages/trueforge/tests/unit/modelRouting/scores.test.ts deleted file mode 100644 index b6b2f02c3..000000000 --- a/packages/trueforge/tests/unit/modelRouting/scores.test.ts +++ /dev/null @@ -1,22 +0,0 @@ -import { lookupScore } from '../../../src/modelRouting/scores'; - -const scores = { - 'model-a': { name: 'Model A', score: 40 }, - 'model-a-low': { name: 'Model A', score: 10 }, - 'model-a-medium': { name: 'Model A', score: 20 }, - 'model-b-low': { name: 'Model B', score: 5 }, - 'model-b-high': { name: 'Model B', score: 15 }, - 'model-a-5-high': { name: 'Model A 5', score: 99 }, -}; - -describe('lookupScore', () => { - it.each([ - { case: 'the exact effort variant', slug: 'model-a', effort: 'medium', score: 20 }, - { case: 'the unlabelled slug when the effort has no variant', slug: 'model-a', effort: 'max', score: 40 }, - { case: 'the nearest effort, ties going higher', slug: 'model-b', effort: 'medium', score: 15 }, - { case: 'its own score, never a longer slug that starts with it', slug: 'model-a', effort: 'high', score: 40 }, - { case: 'nothing for an unknown slug', slug: 'model', effort: 'medium', score: undefined }, - ])('returns $case', ({ slug, effort, score }) => { - expect(lookupScore({ slug, effort, scores })?.score).toBe(score); - }); -}); diff --git a/packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts b/packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts deleted file mode 100644 index 710ebe356..000000000 --- a/packages/trueforge/tests/unit/truefoundry/mapEnabledModels.test.ts +++ /dev/null @@ -1,22 +0,0 @@ -import { mapEnabledModels } from '../../../src/truefoundry/mapEnabledModels'; - -describe('mapEnabledModels', () => { - it('still lists a model whose routing metadata has an unexpected shape', () => { - const models = mapEnabledModels({ - integrations: [ - { - name: 'model-a', - manifest: { model_types: ['chat'] }, - providerAccount: { name: 'acct' }, - metadata: { - external_ids: 'not-an-object', - public_cost: 5, - costs: [{ input_cost_per_token: 'free' }], - modalities: null, - }, - }, - ], - }); - expect(models).toEqual([expect.objectContaining({ accountName: 'acct', modelName: 'model-a' })]); - }); -}); From 97a4952ec9ec6c20255a6eed4aa9ace0b1ca0e57 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 15:45:19 +0530 Subject: [PATCH 11/14] docs: drop the routing LLD from this PR Co-authored-by: Cursor --- docs/subagent-auto-routing/lld.md | 313 ------------------------------ 1 file changed, 313 deletions(-) delete mode 100644 docs/subagent-auto-routing/lld.md diff --git a/docs/subagent-auto-routing/lld.md b/docs/subagent-auto-routing/lld.md deleted file mode 100644 index c50b7430c..000000000 --- a/docs/subagent-auto-routing/lld.md +++ /dev/null @@ -1,313 +0,0 @@ -# Subagent model routing with Jev: low-level design - -Status: ready for review. Scope: a PoC on internal TrueFoundry tenants, in TrueFoundry mode only. - -When a subagent is spawned, one Jev call picks the model it runs on. Root-agent routing can reuse the same router later (§8). The modules are split so that model routing can later become a capability that also works outside TrueFoundry mode (§9). - -**Related work, tracked separately:** - -- **`tfg.thread_id` on every model call** (AGE-2837, PR #1392). Routing analysis joins on it. -- **Subagent model overrides** (AGE-2841). A subagent on a different model must use that model's own default params, not the parent's, and `THREAD_CREATED` must save the model the subagent actually runs on. Shadow mode doesn't need this. "On" mode does. - -## Decisions at a glance - -**Fixed by this design.** Changing any of these means a new design review. - -| # | Invariant | Where | -| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | -| I1 | **One Jev call per subagent, made before it starts.** Never during the agent loop. | §1 | -| I2 | **Jev sees only the task and the candidate pool.** One yes/no question per candidate (name, effort, score), plus one per benchmark. No parent model, prices or comparisons. | §3 | -| I3 | **The pick is the candidate with the highest yes-probability,** the cheaper one on a tie. There's no threshold and no price ceiling yet. | §4 | -| I4 | **The fallback is the parent's model.** It runs only when routing can't produce a pick. | §4 | -| I5 | **Candidates are the caller's models that have a score.** No other metadata filters candidates. Other fields are read only for the log line. | §2 | -| I6 | **Scores are bundled data,** keyed by Artificial Analysis slug. No Artificial Analysis call while a request runs. | §2 | -| I7 | **Jev sees each candidate's score at the effort it will actually run at:** its default effort on TrueFoundry. | §2 | -| I8 | **The router never throws.** Any error, timeout or unusable pick means the parent's model runs, as today. | §4 | -| I9 | **The routed model is applied through the existing `request.model` path** and saved on `agent_info.model`. Resume and rebuild never call the router. | §1 | -| I10 | **One optional core hook,** `routeModel`. The router's logic lives in the server, split into provider-neutral and TrueFoundry-specific modules. | §1, §9 | -| I11 | **Observability:** gateway traces joined on `tfg.thread_id`, plus one info log line per decision. No new tables or public API. | §6 | -| I12 | **Mode per tenant:** off, shadow or on. Off by default. | §5 | - -**Tunable without a design change:** - -- **Config:** the Jev timeout (2 s), `max_candidates` (60), and each tenant's mode and decision model. -- **Data:** the score table, and the benchmark-resemblance list. - -## What it does - -1. **The parent calls `create_sub_agent`** with a name and a task. -2. **The router builds the candidates:** the caller's models that have a score (§2). -3. **One Jev call** asks, for each candidate, "can this model complete this task fully and correctly?" (§3). It also asks whether the task resembles each of the nine benchmarks in the Intelligence Index. Those answers are only recorded, to learn later which kind of work our subagents do. -4. **The router picks** the candidate with the highest yes-probability (§4). In shadow mode it logs the pick and runs the parent's model. In "on" mode the subagent runs on the pick. -5. **If anything fails,** the subagent runs on the parent's model. - -```mermaid -flowchart TD - S["create_sub_agent call
(makeCreateDynamicSubAgentThread)"] --> R{"routeModel wired?
(tenant's subagent mode is shadow/on)"} - R -- no --> K["parent's model, as today"] - R -- yes --> C["candidates: caller's models with a score"] - C --> J["Jev: POST {gateway}/v1/systemone
one question per candidate + 9 benchmark questions"] - J --> P["pick: highest P(yes), cheaper on a tie"] - P --> L["one info log line"] - L --> A["on: request.model = pick · shadow: parent's model"] -``` - -## 1. The router contract - -```ts -// trueforge-core: agent-session/ITurnResourceResolver.ts -interface RouteModelInput { - threadId: string; // the subagent being routed - fallbackModel: string; // the parent's model (account/model) - task: { - name?: string; // subagent name - instructions: string; // the delegated task (self-contained by the create_sub_agent contract) - tools: string[]; // MCP server names, web_search, sandbox - }; - signal: AbortSignal; // the subagent's abort signal: cancelling the turn cancels the Jev call -} - -interface RouteModelResult { - model?: string; // set only in "on" mode when a candidate was picked -} - -interface ITurnResourceResolver { - // ...existing methods - routeModel?(input: RouteModelInput): Promise; // never throws; undefined = today -} -``` - -**The call site:** `SessionHandle.makeCreateDynamicSubAgentThread`. If `request.model` is unset, it builds the input and calls `routeModel`. If a model comes back, it continues with `{ ...request, model }`. From there, the existing override path does the rest: `resolveAgentDefinition` resolves `agent_info.model`, and `THREAD_CREATED` saves it (after AGE-2841). A rebuilt or resumed subagent uses its saved model and never calls the router. - -## 2. Candidates and scores - -**The score is the Artificial Analysis Intelligence Index, v5** (0–100). It's the closest single score to agentic work: six of its nine evaluations use tools. It averages four equal categories: Knowledge Work, Coding, Science and General. - -**The score table** is a typed TypeScript constant in the server package, `modelRouting/modelRoutingScores.ts`. It mirrors Artificial Analysis: one entry per Artificial Analysis slug, so each effort variant is its own entry: - -```ts -'claude-haiku-5-5': { name: 'Claude Haiku 5.5', score: 36.5 }, // plain slug: Artificial Analysis labels it (Max) -'claude-haiku-5-5-medium': { name: 'Claude Haiku 5.5', score: 24.6 }, -'mimo-v2-6-pro': { name: 'MiMo-V2.6-Pro', score: 29.9 }, -``` - -- **How it's produced:** generated once for the PoC from the Artificial Analysis free endpoint (`/api/v2/language/models/free`), and reviewed like code. A refresh script can come later. -- **Why a TypeScript constant:** it's bundled like the shipped catalogues, and a malformed entry fails at compile time. No loader is needed. -- **The name** is Artificial Analysis's model name without the effort label. It's what Jev sees in the question. - -**Candidates are built once per turn from the caller's model list.** An internal `listRoutingCandidates()` on `TrueFoundryModelProviderStore` shares `listModels`' `provider-integrations` fetch. It reads these fields from each model's metadata: - -| Field | Used for | -| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | -| `external_ids.artificial_analysis_slug` | Score lookup. A model without a slug, or with one that isn't in the table, isn't a candidate. | -| `params[reasoning_effort].defaultValue` | Which effort's score Jev sees. | -| Price (`public_cost` or `costs`), blended 0.8 × input + 0.2 × output | Breaking ties, and the log line. It's also needed if a threshold or price ceiling is added. | -| `modalities.input`, `features` | The log line only. No filtering. | - -- **The internal type is widened:** `TrueFoundryEnabledModel` gains these fields. `listModels` and the public `AvailableModel` are unchanged. -- **How a score is looked up:** the exact variant `-` (`-non-reasoning` for `none`). If there isn't one, use the plain slug, then the nearest effort variant, with ties going to the higher one. Variants match exactly, never by prefix. A prefix match would let `claude-sonnet-5` read `claude-sonnet-5-5-*`. -- **A model served by several accounts is asked about once** (same slug), preferring the parent model's account, then its provider, then the first by name. -- **The parent's model is a candidate like any other** if it has a score. If it doesn't, routing still runs among the scored candidates. -- **Revoked access:** candidates come from the caller's model list for this turn, so a pick is a model the caller could use when the turn started. Access revoked mid-turn behaves as it does today for any model: the gateway rejects the call. - -**The internal pool today:** 44 chat models, 18 of them scored. - -| Model | Score | TrueFoundry default effort | Score taken from | $/M blended | -| ----------------- | ----- | -------------------------- | ---------------- | ----------- | -| GPT-6 Astra | 51.6 | medium | medium | 18.00 | -| Claude Opus 5.5 | 48.2 | medium | medium | 7.20 | -| GPT-6.1 Sol | 46.3 | medium | medium | 3.60 | -| Claude Opus 5 | 44.2 | high | unlabelled | 9.00 | -| Claude Sonnet 5.5 | 41.5 | high | high | 3.60 | -| Grok 4.7 | 35.2 | high | high | 2.80 | -| Qwen 3.8 Max | 33.7 | xhigh | unlabelled | 2.80 | -| GPT-5.6 Terra | 32.3 | medium | unlabelled | 4.00 | -| GLM 5.3 | 31.7 | max | unlabelled | 4.00 | -| MiMo V2.6 Pro | 29.9 | medium | unlabelled | 0.52 | -| GLM 5.3 Flash | 27.8 | max | unlabelled | 0.22 | -| MiMo V2.6 Flash | 26.7 | medium | unlabelled | 0.17 | -| Claude Haiku 5.5 | 24.6 | medium | medium | 0.18 | -| GPT-5.6 Luna | 23.8 | medium | unlabelled | 0.40 | -| MiniMax M3 | 18.9 | medium | unlabelled | 0.96 | -| GPT-6 Luna | 18.5 | medium | medium | 0.18 | -| Kimi K3 | 17.4 | max | low (nearest) | 5.40 | -| gpt-oss-120b | 3.8 | medium | unlabelled | 0.43 | - -"Unlabelled" means Artificial Analysis publishes a single score with no effort label. That score is usually the model's maximum effort: Haiku 5.5's is 36.5, above its xhigh 34.6. So models that only have an unlabelled score may be overrated at a lower default. "Nearest" means only other efforts are published. Kimi K3, for example, has only a `low` score, so it's likely underrated at its `max` default. - -## 3. The Jev request - -One call per subagent to `POST {gateway}/v1/systemone`, using Jev's own provider config: gateway URL, token and turn headers. The TrueFoundry store's by-name lookup resolves decision-type models as well as chat models, so `getModelDetails` works for Jev exactly as for any model. Model listings stay chat-only, so Jev never shows up in the model picker. There's no call when there are no candidates. - -```json -{ - "model": "typesafe/jev-1.13.0", - "state": { - "name": "", - "task": "", - "tools": ["", "web_search", "sandbox"], - "benchmark_note": "Artificial Analysis Intelligence Index (0-100, higher is better) averages agentic knowledge work, coding, science and general reasoning evaluations." - }, - "questions": { - "can_0": { - "type": "noul", - "instructions": "Can Claude Haiku 5.5 at medium effort (Intelligence Index 24.6) complete this task fully and correctly?", - "criteria": { - "true": "It would produce a result that can be used as-is: complete, accurate, and with the needed tool use done.", - "false": "It would likely miss parts, make errors, or stop before the task is done." - } - }, - "can_1": { - "type": "noul", - "instructions": "Can Claude Sonnet 5.5 at high effort (Intelligence Index 41.5) complete this task fully and correctly?", - "criteria": { "...": "same as above" } - }, - "resembles_automationbench": { - "type": "noul", - "instructions": "Does this task resemble the benchmark AutomationBench? It tests whether models can complete realistic SaaS workflows that span multiple simulated business apps, using REST APIs as the tool interface.", - "criteria": { "true": "Same kind of work as the benchmark.", "false": "A different kind of work." } - } - } -} -``` - -- **The decision model is configured per tenant,** because it's a model on the tenant's own gateway: `typesafe/jev-1.13.0` on internal. If the tenant can't reach it, the call fails and the parent's model runs (`decision_error`). -- **Candidate questions** are keyed `can_`, and the router maps them back to models. -- **Limits:** one call, with a 2-second timeout and at most `max_candidates` (60) candidate questions. -- **Benchmark-resemblance questions:** the nine evaluations in the Artificial Analysis Intelligence Index v5, the score routing uses, so the answers say which part of the index our subagent work resembles. The answers are recorded only. Each description is quoted verbatim from the benchmark's official page (`modelRouting/benchmarks.ts`): the [Artificial Analysis methodology](https://artificialanalysis.ai/methodology/intelligence-benchmarking), and [lastexam.ai](https://lastexam.ai/) for HLE. - -| Category | Benchmark | -| -------------- | ---------------------- | -| Knowledge Work | AA-Briefcase | -| Knowledge Work | AutomationBench | -| Knowledge Work | GDP.pdf | -| Coding | FrontierCode | -| Coding | Terminal-Bench | -| Science | Terminal-Bench-Science | -| Science | CritPt | -| General | AA-Omniscience | -| General | Humanity's Last Exam | - -**Offline test** on 49 real subagent tasks × 11 candidates, checked against how each model actually did: - -- **Jev doesn't say yes to everything.** Average yes-probability per model was 0.17–0.42. -- **It ranks models sensibly on average** (rank correlation 0.53 with how often each model passed). On a single task, though, it only weakly tells whether a model will do well: 0.6, where 0.5 is a coin flip. -- **Showing Jev the parent model made it worse.** Jev then mostly compared the two scores. That's why the request has no parent model. -- **The number of questions doesn't matter.** Answers moved by about 0.01 between 3, 11 and 16 candidates. Calls took 0.4–1.2 s for up to 60 questions. - -## 4. Picking the model - -1. **Pick** the candidate with the highest yes-probability. On a tie, the cheaper one wins. -2. **Fall back to the parent's model** when there's no usable pick. The reason is logged: - - `no_candidates`; - - `decision_error`: the call failed, or Jev answered none of the candidate questions; - - `decision_timeout`. -3. **Shadow mode** logs the pick but runs the parent's model. **On mode** runs the pick. - -**In the offline test** (candidates cheaper than Opus 5.5, judged against Opus): Sonnet 5.5 was picked for 47 of 49 tasks. It was as good as Opus on 76% of tasks, at 53% of Opus's cost. - -**Expected risk on internal:** with no price ceiling, the pool includes GPT-6 Astra, the highest score at $18/M, which is 2.5× Opus. Jev's answers rise with the score, so many subagents may be picked onto Astra. Shadow mode measures the pick distribution and how often the pick costs more than the parent's model. If picks go up in price too often, the next step is a threshold rule: the cheapest candidate at or above a yes-probability. Prices are already on each candidate, so that's a change to the pick function and config only. - -## 5. Modes and config - -```json -TRUEFOUNDRY_MODEL_ROUTING = { - "timeout_ms": 2000, - "max_candidates": 60, - "tenants": { - "": { "subagent": "shadow", "decision_model": "typesafe/jev-1.13.0" } - } -} -``` - -- **off** (the default, or when the tenant isn't listed): the hook isn't wired, and nothing changes. -- **shadow:** ask Jev and log the decision, but run the parent's model. -- **on:** run the pick. - -## 6. Observability - -- **Gateway traces,** joined on `tfg.thread_id`: - - every model call a subagent makes (AGE-2837); - - the Jev call, which carries the turn metadata plus `tfg.thread_id` (the subagent being routed). Its model name identifies it as a routing call. - - The Jev request and response hold every candidate's question and probability, and the benchmark-resemblance answers. - -- **One log line per decision:** - - ```ts - logger.info('model routing', { - thread_id, - mode, - reason, - fallback_model, - picked_model, - chosen_model, - picked_p_yes, - picked_price, - fallback_price, - price_direction, - picked_image_input, - candidate_count, - }); - ``` - - - `price_direction` is `up`, `down` or `same`: the pick's price against the parent's model. - - `picked_image_input` shows whether the pick can read images. That's useful if routed subagents turn out to receive images. - - The line covers what the traces can't: decisions where Jev wasn't called, why routing fell back, and shadow-mode picks that weren't applied. - - It never includes task text. - -## 7. Rollout - -1. **Merge** with the config unset, so nothing changes. -2. **Shadow mode** on a few internal tenants. Check latency, the pick distribution, how often the pick costs more or less than the parent's model, and the fallback reasons. -3. **Switch on** for those tenants once AGE-2841 has merged. Measure quality with user feedback (§10). - -## 8. Root agent (later) - -The same router, score table, Jev request and pick rule. The root needs four things on top: - -- **A call site** where the root thread's definition is built for a turn, with the spec's model as the fallback, and a `scope` field on the input to tell the two apart. -- **Once per turn,** never mid-turn. -- **Persistence:** save the routed model with the turn, so a resumed turn keeps it. -- **Opt-in:** a per-tenant or per-agent opt-in. A model the user picked explicitly is never overridden. - -## 9. Code changes - -**Core (`packages/trueforge-core`):** only the hook and the call site. Core has no access to the model list, Jev or the scores. - -| File | Change | -| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `agent-session/ITurnResourceResolver.ts` | `RouteModelInput`, `RouteModelResult` and the optional `routeModel` hook (§1). | -| `agent-session/TurnResourceResolver.ts` | Accept `routeModel` in its deps and expose it. | -| `agent-session/SessionHandle.ts` | In `makeCreateDynamicSubAgentThread`, call `routeModel` when `request.model` is unset, and continue with `{ ...request, model }` if one comes back. | - -**Server: provider-neutral routing (`packages/trueforge/src/modelRouting/`).** None of this knows about TrueFoundry. It's the part a future routing capability would reuse or move into core. - -| File | Change | -| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **New** `types.ts` | What a provider supplies: `RoutingCandidate` (`model`, `slug`, `defaultEffort`, `price`, `imageInput`), a function that lists them, and `DecisionClient`, which asks yes/no questions and returns probabilities. Also the routing modes. | -| **New** `scores.ts` + `modelRoutingScores.ts` | The score table, and the lookup by slug and effort (§2). `benchmarks.ts` holds the resemblance benchmarks (§3). | -| **New** `pick.ts` | The pick rule, as a pure function from candidates and probabilities to a pick (§4). A threshold or price ceiling would change only this file and the config. | -| **New** `router.ts` | `createModelRouter({ candidates, decision, scores, config, logger })`. It builds the questions (§3), calls the decision client, picks, logs (§6), and returns `routeModel`. It never throws. | - -**Server: TrueFoundry implementations.** - -| File | Change | -| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `db/modelProviderStore.ts` | An optional `listRoutingCandidates()` on `IModelProviderStore`. Stores that can't supply catalogue metadata omit it, and routing isn't wired for them. | -| `truefoundry/mapEnabledModels.ts` | Widen the internal `TrueFoundryEnabledModel` with the fields in §2. Each new field falls back to undefined on an unexpected shape, so it can never break model listing. | -| `truefoundry/TrueFoundryModelProviderStore.ts` | `listRoutingCandidates()` for TrueFoundry mode, from the same `provider-integrations` fetch. The by-name lookup (`getProvider`) also resolves decision-type models, so Jev's config resolves like any model's; listings stay chat-only. | -| `truefoundry/InlineModelProviderStore.ts` | Forwards `listRoutingCandidates()` to the wrapped store. | -| **New** `truefoundry/jevDecisionClient.ts` | The `DecisionClient` for Jev: `POST {gateway}/v1/systemone` with Jev's own gateway URL, token and turn headers. | -| `config.ts` | `TRUEFOUNDRY_MODEL_ROUTING` (§5). | -| `apis/turns.ts` | When the tenant's subagent mode isn't off, create the router per turn from the TrueFoundry candidate source and the Jev client, and pass `routeModel` into the resolver. | - -No database schema or public API changes. - -**Extending outside TrueFoundry mode later:** implement `listRoutingCandidates()` on the configured-provider stores (slug from the shipped catalogue), and add a `DecisionClient` for whatever decision model is available. Then wire them where the resolver is built. The score table, pick rule, router and core hook stay the same. - -## 10. Open questions - -1. **How is quality measured?** The TrueForge backend doesn't store user feedback. Only the UI runtime passes a feedback adapter through. Feedback is also given per turn, on the root agent's answer, not per subagent. Before "on" mode we need to know where feedback lands on internal tenants, and how to attribute it to subagents. -2. **Model names in the questions:** keep them, or say only "a model with Intelligence Index S" to avoid Jev's preconceptions about names? -3. **Threshold or price ceiling:** likely needed soon, if shadow data shows picks go up in price too often (§4). It's a change to `pick.ts` and config only. From 5e19f65e1b3d3de82b5598d04303ae613cbc6e9c Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 16:02:38 +0530 Subject: [PATCH 12/14] refactor: Jev client takes the decision model's provider config Co-authored-by: Cursor --- packages/trueforge/src/apis/turns.ts | 25 ++++++---------- .../src/truefoundry/jevDecisionClient.ts | 30 +++++++++++-------- 2 files changed, 27 insertions(+), 28 deletions(-) diff --git a/packages/trueforge/src/apis/turns.ts b/packages/trueforge/src/apis/turns.ts index 5a6c8f145..f7a163415 100644 --- a/packages/trueforge/src/apis/turns.ts +++ b/packages/trueforge/src/apis/turns.ts @@ -387,22 +387,15 @@ function subagentModelRouter(input: { mode: tenantRouting.subagent, listCandidates: () => modelProviderStore.listRoutingCandidates?.({ tenant_id }) ?? Promise.resolve([]), decision: createJevDecisionClient({ - model: tenantRouting.decision_model, - resolveGateway: async model => { - const { providerConfig } = await getModelDetails({ - tenant_id, - name: model, - store: modelProviderStore, - turnMetadata, - }); - if (providerConfig.provider.type !== 'truefoundry' || providerConfig.baseUrl === undefined) { - throw new Error(`Model "${model}" is not served by the TrueFoundry gateway`); - } - return { - baseUrl: providerConfig.baseUrl, - headers: { Authorization: `Bearer ${providerConfig.apiKey}`, ...providerConfig.headers }, - }; - }, + resolveProviderConfig: async () => + ( + await getModelDetails({ + tenant_id, + name: tenantRouting.decision_model, + store: modelProviderStore, + turnMetadata, + }) + ).providerConfig, }), timeoutMs: routing.timeout_ms, maxCandidates: routing.max_candidates, diff --git a/packages/trueforge/src/truefoundry/jevDecisionClient.ts b/packages/trueforge/src/truefoundry/jevDecisionClient.ts index ad0d0db82..a3417fc6e 100644 --- a/packages/trueforge/src/truefoundry/jevDecisionClient.ts +++ b/packages/trueforge/src/truefoundry/jevDecisionClient.ts @@ -1,4 +1,4 @@ -import { modelSsrfFetch } from '@truefoundry/trueforge-core/core'; +import { modelSsrfFetch, type VercelAIProviderConfig } from '@truefoundry/trueforge-core/core'; import { z } from 'zod'; import type { DecisionClient } from '../modelRouting/types'; import { gatewayMetadataHeaders, headerValue, parseGatewayMetadataHeader, X_TFY_METADATA } from './gatewayMetadata'; @@ -7,29 +7,35 @@ const SystemOneResponseSchema = z.object({ answers: z.record(z.string(), z.object({ noul: z.number().nullish() })), }); -/** The model's gateway base URL and invoke headers (auth plus the turn's `x-tfy-metadata`). */ -export type ResolveGateway = (model: string) => Promise<{ baseUrl: string; headers: Record }>; - -/** Yes/no questions to a TrueFoundry decision model (Jev) through the gateway's `/v1/systemone`. */ -export function createJevDecisionClient(input: { model: string; resolveGateway: ResolveGateway }): DecisionClient { +/** + * Yes/no questions to a TrueFoundry decision model (Jev) through the gateway's `/v1/systemone`. + * `resolveProviderConfig` is the decision model's provider config, resolved on first use. + */ +export function createJevDecisionClient(input: { + resolveProviderConfig: () => Promise; +}): DecisionClient { return { async ask({ threadId, state, questions, signal }) { - const gateway = await input.resolveGateway(input.model); - const turnMetadataHeader = headerValue(gateway.headers, X_TFY_METADATA); + const config = await input.resolveProviderConfig(); + if (config.provider.type !== 'truefoundry' || config.baseUrl === undefined) { + throw new Error(`Decision model "${config.name}" is not served by the TrueFoundry gateway`); + } + const turnMetadataHeader = headerValue(config.headers, X_TFY_METADATA); const headers = { - ...Object.fromEntries(Object.entries(gateway.headers).filter(([key]) => key.toLowerCase() !== X_TFY_METADATA)), + ...Object.fromEntries(Object.entries(config.headers).filter(([key]) => key.toLowerCase() !== X_TFY_METADATA)), ...gatewayMetadataHeaders({ ...(turnMetadataHeader === undefined ? {} : parseGatewayMetadataHeader(turnMetadataHeader)), 'tfg.thread_id': threadId, }), + Authorization: `Bearer ${config.apiKey}`, 'content-type': 'application/json', }; - const root = gateway.baseUrl.replace(/\/+$/, '').replace(/\/v1$/, ''); + const root = config.baseUrl.replace(/\/+$/, '').replace(/\/v1$/, ''); const response = await modelSsrfFetch(`${root}/v1/systemone`, { method: 'POST', headers, body: JSON.stringify({ - model: input.model, + model: config.model.id, state, questions: Object.fromEntries( Object.entries(questions).map(([key, question]) => [key, { type: 'noul', ...question }]), @@ -38,7 +44,7 @@ export function createJevDecisionClient(input: { model: string; resolveGateway: signal, }); if (!response.ok) { - throw new Error(`Decision model ${input.model} returned HTTP ${String(response.status)}`); + throw new Error(`Decision model "${config.name}" returned HTTP ${String(response.status)}`); } const { answers } = SystemOneResponseSchema.parse(await response.json()); return Object.fromEntries( From 94e07282448fdc96ed39ec96589fee1155027b5c Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 17:31:51 +0530 Subject: [PATCH 13/14] fix: bound the whole routing step by the timeout; log candidate probabilities Co-authored-by: Cursor --- packages/trueforge/src/modelRouting/router.ts | 32 +++++++++++++++++-- .../tests/unit/modelRouting/router.test.ts | 19 +++++++++-- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/packages/trueforge/src/modelRouting/router.ts b/packages/trueforge/src/modelRouting/router.ts index f67febc46..e5bd250b3 100644 --- a/packages/trueforge/src/modelRouting/router.ts +++ b/packages/trueforge/src/modelRouting/router.ts @@ -1,4 +1,5 @@ import type { RouteModelInput, RouteModelResult } from '@truefoundry/trueforge-core/agent-session'; +import { describeUnknownError } from '@truefoundry/trueforge-core/core'; import type { Logger } from 'winston'; import { RESEMBLANCE_BENCHMARKS } from './benchmarks'; import { pickCandidate } from './pick'; @@ -119,13 +120,13 @@ export function createModelRouter(input: { return {}; }; - const timeout = AbortSignal.timeout(input.timeoutMs); - try { + const decide = async (timeout: AbortSignal): Promise => { models ??= input.listCandidates().catch((error: unknown) => { models = undefined; throw error; }); const all = await models; + timeout.throwIfAborted(); const candidates = scoredCandidates({ models: all, fallbackModel: route.fallbackModel, @@ -145,6 +146,7 @@ export function createModelRouter(input: { questions: questions(candidates), signal: AbortSignal.any([route.signal, timeout]), }); + timeout.throwIfAborted(); const picked = pickCandidate( candidates.flatMap((candidate, index) => { const pYes = answers[`can_${String(index)}`]; @@ -166,10 +168,34 @@ export function createModelRouter(input: { price_direction: priceDirection({ picked: picked.candidate.price, fallback: fallbackPrice }), picked_image_input: picked.candidate.imageInput, candidate_count: candidates.length, + candidates: Object.fromEntries( + candidates + .map((candidate, index) => [candidate.model, answers[`can_${String(index)}`]] as const) + .sort(([, a], [, b]) => (b ?? -1) - (a ?? -1)), + ), }); return input.mode === 'on' ? { model: picked.candidate.model } : {}; + }; + + // The budget covers the whole step, including candidate listing and gateway lookups, not just the Jev call. + const timeout = AbortSignal.timeout(input.timeoutMs); + const timedOut = new Promise((_, reject) => { + timeout.addEventListener( + 'abort', + () => { + reject(new Error(`Routing timed out after ${String(input.timeoutMs)} ms`, { cause: timeout.reason })); + }, + { once: true }, + ); + }); + const work = decide(timeout); + // Whichever promise loses the race settles later; its outcome is already reported. + work.catch(() => undefined); + timedOut.catch(() => undefined); + try { + return await Promise.race([work, timedOut]); } catch (error) { - return fallback(timeout.aborted ? 'decision_timeout' : 'decision_error', { err: error }); + return fallback(timeout.aborted ? 'decision_timeout' : 'decision_error', { err: describeUnknownError(error) }); } }; } diff --git a/packages/trueforge/tests/unit/modelRouting/router.test.ts b/packages/trueforge/tests/unit/modelRouting/router.test.ts index 64ba50745..c69f053f4 100644 --- a/packages/trueforge/tests/unit/modelRouting/router.test.ts +++ b/packages/trueforge/tests/unit/modelRouting/router.test.ts @@ -10,10 +10,14 @@ const candidates = [ { model: 'acct/claude-sonnet-5-5', slug: 'claude-sonnet-5-5', defaultEffort: 'high', price: 3.6 }, ]; -function route(decision: DecisionClient, mode: 'shadow' | 'on') { +function route( + decision: DecisionClient, + mode: 'shadow' | 'on', + listCandidates: () => Promise = () => Promise.resolve(candidates), +) { const routeModel = createModelRouter({ mode, - listCandidates: () => Promise.resolve(candidates), + listCandidates, decision, timeoutMs: 1_000, maxCandidates: 60, @@ -53,6 +57,17 @@ describe('createModelRouter', () => { expect(decision.ask).toHaveBeenCalledTimes(1); }); + it('keeps the fallback model when listing candidates outlasts the timeout', async () => { + jest.useFakeTimers(); + try { + const pending = route(prefersSonnet(), 'on', () => new Promise(() => undefined)); + await jest.advanceTimersByTimeAsync(1_000); + await expect(pending).resolves.toEqual({}); + } finally { + jest.useRealTimers(); + } + }); + it('keeps the fallback model when the decision model fails', async () => { const failing: DecisionClient = { ask: () => Promise.reject(new Error('HTTP 403')) }; await expect(route(failing, 'on')).resolves.toEqual({}); From 3adbb05780c85ca25b7480f5531e982813b93e20 Mon Sep 17 00:00:00 2001 From: Hary Ahuja Date: Mon, 16 Nov 2026 17:33:54 +0530 Subject: [PATCH 14/14] fix: log routing candidates as a ranked list Co-authored-by: Cursor --- packages/trueforge/src/modelRouting/router.ts | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/packages/trueforge/src/modelRouting/router.ts b/packages/trueforge/src/modelRouting/router.ts index e5bd250b3..aa613f3bb 100644 --- a/packages/trueforge/src/modelRouting/router.ts +++ b/packages/trueforge/src/modelRouting/router.ts @@ -168,11 +168,10 @@ export function createModelRouter(input: { price_direction: priceDirection({ picked: picked.candidate.price, fallback: fallbackPrice }), picked_image_input: picked.candidate.imageInput, candidate_count: candidates.length, - candidates: Object.fromEntries( - candidates - .map((candidate, index) => [candidate.model, answers[`can_${String(index)}`]] as const) - .sort(([, a], [, b]) => (b ?? -1) - (a ?? -1)), - ), + // A list, not an object: the JSON log format sorts object keys, which would lose the ranking. + candidates: candidates + .map((candidate, index) => ({ model: candidate.model, p_yes: answers[`can_${String(index)}`] })) + .sort((a, b) => (b.p_yes ?? -1) - (a.p_yes ?? -1)), }); return input.mode === 'on' ? { model: picked.candidate.model } : {}; };