[Bug] Routing profile optimize.latency has no latency score term and only becomes the residual priorityWeight #1834

Summary

A routing profile's optimize.latency weight is documented as a tuning knob that shifts candidate
selection toward low-latency providers (the dashboard editor exposes it as a "scoring weight" and the
default profile weights make it the most-weighted slot at latency: 0.55), but the routing
evaluator never scores latency as an independent objective. The value is normalized into the weight
sum and then leaks out only as the residual priorityWeight (an indirect, unintentional effect), while
real measured latency is folded into the health dimension instead. The selection step is a
deterministic argmax over the scored candidates, so with only latency-relevant evidence the winner
is chosen purely by configured declaration order, independent of any actual latency measurement.
Raising optimize.latency therefore does not make the router prefer faster providers; it mainly
increases the influence of the configured-candidate-priority term, which is not what an operator
setting this knob expects.

Why it is a real issue (file:line evidence)

• src/routing/profile.ts:26-31 — DEFAULT_PROFILE_WEIGHTS = { latency: 0.55, health: 0.25, cost: 0.10, quota: 0.10 }.
  Latency is the single most-weighted slot, yet it has no score term (see below) — the largest
  weight is the knob doing something unrelated (configured priority).
• src/routing/profile.ts:504-521 — normalizeRoutingProfile reads raw.optimize, sums all four
  weights and normalizes (each / safeSum), and stores optimize.latency as a normalized weight only.
  It never reads a latency score.
• src/routing/evaluator.ts:397-448 — the candidate score is built only from
  profile.optimize.health/quota/cost plus priorityWeight * configuredPriorityScore(...); there is
  no latency score term and grep -c "latency" src/routing/evaluator.ts returns 0. The remaining
  budget is const priorityWeight = Math.max(0, 1 - spentHealth - spentQuota - spentCost);
  (evaluator.ts:404). Because the normalized weights sum to 1, priorityWeight === optimize.latency
  only when health, quota, and cost each carry concrete evidence (each spent* equals its
  weight); otherwise it is the residual budget and can exceed optimize.latency. So latency is not
  ignored wholesale, but its only role is to become the weight of the configured-priority term
  (configuredPriorityScore(index, total), evaluator.ts:183-185), never a latency term.
• src/routing/evaluator.ts:444-447 — selection is a deterministic argmax: if (evaluated.eligible && score.total > bestScore). No shuffle, no weighted-random, no latency re-selection; ties favor the earlier index (strict >). The dispatch path then routes the selected candidate directly (src/router.ts:552-564). So "ranked purely by configured order" is exactly what happens for the priority-only case.
• Latency is captured only inside the health score:
  • Real path: src/routing/compatibility/assemble.ts:52-72 → policyCandidateHealthEvidence →
    src/routing/health.ts:184-204, where the persisted duration_ms samples are folded into a p50
    and carried as RouteHealthEvidence.recentLatencyMs (health.ts:323-325).
  • src/routing/health.ts:386-400 — latencyScore (from recentLatencyMs) is folded into the
    healthScore composite and (on the routing side, evaluator.ts:407-410) scaled by the health
    weight profile.optimize.health, not by profile.optimize.latency.
  • Net influence: under default weights, raw latency reaches the final score only as
    optimize.health (0.25) × HEALTH.LATENCY_WEIGHT (0.20) = 0.05, while the "latency" knob's 0.55 goes
    entirely to configured priority. The most-weighted knob has ~11× less real latency influence than
    an operator would infer.
• src/routing/trace.ts:133 — RouteScoreEvidence.components already declares a latent latency?: number slot, but the evaluator never fills it (no components.latency anywhere). The schema was
  designed for a latency score component that the implementation omits.
• Existing test pins the current behavior — tests/routing-profile.test.ts:317-335
  ("dry-run evaluator: deterministic priority picks the earlier candidate") asserts
  components: { configuredPriority: 1, health: 0.3, quota: 0.3, cost: 0.3 } (total 0.685) with
  no latency component, and that the earlier candidate wins. This is not an accidental gap: the
  behavior is intentional and test-locked, which strengthens the case that the knob is mis-documented
  rather than merely unimplemented.

Reproduction / demonstration

Repro probe: repro/routing_latency_probe.mjs (a faithful self-contained mirror of the scoring block
evaluator.ts:397-448 + healthScore latency math health.ts:386-400, using the real constants
DEFAULT_PROFILE_WEIGHTS and HEALTH: { LATENCY_WEIGHT: 0.20, LATENCY_TARGET_MS: 60_000, ... }).
Run: node repro/routing_latency_probe.mjs.

=== Scenario A: optimize { latency:1, health:0, cost:0, quota:0 } ===
config order: [fast, slow]  |  fast p50=800ms, slow p50=40s
  fast-provider: priorityWeight=1 total=1.000 (configuredPriority=1)
  slow-provider: priorityWeight=1 total=0.500 (configuredPriority=0.5)
  SELECTED: fast-provider  (p50s 800,40000 never referenced in scoring)

=== Scenario A2: same weights, config order REVERSED [slow, fast] ===
  slow-provider: priorityWeight=1 total=1.000
  fast-provider: priorityWeight=1 total=0.500
  SELECTED: slow-provider  <- pure configured order, NOT latency

=== Scenario B (default weights): latency ~0.55, health ~0.25 ... ===
  net latency influence = optimize.health * HEALTH.LATENCY_WEIGHT = 0.0500
  vs priorityWeight 0.55 (the "latency" knob -> CONFIGURED PRIORITY)

=== Identity caveat (numeric) ===
  optimize { latency:.4, health:.3, cost:.2, quota:.1 }, only health evidence -> priorityWeight=0.7
  (NOT 0.4) -- identity priorityWeight===latency holds only when health/quota/cost all have evidence

Takeaways:

1. With optimize: { latency: 1, ... }, priorityWeight = 1 - 0 - 0 - 0 = 1 and the score reduces to
   1 * priorityScore, an argmax on configured order — the faster provider is chosen only if listed
   first, and reversing the config order picks the slower one. No measured latency is consulted.
2. With optimize: { latency: 0, health: 1, ... }, priorityWeight = 0 and only the latency-informed
   healthScore term is used — this works, but only because latency is embedded in health, and it
   costs the configured-priority signal entirely.
3. The exact identity priorityWeight === optimize.latency (from the original draft) is not the
   general case: it equals 1 - spentHealth - spentQuota - spentCost; it matches optimize.latency
   only when all three other dimensions carry concrete evidence.

Proposed direction

Either (a) implement a real latency score term in evaluator.ts scaled by profile.optimize.latency
(reusing the p50 latency already carried in RouteHealthEvidence.recentLatencyMs, and filling the
latent components.latency slot in trace.ts:133), keeping the health composite independent and
updating the pinned test at tests/routing-profile.test.ts:317-335; or (b) drop latency from the
optimize block and document that latency is already folded into the health dimension, so the
confusing knob cannot silently mutate priority weight. Existing tests should pin the chosen behavior.

Checks

• src/routing/evaluator.ts:397-448 confirms no latency score term, priorityWeight = residual,
  and argmax selection (>, earlier-index tie-break).
• src/routing/profile.ts:26-31,504-521 confirms latency: 0.55 default and that latency
  participates only in weight normalization.
• src/routing/health.ts:184-204,323-325,386-400 confirms latency is only scored through the
  health dimension (scaled by optimize.health, weight 0.20 in the composite).
• tests/routing-profile.test.ts:317-335 pins the no-latency-component behavior (intentional).
• repro/routing_latency_probe.mjs reproduces the bias numerically (run node ...).
• No existing open issue describes an ineffective optimize.latency knob (searched title scope).

References (web)

• opencodex Routing Profile Editor guide — documents the optimize block with
  "latency": 0.55 as the dominant weight in a "fast" profile example, with no note that latency is
  not independently scored: https://github.com/lidge-jun/opencodex/blob/main/docs-site/src/content/docs/guides/routing-profile-editor.md
• OpenRouter Provider Routing — latency is a first-class routing dimension
  (sort: "latency" prioritizes lowest latency; preferred_max_latency filters by p50/p90/p99 over a
  rolling window): https://openrouter.ai/docs/guides/routing/provider-selection
• LiteLLM Router — native latency-based-routing strategy and lowest_latency_buffer, i.e. latency
  as a real scoring criterion: https://docs.litellm.ai/docs/routing
• opencodex Model Routing guide — documents the resolver as order-based ("the first match wins"), with
  no latency-aware selection: https://github.com/lidge-jun/opencodex/blob/main/docs-site/src/content/docs/guides/model-routing.md