Add tuned routing evaluation levers by devin-ai-integration[bot] · Pull Request #916 · workweave/router

devin-ai-integration · 2026-08-14T13:15:34Z

Summary

Three default-off levers for routing experiments, reimplemented from #902 by @EllAchE (credited via co-author trailer) to fit current main and repo conventions.

ROUTER_RESPECT_REQUESTED_MODEL — allowlisted requested models are served verbatim, skipping planner and scorer (Decision.Reason = requested_model_respected).
ROUTER_STATIC_CLUSTER_PIN — clusterID:model pairs short-circuit blendScoresV2 when the nearest cluster matches, tagging Reason with cluster-pin:.
ROUTER_TOOL_RESULT_TIER_CEILING — on ToolResult turns, excludes models above the requested model's tier.

Behavior worth knowing beyond the diff:

The ceiling is applied in runTurnLoop right after turn classification, so Messages, OpenAI and Gemini surfaces share it rather than only the Anthropic path.
It fails open: if no at-or-below-tier model survives (survivors == 0), the original exclusion set is returned unchanged, so an evaluation constraint can never turn a routable request into a hard failure. Model universe is s.availableModels when set, else the catalog — matching restrictToTier.
The sticky-pin context-window fit-check can lift context-derived exclusions, but must not lift a ceiling exclusion; an above-ceiling pin is evicted and routing falls to the scorer:

if !policyExcludes && !compatibilityExcludes && !ceilingExcludes {
    // ...existing lift
} else if ceilingExcludes {
    pinFound, pin = false, sessionpin.Pin{}
}

Precedence: an explicit user force-model pin (/force-model, x-weave-force-model) wins over the honor list. An honored model that is unknown or ineligible falls back to automatic routing rather than erroring, since the list is deployment config rather than a per-request override.
Static pin parsing fails fast at startup on malformed pairs or cluster IDs; syntactically valid pins naming models absent from the catalog warn and are skipped.

docs/CONFIGURATION.md documents all three. Regression tests cover force-model precedence, surface parity, the fail-open path, sticky-pin eviction, and pin fall-through when the pinned model is ineligible or unmatched. make precommit passes.

Link to Devin session: https://app.devin.ai/sessions/990590584e2e4f578c48064c4dc16fa1
Requested by: @steventohme

Co-authored-by: EllAchE <EllAchE@users.noreply.github.com>

greptile-apps · 2026-08-14T13:15:39Z

PR author is not in the allowed authors list.

devin-ai-integration · 2026-08-14T13:15:39Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment, CI, and merge conflict monitoring

claude · 2026-08-14T13:15:48Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Posted a comment-length advisory review with two suggestions.

workweave-bot

Advisory only — comment-length nits. Won't block merge.

workweave-bot · 2026-08-14T13:17:33Z

+	// ROUTER_TOOL_RESULT_TIER_CEILING (default off): excludes above-tier models
+	// from ToolResult turns before the scorer runs, capping candidates at or
+	// below the turn's requested-model tier. Eval/debug lever — see
+	// Service.applyToolResultTierCeiling.


Suggested change

// ROUTER_TOOL_RESULT_TIER_CEILING (default off): excludes above-tier models

// from ToolResult turns before the scorer runs, capping candidates at or

// below the turn's requested-model tier. Eval/debug lever — see

// Service.applyToolResultTierCeiling.

// ROUTER_TOOL_RESULT_TIER_CEILING: caps ToolResult candidates at the requested-model tier (default off).

Was 4 lines restating the env-var name; the key WHY fits in one.

Agreed — collapsed to a single why line in 3e926b8, matching the terseness of the neighbouring flag lines. The env var name is on the assignment immediately below, so the comment no longer repeats it.

workweave-bot · 2026-08-14T13:17:33Z

+// applyToolResultTierCeiling fails open when no eligible survivor remains so
+// an evaluation constraint cannot turn a routable request into a hard failure.
+// It uses the deployed model set when available, otherwise the full catalog.


Suggested change

// applyToolResultTierCeiling fails open when no eligible survivor remains so

// an evaluation constraint cannot turn a routable request into a hard failure.

// It uses the deployed model set when available, otherwise the full catalog.

// applyToolResultTierCeiling fails open: if no at-or-below-tier model survives,

// returns excluded unchanged so an eval constraint can never block a routable request.

Was 3 lines; line 3 restated the availableModels/catalog branch visible in the code.

Agreed — dropped the third line in 3e926b8; the godoc now carries only the fail-open rationale, since the availableModels/catalog branch is right there in the code.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 164d949. Configure here.}

cursor · 2026-08-14T13:19:25Z

+		// frontmatter pin must not anchor the session it happens to run inside.
+		res.HardPinned = true
+		res.PinTier = reasonRequestedModelRespected
+		return res, nil


Honor path skips thinking strip

High Severity

honoredRequestedModel returns like a hard pin: it sets HardPinned and exits before session pin / HMM history load, so PriorServedModel and SessionEverSwitched stay empty. On Anthropic emit, modelSwitched() then stays false and stale thinking signatures are not stripped after a real model change, which can 400 the upstream request. Usage-bypass already reloads switch history for this reason; the honor path does not.

Additional Locations (1)

internal/proxy/turnloop.go#L442-L466

^{Triggered by learned rule: Thinking block format must be capability-matched on same-format Anthropic emit}

^{Reviewed by Cursor Bugbot for commit 164d949. Configure here.}

cursor · 2026-08-14T13:19:25Z

+		res.StickyHit = true
+		// Bypasses pin lookup/write like the hard pins above: a per-request
+		// frontmatter pin must not anchor the session it happens to run inside.
+		res.HardPinned = true


Requested-model marker never shows

Medium Severity

The honor path sets HardPinned = true, and routingMarkerFor returns empty whenever HardPinned is set. The new markerReasonRequested / reasonRequestedModelRespected wiring in routingReasonShort is therefore unreachable for the path it was added for, so users never see “served the model you asked for”.

Additional Locations (2)

internal/proxy/service.go#L520-L521

internal/proxy/service.go#L543-L546

^{Reviewed by Cursor Bugbot for commit 164d949. Configure here.}

Co-authored-by: EllAchE <EllAchE@users.noreply.github.com>

claude · 2026-08-14T13:21:17Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Claude finished @devin-ai-integration[bot]'s task — View job

Posted a comment-length advisory review with one suggestion.

Run git diff and analyze added comment blocks
Flag verbose 3+ line comment blocks or stay silent

workweave-bot

Advisory only — comment-length nit. Won't block merge.

workweave-bot · 2026-08-14T13:23:18Z

+// honoredRequestedModel returns an eligible allowlisted requested model.
+// Unknown or ineligible entries fall back to routing because the list is not
+// an explicit per-request override like force-model.


Suggested change

// honoredRequestedModel returns an eligible allowlisted requested model.

// Unknown or ineligible entries fall back to routing because the list is not

// an explicit per-request override like force-model.

// honoredRequestedModel returns an eligible allowlisted requested model.

// Unknown or ineligible entries fall back to routing; the list is config, not a per-request override like force-model.

Was 3 lines; the why fits in one sentence on line 2.

Co-authored-by: EllAchE <EllAchE@users.noreply.github.com>

claude · 2026-08-14T13:24:36Z

Claude finished @devin-ai-integration[bot]'s task —— View job

Claude finished @devin-ai-integration[bot]'s task — View job

Posted a comment-length advisory review with one suggestion.

Run git diff and analyze added comment blocks
Flag verbose 3+ line comment blocks or stay silent

workweave-bot

Advisory only — comment-length nit. Won't block merge.

workweave-bot · 2026-08-14T13:26:14Z

+// honoredRequestedModel returns an eligible allowlisted requested model.
+// Unknown or ineligible entries route normally because this config list is not
+// a per-request override like force-model.


Suggested change

// honoredRequestedModel returns an eligible allowlisted requested model.

// Unknown or ineligible entries route normally because this config list is not

// a per-request override like force-model.

// honoredRequestedModel returns an eligible allowlisted requested model.

// Unknown or ineligible entries fall back to routing; the list is config, not a per-request override like force-model.

Still 3 lines after the previous tightening; lines 2–3 are one sentence that fits on one.

Tightened this one already in the previous round — the remaining two lines are a single why-only sentence wrapped at the file's column width, which is what root AGENTS.md asks for ("Only when why is non-obvious… no multi-paragraph"). Collapsing it to one long line would only move the wrap, not remove information, so I'll leave it here. Noting these are advisory and non-blocking.

steventohme and others added 2 commits August 14, 2026 13:09

Add tuned routing evaluation levers

1cc410b

Co-authored-by: EllAchE <EllAchE@users.noreply.github.com>

Fix tuned routing lever review findings

164d949

Co-authored-by: EllAchE <EllAchE@users.noreply.github.com>

devin-ai-integration Bot mentioned this pull request Aug 14, 2026

Add tuned-config retrial levers: respect-requested-model, static cluster pin, tool_result tier ceiling #902

Open

4 tasks

workweave-bot reviewed Aug 14, 2026

View reviewed changes

cursor Bot reviewed Aug 14, 2026

View reviewed changes

Tighten routing lever comments

3e926b8

Co-authored-by: EllAchE <EllAchE@users.noreply.github.com>

workweave-bot reviewed Aug 14, 2026

View reviewed changes

Tighten honored model godoc

7f08587

Co-authored-by: EllAchE <EllAchE@users.noreply.github.com>

workweave-bot reviewed Aug 14, 2026

View reviewed changes

Uh oh!

Conversation

devin-ai-integration Bot commented Aug 14, 2026

Summary

Uh oh!

greptile-apps Bot commented Aug 14, 2026

Uh oh!

devin-ai-integration Bot commented Aug 14, 2026

🤖 Devin AI Engineer

Uh oh!

claude Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

workweave-bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

workweave-bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 14, 2026

Choose a reason for hiding this comment

Honor path skips thinking strip

Uh oh!

cursor Bot Aug 14, 2026

Choose a reason for hiding this comment

Requested-model marker never shows

Uh oh!

claude Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

workweave-bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

workweave-bot left a comment

Choose a reason for hiding this comment

Uh oh!

workweave-bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Aug 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

claude Bot commented Aug 14, 2026 •

edited

Loading

claude Bot commented Aug 14, 2026 •

edited

Loading

claude Bot commented Aug 14, 2026 •

edited

Loading