Pre-Routing Target Filtering: Eligibility vs. Preference in Route Selection (Design Discussion)

[drewid74]

Pre-Routing Target Filtering: Eligibility vs. Preference in Route Selection
Repo: NVIDIA NeMo Switchyard
Type: Design discussion / feature proposal
Context: Validating the pre-dispatch abstraction before proposing implementation changes to libsy and the algorithm dispatch path.

Question
Is there an existing or planned mechanism for applying hard eligibility rules to candidate targets before a routing algorithm executes, rather than requiring those rules to be implemented within individual routing algorithms?
Today, the primary extension point in Switchyard is the routing algorithm layer, such as the LLM classifier, stage router, escalation router, or custom implementations conforming to the routing dispatch interface.
That layer is well suited to express preference: which target is best for a given turn. However, it is an awkward place to express eligibility: which targets are legally, structurally, or operationally allowed to be considered at all.

Preference and Eligibility Are Different Concerns
Consider a standard enterprise policy constraint:
Requests containing EU user data may only be dispatched to targets hosted in eu-west.
If the intended path is to implement a custom routing algorithm, that works mechanically, but it couples hard filtering with soft heuristic ranking.
• Re-implementation overhead: A team wanting the tier promotion logic of stage router or escalation must wrap or duplicate the scoring algorithm simply to bolt a candidate filter in front of it, inheriting maintenance risk from upstream algorithm changes.
• Distinct failure modes: Preference is heuristic, tunable, and tolerates probabilistic variance while optimizing for latency, cost, or output quality. Eligibility is invariant enforcement. A routing failure here represents a compliance, licensing, or execution defect.
• Operational ownership: Changes to ranking heuristics belong to ML/product teams tuning cost versus quality. Changes to data residency, licensing, or compliance boundaries belong to security and legal stakeholders. Keeping these concerns distinct makes both easier to reason about, audit, and review in isolation.
• Target-aware boundary: Coarse network filtering can live in an ingress API gateway, but fine-grained routing constraints require route-level awareness of target properties, such as target-specific token cost ceilings, hardware quantization capabilities, or dynamic endpoint health states. An upstream gateway lacks that context, and individual routing algorithms should not need to re-implement it.

Use Cases
These operational requirements all share the same structural pattern:

1. Cost ceilings: A route may need to prevent dispatch to targets exceeding a dynamic budget threshold, such as max cost per mtok out > N. Removing expensive targets statically from route configuration is too coarse because budget constraints are often tenant-, request-, or task-specific.
2. Capability invariants: If an incoming request payload specifies tool calls, function schemas, multimodal inputs, or structured JSON schema outputs, targets lacking backend support for those features must be removed from the candidate set before ranking.
3. Dynamic health / rate-limit backoff: Targets encountering 429s, saturation, circuit breaker trips, or degraded health telemetry should be temporarily subtracted from candidate sets without mutating declarative route configs.
4. Data residency / sovereignty: Payloads tagged with jurisdictional attributes, such as x-region: eu, must strictly match target infrastructure attributes before any heuristic evaluation occurs.
5. Output-use licensing: Workflows generating public artifacts or training datasets may be restricted from using models whose terms of service prohibit competitive distillation or specific commercial downstream uses.

Common Shape Abstraction
All of the above reduce to a single invariant predicate executed over the request context and candidate target set.
Rust: individual target evaluation predicate

request context + target -> Result<(), ExclusionReason>

Candidate set filtering

filter(request context, candidates: Vec<Target>) -> Result<Vec<Target>, RouteExhaustedError>

Proposed Pipeline Flow

route resolution config + headers -> constraint filtering -> set reduction -> routing algorithm dispatch -> target execution

Declarative Example Sketch

[constraints.budget]
kind = "cost_ceiling"
max_cost_per_mtok_out = 100

[constraints.residency]
kind = "attribute_match"
request_attribute = "x-region"
target_attribute = "region"

[targets.frontier_us]
id = "deepseek-r1-us"
region = "us"
cost_per_mtok_out = 80

[routes.agent]
algorithm = "escalation"
constraints = ["budget", "residency"]

Composition Semantics
Monotonically Subtract Only

Candidates_final subseteq Candidates_initial

Constraints can only eliminate candidates from the route pool. They can never introduce or re-add candidates. This ensures determinism and avoids order-dependent resolution loops.

Fail Closed on Empty Sets
If constraint filtering eliminates all candidates, Switchyard should return an explicit routing error, such as RouteExhaustedError, rather than falling back to an unconstrained target. Bypassing constraints should require explicit opt-in policy.

Interaction with Escalation and Multi-Turn Latching
In Switchyard escalation routing, the escalation router and LlmTaskClassifier in escalation mode track session progression and latch target affinity once escalation thresholds, such as consecutive task-failure signals, are crossed.
Because Switchyard’s internal session state manages target affinity and execution signals rather than content compliance history, a subtle multi-turn edge case emerges when requests forward conversational context, such as full OpenAI or Anthropic messages arrays.

1. Turns 1–2 are evaluated on Tier 1, such as a local model.
2. Turn 3 contains EU-restricted PII or proprietary code that disallows frontier external targets.
3. Turn 7 is a generic syntax query with no turn-specific constraint flags.
4. Turn 8 crosses the escalation threshold, and Switchyard latches affinity to a frontier target.
5. The entire message transcript, including Turn 3 content, is forwarded to the promoted target.
   If eligibility constraints evaluate only against the active turn’s immediate payload delta, Turn 8 evaluates as compliant and dispatches the accumulated transcript to an ineligible target.

Proposed Session Constraint Model: Monotonic Join Lattice
Rust trait sketch

trait SessionConstraint {
    fn permits(&self, ctx: RequestContext, target: Target) -> Result<(), ExclusionReason>;
    fn join(&self, prev_state: ConstraintState, current: ConstraintState) -> ConstraintState;
}

• Session-scoped constraints: Track a monotonic state that can narrow the candidate set over the session lifecycle, but never widen it.
• Stateless turn-scoped constraints: Evaluate per turn for transient conditions, such as target health or payload token limits.

Questions for Maintainers

1. Does adding a pre-routing constraint filter stage align with Switchyard’s architectural roadmap, or is the preferred pattern to keep candidate filtering encapsulated inside custom routing algorithms?
2. Does the subtract-only candidate model cover all expected routing patterns, or are there scenarios where constraints need additive capability?
3. For multi-turn escalation and latched sessions, is a monotonic constraint lattice a desirable approach to prevent transcript leakage across target promotions, or is session-level compliance handled at a different abstraction layer in libsy?

I would be happy to put together an initial RFC and prototype against libsy if this direction aligns with the repository’s design goals.

[gschaetz]

We hit this same eligibility-versus-preference problem building a gateway on top of switchyard-libsy, and solved it at the host rather than inside libsy. Sharing what worked, in case it's useful to the design.

Approach. Before an algorithm runs, we build a per-request RuntimeModels that lists only the eligible targets, so the algorithm (stage router, escalation, ...) never sees the others. It follows your subtract-only and fail-closed shape:

• Constraints only remove candidates, and nothing later (health, budgets, selector rules, headers) can add one back.
• A route configured so that no target could ever qualify is rejected at config load, and an empty set at request time is an explicit refusal, not a fallback to an unconstrained target.
• The existing tier substitution in the router stays inside the allowed set, so a degraded request cannot leak out through a "borrow from the other tier" path.

What we found while doing it.

• Enforcing at the moment of fallback is much weaker than filtering before the algorithm: it depends on the order of attempts and leaves retries and substitution as other paths. Filtering up front made "never calls X" provable with a test where the excluded upstream records zero calls.
• Failing closed at load matters: a target with no property (for us, no class label) on a constrained route must be an error, not a silent exclusion.
• Recording exclusions per request, with a counter of requests that failed because a constraint removed the fallback, turned out to be the useful operational signal.

Where we fall short of what you describe. Ours is static labels on targets plus a small budget/health policy, not a general predicate over request attributes (cost ceiling, region, capability). And we don't have your multi-turn case: a restricted turn 3 doesn't limit where turn 8 can be promoted. Strictness is per route, so a request can't escalate out of a strict route, but nothing latches from content. That seems like the hard part and a reason this belongs in libsy rather than in every host.

Design notes: https://github.com/gschaetz/humpyard/blob/main/docs/adr/0017-strict-routes-exclude-disallowed-classes-before-routing.md

Happy to answer questions or help prototype.