feat: per-model and per-class toleration presets in account files by hachall · Pull Request #2081 · doublewordai/control-layer

hachall · 2026-10-21T17:14:40Z

What

Account files can set a scheduling-toleration preset on a model, or on one class of it:

account: acme
models:
  example/chat-model:
    toleration_preset: no-overflow   # this model only
    classes:
      standard:
        toleration_preset: gpu       # overrides the model's

For requests to that model, the preset replaces the account's own preset (users.toleration_preset, set by a platform manager). The more specific setting wins: class, then model, then the account, then none. "No preset" stays distinct from an empty toleration list. Only standard can be configured until models carry other classes, as with the rest of the account file.

The account-wide preset stays owned by the admin console; the files only add model and class overrides.

How

Migration (additive): nullable account_model_policies.toleration_preset. Written by the account-file reconcile like the other policy columns; the table's existing notify trigger refreshes caches.
Per-key policy map: each key's KeyPolicy carries its account's per-model preset map (class folded over model), shared by Arc across the account's keys. One extra small query per refresh.
Inference middleware: the preset is now resolved after the model is known (after the async suffix strip), so realtime, flex and background requests get the list for their model.
Batch upload: each line is stamped by its own model, from the same policy (or the direct DB read used for a just-created hidden key).
Unknown ids fail closed to the empty list at request time, exactly as account presets do today. Startup logs a warning for any id the files name that the deployment's scheduling.toleration_presets doesn't define (the offline validator has no access to config).

Rollout

Dormant: nothing changes until an account file sets toleration_preset. Older images ignore the new column.

Tests

just lint rust: green. just test rust: 4241 passed.
New tests:
- realtime, Responses and batch: a model preset overrides the account preset on that model only, and a class preset overrides the model's;
- a model preset pins an otherwise unpinned account on that model only;
- per-key resolution (model over account, "no preset" vs empty list);
- the reconcile writes, folds and clears presets, and validation rejects empty or padded ids.

🤖 Generated with Claude Code

The interactive/throughput presets, per-account class grants and account or overlay default classes are unused. Removing them now keeps the routing path small before model class routing lands. - Onwards resolves only overlay targets (`custom`) or nothing (`standard`) and the self-hosted restriction; class suffixes are no longer parsed. - dwctl no longer strips or interprets realtime class suffixes. Flex, background and batch uploads still discard a model suffix after a shape check, unchanged. - The admin API drops the grant and default-class fields; the organisation serving view and model overlays keep targets and self_hosted_only. - The model catalog accepts `serving_classes` only as an empty map. The organisation catalog prices `standard` and `custom` only, and retires any rows it owned under the old class names. - The dashboard drops the grant, default-class and preset views. Stored onwards configuration that still carries the old fields loads with them ignored. Database columns are left in place for a later migration.

Assert that a PATCH carrying the retired grant and default-class fields is accepted, writes neither column and returns neither field. Restrict the organisation pricing view to the classes billing can select (standard, and custom when an overlay's targets apply), so a manual price under a retired class name is no longer shown as if it applied.

…rving-classes

…iles

Each account file can now hold one account's whole serving deal, per model and per serving class, alongside its realtime in-flight limits: - an account-wide scheduling priority, with model and class overrides; - ttft/itl latency targets per class; - batch prices per model, realtime/playground prices and a cache tariff per class. The older top-level `realtime_inflight` shape is still accepted and means `classes.standard.realtime_inflight`. Only `standard` can be configured until models carry other classes. The account files are reconciled in one transaction: limits keep their whole-table semantics; priorities, policies and prices are owned per row by the `account-catalog` source, so hand-written rows are adopted only when the pair is declared and rows owned by the organisation catalog are refused. An account/model pair may be configured in an account file or an organisation overlay, not both. Onwards stamps the resolved priority and targets on realtime requests to dynamo members as hints that keep the request billed as its own class; legacy overlay targets keep their custom-class behaviour and daemon traffic keeps its deadline priority. The migration is additive (a nullable users column and a new table); older images ignore both, and today's files reconcile to exactly what they do now.

Reject the background tier's reserved priority and latency targets that do not fit the stored integer, and refuse to overwrite a policy row another writer owns (unowned rows are adopted, as manual prices are). Clarify that latency targets apply to realtime requests only and when account-file failures stop startup.

An account file can now set `toleration_preset` on a model or on one of its classes. For requests to that model the preset replaces the account's own (console-set) preset; the more specific setting wins: class, then model, then the account, then none. "No preset" stays distinct from the empty list. The per-key policy map carries each account's per-model presets, so the inference middleware resolves the list after the model is known (realtime, flex and background) and batch upload stamps each line by its model. An id the deployment does not configure fails closed to the empty list, as account presets already do, and startup logs any such id the files name. Dormant until an account file sets the field.

…on-presets # Conflicts: # docs/src/reference/model-provisioning.md # dwctl/src/account_limits.rs # dwctl/src/lib.rs

cloudflare-workers-and-pages · 2026-10-21T17:14:42Z

Deploying control-layer with Cloudflare Pages

Latest commit:	`c5338a0`
Status:	✅ Deploy successful!
Preview URL:	https://6ca26b88.control-layer.pages.dev
Branch Preview URL:	https://feat-account-toleration-pres.control-layer.pages.dev

View logs

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

cubic-dev-ai

All reported issues were addressed across 11 files

_{Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic}

…on-presets

hachall added 8 commits October 21, 2026 12:17

Merge remote-tracking branch 'origin/main' into feat/remove-legacy-se…

bf23ccf

…rving-classes

Merge remote-tracking branch 'origin/main' into feat/account-policy-f…

4c4a04a

…iles

Merge remote-tracking branch 'origin/main' into feat/account-tolerati…

0a74cc3

…on-presets # Conflicts: # docs/src/reference/model-provisioning.md # dwctl/src/account_limits.rs # dwctl/src/lib.rs

Copilot AI balanced review requested due to automatic review settings October 21, 2026 17:14

Copilot started reviewing on behalf of hachall October 21, 2026 17:15

View session

Copilot AI reviewed Oct 21, 2026

cubic-dev-ai Bot reviewed Oct 21, 2026

View reviewed changes

Comment thread docs/src/reference/model-provisioning.md Outdated

hachall added 2 commits October 21, 2026 18:22

docs: use a configured preset id in the account file example

0a24cde

Merge remote-tracking branch 'origin/main' into feat/account-tolerati…

c5338a0

…on-presets

rushilbhat approved these changes Oct 21, 2026

View reviewed changes

hachall added this pull request to the merge queue Oct 21, 2026

Merged via the queue into main with commit d916973 Oct 21, 2026
27 checks passed

pjb157 mentioned this pull request Oct 21, 2026

release(main): release 15.2.0 #2084

Merged

Navigation Menu

Repository navigation

feat: per-model and per-class toleration presets in account files - #2081

feat: per-model and per-class toleration presets in account files#2081
hachall merged 10 commits into
mainfrom
feat/account-toleration-presets

hachall commented Oct 21, 2026 •

edited by cubic-dev-ai Bot

Loading

Uh oh!

cloudflare-workers-and-pages Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

hachall commented Oct 21, 2026 • edited by cubic-dev-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Rollout

Tests

Uh oh!

cloudflare-workers-and-pages Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying control-layer with Cloudflare Pages

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hachall commented Oct 21, 2026 •

edited by cubic-dev-ai Bot

Loading

cloudflare-workers-and-pages Bot commented Oct 21, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading