Repository navigation
feat: overlay targets are scheduling hints, not a serving class - #2087
Conversation
A serving class is where a request is routed. Overlay `targets` used to make a request resolve as a `custom` class, which changed its analytics class and its price lookup: an account's standard-class price was skipped whenever its overlay carried targets. Targets now travel exactly like account-file hints: the priority and latency targets are sent to dynamo members, and the request keeps its routed class (standard) and its price. - onwards: `ServingClass` has only `standard`; the separate targets path and `ServingTargets` are gone. Hints are not sent to a named-pool member that refuses scheduling fields, matching how its `nvext` is already stripped. The request span records the hinted priority and whether targets were sent. - dwctl: the sync turns stored overlay targets into hints. Overlay `class_pricing` prices one of the model's classes (`standard`); `custom` is refused, and rows the catalog owns under it are retired on the next apply. - dashboard: pricing shows the model's classes only; overlay details label targets as scheduling hints.
|
Running ultrareview automatically — this PR appears to change payment, billing, or quota logic, where a missed bug can charge customers the wrong amount, so a deeper multi-pass review is worth running. I'll post findings when complete. |
Deploying control-layer with
|
| Latest commit: |
20ee974
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://02d29d3e.control-layer.pages.dev |
| Branch Preview URL: | https://feat-targets-are-hints.control-layer.pages.dev |
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
Ultrareview ran automatically — this PR appears to change payment, billing, or quota logic, where a missed bug can charge customers the wrong amount, so a deeper multi-pass review is worth running.
All reported issues were addressed across 15 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
… policies Name the request span's targets field for what it is (configured) and record onwards.hints_sent on the attempt that actually carried hints. A priority-only account policy no longer clears an overlay's latency targets in the sync. The overlay docs name dynamo members that accept scheduling fields, and a test covers retiring catalog-owned custom prices on a still-declared model.
There was a problem hiding this comment.
0 issues found across 5 files (changes from recent commits).
Requires human review: Cross-cutting serving behaviour change: overlay targets become scheduling hints, the custom serving class and its pricing are removed, altering production class resolution, tariff selection and dashboards. Affects the production request path, so it requires human approval.
View guided diff | Turn on auto-fix | Re-trigger cubic
What
A serving class is where a request is routed. Until now, an organisation overlay's
targetsmade the request resolve as acustomclass. This PR removescustom: overlaytargetsbecome scheduling hints, exactly like account-file priority and targets, and the request keeps its routed class (standardtoday).targets→ resolved classcustom(analyticsresolved_serving_class, tariff and cache lookup)standard)dynamomembers that accept scheduling fieldsclass_pricingcould pricecustomclass_pricingprices one of the model's classes (standard).customis refused by validation, and rows the catalog owns under it are retired on the next applyWhy
modelis standard and, later,model:fastis fast. Customer deals (priority, limits, price, tolerations) sit on top of whichever class a request hits. Latency targets are customer labels for alerting and diagnostics; nothing routes on them.customwas left over from the earlier target-routing design.customand skipped an account'sstandard-class price, falling back to the model default. The new e2e testoverlay_targets_change_neither_the_class_nor_the_pricecovers it.Changes
ServingClasshas onlyStandard.ServingTargets, the overlaytargetsfield and the separate stamping path are removed. A stored config that still carries overlaytargetsloads, and the field is ignored.nvextis already stripped.onwards.serving.priorityandonwards.serving.targets(whether latency targets were sent), separately fromonwards.serving.resolved.model_overlays.targets(ttft_ms,itl_ms,priority) onto the overlay's hints.PricingClassisstandardonly;customjoins the retired class names.standardmade redundant is removed.Compatibility
targets: {ttft_ms, itl_ms, priority}is unchanged on the wire and now applies as hints.class_pricing: customfails validation (startup and the offline validator). Move such a price tostandard.onwardsdrops public items (ServingTargets,ServingClass::Custom). Its next release is already a major.Tests
just lint rust,just lint ts: green.just test ts: 805 passed.just test rust: 4244 of 4245 passed. The failure wastest_e2e_ai_proxy_streaming_chat_completions_with_fusillade_header, a known load-timing flake; it passed 3/3 on its own and in the previous full run.main(deps/CA bundle): onwards 676 passed, and the affected dwctl suites passed.🤖 Generated with Claude Code