Improve preset creation harness

[peterschmidt85]

dstack preset create now requires max_ttft, min_context_length and concurrency. They were optional or defaulted, so two creations from the same configuration could produce very different presets: one run served 64K where another served 1M, and a trial whose p50 TTFT was 22.7s against a 10s cap was recorded as a normal trial and was eligible for promotion. This PR also pins the benchmark workload with input_tokens and output_tokens, so trials and the final service measure the same thing, and records what each trial taught on the trial record.

type: preset
name: dsv4-flash

# The agent picks a compatible variant of the base model
base: deepseek-ai/DeepSeek-V4-Flash

# Consider only the specified fleets
fleets: [b200-2x]

# The requirements the preset must meet (time to first token is in milliseconds)
min_context_length: 1048576
max_ttft: 675

# The number of simultaneous requests every benchmark uses
concurrency: 1

# The request shape every benchmark uses (defaults to 1024 and 1024)
input_tokens: 10000
output_tokens: 1500

# Make the first trial a reference point rather than an optimization attempt
baseline: true

# The number of benchmarked trials
trials: 10

dstack preset shows the constraints a preset was created against next to what was measured, with one sparkline glyph per trial:

$ dstack preset -a --base deepseek-ai/DeepSeek-V4-Flash
 ID        BASE                           GPU           CONSTRAINTS                   BENCHMARK                          STATUS              SUBMITTED
 c83375b4  deepseek-ai/DeepSeek-V4-Flash  B200:180GB:2  io=10000/1500 conc=1          tok/s/user=319 ttft=220ms ctx=1M    interrupted (5/10)  8 hours ago
 de7a07c1  deepseek-ai/DeepSeek-V4-Flash  B200:2        io=10000/1500 conc=1          tok/s/user=140 ttft=519ms ctx=1M    verified (5)        19 hours ago
 014c3216  deepseek-ai/DeepSeek-V4-Flash  H100:80GB:4   io=8K/1K prefix=90% conc=648  tok/s/user=11.2 ttft=4.89s ctx=1M   verified (8)        yesterday
 7fe54fdf  deepseek-ai/DeepSeek-V4-Flash  H100:80GB:4   io=8K/1K prefix=90% conc=648  tok/s/user=14.2 ttft=4.15s ctx=32K  verified (7)        2 days ago

Main changes:

• max_ttft, min_context_length and concurrency are now required. Without a latency bound, maximising throughput has a degenerate optimum, and without a context floor two runs under identical constraints are not comparable.
• input_tokens and output_tokens pin the benchmark workload, so trials and the final service measure the same thing. Both default to 1024.
• baseline: true makes the first trial a reference point rather than an optimization attempt, so an improvement percentage means something.
• The trial record gains learned, required for every trial including a failed one, and failed for a benchmark that broke a constraint. A failed trial keeps its benchmark, since that is what the next trial learns from, and is excluded from best-trial selection.
• findings.md is removed. It asked the agent to enumerate what it had not tried, which is unbounded, so the agent produced an arbitrary subset and presented it as complete. What a trial taught now lives on the trial record.
• The listing always shows prefix, including prefix=0%. It decides how much of each request
  the engine serves from its prefix cache, so two rows are only comparable when it matches.
• A run where no trial met the constraints showed neither hardware nor a number, since both were
  read from the best trial and a failed trial cannot become best. It now shows its fastest failed
  benchmark, dimmed and marked * so it does not read as a result where styling is absent.
• dstack preset gains ps-style filtering with -a, -n, --base and --repo, the CONSTRAINTS and BENCHMARK columns above, and one glyph per trial: yellow where a trial broke a constraint, a red dot where it produced no benchmark. Pass -v to also see ctx>= and ttft<=.

Behavior changes:

• Per-user speed is the steady decode rate, 1/TPOT, as the serving literature defines it. Dividing aggregate throughput by concurrency folds in time to first token and read about 9% low. The preset row and the session row previously disagreed on the same run.
• The final benchmark runs inside the service replica, directly against the engine, so it is comparable with the trial benchmarks instead of measuring the proxy and a different client.
• A benchmark may not reuse the previous benchmark's prompts. Consecutive runs against a live engine were hitting its prefix cache and inflating later trials. Trial and final-service benchmarks of the same configuration disagreed by 8.6% before the fix, and by 2.5% and 0.4% after.

Breaking changes:

• max_trials is now trials in the configuration, and trials_num in the constraints the agent reads, since it is the number of trials rather than a ceiling.
• context_length in the configuration is now min_context_length, since it is a requirement. context_length keeps the measured value in the saved preset.
• max_ttft, min_context_length and concurrency no longer have defaults.

Existing configurations fail with extra fields not permitted. Presets are experimental, so no aliases were added.

Internal changes:

• Final service attempts are recorded in verifications.jsonl, so the CLI reads the phase instead of inferring it from a spent trial budget. That inference was wrong for every run that stopped early.
• Session constraints are read from the session directory rather than the agent workspace, which is deleted when a session finishes. A verified preset could not show what it was created against.
• shared_prefix_tokens is documented on the concepts page. It existed only in the generated
  schema reference, so the property that decides the benchmark's prefix cache hit rate was
  invisible to anyone reading the concept.
• The endpoints to presets rename in [Feature] Presets #4058 deleted docs/concepts/endpoints.md and
  docs/reference/cli/dstack/endpoint.md without adding redirects, so both URLs returned
  404. They now redirect to their preset equivalents.
• Maintainer notes written as <!--!...--> are stripped from the rendered agent prompt, so the prompt can carry TODOs without the agent seeing them.

What the agent found for that configuration

$ dstack preset
 ID        BASE                           GPU           CONSTRAINTS           BENCHMARK                          STATUS
 c83375b4  deepseek-ai/DeepSeek-V4-Flash  B200:180GB:2  io=10000/1500 conc=1  tok/s/user=319 ttft=220ms ctx=1M  ▂▁██▇  interrupted (5/10)

The baseline trial measured 157 tok/s/user. The best that held up was trial 4 at 309, roughly 2x the baseline, at the full 1M context and inside the 675ms bound.

The agent arrived at SGLang with EAGLE speculative decoding, the dsv4 attention backend, the flashinfer_mxfp4 MoE runner and an fp8 KV cache, none of which was suggested to it:

python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V4-Flash --tp 2 \
  --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 --attention-backend dsv4 \
  --moe-runner-backend flashinfer_mxfp4 --enable-deepseek-v4-fp4-indexer --kv-cache-dtype fp8

Two things in that row come from this PR. Trial 3 benchmarked highest at 319 but ran out of memory prefilling 1M tokens as a service, so verification rejected it and trial 4 was promoted instead. Trial 5 is the yellow glyph: it benchmarked at 289 but broke a constraint, so it keeps its benchmark for the next trial to learn from and is excluded from selection. The session reads interrupted because it was stopped after 5 of the 10 trials.

The Presets concept page is updated to match, including a roadmap of what is not supported yet: changing code, multi-node and P/D disaggregation, --previous, and concurrency ranges.

Rebased onto Pydantic v2 (#4077).

---

AI assistance: written with Claude Code.