Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions edge0-strata/docs/BENCHMARK_PROTOCOL.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Strata standard benchmark protocol

**Status: draft, implemented in `strata bench run`; pilot validation pending.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`.
**Status: draft, implemented in the Terminal's `strata bench run`; pilot validation pending.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`.
Result format: `strata-standard-benchmark`, schema version `1`.

This document specifies a repeatable community benchmark and the data a future website can consume. The
Expand All @@ -27,13 +27,15 @@ output or add a protocol ID to make it appear standard. The draft becomes final

## Run it

Install Strata, set up the Flash-Next model you want to measure, and close other programs that use the GPU. Then:
Install Strata, set up the Flash-Next model you want to measure, and close other programs that use the GPU. Then, with the Terminal launcher (`bin/strata` from a source checkout, or `strata` from the Terminal archive):

```sh
strata bench run --profile NAME --track default --track-notes "0.2.1 release, setup defaults"
strata bench run --url http://127.0.0.1:8080 # a server that is already running
```

The engine binary (`strata.exe`) has no `bench` subcommand, so running these against it fails with `unknown argument: bench`.

With `--profile` the runner starts that profile's engine, waits for the model to load, runs the protocol, and stops
the engine it started. If the profile is already serving, the runner uses that server and leaves it running; a
server that has already answered requests is not freshly loaded, and the report says how many it had answered.
Expand Down
2 changes: 2 additions & 0 deletions edge0-strata/docs/COMMUNITY_BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,8 @@ and their limits. Report what you actually measured and label estimates separate

## Run the standard benchmark

Run it with the Terminal launcher (`bin/strata` from a source checkout, or `strata` from the Terminal archive). The engine binary (`strata.exe`) has no `bench` subcommand.

```sh
strata bench run --profile NAME --track default --track-notes "release 0.2.1, setup defaults"
strata bench run --url http://127.0.0.1:8080 # a server that is already running
Expand Down
1 change: 1 addition & 0 deletions edge0-strata/src/program/generate.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -1188,6 +1188,7 @@ void usage() {
" --peer-device N a second GPU as an adaptive expert-cache tier (rows over P2P; it also\n"
" computes its experts' prompt rows). Not with --layer-split or\n"
" --expert-cache-device1..3. Default output unchanged without it.\n"
" --vram-reserve-mib N VRAM left free for the desktop, the driver and other processes (default 700)\n"
" --peer-reserve-mib N VRAM the peer tier leaves free on its card (default 600)\n"
" --peer-slots N expert slots on the peer (default: what fits)\n"
" --peer-adapt-swaps N peer cache swaps per adaptation step (default: --adapt-swaps)\n"
Expand Down
2 changes: 1 addition & 1 deletion edge0-strata/tools/strata_bench.py
Original file line number Diff line number Diff line change
Expand Up @@ -1125,7 +1125,7 @@ def main(argv=None) -> int:
say(f"\nReport: {shown / 'README.md'}\nShare it: open a pull request that adds the folder {shown} (see docs/COMMUNITY_BENCHMARKS.md), "
f"or paste README.md into a GitHub issue. Home folders, user and host names and IP addresses are already replaced; "
f"read it once before you share it.")
say("For results other people can compare with, run the standard benchmark: strata bench run (docs/BENCHMARK_PROTOCOL.md).")
say("For results other people can compare with, run the standard benchmark with the Terminal launcher: strata bench run (docs/BENCHMARK_PROTOCOL.md). The engine binary has no bench subcommand.")
return 1 if out["failures"] and not any(out[k] for k in ("chat", "prompts", "agent")) else 0
finally:
if started:
Expand Down