Skip to content

docs(strata): say the bench subcommand is not in this build #212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
akshathayadav378-boop wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs(strata): say the bench subcommand is not in this build #212

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions edge0-strata/docs/BENCHMARK_PROTOCOL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,10 @@
**Status: draft, implemented in `strata bench run`; pilot validation pending.** Protocol ID: `strata-qwen38-thinking-v1-draft.1`.
Result format: `strata-standard-benchmark`, schema version `1`.

> **Availability in this repository:** the `edge0-strata` build here has no `bench` subcommand, so
> `strata bench run` cannot be used yet. Use the community scripts in
> [COMMUNITY_BENCHMARKS.md](COMMUNITY_BENCHMARKS.md) until the runner is ported.

This document specifies a repeatable community benchmark and the data a future website can consume. The
required suite took **2 to 4 minutes after model loading** on the two machines it has been run on (RTX 3060 and
2x RTX A4000, IQ3_XXS, Strata 0.2.1: 3 min 22 s and 2 min 22 s; the optional concurrency stage added 1.5 to 2 minutes).
Expand Down
3 changes: 3 additions & 0 deletions edge0-strata/docs/COMMUNITY_BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ sampling, twelve sanity checks, about 2 to 4 minutes after the model has loaded
the hardware, versions and every request. For a quick look at your own machine, [`BENCH.bat` / `./bench.sh`](#run-the-benchmark)
takes 3 to 10 minutes and writes a report in the format below.

> The `edge0-strata` build in this repository has no `bench` subcommand yet, so the standard benchmark
> cannot be run here; the community scripts above are the way to measure for now.

The protocol is still a draft until a pilot has run on two kinds of hardware. The reports listed below were made before
it existed and are not standard results; they stay valid under this guide.

Expand Down