Skip to content

About

Open, daily tracker of decision models (Jev, Clef, Decider, Luna…): drift, noise, robustness and calibration

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

55 Commits

Folders and files

Repository files navigation

decision-drift

An open, reproducible tracker for decision models: models such as TypeSafe Jev, Cloudflare Clef, Perplexity Decider and OpenAI Luna Decisions, which return typed answers with probabilities instead of text. Every hosted decision model on OpenRouter gets the same frozen probe suite every day. The tracker reports:

  • Drift: whether answers changed relative to day 1, beyond each model's own noise floor
  • Noise floor: how much answers change between identical requests sent minutes apart
  • Silent snapshot swaps: whether an alias quietly started pointing at a new version
  • Robustness: sensitivity to option order and to paraphrased instructions
  • Accuracy and calibration on gold-labeled questions
  • Latency, error rate and cost, as measured from the runner

Layout

Path What
suite/core-v1.public.jsonl Public split of the v1 core suite. Append-only; items are never edited.
suite/manifest-v1.json Coverage, labeling method, and SHA-256 hashes of the public and private splits
suite/noise-ids-v1.json States re-sent for the noise-floor measurement
suite/models-excluded.json Models that can't answer the suite's question types, with the reason
private/ Private split and drafts (git-ignored)
results/<date>/<run>/<model>.jsonl Raw responses for public items
reports/ Per-run reports
src/ Zero-dependency TypeScript (Node ≥ 23 runs it directly)

Running

# put OPENROUTER_API_KEY=... in .env (never committed)
node src/run.ts --items suite/core-v1.public.jsonl,private/core-v1.private.jsonl --run day1 --variants shuffle,para
node src/run.ts --items <noise subset> --run day1-dup
node src/report.ts --date YYYY-MM-DD --run day1 --dup day1-dup

The runner stops when the key's spend for the current UTC day reaches BUDGET_STOP (default $5; OpenRouter's daily window resets at 00:00 UTC).

Before pushing a code change, run the check. It sends one public item per family to every model, confirms the answers parse, and builds the site. It takes about a minute and costs a few cents on your key. A single model's outage is listed as a warning and does not fail the check.

node src/check.ts [--models id1,id2] [--deadline 60]

Method notes

  • Questions follow the published System One conventions (instructions, criteria, an explicit none option). They are identical for every model and are never tuned against results.
  • Gold labels: a question is gold only when the reference label agrees with both independent LLM labelers (GPT-6.1 Sol and Gemini 4 Argon). The reference label comes from the item's author, the source dataset, or the generator for synthetic logs.
  • The private split (~30%) exists so vendors can't train on the whole suite. Only its hash and aggregate scores are published.

Sources and licenses

  • Banking77 test set: PolyAI, CC BY 4.0
  • CLINC150 ("plus" test set): CC BY 3.0
  • Authored items, synthetic logs, results and reports: CC BY 4.0
  • Code: MIT (see LICENSE; data terms in DATA-LICENSE.md)

About

Open, daily tracker of decision models (Jev, Clef, Decider, Luna…): drift, noise, robustness and calibration

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages