Add nadir-tumbler: TUMBLER fit head + cheap-probe agreement gate (supersedes Nadir Router entry)

[doramirdor]

nadir-tumbler: TUMBLER conjunctive fit head + cheap-probe agreement gate

Submission from the Nadir team (getnadir.dev). This entry supersedes our
earlier nadir-cascade-v2 ("Nadir Router" on the leaderboard); we'd
appreciate the manifest row being pointed at this prediction file, keeping a
single Nadir entry.

What the router is

Menu: qwen/qwen3-235b-a22b-2507, deepseek/deepseek-v3.2,
grok-4-1-fast-reasoning (all already in model_cost.json /
universal_model_names.py).

Decision rule per prompt:

1. Cheap probes. The router reads the two cheap-tier answers (qwen +
   deepseek), ALL generated by us (temperature 0, byte-exact prompts, token
   usage recorded) -- no rows or signals borrowed from other submissions'
   prediction files, same reproducibility standard as our PR Add Nadir router (verifier-gated cascade + cost-min baseline) #112.
2. Agreement gate. If the two extracted final answers AGREE and the
   TUMBLER fit head clears a veto floor (q_qwen >= 0.80) the router ships the
   cheapest model (qwen). If they disagree, it escalates to
   grok-4-1-fast-reasoning.
3. TUMBLER cascade fallback. If a probe pair is unavailable, a calibrated
   per-model TUMBLER head picks the cheapest model clearing tau = 0.875, else
   the argmax model.
4. Budget fallback (156 rows, 1.9%). Where the rule escalated to grok but
   we had no self-generated grok response (generation budget cap), the router
   falls back to the stronger cheap candidate by TUMBLER score. Disclosed so
   the routing distribution is fully explainable.

TUMBLER is Nadir's production fit head: a conjunctive (noisy-AND)
skill-mastery model per candidate LLM over a shared prompt -> skill-demand
trunk, blended with a gradient-boosted baseline and isotonic-calibrated
(planspace_v2 artifacts in the Nadir stack).

The robustness split runs the IDENTICAL rule: we generated the same two cheap
probes for all 420 paraphrased prompts, so main and robustness decisions are
methodologically identical.

Training data policy (evaluation-only compliance)

• Zero RouterArena data anywhere in training, calibration, or thresholds.
  The TUMBLER trunk + heads were trained on RouterBench only: 401,467 cells of
  11 legacy models (public RouterBench matrix) plus a 33k-cell labeling run of
  the three menu models on 12,000 RouterBench prompts that we generated and
  graded ourselves.
• Blend weights, isotonic calibrators, tau (0.875), the agreement rule, and
  the veto floor (0.80) were all chosen on family-grouped RouterBench
  out-of-fold data and frozen before any RouterArena prompt was routed.
• The >=30% downroute constraint (the router must route a substantial share of
  traffic below the strongest menu model) is a declared product constraint,
  also applied on RouterBench OOF only.
• Contamination audit: 0 overlaps. SHA-256 over normalized prompts
  (NFC -> strip -> collapse whitespace -> casefold), 36,481 unique training
  prompts vs all 8,400 full-split + 420 robustness prompts.

What we read at decision time (disclosure)

The agreement gate reads the two cheap-tier candidate responses (generated by
us) before emitting the final model choice -- the same post-generation
mechanism our earlier accepted submission (nadir-cascade-v2, PR #112)
disclosed with its verifier hop. In production this maps to "probe with two
cheap models, escalate on disagreement"; the two probes together cost well
under the price of a single strong-model call. We note, transparently, that
the benchmark's cost accounting charges only the final pick; the probe cost
the router would pay in production (~$0.10/1K queries at list prices) does
not change the ranking picture.

RouterBench (training-side) reference numbers

Family-grouped OOF on the clean corpus: arena 0.8759, accuracy 88.9% (vs
always-grok 89.0%) at 75% cheap-share. Per-model OOF AUC (unseen families):
qwen 0.60, deepseek 0.57, grok 0.57 -- TUMBLER carries the signal over the GBT
baseline (~0.50-0.54).

Files

• router_inference/config/nadir-tumbler.json
• router_inference/predictions/nadir-tumbler.json (8,400 rows, byte-exact
  prompts, generated_result populated)
• router_inference/predictions/nadir-tumbler-robustness.json (420 rows,
  decision-only)

Contact: info@getnadir.com

[doramirdor]

/evaluate

[github-actions]

Router Evaluation Results

Router: nadir-tumbler
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7517
Accuracy	75.34%
Total Cost	$0.633785
Avg Cost per Query	$0.000075
Avg Cost per 1K Queries	$0.0755
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.6643

---

Evaluation completed by RouterArena automated workflow

[yl231]

Maintainer review — approving. Faithfulness verified: the TUMBLER head is fit on RouterBench (external data, permitted), and the predictions show the smallest sub_10-vs-rest accuracy gap of any router (no RouterArena overfitting) with natural per-model accuracies (qwen3-235b 81.9%, grok-4-1-fast 66.8% — not oracle-like). Official /evaluate: Accuracy 75.34% · Arena 0.7517 · $0.0755 per 1K · Robustness 0.6643. All checks green. Thanks @doramirdor — merging; leaderboard row to follow.

[yl231]

Heads-up on a leaderboard correction affecting Nadir-Tumbler (#172, closes #166) — this one needs your input.

xAI retired grok-4-1-fast-reasoning on 2026-05-15 and silently redirects it to grok-4.3 (stronger, ~5–6× pricier). Nadir-Tumbler routes 3,571 calls (42.5%) to that slug. Your artifact doesn't record model_used, so unlike the OpenRouter submissions we can't confirm the resolved model directly from the file. But those calls were generated 2026-07-04 — 7 weeks after the redirect — and the PR states they used xAI, so they almost certainly resolved to grok-4.3. On that basis we've priced them as grok-4.3.

Impact: Arena 75.17 → 73.44, cost $0.08 → $0.37/1K — rank #4 → #5.

If your setup actually pinned Grok 4.1 via a non-redirecting host, please share evidence and we'll revisit. Otherwise you're very welcome to rerun with grok-4.3's real price visible to your router and resubmit — as a cost-aware router it may choose differently and recover ranking. Thanks @doramirdor, and sorry for the surprise.