Activity

1. Dong1017

   added

   area:workbenchCLI workbench: profile, A/B loop, quality gates, evidence reportsCLI workbench: profile, A/B loop, quality gates, evidence reports

   help:wantedNeeds an owner or contributorNeeds an owner or contributor

   kind:trackingTracking umbrella that follows child workTracking umbrella that follows child work

   on Oct 29, 2026
2. Dong1017

   added a parent issue on Oct 29, 2026

   • task: measure whether InferRSI helps Claude Code and Codex optimize inference tasks #248
3. amy-why-3459 commented on Oct 29, 2026

   

   amy-why-3459

   on Oct 29, 2026

   Collaborator

   More actions

   [RFC] Optimization search policy: candidate admission, cost ladder and stop contract

   Status: Proposed. Base revision a8215b3.
   Creates no threshold, revision pin, campaign, promotion or execution authority.
   Related: this comment on #252; depends on #252 for task/oracle packages, orthogonal to #249 (parallel lanes), extends row 4 of the campaign plan RFC #30.

   Item	Value
   Scope	Machine-readable search-space projection; candidate card schema; admission gates; cost tier → RunSpec mapping; phase-locked round recording; stop evaluator; negative-result ledger
   Out of scope	Candidate synthesis, code editing by a model, autonomous execution, agent-side hook enforcement, evaluation.py statistics, parallel lanes (#249), benchmark task packages (#252)
   Reuses	Campaign plan row 4 field set; RunSpec (src/inferrsi/runspec.py); AB/BA runner; assessment() short-circuit shape (src/inferrsi/harness/improvement.py); existing rejected-direction cases
   Adds	No 🟢 SearchSpace cell, no measured coverage, no skill, no claim

   ---

   1. Problem

   optimization-search-policy already states a complete policy: four admission considerations, a seven-phase locked round, a six-tier cost ladder and four stop criteria. Every one of those is prose. Nothing in src/inferrsi/ reads it.

   The gap is visible in three concrete places:

   1. Row 4 of the campaign plan already specifies the candidate card — ID, plane, target stage and its measured exposed share of E2E, conditional headroom, diagnostic, hypothesis, mechanism class, risk class, cost tier, prior art or negative record, kill criterion, status, evidence link. It is a human-filled table cell. There is no schema, no validator and no way to ask "which cells are still admissible".
   2. Prior-art and negative-record lookup is unimplemented and, as written, contradicted. The policy requires recall-complete free-text grep over full case bodies, while src/inferrsi/catalog.py:127 scores records by token-AND (every token must appear) and ranks by raw occurrence count. That is a precision-ordered substring index, not a recall-complete case search. The three existing rejected-direction cases are plain Markdown with no query interface.
   3. The stop criteria have no inputs. S1 needs measured uncertainty and removable time; today evaluation.py:36 reports range_pct_of_median as descriptive only and asserts "statistical_significance": "not established". S3 needs a verdict history across rounds; nothing aggregates rounds.

   The result: the policy is correct and unusable. An agent can read it and still re-enter a recorded dead end, because nothing makes the dead end queryable at the moment of candidate generation.

   The narrow claim of this RFC: the admission, ordering and stopping decisions can be made deterministic over declared inputs and existing evidence, without synthesizing candidates and without spending compute. That is a contract problem, not a modelling problem.

   ---

   2. Design principles

   1. Row 4 is the candidate card. Do not introduce a second ontology. This RFC defines a machine-readable form of row 4 plus the evaluation over it, and defers to docs/searchspace-matrix.md as the authoritative coverage matrix. No third matrix.
   2. Gates produce refusals, not rankings. A gate can kill a candidate, mark it unadmissible without a diagnostic, or raise its burden of justification. It never scores two candidates against each other. Ranking by expected value is a further decision that needs the measurement contract to exist first.
   3. Absent input stays absent. Every gate that lacks an input returns unverified with the cheapest check that would resolve it. No default fraction, no assumed share, no fallback threshold. This matches the existing INCONCLUSIVE > FAIL > PASS precedence.
   4. Immutability follows row 4's freeze semantics. A kill criterion and a cost tier are fixed once the candidate has run; changing either requires a new candidate ID, not an edit.
   5. The policy governs records, not agent behaviour. vllm-evolve enforces phase order with a PreToolUse hook. InferRSI cannot and should not: it is not in the agent's tool path. The enforceable analog is that a round which skips a phase cannot be recorded as a qualifying round, and an unrecorded round cannot feed a promotion or a stop decision.

   ---

   3. Objects

   3.1 Search-space cell index

   Generated from docs/searchspace-matrix.md by extending tools/build_skill_coverage.py (which already resolves SEARCHSPACE = Path("docs/searchspace-matrix.md")), never hand-authored. The matrix remains authoritative; the index is a derived artifact with a --check mode that fails on drift.

   {
     "schema_version": 1,
     "id": "accelerator.kv.admission",
     "layer": "accelerator",
     "stages": ["diagnose", "configure", "verify"],
     "target_classes": ["serving", "kv-cache"],
     "skill_ids": ["diagnose-vllm-admission-pressure"],
     "evidence_pins": [{"source": "<repo>", "commit": "<40-hex>"}],
     "maturity": "source_candidate",
     "plane_hint": "model",
     "pruning_claims": [
       "Queue length and KV usage do not establish allocation pressure",
       "stats are gated and incomplete"
     ],
     "rejected_direction_refs": ["casebook/cases/qwen-ascend-rejected-directions.md"]
   }

   pruning_claims is the load-bearing field. The matrix's Cases / Experience column currently holds falsifying statements that only a human can read — "Shape agreement is not conversion", "CPU bypasses accelerator capacity and segmentation gates", "matching total tokens does not preserve peer rows". Those are pre-registered counter-claims. Surfacing them as queryable constraints is the cheapest available reduction of the candidate space, and it costs no measurement.

   maturity carries the existing key unchanged (covered / partial / gap); source_candidate and not_run remain distinct states and neither becomes an admissibility signal.

   3.2 Candidate card

   The machine form of row 4. Field names are the campaign plan's; this adds typing, not vocabulary.

   {
     "schema_version": 1,
     "id": "D-01",
     "plane": "model",
     "cell": "accelerator.kv.admission",
     "hypothesis": "If admission accounting charges <mechanism>, <metric> improves because <causal step>",
     "mechanism_class": "ownership",
     "risk_class": "lossless-exact",
     "cost_tier": "profile-run",
     "stage": {"name": "<stage>", "exposed_share_pct": null, "share_source": "unmeasured"},
     "headroom": {"removable_ms": null, "assumptions": [], "uncertainty_ms": null},
     "diagnostic": {"cost_tier": "profile-run", "kill_criterion": "<observable>"},
     "prior_art": [],
     "negative_record": [],
     "falsification": {
       "metric_key": "<dotted.positive.key>",
       "direction": "minimize",
       "criterion": "<the observation that would reject this hypothesis>"
     },
     "kill_criterion": "<immutable once run>",
     "status": "planned"
   }

   Validation rules (all reject, none warn):

   • falsification.criterion nonempty, and falsification.metric_key resolves to a positive finite number under the existing metric contract (model/protocols/metrics.py). A candidate that no single metric could reject is not a hypothesis; it is a wish.
   • kill_criterion nonempty and not equal to the falsification criterion.
   • plane ∈ harness | model | infra, one per candidate (AGENTS.md RSI contract).
   • cell must resolve in the generated index; mechanism_class must be one of the seven declared tags.
   • stage.exposed_share_pct may be null only when stage.share_source == "unmeasured" and a diagnostic is attached.

   3.3 Admission gates

   Ordered cheapest-first. Each returns admit | refuse | admit_conditional | unverified, with a reason and, where applicable, the cheapest resolving action.

   G0 — Amdahl plausibility.

   upper_bound = exposed_share(cell) × plausible_removal(mechanism_class)
   refuse if upper_bound < residual_target
   

   residual_target is campaign-declared, never a tree default. This is an upper bound on what the stage could yield, not a prediction, and it is the cheapest way to discard a candidate before any compute is spent. Missing share → unverified with required_action: "run the cheapest diagnostic in this cell" (typically a profile run, which is a measurement, not a comparison).

   G1 — Mechanism class alignment. The declared mechanism_class must be reachable from the measured bottleneck class. A residency candidate against a compute-bound stage is refused with reason: class_mismatch. The five/six families the policy names — ownership, work elimination, residency, algorithm, configuration, fusion/operator, graph/launch, parallelism/communication, precision/approximation — are the declared tag set from row 4 and are not extended here.

   G2 — Prior art and negative record. Two lookups, deliberately different:

   • Prior art (recall-oriented). Full-text grep across case bodies, per the policy's explicit requirement. Result is a list of leads.
   • Negative record (classification-oriented). Structured query over the ledger (§3.7), keyed by (cell, mechanism_class, backend, device, model, workload, evidence_level).

   On a negative-record hit the gate returns admit_conditional, never refuse. The card must then name:

   1. the rejection class — one of quality_rejection, no_demonstrated_gain, resource_failure, confounded_control, unintegrated_lead, analytical_rejection;
   2. what changed since that record (revision, device, workload, evidence level);
   3. the cheapest check that would distinguish the two conditions.

   A keyword match is a lead, not a causal diagnosis and not an automatic veto. A different implementation failing does not ban a mechanism family.

   G3 — Cost tier assignment. The cheapest tier that can separate the hypothesis from local noise, declared before measurement. A tier above the minimum needs a stated reason. Escalation is one-way and only on a positive result at the current tier.

   3.4 Cost ladder → RunSpec

   The policy's six tiers map onto RunSpec knobs. The mapping is shape, not numbers; tier presets are campaign-scoped in rows 4 and 6.

   Tier	RunSpec shape	May produce a verdict?
   static/config	no benchmark command; validation command only	correctness only — never a performance verdict
   microbench	narrow metric_key, single scenario, minimal repetitions	screening only
   profile-run	measurement run, no A/B pair	no verdict; supplies exposed_share for G0
   scenario@c1	full paired RunSpec, one declared regime	screening PASS/FAIL; not acceptance
   multi-regime	paired RunSpec, declared regime set	may support acceptance if rows 3/5 declare it
   full-cohort	the frozen acceptance regime	the only tier that can feed a public claim

   Two hard rules:

   • Escalate only on positive. A negative or inconclusive result at tier k does not buy a run at tier k+1.
   • Screening may reject early; it may never substitute for the acceptance regime. A scenario@c1 PASS is not an acceptance PASS, and no promotion rule may treat it as one. This mirrors the CRISP adaptation boundary already recorded in the policy: screening rejects, it does not conclude.

   3.5 Phase-locked round

   Seven phases, with the existing runner as P5's executor. The lock is enforced on the record:

   flowchart LR
     P1["P1 Read context<br/>cell + prior art + negative record"] --> P2["P2 Design<br/>candidate card"]
     P2 --> P3["P3 Apply<br/>bounded, one plane"]
     P3 --> P4["P4 Verify correctness"]
     P4 --> P5["P5 Evaluate<br/>AB/BA at declared tier"]
     P5 --> P6["P6 Compare<br/>better / worse / inconclusive"]
     P6 --> P7["P7 Keep or discard<br/>ledger entry, unconditional"]
     P7 --> P1
     P4 -.->|"fails"| L["Ledger: correctness_rejection<br/>no benchmark budget spent"]
     L --> P1
   

   Loading

   A ledger entry is qualifying only if it records P1 inputs (the cell and the negative-record lookup result), a P2 card, a P4 outcome, and a P5/P6 result. An entry missing P4 is recorded as unqualified and cannot be cited by a stop decision or a promotion.

   This is weaker than a hook by design: it cannot stop an agent from running a benchmark out of order, it can only refuse to treat the result as evidence. Given the repo's non-autonomous posture, that is the correct strength.

   3.6 Stop evaluator

   Deterministic over the ledger. Each criterion yields {decision: stop | continue | unverified, reason, required_action}.

   ID	Criterion	Inputs	Behaviour when input missing
   S1	Current-mechanism headroom exhausted	removable exposed time, uncertainty	unverified — never infer headroom from a roofline
   S2	Amdahl ceiling reached	residual target, plausible removal over remaining stages	unverified
   S3	Repeated inconclusive on the same hypothesis	count of consecutive inconclusive for one hypothesis hash	continue with required_action: "repair the measurement protocol or change hypothesis"
   S4	Negative record saturated	every cell in the declared scope has a terminal ledger entry	stop with required_action: "escalate abstraction level"

   S1 must not be satisfied by proximity to a roofline. The existing policy says this explicitly and it matters: changing the algorithm changes both work and traffic, so proximity to the old roof does not rule out improvement. The evaluator encodes that by requiring a measured removable quantity, and returning unverified otherwise.

   S3's repetition count is campaign-declared. No tree default.

   3.7 Negative-result ledger

   One record per terminal round outcome, including worse and inconclusive. This is the compounding asset: the repository already keeps three rejected-direction cases by hand, and each one exists because someone paid to learn it.

   {
     "schema_version": 1,
     "candidate_id": "D-01",
     "cell": "accelerator.kv.admission",
     "mechanism_class": "ownership",
     "context": {
       "repo": "<repo>", "revision": "<40-hex>", "backend": "ascend",
       "device": "<exact SKU>", "model": "<model>", "workload": "<id>"
     },
     "evidence_level": "scenario@c1",
     "hypothesis": "<as tested>",
     "metric_key": "<key>",
     "verdict": "FAIL",
     "rejection_class": "no_demonstrated_gain",
     "observed": "<measured values and uncertainty>",
     "constraint_derived": "<the search constraint this failure establishes>",
     "reopen_condition": "<what changed would justify retrying>"
   }

   constraint_derived is mandatory. A negative result that establishes no constraint has not narrowed the space and cannot satisfy S4.

   ---

   4. Relationship to existing work

   Work	Relationship
   #252	Supplies the task package and oracle that P4/P5 consume. This RFC defines which candidate is admissible and what tier measures it; #252 defines what the measurement runs against. Neither duplicates the other's truth.
   #249	Parallel lanes are an orthogonal scheduling change. A lane submits a candidate card; the round records it. Nothing here assumes one lane.
   #30 (campaign plan RFC)	This RFC is the schema and evaluator for row 4, and defers to rows 1/2/3/5/6/7 unchanged. It proposes no new frozen row.
   #248	Harness benefit experiment. The ledger feeds it with per-round outcomes; this RFC does not run it.
   #91	SearchSpace knowledge coverage. The cell index is derived from that coverage, not a second copy of it.
   #88	SkillEval. Its assessment() short-circuit ordering (incomplete_evidence → unknown_usage → quality_floor → regression_limit → no_improvement) is the precedent for §3.3's ordering, not a component to reuse: it scores skill-text edits, not inference candidates.
   optimization-search-policy skill	Promoted from prose to contract. The skill remains the human-readable rationale and gains references to the generated schemas.

   ---

   5. Dependency and ordering

   The gates can land first; multi-round search cannot.

   Two preconditions for using this contract across more than one round:

   1. Measurement resolution. harness/evaluation.py:36 compares two independent medians and discards the pairing that the AB/BA schedule already established. Over repeated rounds, selecting the best of N unvalidated comparisons is selection bias, not tuning, and it will deposit noise into the ledger as if it were a constraint. The minimum repair — paired difference plus a resolvability gate that refuses PASS when baseline spread is on the order of min_improvement_pct — is a change to evaluation.py, not to this RFC, and is not proposed here as a slice.
   2. Candidate-space indexing. Without the cell index and the negative-record query, G0/G1/G2 return unverified for everything and the gates reduce to a schema validator.

   So the slices below are ordered accordingly, and no slice depends on statistical work.

   I am not claiming that fixing resolution is sufficient. It is a precondition for round count > 1, and stating it here is cheaper than discovering it after a campaign.

   ---

   6. Delivery slices

   Slice	Content	Depends on	Hardware
   A	Cell index generated from the matrix; --check drift gate	—	none
   B	Candidate card schema + validator; rejects unfalsifiable cards	A	none
   C	Negative-record ledger schema + query over the three existing cases	A	none
   D	G0–G3 evaluation; unverified paths	A, B, C	none
   E	Cost-tier → RunSpec mapping + escalation refusal	B	none
   F	Round recorder: qualifying vs unqualified entries	B, E	none
   G	Stop evaluator S1–S4 over a ledger fixture	C, F	none
   H	One synthetic end-to-end fixture: cards in, gate decisions and stop decision out	D, G	none

   Every slice is model-free and hardware-free. No slice spends compute, allocates a device, calls a paid model, or produces a measured claim.

   ---

   7. Acceptance criteria for this RFC

   • A1. The generated index round-trips against the matrix, and --check fails on an unreconciled edit. The matrix remains the only authoritative source; the index cannot be edited by hand.
   • A2. The card validator refuses: no resolvable positive-finite metric; empty kill criterion; kill criterion equal to the falsification criterion; more than one plane; an unresolvable cell; a null stage share with no diagnostic attached.
   • A3. A negative-record hit yields admit_conditional, never refuse, and a card without what changed fails to validate.
   • A4. A card cannot be assigned a tier above the minimum without a stated reason, and a scenario@c1 result cannot satisfy a promotion rule.
   • A5. A round missing P4 is recorded unqualified and is rejected as input to S3/S4.
   • A6. S1 and S2 return unverified — not continue — when headroom inputs are absent.
   • A7. A negative entry with an empty constraint_derived cannot satisfy S4.
   • A8. One synthetic fixture demonstrates that the gates select the subset a human would, and that the fixture contains no measured performance value.
   • A9. No new 🟢 cell, no measured coverage, no skill promotion, no claim. Every new artifact is status: candidate or a schema.

   ---

   8. What this RFC does not authorize

   • No execution, device allocation, paid model call or benchmark run.
   • No change to evaluation.py, the runner, the AB/BA schedule or any frozen row.
   • No numeric default enters the tree. Repetition counts, tier presets, residual_target and S3's N are campaign-scoped in rows 4 and 6.
   • No promotion of any knowledge, skill or SearchSpace cell.
   • No claim that the policy improves search outcomes. The policy's own benefit is unmeasured and this RFC does not establish it.

   ---

   9. Open questions

   1. Does the cell index belong in src/inferrsi/data/ as package data, or stay a tools/ artifact? Package data would let knowledge search join on cells; it would also create a second shipped copy of the matrix.
   2. Should G2's prior-art lookup use the existing token-AND catalog or a separate grep path? The policy demands recall-complete free-text search, and catalog.py:127 is precision-ordered by construction. Separate paths risk the two disagreeing.
   3. How is plausible_removal(mechanism_class) bounded without inventing numbers? Options: campaign-declared per class, derived only from recorded positive results in the ledger, or left unverified until a campaign declares it. The third is honest and the least useful; the second cannot bootstrap on an empty ledger.
   4. Does an unqualified round still write a negative ledger entry? It carries information but fails the P4 precondition. Recording it pollutes S4; discarding it loses a real dead end.
   5. Should the ledger be issue-scoped or in-tree? Campaign parameters and numeric gates stay issue-scoped per AGENTS.md; a ledger that only lives in an issue is not queryable by the gate that needs it.

   ---

   Proposed for review. Comment on scope and slice boundaries before any implementation claim. No part of this proposes reopening #248, #249 or #252.
4. Dong1017 commented on Oct 29, 2026

   

   Dong1017

   on Oct 29, 2026

   CollaboratorAuthor

   More actions

   Issue/design review of the body updated 2026-10-29T11:14:20Z and the optimization-search-policy RFC, against main 7d1d82f55cda89842bd23ee2da91b521ee60eb08. Source/document inspection only; no benchmark, model, device or workflow execution.

   The shared task/environment/oracle foundation remains a sound bounded goal. The RFC proposes a separate search-policy contract; it should not silently become a prerequisite or expand this tracker's benchmark acceptance. Please revise these three design points before implementing that contract:

   1. P2 — G0 rejects useful incremental candidates (§3.3). upper_bound < residual_target compares one candidate's ceiling with the entire remaining campaign target. In a hypothetical campaign targeting a 10% latency reduction, two independent changes could each remove 6% of the original E2E time and jointly meet the goal, yet G0 refuses both before either runs. Preserve candidate contribution/composition, and reserve the whole-target impossibility test for the combined remaining opportunity. The existing search policy applies its campaign ceiling to all remaining stages together; the campaign template also retains conditional headroom, overlap, queueing, capacity and uncertainty. An unknown or inapplicable bound should remain unverified.

   2. P2 — Static/profile tiers cannot be represented by RunSpec presets alone (§3.4–3.5). static/config supplies only validation, and profile-run explicitly has no A/B pair. Current RunSpec.validate() requires nonempty baseline and candidate commands, and the runner executes paired measurements before optional validation. Define diagnostic/correctness records and their transitions/qualification separately, keeping RunSpec for actual comparisons. Do not fabricate a baseline or performance verdict to pass validation. Clarify how a no-verdict diagnostic can advance under the positive-only escalation rule without pretending it completed P5/P6 comparison.

   3. P2 — S4's per-cell terminal-entry test does not establish search exhaustion (§3.6–3.7). One matrix cell can contain several mechanisms and candidates. One failed candidate in each cell still leaves other candidates untried; G2 itself correctly says a negative hit is conditional evidence, not a veto. Scope exhaustion to an explicitly declared, versioned candidate/hypothesis set and matching context, consistent with the existing candidate-based stopping rule. Retain all terminal records, including correctness failures and inconclusive runs, but keep retention separate from eligibility to exclude a candidate or satisfy S4; inconclusive evidence need not establish a causal search constraint.

   Next: @Dong1017 should retain #252's small shared-task scope, identify an accountable owner for the proposed policy contract, and settle these admission/execution/stop semantics before delegating it. Implementation owner and initial benchmark subset remain unassigned; neither this review nor the RFC demonstrates search benefit, accepts a benchmark delivery, or authorizes execution.
5. Dong1017

   added

   status:proposedWaiting for the decision named in the status blockWaiting for the decision named in the status block

   on Nov 2, 2026
6. Dong1017

   assigned

   Bounty-hunter

   on Nov 2, 2026