discussion: what trace evidence should InferRSI persist beyond per-run artifacts?

Current state: Existing per-run result/trace evidence is hash-bound, with the comparison contract requiring preservation of the complete output directory. goal_checkpoint remains a disposable, read-only view with no scheduler or persistent registry; no new durable store, index or shared event schema has been accepted here. The earlier proposal suggests starting with a concrete failed/inconclusive SkillEval handoff, without yet selecting one to separate evidence availability, bounded diagnosis/reverification and exact stochastic replay. It suggests deferring schema/ledger/index choices until a demonstrated gap warrants them; this is discussion, not an accepted scope or implementation commitment. The subsequent review identifies PR #160 and #133 as possible examples. GitHub metadata confirms their plan/results/report artifacts remain unexpired: PR #160 run, expiring 2026-11-26, and PR #133 run, expiring 2026-11-28. The review reports successful historical-controller report regeneration; metadata verification alone does not verify artifact contents or replay. Neither example is selected or accepted. The newer discussion proposal recommends prioritizing the normalized trace envelope, separately considering per-trial environment snapshots outside the candidate workspace, and explicitly deciding whether to accept or decline a bounded cross-run directory index. It raises feedback visibility, receipt isolation, deadlines, cost and redaction as design considerations. The incremental design review corrects two premises: existing run-level environment.json and per-trial source.json already record environment/provenance, and current inspection does not capture a hostname. It distinguishes the proposed per-trial non-source drift gap from those existing records, and notes that a metadata-only directory lookup need not depend on normalized trace payloads. These are the review's static findings, reused here without a new source audit. These remain proposals; no candidate, new task, implementation or acceptance owner has been authorized by that comment.
Decision needed: Whether any persistence change is needed, which concrete retained pair and bounded claim would inform that decision, and who will own acceptance. The goal/evidence acceptance owner remains unnamed; no candidate or exercise has been accepted.
Next action: Discuss the corrected sequencing proposal alongside the proposed handoff example and its evidence location, producing controller/verifier identity, authorized readers, retention/expiry and redaction limits in comments. Do not infer execution or implementation authorization from the proposal; a named owner's scoped decision is still pending.

Goal and acceptance

Problem / desired outcome: A returning agent or human can reconstruct what happened only by being handed the exact --comparison <dir> path and goal file. If that directory is lost, moved, or never named again, the trace is gone even when the goal is still open. The Codex client trace (turn.completed, item.completed, error) and the Anthropic agent trace (agent.request, agent.response, agent.tool, agent.completed, agent.failed) also use different shapes, so cross-condition tooling has to special-case each client.
Completion criteria: Not yet defined. This issue is to agree on scope before any criteria are written.

Scope and constraints

In scope / out of scope: Discuss what, if anything, should be persisted and why. This does not propose a new scheduler, authorization path, or acceptance authority. A goal or checkpoint view still cannot execute, resume, authorize, or accept work.
Relevant dependencies, shared interfaces or limits: tools/goal_checkpoint.py, docs/goal-checkpoints.md, docs/skill-eval-comparison.md (evidence hash and receipt contract), src/inferrsi/model/anthropic_agent.py, src/inferrsi/harness/skill_evaluation.py.

Candidates to discuss

Goal-checkpoint ledger. Append-only record of each checkpoint build: timestamp, goal input_sha256, recommendation action and reasons, referenced plan_sha256. The view stays read-only and non-authoritative; the ledger keeps the decision history after the last markdown or JSON snapshot is discarded.
Normalized agent-trace envelope. A shared minimal event shape (sequence, type, timestamp, request, response, tool, error) that every model client's trace_metrics maps onto. This would simplify cross-condition analysis and new client onboarding, and it touches the existing clients' event formats.
Explicit retention contract. State in evidence/README.md or docs/contributing.md which files under evidence/<campaign-id>/ must be kept and which are disposable. The current rule is only "preserve the complete output directory to keep its evidence links usable" (docs/skill-eval-comparison.md).
Cross-run index. An optional manifest listing comparison directories known to a given goal id, so goal_checkpoint does not require the caller to remember --comparison <dir>. This is the closest to the persistent registry the current design declined, and it needs the most scrutiny.

Ownership and authorization

Proposed by / actual requester: xinwen (drafted in chat; posted from the Dong1017 GitHub account)
Status: Proposed
Accountable parent (Issue-level owner, if assigned): TBD
Slice implementer(s) / acting contributor (if assigned): TBD
Goal-acceptance owner: TBD
Accepted decision or scoped delegation: Not yet granted

Inputs and deliverables

Relevant sources, revisions or linked issues: docs/goal-checkpoints.md, docs/skill-eval-comparison.md, tools/goal_checkpoint.py, src/inferrsi/model/anthropic_agent.py, src/inferrsi/harness/skill_evaluation.py
Expected artifacts: None yet. Close with a recorded decision to do nothing, or open a follow-up kind:task for whichever candidate is accepted.

Completion and closure

Who accepts completion and may close: TBD (goal-acceptance owner)
Remaining uncertainty / next decision: Which candidate, if any, and whether candidate 4 is acceptable given the stated design of no persistent registry.

The Issue asks the goal/evidence owner to decide whether any trace record should outlive the per-run comparison directory, and to pick among four candidates (checkpoint ledger, normalized agent-trace envelope, retention contract, cross-run index) before any work starts. Verdict: human decision required — the Issue explicitly defers implementation ("No implementation until a decision is recorded") and the decision owner, scope and superseded prior decision are not settled. Labels: priority:P2 (proposal awaiting a decision; P2 is a guess, since no urgency is stated and the repository wants one priority while status:proposed).

Missing information — the decision owner is a role, not a person

Where: Issue status block; docs/agent-collaboration.md:193, :215-218
Problem: The status block names the decision as coming "from the goal/evidence owner", and the ownership table leaves "Accountable parent", "Goal-acceptance owner" and "Who accepts completion" as TBD. The repository's label rule requires that a status:proposed Issue's status block "names a concrete question and its decision owner", and only that owner can move it to status:accepted (line 194). As written, nobody is on the hook to record the decision, so the Issue can sit in status:proposed indefinitely.
Why it is needed: The whole next action is "Discuss the candidates in comments"; without a named person who may accept, a comment cannot be distinguished from an opinion, and the follow-up kind:task the Issue promises cannot be authorized.
Fix: Replace the role with a GitHub handle in the Decision needed: line and in the goal-acceptance owner field.

Scope risk — candidates 1 and 4 reverse an already accepted design decision, and only candidate 4 is flagged

Where: docs/goal-checkpoints.md:18-21, :120; accepted decision comment on [RFC] Goal checkpoints: explicit progress and next actions across agent sessions #137 (2026-10-26)
Problem: The Issue frames candidate 4 as "the closest to the persistent registry the current design declined" and asks for the most scrutiny, but candidate 1 (append-only goal-checkpoint ledger) is the same class of artifact: docs/goal-checkpoints.md:18 says the view is "disposable: rebuild it from the same pinned inputs instead of updating a second status history", and :120 says "no parallel histories are required". The accepted [RFC] Goal checkpoints: explicit progress and next actions across agent sessions #137 decision explicitly excluded "no new runner, registry, ledger, scheduler, dashboard" and "no new ... accounting ledger, registry, scheduler". So accepting candidate 1 or 4 would supersede a recorded decision, not just add an option.
Impact: If only candidate 4 carries that caveat, a later implementer could read candidate 1 as pre-approved and add a decision history the accepted design deliberately excluded.
Fix: State the [RFC] Goal checkpoints: explicit progress and next actions across agent sessions #137 exclusion next to both candidates and require the recorded decision to say explicitly that it supersedes it.

Scope risk — persistence candidates omit the repository's trace redaction rule

Where: docs/privacy-security.md:52-53, docs/goal-checkpoints.md:47-49
Problem: The privacy rule says to "Filter commands, process arguments, errors, traces and generated manifests before persisting or displaying them", and the goal-checkpoint doc requires that only files the caller is authorized to read are named as inputs, with no private sessions or credentials. None of the four candidates says what may be stored (raw request bytes vs. request_sha256, tool paths, workspace paths, feedback content) or who may read a durable record later. Candidate 4 in particular would make previously ad-hoc artifacts discoverable by anyone who knows a goal id.
Impact: Deciding retention without a redaction/access rule risks turning an ad-hoc local trace into a durable record containing private paths, endpoints or credentials-bearing text.
Fix: Make redaction scope, retention period and reader authorization explicit acceptance conditions for whichever candidate is chosen.

Incomplete evidence — the listed event inventory misses event types a shared envelope must carry

Where: src/inferrsi/model/anthropic_agent.py:406, :456, :556-562; src/inferrsi/model/anthropic_edit_agent.py:522-535, :561; src/inferrsi/harness/skill_evaluation.py:100
Problem: Candidate 2 is scoped from the events listed in the Issue (agent.request/response/tool/completed/failed, turn.completed/item.completed/error). The code also emits agent.context (full-context manifest, anthropic_agent.py:406) and agent.protocol (:456), both consumed by the Anthropic trace_metrics (:579); the Codex parser treats turn.failed as an error (skill_evaluation.py:100); and the edit agent emits agent.tool with heterogeneous payloads (receipt, result_sha256, artifact_sha256, revision, :535-540) and agent.failed with reason interrupted (:561).
Impact: A "minimal shared envelope" designed from the Issue's list would drop context/protocol identity and the turn.failed distinction, so cross-condition analysis would lose the evidence the existing verdict logic depends on (task_verdict, skill_evaluation.py:121).
Fix: Add these event types and the variable agent.tool payloads to the inventory before candidate 2 is costed.

Premise refinement — some trace data is already retained and hash-bound; the real gap is raw retrieval and discoverability

Where: src/inferrsi/harness/skill_evaluation.py:341; docs/skill-eval-comparison.md:211-212, :233; evidence/README.md
Problem: The Issue says trace evidence "exists only as ad hoc per-run artifacts". In fact result.json already persists the normalized trace summary ("observations": trace, i.e. usage, commands, errors, completed_turns), evidence_sha256 already binds the execution record, agent trace and error log, and docs/skill-eval-comparison.md:233 already states the preserve rule for the comparison output directory. What is missing is the raw event stream once that directory is lost or unnamed, plus any contract for the evidence/<campaign-id>/ tree, whose README defines layout only.
Impact: Without this distinction, candidate 3 looks like a new rule when it is mostly an existing rule scoped to the comparison controller, and candidate 2 looks like it must introduce a summary format that partly exists.
Fix: State in the Issue which of the four gaps (raw events, discoverability, retention rule, cross-client shape) each candidate closes.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

I would start from usability, diagnosis and reproducibility of a concrete handoff, then choose the smallest persistence change that helps. The bot review's retention and scope concerns are useful, but the four candidates address different problems.

Three outcomes are worth separating:

Find and retain the evidence. A returning reviewer needs a usable entry point to the right run and its surviving bytes. An index cannot recover a deleted bundle. We already retain observations in result.json, and the comparison contract binds trace/result evidence and requires preserving the complete output directory. The question is what makes that existing evidence reliably available at handoff.
Understand a failure and reverify a bounded claim. Preserve the exact input/output or artifact, relevant failure evidence, frozen controller/grader identity and verification limits. Distinguish integrity checking/report rendering, regrading a retained answer, and executing a retained artifact. The existing parser replay already checks artifacts without model calls and presents criterion-level guidance, while preserving producer failures and avoiding causal or acceptance claims.
Reproduce the exact stochastic agent trajectory. This is a much stronger, separate goal. A hash identifies bytes; it does not recreate missing content or guarantee the same future model/tool behavior. For example, the Anthropic trace records request identity and response metadata rather than a complete request/response transcript. A common event envelope alone would not establish exact replay.

Suggested first discussion example: take one existing failed or inconclusive SkillEval pair and hand another authorized reviewer a single entry point plus its retained evidence, without prior chat context. Can they:

locate the exact plan, pair/trial and relevant artifacts;
see which assertion or execution stage failed, what evidence supports that diagnosis, and what remains unknown;
repeat the named supported check with the pinned controller/verifier and required isolation, without rerunning the model or rewriting the original result?

That example could show whether clearer retention and handoff documentation is enough, or identify a specific missing field/link. I would defer a universal event schema until a concrete cross-client question needs it, and a ledger/index until this exercise demonstrates a gap that simpler links cannot address. Both ledger and registry would need an explicit scope decision against the accepted disposable-checkpoint boundary.

Any retained handoff should state necessary fields, authorized readers, storage/retention and expiry behavior. Apply data minimization and redaction before persistence; keep originals and labelled redacted copies distinct, and mark unavailable evidence rather than inventing completeness.

Which existing SkillEval failure would be the most useful first handoff example, and what specific claim should its next reviewer be able to reverify? This is a proposal for discussion, not scope acceptance or an implementation commitment.

Intent: decide whether any trace record should outlive the per-run comparison directory and, if so, which candidate to adopt; the new comment reframes this as a usability/diagnosis-first handoff exercise, defers the ledger, event-schema and index candidates, and asks which existing SkillEval failure should be the first handoff example. Verdict: human decision required — the comment explicitly grants no scope acceptance or implementation, so no code change is requested and the Issue still has no named decision owner. Status of earlier findings: unnamed-decision-owner open (F1); ledger/registry-vs-accepted-boundary answered — the comment now extends the disposable-checkpoint caveat to both ledger and index; missing redaction/access rule answered — fields, authorized readers, retention/expiry and redaction are now stated as prerequisites for any handoff; event-inventory gap open but deferred with candidate 2; "trace data already partly retained" premise answered and adopted. No text in the Issue or comments attempts to steer automated review. Labels: area:usability, area:process (the new direction is a reviewer-handoff usability question whose acceptance gate is a retention/privacy rule; kind:rfc, area:workbench and priority:P2 are already human-set and left untouched).

F1 — still open, blocking — the decision owner is still a role, and the exercise now proposed has no owner either

Where: Issue status block (**Decision needed:** ... from the goal/evidence owner) and Ownership table (Goal-acceptance owner: TBD); docs/agent-collaboration.md:193-194, :213-216
Problem: The repository rule keeps an Issue in status:proposed only while "the status block names a concrete question and its decision owner", and allows status:accepted/status:deferred only after that owner records the decision in a comment (docs/agent-collaboration.md:193-194). The latest comment adds a research exercise (pick a failed/inconclusive pair, hand it to an independent reviewer) but names no person to run it, no timebox, and no owner who may still record the decision. So the Issue is now waiting on an unowned exercise and an unnamed owner.
Conditions: Any attempt to close the decision or authorize the follow-up kind:task the Issue promises.
Impact: The Issue can sit in status:proposed indefinitely; a comment from the author cannot be distinguished from the accepting decision.
Fix: Put a GitHub handle in the Decision needed: line and the goal-acceptance-owner field, plus who runs the handoff exercise and by when.

F2 — new, blocking for the proposed exercise — no reachable retained example exists, so the exercise silently presupposes the durable store it is meant to test

Where: docs/skill-eval-pr.md:227-241 (retention table and expiry rule); evidence/skill-eval-v1-20261021/results/v1/.../record.json (legacy pilot layout); tools/skill_eval_compare.py:439 (load_plan reads plan/freeze.json)
Problem: The comment proposes starting from "one existing failed or inconclusive SkillEval pair" handed over with its retained evidence. Today the only SkillEval evidence committed to the repository is the v1 pilot tree, whose trials are record.json/answer.json/task.json with no plan/freeze.json, result.json or receipt.json. I copied that tree and ran python -m tools.skill_eval_compare report --output /tmp/handoff-v1; it aborts with FileNotFoundError: .../plan/freeze.json from load_plan, so the "named supported check" cannot be repeated on it. Current-controller comparisons are local, uncommitted directories (docs/skill-eval-comparison.md:233-234), and PR-published results are Actions artifacts that expire after 30 days and become EXPIRED/unverified afterwards (docs/skill-eval-pr.md:233-241).
Conditions: Whenever the chosen example is more than 30 days old, was run locally on an operator's disk, or is the committed pilot evidence.
Impact: The exercise as written cannot be performed by an independent reviewer without someone first publishing or naming a directory, which is exactly the persistence decision still open. Choosing the example therefore also chooses the storage answer, so it has to be stated rather than assumed.
Fix: In the Issue, name the concrete pair (task id, condition, repeat) and its evidence location (commit path, Actions run/artifact, or a redacted evidence/<campaign-id>/ copy), its retention/expiry, and the authorized reader. Prefer a current-controller case on the default audit/diagnostic/engineering exam, since only those can be re-graded in tree — custom graders are reported unverified (docs/skill-eval-pr.md:318, tools/skill_eval_compare.py:907-914). The committed v1 pilot (holdout-3 INCONCLUSIVE, holdout-4/holdout-5 FAIL) can motivate the example but cannot serve as its artifact.

F3 — new, missing field — the handoff does not record which controller checkout produced the comparison, so the reviewer cannot repeat the pinned check

Where: tools/skill_eval_compare.py:398, :409-419 (freeze stores per-file controller hashes only), :440-442 (load_plan raises Controller changed after freeze on any byte difference); docs/goal-checkpoints.md:29-31 ("Historical comparisons need their original controller checkout")
Problem: The comment asks the next reviewer to "repeat the named supported check with the pinned controller/verifier". A comparison directory records SHA-256 for each controller file, but no repository revision or checkout pointer, and any controller whose bytes differ is rejected outright. Nothing in result.json, receipt.json or plan/freeze.json tells the reviewer which checkout to check out, and the goal file pins only plan_sha256 (docs/goal-checkpoints.md:38), i.e. the freeze bytes, not the producing revision.
Conditions: Handing a comparison directory to a reviewer who does not already have the producing checkout, which is the stated scenario ("without prior chat context").
Impact: The most valuable part of the proposed exercise — independently repeating the check — fails at its first step, and the reviewer can only recover the checkout by brute-force scanning history for a commit matching the recorded hashes.
Fix: Make "producing controller revision plus a pointer to the pinned verifier" a required handoff field (for example, record the controller git rev-parse HEAD alongside controllers in plan/freeze.json, or in the chosen candidate's handoff manifest), and state it as an acceptance condition when a candidate is chosen.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Intent: decide whether any trace evidence must outlive a per-run comparison directory, and if so which candidate — the edit reframes this as a usability/diagnosis handoff exercise and now asks for a concrete retained pair plus an acceptance owner; it grants no implementation. Verdict: human decision required — no code change is requested, and the edit folded most earlier points into the body, but the decision owner is still unnamed and the concrete pair the exercise needs is still absent from the Issue. Earlier findings: F1 still open; F2 answered in part — retained unexpired pairs do exist and re-report identically at their producing controller revision; F3 answered — the Next action now requires producing controller/verifier identity; round-1 items on the disposable-checkpoint boundary, redaction/access rules and the hash-bound premise are reflected in the body; the candidate-2 event-inventory gap stays open but deferred. All axes are already human-set (kind:rfc, area:workbench/process/usability, priority:P2, status:proposed), so no label is suggested.

F1 — still open, blocking — the decision owner is unnamed, and owner selection now sits inside the open question

Where: Issue status block (**Decision needed:** … and who will own acceptance. The goal/evidence acceptance owner remains unnamed); Ownership table (Goal-acceptance owner: TBD, Who accepts completion: TBD); docs/agent-collaboration.md:193-194, :217, :178
Problem: the repository keeps status:proposed while "the status block names a concrete question and its decision owner" and allows status:accepted/status:deferred only after that owner records the decision (lines 193-194); the prescribed second line is **Decision needed:** <question>, from <decision owner> / none (line 217). The edited block names no owner and instead makes "who will own acceptance" part of the undecided question, so nobody is chartered to answer it. Line 178 does allow "unknown fields … TBD while proposed", so the TBD ownership table alone was never the problem — the missing named decider plus the self-referential owner question is.
Conditions: any attempt to record the decision, or to authorize the follow-up kind:task the Issue promises as its expected artifact.
Impact: the Issue stays status:proposed indefinitely, and no comment can be distinguished from the accepting decision, so the exercise it now proposes has no one to accept its outcome.
Fix: name a handle — **Decision needed:** … , from <handle> — or write from none and decide owner selection as a separate step; fill Goal-acceptance owner at the same time.

F2 — answered in part; the concrete pair exists today and is unexpired, but the Issue never names it

Where: Issue "Current state" and "Decision needed"; .github/workflows/skill-eval-pr.yml:74, :87-92, :124-129; docs/skill-eval-pr.md:227-239
Problem: my previous review concluded no reachable retained example exists. That is wrong, and the correction matters: executed comparisons are published as skill-eval-plan / skill-eval-results / skill-eval-report Actions artifacts, and several are unexpired now. Two candidates:
- PR 160 (inference-benchmark-design, update): missing_pair: 1 — expected 2 trials, 1 recorded, stop_reason: client_failed. Run 40496441263; skill-eval-results 12730713778 and skill-eval-report 12730870998 expire 2026-11-26.
- PR 133 (optimization-search-policy, update): case-05 baseline FAIL / candidate FAIL, unchanged_incorrect: 1. Run 40814444430; skill-eval-results 12872245696 and skill-eval-report 12871464282 expire 2026-11-28.
  I reassembled plan/ + validator/ (from skill-eval-plan) with results/ (from skill-eval-results) in a scratch directory and ran python -m tools.skill_eval_compare report --output <dir>: at the producing checkout (e21b739… for PR 160, a0a76b3… for PR 133) it regenerates the same counts with no model call, while at the reviewed default-branch checkout it aborts with ValueError: Controller changed after freeze: tools/skill_eval_compare.py (tools/skill_eval_compare.py:438-442). Also, the "Current state" sentence says the cited proposal "starts with a concrete failed/inconclusive SkillEval handoff", but that comment names no pair — it asks which failure to use.
Conditions: a reviewer trying to run the proposed handoff with the Issue exactly as edited.
Impact: the Issue's central question ("which concrete retained pair and bounded claim") has a verifiable answer that is not in the Issue, so a pair that already demonstrates the real gap — 30-day artifact expiry plus an unrecorded producing revision — lapses in about three weeks while the discussion weighs ledger/index/schema options.
Fix: record one pair in the status block with run and artifact IDs, expiry, authorized readers and redaction scope, and phrase the bounded claim narrowly (e.g. "the update's effect is unestablished because the baseline arm never completed"). Choosing the pair is the owner's call; no new persistence is needed to start.

F3 — answered; "producing controller/verifier identity" is now required, but it is still recorded nowhere in the artifacts

Where: Issue "Next action"; tools/skill_eval_compare.py:398, :440-442; docs/skill-eval-pr.md:227-229
Problem: the edit adopts my earlier requirement, so it is closed at Issue level. The residual fact is that the identity is only derivable from the freeze: plan/freeze.json stores a controllers SHA-256 per file with no repository revision, and load_plan rejects any checkout whose bytes differ. I recovered both producing checkouts by hashing candidate commits (e21b739… for PR 160's freeze, a0a76b3… for PR 133's) — a reviewer can do it, but only by scanning history, and a wrong guess fails as "controller changed" rather than as a missing field.
Conditions: handing a comparison directory or its artifacts to a reviewer who does not already hold the producing checkout, which is the stated scenario ("without prior chat context").
Impact: the most valuable step of the exercise — independently repeating the check — silently depends on undisclosed knowledge; if the handoff later records a stale revision, the check dies with a misleading error.
Fix: make the producing commit a required handoff field (the workflow's ${{ github.workflow_sha }} checkout revision, or a revision beside controllers), and state "reports identically at the producing revision" as an acceptance condition for whichever candidate is chosen.

F4 — new — candidate 3's "the current rule is only …" misstates the existing retention rules and would scope the contract to the wrong tree

Where: Issue candidate 3; docs/skill-eval-comparison.md:234; docs/skill-eval-pr.md:227-239; docs/privacy-security.md:61-64; evidence/README.md; .github/workflows/skill-eval-pr.yml:124-129
Problem: candidate 3 says the current rule is only "preserve the complete output directory to keep its evidence links usable". Three other rules already exist: the PR lane defines per-artifact retention for skill-eval-plan, skill-eval-results and skill-eval-report (30 days) plus EXPIRED/unverified semantics once an artifact lapses (docs/skill-eval-pr.md:227-239); the privacy rules already require sensitive originals to stay in access-controlled storage outside the repository with a labelled redacted copy (docs/privacy-security.md:61-64); evidence/README.md defines layout only. Today's SkillEval evidence lives in that PR/artifact lane — there is no results/ or report.json under evidence/ at all, and the only committed SkillEval tree (evidence/skill-eval-v1-20261021) cannot even be re-reported because it has no plan/freeze.json (FileNotFoundError from tools/skill_eval_compare.py:439).
Conditions: an implementer acting on candidate 3 as written.
Impact: the resulting contract would govern evidence/<campaign-id>/, where none of the SkillEval evidence sits, and would leave the 30-day artifact expiry unreconciled with the "preserve the complete output directory" rule — the exact gap the exercise is meant to expose.
Fix: restate candidate 3 as consolidating the retention/redaction rules that already exist across the comparison contract, the PR artifact table and the privacy rules, and say which copy must survive artifact expiry; add the workflow/artifact lane to the "relevant dependencies" list.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

Intent: decide whether any trace evidence should outlive a per-run comparison directory and, if so, which of the four candidates — the edit adds two concrete retained examples (PR #160, PR #133) with verified expiry dates but grants no implementation. Verdict: human decision required — the Issue still has no named accepting owner, no chosen example, reader/redaction scope, bounded claim or deadline, and explicitly asks not to infer implementation authorization. Earlier findings: F1 open; F2 partially fixed — I confirmed all six artifacts are unexpired and that PR #160's plan+results regenerate missing_pair: 1 at 4817f92, but the pair is still unselected and unrecorded; F3 open — I confirmed two different commits satisfy every frozen controller hash, so plan/freeze.json cannot identify the producing revision; F4 open — candidate 3 still misstates the existing retention rules; the round-1 points on the disposable-checkpoint boundary and redaction/access rules are answered in the discussion and the candidate-2 event inventory stays deferred by the author. No text in the Issue or comments attempts to steer automated review. Labels: help:wanted (the Issue states the acceptance owner and implementers remain unassigned; every type/area/priority/status axis is already human-set).

F1 — open, blocking — the status block still names no decision owner, and the edit folds owner selection into the open question

Where: Issue status block (**Decision needed:** … and who will own acceptance. The goal/evidence acceptance owner remains unnamed), Ownership table (Goal-acceptance owner: TBD, Who accepts completion: TBD); docs/agent-collaboration.md:193-194, :217
Problem: status:proposed applies only while "the status block names a concrete question and its decision owner" (docs/agent-collaboration.md:193), status:accepted/status:deferred may replace it "only after the decision owner records that decision in a comment" (:194), and the prescribed format is **Decision needed:** <question>, from <decision owner> / none (:217). The edited block names no owner and instead makes "who will own acceptance" part of the question awaiting a decision.
Conditions: any attempt to record the decision, close the Issue, or open the follow-up kind:task the Issue names as its expected artifact.
Impact: nobody is chartered to answer the question, so the Issue can sit in status:proposed indefinitely and the handoff exercise it proposes has no one to accept its outcome.
Fix: write **Decision needed:** …, from <handle> (or from none, with owner selection as a separate, explicitly chartered step) and fill the goal-acceptance/closure fields at the same time.

F2 — partially fixed — examples are now named and unexpired, but the pair, readers, redaction scope and bounded claim are still missing, and nothing captures the evidence before it lapses

Where: Issue "Current state" paragraph; docs/skill-eval-pr.md:229-231, :237-239
Problem: the edit now names PR skill: clarify fixed-workload report aggregation #160 (run 40496441263) and PR knowledge: add bottleneck-driven mechanism decision cases #133 (run 40814444430) and I verified via the API that all six artifacts are unexpired: 12731025393 / 12730713778 / 12730870998 expire 2026-11-26, and 12871358110 / 12872245696 / 12871464282 expire 2026-11-28. But the Issue still does not say which example is chosen, does not give the artifact IDs, authorized readers, redaction scope, or the bounded claim, and names no step to copy the evidence before it lapses. docs/skill-eval-pr.md:237-239 states that a lapsed artifact becomes EXPIRED/unverified with no background expiry monitor, and priority:P2 means "decide or start after this week" while the Next action sets no deadline. I also verified the claim that is actually available for PR skill: clarify fixed-workload report aggregation #160: after reassembling its plan and results, PYTHONPATH=src python -m tools.skill_eval_compare report --output <dir> in a worktree at 4817f92 regenerates missing_pair: 1 (1 of 2 trials recorded, baseline INCONCLUSIVE with evaluation_errors: 1, stop_reason: client_failed).
Conditions: the decision is still open on 2026-11-26.
Impact: the only concrete evidence the exercise can use disappears while the discussion weighs ledger/index/schema options, and a later reader cannot tell which pair the discussion ever concerned.
Fix: record one pair with its artifact IDs, expiry, authorized readers, redaction scope and a narrow bounded claim (for example "PR skill: clarify fixed-workload report aggregation #160's effect is unestablished because the baseline arm never completed"), plus either a redacted copy with its own SHA-256 or a stated decision deadline before the expiry date.

F5 — new — the named entry points are not a usable comparison directory, and the reassembly step is documented nowhere

Where: Issue "Current state" ("their plan/results/report artifacts remain unexpired"); docs/skill-eval-comparison.md:233-234
Problem: the comparison contract treats one output directory as the unit of evidence, but the runs the Issue points at publish two disjoint halves: skill-eval-plan contains plan/, validator/, pr-plan.json and approval.json, while skill-eval-results contains only results.zip, whose entries are inference-benchmark-design/results/.... No document describes merging them — a search over docs/, tools/ and .github/ finds only the artifact table (docs/skill-eval-pr.md:230) and the tool code. I had to merge the two artifacts manually (moving the unpacked results/ into place) before report would run at all.
Conditions: a reviewer handed only the run link, which is the stated scenario ("without prior chat context").
Impact: the proposed exercise fails at its first step for anyone following the Issue as written, which is exactly the kind of undocumented handoff knowledge the Issue is trying to remove.
Fix: state the reassembly recipe (which artifact supplies plan/+validator/, which supplies results/, and the resulting directory layout) as part of the example, or hand over a prepared directory/zip plus its SHA-256.

F3 — open residual — the required producing controller identity is recorded nowhere, and the freeze hashes do not identify it

Where: tools/skill_eval_compare.py:398 (controllers stores only per-file SHA-256), :438-442 (load_plan rejects any checkout whose bytes differ); .github/workflows/skill-eval-pr.yml:74, :106, :144; Issue "Next action"
Problem: the edit now requires "controller/verifier identity" as an output of the discussion, but nothing in plan/freeze.json, result.json or receipt.json records the producing checkout. Worse, the recorded hashes are ambiguous: all seven frozen controller hashes for PR skill: clarify fixed-workload report aggregation #160 match both 4817f92 (the run's own head_sha) and e21b739, so a reviewer cannot recover the producing revision by hash matching. The workflow checks out ${{ github.workflow_sha }} and records it nowhere. Running against the reviewed default-branch checkout fails with ValueError: Controller changed after freeze: tools/skill_eval_compare.py (tools/skill_eval_compare.py:442), which reads as a changed controller rather than a missing field. Neither candidate 2 nor the "Relevant dependencies" list covers adding this field.
Conditions: handing a comparison directory or its artifacts to a reviewer without the producing checkout.
Impact: the most valuable step of the exercise — independently repeating the check with the pinned controller — depends on undisclosed knowledge, and the Issue would close without any candidate owning the fix.
Fix: make the producing revision (the workflow's github.workflow_sha, or a revision beside controllers) a required handoff field, and name the artifact/workflow change as a deliverable of whichever candidate is accepted.

F4 — open — candidate 3 still misstates the existing retention rules and would scope the contract to the wrong tree

Where: Issue candidate 3; docs/skill-eval-comparison.md:233-234; docs/skill-eval-pr.md:229-231, :237-239; docs/privacy-security.md:61-64; evidence/README.md
Problem: candidate 3 says "the current rule is only 'preserve the complete output directory'", but three other rules already exist: the PR lane defines 30-day per-artifact retention for skill-eval-plan/skill-eval-results/skill-eval-report with EXPIRED/unverified semantics (docs/skill-eval-pr.md:229-239), and the privacy rules already require sensitive originals to stay in access-controlled storage with a labelled redacted copy (docs/privacy-security.md:61-64). Today's SkillEval evidence lives entirely in that artifact lane: the only SkillEval tree in evidence/ is the legacy pilot, and no result.json or receipt.json exists under evidence/ at all (the sole evidence/**/plan/freeze.json is evidence/drama3-b300-scale150/plan/freeze.json). The dependency list still omits the workflow and PR lane though the proposed contract would have to reconcile them.
Conditions: an implementer acting on candidate 3 as written.
Impact: the resulting contract would govern evidence/<campaign-id>/, where none of this evidence sits, and would leave the 30-day artifact expiry unreconciled with the "preserve the complete output directory" rule — the gap the exercise is meant to expose.
Fix: restate candidate 3 as consolidating the rules that already exist across the comparison contract, the PR artifact table and the privacy rules, say which copy must survive artifact expiry, and add .github/workflows/skill-eval-pr.yml and docs/skill-eval-pr.md to the dependency list.

Reviewed commit: 663f3b3
Coverage: Checkout: 2645 files at 663f3b3 in a git worktree with the repository's history

A smaller starting slice, and why candidate 2 comes first

Static verification against 9dded7ad; no tests, models or hardware were run. This is discussion input on where to start, not a claim of implementation authorization.

The framing I would add: the thing that currently requires a human is not reasoning about a failure, it is cross-run memory — recognising that this run looks like one from two weeks ago. That is candidate 4. Everything else below is about what can be decided without settling candidate 4.

1. Candidate 2 is a prerequisite, not a parallel option

The issue already states the blocker: the Codex client emits turn.completed / item.completed / error while the Anthropic agent emits agent.request / agent.response / agent.tool / agent.completed / agent.failed. Any tool that diagnoses across conditions has to special-case each client.

That makes candidate 2 load-bearing for candidates 1 and 4 as well: a ledger or an index over two incompatible event shapes inherits the special-casing instead of removing it. Candidate 2 can also be accepted without creating a durable store, which keeps it clear of the design decision this issue declined.

2. There is a smaller slice none of the four candidates covers

src/inferrsi/infra/inspection.py already composes target identity, dependency metadata and platform facts in 66 lines. Its only production call site is the doctor path at cli.py:283; the runner does not call it per trial. The architecture review states the same division (doctor owns environment inspection).

The consequence is specific: in an AB/BA comparison there is currently no recorded basis for separating "the candidate change caused the regression" from "the environment drifted between the two trial groups". That is one of the most common attribution questions, the collector for it already exists, and it is simply not invoked at the point where the answer would be captured.

Proposed first slice: a per-trial environment snapshot written outside the candidate workspace, diffed by the reporting layer. It needs no new schema, no ledger and no index, so it does not pre-empt any of the four candidates. It would also produce the concrete retained pair this issue is asking for, since the first comparison that shows an environment delta is itself the demonstrated gap.

3. Four constraints any collector has to satisfy

These apply to the per-trial snapshot and to anything larger. Listed by consequence, not by difficulty:

Constraint	Consequence if violated
Feedback visibility	The evaluation protocol declares feedback scope, and SkillEval isolates development from holdout. A collector that forwards whatever it gathered can leak holdout content and invalidate the comparison without raising anything. A collector must be a declared allowlist projection, not a general tap.
Receipt invalidation (INV-2)	`_Workspace.write` increments `revision` and clears both receipts on every write, including a byte-identical one. Anything a collector writes inside the candidate workspace invalidates receipts. Collector output must be strictly outside it.
Deadline propagation (INV-3)	A collector that does not inherit the run deadline outlives the run, and its records land after finalization — which then becomes an INV-4 late-result question.
Budget	A collector that calls a provider to summarize charges a reservation with no refund. "An agent watching an agent" doubles per-iteration cost; the deterministic form costs nothing.

There is also a redaction constraint that has to be settled before per-trial collection, not after: the source audit records that inspect_environment() unconditionally places platform.node() in host.hostname, which conflicts with docs/privacy-security.md when artifacts are published unredacted. Collecting per trial multiplies that by the trial count.

4. Reuse the projection rule that already exists

Three modules already produce agent-readable diagnosis, and they share one rule worth stating explicitly in whatever is accepted here:

Module	Lines	Its own stated boundary
`harness/skill_eval_diagnostic.py`	118	"Only observations cross the task boundary. The oracle stays in the validator, not in the task or selected knowledge bundle."
`tools/skill_eval_feedback.py`	195	"Bounded public projections, never raw answers or hidden expected answers."
`tools/skill_eval_artifact_guidance.py`	239	"Deterministic advice over trusted artifact checks; never a judge or acceptance gate."

The third is already the shape a diagnostic aid should take: a fixed mapping from failure code to next action (stream_error, missing_terminal, http_failure, length_mismatch), with no provider call and a closed vocabulary. All three are currently reachable only from the SkillEval path, and all three cover verdict failure codes. None covers performance attribution, which is where human assistance is actually being spent.

5. On candidate 4

Naming it plainly: the reason diagnosis needs a human today is that nobody else holds the mapping from a goal to the comparison directories that belong to it. Candidate 4 is that mapping, and it is the one the current design declined.

Two options are consistent, and the choice belongs to the goal-acceptance owner once named:

Accept a bounded form of candidate 4 (goal id → known comparison directories, read-only, no scheduler, no authority) and stop paying for it in human attention.
Decline it again, and record that reconstruction requires being handed the directory path — so that it is a documented constraint rather than an unowned gap.

What should not happen is leaving it undecided while adding candidates 1, 2 or 3, because none of those removes the need for the human.

Suggested sequence

Accept candidate 2 (normalized trace envelope) — unblocks cross-condition tooling, creates no durable store.
Open a separate small task for the per-trial environment snapshot, including its redaction rule.
Decide candidate 4 explicitly, either way.
Revisit candidates 1 and 3 only after 1–3 produce a retained example that shows what is still missing.

Incremental design review — the 5 Nov sequencing proposal identifies a useful attribution gap, but two implementation premises need correction before scoping new collection. This is static review of main@9dded7ad, not an implementation/experiment authorization.

Existing evidence is broader than doctor. cli.py calls inspect_environment() before dispatching doctor, plan and the actual run. harness/runner.py already writes a run-level environment.json, and execute_checked() writes each trial's before/after repository identity to source.json. What is not established is per-trial non-source environment drift (Python packages, devices, visibility), rather than a complete absence of environment/provenance records. Please use these existing records as the baseline for the proposed gap/acceptance example to avoid another overlapping evidence path.
The hostname assertion is contradicted by current code and a regression test. infra/inspection.py emits host.platform, not host.hostname, and does not call platform.node(); test_hostname_is_not_captured_in_environment_or_report explicitly asserts both. Redaction remains necessary for other raw probe/path fields and for any new collector, but the claimed current hostname leak is not a supported reason to change this implementation.
Candidate 2 is not a strict dependency of candidate 4. A bounded goal_id → comparison directory + freeze/receipt references lookup can remain opaque to Codex/Anthropic event payloads; normalized trace events become necessary when diagnosing inside those records across clients, not merely discovering them. I suggest treating discovery and cross-client diagnosis as separate acceptance decisions, rather than making trace normalization an automatic prerequisite to a metadata-only lookup.

Smallest next decision for the named owner (still pending): use one retained comparison to show a concrete drift/diagnosis question that environment.json and per-trial source.json cannot answer; only then specify whether a new per-trial snapshot is justified, with bounded probe overhead, deadline, redaction and visibility. No change to status:proposed, owners or gate is implied.

Navigation Menu

Repository navigation

discussion: what trace evidence should InferRSI persist beyond per-run artifacts? #333

Goal and acceptance

Scope and constraints

Candidates to discuss

Ownership and authorization

Inputs and deliverables

Completion and closure

A smaller starting slice, and why candidate 2 comes first

1. Candidate 2 is a prerequisite, not a parallel option

2. There is a smaller slice none of the four candidates covers

3. Four constraints any collector has to satisfy

4. Reuse the projection rule that already exists

5. On candidate 4

Suggested sequence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

discussion: what trace evidence should InferRSI persist beyond per-run artifacts? #333

Description

Goal and acceptance

Scope and constraints

Candidates to discuss

Ownership and authorization

Inputs and deliverables

Completion and closure

Activity

agent-ci-infra-bot commented on Nov 3, 2026

Dong1017 commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

agent-ci-infra-bot commented on Nov 3, 2026

Dong1017 commented on Nov 5, 2026

A smaller starting slice, and why candidate 2 comes first

1. Candidate 2 is a prerequisite, not a parallel option

2. There is a smaller slice none of the four candidates covers

3. Four constraints any collector has to satisfy

4. Reuse the projection rule that already exists

5. On candidate 4

Suggested sequence

Dong1017 commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions