Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites

[agent] Filed by the scheduled architecture audit routine (CLI and core). Register: discussion #560 register.

Kind: tracking. Source: Part 8.1; recommendations 8.5 G and H (register C32). Measured on main @ 05ecc6e.

Problem

CLI tests pin whole English sentences rather than behavior:

About 478 single-line .contains("…") assertions in crates/socket-patch-cli/tests match a phrase of four or more words, across 114 files (the review counted 328 by a stricter rule). Multi-line assertions add to that.
245 of them are in the 27 covgap_*/coverage_fix_* files: 26,086 lines and 391 tests.

The heaviest files:

File	Sentence assertions
`covgap_commands_scan_mod.rs`	46
`covgap_commands_rollback.rs`	33
`apply/covgap_commands_apply.rs`	30
`covgap_commands_vendor.rs`	28
`remove/covgap_commands_remove.rs`	27
`covgap_commands_scan_hosted.rs`	21
`covgap_commands_get.rs`	21

Many covgap tests come in _json/_human twins that drive the same code path twice. An example is remove_corrupt_vendor_ledger_fails_closed_json and …_human (covgap_commands_remove.rs#L332-L405).
The repo has no snapshot tooling.

Impact:

A copy edit touches dozens of test files. The output-polish PR Consolidated terminal UI and CLI-wide output polish #248 touched 65 files, and the shared remedy/UI-text work in Break command-module cycles and share remedy and UI text #1043 meets the same cost.
Sentence assertions also hide whether the behavior (exit, errorCode, events) is covered at all.

Target design

Behavior is asserted through --json: status, error.code/errorCode, events[].action/code and summary counts, through shared helpers in tests/common (assert_error_code(&out, "…"), event_codes(&out)).
Human output is asserted only in one render suite per command, keyed by message family. These use a few insta snapshots if a maintainer accepts the dev-dependency, or exact-match fixtures otherwise.
Covgap twins collapse: where a _json test covers a path, its _human twin is deleted or reduced to the one render check that the render suite doesn't already have.
Covgap files are renamed into their command's suite once their tests assert behavior, so that "covgap" stops being a test category.

Children (one file family per PR, each deleting what it replaces)

Child 1: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers #1090, remove/covgap_commands_remove.rs, plus the shared tests/common assertion helpers.
Child 2: covgap_commands_vendor.rs and repair/covgap_commands_repair*.rs.
Child 3: apply/covgap_commands_apply.rs and the coverage_fix_*_silent_mute_exit.rs pair.
Child 4: covgap_commands_rollback.rs, after Decide: one shape for the --json top-level error (scan and get emit both a string and a {code, message} object) #704/Make every --json top-level error a {code, message} object (#704) #1027 gives rollback --json a {code, message} error.
Child 5: covgap_commands_scan_mod.rs, covgap_commands_scan_hosted.rs and covgap_commands_get.rs, after Decide: one shape for the --json top-level error (scan and get emit both a string and a {code, message} object) #704 (both scan and get still emit a bare-string error).
Child 6: the render suites (optional insta) and the remaining non-covgap files (cli/output_modes_e2e.rs, cli/interactive_prompts_e2e.rs, …).

Acceptance criteria

Every child keeps the covered production lines covered: the coverage job's line count for the touched command module doesn't drop.
The count of 4+-word .contains("…") assertions in crates/socket-patch-cli/tests falls by every child's share, ending below 60, all of them in render suites.
No covgap_*/coverage_fix_* file names remain.

Dependencies

Children 4 and 5 are blocked by Decide: one shape for the --json top-level error (scan and get emit both a string and a {code, message} object) #704 (Make every --json top-level error a {code, message} object (#704) #1027).
This is independent of C31 (test binary count, Tracking: build one RunCtx in main and stop mirroring CLI flags into the process environment #793), but the folded files reduce the binary count too.

Consolidated work — backlog review, 2026-10-08

The following standalone issues are now tracked here. Their closure consolidates scheduling; it does not mean their implementation is complete. Original reports and discussion remain linked below.

#1090: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers

Track the remaining scope from Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers #1090 in this issue. Explicit remove-test child of the stable-error-code test tracker; keep it as a checklist step.

Preserved scope and acceptance criteria from #1090

Proposed change

Add tests/common/envelope.rs with:
- parse_envelope(stdout);
- assert_error_code(&Value, &str);
- event_codes(&Value) -> Vec<(action, code)>;
- event_purls(&Value, action).
Delete this file's private copies.
Rewrite each sentence assertion as the equivalent --json assertion: the error code, event action/code, summary count, or the file bytes on disk.
Delete each _human twin whose _json sibling covers the path. Where the human rendering is the point (the prompt text at L1854-L1862), keep one assertion per message family.
Rename the file into the remove suite (for example remove/remove_failure_paths.rs) when done.

Size and scope

About −400/+250 test lines and no production change.
Out of scope: other covgap files (later children of Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089) and any message wording change.

Acceptance criteria

At most 3 sentence (4+-word) .contains assertions remain in the file, each in a render-focused test.
cargo test -p socket-patch-cli --test remove (or the renamed target) is green, as are the whole CLI tests.
Coverage of commands/remove.rs is not lower than on main (coverage job).
tests/common/envelope.rs is used by this file and documented for the next children.

Remove suite asserts codes, not sentences

[agent] Child 1 merged in #1535. remove/covgap_commands_remove.rs is now remove/remove_failure_paths.rs and reads its envelopes through common::envelope, which gains event_purls and has_event. Its --json tests now check codes, events and disk state instead of sentences: 4+-word .contains checks in the file went from 24 to 13, and 11 of the 13 are in human-mode render tests. The file's ≤3 target waits on child 6's render suite. Releasing agent:claimed so children 2 and 3 can be picked up.

Generated by Claude Code

[agent] Claiming the repair slice of child 2 of this issue (repair/covgap_commands_repair.rs, repair/covgap_commands_repair_vendor.rs, repair/coverage_fix_repair_vendor_predelete.rs; shared root cause: CLI tests pin English sentences instead of the --json codes). covgap_commands_vendor.rs stays for a later slice. Branch: agent/fix-ci-repair-envelope-asserts. Claim-ID: 2026-10-14T14:51:08Z-1f3f92

Generated by Claude Code

[agent] Repair slice draft PR: #1539

Generated by Claude Code

Repair suite asserts codes, not sentences

[agent] Repair slice of child 2 merged in #1539. The repair suite's covgap files are now repair/repair_render.rs (human render checks), repair/repair_vendor_failure_paths.rs and repair/repair_vendor_predelete.rs. They read envelopes through common::envelope and assert codes, events and disk state. The private-reader ratchet also catches parse_env/events_of. Still open in child 2: covgap_commands_vendor.rs. Releasing agent:claimed.

Generated by Claude Code

[agent] Claiming the vendor slice of child 2 of this issue (covgap_commands_vendor.rs moves onto common::envelope and asserts codes, events and disk state instead of sentences; shared root cause: CLI tests pin English sentences instead of the --json codes). Branch: agent/fix-ci-vendor-envelope-asserts. Claim-ID: 2026-10-14T18:50:35Z-e06281

Generated by Claude Code

[agent] Vendor slice draft PR: #1544

Generated by Claude Code

Vendor suite asserts codes, not sentences

Fix vendor suite review findings

[agent] Claiming the apply slice of child 3 of this issue (apply/covgap_commands_apply.rs and coverage_fix_apply_silent_mute_exit.rs move onto common::envelope and assert codes, events and disk state instead of sentences; shared root cause: CLI tests pin English sentences instead of the --json codes). coverage_fix_vendor_silent_mute_exit.rs waits for #1544 so the two don't conflict. Branch: agent/fix-ci-apply-envelope-asserts. Claim-ID: 2026-10-14T19:50:59Z-6c1588

Generated by Claude Code

[agent] Apply slice draft PR: #1546

Generated by Claude Code

[agent] Vendor slice of child 2 merged in #1544, which completes child 2. covgap_commands_vendor.rs is now part of the vendor suite binary as vendor/vendor_failure_paths.rs (--json codes, events and disk state, with 0 sentence asserts of 4+ words), vendor/vendor_render.rs (human render checks, 25 sentence asserts) and vendor/npm_fixture.rs. The read-only-directory tests now skip under root. coverage_fix_vendor_silent_mute_exit.rs is left for child 3 with its apply twin. Releasing agent:claimed.

Generated by Claude Code

Apply suite asserts codes, not sentences

Apply suite: review fixes for the split

[agent] Apply slice of child 3 merged in #1546. apply/covgap_commands_apply.rs is now apply/apply_failure_paths.rs (--json codes, events and disk state through common::envelope, no 4+-word sentence asserts) and apply/apply_render.rs (18 human render checks), with shared fixtures in apply/apply_fixture.rs. coverage_fix_apply_silent_mute_exit.rs is now apply/apply_silent_staging_errors.rs. Still open in child 3: coverage_fix_vendor_silent_mute_exit.rs, which waits for #1544. agent:claimed stays while #1544 is open.

Generated by Claude Code

[agent] Claiming the last open piece of child 3 of this issue (coverage_fix_vendor_silent_mute_exit.rs joins the vendor suite binary next to its apply twin, reads envelopes through common::envelope and the shared npm_fixture runner; shared root cause: CLI tests pin English sentences instead of the --json codes). #1544 has merged, so it no longer conflicts. Branch: agent/fix-ci-vendor-silent-staging-errors. Claim-ID: 2026-10-14T21:51:15Z-3c9e1a

Generated by Claude Code

Vendor silent staging tests join vendor suite

Pin vendor's real --silent error line

[agent] Vendor silent slice PR: #1550 (ready). It moves coverage_fix_vendor_silent_mute_exit.rs into the vendor suite as vendor/vendor_silent_errors.rs, where it asserts the failed/apply_failed event and the per-patch Error: line instead of free-form words. Once it merges, child 3 is complete.

Generated by Claude Code

[agent] Claiming child 4 of this issue: covgap_commands_rollback.rs and coverage_fix_rollback_ecosystem_scoped_hosted.rs move into the rollback suite and assert --json codes, events and disk state through common::envelope instead of sentences. #704/#1027 have closed, so rollback's --json error is now a {code, message} object and this child is unblocked. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-rollback-envelope-asserts. Claim-ID: 2026-10-14T22:50:45Z-36e383

Generated by Claude Code

[agent] Rollback slice draft PR (child 4): #1552

Generated by Claude Code

CI: rollback tests assert codes, not sentences

CI: address review of rollback test split

[agent] Claiming the get slice of child 5 of this issue. covgap_commands_get.rs moves into the get suite binary and asserts --json codes, events and disk state through common::envelope instead of sentences. get --json already emits a {code, message} error, so this slice isn't blocked. covgap_commands_scan_mod.rs and covgap_commands_scan_hosted.rs stay open for later slices. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-get-envelope-asserts. Claim-ID: 2026-10-14T23:51:19Z-05be0a

Generated by Claude Code

[agent] get slice draft PR (child 5): #1553

Generated by Claude Code

[agent] #1550 merged, so child 3 is complete. coverage_fix_vendor_silent_mute_exit.rs now lives in the vendor suite binary as vendor/vendor_silent_errors.rs. Its --json test reads the envelope through common::envelope (failed/apply_failed for the patch purl), and its --silent tests require an Error: line naming the patch rather than free-form words. dead_endpoint() is now shared in vendor/npm_fixture.rs. Children 4–6 remain.

Generated by Claude Code

CI: get tests assert codes, not sentences

[agent] Claiming the scan_hosted slice of child 5 of this issue. covgap_commands_scan_hosted.rs moves into the scan suite binary and asserts --json codes, events and disk state through common::envelope instead of sentences. covgap_commands_scan_mod.rs stays open for a later slice. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-scan-hosted-envelope-asserts. Claim-ID: 2026-10-15T00:50:50Z-304ce8

Generated by Claude Code

[agent] scan_hosted slice draft PR (child 5): #1556

Generated by Claude Code

[agent] Rollback slice (child 4) merged in #1552. covgap_commands_rollback.rs is now part of the rollback suite binary:

rollback/rollback_failure_paths.rs checks --json codes, events, warning codes and disk state, with 0 sentence asserts of 4+ words.
rollback/rollback_render.rs holds 27 human render checks.
rollback/rollback_fixture.rs holds the shared fixtures.

coverage_fix_rollback_ecosystem_scoped_hosted.rs also joined the same binary as rollback/rollback_ecosystem_scoped_hosted.rs.

Still open: child 5 (covgap_commands_scan_mod.rs, covgap_commands_scan_hosted.rs, covgap_commands_get.rs, now unblocked because #704 is closed) and child 6. agent:claimed stays while #1550 (vendor silent slice) is open.

Generated by Claude Code

Assert codes in hosted scan tests, fold binary

Tighten hosted scan test checks after review

[agent] #1556 (scan_hosted slice of child 5) is ready for review. covgap_commands_scan_hosted.rs now lives in the scan suite binary as scan/hosted_failure_paths.rs (26 --json tests, 0 sentence asserts, down from 20) and scan/hosted_render.rs (14 human-output tests, one check per message family), with shared helpers in scan/hosted_fixture.rs. With #1553 (get), only covgap_commands_scan_mod.rs remains in child 5.

Generated by Claude Code

[agent] get slice of child 5 merged in #1553. covgap_commands_get.rs is now part of the get suite binary as get/get_failure_paths.rs (--json codes, events and disk state through common::envelope, with 0 sentence asserts of 4+ words), get/get_render.rs (human render checks, 18 sentence asserts) and get/get_fixture.rs. coverage_fix_get_double_json.rs is renamed get/get_single_json_document.rs. Still open in child 5: covgap_commands_scan_mod.rs and covgap_commands_scan_hosted.rs.

Generated by Claude Code

[agent] Claiming the scan_mod slice of child 5 of this issue: covgap_commands_scan_mod.rs moves into the scan suite binary and asserts --json codes, events and disk state through common::envelope instead of sentences, with human output kept in one render file. It is the last covgap file in child 5. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-scan-mod-envelope-asserts. Claim-ID: 2026-10-15T02:51:05Z-0bbbf3

Generated by Claude Code

[agent] scan_mod slice draft PR (child 5): #1560

Generated by Claude Code

[agent] #1556 merged, so the scan_hosted slice of child 5 is done. covgap_commands_scan_hosted.rs now lives in the scan suite binary as scan/hosted_failure_paths.rs (--json codes, no sentence asserts), scan/hosted_render.rs (one human-render check per message family) and scan/hosted_fixture.rs. With #1553 (get) merged too, only covgap_commands_scan_mod.rs remains in child 5.

Generated by Claude Code

CI: scan mod tests assert codes, not sentences

[agent] #1560 (scan_mod slice of child 5) is ready for review. covgap_commands_scan_mod.rs now lives in the scan suite binary as scan/scan_failure_paths.rs (24 --json tests, 0 sentence asserts, down from 37), scan/scan_render.rs (35 human-output tests, one check per message family), scan/scan_tty.rs (4 PTY tests) and scan/scan_fixture.rs. Once it and #1556 merge, child 5 is complete. The covgap files left after that are covgap_commands_vex.rs and covgap_utils_socket_cli_config.rs.

Generated by Claude Code

[agent] Claiming the vex slice of child 6 of this issue: covgap_commands_vex.rs and vex_terminal_output.rs move into one vex suite binary, with the --json contracts asserting codes, events and disk state through common::envelope, and human output kept in one render file. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-vex-envelope-asserts. Claim-ID: 2026-10-15T04:51:16Z-13a68c

Generated by Claude Code

[agent] vex slice draft PR (child 6): #1563

Generated by Claude Code

CI: fold vex tests into one suite on codes

[agent] #1563 (vex slice of child 6) is ready for review. covgap_commands_vex.rs and vex_terminal_output.rs now live in one vex suite binary: vex/vex_failure_paths.rs (9 --json contract tests, no sentence asserts), vex/vex_go_patches.rs (the go-patches guards, now read from --json skipped events), vex/vex_render.rs (15 human-output tests, one per message family) and vex/vex_fixture.rs. The top-level covgap files left after it and #1560 are covgap_utils_socket_cli_config.rs and coverage_fix_scan_hosted_dryrun_vendored.rs.

Generated by Claude Code

[agent] Claiming the cli slice of child 6 of this issue: cli/covgap_api_client.rs, cli/covgap_output.rs and cli/covgap_commands_list.rs get command-named files in the cli suite binary, the --json tests assert warnings[]/event codes through common::envelope instead of sentences, and the human/PTY render checks keep one assertion per message family. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-cli-envelope-asserts. Claim-ID: 2026-10-15T05:51:07Z-e0dc58

Generated by Claude Code

[agent] cli slice draft PR (child 6): #1565

Generated by Claude Code

[agent] scan_mod slice of child 5 merged in #1560, which completes child 5 (with #1553 get and #1556 scan_hosted). covgap_commands_scan_mod.rs now lives in the scan suite binary:

scan/scan_failure_paths.rs: --json codes, events, warning codes and disk state, with 0 sentence asserts of 4+ words.
scan/scan_render.rs: the human render checks.
scan/scan_tty.rs: the PTY tests.
scan/scan_fixture.rs: shared fixtures.

The covgap files left are covgap_commands_vex.rs and covgap_utils_socket_cli_config.rs.

Generated by Claude Code

CI: cli suite tests assert codes, not sentences

CI: keep warn-once and remedy checks in --json

[agent] #1565 (cli slice of child 6) is ready for review. The three cli/covgap_* files are now cli/api_client_org_fallback.rs, cli/prompts_tty.rs and cli/list_render.rs. The --json tests assert the api_auth_fallback code through common::envelope (new shared output_envelope, which shows stderr on a parse failure), and the human/PTY tests keep one check per message family: 10 → 4 sentence asserts. Head 4c2517a is green on lean and full-scope CI, and Bugbot is clean. Remaining after #1560/#1563: the update/ and scan/ covgap files, covgap_utils_socket_cli_config.rs and coverage_fix_scan_hosted_dryrun_vendored.rs.

Generated by Claude Code

[agent] Claiming the last two top-level covgap binaries of this issue: covgap_utils_socket_cli_config.rs folds into cli_config_fallback.rs (its harness was a copy of that suite's private helpers), and coverage_fix_scan_hosted_dryrun_vendored.rs moves into the scan suite binary as scan/hosted_dryrun_vendored.rs. Shared root cause: covgap files kept as standalone test binaries with cloned harnesses instead of living in their command's suite. Branch: agent/fix-ci-socket-cli-config-asserts. Claim-ID: 2026-10-15T06:51:47Z-7c3e1a

Generated by Claude Code

[agent] Draft PR for this slice: #1568

Generated by Claude Code

CI: fold last top-level covgap test binaries

Reuse scan suite helpers in hosted dry run tests

[agent] #1568 is ready for review. It removes the last two top-level covgap binaries:

covgap_utils_socket_cli_config.rs folds into cli_config_fallback.rs.
coverage_fix_scan_hosted_dryrun_vendored.rs becomes scan/hosted_dryrun_vendored.rs and uses the scan suite's shared helpers.

Head 777911b is green (86 checks) and Bugbot is clean. Still to do: the update/ and scan/ covgap files, plus the core crate's covgap_* files.

Generated by Claude Code

[agent] Claiming the update/ slice of this issue: update/covgap_commands_update.rs, update/covgap_update_download.rs and update/covgap_update_swap.rs get command-named homes. The --json tests assert codes, events and disk state through common::envelope, and the human/PTY output keeps one check per message family in update/update_render.rs. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-update-envelope-asserts. Claim-ID: 2026-10-15T15:51:11Z-31a633

Generated by Claude Code

[agent] update slice draft PR (child 6): #1580

Generated by Claude Code

Update tests assert codes, not sentences

Tighten update JSON checks from review

[agent] #1580 (update slice of child 6) is ready for review. The three update/covgap_* files are now update/update_json_contracts.rs (--json codes, events, details and disk state through common::envelope), update/update_render.rs (human and PTY wording, one check per message family) and a download_failed test in self_update_failures_e2e.rs. Head 9dd5b35 is green (86 checks) and Bugbot is clean. Still to do: the scan/ covgap files (covgap_commands_scan_vendor_flow.rs, covgap_ecosystem_dispatch.rs, covgap_commands_fetch_stage.rs, coverage_fix_scan_discovery_corrupt_ledger.rs) and the core crate's covgap_* files.

Generated by Claude Code

[agent] Claiming the scan/ slice of this issue: scan/covgap_commands_fetch_stage.rs, scan/covgap_commands_scan_vendor_flow.rs, scan/covgap_ecosystem_dispatch.rs and scan/coverage_fix_scan_discovery_corrupt_ledger.rs get command-named homes, with --json behaviour asserted through common::envelope and human wording kept to one check per message family. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-scan-envelope-asserts. Claim-ID: 2026-10-15T16:50:49Z-a4ac23

Generated by Claude Code

[agent] update slice of child 6 merged in #1580. The update suite has no covgap files left: update/update_json_contracts.rs checks --json codes, events and disk state, update/update_render.rs holds the human and PTY wording, and the download_failed test lives in self_update_failures_e2e.rs. Remaining covgap files: scan/covgap_commands_scan_vendor_flow.rs, scan/covgap_ecosystem_dispatch.rs, scan/covgap_commands_fetch_stage.rs, scan/coverage_fix_scan_discovery_corrupt_ledger.rs, and the core crate's covgap_* files.

Generated by Claude Code

[agent] scan slice draft PR (child 6): #1583

Generated by Claude Code

CI: fold review fixes into the apply staging tests

[agent] Claiming the core-crate slice of this issue: crates/socket-patch-core/tests/covgap_{api_blob_fetcher,crawlers_composer_crawler,crawlers_npm_crawler,patch_apply,update_state}.rs get named homes in their existing companion suites (sharing those suites' helpers instead of mirrored copies), which also drops four test binaries. Shared root cause: coverage-gap files kept as a test category with duplicated helpers. Branch: agent/fix-ci-core-covgap-homes. Claim-ID: 2026-10-15T17:51:03Z-c905e3

Generated by Claude Code

[agent] Core-crate slice draft PR (child 6): #1585

Generated by Claude Code

CI: core covgap tests join their suites

CI: share npm/composer test fixtures

[agent] Core-crate slice of child 6 is ready in #1585: the five crates/socket-patch-core/tests/covgap_*.rs files moved into their companion suites, which removes four test binaries. CI on 709b391 is green, including a full-scope Windows/macOS run, and Bugbot found no issues. After #1585 and #1583 merge, no covgap_* or coverage_fix_* file is left in the repo.

Generated by Claude Code

[agent] Re-verified on origin/main @ 14eee88 (CI/test lane).

Done: no covgap_* or coverage_fix_* file is left (CI: scan suite covgap files get real homes (#1089) #1583 and CI: core covgap tests join their suites (#1089) #1585 merged 10-15). Per-command render suites now exist (apply_render.rs, get_render.rs, rollback_render.rs, scan_render.rs, hosted_render.rs, vendor_render.rs, repair_render.rs). The shared envelope helpers live in tests/common/envelope.rs.
Left (child 6): 312 single-line .contains("…") assertions with a 4+-word phrase remain in crates/socket-patch-cli/tests. 108 are in *_render.rs suites, where they belong. Of the other ~204, 35 are in the silent/TTY/prompt/output-mode suites (cli_*_silent.rs, prompts_tty.rs, interactive_prompts_e2e.rs, output_modes_e2e.rs) and 44 are in e2e_*/docker_e2e_*. The rest are spread thinly, 1–11 per file across ~70 files. The largest are remove/remove_failure_paths.rs (11), redirect_npm_allow_remote.rs (8) and cli_gem_variant_mismatch_policy.rs (8). remove has no remove_render.rs yet.
Open question before more slices: the silent/TTY/prompt suites deliberately assert human output, since the behavior they test is the text: loud prints it, --silent doesn't, and a prompt appears. Should those count as render suites for the "<60, all in render suites" target, or should their sentences move into per-command render suites? The answer changes the remaining target from ~200 assertions to ~165.

No open PR holds the earlier agent:claimed, so I'm releasing it. The next slice that fits is remove: add remove_render.rs and move the 21 sentences from remove_failure_paths.rs, remove_duality_invariants.rs and cli_remove_silent.rs into it.

Generated by Claude Code

[agent] Claiming the remove slice of child 6 of this issue: add remove/remove_render.rs and move the human-wording checks out of remove_failure_paths.rs and remove_duality_invariants.rs into it, so the behaviour tests assert --json codes, events and disk state through common::envelope. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-remove-render-suite. Claim-ID: 2026-10-20T16:51:40Z-5e2a91

Generated by Claude Code

[agent] remove slice draft PR (child 6): #1641

Generated by Claude Code

Move remove's human-output checks to a render file

[agent] Claiming the apply gem-variant slice of child 6: apply/cli_gem_variant_mismatch_policy.rs asserts 8 English sentences on human stderr. The behaviour tests move to --json codes and events; the wording checks move to apply_render.rs. Separate files from #1641 (the remove slice). Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-gem-variant-json-codes. Claim-ID: 2026-10-20T18:50:46Z-4f0b6d

Generated by Claude Code

[agent] Draft PR for the apply gem-variant slice (child 6): #1642

Generated by Claude Code

Check gem variant apply tests by JSON code

Share gem variant fixtures, tighten asserts

[agent] #1641 merged the remove slice of child 6. The human-mode and PTY wording checks now sit in remove/remove_render.rs, with shared fixtures in remove/remove_fixture.rs. remove_failure_paths.rs checks --json codes, events and disk state; the only full sentence it still asserts is the "manifest was not modified" promise.

Remaining remove follow-ups:

remove_cleanup_failures_warn_not_fatal still asserts human-mode stderr. It should get a --json twin that pins the three cleanup_failed warnings.
cli_remove_silent.rs keeps its loud/silent sentence checks until someone answers the open question above: do silent/TTY suites count as render suites?

No agent claim is held on this issue.

Generated by Claude Code

[agent] Claiming the remove cleanup-warning follow-up of child 6: remove_cleanup_failures_warn_not_fatal checks human stderr only. It moves to --json (three cleanup_failed warnings, exit 0, manifest emptied) and its stderr wording check moves to remove/remove_render.rs. Shared root cause: CLI tests pin human output instead of the --json codes. Branch: agent/fix-ci-remove-cleanup-json. Claim-ID: 2026-10-22T17:51:19Z-4ffe39

Generated by Claude Code

[agent] Draft PR for the remove cleanup-warning follow-up: #1669

Generated by Claude Code

Pin remove's cleanup warnings via --json

Check removed count, not store wording, in JSON

[agent] The apply gem-variant slice of child 6 merged in #1642. cli_gem_variant_mismatch_policy.rs now checks --json codes and has no sentence assertions left (it had 8). The wording checks are in apply_render.rs. Follow-up for the product lane: in --json, a no-matching-variant miss exits 1 with partialFailure and summary.failed: 0, and no event names the base purl. Details are in the #1642 description.

Generated by Claude Code

[agent] #1669 merged the remove cleanup-warning follow-up. remove_cleanup_failures_warn_not_fatal now asserts the --json envelope: three cleanup_failed warnings, summary.removed, the removed event and the manifest. The human Warning: <store> cleanup failed: lines are checked in remove/remove_render.rs.

Remaining remove follow-up: cli_remove_silent.rs keeps its loud/silent sentence checks until the open question above is answered (do silent/TTY suites count as render suites?). No agent claim is held on this issue.

Generated by Claude Code

[agent] Claiming the rollback_invariants.rs slice of child 6: its --json tests already pin manifest_not_found, patch_not_found, missing_blob and rollback_failed, but also assert the English messages on top (7 sentence checks), and its human-mode offline missing-blob test belongs in rollback/rollback_render.rs. Separate files from every open PR. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-rollback-invariants-codes. Claim-ID: 2026-10-22T21:51:02Z-394e96

Generated by Claude Code

[agent] Draft PR for the rollback_invariants.rs slice (child 6): #1674

Generated by Claude Code

Assert rollback codes, not sentences

Render rollback error lines and remedies

[agent] #1674 merged the rollback_invariants.rs slice of child 6. That file now asserts --json codes (manifest_not_found, patch_not_found, missing_blob, rollback_failed) instead of English messages. Its 4+-word sentence checks went from 9 to 2. rollback/rollback_render.rs gained render checks for the no-manifest and unknown-identifier Error: lines and for both missing-blob remedies (offline and online). The online remedy had no test before.

Largest files left: vendor/redirect_npm_allow_remote.rs (9), cli/output_modes_e2e.rs (9), cli/interactive_prompts_e2e.rs (9) and cli_remove_silent.rs (7). No agent claim is held on this issue.

Generated by Claude Code

[agent] CI lane, 2026-10-23 22:5x UTC: looked at the four files the last comment lists as largest, and not claiming any of them.

cli_remove_silent.rs, cli/output_modes_e2e.rs and cli/interactive_prompts_e2e.rs are contract tests for human output itself (--silent muting, output modes, prompts). Their sentence checks are mostly the "loud" baseline that proves a silent or JSON run actually dropped something. Under the target design these are the render checks that are meant to stay, so moving them to --json codes would remove what they test.
vendor/redirect_npm_allow_remote.rs could move to its event codes (redirect_npm_allow_remote, npm_allow_remote_left), but 9 checks in one hosted-npm suite isn't worth a PR plus a CI cycle on its own.

I'd suggest a maintainer decide whether to close this tracker as done for practical purposes, with the human-output suites left as the render layer.

Generated by Claude Code

[agent] Re-verified (CI lane, 2026-10-25, origin/main @ 3db3cdf).

Criterion 3 is met: no covgap_* or coverage_fix_* files remain under crates/socket-patch-cli/tests (git ls-tree -r --name-only origin/main crates/socket-patch-cli/tests | grep -E 'covgap|coverage_fix' is empty). Children 1, 2, 3 and 5 landed as per-command *_render.rs suites (rollback_render.rs, vendor_render.rs, apply_render.rs, scan_render.rs, hosted_render.rs, get_render.rs, remove_render.rs, repair_render.rs).
Criterion 2 is not met yet: using the issue's rule (single-line .contains("…") with 4+ words), there are 435 sentence assertions, down from 478. 161 of them are in render suites and 274 are outside them. The largest non-render files are vendor/redirect_npm_allow_remote.rs (11), cli/output_modes_e2e.rs (9), cli/interactive_prompts_e2e.rs (9), update/self_update_channels_e2e.rs (8) and cli_remove_silent.rs (8).
Not claiming a slice this run. I checked the biggest remaining file, redirect_npm_allow_remote.rs. It already asserts the EALLOWREMOTE warning code. Its sentence checks tell apart the .npmrc created, appended and already-set outcomes, and all three share one code and differ only in detail. Moving those checks to codes would need distinct warning codes or a structured field in the product first, which is outside a test-only slice. The cli/* files are render tests by design (output modes, prompts). What's left is mostly child 6.

Generated by Claude Code

Navigation Menu

Repository navigation

Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089

Problem

Target design

Children (one file family per PR, each deleting what it replaces)

Acceptance criteria

Dependencies

Consolidated work — backlog review, 2026-10-08

#1090: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers

Proposed change

Size and scope

Acceptance criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089

Description

Problem

Target design

Children (one file family per PR, each deleting what it replaces)

Acceptance criteria

Dependencies

Consolidated work — backlog review, 2026-10-08

#1090: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers

Proposed change

Size and scope

Acceptance criteria

Activity

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 20, 2026

mikolalysenko commented on Oct 20, 2026

mikolalysenko commented on Oct 20, 2026

mikolalysenko commented on Oct 20, 2026

mikolalysenko commented on Oct 20, 2026

mikolalysenko commented on Oct 22, 2026

mikolalysenko commented on Oct 22, 2026

mikolalysenko commented on Oct 22, 2026

mikolalysenko commented on Oct 22, 2026

mikolalysenko commented on Oct 22, 2026

mikolalysenko commented on Oct 22, 2026

mikolalysenko commented on Oct 22, 2026

mikolalysenko commented on Oct 23, 2026

mikolalysenko commented on Oct 23, 2026

mikolalysenko commented on Oct 25, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone