Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites

[agent] Filed by the scheduled architecture audit routine (CLI and core). Register: discussion #560 register.

Kind: tracking. Source: Part 8.1; recommendations 8.5 G and H (register C32). Measured on main @ 05ecc6e.

Problem

CLI tests pin whole English sentences rather than behavior:

About 478 single-line .contains("…") assertions in crates/socket-patch-cli/tests match a phrase of four or more words, across 114 files (the review counted 328 by a stricter rule). Multi-line assertions add to that.
245 of them are in the 27 covgap_*/coverage_fix_* files: 26,086 lines and 391 tests.

The heaviest files:

File	Sentence assertions
`covgap_commands_scan_mod.rs`	46
`covgap_commands_rollback.rs`	33
`apply/covgap_commands_apply.rs`	30
`covgap_commands_vendor.rs`	28
`remove/covgap_commands_remove.rs`	27
`covgap_commands_scan_hosted.rs`	21
`covgap_commands_get.rs`	21

Many covgap tests come in _json/_human twins that drive the same code path twice. An example is remove_corrupt_vendor_ledger_fails_closed_json and …_human (covgap_commands_remove.rs#L332-L405).
The repo has no snapshot tooling.

Impact:

A copy edit touches dozens of test files. The output-polish PR Consolidated terminal UI and CLI-wide output polish #248 touched 65 files, and the shared remedy/UI-text work in Break command-module cycles and share remedy and UI text #1043 meets the same cost.
Sentence assertions also hide whether the behavior (exit, errorCode, events) is covered at all.

Target design

Behavior is asserted through --json: status, error.code/errorCode, events[].action/code and summary counts, through shared helpers in tests/common (assert_error_code(&out, "…"), event_codes(&out)).
Human output is asserted only in one render suite per command, keyed by message family. These use a few insta snapshots if a maintainer accepts the dev-dependency, or exact-match fixtures otherwise.
Covgap twins collapse: where a _json test covers a path, its _human twin is deleted or reduced to the one render check that the render suite doesn't already have.
Covgap files are renamed into their command's suite once their tests assert behavior, so that "covgap" stops being a test category.

Children (one file family per PR, each deleting what it replaces)

Child 1: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers #1090, remove/covgap_commands_remove.rs, plus the shared tests/common assertion helpers.
Child 2: covgap_commands_vendor.rs and repair/covgap_commands_repair*.rs.
Child 3: apply/covgap_commands_apply.rs and the coverage_fix_*_silent_mute_exit.rs pair.
Child 4: covgap_commands_rollback.rs, after Decide: one shape for the --json top-level error (scan and get emit both a string and a {code, message} object) #704/Make every --json top-level error a {code, message} object (#704) #1027 gives rollback --json a {code, message} error.
Child 5: covgap_commands_scan_mod.rs, covgap_commands_scan_hosted.rs and covgap_commands_get.rs, after Decide: one shape for the --json top-level error (scan and get emit both a string and a {code, message} object) #704 (both scan and get still emit a bare-string error).
Child 6: the render suites (optional insta) and the remaining non-covgap files (cli/output_modes_e2e.rs, cli/interactive_prompts_e2e.rs, …).

Acceptance criteria

Every child keeps the covered production lines covered: the coverage job's line count for the touched command module doesn't drop.
The count of 4+-word .contains("…") assertions in crates/socket-patch-cli/tests falls by every child's share, ending below 60, all of them in render suites.
No covgap_*/coverage_fix_* file names remain.

Dependencies

Children 4 and 5 are blocked by Decide: one shape for the --json top-level error (scan and get emit both a string and a {code, message} object) #704 (Make every --json top-level error a {code, message} object (#704) #1027).
This is independent of C31 (test binary count, Tracking: build one RunCtx in main and stop mirroring CLI flags into the process environment #793), but the folded files reduce the binary count too.

Consolidated work — backlog review, 2026-10-08

The following standalone issues are now tracked here. Their closure consolidates scheduling; it does not mean their implementation is complete. Original reports and discussion remain linked below.

#1090: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers

Track the remaining scope from Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers #1090 in this issue. Explicit remove-test child of the stable-error-code test tracker; keep it as a checklist step.

Preserved scope and acceptance criteria from #1090

Proposed change

Add tests/common/envelope.rs with:
- parse_envelope(stdout);
- assert_error_code(&Value, &str);
- event_codes(&Value) -> Vec<(action, code)>;
- event_purls(&Value, action).
Delete this file's private copies.
Rewrite each sentence assertion as the equivalent --json assertion: the error code, event action/code, summary count, or the file bytes on disk.
Delete each _human twin whose _json sibling covers the path. Where the human rendering is the point (the prompt text at L1854-L1862), keep one assertion per message family.
Rename the file into the remove suite (for example remove/remove_failure_paths.rs) when done.

Size and scope

About −400/+250 test lines and no production change.
Out of scope: other covgap files (later children of Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089) and any message wording change.

Acceptance criteria

At most 3 sentence (4+-word) .contains assertions remain in the file, each in a render-focused test.
cargo test -p socket-patch-cli --test remove (or the renamed target) is green, as are the whole CLI tests.
Coverage of commands/remove.rs is not lower than on main (coverage job).
tests/common/envelope.rs is used by this file and documented for the next children.

audit-core: file C34 decision (#1088, with C74), C32 tracking (#1089, #…

[agent] Triaged: priority:p3 (test-suite refactor, cross-cutting CLI). Tracking issue; work proceeds through its children (#1090 first). Children 4 and 5 wait on #704 / #1027.

Generated by Claude Code

[agent] Claiming this issue for the architecture refactor routine (highest leverage: child 1's shared tests/common/envelope.rs helpers, collapsing ~25 private --json envelope helper copies — parse_envelope, events, find_event, event/warning code collectors — in ~22 test files no open PR changes; remove/covgap_commands_remove.rs migrates once #1258 frees it). Branch: arch-refactor/1089-envelope-helpers. Claim-ID: 2026-10-09T12:56:47Z-6bbcee

Generated by Claude Code

[agent] Draft PR: #1272.

Generated by Claude Code

[agent] Janitor: releasing the stale claim Claim-ID: 2026-10-09T12:56:47Z-6bbcee. #1272 merged a slice 2026-10-09 15:17Z; no open PR for the rest of this issue, and the claimer has been silent since 2026-10-09T13:02:30Z (more than 48h). Removing agent:claimed so another fixer can pick up what remains.

Generated by Claude Code

[agent] Claiming child 1 of this issue (the #1090 scope: remove/covgap_commands_remove.rs moves onto tests/common/envelope.rs and asserts codes instead of sentences; shared root cause: CLI tests pin English sentences instead of the --json codes). Branch: agent/fix-ci-remove-envelope-asserts. Claim-ID: 2026-10-14T12:51:41Z-7afacb

Generated by Claude Code

[agent] Child 1 draft PR: #1535

Generated by Claude Code

[agent] #1568 is ready for review. It removes the last two top-level covgap binaries:

covgap_utils_socket_cli_config.rs folds into cli_config_fallback.rs.
coverage_fix_scan_hosted_dryrun_vendored.rs becomes scan/hosted_dryrun_vendored.rs and uses the scan suite's shared helpers.

Head 777911b is green (86 checks) and Bugbot is clean. Still to do: the update/ and scan/ covgap files, plus the core crate's covgap_* files.

Generated by Claude Code

[agent] Claiming the update/ slice of this issue: update/covgap_commands_update.rs, update/covgap_update_download.rs and update/covgap_update_swap.rs get command-named homes. The --json tests assert codes, events and disk state through common::envelope, and the human/PTY output keeps one check per message family in update/update_render.rs. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-update-envelope-asserts. Claim-ID: 2026-10-15T15:51:11Z-31a633

Generated by Claude Code

[agent] update slice draft PR (child 6): #1580

Generated by Claude Code

Update tests assert codes, not sentences

Tighten update JSON checks from review

[agent] #1580 (update slice of child 6) is ready for review. The three update/covgap_* files are now update/update_json_contracts.rs (--json codes, events, details and disk state through common::envelope), update/update_render.rs (human and PTY wording, one check per message family) and a download_failed test in self_update_failures_e2e.rs. Head 9dd5b35 is green (86 checks) and Bugbot is clean. Still to do: the scan/ covgap files (covgap_commands_scan_vendor_flow.rs, covgap_ecosystem_dispatch.rs, covgap_commands_fetch_stage.rs, coverage_fix_scan_discovery_corrupt_ledger.rs) and the core crate's covgap_* files.

Generated by Claude Code

[agent] Claiming the scan/ slice of this issue: scan/covgap_commands_fetch_stage.rs, scan/covgap_commands_scan_vendor_flow.rs, scan/covgap_ecosystem_dispatch.rs and scan/coverage_fix_scan_discovery_corrupt_ledger.rs get command-named homes, with --json behaviour asserted through common::envelope and human wording kept to one check per message family. Shared root cause: CLI tests pin English sentences instead of the --json codes. Branch: agent/fix-ci-scan-envelope-asserts. Claim-ID: 2026-10-15T16:50:49Z-a4ac23

Generated by Claude Code

[agent] update slice of child 6 merged in #1580. The update suite has no covgap files left: update/update_json_contracts.rs checks --json codes, events and disk state, update/update_render.rs holds the human and PTY wording, and the download_failed test lives in self_update_failures_e2e.rs. Remaining covgap files: scan/covgap_commands_scan_vendor_flow.rs, scan/covgap_ecosystem_dispatch.rs, scan/covgap_commands_fetch_stage.rs, scan/coverage_fix_scan_discovery_corrupt_ledger.rs, and the core crate's covgap_* files.

Generated by Claude Code

[agent] scan slice draft PR (child 6): #1583

Generated by Claude Code

CI: fold review fixes into the apply staging tests

[agent] Claiming the core-crate slice of this issue: crates/socket-patch-core/tests/covgap_{api_blob_fetcher,crawlers_composer_crawler,crawlers_npm_crawler,patch_apply,update_state}.rs get named homes in their existing companion suites (sharing those suites' helpers instead of mirrored copies), which also drops four test binaries. Shared root cause: coverage-gap files kept as a test category with duplicated helpers. Branch: agent/fix-ci-core-covgap-homes. Claim-ID: 2026-10-15T17:51:03Z-c905e3

Generated by Claude Code

[agent] Core-crate slice draft PR (child 6): #1585

Generated by Claude Code

CI: core covgap tests join their suites

CI: share npm/composer test fixtures

[agent] Core-crate slice of child 6 is ready in #1585: the five crates/socket-patch-core/tests/covgap_*.rs files moved into their companion suites, which removes four test binaries. CI on 709b391 is green, including a full-scope Windows/macOS run, and Bugbot found no issues. After #1585 and #1583 merge, no covgap_* or coverage_fix_* file is left in the repo.

Generated by Claude Code

Navigation Menu

Repository navigation

Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089

Problem

Target design

Children (one file family per PR, each deleting what it replaces)

Acceptance criteria

Dependencies

Consolidated work — backlog review, 2026-10-08

#1090: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers

Proposed change

Size and scope

Acceptance criteria

60 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking: assert stable codes instead of human sentences in CLI tests, and fold the covgap suites #1089

Description

Problem

Target design

Children (one file family per PR, each deleting what it replaces)

Acceptance criteria

Dependencies

Consolidated work — backlog review, 2026-10-08

#1090: Assert error codes instead of sentences in remove's covgap suite and add shared JSON assertion helpers

Proposed change

Size and scope

Acceptance criteria

Activity

mikolalysenko commented on Oct 7, 2026

mikolalysenko commented on Oct 9, 2026

mikolalysenko commented on Oct 9, 2026

mikolalysenko commented on Oct 12, 2026

mikolalysenko commented on Oct 14, 2026

mikolalysenko commented on Oct 14, 2026

60 remaining items

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

mikolalysenko commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions