Skip to content

fix(smoke): validate remote mktemp -d before using it as an rm -rf target - #33

Merged
andrei-hasna merged 6 commits into
mainfrom
fix/smoke-guard-remote-mktemp
Jul 27, 2026
Merged

fix(smoke): validate remote mktemp -d before using it as an rm -rf target#33
andrei-hasna merged 6 commits into
mainfrom
fix/smoke-guard-remote-mktemp

Conversation

@andrei-hasna

@andrei-hasna andrei-hasna commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Secondary remediation for the 2026-07-24 data-destruction incident. The primary fix is
hasna/hooks#10; this closes the same class of
defect in this repo. Separate PR by design.

The defect

scripts/smoke-machine-sync-release.mjs set remoteDir from an unchecked remote
mktemp -d, then later deleted it over ssh:

const remoteDir = runRemote(options.remote, `mktemp -d …`).trim();   // ~line 414

run('ssh', [options.remote, `rm -rf ${shellQuote(remoteDir)}`]);      // ~line 579

mktemp -d prints its error on stderr and nothing on stdout, so a failure leaves
remoteDir empty — the same failure mode as bun pm cache, which is what turned
rm -rf "$(bun pm cache)"/* into rm -rf /* on station02.

This is inert today only because shellQuote('') renders as '', so the delete degrades
to rm -rf ''. It is one appended /* away from catastrophic, and the value is also
interpolated into --peer-workspace. Relying on a quoting helper to neutralise a value that
should never have been accepted is exactly the assumption this incident punished.

The fix

New scripts/lib/remote-temp-dir.mjs exporting assertRemoteTempDir, applied at the
assignment
and re-asserted immediately before the rm -rf.

It accepts only what the template could actually have produced. mktemp replaces the
trailing run of X in place, so a valid result is the template's fixed prefix followed
by a single non-empty path component. Checking only the template's parent directory is not
enough — that would accept /tmp/somebody-elses-dir and delete it. The bound is derived from
the template rather than hardcoded, so it stays correct if the template moves.

Rejected: empty, whitespace-only, multi-line output, non-absolute paths, /, /tmp, /usr,
/home/hasna, a sibling directory sharing the temp parent, a path escaping via .., and a
malformed template (relative, or without at least three X).

Packaging — a second, real bug this surfaced

smoke-machine-sync-release.mjs is a published file. Adding an import to it without
registering the new module would ship a package that throws ERR_MODULE_NOT_FOUND for every
installed copy. The new module is therefore registered in all three deliberate allowlists
(package.json files, scripts/validate-public-package.mjs, tests/package-release.test.ts)
— that triplication is a review tripwire, so it is honoured rather than worked around.

The existing allowlists only catch the opposite mistake (shipping something unreviewed).
Nothing caught "a published script imports something unpublished", so this PR adds that test.

It immediately found pre-existing breakage. scripts/apply-cloud-migrations.mjs:23
statically imports ../src/storage.ts, but src/ has never been in files. Verified against
a packed tarball:

$ npm pack && tar xzf hasna-knowledge-0.2.91.tgz && cd package
$ bun scripts/apply-cloud-migrations.mjs --help
error: Cannot find module '../src/storage.ts' from '…/package/scripts/apply-cloud-migrations.mjs'

That path carries a single exemption named in the test, tracked as todos task 1eb481d5;
removing the exemption is that task's acceptance criterion. It is not fixed here because
choosing between repointing at dist, shipping src, or unpublishing the script is a
separate decision that deserves its own review — and silently deleting the test to make it
green would be precisely the "guard that passes while protecting nothing" trap.

Evidence

$ bun test tests/machine-smoke-script.test.ts tests/package-release.test.ts
  14 pass, 0 fail

$ node scripts/validate-public-package.mjs
  Public package validation passed for hasna-knowledge-0.2.91.tgz.
  Public scripts included: 7.

$ npm pack && tar xzf … && bun scripts/smoke-machine-sync-release.mjs --help
  resolves and runs from the tarball

$ gitleaks git --staged --no-banner --redact …
  no leaks found

Full suite with HASNA_KNOWLEDGE_API_URL/_KEY unset: 227 pass, 2 skip, 1 fail. That one
failure (tests/cli.test.ts — "context pack and proposal context commands return bounded
agent JSON") reproduces on the untouched baseline via git stash, so it is pre-existing and
unrelated. With those env vars set, 99 tests fail because the cloud-API flip disables the
local sqlite catalog — also environmental, also unrelated.

Negative controls

Guards are proven to fail on bad input, not merely to pass on good:

  • assertRemoteTempDir throws for every rejected value listed above, each an explicit test case.
  • Removing the new module from the allowlist makes the packaging test fail with the exact
    missing path:
    error: scripts/smoke-machine-sync-release.mjs imports ./lib/remote-temp-dir.mjs,
           which is not in the published files list
    

No rm is executed by any test, at any scope. Only the validator is exercised.

Scope

grep confirms this was the only remote mktemp -d and the only ssh … rm -rf in the repo.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

…rget

scripts/smoke-machine-sync-release.mjs set remoteDir from an unchecked remote
`mktemp -d` and later ran `ssh <remote> rm -rf ${shellQuote(remoteDir)}`.
`mktemp -d` prints its error on stderr and nothing on stdout, so a failure left
remoteDir empty. That is inert today only because shellQuote renders '' as ''
-- one appended `/*` at the cleanup site turns it into the 2026-07-24 incident,
where an empty command substitution made `rm -rf "$(cmd)"/*` run as `rm -rf /*`
and destroyed a machine's checkouts. The check belongs at the assignment, not at
the delete, and it must not lean on quoting.

New scripts/lib/remote-temp-dir.mjs exports assertRemoteTempDir, which accepts
only what the template could actually have produced: `mktemp` replaces the
trailing X run in place, so the result must be the template's fixed prefix plus a
single non-empty path component. Checking only the template's parent directory is
not enough -- it would accept /tmp/somebody-elses-dir and delete it. The bound is
derived from the template rather than hardcoded, so it stays correct if the
template moves. Applied at the assignment and re-asserted immediately before the
`rm -rf`.

Packaging: smoke-machine-sync-release.mjs is a published file, so the new module
is registered in all three deliberate allowlists (package.json files,
scripts/validate-public-package.mjs, tests/package-release.test.ts). Without that
the published script would fail with ERR_MODULE_NOT_FOUND for every installed
copy.

Also adds a test asserting that every relative import in a published script is
itself published. The existing allowlists only caught the opposite mistake
(shipping something unreviewed). This one immediately found pre-existing
breakage: scripts/apply-cloud-migrations.mjs:23 imports ../src/storage.ts, but
src/ has never been in `files`, so the published script already throws
ERR_MODULE_NOT_FOUND. Verified against a packed tarball. That path carries a
single exemption named in the test and tracked as todos task 1eb481d5; removing
the exemption is that task's acceptance criterion. It is not fixed here because
choosing between repointing at dist, shipping src, or unpublishing the script is
a separate decision.

Evidence:
  bun test tests/machine-smoke-script.test.ts tests/package-release.test.ts
    -> 14 pass, 0 fail
  node scripts/validate-public-package.mjs
    -> passed; public scripts included: 7
  npm pack + extract + `bun scripts/smoke-machine-sync-release.mjs --help`
    -> resolves and runs from the tarball
  full suite (HASNA_KNOWLEDGE_API_* unset) -> 227 pass, 2 skip, 1 fail; that one
    failure reproduces on the untouched baseline via git stash, so it is
    pre-existing and unrelated.
  gitleaks staged scan -> no leaks found

Negative controls, proving the guards fail on bad input rather than only passing
on good: assertRemoteTempDir rejects '', whitespace-only, multi-line output,
relative paths, /, /tmp, /usr, /home/hasna, a sibling dir sharing the temp
parent, a path escaping via .., and a malformed template. Removing the new module
from the allowlist makes the packaging test fail with the exact missing path.
No `rm` is executed by any test.
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

CI on this PR is red. It is not caused by this PR — both failing tests already fail on main, and one of them is a dated time bomb that went off on 2026-07-26. Evidence below so a reviewer/merger does not have to re-derive it.

Failure 1 — knowledge cli > context pack and proposal context commands return bounded agent JSON

tests/cli.test.ts:2667-2668 seed evidence hardcoded to 2026-06-25T00:00:00.000Z; line 2675 queries it with --since 30d; line 2680 asserts the evidence is found. The fixture simply aged out of the window:

date fixture age inside --since 30d? observed
2026-07-24 29d yes main CI ubuntu-bun passed
2026-07-26 31d no this PR failed
2026-07-27 32d no fails locally

Controlled A/B on untouched main @ 2023a26 in a detached worktree, changing only the fixture date:

A) main as-is, today
   bun test tests/cli.test.ts -t "context pack and proposal context"
   -> 1 fail

B) main + ONLY 2026-06-25 -> 2026-07-24 in the fixture, nothing else
   same command
   -> 0 fail

So the calendar is the sole variable. Tracked as todos d1b93867. The fix should seed the fixture relative to now or freeze time — bumping the date only re-arms the bomb.

Failure 2 — knowledge machine sync ledger > sync export redacts local file and workspace refs from bundles (macOS only)

Also fails on main’s own CI run 30106566377, same commit as this PR’s base. Not caused by this PR; needs its own diagnosis.

This PR’s own tests pass in CI

(pass) remote temp dir guard > rejects the empty output a failing mktemp -d leaves behind
(pass) remote temp dir guard > the failure names the path and says why it matters

This PR touches only scripts/lib/remote-temp-dir.mjs, scripts/smoke-machine-sync-release.mjs, scripts/validate-public-package.mjs, the two test files, and package.json. None of them are imported by the CLI or by tests/cli.test.ts.

Adversarial review found the guard silently degraded into the parent-directory-only
check this module exists to avoid, and that the defect narrative described an
unreachable code path. Both fixed.

F1 (high): when a template's final component is nothing but X, the derived prefix
is a bare directory and every sibling under it validated. `/tmp/XXXXXX` accepted
`/tmp/somebody-elses-dir`, `/XXXXXX` accepted `/etc` and `/home`, and
`/home/hasna/XXXXXX` accepted `/home/hasna/.ssh` -- all reproduced. `/tmp/XXXXXX`
is an ordinary mktemp template. The guard that should have caught this,
`prefix.includes('/') === false`, could never fire, because '/tmp/'.includes('/')
is true; mutation testing confirmed that line was both ineffective and untested.
Templates without a fixed prefix in their final component are now refused.

F2 (medium): the stated defect was not reachable. runRemote goes through
runChecked, which throws on a non-zero remote exit, so a FAILING `mktemp -d`
(exit 1, empty stdout) aborts the run and never assigns "". Verified against GNU
coreutils 9.4. The reachable hole is the remote exiting ZERO with stdout that is
not the path -- an ssh Banner, a chatty login profile, or a ForceCommand wrapper.
Verified: `echo banner; mktemp -d ...` exits 0 and yields a two-line value the
guard rejects. The claim appeared in four places (commit, PR body, module
docstring, test comment); the docstring and test comment are corrected here and
the PR body separately. An incident-remediation artifact that misstates the defect
it fixes corrupts the incident record, which is worse than the bug.

F3 (medium): the bound is derived from a template built from a caller-supplied
version string, and templates containing `..` passed validation with a meaningless
bound. Templates must now be normalized.

F4 (medium): the re-assertion before the `rm -rf` sits in a `finally`, where a
throw REPLACES any in-flight exception from the try block and skips the cleanup --
masking a real smoke failure and guaranteeing a remote leak. It is now contained:
it reports and skips the delete, and can never mask. Its actual purpose (catching a
future refactor that makes the value reassignable) is stated plainly.

F5 (medium): the new "published imports must be published" test only scanned
relative specifiers, so it missed that scripts/apply-cloud-migrations.mjs has a
SECOND unresolvable import -- `@hasna/contracts/auth`, a devDependency only. Task
1eb481d5 would have closed on a false green with the published script still
throwing. The test now checks bare specifiers against runtime dependencies too,
both are exempted by name, and both must be removed for that task to be done.

F6 (low-medium): the import regex missed side-effect imports (`import './x.mjs'`)
and fired inside comments. Comments and template literals are now stripped, and
side-effect imports are matched. Runtime builtins (node:, bun:) are excluded --
found by the new check flagging bun:sqlite.

F7/F8 (low): the GNU suffix form is documented as deliberately refused; the local
temp dir can no longer leak on the guard's throw path, because the remote dir is
now created first.

Evidence: 18 pass, 0 fail on the touched tests (was 14). The three mutations that
SURVIVED the reviewer's mutation testing (parent-only bound, bare prefix accepted,
absolute check dropped) are now all CAUGHT, as is dropping the new normalization
check. Package validator passes, packed tarball still resolves and runs. Full suite
231 pass / 1 fail, that failure being the pre-existing dated time bomb tracked as
d1b93867. No rm executed by any test.
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Round 2 — review findings fixed in 0902c4b

F2 first, because it corrects this PR’s own premise

The defect I originally described was not reachable. runRemoterunChecked throws on a non-zero remote exit, so a failing mktemp -d (exit 1, empty stdout) aborts the run and never assigns "". Verified:

failing mktemp -d : exit=1  stdout=""  -> runChecked THROWS; the pre-PR line never assigned ""

The reachable hole is the remote exiting zero with stdout that is not the path — an ssh Banner, a chatty login profile, or a ForceCommand wrapper. Verified:

echo banner; mktemp -d ... : exit=0  stdout="Welcome to linux-node-a\n/tmp/knowledge-banner-uqj7wT"
guard: REJECTED -> path spans multiple lines

The wrong claim appeared in four places. Corrected in the module docstring and the test comments here; this comment corrects the PR narrative. An incident-remediation artifact that misstates the defect it fixes corrupts the incident record — that is worse than the bug.

F1 — the guard degraded into the parent-only check it exists to avoid

When a template’s final component is nothing but X, the derived prefix is a bare directory and every sibling validated. Reproduced:

!! ACCEPT  template=/tmp/XXXXXX          dir=/tmp/somebody-elses-dir
!! ACCEPT  template=/XXXXXX              dir=/etc
!! ACCEPT  template=/XXXXXX              dir=/home
!! ACCEPT  template=/home/hasna/XXXXXX   dir=/home/hasna/.ssh

/tmp/XXXXXX is an ordinary template. The line that should have caught it — prefix.includes("/") === false — can never fire, since "/tmp/".includes("/") is true; mutation testing showed it was both ineffective and untested. Templates without a fixed prefix in their final component are now refused; all five attack cases reject, the legitimate template still passes.

Remaining findings

  • F3 — templates containing .. passed with a meaningless bound, and the template is built from a caller-supplied version string. Templates must now be normalized.
  • F4 — the re-assertion sits in a finally, where a throw replaces the in-flight exception and skips the cleanup: it would mask a real smoke failure and guarantee a remote leak. Now contained — reports and skips the delete, never masks.
  • F5 — the new import test only scanned relative specifiers, so it missed that apply-cloud-migrations.mjs has a second unresolvable import: @hasna/contracts/auth, a devDependency only. Task 1eb481d5 would have closed on a false green with the published script still throwing. Bare specifiers are now checked against runtime dependencies; both imports are exempted by name and both must go for that task to be done.
  • F6 — regex missed side-effect imports and fired inside comments. Comments and template literals are stripped, side-effect imports matched, node:/bun: builtins excluded — found by the new check correctly flagging bun:sqlite.
  • F7/F8 — the GNU suffix form is documented as deliberately refused; the local temp dir can no longer leak on the throw path, because the remote dir is created first.

Evidence

bun test <touched files>   18 pass, 0 fail   (was 14)
validate-public-package    passed, 7 public scripts
npm pack + extract + --help  resolves and runs
full suite                 231 pass, 2 skip, 1 fail  <- the dated time bomb, task d1b93867
gitleaks --staged          no leaks found

Mutation retest — the three mutations that SURVIVED the reviewer’s run are now all caught:

CAUGHT | M5 parent-only bound        CAUGHT | M6 bare prefix accepted
CAUGHT | M7 absolute check dropped   CAUGHT | M8 template normalization dropped

No rm executed by any test. Requesting re-review of bb9cf2b..0902c4b.

…d-rail test itself

Re-review returned APPROVE-WITH-FIXES and found that round 2 introduced a defect in
the same class it was closing.

N1 (new, mine): the import test's comment stripper was regex-based, so
`/\*[\s\S]*?\*/` treated the `/*` inside an ordinary glob string like '**/*.ts' as a
comment opener and deleted everything up to the next `*/`. One innocuous glob line
before a broken import made the check go green -- precisely the
passes-while-broken failure that test exists to catch. Reproduced, then replaced
with a character scanner that tracks string and template literals, so a `/*` or
`//` inside a string is no longer a comment. Verified by planting
`const g = '**/*.ts'` + a broken import: now fails with the exact missing specifier.

N9: the bare-specifier check false-positived on legitimate forms. Builtins imported
without the `node:` prefix, the package's own name (valid via `exports`), and
`#`-prefixed `imports` subpaths are now allowed. The self-reference case was live,
not hypothetical: scripts/smoke-machines-adapter.mjs already imports
'@hasna/knowledge' and was passing only by accident of the old template-literal
strip.

N3a: the docstring claimed the guard accepts "only what this exact template could
have produced", which was false -- any suffix of any length and charset was
accepted. mktemp replaces the X run with exactly that many [A-Za-z0-9] characters,
so that is now enforced and the claim is true. Rejects one-char, over-long,
`$(id)`, `..` and space-containing suffixes.

N3b: a comment presented a paraphrase as a verbatim transcript -- `echo banner`
cannot print "Welcome to linux-node-a", and mktemp never prints the unexpanded
XXXXXX. Replaced with the actual measured output. This is the same defect F2 was
raised for, in text rewritten to fix F2.

N3c: "interpolated into --peer-workspace" was loose; that site passes a separate
argv element. The real shell interpolations are the remote `cd <value> && ...` and
`rm -rf <value>`. Corrected.

N4/N5/N6: the finally block is now fully contained. Round 2 wrapped only the
assertion -- the statement that cannot currently fail -- while leaving rmSync and
the ssh spawn able to throw, replace the in-flight exception from the try block,
and skip the remaining cleanup. All three are individually contained now, and a
non-zero remote cleanup exit is reported instead of discarded. Both temp dirs are
created inside the try with null guards in finally, so whichever one exists is
cleaned up; round 2's reordering had only moved the leak from local to remote.

N7/N8: added the missing fixtures -- the banner-first shape the docstring calls the
live case, the `X{3,}` POSIX boundary, and suffix shape. Bare `.toThrow()`
assertions now pin a message pattern.

Evidence: mutation retest of all 10 checks -- every one CAUGHT, including M8
(`X{3,}` -> `X{1,}`) which survived the reviewer's run, and M5 which my own charset
check had subsumed until a message-ordering test pinned it. Touched-file tests 22
pass / 0 fail (was 18). Validator passes, packed tarball resolves and runs. Full
suite 235 pass / 2 skip / 1 fail, that failure being the pre-existing dated time
bomb (task d1b93867).

CI note: PR #33 is red with TWO failures, not one. Both reproduce on main's own CI
for the same base commit, and `git diff origin/main...HEAD -- src/ tests/cli.test.ts`
is empty, so neither is caused by this PR. The macOS-only redaction failure is
recorded on d1b93867 alongside the time bomb.
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Round 3 — re-review findings fixed in a15850f

The re-review returned APPROVE-WITH-FIXES and caught that round 2 introduced a defect in the same class it was closing. That one first.

N1 — my own guard-rail test could be silenced by one glob string

The import test stripped comments with /\*[\s\S]*?\*\//g, which treats the /* inside an ordinary glob string like **/*.ts as a comment opener and deletes everything up to the next */. Reproduced:

raw specifiers:             [ "ghost-package", "./ok.mjs" ]
what the test actually saw: [ "./ok.mjs" ]

One innocuous glob line before a broken import and the check goes green — precisely the passes-while-broken failure this test exists to catch, in the test itself. Replaced with a character scanner that tracks string and template literals. Verified by planting const g = **/*.ts plus a broken import: now fails with the exact missing specifier.

N9 — the bare-specifier check false-positived on legitimate imports

Builtins imported without the node: prefix, the package’s own name (valid via exports), and #-prefixed imports subpaths are now allowed. The self-reference case was live, not hypothetical: scripts/smoke-machines-adapter.mjs already imports @hasna/knowledge and was passing only by accident of the old template-literal strip.

N3 — three accuracy defects in text rewritten for accuracy

  • (a) The docstring claimed the guard accepts “only what this exact template could have produced.” False — any suffix of any length or charset was accepted (Z, ZZZZZZZZZZ, $(id), ..). mktemp emits exactly N characters from [A-Za-z0-9]; that is now enforced, so the claim is true.
  • (b) A comment presented a paraphrase as a verbatim transcript. echo banner cannot print Welcome to linux-node-a, and mktemp never prints the unexpanded XXXXXX. Replaced with the actual measured output. This is the same defect F2 was raised for, in text I rewrote to fix F2.
  • (c) “interpolated into --peer-workspace” was loose — that site passes a separate argv element. The real shell interpolations are the remote cd <value> && … and rm -rf <value>.

N4/N5/N6 — the finally block is now actually contained

Round 2 wrapped only the assertion — the one statement that cannot currently fail — while leaving rmSync and the ssh spawn able to throw, replace the in-flight exception, and skip the remaining cleanup. The reviewer modelled it: rmSync throw masks; ssh NUL-arg throw masks. All three are individually contained now, and a non-zero remote cleanup exit is reported instead of silently discarded. Both temp dirs are created inside the try with null guards — round 2’s reordering had only moved the leak from local to remote.

Evidence

Mutation retest — all 10 checks CAUGHT, including M8 (X{3,}X{1,}) which survived the reviewer’s run, and M5 which my own new charset check had silently subsumed until a message-ordering test pinned it.

bun test <touched files>   22 pass, 0 fail   (was 18)
validate-public-package    passed, 7 public scripts
npm pack + extract + --help  resolves and runs
full suite                 235 pass, 2 skip, 1 fail
gitleaks --staged          no leaks found

CI correction (N2) — stated plainly

My earlier “231 pass / 1 fail” was Linux-only. CI is red with two failures. The second is macOS-only: knowledge machine sync ledger > sync export redacts local file and workspace refs from bundles. Both reproduce on main’s own CI run 30106566377 for this PR’s base commit, and git diff origin/main...HEAD -- src/ tests/cli.test.ts is empty — so neither is caused by this PR. Both are recorded on todos d1b93867.

This PR cannot merge on a green CI until those two are fixed independently. Saying that rather than reporting a clean-looking local number.

Also surfaced: test-matrix (windows-latest, bun) is cancelled by fail-fast in every recent run, so no test has executed on Windows for some time — the new tests included.

…k to itself

Round-3 review REJECTED a15850f: my fix for N1 reintroduced N1, and nothing in the
suite pinned either round-3 change.

THE REAL DEFECT WAS STRUCTURAL, not the parsing bug. Three versions of this check
shipped in a row, each replaceable with something that reports nothing while the
whole suite stayed green, because the tests meant to pin them exercised a private
copy of the logic instead of the logic itself. Round 3's mutation run proved it:
stripComments could be replaced with `s => ""`, or reverted to the exact regex N1
condemned, and every test passed. Moving the implementation never fixed the
operating mode, which was silent-green.

Fixed by extracting `unresolvableImportsIn(script, source)` and having BOTH the
enforcement test and the pinning tests call it. Mutation retest, all now CAUGHT and
previously all SURVIVED: isAlwaysResolvable always true; extractor reports nothing;
exemptions widened to everything; relative check disabled; bare-dep check disabled;
`bun:` blessed as a class again.

Parsing: replaced the hand-written scanner with Bun.Transpiler().scan(). Two
hand-rolled strippers had the same defect - a `/*` inside an ordinary string opened
a phantom comment and deleted real imports. The second only moved the trigger: a
regex literal containing a quote (`/['"]/`) shifted it into string state and
everything after it vanished, which the reviewer demonstrated on the real published
smoke script with two innocuous added lines. Extracting imports is a parsing
problem; anything short of a parser is a guess that fails silently.

The parser also removed the reason for three allowances, so all three are deleted
rather than justified:
- `ownName`: existed only to mask a phantom the regex stripper invented from a
  template literal that GENERATES a child script. A real parser does not report it.
  The allowance also waved through `@hasna/knowledge/<subpath>` for subpaths absent
  from `exports`, which fail with ERR_PACKAGE_PATH_NOT_EXPORTED.
- unprefixed builtins: `builtinModules` under `bun test` is BUN's list and contains
  `ws`, `undici` and `bun` - real npm names, NOT Node builtins. Verified: all three
  are absent from Node 22's list and throw ERR_MODULE_NOT_FOUND from the tarball.
- `#`-prefixed: package.json has no `imports` map, so every one is guaranteed to
  fail. It could only ever hide breakage.

`bun:` is no longer blessed as a class either. Verified from the packed tarball:
`node scripts/smoke-open-files-installed-boundary.mjs` throws
ERR_UNSUPPORTED_ESM_URL_SCHEME. Exempted by name and filed as task 104f993d.

F6/F7 - containment was correct in runSyncSmoke but violated one frame up. Its two
callers' finally blocks used a raw rmSync, discarding the exception runSyncSmoke
takes care to preserve, in exactly the --no-machines-sync paths the release smoke
runs. And `report` itself was unguarded: process.stderr.write is synchronous on a
pipe and throws EPIPE when the consumer exits, masking the in-flight error and
skipping the remote cleanup. Both are now shared, individually-contained helpers
(reportCleanupProblem, removeTempDirSafely) used at every finally; zero raw rmSync
remains in any finally.

F8 - the two validator checks that survived round 3 are now pinned: a lone \r
(the multi-line check could be narrowed to \n unnoticed) and the suffix charset,
with 6-character fixtures so only the charset rule can reject them - the earlier
fixtures were all the wrong length, leaving the shell-metacharacter backstop pinned
by nothing.

Evidence: touched-file tests 24 pass / 0 fail (was 22). All 6 import-check
mutations CAUGHT (all 6 previously survived); both validator mutations CAUGHT.
Validator passes; packed tarball resolves and runs UNDER NODE, not just Bun. Full
suite 239 pass / 2 skip / 1 fail, that failure being the pre-existing dated time
bomb (task d1b93867). gitleaks clean.
@andrei-hasna

Copy link
Copy Markdown
Contributor Author

Round 4 — REJECTED at a15850f; fixed in 17f328f

My fix for N1 reintroduced N1. But the finding that matters is structural, not the parsing bug.

F2 — three versions of this check shipped in a row, none of them pinned

Mutation run on a15850f:

SURVIVED  isAlwaysResolvable() always true  (guard fully disabled)
SURVIVED  drop the "#" allowance
SURVIVED  drop the builtinModules allowance
SURVIVED  stripComments -> identity
SURVIVED  stripComments -> ""            (strips the entire file)
SURVIVED  stripComments -> the OLD regex from 0902c4b

You could restore the exact regex N1 condemned and every test stayed green. The tests meant to pin the check exercised a private copy of the logic instead of the logic itself. The operating mode — silent green — never changed; only the implementation moved. That is why the same defect class shipped three rounds running.

Fix: extracted unresolvableImportsIn(script, source), and both the enforcement test and the pinning tests now call it. Mutation retest — all CAUGHT, all previously SURVIVED:

CAUGHT  isAlwaysResolvable always true      CAUGHT  extractor reports nothing
CAUGHT  exemptions widened to everything    CAUGHT  relative check disabled
CAUGHT  bare-dep check disabled             CAUGHT  "bun:" blessed as a class again

F1 — stopped hand-rolling a JS parser

The scanner had no regex-literal state. A quote inside a regex literal enters string mode; the phase shift makes a later ordinary string's /* read as a block-comment opener — and that branch deletes to EOF. Now Bun.Transpiler().scan(), verified against every shape that defeated both strippers. Extracting imports is a parsing problem; anything short of a parser is a guess that fails silently.

F3/F4/F5 — the parser removed the reason for three allowances, so all three are deleted

  • ownName existed only to mask a phantom the regex stripper invented from a template literal that generates a child script. It also waved through @hasna/knowledge/<subpath> for subpaths absent from exports (ERR_PACKAGE_PATH_NOT_EXPORTED).
  • unprefixed builtinsbuiltinModules under bun test is Bun's list and contains ws, undici, bun: real npm names, not Node builtins. Verified absent from Node 22's list; all three ERR_MODULE_NOT_FOUND from the tarball.
  • #-prefixed — no imports map exists, so every one is guaranteed to fail. It could only ever hide breakage.

bun: is no longer blessed as a class either — verified ERR_UNSUPPORTED_ESM_URL_SCHEME under node from the packed tarball. Exempted by name, filed as 104f993d.

F6/F7 — containment was correct, one frame too low

runSyncSmoke's finally was contained, but its two callers used a raw rmSync, discarding the exception it takes care to preserve — in exactly the --no-machines-sync paths the release smoke runs. And report itself was unguarded: process.stderr.write is synchronous on a pipe and throws EPIPE when the consumer exits, masking the in-flight error and skipping the remote cleanup. Both are now shared, individually-contained helpers; zero raw rmSync remains in any finally.

F8 — the two unpinned validator checks

A lone \r, and the suffix charset with 6-character fixtures so only the charset rule can reject them. The earlier fixtures were all the wrong length, leaving the shell-metacharacter backstop pinned by nothing.

Evidence

touched-file tests       24 pass, 0 fail   (was 22)
import-check mutations   6/6 CAUGHT        (6/6 previously SURVIVED)
validator mutations      2/2 CAUGHT
validator                passes
npm pack + extract + --help under NODE     resolves and runs
full suite               239 pass, 2 skip, 1 fail

The one failure remains the pre-existing dated time bomb (d1b93867), untouched by this branch.

Left standing as sound by the reviewer: the N3a suffix tightening (verified against 400 GNU mktemp samples and against glibc/musl/BSD padding sets), the runSyncSmoke control flow across all 8 modelled paths, and the N3b docstring correction.

Requesting re-review of a15850f..17f328f.

andreihasna and others added 2 commits July 27, 2026 01:49
The most important change came out of a review accident, not a finding. Importing
scripts/smoke-machine-sync-release.mjs executed it: `main()` ran at module scope, so
merely loading the PUBLISHED script performed `bun install -g @hasna/knowledge` on
the local machine and attempted `ssh <remote> bun install -g` - before any argument
was inspected. It fired for real during review of this PR. Any tool that imports the
file for linting or dependency analysis - including the packaging check this PR adds
- triggers it.

`main()` now runs only when invoked directly, compared against `process.argv[1]`
rather than `import.meta.main`, which Node did not support until v24 and which would
have silently turned the script into a no-op there. Verified: still works under both
bun and node when invoked; importing is inert in 9ms with no install and no ssh.
Incident recorded with blast radius (local install only; the remote host has no DNS
or ssh-config entry, so nothing on the fleet was touched).

F-1: the exemption list was duplicated - the test meant to pin it iterated a
hardcoded literal array rather than the Set - so ADDING an exemption was checked by
nothing. A genuinely broken import plus a matching entry left the whole file green.
Same silent-green class as round 3, one level down. The loop now derives from the
Set, and every entry must name a published script. Verified: that exact combined
mutation is now CAUGHT.

F-2: deleting the allowances over-corrected and created two false positives, both
proven against an extracted tarball under node rather than argued:
  import { sep } from 'path'          RESOLVES  (unprefixed Node builtin)
  @hasna/knowledge/storage            RESOLVES  (declared in exports)
Both are restored, narrowly: unprefixed builtins are matched against a hardcoded
NODE list (not `builtinModules`, which under `bun test` returns BUN's list including
ws/undici/bun), and self-reference only for subpaths the `exports` map declares.
Still refused, also verified: @hasna/knowledge/<undeclared>
(ERR_PACKAGE_PATH_NOT_EXPORTED), `#`-prefixed (ERR_PACKAGE_IMPORT_NOT_DEFINED),
ws/undici (ERR_MODULE_NOT_FOUND). The previous comment claimed each deletion was
"verified against this repo"; the self-reference one was assumed, and was wrong.

F-3: the relative branch checked a hand-maintained 7-script list instead of what the
package ships, so `../package.json` and `../dist/index.js` - both published, both
verified resolvable - were reported as breakage. Now checked against `files`, with
package.json treated as always shipped since npm includes it regardless.

F-4: documented the extractor's real scope. It reports import/import()/export-from,
not require(), import.meta.resolve() or new URL(). None are used today; saying so
beats letting the test name imply more than it covers.

F-5: the EPIPE rationale was factually wrong. Measured on Linux, writing to a stderr
pipe whose reader has exited raises an asynchronous 'error' event that kills the
process - not a synchronous throw - so the try/catch cannot intercept it. The
comment now says what the guard does and does not cover.

Evidence: touched-file tests 26 pass / 0 fail. Mutation battery all CAUGHT, including
X1 (broken import + matching exemption) which SURVIVED round 5, plus the bogus-extra-
exemption, check-returns-nothing, always-resolvable, bare-dep-disabled and
builtins-widened mutations. Validator passes; full suite 239 pass / 2 skip / 1 fail,
that failure being the pre-existing dated time bomb (task d1b93867).

Corrected evidence claim: the touched-file count is 26, not the 24 stated last round.
…both fixes with tests

Closes the three round-6 review findings, all of the same class: the two fixes
this PR exists to make were not observable by any test, and the guard added for
one of them introduced a third silent failure.

1. The direct-invocation guard compared `resolve(process.argv[1])` with
   `fileURLToPath(import.meta.url)`. Node resolves the main module through
   symlinks before deriving `import.meta.url`; `resolve()` is purely lexical. So
   through any symlinked path - a pnpm-style `node_modules/@hasna/knowledge ->
   <store>/package`, or macOS `/tmp -> /private/tmp`, which is the PR's own
   documented `npm pack && tar xzf` evidence procedure - `main()` never ran, the
   script printed nothing and exited 0. A release gate checking only the exit
   status would have recorded a green smoke that did nothing. Compare realpaths
   instead, falling back to the lexical comparison only when a path cannot be
   resolved at all. Verified against the packed tarball: through a pnpm-style
   symlink under node, `--dry-run --json` now produces output and
   `--totally-bogus` raises "Unknown argument"; before, both exited 0 silently.

2. Nothing pinned that `assertRemoteTempDir` was called before the remote
   `rm -rf`; deleting both calls left the whole suite green. `createRemoteTempDir`
   and `removeRemoteTempDir` now live in the side-effect-free `scripts/lib`
   module and take their ssh work as a parameter, so tests drive the functions
   that decide whether the validator runs, including the negative that matters -
   no delete is attempted when the value does not validate. They are kept out of
   the smoke script so this test file never has to import it.

3. Nothing pinned "does not execute on import"; reverting to a bare `main();`
   left the suite green. A new test imports the script in a child process whose
   PATH puts recorders in front of bash/bun/ssh, with a control proving the
   recorders are reachable, and asserts nothing was installed and nothing was
   sent to the remote. Both argv shapes are covered (`node -e`, and an ordinary
   importer module).

Also corrects the guard's comment, which claimed Node had no `import.meta.main`
until v24; measured, v22.22.3 defines it as a boolean.

Mutation-verified - each fix reverted individually, each caught by exactly one
test: lexical compare restored -> symlink test fails; `main()` unconditional ->
import test fails; create-side assert deleted -> createRemoteTempDir test fails;
delete-side re-check deleted -> "never attempts a delete" test fails; helpers
bypassed in the script -> wiring test fails.

Tests: 249 pass, 2 skip, 1 fail (bun test), up from 239 pass, 2 skip, 1 fail.
The one failure is tests/cli.test.ts:2680, reproduced identically at base main
2023a26 (66 pass / 1 fail) and unrelated to this PR.
@andrei-hasna
andrei-hasna merged commit 354a855 into main Jul 27, 2026
0 of 7 checks passed
@andrei-hasna
andrei-hasna deleted the fix/smoke-guard-remote-mktemp branch July 27, 2026 02:17
andrei-hasna added a commit that referenced this pull request Jul 27, 2026
59d1d65

Reconciles an independent adversarial review of the previous commit. Kept as a
second commit rather than an amend: the previous commit's message contains a wrong
sentence (see 2 below), and rewriting history to hide it would be the same mistake
as editing a merged PR body instead of commenting on it.

1. the corrected claim survived in a third file (review F1, also self-found)

tests/cli.test.ts:722-725 still carried the unreachable comma justification that
this PR corrected in README.md and src/cli.ts. Repo-wide, the claim had four sites,
not three. Comment-only; the assertions were already correct and are unchanged.

2. "verify:generated still exits 0" was wrong (review F2)

The previous commit message and the PR body both said it. Measured:

  bun scripts/verify-generated-artifacts.mjs   -> exit 0
  bun run verify:generated                     -> exit 1

package.json:58 makes verify:generated rebuild first, and the rebuild does not
reproduce the committed bin/knowledge-mcp.js or dist/index.js. Two causes, both
pre-existing on main and neither introduced here: package.json is inlined into the
bundles and gained a `files` entry in #33 that the committed mcp bundle predates,
and zod's codegen has drifted. bin/knowledge.js is unaffected because it is
minified. Confirmed not a regression of this branch: git diff --exit-code 64d05ae
59d1d65 -- bin/knowledge-mcp.js dist/index.js exits 0, and this PR does not touch
package.json or src/mcp.js. Tracked separately; main ships a stale mcp bundle.

3. the replacement for the old overclaim was itself an overclaim (review F3)

"one tag named `p q` reads exactly like a pair" is false: a pair renders `p, q`.
Since no not_found entry can hold a comma, joining on ', ' is injective, so a plain
space is a legibility problem, not a collision — and calling it an "ambiguity"
contradicted the injectivity the same comment had just established.

The review found the case that IS load-bearing, and it is stronger: trim() strips
only the ends, so whitespace inside a name survives. `-t $'p\nq'` yields
not_found ["p\nq"], and joined raw that breaks the single-line message in two:

  unquoted: 'Removed 1 tag from k_w (not found: p\nq)'   -> 2 lines
  quoted:   'Removed 1 tag from k_w (not found: "p\\nq")' -> 1 line

Tab behaves the same. Now cited instead, and pinned by a new assertion rather than
left as prose — the one added test assertion in this PR.

4. dependency-state caveat on the rebuild-and-compare check (review F4)

@hasna/events is imported by src/cli.ts and is NOT in the build's --external list,
so it is bundled; it is declared ^0.1.3 and this repo commits no lockfile. 0.1.14
reproduces the committed bundle, 0.1.13 does not (1040485 vs 1045325 stripped). The
CHANGELOG now carries that caveat, since it recommends the check. Tracked separately.

5. smaller review items

The `list` block's comment said "repeated -t still accumulates there" while sitting
inside `list`, where repeated -t NARROWS (measured: 3 -> 2 -> 1 matches as values
are added). Names the command explicitly now, in both README and source. The untag
example block now says its three lines are each run against the same starting state
rather than as a sequence. The 9-line bullet added to the unrelated "inventory
paths block fix" section is back to a 4-line pointer.

bin/knowledge.js is deliberately NOT rebuilt: every source edit here is a comment,
and rebuilding to a temp outfile reproduces the committed bundle byte-for-byte after
stripping (sha256 0630038c6ab4c200), which is the evidence that no shipped
behaviour moved.
andrei-hasna added a commit that referenced this pull request Jul 27, 2026
…e the check say so

`bun run verify:generated` was RED on main at 64d05ae while
`bun scripts/verify-generated-artifacts.mjs` was GREEN. One check, two names, two
answers — and the passing one had been cited in CHANGELOG.md as evidence that a
rebuilt bundle matched its source, which it never checked.

## The stale artifacts

    bun scripts/verify-generated-artifacts.mjs   -> exit 0
    bun run verify:generated                     -> exit 1

package.json:58 prefixed a `bun run build`, and the rebuild did not reproduce the
committed files. Rebuilding from clean main leaves exactly two artifacts modified —
`bin/knowledge-mcp.js` and `dist/index.js` — and the whole diff is 9 lines:

  * `"scripts/lib/remote-temp-dir.mjs"` added. package.json is inlined into the
    bundles and its `files` array gained that entry in 354a855 (#33), so the
    committed mcp bundle predates #33. Occurrences of `remote-temp-dir` in the
    committed artifacts at 64d05ae: bin/knowledge.js 1, bin/knowledge-mcp.js 0.
  * three zod codegen collapses of empty else blocks,
    `} else if (ctx.target === "openapi-3.0") {} else {}` -> `} else {}` removed.
    Dead-code elimination, semantically identical, absent from bin/knowledge.js
    only because that bundle is --minify'd.

Both are absorbed rather than pinned around. The rebuild is deterministic: two
consecutive builds produce byte-identical output for all six bundles (`cmp` clean),
and the four unaffected bundles reproduce the committed bytes exactly.

`bin/knowledge.js` is also rebuilt here, for one reason worth stating: package.json
is inlined, so changing the `verify:generated` script line below changes that bundle
too. Its only content change is that script string.

CI never caught this because .github/workflows/ci.yml runs "Verify generated
artifacts" AFTER "Run tests", and the pre-existing context-pack test failure aborts
the job first.

## Making the check mean something

The old script never rebuilt. On a clean checkout its exit 0 meant only "the files I
did not touch are untouched". Measured against it earlier: exit 0 with src/cli.ts
diverged from bin/knowledge.js, and exit 0 with bin/knowledge.js corrupted by 30
junk bytes.

  * ONE ENTRY POINT. The build moved inside the script, so it cannot be run without
    the rebuild and read as a sync check. `verify:generated` is now the script alone,
    and CI calls `bun run verify:generated` instead of reassembling the two halves.
  * NO HARDCODED ARTIFACT LIST. The diff gate covers the `bin` and `dist` directories
    instead of two paths, and the stale-pattern scan derives its files from
    `git ls-files`. The old four-file list had already drifted: it omitted
    bin/knowledge-serve.js and dist/serve.js.
  * A DIRTY-BEFORE PRECONDITION. A post-rebuild `git diff` only means something if the
    generated paths matched the index beforehand. It now fails with the offending
    paths named instead of reporting drift it cannot attribute.
  * THE PATTERNS PROVE THEMSELVES BEFORE A CLEAN RESULT IS TRUSTED. Each stale pattern
    carries a fixture it must match and a counter-fixture it must not; if either fails
    the script exits 1 rather than reporting every artifact clean. A regex that can no
    longer match anything reports "clean" forever.

That self-check earned its place immediately: the first fixture written for
`/path:\s*decodeURIComponent\([^)]*\.pathname\)/` used
`decodeURIComponent(new URL(uri).pathname)` and the script rejected it, because
`[^)]*` cannot cross a nested `)`. The fixture is now verbatim the line 2bea200
removed from src/source-ref.ts, and the known bound is documented at the pattern
rather than left as a surprise.

tests/generated-artifacts.test.ts covers the pieces: directory-wide gate, derived
file list containing all six shipped bundles and nothing outside bin/dist, every
pattern firing on its fixture and rejecting its counter-fixture, a synthetic stale
bundle in a temp root producing one problem per pattern while the counter-fixtures
come back clean, and `verify:generated` staying a single command. End-to-end script
behaviour is deliberately not tested there — it would need a real build, which
overwrites bin/ and dist/ while other test files run.
andrei-hasna added a commit that referenced this pull request Jul 27, 2026
…e the check say so

`bun run verify:generated` was RED on main at 64d05ae while
`bun scripts/verify-generated-artifacts.mjs` was GREEN. One check, two names, two
answers — and the passing one had been cited in CHANGELOG.md as evidence that a
rebuilt bundle matched its source, which it never checked.

## The stale artifacts

    bun scripts/verify-generated-artifacts.mjs   -> exit 0
    bun run verify:generated                     -> exit 1

package.json:58 prefixed a `bun run build`, and the rebuild did not reproduce the
committed files. Rebuilding from clean main leaves exactly two artifacts modified —
`bin/knowledge-mcp.js` and `dist/index.js` — and the whole diff is 9 lines:

  * `"scripts/lib/remote-temp-dir.mjs"` added. package.json is inlined into the
    bundles and its `files` array gained that entry in 354a855 (#33), so the
    committed mcp bundle predates #33. Occurrences of `remote-temp-dir` in the
    committed artifacts at 64d05ae: bin/knowledge.js 1, bin/knowledge-mcp.js 0.
  * three zod codegen collapses of empty else blocks,
    `} else if (ctx.target === "openapi-3.0") {} else {}` -> `} else {}` removed.
    Dead-code elimination, semantically identical, absent from bin/knowledge.js
    only because that bundle is --minify'd.

Both are absorbed rather than pinned around. The rebuild is deterministic: two
consecutive builds produce byte-identical output for all six bundles (`cmp` clean),
and the four unaffected bundles reproduce the committed bytes exactly.

`bin/knowledge.js` is also rebuilt here, for one reason worth stating: package.json
is inlined, so changing the `verify:generated` script line below changes that bundle
too. Its only content change is that script string.

CI never caught this because .github/workflows/ci.yml runs "Verify generated
artifacts" AFTER "Run tests", and the pre-existing context-pack test failure aborts
the job first.

## Making the check mean something

The old script never rebuilt. On a clean checkout its exit 0 meant only "the files I
did not touch are untouched". Measured against it earlier: exit 0 with src/cli.ts
diverged from bin/knowledge.js, and exit 0 with bin/knowledge.js corrupted by 30
junk bytes.

  * ONE ENTRY POINT. The build moved inside the script, so it cannot be run without
    the rebuild and read as a sync check. `verify:generated` is now the script alone,
    and CI calls `bun run verify:generated` instead of reassembling the two halves.
  * NO HARDCODED ARTIFACT LIST. The diff gate covers the `bin` and `dist` directories
    instead of two paths, and the stale-pattern scan derives its files from
    `git ls-files`. The old four-file list had already drifted: it omitted
    bin/knowledge-serve.js and dist/serve.js.
  * A DIRTY-BEFORE PRECONDITION. A post-rebuild `git diff` only means something if the
    generated paths matched the index beforehand. It now fails with the offending
    paths named instead of reporting drift it cannot attribute.
  * THE PATTERNS PROVE THEMSELVES BEFORE A CLEAN RESULT IS TRUSTED. Each stale pattern
    carries a fixture it must match and a counter-fixture it must not; if either fails
    the script exits 1 rather than reporting every artifact clean. A regex that can no
    longer match anything reports "clean" forever.

That self-check earned its place immediately: the first fixture written for
`/path:\s*decodeURIComponent\([^)]*\.pathname\)/` used
`decodeURIComponent(new URL(uri).pathname)` and the script rejected it, because
`[^)]*` cannot cross a nested `)`. The fixture is now verbatim the line 2bea200
removed from src/source-ref.ts, and the known bound is documented at the pattern
rather than left as a surprise.

tests/generated-artifacts.test.ts covers the pieces: directory-wide gate, derived
file list containing all six shipped bundles and nothing outside bin/dist, every
pattern firing on its fixture and rejecting its counter-fixture, a synthetic stale
bundle in a temp root producing one problem per pattern while the counter-fixtures
come back clean, and `verify:generated` staying a single command. End-to-end script
behaviour is deliberately not tested there — it would need a real build, which
overwrites bin/ and dist/ while other test files run.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant