Add an ast-grep vendor-contract test #108

What a vendor-contract test is here

PR #93 added packages/cli/test/vale-vendor-contract.test.ts. It invokes the vendored Vale binary directly — deliberately not through runVale — and pins the givens: the exit-code semantics, the JSON shape and field names, the Span base, the config-error channel, matcher precedence. Every case names the code that rests on it, so a Vale upgrade that changes a default fails there with an instruction ("go change normalizeSeverity") instead of surfacing downstream as a mysterious mapping bug.

That file is currently the only vendor-contract test in the stack. There is no ast-grep equivalent, even though ast-grep is the executor for every sg rule and the CLI's logic is built on just as many observed-but-undocumented behaviors.

Why it matters

@ast-grep/cli and all seven platform packages are exact-pinned at 0.41.0 in packages/cli/package.json. An exact pin is the right call, but it means the only moment these assumptions can change is a deliberate bump — and at that moment nothing tells us which ones moved. The pin makes the bump the single checkpoint; a contract test is what makes the checkpoint say something.

The failure mode that hides is worse than for Vale: an empty or mis-scoped ast-grep scan reports success. runAstGrepScan in packages/cli/src/rules/scan.ts treats exit 0 and exit 1 as normal, parses whatever JSON lines arrive, and silently discards any stdout line that does not parse. A scan that matched nothing because discovery changed is indistinguishable from a clean codebase. Vale at least fails loudly on a bad config; here, a broken assumption reads as green.

Behaviors worth pinning

Confirmed against the source in this repo:

1. --json=stream line protocol and match shape. scan.ts parses one JSON object per line and maps it via AstGrepMatch in packages/cli/src/types/check.ts — ruleId, severity, message, note?, text, file, range.start/end.{line,column}, replacement?. A rename arrives as undefined in a CheckResult, not as an error. Worth pinning that the stream is newline-delimited JSON with no interleaved status lines, since scan.ts swallows unparseable lines by design.

2. range line/column are 0-based. packages/cli/src/util/format.ts renders range.start.line + 1 / column + 1, and packages/cli/src/rules/runtime/narrow.ts does the same. If ast-grep ever emitted 1-based positions, every reported location would silently be off by one — no error, just wrong.

3. severity vocabulary. AstGrepMatch.severity is typed as exactly error | warning | info | hint. Nothing verifies ast-grep cannot emit a fifth value.

4. Scan exit codes. scan.ts encodes "exit 1 means error-severity matches were found; exit > 1 means the binary or config failed." That is the entire boundary between "findings" and "engine failure" and it is asserted nowhere against the binary.

5. sg test output format and exit code. runTests in packages/cli/src/rules/verify.ts scrapes counts out of the human-readable line test result: ok. 3 passed; 0 failed; with /(\d+)\s+passed/i and /(\d+)\s+failed/i, and keys success off exit 0. This is prose parsing against an unpinned format — if the wording changes, verify reports 0 passed on a passing run. Also worth pinning: --skip-snapshot-tests and that --filter takes an anchored regex (^id$).

6. --version identifies itself as ast-grep. AST_GREP_BINARY.identity is /ast-grep/i (scan.ts). If the string changes, findSgBinary rejects the real binary as a placeholder and throws "ast-grep binary not found" — the resolver exists precisely because @ast-grep/cli's postinstall can leave a placeholder text file at the binary path.

7. ruleDirs recursion vs. .tests/ — already pinned, worth relocating. ruleDirs recurses and parses every .yml beneath it as a rule, which is why fixtures live in a dot-prefixed .tests/: measured against 0.41.0, tests/ and __tests__/ both hard-fail the whole scan with Fail to parse yaml as RuleConfig: missing field 'language', while a dot-directory is skipped by discovery and sg test still reads it via testConfigs[].testDir. Documented at packages/cli/src/rules/engines.ts (RULE_TESTS_DIRECTORY) and packages/cli/src/rules/assemble.ts (assembleSgConfig), and pinned today by packages/cli/test/engine-layout.test.ts. That test is the existing proof this class of check pays for itself; the contract file is where it belongs alongside the rest.

Noted but not currently depended on: files / ignores scoping inside a rule. packages/cli/src/rules/engines.ts cites it as the reason sg needs no per-rule config file, but no code path exercises it yet. Pin it if and when rule scope ships.

Suggested shape

packages/cli/test/ast-grep-vendor-contract.test.ts, mirroring the Vale file: resolve the binary via findSgBinary(), skip the suite if absent, build throwaway projects in mkdtemp, invoke the binary directly rather than through runAstGrepScan, and comment each case with the code it protects.

Refs #93