Skip to content

test(docs): CLI.md context/audit --quick section headers checked against real re-runs - #2510

Merged
carlos-alm merged 1 commit into
mainfrom
fix/issue-2369-cli-output-content-diff
Aug 14, 2026
Merged

test(docs): CLI.md context/audit --quick section headers checked against real re-runs#2510
carlos-alm merged 1 commit into
mainfrom
fix/issue-2369-cli-output-content-diff

Conversation

@carlos-alm

Copy link
Copy Markdown
Contributor

Problem

#2212 extended docs-drift protection to `docs/examples/CLI.md` with two tiers (static command/flag validity, and representative real execution), but neither verifies that a documented example's output content/shape still matches what the command actually produces — only that it's syntactically valid and doesn't crash.

Fix

Added a Tier 3, deliberately scoped to just the two commands whose documented output already uses clear `## Section` markdown-style headers (`context`, `audit --quick`), per the issue's own suggested starting point:

  • Extract every `## Section Name` header from a block of text, normalizing away a trailing dynamic count (`## Callers (3)` → `## Callers`) — otherwise these are pure static labels, unlike the H1 title line (`# () — :`), which fully embeds dynamic content and would need real normalization rules to compare fairly (deliberately left out of scope — see below).
  • Find the documented output block immediately following a specific ```bash ``` block in CLI.md.
  • Run the real command against a small fixture graph (reusing the existing Tier 2 fixture, extended with one internal/non-exported symbol so `audit --quick` produces a real `## Internal` section) and compare header lists.

Real bug caught during development: the documented `context` example was missing two sections that exist in the current CLI (`## Children`, `## Complexity`) and had stale names for two others (`## Dependencies` → `## Direct Dependencies (N)`, `## Callers` → `## Callers (N)`). Fixed in this same PR, mirroring #2212's own precedent of fixing what its own new check finds. `audit --quick`'s documented headers already matched exactly — no doc change needed there.

Deliberately out of scope

  • Full output-content diffing (file paths, line numbers, source snippets, dependency-list contents) — needs real dynamic-content normalization rules and a strictness-level decision, explicitly flagged in the issue as "the harder half," not decided unilaterally here.
  • MCP.md's equivalent (tool-call response shape vs. a real MCP server call) — entirely different mechanics (needs standing up/driving an MCP server, not just `execFileSync`), filed as follow-up: MCP.md tool-call output structural comparison (entirely unaddressed) #2509.

Closes #2369

Test plan

  • New tests pass against the current (now-fixed) CLI.md content
  • Verified by hand that the old `## Dependencies`/`## Callers` (no count) headers would have failed against real current output before the doc fix
  • Full `vitest run` suite (5285 passed, 328 files) after rebuilding the native addon (stale in the fresh worktree, unrelated to this change)
  • `tsc --noEmit`, `biome check` clean
  • TS-only test/docs change — no Rust files touched

…nst real re-runs

Extends #2212's docs-drift protection for CLI.md with a third tier: for
the two commands whose documented output already uses clear ## Section
headers (context, audit --quick), extract just those header labels
(normalizing away a trailing dynamic count) and compare against a real
re-run against a small fixture graph. Deliberately scoped to header
labels only, not full output content, which would need real dynamic-
content normalization rules (file paths, line numbers, source snippets) —
the harder half of #2212's original ask, still deferred.

Already caught a real bug during development: the documented context
example was missing the Children/Complexity sections added to the CLI
since the doc was last updated, and had stale names for two others
(Dependencies -> Direct Dependencies (N), Callers -> Callers (N)) — fixed
in this same change, mirroring #2212's own precedent.

MCP.md's equivalent (tool-call response shape vs. real MCP server output)
is a separate, unaddressed piece of scope, filed as #2509.

Closes #2369

docs check acknowledged
@greptile-apps

greptile-apps Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds a third tier of CLI documentation-drift coverage that reruns context and audit --quick, normalizes their section counts, and compares the resulting section labels with documented examples.

  • Updates the context example with current Children, Complexity, Direct Dependencies, and Callers sections.
  • Shares and extends the CLI fixture with an internal symbol required to exercise audit --quick.
  • Adds helpers and integration tests for extracting and comparing documented versus live section headers.

Confidence Score: 5/5

The PR appears safe to merge, with no actionable correctness or security issues identified.

The updated examples match the section conditions exercised by the fixture, and the new tests consistently compare the intended documented and live CLI output shapes.

Important Files Changed

Filename Overview
docs/examples/CLI.md Refreshes the documented context output sections and labels to match the current CLI shape.
tests/integration/docs-examples-cli-drift.test.ts Adds focused structural-output drift tests using a representative WASM-built fixture and normalized section-header comparisons.

Reviews (1): Last reviewed commit: "test(docs): cli.md context/audit --quick..." | Re-trigger Greptile

@carlos-alm
carlos-alm merged commit 23e4c80 into main Aug 14, 2026
39 of 41 checks passed
@carlos-alm
carlos-alm deleted the fix/issue-2369-cli-output-content-diff branch August 14, 2026 23:07
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 14, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

follow-up: full CLI.md/MCP.md output-content diffing against real invocations

1 participant