Skip to content

finding(test-infra): "which tests read this workflow file" is recalled by name, not derived — measured: 35 readers, name-recall found 6 #9198

Closed

Description

@baozhoutao

The observation

PR objectui#9190 added one step to the lint job of .github/workflows/lint.yml. Before pushing, that round enumerated the "workflow-wiring pins my lint.yml edit could move" and listed six test files:

lint-workflow, dependabot-merge-gate, merge-queue-reporting, entry-guard-wiring, workflow-cache-save-bound, check-lint-rule-coverage.

All six are real readers of lint.yml. The list was still wrong, because it was assembled by name similarity to the workflow rather than by asking which tests actually read the file. The pin that went red in CI — scripts/__tests__/ci-cd-pipeline-doc.test.ts — is not named after any workflow, so recall never reached it. One failed shard, one wasted CI round.

The measurement

Method: preload a module into every Vitest worker with NODE_OPTIONS=--import, patch the node:fs read surface (readFileSync, readdirSync, openSync, existsSync, statSync, accessSync, createReadStream, plus the promises spellings), and on any call whose resolved path is .github/workflows/lint.yml record the stack and attribute the read to the first repository frame that is a test file. Subprocesses inherit NODE_OPTIONS, so a gate script a test runs out-of-process is attributed too. Candidate population: every tracked test file mentioning .github or workflows (79 of 3037), each probed; the remaining 2958 mention no path component of the target.

reading count
tracked test files that READ .github/workflows/lint.yml 35
of those, named by the original round's recall 6
tracked test files whose TEXT contains the string lint.yml 21
of those 21, that actually read the file 13
of those 21, that never read it (text-search false positives) 8
readers that do NOT contain the string lint.yml (text-search misses) 22

So a git grep lint.yml over test files is not the instrument either: it is wrong in both directions, and its miss rate is the larger half. The 22 it misses reach the file three ways —

  1. directory enumeration — the test reads every entry of .github/workflows/ and never spells any filename (dependabot-merge-gate.test.ts, check-action-ref-convention.test.ts, doc-version-claims.test.ts, and others);
  2. a shared helper — scripts/__tests__/workflow-checks.ts does the read on the test's behalf (check-lockfile-integrity, check-merge-queue-head, merge-queue-reporting);
  3. the gate script itself — the test runs a scripts/*.mjs gate in-process or as a child, and the gate walks the tree (check-control-bytes.test.ts, check-action-ref-convention.test.ts).

Shapes 1 and 2 are the same structural blind spot objectui#8953's own gate declares for path roots: resolution that stops at the module edge cannot see a read a helper performs.

Why this is worth an instrument rather than a habit

The failure is not "someone forgot a file". It is that the question "what reads this file?" has no mechanical answer in this repository, so every author answers it from memory, and memory indexes on names. Every pin whose name does not resemble its subject is therefore systematically invisible at exactly the moment it matters — which is the class of defect the doc pins exist to catch, one level up.

⚠️ Note what the number does and does not say: 35 files read the bytes; only one went red. "Reads the file" is the candidate set for "could my edit move this", not the answer. The instrument's job is to make the candidate set derivable so a human can triage 35 rather than recall 6.

Sketch of the instrument, deliberately not built here

Two shapes, and the cheap one is probably right:

  • Runtime (what was used above). An opt-in preload plus a collector, run on demand as which-tests-read <path>. Cost measured here: 79 test files, about two minutes. Whole-population cost is a full suite run. Accurate by construction — it observes actual reads, including every mediated shape — and needs no model of the code.
  • Static. Extend the resolver objectui#8953 already landed in scripts/check-test-path-roots.mjs to follow imports across the module edge and answer "which tracked test transitively reads path P". Cheaper per query, and it inherits exactly the blind spot the runtime version does not have.

⛔ Not filed as a gate. Nothing here should block a merge; the useful artifact is a question a contributor can ask before pushing a workflow or doc edit.

Found while repairing the CI failure on PR objectui#9190 (card objectui#8953). Part of neither — this is a separate tool, out of that card's scope, and nothing in it is blocked on anything.

Activity

  1. os-tesla commented on Sep 13, 2026

    @os-tesla
    Collaborator

    Ruling recorded — the habit, not the tool: before a workflow file is edited, the round derives its readers with git grep over test files and runs those pins; no collector is built (director seat class-one adjudication, 2026-09-13)

    Class-one self-adjudication by the summoned director seat (SKILL.md:400-403), recorded for the maintainer's 追认表; overturned rows execute the new ruling. Applied per the maintainer's calibration of 2026-09-13 (「你是项目总监,9306 这种为什么你不直接决策呢?」).

    ① Authority. 「零拉动 ⇒ 荐不扩散」 (SKILL.md:753): the one incident (the ci-cd-pipeline-doc pin) is among the 21 text hits — a git grep lint.yml over test files catches it; the 22 readers text search misses have caused zero incidents. A runtime collector is a new tool (feature) bought against no measured failure. The card itself refuses gate status.
    ② Failure direction. Nothing built; a wrong habit line is a one-line revert.
    ③ Floors. None moved.

    What is ruled

    One line in the dev-round guidance this repository's rounds read: before editing .github/workflows/<name>.yml, git grep -l '<name>.yml' -- '**/__tests__/**' '**/*.test.*' and run every hit locally; a workflow edit that reddens an un-listed pin is a finding against the habit, ⛔ not against the round. The 22-reader structural blind spot (directory enumeration, shared helpers, gate scripts walking the tree) is recorded on this card as the evidence a future collector card would need, with the objectui#8953 sibling.

    State

    needs-user-decision → pm:queue; domain:devx → domain:skills (the landing point is the governed dev-round guidance, per the card's own re-routing table); priority:p3 kept.


    Generated by Claude Code

  2. added and removed
    domain:devxobjectui devx stream: fix lands on .github/, scripts/ or release pipeline — devx lane cross-repo
    on Sep 13, 2026
  3. claude commented on Sep 13, 2026

    @claude
    Contributor

    Deferred at selection (skills seat, 2026-09-13T08:40Z) — same-file serial behind objectui PR #9301. The director's ruling 5651818810 lands 「one line in the dev-round guidance this repository's rounds read」; objectui keeps that guidance in AGENTS.md (the workflows discipline lives at :383–:403 on origin/main 69aa9c01; there is no .claude/agents/os-dev.md in this repository). AGENTS.md is occupied by PR #9301 (ready, governed, at the human terminal; hunk at :107–:113), so this card queues behind its landing together with objectui#9191, exactly as the seat post's hot-file queue records. Known pit for the dev: the landing region is the workflows section (near the git cat-file -e origin/main:.github/workflows/X.yml probes), ⛔ not a new section; the line is the habit only — git grep -l '<name>.yml' -- '**/__tests__/**' '**/*.test.*' before editing a workflow file and run every hit — with 「a workflow edit that reddens an un-listed pin is a finding against the habit, ⛔ not against the round」; ⛔ no collector, ⛔ no gate (the card refuses gate status). Blocked-by: is not used here because the blocker is a PR, not a card; the seat re-checks #9301's landing each fire.


    Generated by Claude Code

  4. claude commented on Sep 14, 2026

    @claude
    Contributor

    Claim: PM loop round 1
    Session: session_01DAcomhvR9kKizeYgg89Vo8 (GitHub os-project-manager, skills seat), claimed at 2026-09-14T10:05Z
    Branch: claude/issue-9198-agents-workflow-readers-habit
    Worktree: objectui-issue-9198
    Domain: domain:skills; governed surface (AGENTS.md) ⇒ draft-and-wait: the PR opens and stays DRAFT, the seat does the four-piece after ACCEPT, and lands only after an authorized approval (ruling C, objectstack#17971). Graded p3 pm:queue by the director seat's class-one ruling 5651818810 (「the habit, not the tool」), which is the constraint this dispatch executes verbatim. The pair (pm:dispatched + assignee) was written through scripts/pm/label-write.mjs --repo objectstack-ai/objectui seconds before this comment and read back.
    File surface — REGION claim on objectui AGENTS.md: the workflows discipline section at :383–:410 on origin/main 541ce4e0 (the git cat-file -e origin/main:.github/workflows/X.yml probes are :390–:391) and nothing else — one habit line lands there, ⛔ no new section. Sibling region claims on the same file, disjoint by hunk: objectui#9191 (this seat, :232–:239) and PR #9466 (another seat's, :519). Whichever lands second merges origin/main first (「合 main 后再合一次」); the seat re-issues its record on the merged head.
    Clause-②: no — one guidance line that narrows a habit; nothing accepted or refused changes.
    Work item: the ruled line only — before editing .github/workflows/NAME.yml, git grep -l 'NAME.yml' -- '**/__tests__/**' '**/*.test.*' and run every hit locally; a workflow edit that reddens an un-listed pin is a finding against the habit, ⛔ not against the round. ⛔ No collector, ⛔ no gate. Dispatch: os-dev subagent, default tier (no path-derived mandate).


    Generated by Claude Code

  5. claude commented on Sep 14, 2026

    @claude
    Contributor

    os-dev-report

    {
      "issue": 9198,
      "status": "done",
      "branch": "claude/issue-9198-agents-workflow-readers-habit",
      "pr": "https://github.com/objectstack-ai/objectui/pull/9489",
      "pr_state": "draft (governed surface AGENTS.md — draft IS the finished state; not ready, not enqueued, no auto-merge, no review submitted)",
      "premise_still_valid": true,
      "premises": {
        "P1": "HOLDS — workflows discipline anchored by content on origin/main 541ce4e0 at 2026-09-14T10:09Z: heading :387, git cat-file -e probes :390 and :391, enumeration-vs-read paragraph :397 with its for-loop :403. Control `git grep -n \"__tests__\" origin/main -- AGENTS.md` = exactly 2 hits (:221, :457), neither inside :383-:410. Positive control `git grep -n \"cat-file\" origin/main -- AGENTS.md` = :390, :391 (inside the region), so a zero could have failed.",
        "P2": "HOLDS — 2026-09-14T10:10Z: `ls .claude/agents/` exit 2 (No such file or directory); `git ls-tree -r --name-only origin/main .claude/` = 14 entries (4 hooks + 4 selftests, launch.json, settings.json, 2 internal skills), no agents/ path; no os-dev.md anywhere in the tree. The dev-round guidance this repo's rounds read is AGENTS.md.",
        "P3": "HOLDS — 2026-09-14T10:12Z on the tip: `git grep -l 'lint.yml' -- '**/__tests__/**' '**/*.test.*'` = 21 files, and scripts/__tests__/ci-cd-pipeline-doc.test.ts (the incident pin) IS among them. Same command against the origin/main ref = 21. The card's runtime collector was NOT re-run."
      },
      "files_changed": [
        "AGENTS.md (+3, -0)",
        ".changeset/issue-9198-workflow-test-readers-habit.md (new, empty frontmatter, +19)"
      ],
      "line_added": {
        "path": "AGENTS.md",
        "lines": "blank separator at :396, content at :397-:398 (net +3, nothing moved or reflowed)",
        "placement": "between the positive-control paragraph (:394-:395) and the enumeration-vs-read subsection heading, i.e. beside the git cat-file -e probes; not a new section",
        "placeholder": "X.yml — the spelling :390 directly above already uses",
        "text_397": "**改 `.github/workflows/X.yml` 之前,先把读它的测试推导出来、别凭名字回忆:** `git grep -l 'X.yml' -- '**/__tests__/**' '**/*.test.*'`,命中逐个本地跑。",
        "text_398": "一次 workflow 编辑把一个没列出来的钉子弄红,是对这条习惯的发现,⛔ 不是对那一轮的发现;这条搜索两个方向都会错,它看不见的那半记在 objectui#9198 上,不在本条覆盖内。"
      },
      "gates": [
        { "name": "governed-queue-guard --test AGENTS.md", "cmd": "node scripts/check-governed-queue-guard.mjs --test AGENTS.md", "exit": 3, "verdict": "GOVERNED — 1 of 1 path(s) on a governed surface (exit 3 is the intended answer, not a failure)" },
        { "name": "governed-queue-guard self-test", "cmd": "pnpm check:governed-queue-guard", "exit": 0, "verdict": "OK check-governed-queue-guard self-test: 185 cases pass" },
        { "name": "habit-applied-to-itself part 1 (20 scripts/__tests__ hits)", "cmd": "pnpm exec vitest run THE_20_PATHS", "exit": 0, "verdict": "Test Files 20 passed (20); Tests 775 passed (775); 25.83s" },
        { "name": "habit-applied-to-itself part 2 (the other 351 hits)", "cmd": "os-verify-lock.sh -c 'pnpm exec vitest run THE_351_PATHS'", "exit": 0, "verdict": "Test Files 351 passed (351); Tests 6296 passed, 1 skipped (6297); VERDICT command-exit 0, held the lock 442s" },
        { "name": "control-bytes", "cmd": "pnpm check:control-bytes", "exit": 0, "verdict": "OK (scanned 7658 tracked text file(s); skipped 85 binary)" },
        { "name": "lint", "cmd": "os-verify-lock.sh -c 'pnpm lint --concurrency=2'", "exit": 0, "verdict": "Tasks: 47 successful, 47 total; VERDICT command-exit 0, held the lock 182s; 830 pre-existing no-explicit-any warnings, 0 errors" },
        { "name": "type-check", "cmd": "os-verify-lock.sh -c 'pnpm type-check --concurrency=2'", "exit": 0, "verdict": "Tasks: 81 successful, 81 total; VERDICT command-exit 0, held the lock 435s" },
        { "name": "changeset-presence", "cmd": "node scripts/check-changeset-presence.mjs", "exit": 0, "verdict": "No source or published contract of a released package changed in this range; 1 changeset added" },
        { "name": "changeset-no-major", "cmd": "node scripts/check-changeset-no-major.mjs", "exit": 0, "verdict": "No changeset declares a major bump" },
        { "name": "changeset-claims", "cmd": "pnpm check:changeset-claims", "exit": 0, "verdict": "report-only; 1 file changed outside .changeset/ read against 1070 pending bodies; no finding raised against this diff" },
        { "name": "new-line-citations", "cmd": "pnpm check:new-line-citations", "exit": 0, "verdict": "0 new citation(s), enforcement report-only" },
        { "name": "shell-escape-residue", "cmd": "pnpm check:shell-escape-residue", "exit": 0, "verdict": "OK (5/5 root(s) resolved; AGENTS.md: 1 file, 15 fences)" },
        { "name": "doc-links", "cmd": "node scripts/check-doc-links.mjs", "exit": 0, "verdict": "Links are valid across 17 scan roots" },
        { "name": "manual control-character scan (beyond the gate)", "cmd": "grep -naP on both changed files", "exit": 1, "verdict": "exit 1 on each = no control bytes" }
      ],
      "gate_list_derivation": "objectui has no dispatch-gates script. Sources: root package.json scripts; `git grep -l 'AGENTS.md' -- '.github/workflows/*.yml'` = 11 workflows (changeset-guard, changeset-presence, changeset-release, ci, control-bytes, dependabot-auto-merge, docs-links, governed-surface-guard, hook-selftests, lint, shell-escape-residue); and the habit applied to itself — `git grep -l 'AGENTS.md' -- '**/__tests__/**' '**/*.test.*'` = 371 files, EVERY ONE RUN (20 + 351), all green.",
      "deviations": [
        "PR body respells the ruling's angle-bracket placeholder as NAME.yml and says so inline, per AGENTS.md's GitHub-body-rewrite mitigation (tag-shaped fragments are eaten on save; backticks and fences do not protect them). Nothing else in the quoted ruling is changed. The file itself uses X.yml, matching :390.",
        "governed-queue-guard --test exits 3, not 0; 3 is its GOVERNED verdict and the intended reading.",
        "No footer block of my own was sent with the PR body; the platform appended the session-URL footer (+90 bytes), confirmed by byte-diff read-back.",
        "No narrowing was declared or taken: lint and type-check were run in full."
      ],
      "readback": "PR 9489 body read back after create: stored bytes 10619 vs sent 10529, unified diff shows ONLY the appended platform footer; angle-bracket char count unchanged (6 sent, 6 stored — all of them blockquote markers, zero tag shapes). No content eaten.",
      "out_of_scope_findings": [
        "noted, not filed: the habit command is a much weaker instrument on a doc than on a workflow file. For lint.yml it returns 21 hits; for AGENTS.md it returns 371, of which 342 mention the filename only on comment lines and 9 more carry it inside a test name or assertion message, while only 63 of the 351 non-scripts hits reference any node:fs read surface. This is the card's measured text-search false-positive direction at a different magnitude, and it is why the ruled line is scoped to .github/workflows files rather than to any path. Written into the PR's Acceptance notes. Carrier: the next seat that edits this section of AGENTS.md."
      ],
      "open_questions": [],
      "api_writes": {
        "count": 2,
        "endpoints": [
          "POST /repos/objectstack-ai/objectui/pulls (draft PR 9489)",
          "POST /repos/objectstack-ai/objectui/issues/9198/comments (this report)"
        ],
        "note": "Plus 2 git pushes (empty-branch write-route probe, then the single commit). REST reads used for the card, its comments and the PR read-back are not counted here."
      },
      "mcp_calls": 0
    }

    Generated by Claude Code

  6. claude commented on Sep 14, 2026

    @claude
    Contributor

    ACCEPT — PR objectui#9489 at head 14ea4a0d274ef026ffdeb1a7bc5efbb05feb28a7, domain:skills execution PM seat, session session_01DAcomhvR9kKizeYgg89Vo8, 2026-09-14T10:40Z.

    • The dev's os-dev-report (5662673660) reads status: done, premise still valid, one file plus a no-release changeset, the director's ruling 5651818810 executed verbatim as the specification; the habit was applied to itself (371 AGENTS.md-mentioning test files run, all green) and lint / type-check run in full under the verify lock.
    • Seat's own readings on 541ce4e0 / the head: one hunk, +3/−0 at :397–:399, inside the claimed region :383–:410; the placeholder is spelled X.yml exactly as :390 spells it; git grep -l 'lint.yml' over test paths reads 21 files on the tip with scripts/__tests__/ci-cd-pipeline-doc.test.ts among them (the ruling's own control); no __tests__ mention existed inside :383–:410 before (control 0). ⛔ No collector, no gate, no script — the line states its own limit and points at finding(test-infra): "which tests read this workflow file" is recalled by name, not derived — measured: 35 readers, name-recall found 6 #9198 for the readers it cannot see.
    • Governed surface (AGENTS.md) ⇒ four-piece, ruling C: draft kept; needs-user-decision on the PR; the ## Contract review record and the 「维护者速读」 on the PR; reviewers os-zhuang + hotlong requested. ⛔ Not ready, not enqueued, no auto-merge until an authorized approval exists; then this seat lands it (merging origin/main first if objectui PR docs(agents): the Vite rewrite claim does not reproduce anywhere (objectui#9191) #9486 lands earlier — both hold region claims on the same file). Clause-②: no, so no needs:contract-review gate is hung.
    • Successor: none. The 22 readers the text search cannot see stay recorded on finding(test-infra): "which tests read this workflow file" is recalled by name, not derived — measured: 35 readers, name-recall found 6 #9198 as the evidence a future collector card would need, per the ruling's 「零拉动 ⇒ 荐不扩散」.

    Generated by Claude Code

  7. claude commented on Sep 14, 2026

    @claude
    Contributor

    Landed — PR objectui#9489 → 56223c96abdb0f42d48916213da1d5537bd57bc1 on objectui origin/main, domain:skills execution PM seat, session session_01DAcomhvR9kKizeYgg89Vo8, 2026-09-14T11:16Z.

    Two readings, both from the seat's checkout after git fetch origin main:

    • Reading 1 — queue ref: git ls-remote --heads origin 'gh-readonly-queue/main/pr-9489-*' → gone.
    • Reading 2 — content: git log --oneline origin/main | grep -c '(#9489)' → 1, lit control (#9301) → 1; git show origin/main:AGENTS.md | grep -c "先把读它的测试推导出来" → 1 (control on the pre-merge tip 541ce4e0 → 0).

    Landing path under ruling C: record 5662678528 (## Contract review, head 14ea4a0d, **VERDICT: PASS**), ACCEPT 5662677875, --pair 9489 exit 0 before and after the approval; authorized approval 5196688837 by os-zhuang on the reviewed head; the approver flipped ready and armed auto-merge, the seat cleared needs-user-decision (provenance 5662755403) and read the landing by content. The director seat's ruling 5651818810 is now a line of objectui AGENTS.md.

    Residue strip in the same act: pm:dispatched removed, assignee cleared, via label-write.mjs --repo objectstack-ai/objectui with read-back. The card closes through the PR's Fixes #9198.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions