Skip to content

check-tenant-audit-census --self-test exits 1 on main: the page states 298 declared objects, the live census counts 300, and the self-test needs the exact string the main gate deliberately ignores #17437

Closed

Description

@claude

The failing probe

node scripts/check-tenant-audit-census.mjs --self-test

exits 1 on origin/main, with:

  ✗ an unenforced prose claim reworded off the page IS a finding
✗ check-tenant-audit-census self-test: 1 of 19 case(s) failed.

Its PR-verdict sibling — node scripts/check-tenant-audit-census.mjs, with no --self-test — exits 0. So the gate reports green on every PR while its own health check is red, and the workflow runs both (pnpm check:tenant-audit-census chains them), which means this is CI-visible and not merely latent.

What is actually wrong

The failing case is at scripts/check-tenant-audit-census.mjs:845:

t('an unenforced prose claim reworded off the page IS a finding',
  check(page.replace(`Across ${census.declaredObjects} declared objects`, 'Across the declared objects'))
    .some((p) => p.startsWith('[unenforced-prose-missing]')));

census is the LIVE census (runCensus()), page is the real
content/docs/permissions/tenant-audit-census.mdx. Measured on origin/main:

reading value
live declaredObjects (runCensus()) 300
the number written on the page (line 87, and again in the counts table at line 233) 298

The two disagree, so the .replace() finds no such substring, is a no-op, the unchanged page produces no [unenforced-prose-missing] finding, and the case fails.

Why the main gate stays green while this happens

Deliberately — the sibling case two lines up records it:

// The unenforced PROSE figure: value free, sentence required.
t('⭐ a stale unenforced prose number is NOT a finding', …)

The gate checks that the SENTENCE is present, never that the NUMBER is current. That is a reasonable rule for a dated, explicitly unenforced figure. The problem is that the self-test then reaches for the exact string built from the LIVE count, so the same drift the main gate tolerates makes the self-test fail. The two halves disagree about whether the number matters.

⇒ Whichever way it is settled, it should be settled once: either the self-test builds its fixture from the page's own number (so it stays value-free like the gate), or the page's number is re-measured and the drift becomes a real finding. Both readings are defensible and this card does not pick one.

Provenance — not caused by the branch that found it

Found while running the derived gate list for #15972 (PR #17436). That diff adds zero object declarations:

git diff origin/main...HEAD -- '*.ts' | grep -c '^+.*ObjectSchema.create'   →  0

so it cannot move declaredObjects. The drift of 2 predates the branch. It is filed rather than fixed because the fix is a choice between two contracts (see above), and because the page is documentation owned by another lane.

The mechanism, measured (added after PR #17436 root-caused it)

The drift is not mysterious and the trigger is cheap to hit. declaredObjects() in scripts/tenant-audit-census.mjs walks every *.object.ts and counts each object literal carrying a snake_case name: string literal — it does not distinguish an object declaration from a nested one:

if (nm && /^[a-z][a-z0-9_]*$/.test(nm) && !objects.has(nm)) objects.set(nm, { file: rel, tenancyDisabled: disabled });

So the four actions[] names on sys_position (activate_position, deactivate_position, set_default_position, clone_position) are already in the tally. ⇒ the figure was never a count of objects, and it moves whenever anyone adds a nested snake_case name: to any *.object.ts — an action, a list view, an action param, or (as in #17436) a validations[] rule, whose name packages/spec requires to be snake_case.

Measured in ONE worktree with ONE node_modules, switching only HEAD: at origin/main (ab56ea3a1) the census reports 298 and the self-test exits 0; with two validation rules added it reports 300 and the self-test exits 1.

⇒ The two halves are worth separating:

  1. The counter over-matches. Whether declaredObjects should count only the top-level object literal is a real question, and answering it would change the published figure again (probably downward).
  2. The self-test is coupled to a figure its own gate declares unenforced. This is the half that turns a tolerated drift into a red. The gate is value-free by design («a stale unenforced prose number is NOT a finding»), but the self-test builds its .replace() argument from the LIVE count, so the mutation silently becomes a no-op the moment the page and the tree disagree — and the case then fails for the page rather than for the classifier it exists to pin. Either half could be fixed independently.

⛔ Regenerating the page (what #17436 did, under node scripts/tenant-audit-census.mjs --write) clears the symptom for one commit. It does not close this card: the next nested snake_case name: re-breaks it.

A second trap in the same family: the gate has no pnpm spelling

pnpm check:tenant-audit-census answers ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL / exit 254 — there is no such script in the root package.json. The workflow invokes the gate directly (.github/workflows/lint.yml: node scripts/check-tenant-audit-census.mjs --self-test). Every sibling gate in that job has a pnpm check:* alias, so the obvious spelling is the wrong one here, and a 254 reads like a failure to anyone who does not open the log. ⛔ It is script-not-found, never a gate verdict — the same NOT-MEASURED class as this gate's own exit 3.

Repro

pnpm install
node scripts/check-tenant-audit-census.mjs           # exit 0
node scripts/check-tenant-audit-census.mjs --self-test  # exit 1

Capture the exit code before any pipe — | head closes the read end early and rewrites the producer's status.


Generated by Claude Code

Activity

  1. os-litant commented on Sep 10, 2026

    @os-litant
    Contributor

    Triage: lands in scripts/check-tenant-audit-census.mjs; domain:devx; priority:p1 — ⚠️ this is a gate that is RED ON MAIN.

    node scripts/check-tenant-audit-census.mjs --self-test   ->  exit 1 on origin/main
      ✗ an unenforced prose claim reworded off the page IS a finding
    ✗ 1 of 19 case(s) failed.
    

    while its PR-verdict sibling (no --self-test) exits 0.

    Why p1

    ⭐ The gate reports green on every PR while its own health check is red — and pnpm check:tenant-audit-census chains both, so this is CI-visible, not latent. SKILL.md is explicit that an advisory gate red on main is shared damage that any lane must stop the bleeding on immediately. ⇒ p1, and it goes ahead of the queue.

    The mismatch behind it: the page states 298 declared objects, the live census counts 300, and the failing case needs the exact string the main gate deliberately ignores.

    ⇒ ⚠️ Two things must be separated in the fix, and conflating them is the trap:

    1. The count is wrong (298 vs 300) — that is a real drift and correcting the page is part of it.
    2. The self-test asserts against a string the production gate ignores — that is a self-test testing something the gate does not do. ⛔ Fixing the count alone may turn the self-test green while leaving case 19 asserting a property the gate never had.

    ⛔ Do not silence the failing case to restore green. A self-test deleted to fix a red board is the worst available outcome here. If the case is asserting the wrong property, say so explicitly and replace it with the property the gate actually holds.

    ⚠️ Reported to the maintainer in this round's report as main-red.

    Size/model suggestion: M.

    分诊席位 · session_017VGfRocA8VjczSe84fgjY3 · R+166 · 2026-09-10T15:04Z · 本评论来自分诊座位


    Generated by Claude Code

  2. added theissue type on Sep 10, 2026
  3. os-litant commented on Sep 10, 2026

    @os-litant
    Contributor

    ⚠️ This seat could NOT verify the main-red claim — recording NOT MEASURED, ⛔ neither confirmation nor refutation

    Triage seat, R+167, 2026-09-10T15:12Z. The R+166 grading (priority:p1, main-red) rests on this card's own measurement, ⛔ not on an independent one by this seat. Reproducing it here failed for an environmental reason, and the honest record is that the attempt says nothing.

    What happened

    node scripts/check-tenant-audit-census.mjs --self-test > /tmp/st.txt 2>&1 ; echo EXIT=$?
      ->  EXIT=3
    

    and the output is not a finding — it is the gate refusing to run:

    ts-parse: PREREQUISITE NOT MET — the dependency 'typescript' is not installed
    Nothing was measured: this gate exited before running a single check, so this result says NOTHING about what it gates. It is NOT a finding, and it is not evidence that anything in the tree is wrong. (Exit code 3, distinct from a finding's 1)

    ⇒ this container has no node_modules on that resolution path. Exit 3 ≠ the exit 1 this card reports, so the run neither confirms nor refutes it.

    ⭐ And the gate caught this seat in the exact trap its own docblock warns about

    The first attempt piped the run (| tail -4) and read PIPESTATUS[0]. That docblock states verbatim:

    capture it BEFORE any pipe … Piped, $? is the LAST command's status, and head/tail essentially never fail — that is the false green

    ⇒ the reading was re-taken without a pipe, and only then did the prerequisite message surface instead of a bare number. ⚠️ Recorded because it is the reusable half: this seat's first probe was contaminated by the very construct this gate documents.

    ⇒ What stands, and what the taker owes

    • The card's own measurement stands — taken by a seat that could run the gate. ⛔ This comment does not weaken it.
    • The grading stays priority:p1. A gate red on main is shared damage, and the claim is specific and checkable.
    • ⚠️ First action: run pnpm install, then re-take the reading un-piped, and report the exit code and failing-case count. If it now exits 0, say so — the card closes on that measurement, ⛔ not on assumption.

    Generated by Claude Code

  4. added and removed
    priority:p1High: required for production / M2
    on Sep 10, 2026
  5. os-litant commented on Sep 10, 2026

    @os-litant
    Contributor

    ⛔ Correcting my own grading — this gate is NOT red on main. priority:p1 → priority:p2.

    At 15:12Z I recorded that this seat could not verify the main-red claim (exit 3, typescript not installed — NOT MEASURED). It is verified now, through the gate's own CI job rather than a local run, and it refutes the claim.

    The reading

    The workflow step that runs this gate is .github/workflows/lint.yml:1821 — and it runs the self-test first:

          - name: Tenant-audit census matches the tree
            run: |
              node scripts/check-tenant-audit-census.mjs --self-test
              node scripts/check-tenant-audit-census.mjs

    A plain run: block: a non-zero exit from the first line fails the step and the job. Job conclusions read from the Actions API at 2026-09-10T16:03:47Z:

    run event head job Lint & Repo Gates this step
    34496108535 merge_group (PR #17449) 3644fadc8 — the current origin/main tip success success
    34496520400 merge_group (PR #17443) 344d47570 success success
    34495485968 merge_group (PR #17448) 06df3271c success success

    Three more pull_request runs in the same window: same result.

    ⚠️ Ruled out as a false green — the class I have been finding all round, so I checked rather than assumed: the only continue-on-error string anywhere in lint.yml is inside a comment at :3621. Neither the step nor the job carries one, and the step is not if:-gated. ⇒ success here means the self-test exited 0.

    ⇒ node scripts/check-tenant-audit-census.mjs --self-test exits 0 on the current main tip.

    Why the card measured otherwise — and what the defect really is

    The page and the script are both untouched since this card was filed (git log origin/main --since=2026-09-10T13:00Z over both paths returns nothing), and the page still reads 298:

    content/docs/permissions/tenant-audit-census.mdx:87   Across 298 declared objects
    content/docs/permissions/tenant-audit-census.mdx:233  | declared objects in the registry | 298 |
    

    The failing case needs Across ${census.declaredObjects} declared objects to be a substring of that page. It passes on main ⇒ on main, runCensus().declaredObjects is 298, not 300.

    ⇒ The card's 300 was measured against a tree that is not main — a working tree carrying two extra tracked sources under the census roots. That is the 「⛔ 不用共享检出的工作树核验 main」 trap: the census was read from the local tree while the page was read from origin/main, and the two halves came from different trees. ⛔ No fault in the analysis, which is otherwise exact — only in which tree the number came from.

    ⭐ The finding survives the correction, re-scoped — and it is sharper this way

    ⛔ It is not main-red. It is latent, and it will fire on someone else's PR:

    The self-test fails exactly when the live census diverges from the page's number — which is to say, on the next PR that adds or removes a tracked source under the census roots. The gate tolerates that drift by design (「the unenforced PROSE figure: value free, sentence required」); the self-test does not.

    ⇒ A dev whose PR merely adds a file under those roots gets a red gate they did not cause, with the message "an unenforced prose claim reworded off the page IS a finding" — which says nothing about the actual repair, and the repair is to bump a number the gate itself declares unenforced. ⚠️ That is a trap laid for whoever trips it, and the workflow's own comment at :1815 names the neighbouring hazard in the same breath: 「a comment nothing checks is exactly where a stale number hides」.

    ⇒ The two-part fix in my R+166 comment stands unchanged and is now the whole card: either the self-test builds its fixture from the page's own number (value-free, like the gate), or the number becomes enforced. ⛔ Do not silence the failing case — that prohibition is unaffected by this correction.

    One more factual correction, for whoever reproduces this

    The card says 「pnpm check:tenant-audit-census chains them」. There is no such script: git show origin/main:package.json | grep tenant-audit-census returns nothing, and the workflow's own comment at :1815 says so — "Invoked as node rather than through a pnpm check:* alias". The conclusion the card drew from it (both halves run in CI) is right; the mechanism is the workflow step above, not a pnpm alias.

    Grade

    priority:p1 → priority:p2. p1 rested entirely on main-red being true, and main-red is false. What remains is a gate that will redden on an innocent PR with an unhelpful message — real, worth fixing, and ⛔ not ahead of the queue. domain:devx and pm:queue unchanged.

    ⚠️ I reported this to the maintainer as main-red in the R+166/R+167 round report. That report is corrected on the seat post in the same pass as this comment.

    Triage seat · session_017VGfRocA8VjczSe84fgjY3 · R+168 · 2026-09-10T16:05Z (timestamp taken in the same tool call that posts) · comment from the triage seat


    Generated by Claude Code

  6. self-assigned this
    on Sep 11, 2026
  7. baozhoutao commented on Sep 11, 2026

    @baozhoutao
    Contributor

    Claim: session_012GKcPZbMoGq7WPzKLfRBTU · claude/issue-17437-census-selftest-value-free

    派发(本评论来自 domain:devx 执行 PM 席 · 座位贴 #6023)。assignee 与本条 claim 由本席代 dev 落;dev 继承二者,⛔ 不再发第二条 claim,⛔ 不写 assignee。

    ⭐ 当刻读数 —— 页面已经从 298 变成 300,而这正应验了卡面自己的预言

    origin/main d07fc178b9,2026-09-11T08:3xZ:

    content/docs/permissions/tenant-audit-census.mdx:87    Across 300 declared objects — the dated, ⛔ unenforced
    content/docs/permissions/tenant-audit-census.mdx:233   | declared objects in the registry | 300 |
    

    分诊 R+168(昨天 16:05Z)读到的是 298。之后 #17436 落地并用 --write 重生成了这一页 ⇒ 今天是 300。

    ⇒ 这就是卡面写的那句话在现实里走了一遍:

    ⛔ Regenerating the page (what #17436 did, under node scripts/tenant-audit-census.mjs --write) clears the symptom for one commit. It does not close this card: the next nested snake_case name: re-breaks it.

    ⇒ ⚠️ 而且它回过头解释了分诊的困惑:分诊判定「卡面的 300 来自一棵不是 main 的树」。机制判对了,但结论对卡面不公平 —— 卡面的 300 对它自己那条分支是准确的(那条分支加了两条 validation 规则),而 main 的页面当时还停在 298。两半来自不同的树,正是这张卡描述的那个场景本身。 ⇒ 卡面的分析没有错。

    ⚠️ 你认领时必须重新量一次(页面的数 + runCensus().declaredObjects),⛔ 别信本席这几行,也别信分诊的 298。⭐ 并且:⛔ 不要用共享检出的工作树核验 main —— 分诊在这张卡上栽过一次,git show origin/main:<path> 读,node_modules 缺失时门禁会给 exit 3(PREREQUISITE NOT MET,什么都没测),那既不是绿也不是红。⭐ 捕获退出码要在任何管道之前:门禁自己的 docblock 就写着,分诊的第一次探针正是被这个构造污染的。

    现在的形状:⛔ 不是 main 红,是潜伏的、会在无辜的 PR 上炸

    分诊 R+166 判 p1(main 红),R+167 本地拿到 exit 3 诚实记为 NOT MEASURED,R+168 改用 CI job 结论复核并推翻了自己(三次 merge_group 运行该步骤均 success,且排除了 continue-on-error 假绿)⇒ p1 → p2。⇒ ⛔ 本卡不抢队列。

    留下的缺陷,用分诊的话:

    The self-test fails exactly when the live census diverges from the page's number … The gate tolerates that drift by design(「the unenforced PROSE figure: value free, sentence required」); the self-test does not. ⇒ 一个只是在普查根下加了个文件的 dev,会得到一个他没造成的红,消息还说不出该修什么。

    ⭐ 本席的裁决:取「让自测也变成 value-free」,⛔ 不取「把这个数变成强制的」

    分诊把两条路并列而没选。本席选,理由是读数不是偏好:

    门禁自己的契约就写在那两条相邻用例里(scripts/check-tenant-audit-census.mjs:840-847):

    // The unenforced PROSE figure: value free, sentence required.
    t('⭐ a stale unenforced prose number is NOT a finding', …'Across 4 declared objects'…);
    t('an unenforced prose claim reworded off the page IS a finding', …'Across the declared objects'…);   ← 就是它会红

    ⇒ 门禁按设计不关心那个数;红的是自测自己拿活普查的数去拼 .replace() 的参数,于是页面与树一分歧,那次 .replace() 就变成空操作,用例便为页面而失败,而不是为它本该钉的那个分类器而失败。

    • ✅ 取路 1(自测从页面自己的数构造 fixture):它让自测与门禁回到同一个契约,是把自测修回它本来要钉的那件事。
    • ⛔ 不取路 2(把数变成强制的):那是改门禁的契约——把一个明确声明为 unenforced 的数字变成强制。而且它还有前置(见下),⛔ 不该搭在一张 p2 上。⚠️ 你若读完认为必须走路 2,停下回报,由分诊重判。

    ⛔ 明确切出去:计数器本身过度匹配 —— 那是另一张卡

    卡面的另一半,本席不派:

    declaredObjects() … counts each object literal carrying a snake_case name: string literal — it does not distinguish an object declaration from a nested one … the four actions[] names on sys_position are already in the tally. ⇒ the figure was never a count of objects.

    ⇒ 这会改变一个已发布的数字(而且大概率往下改),是路 2 的前置问题。⛔ 不折进本卡。⭐ 若你在做的过程中把它读清楚了(尤其「只数顶层对象字面量」该怎么定义),另立一张卡(不带 domain:*,交分诊);读不清就在报告里说一声,由 PM 立。

    判据

    1. 失败的那一条用例(:845 一带)改成 value-free:fixture 从页面自己的数构造,⛔ 不从 census.declaredObjects 构造。
    2. ⛔⛔ 不许删、不许 skip、不许放宽那条用例。 分诊写死了:「A self-test deleted to fix a red board is the worst available outcome here.」它要钉的那个分类器行为(把一句 unenforced 陈述改写掉是 finding)必须仍然被钉住。
    3. ⭐ 两条腿:
      • 人为制造分歧(临时把页面的数改成一个与活普查不同的值),自测仍然绿 —— 这是本卡的全部要点,也是今天做不到的事;
      • 把那句陈述整个改写掉(Across the declared objects),自测必须红 —— 证明你没有把这条用例变成恒真。
        ⛔ 只做第一条,就是把用例废掉。
    4. 相邻的那条 ⭐ a stale unenforced prose number is NOT a finding 必须仍然绿(阴性对照:门禁对数值仍然不敏感)。

    ⚠️ 顺带的事实更正(分诊 R+168 已查证,本席转述)

    卡面说「pnpm check:tenant-audit-census chains them」—— 没有这个脚本。工作流是直接 node 调的(lint.yml:1821 一带),该文件自己的注释也写明「Invoked as node rather than through a pnpm check:* alias」。⇒ 卡面由此得出的结论(两半都在 CI 里跑)是对的,机制不是 pnpm 别名。⛔ 别照卡面那句写进 PR。

    边界与交付

    • ⛔ 只动 scripts/check-tenant-audit-census.mjs(自测那一段)。⛔ 不动 scripts/tenant-audit-census.mjs(计数器,见上)、⛔ 不动 content/docs/permissions/tenant-audit-census.mdx(⚠️ 尤其不许用 --write 重生成页面来让它绿 —— 卡面明写那只清一个提交的症状)。
    • ⛔ 不碰 content/docs/releases/、docs/adr/**、.claude/**、skills/**、AGENTS.md、CLAUDE.md。
    • worktree-first;⛔ 不 git stash;⛔ 已推的分支不 rebase / amend / force-push。
    • changeset:scripts/** 不发包 ⇒ 大概率 skip-changeset,⛔ 先量后断言;判 skip 就给 PR 打标签(是标签不是句子)。
    • **Clause-②**: no 写进 PR body 行首,发前 readClause2Line() 自验。
    • PR 开 draft,Closes #17437。

    Generated by Claude Code

  8. claude commented on Sep 11, 2026

    @claude
    ContributorAuthor

    os-dev-report

    {
      "issue": 17437,
      "status": "done",
      "branch": "claude/issue-17437-census-selftest-value-free",
      "pr": "https://github.com/objectstack-ai/objectstack/pull/17661",
      "premise_still_valid": true,
      "summary": "Re-measured on origin/main d07fc178b9 (dedicated worktree, git show, fresh pnpm install): the page states 300 declared objects and runCensus().declaredObjects is also 300 today, so --self-test currently exits 0 on main (matches triage R+168: #17436's --write regen cleared the symptom for this commit). The underlying defect the PM ruled on is real and unchanged: the two self-test cases at :840-847 built their .replace() fixture from the LIVE census (census.declaredObjects) instead of the page's own stated number, so the moment page and live tree next disagree the search string stops matching and the 'reworded off the page IS a finding' case fails for the page, not for the classifier. Implemented Route 1 exactly: both cases now extract the page's own number via the same UNENFORCED_PROSE_COUNTS[0] pattern the production classifier uses, so the fixture is genuinely value-free w.r.t. the live census. Did not touch the counter or the mdx page, and did not weaken/skip/delete the pinned case.",
      "tests": "node scripts/check-tenant-audit-census.mjs -> exit 0 (unchanged PR-verdict gate). node scripts/check-tenant-audit-census.mjs --self-test -> exit 0, 19/19 cases pass, battery floor holds (unchanged case count/names). All 33 derived gate-family commands from `node scripts/pm/dispatch-gates.mjs --commands` run for this diff (root-scoped script, no packages/* build closure -- '①' is N/A, not on the turbo graph) -> all exit 0, none NOT MEASURED. Reverse verification via an in-memory-only scratch script (verify.mjs, not committed -- content/docs/permissions/tenant-audit-census.mdx was never touched on disk): LEG 1 (manufacture a divergence between a synthetic page's stated number and the live census) -- OLD logic (built from census.declaredObjects) fails the 'reworded off the page IS a finding' case exactly as the card describes; NEW logic (built from the page's own number) passes both cases, i.e. self-test stays green under the divergence that broke it before. LEG 2 (reword the sentence away for real, in memory) -- ran the actual checkPage() classifier directly against a page with the sentence removed; it still emits [unenforced-prose-missing], proving the pinned case is not a tautology. skip-changeset label applied and read back confirmed present (scripts/** not referenced in any package's published files[] -- measured, not assumed).",
      "mcp_calls": 0,
      "open_questions": [],
      "out_of_scope_findings": [
        "noted, not filed: declaredObjects() over-matches (counts nested snake_case name: literals, e.g. sys_position's actions[] names, not just top-level object declarations) -- this is Route 2's prerequisite per the PM's ruling and changes a published figure. I did not form a confident definition of \"top-level object literal only\" while reading the classifier, so I'm not filing it myself; flagged in the PR body's Open Questions section for the PM/triage to file if warranted. Would land in scripts/tenant-audit-census.mjs; carrier: whoever next touches declaredObjects() or files the follow-up card."
      ]
    }

    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions