Skip to content

Commit 87c09eb

Browse files
qa(checklist): clauses for three round-3 defects outside the ledger, plus two RUNNER rules — a pin scores only the layer it drives, and an off-clause probe budget (#24811)
Fixes #24802 Clause-②: no — the internal QA ledger and its runner protocol; nothing published ## Summary Three ledger clauses for the defects round 3 of the 18.0 sweep found outside the ledger, and two RUNNER rules. Each new clause carries a `QA-source:` citation, a `revision` bump, a `history` row, and a `knownGaps` line naming its card. Each stays red until that card's fix lands. - **`api-backend.batch-transactional-discovery` rev 6 (#24781).** New acceptance[5]: an `update` or `delete` op on `POST /api/v1/batch` whose id is the empty string answers 400 `VALIDATION_FAILED`, the same as a missing id, and never a 5xx. Adds steps[6] (the empty-id pair plus their no-id controls) and negative[1]. Source: QA run #24779, observed beside the clauses. - **`records-forms.validation-rule-type-matrix` rev 5 (#24782).** New acceptance[5] (log oracle): an unevaluable `severity: 'warning'` rule lets the write land, and no log entry naming the rule says the write was rejected or failed closed. fixtures.requires[2] carries round 3's recipe: two scratch objects authored through `PUT /api/v1/meta/object`, each with a script rule `int(record.title) > 1`, one at `warning` and one at `error`. That closes round 2's fixture gap (QA run #24616: negative[0] blocked(fixture), "an unevaluable CEL validation cannot be authored through any door"). negative[0] now names its scratch object. Adds steps[6] and negative[3]. - **`access-security.no-active-org-session-semantics` rev 6 (#24784).** New acceptance[7], the walled `group` leg: a member whose membership set is non-empty and whose session has no active organization never gets a 5xx on a tenant-scoped create, and nothing is stamped with a NULL `organization_id`. Whether `group` lands that write or refuses it 4xx is left to #24784. Adds steps[8] (the set-active-null producer) and knownGaps[3]. `automated.ref` now names the layer its pin drives. `no-active-organization-write-refusal.test.ts` runs SecurityPlugin's middleware over a stub terminal that stands in for the engine, so its `group` CONTROL "lands" at the plugin-security layer only. The pin therefore cannot score acceptance[7]. - **RUNNER rule 6, new paragraph: a pin scores only the layer it drives.** Before a pin scores a clause whose oracle is a public door's answer, the runner names the layer the pin reaches (plugin unit, engine, or HTTP), read off the pin's harness. A pin that stops above the oracle's layer scores the clause `partial` and names the missing layer. #24784 is the worked instance. The paragraph sits beside rule 6, not inside the walled-boot entry. - **RUNNER rule 8: keep an off-clause probe budget.** Each run keeps time after its clauses for probing the selector's doors past them: boundary values, the opposite severity or posture, and recipients outside the asserted scope. A probe scores nothing. A finding that passes the filing gate is extracted with `QA-source: … · observed beside the clauses`. The run-record body contract gains the probe line: how long the probes took, what they covered, and what they found. **Not in this PR: the `records-forms.record-discussion-mentions` clause for #24785.** Its fix PR #24794 is still an open draft (checked just before this PR was opened). Under RUNNER rule 2 that clause would publish the reproduction before the fix ships, so it waits until #24794 merges. Out of scope, and not touched here: the round-3 re-points and accuracy findings in #24579 comment 6284472874. These include F3, the `group` wording of this item's acceptance[0] and [1] and its knownGaps[0] and [1] producers. They belong to the anchor lane. ## Verification The clauses were run on a stock showcase at `a43405d596`, this branch's base and current `origin/main`. The build was `pnpm build --concurrency=2` (76/76). The boot was `objectstack dev --seed-admin`, posture `single`, on a fresh DB. Server stopped by process group afterwards. | clause | expected now (card open) | measured | |---|---|---| | batch acceptance[5], update with id empty string | red | 500 `DATABASE_ERROR` "Internal data error"; no-id control 400 `VALIDATION_FAILED` "Operation 'update' on 'showcase_task' requires an id" | | batch acceptance[5], delete with id empty string | red | 500 `INTERNAL_ERROR`; no-id control 400 `VALIDATION_FAILED`. Log: `[REST] Unhandled error: Error: Update requires an ID or options.multi=true` (and the Delete twin) | | validation acceptance[5], warning rule | red | 201, row persisted (re-read 1 row). The log prints two entries naming `qa_unevaluable_warn`, and both say "write rejected". The first ends "a rule that cannot be evaluated fails closed, it is never skipped" after a multi-line CEL caret excerpt, so the verify text now says to read each entry to its end | | validation negative[0], error twin | green | 400 `VALIDATION_FAILED`, fields[0] `{field: _record, code: rule_violation}`, re-read 0 rows | | no-active-org acceptance[7], walled group | red | not measured here: it needs a walled boot (scratch app declaring `@objectstack/organizations`, owner email, membership policy). Its red rests on QA run #24783 and the RUNNER rule 7 re-derivation in #24784. Per the dispatch, written from knownGaps, not hand-tested | The red and green states match the cards: #24781, #24782 and #24784 are open. ## Gates On head `1143504b`: - `node scripts/check-platform-checklist.mjs` → exit 0. Output: `check-platform-checklist: OK — 16 areas, 328 items (323 active, 1 planned)`. It reports 0 line citations surviving and 943/953 symbol anchors resolved, the same counts as on base. - `node scripts/pm/dispatch-gates.mjs --commands` derived 13 commands, each run and each exit 0: - `node scripts/check-ci-filter-parity.mjs` 0 - `node scripts/check-closing-keyword-parity.mjs` 0 - `node scripts/check-closing-keyword-parity.mjs --self-test` 0 - `node scripts/check-comment-mask-corpus.mjs` 0 - `pnpm --filter @objectstack/lint run check:doc-formula-expressions` 0 (ran: the log ends with the checker's own summary) - `pnpm check:cross-package-test-inputs` 0 - `pnpm check:doc-authoring` 0 - `pnpm check:driver-memory-census` 0 - `pnpm check:gitlink-declared` 0 - `pnpm check:nul-bytes` 0 - `pnpm check:platform-checklist` 0 - `pnpm check:refd-timer-probe` 0 - `pnpm check:watch-hint-literal` 0 - `node scripts/pm/dispatch-gates.mjs --ran` → exit 0: `13 derived famil(ies) accounted for — 13 run, 0 NOT-MEASURED`. - `coverage.json` untouched: the checker required no change. ## Acceptance notes - Steps are referenced 1-based in `verify` text, as the existing api-backend clauses do ("step-7"). History rows use 0-based array indices ("steps[6]"). - The batch and validation knownGaps lines record the `580fd4ed` measurement and this PR's re-measurement at `a43405d596`. Each tells the runner to re-run the clause and drop the line once its card's fix merges. --- _Generated by [Claude Code](https://claude.ai/code/session_01MRkuLGosywLTY6EMcP6fGK)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent ddb3360 commit 87c09eb

4 files changed

Lines changed: 107 additions & 17 deletions

File tree

‎docs/qa/platform-checklist/RUNNER.md‎

Lines changed: 41 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,10 +110,49 @@ selector handed you has a verdict; planned ids are listed, not chased.
110110
and cite its output as the evidence; drive the browser only for what the pin doesn't
111111
cover. The reverse also holds: when a sweep hand-proves something repeatedly,
112112
propose promoting it to a permanent test and set `automated` in a revision.
113+
114+
**A pin scores only the layer it drives.** Before a pin scores a clause whose oracle
115+
is a public door's answer (an HTTP status, an error envelope, a row the door left
116+
behind), name the layer the pin reaches — a **plugin unit** (one plugin's hooks or
117+
middleware over a stub), the **engine** (a real ObjectQL engine and driver, no HTTP),
118+
or **HTTP** (a booted server answering the door) — and write that layer in the
119+
clause's evidence. Read it off the pin's own harness, never off its file name or its
120+
`automated.ref` prose: a stub terminal standing in for the engine means "lands" is
121+
"the middleware called `next()`", and nothing below it ran. A pin that stops above
122+
the layer the clause's oracle sits at scores that clause **`partial`**, naming the
123+
missing layer, and the live leg is driven, or recorded `blocked` with its fixture
124+
cited. It never scores `pass`. The instance that made this a rule is
125+
`access-security.no-active-org-session-semantics`: the `group` CONTROL of
126+
`no-active-organization-write-refusal.test.ts` "lands" a write by a member with no
127+
active organization at the plugin-security layer, two rounds scored the walled legs
128+
from it while a walled boot was refused, and on a full walled `group` boot the engine
129+
answers that same write with a 500 (#24784).
113130
7. **Verify pass for high-stakes claims.** For P0 `fail`s and any finding that would
114131
ship or block a release: a second, independent agent re-derives the verdict from the
115132
captured evidence alone (not from the first agent's narrative) before it is acted
116133
on. Disagreement → re-run the item.
134+
8. **Keep an off-clause probe budget.** Clauses only ask what someone already thought
135+
to ask. Every run keeps time AFTER its clauses have verdicts for probing the doors its
136+
selector covers past those clauses — on the same boot, as the same personas. Probe
137+
where a clause stops:
138+
- **boundary values** on the inputs the clauses drive: the empty string, a missing
139+
id, an unknown id;
140+
- **the opposite severity or posture** of a path a clause asserts: the `warning`
141+
twin of an `error` rule, `group` beside `isolated`;
142+
- **recipients or readers outside the asserted scope**: who else a notification,
143+
share or read reaches.
144+
145+
A probe is not a clause, so it scores nothing and no item verdict moves on it. A
146+
finding that passes the filing gate is extracted like any other product defect (see
147+
"Extraction obligation" below), and its card carries
148+
`QA-source: #<run record> · <area.item> · observed beside the clauses`, naming the
149+
item whose doors were probed. A probe that finds a gap the ledger should have asked
150+
about is also a checklist-accuracy finding for the wave's anchor card: the clause it
151+
was missing. The run record states how long the probes took and what they covered
152+
(doors, personas, the boundary classes tried), and says so when the budget was
153+
spent on nothing. A record that omits the probe line reads as a run that skipped it.
154+
Round 3 of the 18.0 sweep (QA run #24779) found #24781 and #24782 this way, beside
155+
items whose clauses all passed.
117156

118157
### Environment facts the runner should not re-derive
119158

@@ -717,7 +756,8 @@ The in-environment JSON scratch (RUNNER shape, never committed):
717756

718757
The issue body is: env fingerprint · scope (selector + per-item `revision`) · the
719758
per-clause verdict table (text oracle evidence) · a reproduction rule per `fail` (rule 2's
720-
carve-out excepted) · derived item verdicts + fixture gaps. The durable,
759+
carve-out excepted) · derived item verdicts + fixture gaps · the off-clause probe line
760+
(rule 8: how long, which doors and boundary classes, what it found). The durable,
721761
version-controlled truth is still the checklist under `areas/`; a run is a dated assertion
722762
about one build, and it lives in its issue, not the tree.
723763

‎docs/qa/platform-checklist/areas/access-security.json‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -2352,7 +2352,7 @@
23522352
"title": "A session with no active organization is a LEGAL state with fail-closed semantics: tenant-scoped reads resolve to nothing, tenant-scoped writes are refused loudly naming the missing org, and nothing is ever NULL-stamped",
23532353
"since": "v17",
23542354
"status": "active",
2355-
"revision": 6,
2355+
"revision": 7,
23562356
"priority": "P0",
23572357
"surface": "api",
23582358
"personas": [
@@ -2371,13 +2371,14 @@
23712371
"`invite-only` is no longer a fixture gap: the walled boot must declare a membership policy anyway (RUNNER.md 'A walled boot is a fixture of its own', prerequisite 3), and declaring `invite-only` gives knownGaps[0]'s producer (1). SSO-JIT-pending still has no stock fixture: that producer is blocked(fixture) unless the run configures SSO.",
23722372
"requires[0]'s tenancy-walled deployment is a walled boot. On 18.0.0 os serve — and os dev / os start, which boot through it — boots OS_TENANCY_POSTURE=group|isolated (and the legacy OS_MULTI_ORG_ENABLED=true): 18.x step 4 (#24580) deleted the walled boot refusal (serve.ts walledPostureRefusal) before 18.0.0, and of the published prereleases only 18.0.0-next.5 carries it. The walled boot is provisioned per RUNNER.md's environment-facts entry 'A walled boot is a fixture of its own': the multi-org runtime declared by the app (no example app declares it), OS_PLATFORM_OWNER_EMAIL, and a declared OS_AUTH_MEMBERSHIP_POLICY; each one absent refuses an os dev boot naming itself, while a bare os serve in production mode judges none of them until OS_AUTH_SECRET is set (that entry). The item stays blocked (fixture, #9334) because every clause is a walled leg and the stock showcase provisions no walled boot, not for want of a producer (knownGaps[0]); its pin (automated.ref) boots the posture in-process.",
23732373
"acceptance[4] has an open question for a user who holds NO membership at all (producer (1) of knownGaps[0]): its login/logout rows in sys_audit_log carry tenant_id null, because no organization exists to stamp and ADR-0123 D3's ordering fix 'must not manufacture a membership that policy says must not exist'. QA run #24783 read those rows only as the platform operator (GET /api/v1/data/sys_audit_log filtered by user_id answered 200), while the user itself gets 403 on sys_audit_log, on both postures. ADR-0123 does not state whether that is the intended answer: D3 forbids a NULL-tenanted row under this state 'in any subsystem' and pays the audit ledger by ordering, which binds nobody under invite-only. The clause stands as written. Until the question is ruled (raised by QA run #24783's finding F3 for wave anchor #24579; no card carries it yet), score acceptance[4] on that user from what is measured (the raw rows, their tenant_id and the readers that see them) and name this question beside the verdict",
2374-
"On `group`, producer (2), a member with no active organization, gets 500 {\"code\": \"ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED\"} on a tenant-scoped create, where `isolated` and a member-less caller answer the acceptance[1] 403 naming the missing organization; nothing is written (QA run #24783: acceptance[1], acceptance[5], acceptance[6] and negative[3] FAIL on `group` only). That is the open product card #24784. The clauses stand as written: score them against it until it lands, and score the member-less caller's write separately"
2374+
"On `group`, producer (2), a member with no active organization, gets 500 {\"code\": \"ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED\"} on a tenant-scoped create, where `isolated` and a member-less caller answer the acceptance[1] 403 naming the missing organization; nothing is written (QA run #24783: acceptance[1], acceptance[5], acceptance[6] and negative[3] FAIL on `group` only). That is the open product card #24784. The clauses stand as written: score them against it until it lands, and score the member-less caller's write separately",
2375+
"acceptance[7] (the walled `group` leg for a member with no active organization) stays red until #24784 lands: on a walled group boot at 580fd4ed (QA run #24783, re-derived under RUNNER rule 7) the create answers 500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED and writes no row. It is scored only on a live walled group boot; a run without one records it blocked(fixture) citing RUNNER.md's 'A walled boot is a fixture of its own' — ⛔ never from automated.ref, whose group CONTROL stops at the plugin-security layer (RUNNER rule 6, a pin scores only the layer it drives). When #24784's fix merges, re-run the clause and drop this line"
23752376
]
23762377
},
23772378
"blocked": { "by": "fixture", "ref": "#9334" },
23782379
"automated": {
23792380
"kind": "unit",
2380-
"ref": "packages/plugins/plugin-security/src/no-active-organization-write-refusal.test.ts — pins every clause of this item at unit level, INCLUDING a control asserting that the `single` posture has no wall, i.e. the test itself agrees the stock showcase configuration makes the item vacuous"
2381+
"ref": "packages/plugins/plugin-security/src/no-active-organization-write-refusal.test.ts — pins every clause of this item at unit level, INCLUDING a control asserting that the `single` posture has no wall, i.e. the test itself agrees the stock showcase configuration makes the item vacuous. The layer it drives is a plugin unit: SecurityPlugin's middleware over a stub terminal that stands in for the engine, so a case that 'lands' there means the middleware called next(), and the engine's own insert path never runs. Its `group` describe's CONTROL 'a NON-EMPTY membership set lands, even with no active organization' therefore proves the plugin-security wall lets that write through and nothing past it: on a full walled group boot the engine refuses the same write with 500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED (#24784). Under RUNNER rule 6 the pin scores a clause whose oracle is the HTTP answer partial at most, naming the engine and HTTP layers as missing, and it cannot score acceptance[7] at all"
23812382
},
23822383
"steps": [
23832384
"establish the state and PROVE it: obtain a session whose `activeOrganizationId` is null, and cite the evidence (session row / session endpoint payload) — do not infer the state from the symptom you are about to test",
@@ -2387,7 +2388,8 @@
23872388
"query the underlying table as admin for rows whose tenant column is NULL, before and after the write attempt",
23882389
"repeat the read and the write against a NON-tenant-scoped object to bound the blast radius",
23892390
"sign in and out with the state in force, then inspect the audit ledger rows the session produced (plugin-audit auth-session-audit login/logout)",
2390-
"run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial"
2391+
"run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial",
2392+
"walled `group` leg (acceptance[7]), on a walled boot with OS_TENANCY_POSTURE=group provisioned per RUNNER.md's 'A walled boot is a fixture of its own': a member of organization A (non-empty membership set) sets A active with POST /api/v1/auth/organization/set-active, creates a showcase_account and a showcase_project in A, then clears it with POST /api/v1/auth/organization/set-active {organizationId: null} and proves the state (GET /api/v1/auth/get-session reads session.activeOrganizationId null). As that member, POST /api/v1/data/showcase_task {title, project: the project in A}; take the admin-side count of showcase_task rows whose organization_id is NULL or empty before and after, and read back any row the create produced"
23912393
],
23922394
"acceptance": [
23932395
{
@@ -2431,6 +2433,12 @@
24312433
"oracle": "api",
24322434
"verify": "the run record states each subsystem's observed behavior side by side; any disagreement is the finding, whichever way it leans",
24332435
"evidence": "the three-way comparison table"
2436+
},
2437+
{
2438+
"clause": "walled `group` leg: a member whose membership set is NON-EMPTY and whose session has no active organization never gets a 5xx on a tenant-scoped create, and nothing is stamped with a NULL organization_id. Which answer `group` gives — the write lands in a member organization, or a 4xx refusal — is #24784's to decide; this clause asserts only those two facts. Red until #24784 lands",
2439+
"oracle": "api",
2440+
"verify": "step-9 create on the live walled group boot: status is NOT 5xx (500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED, logged '[REST] Unhandled error: SystemWriteOrganizationRequiredError', is the #24784 shape); the admin-side NULL-or-empty organization_id count for showcase_task is identical before and after; and if the create answered 2xx, the row it produced carries a real organization_id the member belongs to. Score it only from that live boot: the automated.ref pin's group CONTROL stops above the engine (RUNNER rule 6), so it cannot score this clause",
2441+
"evidence": "the get-session payload + the create response, quoted + both NULL counts + the produced row's organization_id when the create landed"
24342442
}
24352443
],
24362444
"negative": [
@@ -2453,7 +2461,8 @@
24532461
"PR #8471 (ADR-0123 + enforcement)",
24542462
"@objectstack/plugin-security security-plugin.ts step 3.7 + tenant-layer.ts (computeTenantLayer0Filter, RLS_DENY_FILTER)",
24552463
"@objectstack/plugin-auth auth-manager.ts composeDatabaseHooks + reconcile-membership.ts",
2456-
"@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)"
2464+
"@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)",
2465+
"QA-source: #24783 · access-security.no-active-org-session-semantics · acceptance[1] (the walled group leg; #24784)"
24572466
],
24582467
"history": [
24592468
{
@@ -2486,6 +2495,12 @@
24862495
"date": "2026-10-20",
24872496
"change": "producers re-measured and acceptance[0] aligned with its governing text — QA run #24783 (finding F3), round 11 of wave anchor #24579. personas[0]–[1], fixtures.requires[1] and knownGaps[0]–[1] now name the two durable producers on a walled boot (a user the operator creates under invite-only, membershipCreated:false; a member that sets its active organization to null). The signup race is closed by ADR-0123 D3's session-seam settle, and the removed-member path revokes or re-points the session (QA run #24783 measured 401). acceptance[0]'s clause and verify scope 'resolve to nothing' to a caller with no organization scope, which ADR-0123 D2 defines under `group` as an empty membership set, ADR-0105 D2 letting a `group` member's reads span its memberships. acceptance[4] is unchanged: ADR-0123 states no answer for a member-less user's NULL-tenanted audit rows, so knownGaps[3] names the question. knownGaps[4] names #24784, the `group` write 500. knownGaps[2]: the prerequisite sentence holds through os dev only (a bare production os serve needs OS_AUTH_SECRET first, RUNNER.md), and the blocked marker's reason is restated (every clause is a walled leg) because a producer is no longer missing; the marker itself is unchanged",
24882497
"ref": "#24801"
2498+
},
2499+
{
2500+
"revision": 7,
2501+
"date": "2026-10-20",
2502+
"change": "acceptance[7], steps[8] and knownGaps[5] added, and automated.ref now names the layer its pin drives — for #24784, which the walled pass of round 3 (QA run #24783, 580fd4ed) found. Rounds 1 and 2 scored the walled legs from the pin because a walled boot was refused then; the pin's group CONTROL 'lands' a write by a member with no active organization at the plugin-security layer, and on a full walled group boot the engine refuses that write with a 500. The new clause asserts only no 5xx and no NULL stamping (which answer group gives is #24784's to decide) and stays red until #24784's fix lands. Merged onto revision 6 (round 11 of wave anchor #24579, #24801), whose knownGaps[0] producer (2) is the one steps[8] drives and whose knownGaps[4] names the same card for acceptance[1], [5], [6] and negative[3]; knownGaps[5] adds only the new clause's red state and the rule that it is never scored from the pin",
2503+
"ref": "#24802"
24892504
}
24902505
]
24912506
},

0 commit comments

Comments
 (0)