diff --git a/docs/qa/platform-checklist/RUNNER.md b/docs/qa/platform-checklist/RUNNER.md index d9a445ce3a..5606c63035 100644 --- a/docs/qa/platform-checklist/RUNNER.md +++ b/docs/qa/platform-checklist/RUNNER.md @@ -110,10 +110,49 @@ selector handed you has a verdict; planned ids are listed, not chased. and cite its output as the evidence; drive the browser only for what the pin doesn't cover. The reverse also holds: when a sweep hand-proves something repeatedly, propose promoting it to a permanent test and set `automated` in a revision. + + **A pin scores only the layer it drives.** Before a pin scores a clause whose oracle + is a public door's answer (an HTTP status, an error envelope, a row the door left + behind), name the layer the pin reaches — a **plugin unit** (one plugin's hooks or + middleware over a stub), the **engine** (a real ObjectQL engine and driver, no HTTP), + or **HTTP** (a booted server answering the door) — and write that layer in the + clause's evidence. Read it off the pin's own harness, never off its file name or its + `automated.ref` prose: a stub terminal standing in for the engine means "lands" is + "the middleware called `next()`", and nothing below it ran. A pin that stops above + the layer the clause's oracle sits at scores that clause **`partial`**, naming the + missing layer, and the live leg is driven, or recorded `blocked` with its fixture + cited. It never scores `pass`. The instance that made this a rule is + `access-security.no-active-org-session-semantics`: the `group` CONTROL of + `no-active-organization-write-refusal.test.ts` "lands" a write by a member with no + active organization at the plugin-security layer, two rounds scored the walled legs + from it while a walled boot was refused, and on a full walled `group` boot the engine + answers that same write with a 500 (#24784). 7. **Verify pass for high-stakes claims.** For P0 `fail`s and any finding that would ship or block a release: a second, independent agent re-derives the verdict from the captured evidence alone (not from the first agent's narrative) before it is acted on. Disagreement → re-run the item. +8. **Keep an off-clause probe budget.** Clauses only ask what someone already thought + to ask. Every run keeps time AFTER its clauses have verdicts for probing the doors its + selector covers past those clauses — on the same boot, as the same personas. Probe + where a clause stops: + - **boundary values** on the inputs the clauses drive: the empty string, a missing + id, an unknown id; + - **the opposite severity or posture** of a path a clause asserts: the `warning` + twin of an `error` rule, `group` beside `isolated`; + - **recipients or readers outside the asserted scope**: who else a notification, + share or read reaches. + + A probe is not a clause, so it scores nothing and no item verdict moves on it. A + finding that passes the filing gate is extracted like any other product defect (see + "Extraction obligation" below), and its card carries + `QA-source: # · · observed beside the clauses`, naming the + item whose doors were probed. A probe that finds a gap the ledger should have asked + about is also a checklist-accuracy finding for the wave's anchor card: the clause it + was missing. The run record states how long the probes took and what they covered + (doors, personas, the boundary classes tried), and says so when the budget was + spent on nothing. A record that omits the probe line reads as a run that skipped it. + Round 3 of the 18.0 sweep (QA run #24779) found #24781 and #24782 this way, beside + items whose clauses all passed. ### Environment facts the runner should not re-derive @@ -717,7 +756,8 @@ The in-environment JSON scratch (RUNNER shape, never committed): The issue body is: env fingerprint · scope (selector + per-item `revision`) · the per-clause verdict table (text oracle evidence) · a reproduction rule per `fail` (rule 2's -carve-out excepted) · derived item verdicts + fixture gaps. The durable, +carve-out excepted) · derived item verdicts + fixture gaps · the off-clause probe line +(rule 8: how long, which doors and boundary classes, what it found). The durable, version-controlled truth is still the checklist under `areas/`; a run is a dated assertion about one build, and it lives in its issue, not the tree. diff --git a/docs/qa/platform-checklist/areas/access-security.json b/docs/qa/platform-checklist/areas/access-security.json index af6e592d32..0802fe4c7a 100644 --- a/docs/qa/platform-checklist/areas/access-security.json +++ b/docs/qa/platform-checklist/areas/access-security.json @@ -2352,7 +2352,7 @@ "title": "A session with no active organization is a LEGAL state with fail-closed semantics: tenant-scoped reads resolve to nothing, tenant-scoped writes are refused loudly naming the missing org, and nothing is ever NULL-stamped", "since": "v17", "status": "active", - "revision": 6, + "revision": 7, "priority": "P0", "surface": "api", "personas": [ @@ -2371,13 +2371,14 @@ "`invite-only` is no longer a fixture gap: the walled boot must declare a membership policy anyway (RUNNER.md 'A walled boot is a fixture of its own', prerequisite 3), and declaring `invite-only` gives knownGaps[0]'s producer (1). SSO-JIT-pending still has no stock fixture: that producer is blocked(fixture) unless the run configures SSO.", "requires[0]'s tenancy-walled deployment is a walled boot. On 18.0.0 os serve — and os dev / os start, which boot through it — boots OS_TENANCY_POSTURE=group|isolated (and the legacy OS_MULTI_ORG_ENABLED=true): 18.x step 4 (#24580) deleted the walled boot refusal (serve.ts walledPostureRefusal) before 18.0.0, and of the published prereleases only 18.0.0-next.5 carries it. The walled boot is provisioned per RUNNER.md's environment-facts entry 'A walled boot is a fixture of its own': the multi-org runtime declared by the app (no example app declares it), OS_PLATFORM_OWNER_EMAIL, and a declared OS_AUTH_MEMBERSHIP_POLICY; each one absent refuses an os dev boot naming itself, while a bare os serve in production mode judges none of them until OS_AUTH_SECRET is set (that entry). The item stays blocked (fixture, #9334) because every clause is a walled leg and the stock showcase provisions no walled boot, not for want of a producer (knownGaps[0]); its pin (automated.ref) boots the posture in-process.", "acceptance[4] has an open question for a user who holds NO membership at all (producer (1) of knownGaps[0]): its login/logout rows in sys_audit_log carry tenant_id null, because no organization exists to stamp and ADR-0123 D3's ordering fix 'must not manufacture a membership that policy says must not exist'. QA run #24783 read those rows only as the platform operator (GET /api/v1/data/sys_audit_log filtered by user_id answered 200), while the user itself gets 403 on sys_audit_log, on both postures. ADR-0123 does not state whether that is the intended answer: D3 forbids a NULL-tenanted row under this state 'in any subsystem' and pays the audit ledger by ordering, which binds nobody under invite-only. The clause stands as written. Until the question is ruled (raised by QA run #24783's finding F3 for wave anchor #24579; no card carries it yet), score acceptance[4] on that user from what is measured (the raw rows, their tenant_id and the readers that see them) and name this question beside the verdict", - "On `group`, producer (2), a member with no active organization, gets 500 {\"code\": \"ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED\"} on a tenant-scoped create, where `isolated` and a member-less caller answer the acceptance[1] 403 naming the missing organization; nothing is written (QA run #24783: acceptance[1], acceptance[5], acceptance[6] and negative[3] FAIL on `group` only). That is the open product card #24784. The clauses stand as written: score them against it until it lands, and score the member-less caller's write separately" + "On `group`, producer (2), a member with no active organization, gets 500 {\"code\": \"ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED\"} on a tenant-scoped create, where `isolated` and a member-less caller answer the acceptance[1] 403 naming the missing organization; nothing is written (QA run #24783: acceptance[1], acceptance[5], acceptance[6] and negative[3] FAIL on `group` only). That is the open product card #24784. The clauses stand as written: score them against it until it lands, and score the member-less caller's write separately", + "acceptance[7] (the walled `group` leg for a member with no active organization) stays red until #24784 lands: on a walled group boot at 580fd4ed (QA run #24783, re-derived under RUNNER rule 7) the create answers 500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED and writes no row. It is scored only on a live walled group boot; a run without one records it blocked(fixture) citing RUNNER.md's 'A walled boot is a fixture of its own' — ⛔ never from automated.ref, whose group CONTROL stops at the plugin-security layer (RUNNER rule 6, a pin scores only the layer it drives). When #24784's fix merges, re-run the clause and drop this line" ] }, "blocked": { "by": "fixture", "ref": "#9334" }, "automated": { "kind": "unit", - "ref": "packages/plugins/plugin-security/src/no-active-organization-write-refusal.test.ts — pins every clause of this item at unit level, INCLUDING a control asserting that the `single` posture has no wall, i.e. the test itself agrees the stock showcase configuration makes the item vacuous" + "ref": "packages/plugins/plugin-security/src/no-active-organization-write-refusal.test.ts — pins every clause of this item at unit level, INCLUDING a control asserting that the `single` posture has no wall, i.e. the test itself agrees the stock showcase configuration makes the item vacuous. The layer it drives is a plugin unit: SecurityPlugin's middleware over a stub terminal that stands in for the engine, so a case that 'lands' there means the middleware called next(), and the engine's own insert path never runs. Its `group` describe's CONTROL 'a NON-EMPTY membership set lands, even with no active organization' therefore proves the plugin-security wall lets that write through and nothing past it: on a full walled group boot the engine refuses the same write with 500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED (#24784). Under RUNNER rule 6 the pin scores a clause whose oracle is the HTTP answer partial at most, naming the engine and HTTP layers as missing, and it cannot score acceptance[7] at all" }, "steps": [ "establish the state and PROVE it: obtain a session whose `activeOrganizationId` is null, and cite the evidence (session row / session endpoint payload) — do not infer the state from the symptom you are about to test", @@ -2387,7 +2388,8 @@ "query the underlying table as admin for rows whose tenant column is NULL, before and after the write attempt", "repeat the read and the write against a NON-tenant-scoped object to bound the blast radius", "sign in and out with the state in force, then inspect the audit ledger rows the session produced (plugin-audit auth-session-audit login/logout)", - "run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial" + "run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial", + "walled `group` leg (acceptance[7]), on a walled boot with OS_TENANCY_POSTURE=group provisioned per RUNNER.md's 'A walled boot is a fixture of its own': a member of organization A (non-empty membership set) sets A active with POST /api/v1/auth/organization/set-active, creates a showcase_account and a showcase_project in A, then clears it with POST /api/v1/auth/organization/set-active {organizationId: null} and proves the state (GET /api/v1/auth/get-session reads session.activeOrganizationId null). As that member, POST /api/v1/data/showcase_task {title, project: the project in A}; take the admin-side count of showcase_task rows whose organization_id is NULL or empty before and after, and read back any row the create produced" ], "acceptance": [ { @@ -2431,6 +2433,12 @@ "oracle": "api", "verify": "the run record states each subsystem's observed behavior side by side; any disagreement is the finding, whichever way it leans", "evidence": "the three-way comparison table" + }, + { + "clause": "walled `group` leg: a member whose membership set is NON-EMPTY and whose session has no active organization never gets a 5xx on a tenant-scoped create, and nothing is stamped with a NULL organization_id. Which answer `group` gives — the write lands in a member organization, or a 4xx refusal — is #24784's to decide; this clause asserts only those two facts. Red until #24784 lands", + "oracle": "api", + "verify": "step-9 create on the live walled group boot: status is NOT 5xx (500 ERR_SYSTEM_WRITE_ORGANIZATION_REQUIRED, logged '[REST] Unhandled error: SystemWriteOrganizationRequiredError', is the #24784 shape); the admin-side NULL-or-empty organization_id count for showcase_task is identical before and after; and if the create answered 2xx, the row it produced carries a real organization_id the member belongs to. Score it only from that live boot: the automated.ref pin's group CONTROL stops above the engine (RUNNER rule 6), so it cannot score this clause", + "evidence": "the get-session payload + the create response, quoted + both NULL counts + the produced row's organization_id when the create landed" } ], "negative": [ @@ -2453,7 +2461,8 @@ "PR #8471 (ADR-0123 + enforcement)", "@objectstack/plugin-security security-plugin.ts step 3.7 + tenant-layer.ts (computeTenantLayer0Filter, RLS_DENY_FILTER)", "@objectstack/plugin-auth auth-manager.ts composeDatabaseHooks + reconcile-membership.ts", - "@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)" + "@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)", + "QA-source: #24783 · access-security.no-active-org-session-semantics · acceptance[1] (the walled group leg; #24784)" ], "history": [ { @@ -2486,6 +2495,12 @@ "date": "2026-10-20", "change": "producers re-measured and acceptance[0] aligned with its governing text — QA run #24783 (finding F3), round 11 of wave anchor #24579. personas[0]–[1], fixtures.requires[1] and knownGaps[0]–[1] now name the two durable producers on a walled boot (a user the operator creates under invite-only, membershipCreated:false; a member that sets its active organization to null). The signup race is closed by ADR-0123 D3's session-seam settle, and the removed-member path revokes or re-points the session (QA run #24783 measured 401). acceptance[0]'s clause and verify scope 'resolve to nothing' to a caller with no organization scope, which ADR-0123 D2 defines under `group` as an empty membership set, ADR-0105 D2 letting a `group` member's reads span its memberships. acceptance[4] is unchanged: ADR-0123 states no answer for a member-less user's NULL-tenanted audit rows, so knownGaps[3] names the question. knownGaps[4] names #24784, the `group` write 500. knownGaps[2]: the prerequisite sentence holds through os dev only (a bare production os serve needs OS_AUTH_SECRET first, RUNNER.md), and the blocked marker's reason is restated (every clause is a walled leg) because a producer is no longer missing; the marker itself is unchanged", "ref": "#24801" + }, + { + "revision": 7, + "date": "2026-10-20", + "change": "acceptance[7], steps[8] and knownGaps[5] added, and automated.ref now names the layer its pin drives — for #24784, which the walled pass of round 3 (QA run #24783, 580fd4ed) found. Rounds 1 and 2 scored the walled legs from the pin because a walled boot was refused then; the pin's group CONTROL 'lands' a write by a member with no active organization at the plugin-security layer, and on a full walled group boot the engine refuses that write with a 500. The new clause asserts only no 5xx and no NULL stamping (which answer group gives is #24784's to decide) and stays red until #24784's fix lands. Merged onto revision 6 (round 11 of wave anchor #24579, #24801), whose knownGaps[0] producer (2) is the one steps[8] drives and whose knownGaps[4] names the same card for acceptance[1], [5], [6] and negative[3]; knownGaps[5] adds only the new clause's red state and the rule that it is never scored from the pin", + "ref": "#24802" } ] }, diff --git a/docs/qa/platform-checklist/areas/api-backend.json b/docs/qa/platform-checklist/areas/api-backend.json index ed165fe637..9da482b858 100644 --- a/docs/qa/platform-checklist/areas/api-backend.json +++ b/docs/qa/platform-checklist/areas/api-backend.json @@ -9,7 +9,7 @@ "title": "transactionalBatch capability bit matches the mounted /batch behavior — atomicity, rollback codes, size cap", "since": "v16", "status": "active", - "revision": 5, + "revision": 6, "priority": "P1", "surface": "api", "personas": [ @@ -20,6 +20,9 @@ "requires": [ "two batch-writable objects: showcase_private_note (title is enough) and showcase_task — a showcase_task create needs its required `project` lookup (a seeded showcase_project id), or it answers 400 VALIDATION_FAILED 'Project is required'", "default maxBatchSize (200 unless RestServerConfig.batch overrides)" + ], + "knownGaps": [ + "acceptance[5] (an empty-string id on an update or delete op) stays red until #24781 lands: the door's up-front id check tests `op.id == null` only (packages/rest/src/batch-routes.ts), so `\"id\": \"\"` reaches the engine and answers 500 — DATABASE_ERROR for update, INTERNAL_ERROR for delete. Measured at 580fd4ed (QA run #24779) and again at a43405d596 (#24802). When #24781's fix merges, re-run the clause and drop this line" ] }, "steps": [ @@ -28,7 +31,8 @@ "POST /api/v1/batch with the same operations and a TOP-LEVEL atomic: false; capture status + error code. ⛔ Spelled as options {atomic: false} the key is stripped, atomic defaults to true, and the batch simply SUCCEEDS — the clause below then reads as a product failure when nothing failed", "POST /api/v1/batch where a MIDDLE operation must fail (e.g. create showcase_invoice missing its required name), sandwiched between two valid creates; capture the per-operation results and re-read all three would-be rows", "POST /api/v1/batch with operations.length > maxBatchSize (201 no-op creates on the default config); capture the rejection", - "POST /api/v1/data/showcase_private_note/batch (the PER-OBJECT batch door) with a mixed set [valid, invalid, valid], NO atomic flag and options {continueOnError: true}; capture the per-row outcomes. Then repeat the same set with NO options at all (continueOnError defaults false) and capture the stop: the row before the failure written, the failing row its own error, the tail NOT_ATTEMPTED" + "POST /api/v1/data/showcase_private_note/batch (the PER-OBJECT batch door) with a mixed set [valid, invalid, valid], NO atomic flag and options {continueOnError: true}; capture the per-row outcomes. Then repeat the same set with NO options at all (continueOnError defaults false) and capture the stop: the row before the failure written, the failing row its own error, the tail NOT_ATTEMPTED", + "empty-id probe: POST /api/v1/batch with ONE op {object: showcase_task, action: update, id: \"\" (the empty string), data: {done: false}} and a TOP-LEVEL atomic: true, then the same with action: delete and id: \"\". Control: each op again with the id key omitted. Capture status + body for all four, and the server log around the empty-id pair" ], "acceptance": [ { @@ -60,10 +64,17 @@ "oracle": "api", "verify": "continueOnError:true response has per-row success and error entries and re-reads confirm BOTH valid rows persisted; the default-options response has row 0 success, row 1 its own validation error, row 2 errors[0].code NOT_ATTEMPTED, and re-reads find row 0 persisted and row 2 absent", "evidence": "both responses + re-reads" + }, + { + "clause": "an update or delete op whose id is the EMPTY STRING is refused up front exactly as a missing id is — 400 VALIDATION_FAILED (\"Operation 'update' on 'showcase_task' requires an id\") — and never answers a 5xx; the per-object doors (POST /data/:object/batch, /updateMany) already treat a blank id as absent. Red until #24781 lands", + "oracle": "api", + "verify": "step-7 responses: both empty-id ops answer status 400 with error code VALIDATION_FAILED, the same status and code as their no-id controls, and the log carries no '[REST] Unhandled error' for them. Any 5xx is a FAIL (the #24781 shape: update 500 DATABASE_ERROR, delete 500 INTERNAL_ERROR, logged '[REST] Unhandled error: Error: Update requires an ID or options.multi=true'). Score status and code per verb, not the sentence", + "evidence": "the four responses (two empty-id, two no-id controls) + the log excerpt around the empty-id pair" } ], "negative": [ - "the two batch doors must not be conflated: atomic:false must NOT be accepted-and-ignored on /api/v1/batch (silent non-atomic acceptance is the pre-ADR-0119 bug shape), and the per-object door must NOT silently roll back valid rows when a sibling row fails without atomic being requested" + "the two batch doors must not be conflated: atomic:false must NOT be accepted-and-ignored on /api/v1/batch (silent non-atomic acceptance is the pre-ADR-0119 bug shape), and the per-object door must NOT silently roll back valid rows when a sibling row fails without atomic being requested", + "a malformed op must never answer a 5xx: an empty-string id answered 500 tells the caller the server broke, a retry policy keyed on 5xx replays it, and the two verbs disagree on the code (#24781) — boundary values (\"\", a missing id, an unknown id) are the door's own validation, never the engine's" ], "traps": [ "dispatcher-vs-hono-route" @@ -73,7 +84,8 @@ "packages/spec/src/api/batch.zod.ts (ADR-0119 D4)", "packages/spec/src/api/error-code-ledger.zod.ts#BATCH_NOT_ATOMIC (BATCH_NOT_ATOMIC, BATCH_TOO_LARGE, NOT_ATTEMPTED, ROLLED_BACK — #4793)", "packages/rest/src/rest-route-ledger.ts (batch family)", - "packages/spec/src/api/discovery.zod.ts#transactionalBatch (transactionalBatch)" + "packages/spec/src/api/discovery.zod.ts#transactionalBatch (transactionalBatch)", + "QA-source: #24779 · api-backend.batch-transactional-discovery · observed beside the clauses (#24781)" ], "history": [ { @@ -105,6 +117,12 @@ "date": "2026-10-16", "change": "the fixture line said showcase_task accepts a baseline create with only a title (#23878 checklist-accuracy finding); its project lookup is required. Re-measured on main c91af8ed6: a task create without project answers 400 VALIDATION_FAILED fields[project] required, alone or inside an atomic /batch, where the whole batch is refused and the sibling note does not persist; with a project the atomic batch answers 200. The fixture line and step 2's operations now carry the project; the rollback clause is unchanged and its re-run passed", "ref": "#24036" + }, + { + "revision": 6, + "date": "2026-10-20", + "change": "acceptance[5], steps[6], negative[1] and knownGaps[0] added for #24781, which QA run #24779 found beside the clauses (round 3 of the 18.0 sweep, 580fd4ed): no clause asked what the cross-object door answers for an empty-string id, and it answers 500 where a missing id answers 400 VALIDATION_FAILED. The clause asserts the missing-id answer and stays red until #24781's fix lands. No existing clause changed", + "ref": "#24802" } ] }, diff --git a/docs/qa/platform-checklist/areas/records-forms.json b/docs/qa/platform-checklist/areas/records-forms.json index e15277f4b1..c9c69fafe8 100644 --- a/docs/qa/platform-checklist/areas/records-forms.json +++ b/docs/qa/platform-checklist/areas/records-forms.json @@ -1912,7 +1912,7 @@ "title": "All six validation-rule types enforce on the write path with their exact per-type error codes — and advisory severities flag without blocking", "since": "v15", "status": "active", - "revision": 4, + "revision": 5, "priority": "P1", "surface": "api", "personas": [ @@ -1922,10 +1922,12 @@ "app": "showcase", "requires": [ "the seeded per-type rules: showcase_account account_lifecycle (state_machine, events [update] only — transitions, no create-time check), tax_id_format + billing_email_format (format), support_config_shape (json_schema), churn_reason_consistency (conditional); showcase_project end_after_start (cross_field), spent_within_budget (script), project_status_flow (state_machine)", - "the seeded ADVISORY rule: showcase_project project_health_progression — state_machine on `health` with severity:'warning' (green↔yellow↔red one step at a time; a jump is flagged, not blocked — examples/app-showcase/src/data/objects/project.object.ts)" + "the seeded ADVISORY rule: showcase_project project_health_progression — state_machine on `health` with severity:'warning' (green↔yellow↔red one step at a time; a jump is flagged, not blocked — examples/app-showcase/src/data/objects/project.object.ts)", + "for the UNEVALUABLE legs (negative[0], negative[1], acceptance[5]): two runtime-authored scratch objects, each carrying one script rule whose condition `int(record.title) > 1` cannot be evaluated over a non-numeric title — no seeded rule is unevaluable, so the stock doors author them. As admin: PUT /api/v1/meta/object/qa_vr_probe {name: qa_vr_probe, label: QA, sharingModel: public_read_write, fields: {title: {type: text}}, validations: [{type: script, name: qa_unevaluable_warn, severity: warning, message: x, condition: \"int(record.title) > 1\"}]} → 200, and the same body as qa_vr_probe_err with the rule named qa_unevaluable_err at severity: error → 200 (the recipe of QA run #24779 / #24782). A write with title \"abc\" then drives each rule" ], "knownGaps": [ - "no showcase rule declares severity:'info' — it shares the identical non-error branch with 'warning' (rule-validator.ts tests `severity === 'error'` and logs everything else), so the warning leg is the driven proof; author a scratch info rule only if a distinct observation is wanted, and record which severities were actually driven" + "no showcase rule declares severity:'info' — it shares the identical non-error branch with 'warning' (rule-validator.ts tests `severity === 'error'` and logs everything else), so the warning leg is the driven proof; author a scratch info rule only if a distinct observation is wanted, and record which severities were actually driven", + "acceptance[5] stays red until #24782 lands: the unevaluable warning rule's write answers 201 and persists (correct), but the server log prints two lines naming the rule that both say \"write rejected\" — the predicate evaluator's fail-closed sentence, emitted before severity is consulted, and the advisory branch's warn, which embeds that violation message (packages/objectql/src/validation/rule-validator.ts). Measured at 580fd4ed (QA run #24779) and again at a43405d596 (#24802). When #24782's fix merges, re-run the clause and drop this line" ] }, "variants": [ @@ -1943,7 +1945,8 @@ "for each variant: send the happy-path twin (same shape, satisfying values) and capture success", "after every rejection, re-read the row set to confirm nothing persisted", "for state_machine additionally: create with a legal initial state, walk one legal transition, then attempt the illegal one", - "severity probe: PATCH a showcase_project's `health` from green straight to red (skipping yellow — a jump project_health_progression matches); capture the response, re-read the row, and read the server log for the warning line" + "severity probe: PATCH a showcase_project's `health` from green straight to red (skipping yellow — a jump project_health_progression matches); capture the response, re-read the row, and read the server log for the warning line", + "unevaluable probe (the scratch objects of fixtures.requires[2]): POST /api/v1/data/qa_vr_probe {title: \"abc\"} and capture the response, re-read the row, and capture EVERY server-log line naming qa_unevaluable_warn; then POST /api/v1/data/qa_vr_probe_err {title: \"abc\"} and capture the refusal and a re-read showing no row" ], "acceptance": [ { @@ -1975,12 +1978,19 @@ "oracle": "log", "verify": "the 2xx response with its `warnings` entry naming the rule (the write answer reports the write's own advisory hits — packages/metadata-protocol/src/protocol.ts, #22726), the persisted re-read, and the logger.warn line (packages/objectql/src/validation/rule-validator.ts — non-error severities route to opts.logger.warn, never into `errors`). Score the entry's rule, severity, field and code (packages/objectql/src/validation/rule-validator.ts#advisoryEvent copies field and code from the violation, so a state_machine advisory reports invalid_transition), not its message wording; a run wanting a UI-visible advisory is asking for a feature, not verifying this one", "evidence": "response + re-read + the captured log line" + }, + { + "clause": "an UNEVALUABLE severity:'warning' rule lets the write land, and no log line beside it reports a rejection: the write answers 2xx, the row persists, and no server-log line naming the rule says the write was rejected or that the rule failed closed — the log must not state the opposite of what the server did. Red until #24782 lands", + "oracle": "log", + "verify": "step-7 qa_vr_probe write: status 201 and a re-read finding the row; then every log entry naming qa_unevaluable_warn, read to its end — the predicate evaluator's entry spans lines (it embeds the CEL source with a caret) and its verdict sits after the caret — none may contain \"write rejected\" or \"fails closed\" (the fail-closed sentence belongs to the error-severity path; rule-validator.ts header: an unevaluable warning/info rule is logged, never thrown). Score the absence of the rejection wording, not whichever replacement sentence #24782 picks; a line saying the rule could not be evaluated is expected and passes. Control: the qa_vr_probe_err write answers 400 VALIDATION_FAILED naming qa_unevaluable_err and persists nothing — there the rejection wording is true", + "evidence": "both responses + both re-reads + every log line naming either rule, quoted" } ], "negative": [ - "an unevaluable CEL expression must fail CLOSED (rule_violation), never fail-open silently accepting the write — the rule-validator's documented posture", + "an unevaluable CEL expression must fail CLOSED (rule_violation), never fail-open silently accepting the write — the rule-validator's documented posture; driven on the qa_vr_probe_err scratch object (fixtures.requires[2], steps[6])", "the fail-closed posture is SEVERITY-SCOPED: an unevaluable 'warning'/'info' rule stays non-blocking (logged, never thrown — rule-validator.ts 'advisory rules stay advisory'); a warning rule that BLOCKS a write, evaluable or not, is a FAIL in the other direction", - "a matching warning rule that leaves NO log line is a FAIL of the surfacing half — flagged-not-blocked means flagged" + "a matching warning rule that leaves NO log line is a FAIL of the surfacing half — flagged-not-blocked means flagged", + "an unevaluable advisory rule whose log line says the write was REJECTED while the write landed is a FAIL of the surfacing half in the other direction: an operator reading the log for a disputed write is told it was refused when it was stored (#24782)" ], "traps": [ "wrong-persona" @@ -1989,7 +1999,8 @@ "packages/spec/src/data/validation.zod.ts#ValidationRuleSchema (ValidationRuleSchema, 6 discriminated variants; severity enum ['error','warning','info'] default 'error')", "packages/objectql/src/validation/rule-validator.ts#evaluateRule (evaluateRule switch + per-type codes; 'only error blocks', unevaluable-advisory stays non-blocking, the severity fork)", "examples/app-showcase/src/data/objects/{account,project,task}.object.ts (the seeded rules; project.object.ts the advisory project_health_progression)", - "#1475 (declared ≠ enforced history: 9 declared → 6 declared+enforced)" + "#1475 (declared ≠ enforced history: 9 declared → 6 declared+enforced)", + "QA-source: #24779 · records-forms.validation-rule-type-matrix · negative[1] (observed beside the clauses — the write did not block, the log beside it said it did; #24782)" ], "history": [ { @@ -2015,6 +2026,12 @@ "date": "2026-10-19", "change": "variants[0] and fixtures.requires[0] corrected from QA run #24572 (18.0 sweep, 5b14b7f3), re-read at objectstack f89d29e382: account_lifecycle binds `events: ['update']` only (examples/app-showcase/src/data/objects/account.object.ts), so the create-with-non-initial-status leg raises invalid_initial_state from project_status_flow alone (events insert + update, initialStates ['planned'], project.object.ts). The since:v18 run #24558's note that this item describes the advisory hit as server-log only is already on main (revision 3 reads the write answer's warnings) and is not repeated. No clause changed. Wave anchor #24579 §3", "ref": "#24671" + }, + { + "revision": 5, + "date": "2026-10-20", + "change": "acceptance[5], steps[6], negative[3], fixtures.requires[2] and knownGaps[1] added for #24782, which QA run #24779 found beside the clauses (round 3 of the 18.0 sweep, 580fd4ed): an unevaluable warning rule lets the write land, as negative[1] requires, but its log says the write was rejected, and no clause read the log of the unevaluable path. The clause stays red until #24782's fix lands. fixtures.requires[2] is round 3's recipe — a scratch object authored through PUT /api/v1/meta/object carries an unevaluable script rule — and closes round 2's fixture gap (QA run #24616: negative[0] blocked(fixture), 'an unevaluable CEL validation cannot be authored through any door'); negative[0] now names its scratch object. No existing clause changed", + "ref": "#24802" } ] },