feat(metadata-protocol): publishPackageDrafts takes an opt-in backfill of new fields' literal defaults, written as the caller (#24269)

Fixes #24253

Clause-②: yes (widening) (a request key on a published protocol method
and its door, and a response key in `packages/spec/src/**`; owes one
`CONTRACT_REVIEW_TIER` review before the queue)

Dispatched by the `domain:engine` seat 1 PM
(`session_01JtSQQ3EWsbNPzyYMug1tXy`); implemented by the os-dev session
`session_01DtBADZFvLxnZCFFDWCy87D` on the claim in comment 6245957326.

## What changes

`publishPackageDrafts` takes an opt-in `backfill: Array of { object,
field }`. Each entry names a field this publish adds. Once the drafts
are live, that field's literal `defaultValue` is written into the
object's existing rows whose value is absent. The writes run as the
caller, and the response reports the rows written per field under
`backfillApplied`.

- **`packages/metadata-protocol/src/protocol.ts`**
(`publishPackageDrafts` and two private helpers beside it):
- `readPublishBackfillRequest` judges the request's shape before any
server state is read. The shape must be an array of `{ object, field }`
pairs, each named once, with no other keys. A non-empty backfill needs
the caller's `context`.
- `planPublishBackfill` runs after `listDrafts` and before anything is
promoted. An entry is honoured only when all four hold: the package
holds an `object` draft of that name; the draft declares the field; the
object the engine serves today does NOT have the field; and the field
declares a literal `defaultValue`.
- A runtime token (`NOW()`, `current_user`) or an expression envelope is
resolved for each insert against that insert's context. There is no
single value to stamp into rows that other people wrote at other times,
so those defaults are refused too.
- All failing entries are returned in ONE `400 VALIDATION_ERROR`, with
one `issues[]` entry per problem at path `backfill.N`. Nothing is
promoted and no row is written.
- `applyPublishBackfill` runs after every promoted item's side effects.
For each field, it reads `{ field: { $null: true } }` 200 rows at a time
as the caller, then writes that page through `updateManyData` with the
caller's context. `updateManyData` is the bulk face of the by-id PATCH,
so permissions, RLS/FLS, validation and record hooks run per row, as on
that caller's own edit.
- **`packages/runtime/src/domains/packages.ts`**: the `POST
/packages/:id/publish-drafts` door passes `body.backfill` through as
sent, together with `_context.executionContext` (the request's own
principal, never a system one). A body without the key sends neither
key.
- **`packages/client/src/index.ts`**: `packages.publishDrafts(id, {
backfill })`.
- **`packages/spec/src/api/protocol.zod.ts`**:
`PublishPackageDraftsResponseSchema.backfillApplied` declares `{
success, fields: [{ object, field, updated, error?, code? }] }`. Without
it, the declared parse would strip the key. The generated protocol
reference, the authorable-surface shard and the strictness-ledger count
were regenerated by `check:generated --fix` (442 → 444 response
`z.object` sites).
- Four `minor` changesets: `@objectstack/metadata-protocol`,
`@objectstack/runtime`, `@objectstack/client` and `@objectstack/spec`.
`@objectstack/objectql` is touched by tests only, and its `files[]`
ships `dist/` alone.

### Where the write lives, and why (H2 measured)

The dispatch's H2 hypothesis was that only the door holds the caller's
context, so the door would have to perform the writes. Measurement on
`fb6fb24` falsified the premise behind it.

The protocol's own data methods already take the caller's
`ExecutionContext` from every door: `callData` hands
`_context.executionContext` to `protocol.updateData` / `updateManyData`
/ `findData`. plugin-security's CRUD/RLS/FLS middleware and the record
hooks read it there.

So the write lives in the protocol, and the door passes the request's
context with the key. This keeps one implementation for every caller of
the method. A caller without a context is refused, and the write is
never performed as the system. The door half is pinned (it sends
`_context.executionContext`, and `isSystem` is unset).

## Premise measurement: the SQL driver already fills existing rows

Measured on a live `pnpm dev:crm -- --fresh` backend (SQLite,
`SqlDriver(better-sqlite3)`) at base `fb6fb24`, before any change:

- A package was created, an object published, and 12 rows inserted.
- A draft then added `done: boolean, defaultValue: false` and was
published with no backfill.
- Result: all 12 rows read `false`, and `?done=false` matched 12.
- `sqlite_master` showed `done boolean default '0'`, and the stored
values were `integer 0` × 12.

The cause is that `ALTER TABLE … ADD COLUMN` carries the literal default
(`applyDeclaredColumnDefault`), and the database fills existing rows
with it.

The gap is real on the drivers without a column DEFAULT:
- Turso's remote transport: `columnDeclarationSQL` emits no DEFAULT, and
its own docblock records the asymmetry.
- The memory and MongoDB drivers, which are schemaless.

Cloud's case (cloud#2656 / #1586) is the first of these. So on the SQL
driver `backfillApplied.fields[].updated` is a truthful `0`, and the
spec describes that.

The card's Done-when is therefore pinned on a real ObjectQL engine over
a store without column defaults. On a SQL backend it would pass
vacuously.

## Mechanism hypotheses, measured

- **H1, the order: holds.** The backfill runs after
`runPublishSideEffects` has run for every promoted item. That step does
two things: `applyRegistryWriteThrough`, without which the engine would
serve no new field and strip the write; and `ensureObjectStorage` →
`syncObjectSchema`, which adds the column. A sync refused there is
logged at `error` and the publish continues. The backfill's write then
fails and is reported per field. It is never skipped.
- **H2, as the caller: the premise is falsified (see above).** The
caller's context does reach the protocol. Pinned: the CRUD middleware
seam and a `beforeUpdate` hook each see `userId` = the caller, with
`isSystem` not set, once per row.
- **H3, what may be named: holds, plus one case.** "New" means a field
absent from the object the engine serves
(`registry.getObject(object).fields`, injected system fields included)
and present in this batch's draft. Unknown object, unknown field,
not-new, no default, and a token or expression default are each refused
by path, all in one answer, before promotion. The extra case is the
token/expression refusal, decided by the axes below.
- **H4, which rows: holds.** Only rows whose value is absent are
written, a page of 200 at a time. A value already stored is neither
overwritten nor counted (pinned: 11 of 12). With 205 rows, the reads are
two full pages plus a final empty one.
- A row a hook leaves empty stops the field instead of looping on it. So
does a write whose field the engine dropped (`droppedFields`).
- A residual window remains between a page's read and its by-id write. A
value a user writes to that row inside that window would be overwritten.
`updateData` writes by id only; no conditional by-id update exists on
this face.
- **H5, a failure midway: settled as loud and per field.** The promotion
has committed and is not rolled back. A field stops at its first failing
row, and its element carries `updated` (the rows written so far),
`error` (the record and why) and `code` (the refusal's ADR-0112 code,
e.g. `PERMISSION_DENIED`). Other fields still run, and
`backfillApplied.success` is false. The failure reaches the caller in
the 200 body, so by AGENTS.md "Degradation log levels" it is not a
degradation, and the server log carries a `warn` line. Pinned in the
test suite.

## Verification record (head `08c87b8d0`, after merging `origin/main`
`a422f04fe`)

Tests:
- New `packages/objectql/src/publish-package-drafts-backfill.test.ts`
(10 cases, real `ObjectStackProtocolImplementation` on a real `ObjectQL`
engine):
- The card case: a boolean with `defaultValue: false` over 12 rows reads
12 `false`, and the receipt is `updated: 12`.
  - No key, no write.
- A field with no default is refused with `400 VALIDATION_ERROR` at
`backfill.0`, and the draft stays pending.
  - Four entry refusals arrive in one answer.
  - The shape refusals: non-array, unknown key, duplicate, no context.
  - An existing value is kept.
  - Paging over 205 rows.
  - Caller context at the CRUD seam and in the hook.
- A caller the update refuses gets `updated: 0` with `code:
PERMISSION_DENIED`, and the publish is kept.
  - A clearing hook stops the field.
- Producer conformance
(`objectql/src/publish-package-drafts-response-conformance.test.ts`)
gains one case: `backfillApplied` parses unstripped.
- Door conformance
(`runtime/src/domains/packages-publish-drafts-response-conformance.test.ts`)
gains four cases:
- The key and the caller's context reach the protocol, and the receipt
conforms.
  - A stopped field's `error`/`code` conform.
  - Without the key, neither key is sent.
- The protocol's refusal answers 400 `VALIDATION_ERROR` with
`backfill.0`, with no flip and no announce.
  - `backfillApplied` also joins the byte-stability absent-key list.
- Full suites on `08c87b8d0`, every exit code 0:
- `pnpm --filter @objectstack/metadata-protocol test`: 244 files, 20641
passed.
  - `@objectstack/client`: 52 files, 671 passed.
  - `@objectstack/runtime`: 416 files, 6549 passed.
  - `@objectstack/objectql`: 438 files, 8255 passed.
  - `@objectstack/spec`: 693 files, 21002 passed.
- `pnpm --filter … typecheck` for spec, metadata-protocol, runtime,
client and objectql: each exit 0. objectql's and runtime's `typecheck`
reach the test files through `check:test-typecheck`.
- `pnpm --filter @objectstack/spec check:generated`: exit 0, after
`--fix` regenerated the two stale artifacts named above.

Live, over HTTP through the door (a `pnpm dev:crm -- --fresh` backend
built from this branch, SQLite):
- A backfill naming `flagged` (no default) and `title` (not new)
answered `400 VALIDATION_ERROR`. `details.issues` named `backfill.0`
("declares no defaultValue") and `backfill.1` ("already exists on the
published object"), and the live object's fields were unchanged.
- The publish with `backfill: [done, score]` answered `200 published`,
`backfillApplied: { success: true, fields: [{ done, updated: 0 }, {
score, updated: 0 }] }`. The 12 rows read `false` / `7`: the column
DEFAULT had filled them, as measured above.

Ablations, one per mechanism. The mutations went through
`scripts/ablation-replace.mjs` (wrap mode), and the dist legs were
proven with `scripts/ablation-dist-preflight.mjs` (`marker present`,
then `--absent` and `tree clean` on restore):
- **A1, the write** (`applyPublishBackfill` gated off): the backfill
suite went red in 6 cases, the card case included.
- **A2, the no-default refusal** (that guard gated off): red in 1 case,
"a field without a defaultValue is refused".
- **A3, the door's context** (`context` dropped from the pass-through,
runtime source): red in 1 case, "reaches the protocol … with THIS
caller's execution context".
- **A4, the spec declaration.** `backfillApplied.fields[].updated` was
re-typed to a string. The producer conformance went red in 1 case and
the door conformance in 2.
- The first A4 attempt renamed the key. The spec build itself refused
that (`authorable-surface` reports
`api/PublishPackageDraftsResponse:backfillApplied` removed), so nothing
reached `dist/` and that leg was void. The objectql leg ran green
against the old `dist/` and is not counted. The preflight caught it, and
the leg was redone as the type mutation above.

Gates:
- `node scripts/pm/dispatch-gates.mjs --commands` (no paths) on
`08c87b8d0` derived 116 commands. Every line was run, and each exit code
was captured before any pipe. The reconciliation is in the os-dev report
on #24253.
- The earlier run on the pre-merge head (114 commands) found one real
red: `check:query-options-erasure`. The new test cast engine options `as
any` (test surface 236 → 237). It was fixed by typing them, and the gate
is green.
- `check:engine-split-ratio` and `check-plugin-teardown-shape
--self-test` first refused on the shallow clone. After `git fetch
--shallow-since=2026-07-13`, both exit 0.

## Decisions settled by the four axes (recorded for review)

- **Token and expression defaults are refused rather than evaluated per
row.**
- Business need: the measured consumers are booleans and literals
(cloud#2656, the 「可选补 0」 card).
  - Long term: there is one meaning per default.
- AI-proofing: a loud refusal beats stamping publish-time `NOW()` or the
publisher's id into everyone's rows.
- Startup scope: this is the narrowest claim the runtime fully honours.
- **A context is required for any non-empty backfill.** "As the caller"
is unprovable without one, and a silent system write is the failure the
ruling forbids.
- **Fields are independent.** One field's failure does not stop the
others, and each element reports its own outcome.

## Acceptance notes

- **Turso remote, memory and MongoDB fill nothing on ADD COLUMN; the SQL
driver does.** This is the reason the opt-in matters on cloud. The Turso
half is already recorded in `remote-transport.ts`
`columnDeclarationSQL`'s docblock and in
`turso-remote-not-null.test.ts`. Carrier: none — not filed (a recorded,
known asymmetry, outside this card).
- **"As the caller" means the caller's visibility.** On an
organization-partitioned object, the read is the caller's, so rows
outside the caller's RLS/tenant scope are not read and not counted. That
is the ruling's own semantics. `updated` says what was written, never
that the table is complete.
- **The read→write window** between a page's read and its by-id write is
described under H4.
- No objectui or cloud change. `discardPackageDrafts` and every other
verb are untouched.

---
_Generated by [Claude
Code](https://claude.ai/code/session_01DtBADZFvLxnZCFFDWCy87D)_

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: objectstack-fleet[bot]

.changeset/24253-client-publish-drafts-backfill.md

@@ -0,0 +1,9 @@
+---
+"@objectstack/client": minor
+---
+
+feat(client): `packages.publishDrafts(id, { backfill })` sends the opt-in backfill of new fields' defaults
+
+Clause-②: yes (widening)
+
+- `opts.backfill?: Array<{ object, field }>` is sent in the publish body; the response's `backfillApplied` reports the rows written per field. A field the publish does not add, or one without a literal `defaultValue`, refuses the whole request with a `400` before anything is published.

.changeset/24253-metadata-protocol-publish-backfill.md

@@ -0,0 +1,12 @@
+---
+"@objectstack/metadata-protocol": minor
+---
+
+feat(metadata-protocol): `publishPackageDrafts` takes an opt-in `backfill` that writes the literal `defaultValue` of fields the publish adds into the object's existing rows, as the caller
+
+Clause-②: yes (widening)
+
+- **New request keys.** `backfill?: Array<{ object, field }>` names fields this publish adds; `context` is the caller's execution context the writes run as. Absent or `[]`, no row is touched.
+- **Judged before anything is published.** Every entry must name an object drafted in the package, a field its draft declares, a field the published object does not have yet, and a literal `defaultValue` (a runtime token such as `NOW()` or `current_user`, or an expression, is resolved per new record and is refused). A malformed entry, a duplicate, or a backfill without a caller context is refused too. All problems come back in one `400 VALIDATION_ERROR`, one `issues[]` entry per problem with its `backfill.N` path, and nothing is promoted or written.
+- **Written after the publish, as the caller.** Once the drafts are live and the new column exists, rows whose value is absent are read and written 200 at a time through `updateManyData` with the caller's context, so permissions, RLS/FLS, validation and record hooks run as on that caller's own edit. A value already stored is never overwritten.
+- **New response key `backfillApplied`:** `{ success, fields: [{ object, field, updated, error?, code? }] }`, one element per requested field. `updated` counts the rows this publish wrote. A field stops at its first failing row and reports the rows written so far with the error; the publish is not undone. On the SQL driver the new column's DEFAULT already fills existing rows, so `updated` is 0 there.

.changeset/24253-runtime-publish-drafts-backfill.md

@@ -0,0 +1,10 @@
+---
+"@objectstack/runtime": minor
+---
+
+feat(runtime): `POST /packages/:id/publish-drafts` accepts a `backfill` body key and passes it to the protocol with the caller's execution context
+
+Clause-②: yes (widening)
+
+- The body's `backfill` (`[{ object, field }]`) reaches `publishPackageDrafts` as sent, together with the request's own execution context, so the backfill writes run as the signed-in caller, never as the system. A body without the key sends neither.
+- The protocol's request refusal answers `400 VALIDATION_ERROR` with its `issues[]`, before anything is published, flipped or announced. The `backfillApplied` receipt rides the `200` body.

.changeset/24253-spec-publish-drafts-backfill-applied.md

@@ -0,0 +1,9 @@
+---
+"@objectstack/spec": minor
+---
+
+feat(spec): `PublishPackageDraftsResponseSchema` declares `backfillApplied`, the per-field receipt of the publish door's opt-in backfill
+
+Clause-②: yes (widening)
+
+- `backfillApplied?: { success, fields: [{ object, field, updated, error?, code? }] }`, present only when the request named a `backfill` and the batch published. Declared so the door's response parse carries it instead of stripping it.

content/docs/references/api/protocol.mdx

@@ -2524,6 +2524,7 @@ Installed package with runtime lifecycle state
 | **materializeApplied** | `{ success: boolean; inserted: integer; updated: integer; failures: object[] }` | optional | ADR-0086 P2 — aggregate result of publish-time materializers across the batch (e.g. `flow`, whose draft credentials become the live ones), including side-effect failures surfaced by the per-item effects loop. Present ONLY when at least one published item had a registered materializer or a side-effect failure. Best-effort, same contract as `seedApplied`. |
 | **probes** | `any` | optional | ADR-0038 L3 post-publish runtime probe report — one real read per published artifact (seeded objects have rows, views are readable, widget dataset selections execute). DELIBERATELY OPAQUE in this contract: the key is declared and carried through verbatim, but its inner shape is intentionally not modeled until a consumer needs a field of it. Present only when something was publishable; probes never fail the publish. |
 | **commitId** | `string` | optional | ADR-0067 — id of the commit this publish recorded. Absent when nothing published. |
+| **backfillApplied** | `{ success: boolean; fields: object[] }` | optional | The opt-in `backfill` request key's receipt: the literal `defaultValue` of each named field this publish added, written into the object's existing rows whose value is absent, page by page, as the caller (permissions, RLS/FLS and record hooks apply as on their own edit). Present ONLY when the request named at least one field and the batch published; absent means nothing was asked. A request that names a field this publish does not add, or one without a literal default, is refused with a 400 before anything is published, so this key never reports a refusal. |
 | **unhiddenApps** | `string[]` | optional | ADR-0045 §3 — names of the apps whose `_unpublished` gate this publish cleared (publish = live AND visible; a materialized additive build has no drafts, only this flip). Attached by the REST door, not the protocol helper. Present ONLY when at least one app flipped — on a mid-loop failure it names the apps that DID persist, beside `unhideError` ('s split report). Spelling is a permanent wire contract (see the door's note). |
 | **unhideError** | `string` | optional | Present when the ADR-0045 visibility flip failed (wholly or partway): the drafts ARE published, but apps still stored `_unpublished: true` stay externally unobservable. Client-facing text only — undeclared driver text is withheld per ADR-0112; the full cause is in the server log. The route is idempotent: re-run it once the cause is resolved. |
 | **rebindError** | `string` | optional | Present when the post-publish `metadata:reloaded` announce failed: everything is published and stored, but boot-cached consumers keep the pre-publish view until re-run or restart (a newly published record-triggered flow does not bind its trigger). Client-facing text only, same ADR-0112 withhold as `unhideError`. |
@@ -2567,6 +2568,13 @@ Installed package with runtime lifecycle state
 | **updated** | `integer` | ✅ | Data-plane rows updated across the batch. |
 | **failures** | `{ type: string; name: string; error: string }[]` | ✅ | Each item whose data-plane projection did NOT land — named per-item, unlike the single-item door's scalar `error`, because one aggregate boolean over N items would hide WHICH ones never went live. |
 
+### Nested Shape: `PublishPackageDraftsResponse.backfillApplied`
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **success** | `boolean` | ✅ | False when any field stopped short (its entry carries `error`). The publish itself still succeeded and is not undone. |
+| **fields** | `{ object: string; field: string; updated: integer; error?: string; … }[]` | ✅ | One element per requested field, in request order. |
+
 
 ---
 

docs/audits/2026-07-unknown-key-strictness-ledger.counts/api.md

@@ -19,4 +19,4 @@ hand-patch a number here** — fix the code or the verdict and regenerate.
 
 | Dir | Sites |
 |---|---|
-| `api/` | 442 |
+| `api/` | 444 |

packages/client/src/index.ts

@@ -2895,8 +2895,17 @@ export class ObjectStackClient {
      * `unhiddenApps` / `unhideError` / `rebindError`) each report their own
      * outcome: a 200 does not mean the data plane or the visibility flip
      * caught up. `probes` is deliberately opaque in the contract (#9406).
+     *
+     * `opts.backfill` (#24253) names fields this publish adds whose literal
+     * `defaultValue` is written into the object's existing rows, as the
+     * signed-in caller; `backfillApplied` reports the rows written per field.
+     * A field that is not new, or declares no literal default, refuses the
+     * whole request with a 400 before anything is published.
      */
-    publishDrafts: async (id: string, opts?: { actor?: string }): Promise<PublishPackageDraftsResponse> => {
+    publishDrafts: async (
+        id: string,
+        opts?: { actor?: string; backfill?: Array<{ object: string; field: string }> },
+    ): Promise<PublishPackageDraftsResponse> => {
         const route = this.getRoute('packages');
         const res = await this.fetch(`${this.baseUrl}${route}/${encodeURIComponent(id)}/publish-drafts`, {
             method: 'POST',