You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3d8960c
Browse filesBrowse the repository at this point in the historyBrowse files
feat(cli,plugin-audit): os migrate audit-mail-bodies masks the archive datasource too (#24934)
Fixes#24899
Clause-②: yes (widening)
`os migrate audit-mail-bodies` now masks the mail bodies in the
`sys_audit_log` copies the lifecycle Archiver already moved to the
`archive` datasource. Hot copies masked by the run are archived masked
later, so the same run covers those too. The command keeps its dry-run /
`--apply` / `--yes` / `--json` shape, and its report is still counts
only.
> The claim expected `Clause-②: no`. From the diff it is `yes
(widening)`: the `--json` document gains an `archive` key, and
`@objectstack/plugin-audit` gains exports. Both packages take a `minor`
changeset.
## Measured
Every reading below is from `origin/main` at `81e6edd46` and is a count,
a status or a boolean. No body value was printed.
1. **Which deployments can have an `archive` datasource, and who reads
it.**
- There is no special configuration. The Archiver resolves `archive.to`
by driver name
(`packages/objectql/src/lifecycle/lifecycle-service.ts:1344`,
`engine.datasource(name)` → `packages/objectql/src/engine.ts:21671`, the
`drivers` map). A driver is registered under the name `archive` in one
of three ways:
- (a) A stack-config `datasources` entry. Nothing routes objects to an
archive, so the ADR-0062 D2 gate connects it only with `autoConnect:
true`
(`packages/services/service-datasource/src/datasource-connection-service.ts:301-305`,
`packages/spec/src/data/datasource.zod.ts:665`).
- (b) A runtime datasource row: a `sys_metadata` row of type
`datasource`, created in Setup → Datasources and rehydrated by `os
serve` at boot
(`packages/services/service-datasource/src/datasource-admin-plugin.ts:125-139`,
`:734-760`).
- (c) A host plugin that publishes a `driver.archive` service
(`packages/objectql/src/plugin.ts:1060-1076`).
- No shipped template or example declares one. There are zero `name:
'archive'` datasources under `examples/` and
`packages/create-objectstack`. Every deployment that declares none keeps
every row hot: `archive-pending` (`lifecycle-service.ts:1349-1355`).
- Readers: only the Archiver reads the archive through the platform.
`engine.datasource(` has one caller, `lifecycle-service.ts:1344`, and
the engine never routes `sys_audit_log` there. Otherwise there is direct
database access, and the datasource admin door: `GET
/datasources/:name/remote-tables` plus the object-draft flow, gated by
`requireDatasourceAdmin`
(`packages/services/service-datasource/src/admin-routes.ts:706-707`). A
federated object an admin binds over the archived table would serve it
over REST.
2. **Does the archive copy pass through the mask?** No. It is a verbatim
driver-to-driver copy: `hot.find` → `cold.upsert(object, row, ['id'])` →
`hot.bulkDelete` (`lifecycle-service.ts:1471-1520`), with no hook, no
writer and no mask. So a hot copy written before #24688 and archived
before the operator runs the mask lands in the archive in clear. Pinned
below in case 4: a copy masked hot is archived masked.
3. **A supported way to run the mask over the archive?** None before
this PR: the runner took the engine only
(`packages/plugins/plugin-audit/src/mail-log-body-migration.ts`,
`migrateMailLogBodyCopies`). Two facts made one sound:
- The runner's reads and writes fit the `IDataDriver` contract (`find`
with `where` / `fields` / `orderBy` / `limit`, and `update(object, id,
data, { preserveAudit })`).
- `bootSchemaStack` already connects declared datasources when asked to
(`composeDeclaredDatasources`,
`packages/cli/src/utils/schema-migrate.ts:1345-1350`).
4. **`sys_activity`**: no archive. Its lifecycle is `telemetry`, with
14-day retention and rotation
(`packages/plugins/plugin-audit/src/objects/sys-activity.object.ts:29-34`),
so its copies are only ever hot.
## What changed
- **`@objectstack/plugin-audit`.**
- `migrateArchivedMailLogBodyCopies(driver, logger, options)` runs the
same pages and planners through a datasource's driver, as the Archiver
itself does. `preserveAudit` keeps each row's `updated_at`.
- An archive table nothing was archived into yet reads as no rows, by
`isMissingTableError`. Any other refused read is a failure.
- New `objects` option on both runners. The engine runner's behaviour is
unchanged.
- **`os migrate audit-mail-bodies`.**
- After the hot run, it reads each copy table's declared
`lifecycle.archive.to` and runs the archive runner over
`engine.getDriverByName(to)`.
- It boots with `composeDeclaredDatasources: true`, so a declared
archive is connected as `os serve` connects it.
- When the target is not connected but an active runtime datasource row
of that name exists, the archive reads `not-connected` and the command
exits `1`. It does not report a clean archive.
- The success path ends with `exitOneShotCommand`, as `plan` / `apply` /
`unmapped-columns` do. Without it, the pool the declared connection
opened kept the process alive after "Graceful shutdown complete".
Measured here: it hung, and `TaskStop` ended it.
- **Docs:** the command's section of `content/docs/deployment/cli.mdx`,
plus its row in the command table.
- **Changeset:** `.changeset/24899-archived-mail-bodies.md` (`minor` for
both packages).
## Proof
- **End to end.**
`packages/cli/src/commands/migrate/audit-mail-bodies.archive.integration.test.ts`,
7/7.
- The fixture is a project that declares an `archive` SQLite datasource
(`autoConnect: true`). A served-shape boot seeds 18.0-writer copies, and
the real Archiver (`lifecycle.sweep()`) moves 3 rows. 2 of them hold a
clear body.
- Dry run: hot to-mask 1; archive `examined`, scanned 2, to-mask 2;
nothing written.
- `--apply`: archive rewritten 2; clear copies 0 in either store; every
other archived column equal, `updated_at` included.
- Re-run: rewritten 0, archived rows unchanged.
- The copy that was hot during the run, aged and archived by a second
sweep: archived masked (archive rows 4, clear 0).
- No run printed the sentinel.
- Runtime-row archive: `not-connected`, exit 1. No archive at all:
`none`, exit 0.
- **Ablation.** Removing the archive lookup (`const driver = undefined`)
turns cases 1–4 red (4 failed, 3 passed). Restored to the HEAD blob
`b6e1355a`.
- **Runner unit tests** over a real `SqliteWasmDriver` store provisioned
as the Archiver does (`syncSchema` + `upsert`): dry run, apply,
idempotent re-run, an archive with no table yet, a refused read.
## Local verification (head `39fc914f`; the diff since `8a87750c` is one
test-file header comment)
- `@objectstack/plugin-audit`: `pnpm test` 983/983 (60 files);
`typecheck` green.
- `@objectstack/cli`: build + `typecheck` green. The `unit` project:
4634/4635. The one failure was
`test/hook-timeout-override-refusal.test.ts` CONTROL, a 5000 ms timeout
while the gate sweep ran beside it. Re-run alone: 4/4. It spawns a
process this diff does not touch.
- `@objectstack/cli` `integration`, run locally because this diff
touches that layer: the new archive pin 7/7;
`data-commands.absent-database` + `schema-migrate.one-shot-family`
151/151. `test/json-stdout-purity.e2e.test.ts` is left to CI.
- Changed files: `eslint --no-inline-config` clean.
- `node scripts/pm/dispatch-gates.mjs --commands`: 101 derived, every
one run, exit codes captured before any pipe; `--ran` reconciles 101 /
97 run / 4 NOT MEASURED / 0 unrun.
- **The census was exit 1 (2) on `39fc914f`. It is exit 0 on
`4fb9afa1`.**
- This diff adds one write call site (the archive runner's
`driver.update`), so the count goes 232 → 233.
- The seat widened the surface in amendment 6295968920 and again in
6296052814.
- `3340a4ad` ran `node scripts/tenant-audit-census.mjs --write`.
- `105a8110` and `4fb9afa1` made the 11 hand-written prose figures +1,
numbers only.
- On `4fb9afa1`, `node scripts/check-tenant-audit-census.mjs
--self-test` passes 27 cases, and the plain run reports 233 write call
sites certified and 23 prose figures held. Both exit 0.
- The new census row:
`packages/plugins/plugin-audit/src/mail-log-body-migration.ts` ·
`update` · object chosen at run time (the CLI limits the runner to
`sys_audit_log`) · tenancy `undecidable` · context `options unreadable`
· 1. The write goes to the archive datasource through its own driver,
which no tenant scope covers. It masks every organization's archived
copies by design, and only the operator command reaches it.
- **exit 3, prerequisite not met (4), NOT MEASURED, left to CI:**
`check-plugin-teardown-shape --self-test` (shallow clone), `spec
check:skill-examples`, `check:dual-build-cjs-loads`,
`check:i18n-coverage` (each needs the full monorepo build).
- every other family: exit 0.
- `node scripts/pm/check-governed-merges.mjs --branch
origin/claude/issue-24899-archived-mail-bodies`: NOT governed; 791
changed lines.
## Surface question (answered)
The seat widened the card's surface to the census regeneration and its
11 prose figures (6295968920, 6296052814 on #24899, plus the line-210
figure by message). The census is regenerated by the generator only, and
`Lint & Repo Gates`' census step is green on `4fb9afa1`.
## Acceptance notes
- **Behaviour change of the boot:** the command now connects the
datasources the config declares, as `os serve` does. A declared
datasource that fails to connect therefore fails the command's boot, as
it fails `os serve`'s. Before, the command never tried to connect it.
- **Stale comment, fixed here** (`f72911d8`, amendment 6296052814):
`packages/cli/src/utils/schema-migrate.ts:1176-1179` now names `os
migrate audit-mail-bodies` among the callers of
`composeDeclaredDatasources`. This diff had made the old text false.
- **Not covered by this command:** an archive registered only by a host
plugin (`driver.archive`) that the CLI boot does not compose. It reads
`none` unless a runtime datasource row of that name exists. A
code-declared archive *without* `autoConnect: true` is never connected
by `os serve` either, so the Archiver never archived to it.
- **17.x:** the writer mask (#24688) and this command shipped in 18.0.0.
A 17.x line, if cut, has neither, so it would owe the whole mail-body
note (hot and archive), not only this half.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01WQWCrorWxW5zDh4n6jCEF5)_
<sub>Census, surface and stale-comment lines updated by the epic seat on
2026-10-20 (about 16:50Z) to match head `4fb9afa1`.</sub>
---------
Co-authored-by: Claude <noreply@anthropic.com>
feat(cli,plugin-audit): `os migrate audit-mail-bodies` also masks the mail bodies in the audit copies already moved to the `archive` datasource
7
+
8
+
Clause-②: yes (widening)
9
+
10
+
**Re-run `os migrate audit-mail-bodies --apply` on a deployment with an archive datasource.** The 18.0.0 command masked the hot audit tables only.
11
+
12
+
-**What was left in clear.**`sys_audit_log` declares `lifecycle.archive: { after: '90d', to: 'archive', keep: '7y' }`. Past the hot window, the lifecycle Archiver copies its rows to the datasource named `archive` as they are, then deletes them hot. A `sys_email` / `sys_identity_email` copy archived before the audit writer masked the mail body still holds `body_text` / `body_html` in clear there, password-reset, verification, magic-link and invitation links included, for as long as the archive keeps it. The engine never routes `sys_audit_log` to the archive, so the hot run could not reach those rows.
13
+
-**What changed.** After the hot run, the command reads the archive target each copy table declares (`sys_audit_log` only; `sys_activity` declares none) and runs the same mask, in the same dry-run or `--apply` mode, through that datasource's driver. Each archived row keeps every other column, `updated_at` included. The run is idempotent and its report is counts only. A hot copy masked first is archived masked, so one run covers both the copies already archived and the ones still to be.
14
+
-**Which archives it reaches.** The command now connects the managed datasources the stack config declares, through the same connect service `os serve` uses. An archive datasource is declared with `autoConnect: true`, since nothing routes objects to it. It is NOT examined, and the run reports `not-connected` and exits `1` rather than reporting a clean archive, in two cases. In the first, the archive is declared but this boot did not connect it; the entry carries the connect status and reason. In the second, the archive is a runtime datasource created in Setup → Datasources, which `os serve` connects and this command does not; declare it in the config's `datasources` and re-run.
15
+
-**What `none` means.** No datasource of that name is declared and connected on this boot, none was attempted, and no active runtime datasource row names one. That is all it checks. An archive a host registers in code outside the stack config is not reached and reads `none`. To mask it, declare it in the stack config's `datasources` with `autoConnect: true`.
16
+
-**The command's boot can now fail on a declared datasource.** A bound, mapped or boot-critical declared datasource that cannot connect now fails the command's boot with exit `1`, as it fails `os serve`'s. `OS_ALLOW_DRIVER_CONNECT_FAILURE` is the same escape.
17
+
-**Run it with the server stopped**, or re-run it once and read 0 to mask in both halves. A live server's Archiver can move a row between the hot run and the archive run.
18
+
-**New in the `--json` document:** an `archive` array, one entry per declared target: `{ datasource, objects, status, report?, connection? }`, with `status` one of `examined`, `none`, `not-connected` or `unknown`. The `report` key and its shape are unchanged.
19
+
-**New exports from `@objectstack/plugin-audit`:**`migrateArchivedMailLogBodyCopies(driver, logger, options)`, the runner over a datasource's driver, and an `objects` option on both runners, which limits a run to the named copy tables. Types `MailLogBodyArchiveDriver`, `MailLogBodyCopyObject` and `MailLogBodyMigrationOptions`.
Copy file name to clipboardExpand all lines: content/docs/deployment/cli.mdx
+50-4Lines changed: 50 additions & 4 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1204,7 +1204,7 @@ where the data lives.
1204
1204
| `os migrate comment-reactions` | Carry the emoji reactions a 17.x deployment stored in the retired `sys_comment.reactions` column into `sys_comment_reaction` records, list every stored value that does not read, and record the verified run that lets `os migrate apply --allow-destructive` drop the column |
1205
1205
| `os migrate organization-ownership` | The v18 ownership ceremony (ADR-0131 D10): write a plan of what each table's rows will become, then `--apply` that reviewed plan. It upgrades a database 17.x or an 18.0 pre-release (`18.0.0-next.*`) created; a v18 server refuses to start on one until it has run. See [below](#os-migrate-organization-ownership) |
1206
1206
| `os migrate activity-changed-fields` | Stamp the two read-rule columns (`changed_fields`, `milestone_key`) on the `sys_activity` rows written before they existed. Until it runs, those rows are withheld from every reader but the system, so run it at once after upgrading. No flag; idempotent |
1207
-
| `os migrate audit-mail-bodies` | Mask the mail bodies (`body_text`, `body_html`) that `sys_audit_log` and `sys_activity` copies of `sys_email` / `sys_identity_email` rows still hold in clear, written before the audit writer masked them. Reports counts only. No flag; idempotent. See [below](#os-migrate-audit-mail-bodies) |
1207
+
| `os migrate audit-mail-bodies` | Mask the mail bodies (`body_text`, `body_html`) that `sys_audit_log` and `sys_activity` copies of `sys_email` / `sys_identity_email` rows still hold in clear, written before the audit writer masked them. Covers the `archive` datasource `sys_audit_log` archives to as well. Reports counts only. No flag; idempotent. See [below](#os-migrate-audit-mail-bodies) |
1208
1208
| `os migrate comment-parent-pair` | Stamp the parent pair (`object_name`, `record_id`) on the `sys_comment` rows written before it existed. Until it runs, a broad comment read past 2,000 candidate rows omits comments the reader may see, and its total agrees with the short list, so run it at once after upgrading. No flag; idempotent |
1209
1209
| `os migrate meta --stored` | Replay the metadata conversion chain over this deployment's `sys_metadata` rows and rewrite the ones still carrying a pre-protocol shape. Hygiene, not a gate — nothing depends on it having run |
1210
1210
| `os migrate security-catalog-overlays` | List the environment-wide permission-set and position rows (the legacy plurals `permissions` / `positions` included) stored over a name a configured package holds — the rows a v18 cold boot refuses — each with the package that holds its name; `--apply` deletes them, one history tombstone per row. Run it before the first v18 boot, with the flags and environment the deployment boots with: the platform security plugin's sets are held only where `os serve` composes it (an auth secret set, and the `auth` tier on), and the step takes `serve`'s `--preset` and `--dev` with `serve`'s meaning. Adopts nothing; exits 1 while it lists a row |
@@ -1501,9 +1501,55 @@ value.
1501
1501
1502
1502
**Idempotent**: a masked body is the mask, so a second run masks nothing, and
1503
1503
re-running is how an interrupted run is finished. A failed read or write exits
1504
-
`1`. The copies age out with each table's retention (14 days for activity, 90
1505
-
for the audit log, unless the deployment lengthens it); run it once after
1506
-
upgrading rather than wait for that.
1504
+
`1`. Activity copies age out with its 14-day retention. Audit-log copies do not
1505
+
age out: with no archive datasource they are retained hot, and with one they
1506
+
are moved there after 90 days and kept for seven years. Run it once after
1507
+
upgrading.
1508
+
1509
+
**Run it with the server stopped**, or re-run it once and read 0 to mask in
1510
+
both halves. A live server's Archiver sweeps on its own schedule, and a row it
1511
+
moves between the hot run and the archive run can land in the archive
The `--json` document carries one `archive` entry per declared target, with a
1537
+
`status`:
1538
+
1539
+
| `status` | Meaning | Exit |
1540
+
| :--- | :--- | :--- |
1541
+
| `examined` | Connected and masked (or counted); `report` holds the counts | `0`, or `1` on a failed read or write |
1542
+
| `none` | No datasource of that name is declared and connected on this boot, none was attempted, and no active runtime datasource row names one. That is all it checked | `0` |
1543
+
| `not-connected` | The archive was **not** examined. Either the stack config declares it and this boot did not connect it (`connection` carries the connect status and reason), or the database holds an active runtime datasource of that name (one created in Setup → Datasources), which `os serve` connects and this command does not | `1` |
1544
+
| `unknown` | Not connected, and the datasource rows could not be read | `1` |
1545
+
1546
+
On `not-connected` for a runtime datasource, declare the archive datasource in
1547
+
the stack config's `datasources` (with `autoConnect: true`), then re-run. For a
1548
+
declared one, make it reachable from where the command runs, then re-run.
1549
+
1550
+
An archive that a host registers in code outside the stack config (a plugin
1551
+
that publishes the driver itself) is not reached and reads `none`. To mask it,
1552
+
declare it in the stack config's `datasources` with `autoConnect: true`.
Copy file name to clipboardExpand all lines: content/docs/permissions/tenant-audit-census.mdx
+18-18Lines changed: 18 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way.
122
122
123
123
The same holds twice over for the context. An options argument spelled as a
124
124
literal can be read; one spelled `options`, `{ ...opts }`, or handed through a
125
-
forwarding shim cannot, and **50 of the 232 sites are spelled that way**. A
125
+
forwarding shim cannot, and **51 of the 233 sites are spelled that way**. A
126
126
context resolved from an inline literal or a local `const` can be tested for
127
127
`isSystem`; one arriving from a helper call cannot.
128
128
@@ -150,8 +150,8 @@ now **0**: nothing on this surface threads a context that provably lacks the fla
150
150
151
151
**"No tenant context" counted sites it had not read.** An options argument the
152
152
walker could not parse was folded into the same bucket as one it had read and
153
-
found empty. That published **58 sites "carrying no tenant context at all"**
154
-
when 8 said so and 50 were simply unread — an over-claim in the *alarming*
153
+
found empty. That published **59 sites "carrying no tenant context at all"**
154
+
when 8 said so and 51 were simply unread — an over-claim in the *alarming*
155
155
direction, on the very figure this page tells other cards to cite. `carries` is
156
156
now three-valued, and an unreadable argument can never contribute to the
157
157
provable count.
@@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page:
187
187
188
188
| carried figure | where it survives | this census |
189
189
| :--- | :--- | ---: |
190
-
| 175 write call sites | quoted in the merged changeset |**232**|
190
+
| 175 write call sites | quoted in the merged changeset |**233**|
191
191
| 24 carrying no tenant context | quoted in the merged changeset |**2** provable and tenancy-enabled; **29** more whose options argument is unreadable |
192
-
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card |**156 of 232** decidable, **76** undecidable |
193
-
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration**|**not reproduced**: 131 decidably elevated, 0 decidably not, 93 undecidable |
192
+
| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card |**156 of 233** decidable, **77** undecidable |
193
+
| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration**|**not reproduced**: 131 decidably elevated, 0 decidably not, 94 undecidable |
194
194
| 141 and 132, two independent re-derivations | the card that filed this work | — |
195
195
196
196
**The differences are not reconciled, and deliberately so.** The old census's
@@ -207,11 +207,11 @@ would report a smaller number and would not say so.
207
207
208
208
The fourth row is the one worth flagging to anyone citing it. **The 135 / 77%
209
209
figure has no surviving corroboration anywhere in the tree.** This census reads
210
-
131 of 232 (56%) as decidably elevated, with 93 more whose elevation is a
210
+
131 of 233 (56%) as decidably elevated, with 94 more whose elevation is a
211
211
run-time fact — so the claim is neither confirmed nor refuted, and the honest
212
212
answer is that a static reading cannot settle it.
213
213
214
-
⇒ **Cite `2 / 232`, and say what it is**: the sites whose options argument was
214
+
⇒ **Cite `2 / 233`, and say what it is**: the sites whose options argument was
215
215
READ and holds no tenant context, against a decidably tenancy-enabled object.
216
216
That is the control's provable yield surface. ⛔ Do not cite it as "the sites
217
217
without tenant context" — **29 further sites** have an options argument this
@@ -223,31 +223,31 @@ cannot read, and they are neither in nor out.
223
223
224
224
| what | count |
225
225
| :--- | ---: |
226
-
| write call sites on the application surface |**232**|
226
+
| write call sites on the application surface |**233**|
227
227
| …whose object name is statically decidable | 156 |
228
-
| …whose object name is chosen at run time |76|
228
+
| …whose object name is chosen at run time |77|
229
229
| …against an object with tenancy ENABLED | 155 |
230
230
| …against an object that declares tenancy off | 1 |
231
231
| threading a tenant context | 174 |
232
232
| PROVABLY carrying none (options read, no context key) |**8**|
233
233
| …of those, against a decidably tenancy-enabled object |**2**|
234
-
| options argument UNREADABLE — may or may not carry one |50|
234
+
| options argument UNREADABLE — may or may not carry one |51|
235
235
| …of those, against a decidably tenancy-enabled object | 29 |
236
236
| threading a decidably ELEVATED (`isSystem`) context | 131 |
237
237
| threading a context that is decidably NOT elevated | 0 |
238
-
| threading a context whose elevation is a run-time fact |93|
238
+
| threading a context whose elevation is a run-time fact |94|
239
239
240
240
| how the instrument reached the site | count |
241
241
| :--- | ---: |
242
-
| receiver carried a readable engine type |191|
242
+
| receiver carried a readable engine type |192|
243
243
| receiver erased, placed by the object NAME | 31 |
244
244
| receiver erased, placed by an `object: string` PARAMETER | 5 |
245
245
| receiver erased, placed by an `UNTYPED_RECEIVERS` row | 5 |
246
246
247
247
| object name spelled inline | 107 |
248
248
| object name spelled through a `const`| 49 |
249
249
| object name is an `object: string` parameter | 7 |
250
-
| object name is some other run-time expression |69|
250
+
| object name is some other run-time expression |70|
251
251
252
252
### Subtractions the census could NOT defend — enforced
253
253
@@ -297,13 +297,13 @@ holds still. They are required to be HERE and to say WHEN they were true;
297
297
their values are not compared. The reasoning, and the measurement behind it,
0 commit comments