diff --git a/.changeset/24693-audit-mail-bodies.md b/.changeset/24693-audit-mail-bodies.md new file mode 100644 index 00000000000..1f8b014aa9b --- /dev/null +++ b/.changeset/24693-audit-mail-bodies.md @@ -0,0 +1,18 @@ +--- +'@objectstack/plugin-audit': minor +'@objectstack/cli': minor +--- + +feat(cli,plugin-audit): `os migrate audit-mail-bodies` masks the mail bodies that audit and activity copies written before the mail-body mask still hold in clear + +**Run `os migrate audit-mail-bodies --apply` once after upgrading.** Since the mail-body read gate, the audit writer masks `body_text` and `body_html` whenever it copies a `sys_email` or `sys_identity_email` row into `sys_audit_log` (`new_value`, `old_value`) or `sys_activity` (both halves of `metadata`). Copies written before that still hold the body in clear at rest: password-reset, verification, magic-link and invitation links included. Today they are served only to a reader who is also served the body field, so this is no open door; it is a credential copy at rest, and it is what any later relaxation of the mail-body gate would re-open. + +Clause-②: yes + +- **The command.** `os migrate audit-mail-bodies` masks those copies with the writer's own mask: a set body becomes `••••••••`, an empty one stays empty, and a copy that never carried the column gains none. It is a dry run by default (counts, writes no row), `--apply` writes, and its flags are `os migrate audit-metadata-bodies`'s, one for one (`--database-url`, `--apply`, `--yes`, `--force`, `--json`). The report is counts only, per table, and never prints a value. +- **What changes in a row.** Only the body keys of the copy. Every other key of the copy and every other column of the row stay byte-identical, the row's `updated_at` included (the rewrite writes under `preserveAudit` with the stored stamp). +- **Idempotent.** A masked body is the mask, so a second run masks nothing; re-running finishes an interrupted run. A failed read or write exits `1`. +- **No selector, no ordering.** Every copy of both mail logs is masked, as the writer masks new ones, so the command needs no rule telling identity mail apart from tenant mail. It gives the same result before or after `os migrate organization-ownership`: that ceremony re-points moved mail's activity copies to `sys_identity_email`, which is masked the same, and leaves `sys_audit_log` as written. +- **Who sees a difference.** A platform administrator (the one reader the body field is served to) no longer reads a pre-upgrade mail body in the audit trail or the activity feed. The mail log row itself keeps its body, behind its field gate. + +`@objectstack/plugin-audit` exports the rewrite, `migrateMailLogBodyCopies` (with `MAIL_LOG_BODY_COPY_OBJECTS`, `MAIL_LOG_BODY_MIGRATION_PAGE_SIZE` and its engine and report types), its two pure planners `planMailLogBodyAuditRowPatch` and `planMailLogBodyActivityRowPatch`, and `maskMailLogBodyCopy`, the one mask the audit writer and the rewrite both apply. diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index dd6f1128007..6cd38d6f0e2 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -1218,6 +1218,7 @@ where the data lives. | `os migrate comment-reactions` | Carry the emoji reactions a 17.x deployment stored in the retired `sys_comment.reactions` column into `sys_comment_reaction` records, list every stored value that does not read, and record the verified run that lets `os migrate apply --allow-destructive` drop the column | | `os migrate organization-ownership` | The v18 ownership ceremony (ADR-0131 D10): write a plan of what each table's rows will become, then `--apply` that reviewed plan. A v18 server refuses to start until it has run. 18.0 supports fresh `single` installs; upgrades and walled postures arrive in 18.x. In 18.0 the command refuses. See [below](#os-migrate-organization-ownership) | | `os migrate activity-changed-fields` | Stamp the two read-rule columns (`changed_fields`, `milestone_key`) on the `sys_activity` rows written before they existed. Until it runs, those rows are withheld from every reader but the system, so run it at once after upgrading. No flag; idempotent | +| `os migrate audit-mail-bodies` | Mask the mail bodies (`body_text`, `body_html`) that `sys_audit_log` and `sys_activity` copies of `sys_email` / `sys_identity_email` rows still hold in clear, written before the audit writer masked them. Reports counts only. No flag; idempotent. See [below](#os-migrate-audit-mail-bodies) | | `os migrate comment-parent-pair` | Stamp the parent pair (`object_name`, `record_id`) on the `sys_comment` rows written before it existed. Until it runs, a broad comment read past 2,000 candidate rows omits comments the reader may see, and its total agrees with the short list, so run it at once after upgrading. No flag; idempotent | | `os migrate meta --stored` | Replay the metadata conversion chain over this deployment's `sys_metadata` rows and rewrite the ones still carrying a pre-protocol shape. Hygiene, not a gate — nothing depends on it having run | | `os migrate security-catalog-overlays` | List the environment-wide permission-set and position rows (the legacy plurals `permissions` / `positions` included) stored over a name a configured package holds — the rows a v18 cold boot refuses — each with the package that holds its name; `--apply` deletes them, one history tombstone per row. Run it before the first v18 boot, with the flags and environment the deployment boots with: the platform security plugin's sets are held only where `os serve` composes it (an auth secret set, and the `auth` tier on), and the step takes `serve`'s `--preset` and `--dev` with `serve`'s meaning. Adopts nothing; exits 1 while it lists a row | @@ -1236,8 +1237,8 @@ holds nothing, and answers with empty work and exit 0. `os migrate value-shapes` clean over zero records and names the objects it did not read. `os migrate recorded-by` has nothing to convert and `os migrate resume` no interrupted runs. `os migrate meta --stored` has no stored metadata to examine, `os migrate -audit-metadata-bodies` no audit copy to rewrite, and `os migrate account-issuer` no -account to collide. `os secret orphans`, `os secret rewrap` and `os storage orphans` +audit-metadata-bodies` and `os migrate audit-mail-bodies` no audit copy to rewrite, and +`os migrate account-issuer` no account to collide. `os secret orphans`, `os secret rewrap` and `os storage orphans` report no secret and no file. A read the command cannot avoid and that fails for any other reason still refuses and exits 1. Point `--database-url` at the deployment's database, or boot the deployment once first, to see what it holds. @@ -1488,6 +1489,36 @@ the upgraded deployment's schema sync, which an `--apply` run performs itself: a dry run against a database no upgraded process has opened cannot read them, reports the failed read and exits `1`. A failed read or write exits `1`. +#### `os migrate audit-mail-bodies` + +The audit trail keeps a copy of each audited row: `sys_audit_log` in +`new_value` / `old_value`, and the activity feed in both halves of +`sys_activity.metadata`. For a mail-log row (`sys_email`, `sys_identity_email`) +that copy used to carry the mail **body**, so password-reset, verification, +magic-link and invitation links sat in the audit tables in clear. New copies +mask `body_text` and `body_html` the way a credential field is masked +(`••••••••`; an empty body stays empty). This command masks the copies written +before that. + +```bash +os migrate audit-mail-bodies # Dry run: counts, writes no rows +os migrate audit-mail-bodies --apply # Mask the copies (prompts) +os migrate audit-mail-bodies --apply --yes --json # CI / scripts +``` + +Every copy of either mail log is masked; no mail is told apart from another, so +the command does not depend on `os migrate organization-ownership` and gives the +same result run before or after it. Only the body keys of the copy change: every +other key of the copy, and every other column of the row (its `updated_at` +included), stays as it was. The report holds counts only and never prints a +value. + +**Idempotent**: a masked body is the mask, so a second run masks nothing, and +re-running is how an interrupted run is finished. A failed read or write exits +`1`. The copies age out with each table's retention (14 days for activity, 90 +for the audit log, unless the deployment lengthens it); run it once after +upgrading rather than wait for that. + #### `os migrate organization-ownership` The v18 ownership ceremony (ADR-0131 D10). A v18 `os serve` / `os start` refuses diff --git a/content/docs/permissions/tenant-audit-census.mdx b/content/docs/permissions/tenant-audit-census.mdx index 2431dab14da..3b1e71896d9 100644 --- a/content/docs/permissions/tenant-audit-census.mdx +++ b/content/docs/permissions/tenant-audit-census.mdx @@ -122,7 +122,7 @@ are reported as `undecidable` rather than assumed either way. The same holds twice over for the context. An options argument spelled as a literal can be read; one spelled `options`, `{ ...opts }`, or handed through a -forwarding shim cannot, and **50 of the 230 sites are spelled that way**. A +forwarding shim cannot, and **50 of the 231 sites are spelled that way**. A context resolved from an inline literal or a local `const` can be tested for `isSystem`; one arriving from a helper call cannot. @@ -187,10 +187,10 @@ reproduce them. Where it disagrees, it disagrees on the page: | carried figure | where it survives | this census | | :--- | :--- | ---: | -| 175 write call sites | quoted in the merged changeset | **230** | +| 175 write call sites | quoted in the merged changeset | **231** | | 24 carrying no tenant context | quoted in the merged changeset | **2** provable and tenancy-enabled; **29** more whose options argument is unreadable | -| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **156 of 230** decidable, **74** undecidable | -| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 129 decidably elevated, 0 decidably not, 93 undecidable | +| 127 of 175 statically decidable, 48 runtime-parameter-name sites | restated on the `isSystem`-scoping card | **156 of 231** decidable, **75** undecidable | +| 135 (77%) silenced by the `isSystem` guard before the posture gate | the lost issue body — **no surviving corroboration** | **not reproduced**: 130 decidably elevated, 0 decidably not, 93 undecidable | | 141 and 132, two independent re-derivations | the card that filed this work | — | **The differences are not reconciled, and deliberately so.** The old census's @@ -207,11 +207,11 @@ would report a smaller number and would not say so. The fourth row is the one worth flagging to anyone citing it. **The 135 / 77% figure has no surviving corroboration anywhere in the tree.** This census reads -129 of 230 (56%) as decidably elevated, with 93 more whose elevation is a +130 of 231 (56%) as decidably elevated, with 93 more whose elevation is a run-time fact — so the claim is neither confirmed nor refuted, and the honest answer is that a static reading cannot settle it. -⇒ **Cite `2 / 230`, and say what it is**: the sites whose options argument was +⇒ **Cite `2 / 231`, and say what it is**: the sites whose options argument was READ and holds no tenant context, against a decidably tenancy-enabled object. That is the control's provable yield surface. ⛔ Do not cite it as "the sites without tenant context" — **29 further sites** have an options argument this @@ -223,23 +223,23 @@ cannot read, and they are neither in nor out. | what | count | | :--- | ---: | -| write call sites on the application surface | **230** | +| write call sites on the application surface | **231** | | …whose object name is statically decidable | 156 | -| …whose object name is chosen at run time | 74 | +| …whose object name is chosen at run time | 75 | | …against an object with tenancy ENABLED | 155 | | …against an object that declares tenancy off | 1 | -| threading a tenant context | 172 | +| threading a tenant context | 173 | | PROVABLY carrying none (options read, no context key) | **8** | | …of those, against a decidably tenancy-enabled object | **2** | | options argument UNREADABLE — may or may not carry one | 50 | | …of those, against a decidably tenancy-enabled object | 29 | -| threading a decidably ELEVATED (`isSystem`) context | 129 | +| threading a decidably ELEVATED (`isSystem`) context | 130 | | threading a context that is decidably NOT elevated | 0 | | threading a context whose elevation is a run-time fact | 93 | | how the instrument reached the site | count | | :--- | ---: | -| receiver carried a readable engine type | 189 | +| receiver carried a readable engine type | 190 | | receiver erased, placed by the object NAME | 31 | | receiver erased, placed by an `object: string` PARAMETER | 5 | | receiver erased, placed by an `UNTYPED_RECEIVERS` row | 5 | @@ -247,7 +247,7 @@ cannot read, and they are neither in nor out. | object name spelled inline | 107 | | object name spelled through a `const` | 49 | | object name is an `object: string` parameter | 7 | -| object name is some other run-time expression | 67 | +| object name is some other run-time expression | 68 | ### Subtractions the census could NOT defend — enforced @@ -297,12 +297,12 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-19 at `fa6a837e7`. +Measured on 2026-10-20 at `1f2b0cb73`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 672 | -| engine-shaped types recognised | 79 | +| tracked non-test sources scanned | 674 | +| engine-shaped types recognised | 80 | | declared objects in the registry | 118 | | same-named calls subtracted as non-engine | 166 | diff --git a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md index d26ced0960a..eaa6d70fd3f 100644 --- a/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md +++ b/docs/audits/2026-08-tenant-audit-write-call-sites.counts.md @@ -33,17 +33,17 @@ silent, and `node scripts/tenant-audit-census.mjs --write` is the resolution. | Measure | Value | |---|---:| -| Write call sites | 230 | +| Write call sites | 231 | | Object name statically decidable | 156 | -| Object name chosen at run time | 74 | +| Object name chosen at run time | 75 | | Against a tenancy-enabled object | 155 | | Against an object declaring tenancy off | 1 | -| Threading a tenant context | 172 | +| Threading a tenant context | 173 | | Provably carrying none | 8 | | …and decidably tenancy-enabled | 2 | | Options argument unreadable | 50 | | …and decidably tenancy-enabled | 29 | -| Threading a decidably elevated context | 129 | +| Threading a decidably elevated context | 130 | | Threading a decidably non-elevated context | 0 | | Threading a context of undecidable elevation | 93 | @@ -90,12 +90,12 @@ holds still. They are required to be HERE and to say WHEN they were true; their values are not compared. The reasoning, and the measurement behind it, are in `scripts/check-tenant-audit-census.mjs`. -Measured on 2026-10-19 at `fa6a837e7`. +Measured on 2026-10-20 at `1f2b0cb73`. | corpus scale (not enforced) | count | | :--- | ---: | -| tracked non-test sources scanned | 672 | -| engine-shaped types recognised | 79 | +| tracked non-test sources scanned | 674 | +| engine-shaped types recognised | 80 | | declared objects in the registry | 118 | | same-named calls subtracted as non-engine | 166 | @@ -120,6 +120,7 @@ Measured on 2026-10-19 at `fa6a837e7`. | `packages/plugins/plugin-audit/src/activity-changed-fields-backfill.ts` | `update` | `sys_activity` | enabled | elevated | 1 | | `packages/plugins/plugin-audit/src/auth-event-audit.ts` | `insert` | `sys_audit_log` | enabled | options unreadable | 1 | | `packages/plugins/plugin-audit/src/comment-parent-pair-backfill.ts` | `update` | `sys_comment` | enabled | elevated | 1 | +| `packages/plugins/plugin-audit/src/mail-log-body-migration.ts` | `update` | `object` | undecidable | elevated | 1 | | `packages/plugins/plugin-audit/src/read-audit.ts` | `insert` | `sys_audit_log` | enabled | options unreadable | 1 | | `packages/plugins/plugin-audit/src/stored-metadata-body-migration.ts` | `update` | `object` | undecidable | elevated | 1 | | `packages/plugins/plugin-auth/src/admin-import-users.ts` | `insert` | `sys_audit_log` | enabled | options unreadable | 1 | diff --git a/docs/qa/platform-checklist/areas/cli.json b/docs/qa/platform-checklist/areas/cli.json index 4815a651a8d..f5d45a286b3 100644 --- a/docs/qa/platform-checklist/areas/cli.json +++ b/docs/qa/platform-checklist/areas/cli.json @@ -212,7 +212,7 @@ "title": "os migrate: bare command is a read-only plan, apply is safe-by-default, re-runs are idempotent, and --json exits 0 on success (#4873)", "since": "v15", "status": "active", - "revision": 5, + "revision": 6, "priority": "P1", "surface": "cli", "personas": ["operator (local shell)"], @@ -298,14 +298,15 @@ "security-catalog-overlays", "comment-reactions", "activity-changed-fields", - "comment-parent-pair" + "comment-parent-pair", + "audit-mail-bodies" ], "traps": ["stale-dist"], "automated": { "kind": "unit", "ref": "packages/cli/src/utils/format.exit-code.test.ts — covers acceptance[2] ONLY (#4873: emitJson / emitText leave the exit code alone on success and a duration can no longer reach the exit-code slot); acceptance[0], [1] and [3]–[6] need the run" }, "source": [ "packages/cli/src/commands/migrate/index.ts + plan.ts (#2186 bare-command-is-plan; #3917 enforced never-mutates; occupancy warning) + apply.ts (--allow-destructive / --force / --yes / in_sync)", "packages/cli/src/utils/format.ts#CliExitCode (CliExitCode narrows the emitJson exit slot — the #4873 fix, commit 83df2fd)", - "packages/cli/src/commands/migrate/ (the 18 registered subcommands under `os migrate`, besides the bare command, enumerated as variants — oclif pattern strategy, one default-exported Command per non-test file)", + "packages/cli/src/commands/migrate/ (the 19 registered subcommands under `os migrate`, besides the bare command, enumerated as variants — oclif pattern strategy, one default-exported Command per non-test file)", "sibling item cli.migrate-duplicates-inventory (the duplicates subcommand's own contract — cross-referenced, not duplicated)" ], "history": [ @@ -328,6 +329,12 @@ "date": "2026-10-19", "change": "steps[1] and automated.ref corrected — checklist-accuracy findings of the 18.0 QA sweep (QA run #24551, cli.migrate-plan-apply-json step 2; round 2's Tier 1 record #24598, which names this ref among those that over-claim clause coverage; wave anchor #24579 §3). On an os-dev-synced DB the pending plan also lists the lifecycle-classed tables: at base 9a7838edf5 the one-shot boot provisions the telemetry sibling only under isDevelopmentBoot (packages/cli/src/utils/schema-migrate.ts) and resolveTelemetryDbPath declines on a non-dev boot without OS_TELEMETRY_DB. The one pin asserts only the #4873 exit-code contract, so the ref now says it covers acceptance[2] alone. No clause changed", "ref": "#24649" + }, + { + "revision": 6, + "date": "2026-10-20", + "change": "variants: audit-mail-bodies added, the 19th registered subcommand (#24693: the one-off mask of the mail bodies that sys_audit_log / sys_activity copies of sys_email and sys_identity_email rows still hold in clear). The source note's count moves from 18 to 19 with it. No clause changed", + "ref": "#24693" } ] }, diff --git a/packages/cli/src/commands/migrate/audit-mail-bodies.ts b/packages/cli/src/commands/migrate/audit-mail-bodies.ts new file mode 100644 index 00000000000..eeaf91cb9be --- /dev/null +++ b/packages/cli/src/commands/migrate/audit-mail-bodies.ts @@ -0,0 +1,235 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +import { Command, Flags } from '@oclif/core'; +import chalk from 'chalk'; +import { createInterface } from 'node:readline'; +import type { IObjectQLEngine } from '@objectstack/spec/contracts'; +import { + printHeader, + printSuccess, + printWarning, + printError, + printInfo, + printStep, + createTimer, + emitJson, + isExitSignal, + errorCodeFields, +} from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; +import { describeOccupancy, JOURNAL_MODE_SWITCH_NOTE } from '../../utils/sqlite-occupancy.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; +import { absentTableReads } from '../../utils/absent-table-reads.js'; + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((resolve) => rl.question(question, resolve)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} + +/** + * `os migrate audit-mail-bodies` — the one-off mask of the mail-log bodies the + * audit writer copied in clear into `sys_audit_log` / `sys_activity` (#24693). + * + * The audit writer COPIES a `sys_email` / `sys_identity_email` row into + * `sys_audit_log.new_value` / `old_value` and both halves of + * `sys_activity.metadata`. Before #24625 that copy carried the mail BODY in + * clear, reset, verification, magic-link and invitation links included: a + * second, at-rest store of them. The writer masks those columns now, but only + * on NEW writes; rows copied before keep the body. This command masks them with + * the writer's own mask (`@objectstack/plugin-audit`'s + * `migrateMailLogBodyCopies`). Operators run it once after upgrading, dry run + * first. + * + * Its own command, not a step of `os migrate organization-ownership`: it masks + * every copy of both mail logs, so it needs no identity-mail selector and gives + * the same result run before or after the ceremony, on a database the ceremony + * never runs on too. Only the copy column of a copy row is written. + * + * Dry run by default (writes no row), `--apply` to mask. Idempotent: a masked + * body is the mask, so a second run masks nothing — re-running and reading a + * clean report is the verification. The report is counts only. No + * `sys_migration` flag is recorded: nothing gates irreversible behaviour on it. + * The flags are `os migrate audit-metadata-bodies`'s, one for one. + */ +export default class MigrateAuditMailBodies extends Command { + static override description = + 'Mask the mail bodies (body_text, body_html) of sys_email / sys_identity_email rows that the audit writer ' + + 'copied in clear into sys_audit_log / sys_activity before it masked them. Dry run by default; --apply writes. ' + + 'Reports counts only.'; + + static override examples = [ + '$ os migrate audit-mail-bodies', + '$ os migrate audit-mail-bodies --apply', + '$ os migrate audit-mail-bodies --apply --yes --json', + ]; + + static override flags = { + 'database-url': Flags.string({ + description: 'Database URL to migrate (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + apply: Flags.boolean({ + description: 'Mask the copies that hold a mail body in clear (default is a read-only dry run)', + default: false, + }), + yes: Flags.boolean({ char: 'y', description: 'Skip the --apply confirmation prompt', default: false }), + force: Flags.boolean({ + description: 'Apply even when another process is using the database (SQLite occupancy check)', + default: false, + }), + json: Flags.boolean({ description: 'Output as JSON (implies non-interactive; requires --yes to apply)' }), + }; + + async run(): Promise { + const { flags } = await this.parse(MigrateAuditMailBodies); + const timer = createTimer(); + const apply = flags.apply; + + if (!flags.json) printHeader('Migrate · audit-mail-bodies'); + const occupancy = await probeMigrationTarget(flags['database-url']); + if (occupancy.status === 'busy' && apply && !flags.force) { + if (flags.json) { + await emitJson( + { error: 'database_busy', database: occupancy.filename, signal: occupancy.signal, detail: occupancy.detail, hint: OCCUPANCY_HINT }, + 0, + { compact: true }, + ); + this.exit(1); + return; + } + printError(describeOccupancy(occupancy)); + printWarning(OCCUPANCY_HINT); + this.exit(1); + return; + } + if (occupancy.status === 'busy' && !flags.json) { + printWarning(apply + ? `--force: ${describeOccupancy(occupancy)} Masking anyway — the live process may write rows mid-walk.` + : `${describeOccupancy(occupancy)} The dry run below writes no row or table, but its counts may shift while that process is running. ${JOURNAL_MODE_SWITCH_NOTE}`); + } + if (occupancy.status === 'unknown' && !flags.json) { + printWarning(`Could not check whether the database is in use — ${occupancy.detail}`); + } + + if (apply && !flags.yes) { + if (flags.json || !process.stdin.isTTY) { + if (flags.json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); + this.exit(1); + return; + } + printWarning('Apply mode rewrites audit/activity rows. Re-run with --yes to confirm, or run without --apply to preview.'); + this.exit(1); + return; + } + const ok = await confirm( + chalk.bold('\nMask the mail body in every audit/activity copy of a mail-log row on this database? [y/N] '), + ); + if (!ok) { + printInfo('Aborted — no changes made.'); + return; + } + } + + if (!flags.json) { + printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (dry run)…'); + } + + let stack; + try { + // [#21349] The dry run boots READ-ONLY — the same boot `os migrate plan` + // takes: `deferSchemaDdl` holds schema DDL back and suppresses the + // artifact's inline seed loader (whose upserts rewrite every seeded row + // of the app's tables), and `readOnlyProbe` keeps a missing sqlite file + // from being created. `--apply` keeps the plain boot, unchanged. + stack = await bootSchemaStack({ + jsonOutput: flags.json, + databaseUrl: flags['database-url'], + extraPlugins: await buildDataMigrationPlugins({ audit: true }), + ...(apply ? {} : { deferSchemaDdl: true, readOnlyProbe: true }), + }); + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + return; + } + + try { + // `SchemaStack.kernel` is untyped, so the slot's contract is stated on the + // RESULT — the `objectql` slot serves `IObjectQLEngine`, whose `find` / + // `update` are exactly what the rewrite calls. + const engine = stack.kernel.getService('objectql') as IObjectQLEngine | undefined; + if (typeof engine?.find !== 'function' || typeof engine?.update !== 'function') { + throw new Error('No ObjectQL engine on this stack — cannot rewrite audit rows.'); + } + + const { migrateMailLogBodyCopies } = await import('@objectstack/plugin-audit'); + + const logger = flags.json + ? { info: (m: string) => console.error(m), warn: (m: string) => console.error(m) } + : { info: (m: string) => printInfo(m), warn: (m: string) => printWarning(m) }; + + // [#21552] The read view `os migrate audit-metadata-bodies` reads the same + // two tables through: a table the read-only boot MEASURED absent holds no + // copy and answers no rows without the read; `sys_activity`, rotation- + // managed and a view over time-sharded tables, is always read, and only + // the refusal of a table that is not there reads as no rows. ⛔ Any other + // refused read is still counted in `failures` and still exits non-zero. + // `--apply` booted plain; its writes go to the engine itself. + const reads = absentTableReads(stack, (object) => engine.getObject(object)); + const readView: Pick = { + find: (object, query, options) => reads.rows(object, () => engine.find(object, query, options)), + update: (object, data, options) => engine.update(object, data, options), + }; + + const report = await migrateMailLogBodyCopies(readView, logger, { apply }); + + if (flags.json) { + reads.notice(true); + await emitJson({ database: stack.dbLabel, apply, report, duration: timer.elapsed() }); + if (report.failures > 0) this.exit(1); + return; + } + + printInfo(`Database: ${chalk.white(stack.dbLabel)}`); + reads.notice(false); + console.log(''); + for (const [object, stats] of Object.entries(report.byObject)) { + console.log( + ` ${chalk.white(object)}: ${stats.scanned} mail-log cop(ies) scanned, ${stats.rewritten} ${apply ? 'masked' : 'to mask'}`, + ); + } + console.log(''); + + if (report.failures > 0) { + printError(`${report.failures} read(s) or row(s) failed — re-run to finish them.`); + } else if (apply && report.rewritten > 0) { + printSuccess( + `Masked the mail body in ${report.rewritten} audit/activity row(s). Re-run any time — it only revisits copies still holding a body.`, + ); + } else if (!apply && report.rewritten > 0) { + printInfo(`Dry run only — ${report.rewritten} row(s) would be masked. Re-run with --apply.`); + } else { + printSuccess('Nothing to mask — no audit/activity copy of a mail-log row holds a mail body in clear.'); + } + console.log(chalk.dim(` ${timer.display()}`)); + console.log(''); + if (report.failures > 0) this.exit(1); + } catch (error: any) { + if (isExitSignal(error)) throw error; + if (flags.json) { await emitJson({ error: error.message, ...errorCodeFields(error) }, 0, { compact: true }); this.exit(1); } + printError(error.message || String(error)); + this.exit(1); + } finally { + await stack.shutdown(); + } + } +} diff --git a/packages/cli/src/commands/migrate/data-commands.absent-database.integration.test.ts b/packages/cli/src/commands/migrate/data-commands.absent-database.integration.test.ts index 4fb87952a03..5ee50d83a60 100644 --- a/packages/cli/src/commands/migrate/data-commands.absent-database.integration.test.ts +++ b/packages/cli/src/commands/migrate/data-commands.absent-database.integration.test.ts @@ -71,6 +71,11 @@ * [#21573] `os migrate unmapped-columns` joined the roster later, born with * the same answer: it asks `tableAbsent` before the differ. Its control row is * a retired field's column on `os21529_contact`, holding a value. + * + * [#24693] `os migrate audit-mail-bodies` joined it born with the answer too: + * it reads `sys_audit_log` and `sys_activity` through the view + * `audit-metadata-bodies` reads them through. Its control rows are an 18.0 + * writer's copies of a mail-log row, planted below the writer. */ import { describe, it, expect, beforeAll, afterAll } from 'vitest'; @@ -433,6 +438,18 @@ await k(shard.name).insert({ }, }), }); +// [#24693] An 18.0 writer's copies of a mail-log row: the body in clear, a +// sentinel. The activity row carries its read-rule stamp, so it is no row of +// work for \`activity-changed-fields\`. +await k('sys_audit_log').insert({ + id: 'aud_24693', object_name: 'sys_email', record_id: 'em_24693', action: 'create', + new_value: JSON.stringify({ id: 'em_24693', subject: 's', body_text: 'mail-body-sentinel-24693', body_html: null }), +}); +await k(shard.name).insert({ + id: 'act_24693', timestamp: new Date().toISOString(), type: 'create', summary: 'created', object_name: 'sys_email', record_id: 'em_24693', + changed_fields: '[]', + metadata: JSON.stringify({ old: null, new: { id: 'em_24693', subject: 's', body_text: 'mail-body-sentinel-24693', body_html: null } }), +}); await k('sys_comment').insert({ id: 'cmt_23857', thread_id: 'sys_metadata:m_21552', body: 'planted below the writer', author_id: 'u1', created_at: new Date().toISOString(), @@ -495,6 +512,21 @@ const DOORS: readonly Door[] = [ expect(doc.report.byObject.sys_activity.rewritten).toBe(1); }, }, + { + name: 'migrate audit-mail-bodies', + argv: ['migrate', 'audit-mail-bodies'], + empty: (doc) => { + expect(doc).toMatchObject({ apply: false, report: { scanned: 0, rewritten: 0, failures: 0 } }); + expect(Object.keys(doc.report.byObject).sort()).toEqual(['sys_activity', 'sys_audit_log']); + }, + humanEmpty: /Nothing to mask/, + tables: ['sys_audit_log', 'sys_activity'], + work: (doc) => { + // The seeded clear mail-log copy is the one row to mask on each table; + // the rotation-managed table's rows are READ, as for audit-metadata-bodies. + expect(doc.report).toMatchObject({ failures: 0, byObject: { sys_audit_log: { rewritten: 1 }, sys_activity: { rewritten: 1 } } }); + }, + }, { name: 'migrate activity-changed-fields', argv: ['migrate', 'activity-changed-fields'], diff --git a/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts b/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts index efbed27e1e0..af35d2323fe 100644 --- a/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts +++ b/packages/cli/src/utils/schema-migrate.one-shot-family.integration.test.ts @@ -58,6 +58,7 @@ import MigrateAccountIssuer from '../commands/migrate/account-issuer.js'; import MigrateActivityChangedFields from '../commands/migrate/activity-changed-fields.js'; import MigrateCommentParentPair from '../commands/migrate/comment-parent-pair.js'; import MigrateApply from '../commands/migrate/apply.js'; +import MigrateAuditMailBodies from '../commands/migrate/audit-mail-bodies.js'; import MigrateAuditMetadataBodies from '../commands/migrate/audit-metadata-bodies.js'; import MigrateCommentReactions from '../commands/migrate/comment-reactions.js'; import MigrateDuplicates from '../commands/migrate/duplicates.js'; @@ -176,6 +177,11 @@ const CALLERS: Record = { write: [{ label: 'migrate apply --yes', argv: ['--yes', '--database-url', '@DB@', '--json'] }], note: 'the apply step itself: its preview is `os migrate plan`', }, + 'commands/migrate/audit-mail-bodies.ts': { + run: invoke(MigrateAuditMailBodies), + noWrite: [{ label: 'migrate audit-mail-bodies', argv: ['--database-url', '@DB@', '--json'] }], + write: [{ label: 'migrate audit-mail-bodies --apply', argv: ['--apply', '--yes', '--database-url', '@DB@', '--json'] }], + }, 'commands/migrate/audit-metadata-bodies.ts': { run: invoke(MigrateAuditMetadataBodies), noWrite: [{ label: 'migrate audit-metadata-bodies', argv: ['--database-url', '@DB@', '--json'] }], diff --git a/packages/cli/test/json-stdout-purity.e2e.test.ts b/packages/cli/test/json-stdout-purity.e2e.test.ts index 4ee761b6936..c2c269dd467 100644 --- a/packages/cli/test/json-stdout-purity.e2e.test.ts +++ b/packages/cli/test/json-stdout-purity.e2e.test.ts @@ -105,6 +105,10 @@ const FAMILY: Record = { // its own, which `--json` points at stderr — a third pollution source beside // the two in the header, so this member is worth driving, not only listing. 'migrate audit-metadata-bodies': [], + // [#24693] The same shape over the same two tables: a dry run by default, + // reading `sys_audit_log` / `sys_activity` and writing nothing, its rewrite + // runner logging through a sink `--json` points at stderr. + 'migrate audit-mail-bodies': [], // A dry run is its DEFAULT and the only form driven here: without `--apply` // it boots read-only (plugin-audit's objects registered), reads // `sys_comment` below the data door and writes nothing. diff --git a/packages/plugins/plugin-audit/src/audit-writers.ts b/packages/plugins/plugin-audit/src/audit-writers.ts index bc83986939f..9dfac948191 100644 --- a/packages/plugins/plugin-audit/src/audit-writers.ts +++ b/packages/plugins/plugin-audit/src/audit-writers.ts @@ -18,8 +18,8 @@ import type { IDataEngine } from '@objectstack/spec/contracts'; // for the same value, so it honours the same declaration the same way — the // producer's declaration consumed, not a masking rule of this package's own. import { SECRET_MASK, collectInternalReadFields, collectMaskedReadFields } from '@objectstack/objectql/core'; -// [#24625] The mail logs and their body columns, named once beside the objects. -import { MAIL_LOG_BODY_FIELDS, isMailLogObject } from '@objectstack/platform-objects/audit'; +// [#24625] The mask of a mail-log row's body columns in a copy of it. +import { maskMailLogBodyCopy } from './mail-log-body-mask.js'; // [#7230] ADR-0079's record display-name contract, imported rather than // re-derived. `resolveDisplayField` IS the definition of "which field is this // object's human title" (`nameField` → deprecated `displayNameField` alias → @@ -1760,12 +1760,8 @@ export function installAuditWriters( // identity rows still in `sys_email` cannot be told apart from tenant mail // by any column, so both logs' bodies are masked on every row. The log row // keeps the body behind its field gate; the copy never holds it in clear. - if (isMailLogObject(objectName)) { - for (const field of MAIL_LOG_BODY_FIELDS) { - if (!(field in out)) continue; - out[field] = out[field] == null ? null : SECRET_MASK; - } - } + // [#24693] One helper, the one the at-rest rewrite of older copies applies. + maskMailLogBodyCopy(objectName, out); // [#21120] A stored metadata BODY (`sys_metadata` / `sys_metadata_history`) // is not a top-level secret field, so the mask above never touches it. The // audit copy is a credential read exit exactly like `/meta`, so the body is diff --git a/packages/plugins/plugin-audit/src/index.ts b/packages/plugins/plugin-audit/src/index.ts index 859376bf4b3..51606773ee8 100644 --- a/packages/plugins/plugin-audit/src/index.ts +++ b/packages/plugins/plugin-audit/src/index.ts @@ -66,6 +66,19 @@ export type { StoredMetadataBodyMigrationEngine, StoredMetadataBodyMigrationReport, } from './stored-metadata-body-migration.js'; +// [#24693] The one-off rewrite of the mail-log bodies this writer copied in +// clear into `sys_audit_log` / `sys_activity` before it masked them, the pure +// planners the CLI command (`os migrate audit-mail-bodies`) drives, and the +// one mask both it and the writer apply. +export { maskMailLogBodyCopy } from './mail-log-body-mask.js'; +export { + migrateMailLogBodyCopies, + planMailLogBodyAuditRowPatch, + planMailLogBodyActivityRowPatch, + MAIL_LOG_BODY_COPY_OBJECTS, + MAIL_LOG_BODY_MIGRATION_PAGE_SIZE, +} from './mail-log-body-migration.js'; +export type { MailLogBodyMigrationEngine, MailLogBodyMigrationReport } from './mail-log-body-migration.js'; // [ADR-0139 E6] The stored-row backfill of `sys_activity`'s two writer-stamped // read-rule columns, and the one planner it, the CRUD mirror and the // before-insert hook stamp by. The CLI command diff --git a/packages/plugins/plugin-audit/src/mail-log-body-mask.ts b/packages/plugins/plugin-audit/src/mail-log-body-mask.ts new file mode 100644 index 00000000000..a6fb60c7b8a --- /dev/null +++ b/packages/plugins/plugin-audit/src/mail-log-body-mask.ts @@ -0,0 +1,40 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#24625] The one mask a COPY of a mail-log row gets, shared by the two + * places that hold such a copy to it: the audit writer (`ledgerView` in + * `audit-writers.ts`), which masks every new `sys_audit_log` / `sys_activity` + * copy at write time, and [#24693] the at-rest rewrite of the copies written + * before it did (`mail-log-body-migration.ts`). One function, so a copy the + * writer makes today and a copy the rewrite repaired read the same. + * + * The object names and the body columns are `@objectstack/platform-objects`' + * (`mail-log-body.ts`); the mask value is the credential read mask + * (`SECRET_MASK`). Neither is spelled here. + */ + +import { SECRET_MASK } from '@objectstack/objectql/core'; +import { MAIL_LOG_BODY_FIELDS, isMailLogObject } from '@objectstack/platform-objects/audit'; + +/** + * Mask the body columns of one copy of an `objectName` row, IN PLACE, the way + * a credential field is masked: a set body becomes {@link SECRET_MASK}, an + * unset one stays `null` (so "no body" and "a body was sent" remain + * distinguishable), and a column the copy does not carry is never invented. A + * copy of any object other than the two mail logs is left as it is. + * + * Returns whether any column changed — `false` on a copy already masked, which + * is what makes the at-rest rewrite idempotent. + */ +export function maskMailLogBodyCopy(objectName: unknown, copy: Record): boolean { + if (typeof objectName !== 'string' || !isMailLogObject(objectName)) return false; + let changed = false; + for (const field of MAIL_LOG_BODY_FIELDS) { + if (!(field in copy)) continue; + const masked = copy[field] == null ? null : SECRET_MASK; + if (copy[field] === masked) continue; + copy[field] = masked; + changed = true; + } + return changed; +} diff --git a/packages/plugins/plugin-audit/src/mail-log-body-migration.test.ts b/packages/plugins/plugin-audit/src/mail-log-body-migration.test.ts new file mode 100644 index 00000000000..04f777a4a84 --- /dev/null +++ b/packages/plugins/plugin-audit/src/mail-log-body-migration.test.ts @@ -0,0 +1,267 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#24693] The at-rest rewrite of the mail-log bodies the audit writer copied + * in clear before #24625 — pure planners, then the paged runner against a fake + * engine. The rows are 18.0-shaped: the copies the 18.0 writer stored for a + * `sys_email` create and a status update, and for a `sys_identity_email` + * delete, on all four snapshot sides (`new_value`, `old_value`, and the two + * halves of `sys_activity.metadata`). + * + * ⛔ The bodies here are sentinels, never a link, a token or a real mail body, + * and no title or message names one. + */ + +import { isDeepStrictEqual } from 'node:util'; +import { describe, expect, it, vi } from 'vitest'; +// [engine-double-contract] A fake engine's update must refuse exactly what +// ObjectQL refuses — opened with the producer's own predicate, the shape +// `stored-metadata-body-migration.test.ts` takes. +import { assertEngineUpdateDispatch } from '@objectstack/metadata-core'; +import { SECRET_MASK } from '@objectstack/objectql/core'; +import { maskMailLogBodyCopy } from './mail-log-body-mask.js'; +import { + migrateMailLogBodyCopies, + planMailLogBodyActivityRowPatch, + planMailLogBodyAuditRowPatch, + type MailLogBodyMigrationEngine, +} from './mail-log-body-migration.js'; + +type Row = Record; + +const TEXT = 'mail-body-sentinel-text'; +const HTML = 'mail-body-sentinel-html'; + +/** A mail-log row as the 18.0 writer copied it whole (a create snapshot). */ +const mailRow = (id: string, body: { text?: string | null; html?: string | null } = {}): Row => ({ + id, + organization_id: 'org_1', + to: '["someone@example.invalid"]', + subject: 'Subject kept', + status: 'sent', + related_object: 'sys_user', + related_id: 'usr_1', + body_text: body.text === undefined ? TEXT : body.text, + body_html: body.html === undefined ? HTML : body.html, +}); + +/** The 18.0 sys_audit_log rows: a create (new_value) and a delete (old_value) carry the body; a status update does not. */ +const auditRows = (): Row[] => [ + { id: 'al_1', object_name: 'sys_email', action: 'create', record_id: 'em_1', updated_at: '2026-10-19T00:00:00.000Z', old_value: null, new_value: JSON.stringify(mailRow('em_1')) }, + { id: 'al_2', object_name: 'sys_email', action: 'update', record_id: 'em_1', old_value: JSON.stringify({ status: 'queued' }), new_value: JSON.stringify({ status: 'sent' }) }, + { id: 'al_3', object_name: 'sys_identity_email', action: 'delete', record_id: 'ie_1', old_value: JSON.stringify(mailRow('ie_1')), new_value: null }, + // An update diff whose two halves both name a body column (a resend that re-rendered it). + { id: 'al_4', object_name: 'sys_email', action: 'update', record_id: 'em_2', old_value: JSON.stringify({ body_text: TEXT }), new_value: JSON.stringify({ body_text: HTML }) }, + // Not a mail log: never touched, whatever its columns are called. + { id: 'al_5', object_name: 'crm_note', action: 'create', record_id: 'n_1', old_value: null, new_value: JSON.stringify({ id: 'n_1', body_text: TEXT }) }, +]; + +const TEXT_SOURCES = { summary: ['subject'], record_label: ['subject'] }; + +/** The 18.0 sys_activity rows: `metadata` is `{ old, new, …declaration }`. */ +const activityRows = (): Row[] => [ + { id: 'ac_1', object_name: 'sys_email', record_id: 'em_1', summary: 'Created Email "Subject kept"', metadata: JSON.stringify({ old: null, new: mailRow('em_1'), _textSources: TEXT_SOURCES }) }, + { id: 'ac_2', object_name: 'sys_identity_email', record_id: 'ie_1', summary: 'Deleted', metadata: JSON.stringify({ old: mailRow('ie_1', { html: null }), new: null, _textSources: TEXT_SOURCES }) }, + { id: 'ac_3', object_name: 'sys_email', record_id: 'em_1', summary: 'Status: queued → sent', metadata: JSON.stringify({ old: { status: 'queued' }, new: { status: 'sent' }, _textSources: TEXT_SOURCES }) }, + { id: 'ac_4', object_name: 'crm_note', record_id: 'n_1', summary: 'Created', metadata: JSON.stringify({ old: null, new: { body_text: TEXT } }) }, +]; + +/** Every stored string of `rows` that still holds a sentinel body in clear. */ +const clearBodies = (rows: Row[]): number => + rows.filter((r) => r.object_name !== 'crm_note' && JSON.stringify(r).match(/mail-body-sentinel/)).length; + +describe('maskMailLogBodyCopy — the writer\'s mask, the rewrite\'s mask', () => { + it('masks a set body, keeps an unset one null, and invents no column', () => { + const copy: Row = { subject: 's', body_text: TEXT, body_html: null }; + expect(maskMailLogBodyCopy('sys_email', copy)).toBe(true); + expect(copy).toEqual({ subject: 's', body_text: SECRET_MASK, body_html: null }); + }); + + it('is a no-op on a copy already masked, and on any other object', () => { + const masked: Row = { body_text: SECRET_MASK, body_html: null }; + expect(maskMailLogBodyCopy('sys_identity_email', masked)).toBe(false); + const other: Row = { body_text: TEXT }; + expect(maskMailLogBodyCopy('crm_note', other)).toBe(false); + expect(other.body_text).toBe(TEXT); + }); +}); + +describe('planMailLogBodyAuditRowPatch', () => { + it('masks a create\'s new_value and keeps every other key byte-identical', () => { + const [create] = auditRows(); + const patch = planMailLogBodyAuditRowPatch(create!); + expect(Object.keys(patch!)).toEqual(['new_value']); + expect(patch!.new_value).toBe(JSON.stringify({ ...mailRow('em_1'), body_text: SECRET_MASK, body_html: SECRET_MASK })); + }); + + it('masks a delete\'s old_value, and both halves of an update diff that names a body', () => { + const rows = auditRows(); + expect(Object.keys(planMailLogBodyAuditRowPatch(rows[2]!)!)).toEqual(['old_value']); + const both = planMailLogBodyAuditRowPatch(rows[3]!)!; + expect(JSON.parse(String(both.old_value))).toEqual({ body_text: SECRET_MASK }); + expect(JSON.parse(String(both.new_value))).toEqual({ body_text: SECRET_MASK }); + }); + + it('leaves a body-free update diff and another object\'s row alone', () => { + const rows = auditRows(); + expect(planMailLogBodyAuditRowPatch(rows[1]!)).toBeNull(); + expect(planMailLogBodyAuditRowPatch(rows[4]!)).toBeNull(); + }); + + it('is idempotent: the rewritten row plans nothing', () => { + const [create] = auditRows(); + const once = { ...create!, ...planMailLogBodyAuditRowPatch(create!) }; + expect(planMailLogBodyAuditRowPatch(once)).toBeNull(); + }); +}); + +describe('planMailLogBodyActivityRowPatch', () => { + it('masks each half of the pair and keeps the pair\'s other keys', () => { + const [created, deleted] = activityRows(); + const a = JSON.parse(String(planMailLogBodyActivityRowPatch(created!)!.metadata)); + expect(a).toEqual({ old: null, new: { ...mailRow('em_1'), body_text: SECRET_MASK, body_html: SECRET_MASK }, _textSources: TEXT_SOURCES }); + const d = JSON.parse(String(planMailLogBodyActivityRowPatch(deleted!)!.metadata)); + expect(d.old).toEqual({ ...mailRow('ie_1', { html: null }), body_text: SECRET_MASK }); + }); + + it('leaves a body-free pair, another object\'s row and a rewritten row alone', () => { + const rows = activityRows(); + expect(planMailLogBodyActivityRowPatch(rows[2]!)).toBeNull(); + expect(planMailLogBodyActivityRowPatch(rows[3]!)).toBeNull(); + const once = { ...rows[0]!, ...planMailLogBodyActivityRowPatch(rows[0]!) }; + expect(planMailLogBodyActivityRowPatch(once)).toBeNull(); + }); +}); + +/** + * The fake engine's WHERE matcher: the Filter Protocol subset the runner's page + * query speaks — equality, `$in`, `$gt` and `$and`. Any other combinator or + * operator is REFUSED, never read as a field name or as an equality on an object. + */ +function matchesWhere(r: Row, w: Record): boolean { + return Object.entries(w).every(([key, cond]) => { + if (key === '$and') return (cond as Record[]).every((c) => matchesWhere(r, c)); + if (key.startsWith('$')) throw new Error(`fake engine: combinator ${key} is not implemented`); + if (cond && typeof cond === 'object') { + const ops = Object.keys(cond); + if (ops.length !== 1 || !['$in', '$gt'].includes(ops[0]!)) { + throw new Error(`fake engine: operator ${ops.join(',')} is not implemented`); + } + if ('$in' in cond) return (cond.$in as unknown[]).includes(r[key]); + return String(r[key]) > String(cond.$gt); + } + return r[key] === cond; + }); +} + +describe('migrateMailLogBodyCopies (driven, paged)', () => { + /** A fake engine over two in-memory tables, answering the runner's keyset page query. */ + function fakeEngine(tables: Record, opts: { failRead?: string } = {}) { + const reads: Array<{ object: string; limit: unknown }> = []; + const writes: Array<{ object: string; data: Row; context: unknown }> = []; + const engine: MailLogBodyMigrationEngine = { + async find(object, query) { + if (object === opts.failRead) throw new Error('read refused'); + const q = (query ?? {}) as { where?: any; limit?: number }; + reads.push({ object, limit: q.limit }); + const rows = (tables[object] ?? []) + .filter((r) => matchesWhere(r, q.where ?? {})) + .sort((a, b) => String(a.id).localeCompare(String(b.id))); + // The caller's bound, applied AFTER the filter, by presence. + const page = typeof q.limit === 'number' ? rows.slice(0, q.limit) : rows; + return page.map((r) => ({ ...r })); + }, + async update(object, data, options) { + // Refuse exactly what the real engine's update dispatch refuses. + assertEngineUpdateDispatch(data, options === undefined ? undefined : { ...options }); + writes.push({ object, data: data as Row, context: (options as { context?: unknown } | undefined)?.context }); + const { id, ...rest } = data as Row; + const row = (tables[object] ?? []).find((r) => r.id === id); + if (row) Object.assign(row, rest); + return { id, ...rest }; + }, + }; + return { engine, reads, writes }; + } + const logger = () => ({ info: vi.fn(), warn: vi.fn() }); + + it('a dry run counts the clear copies and writes nothing', async () => { + const tables = { sys_audit_log: auditRows(), sys_activity: activityRows() }; + const before = JSON.stringify(tables); + const { engine } = fakeEngine(tables); + const report = await migrateMailLogBodyCopies(engine, logger()); + expect(report).toEqual({ + apply: false, + byObject: { sys_audit_log: { scanned: 4, rewritten: 3 }, sys_activity: { scanned: 3, rewritten: 2 } }, + scanned: 7, + rewritten: 5, + failures: 0, + }); + expect(JSON.stringify(tables)).toBe(before); + }); + + it('--apply leaves 0 clear copies, touches no other column or row, and a re-run is a no-op', async () => { + const tables = { sys_audit_log: auditRows(), sys_activity: activityRows() }; + expect(clearBodies([...tables.sys_audit_log, ...tables.sys_activity])).toBe(5); + const { engine } = fakeEngine(tables); + const report = await migrateMailLogBodyCopies(engine, logger(), { apply: true, pageSize: 2 }); + expect(report.rewritten).toBe(5); + expect(report.failures).toBe(0); + expect(clearBodies([...tables.sys_audit_log, ...tables.sys_activity])).toBe(0); + + // Non-body fields: each row's every other column is byte-identical, and the + // rows of another object are untouched entirely. + const originals = new Map([...auditRows(), ...activityRows()].map((r) => [r.id, r])); + for (const row of [...tables.sys_audit_log, ...tables.sys_activity]) { + const original = originals.get(row.id)!; + for (const key of Object.keys(original)) { + if (['new_value', 'old_value', 'metadata'].includes(key)) continue; + expect(row[key], `${String(row.id)}.${key}`).toBe(original[key]); + } + } + expect(tables.sys_audit_log[4]).toEqual(auditRows()[4]); + expect(tables.sys_activity[3]).toEqual(activityRows()[3]); + + const after = JSON.stringify(tables); + const again = await migrateMailLogBodyCopies(engine, logger(), { apply: true, pageSize: 2 }); + expect(again.rewritten).toBe(0); + expect(again.scanned).toBe(7); + expect(JSON.stringify(tables)).toBe(after); + }); + + it('writes the copy column alone, as the platform keeping the row\'s stored updated_at', async () => { + const tables = { sys_audit_log: auditRows(), sys_activity: activityRows() }; + const { engine, writes } = fakeEngine(tables); + await migrateMailLogBodyCopies(engine, logger(), { apply: true }); + expect(writes.map((w) => Object.keys(w.data).sort())).toEqual([ + ['id', 'new_value', 'updated_at'], + ['id', 'old_value'], + ['id', 'new_value', 'old_value'], + ['id', 'metadata'], + ['id', 'metadata'], + ]); + expect(writes[0]!.data.updated_at).toBe('2026-10-19T00:00:00.000Z'); + expect(writes.every((w) => isDeepStrictEqual(w.context, { isSystem: true, preserveAudit: true }))).toBe(true); + }); + + it('pages: every page is bounded and every mail-log copy is read exactly once', async () => { + const tables = { sys_audit_log: auditRows(), sys_activity: activityRows() }; + const { engine, reads } = fakeEngine(tables); + const report = await migrateMailLogBodyCopies(engine, logger(), { pageSize: 2 }); + expect(reads.every((r) => r.limit === 2)).toBe(true); + expect(report.byObject.sys_audit_log!.scanned).toBe(4); + expect(report.byObject.sys_activity!.scanned).toBe(3); + }); + + it('a table it cannot read is a failure, never a clean report', async () => { + const tables = { sys_audit_log: auditRows(), sys_activity: activityRows() }; + const { engine } = fakeEngine(tables, { failRead: 'sys_activity' }); + const log = logger(); + const report = await migrateMailLogBodyCopies(engine, log, { apply: true }); + expect(report.failures).toBe(1); + expect(report.byObject.sys_audit_log!.rewritten).toBe(3); + // Counts only: no log line names a body. + const lines = [...log.info.mock.calls, ...log.warn.mock.calls].flat().join('\n'); + expect(lines).not.toMatch(/mail-body-sentinel/); + }); +}); diff --git a/packages/plugins/plugin-audit/src/mail-log-body-migration.ts b/packages/plugins/plugin-audit/src/mail-log-body-migration.ts new file mode 100644 index 00000000000..51e8b38cdad --- /dev/null +++ b/packages/plugins/plugin-audit/src/mail-log-body-migration.ts @@ -0,0 +1,249 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * [#24693] One-off rewrite of the mail-log bodies this plugin's writer copied + * in clear before it masked them. + * + * `writeAudit` COPIES the audited row into `sys_audit_log.new_value` / + * `old_value` and into both halves of `sys_activity.metadata`. For a + * `sys_email` / `sys_identity_email` row that copy carried the mail BODY — + * reset, verification, magic-link and invitation links included — into a + * second, at-rest store. The writer masks those columns now (#24625, + * `ledgerView`), but only on NEW writes: rows copied before keep the body in + * clear. This module masks them, with the writer's own helper + * ({@link maskMailLogBodyCopy}), so a repaired copy and a new one read the same. + * + * ## Every mail-log copy, never a selector + * + * Both logs are masked on every row, as the writer does: `sys_identity_email` + * holds identity mail only, and the 18.0 identity rows still in `sys_email` + * cannot be told apart from tenant mail by any column. So the rewrite needs no + * identity selector and does not depend on the organization-ownership + * ceremony: run before it, it masks the `sys_email` copies; run after it, the + * activity copies the ceremony re-pointed name `sys_identity_email`, which is + * masked the same. The `sys_audit_log` copies keep the object name they were + * written under either way. + * + * ## Pure planner, driven runner + * + * The decision is pure ({@link planMailLogBodyAuditRowPatch} / + * {@link planMailLogBodyActivityRowPatch}); {@link migrateMailLogBodyCopies} + * pages the two tables through the engine and rewrites only the rows a planner + * changed — and in a row, only the column holding the copy, under + * `preserveAudit` with the row's stored `updated_at`, so no other column of the + * row moves. Nothing else in the copy moves either: the other keys of the + * snapshot and of the activity pair are re-serialized as they were read. + * + * ## Idempotent, its own verification + * + * A masked body is the mask, so a second run finds nothing to rewrite, and + * re-running and reading a clean report is the check (the posture + * `os migrate audit-metadata-bodies` takes). No `sys_migration` flag is + * recorded: nothing gates irreversible behaviour on it. The report holds counts + * only; no log line names a value. + */ + +import type { IDataEngine } from '@objectstack/spec/contracts'; +import { MAIL_LOG_OBJECTS } from '@objectstack/platform-objects/audit'; +import { maskMailLogBodyCopy } from './mail-log-body-mask.js'; +import type { MigrationLogger } from './stored-metadata-body-migration.js'; + +/** The two audit-family tables the writer copies a mail-log row into. */ +export const MAIL_LOG_BODY_COPY_OBJECTS = ['sys_audit_log', 'sys_activity'] as const; + +/** Rows read per page. */ +export const MAIL_LOG_BODY_MIGRATION_PAGE_SIZE = 500; + +/** The rewrite reads as the platform, and writes as the platform keeping each row's audit stamp. */ +const SYSTEM_CTX = { isSystem: true } as const; +const SYSTEM_PRESERVING_CTX = { isSystem: true, preserveAudit: true } as const; + +/** + * Mask the body columns of one parsed snapshot of an `objectName` row. A + * non-object, or a snapshot with no clear body, comes back unchanged. + */ +function maskedSnapshot(objectName: unknown, snapshot: unknown): { changed: boolean; value: unknown } { + if (!snapshot || typeof snapshot !== 'object' || Array.isArray(snapshot)) return { changed: false, value: snapshot }; + const copy = { ...(snapshot as Record) }; + return maskMailLogBodyCopy(objectName, copy) ? { changed: true, value: copy } : { changed: false, value: snapshot }; +} + +/** Parse → apply `step` → re-serialize one stored JSON string column. A non-string / unparseable value is left as-is. */ +function rewriteSerialized( + serialized: unknown, + step: (parsed: unknown) => { changed: boolean; value: unknown }, +): { changed: boolean; value: unknown } { + if (typeof serialized !== 'string' || serialized === '') return { changed: false, value: serialized }; + let parsed: unknown; + try { + parsed = JSON.parse(serialized); + } catch { + return { changed: false, value: serialized }; + } + const { changed, value } = step(parsed); + return changed ? { changed: true, value: JSON.stringify(value) } : { changed: false, value: serialized }; +} + +/** + * The patch for one `sys_audit_log` row about a mail log — its `new_value` + * (create, and the after half of an update diff) and `old_value` (delete, and + * the before half) with each clear body masked — or `null` when the row is + * about another object or holds no clear body. + */ +export function planMailLogBodyAuditRowPatch( + row: Record, +): { new_value?: unknown; old_value?: unknown } | null { + const patch: { new_value?: unknown; old_value?: unknown } = {}; + for (const column of ['new_value', 'old_value'] as const) { + const out = rewriteSerialized(row[column], (parsed) => maskedSnapshot(row.object_name, parsed)); + if (out.changed) patch[column] = out.value; + } + return Object.keys(patch).length > 0 ? patch : null; +} + +/** + * The patch for one `sys_activity` row about a mail log — the serialized pair + * `{ old, new }` with each half's clear body masked, every other key of the + * pair kept — or `null` when the row is about another object or holds no clear + * body. + */ +export function planMailLogBodyActivityRowPatch(row: Record): { metadata: unknown } | null { + const out = rewriteSerialized(row.metadata, (parsed) => { + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) return { changed: false, value: parsed }; + const pair = parsed as Record; + const next: Record = { ...pair }; + let changed = false; + for (const half of ['old', 'new'] as const) { + if (!(half in pair)) continue; + const masked = maskedSnapshot(row.object_name, pair[half]); + if (!masked.changed) continue; + next[half] = masked.value; + changed = true; + } + return { changed, value: changed ? next : parsed }; + }); + return out.changed ? { metadata: out.value } : null; +} + +/** The engine surface the runner needs — the real data-engine contract's members. */ +export type MailLogBodyMigrationEngine = Pick; + +/** What one migration run did (or would do in dry-run). Counts only. */ +export interface MailLogBodyMigrationReport { + apply: boolean; + /** Per copy table: mail-log copy rows read, and rows rewritten (or that would be). */ + readonly byObject: Record; + /** Mail-log copy rows read, total. */ + scanned: number; + /** Rows holding a clear body, total — rewritten under `apply`. */ + rewritten: number; + /** A failed read, or a row an `update` threw on (apply mode) — re-run to finish. */ + failures: number; +} + +function asArray(result: unknown): Record[] { + if (Array.isArray(result)) return result as Record[]; + if (result && typeof result === 'object' && Array.isArray((result as { records?: unknown }).records)) { + return (result as { records: Record[] }).records; + } + return []; +} + +/** + * Mask the mail-log body in every `sys_audit_log` / `sys_activity` copy of a + * `sys_email` or `sys_identity_email` row that still holds it in clear. + * Dry-run by default; `apply` writes. Idempotent. + */ +export async function migrateMailLogBodyCopies( + engine: MailLogBodyMigrationEngine, + logger: MigrationLogger, + options: { apply?: boolean; pageSize?: number } = {}, +): Promise { + const apply = options.apply === true; + const pageSize = options.pageSize && options.pageSize > 0 ? options.pageSize : MAIL_LOG_BODY_MIGRATION_PAGE_SIZE; + const report: MailLogBodyMigrationReport = { apply, byObject: {}, scanned: 0, rewritten: 0, failures: 0 }; + + const tables: ReadonlyArray<{ + object: (typeof MAIL_LOG_BODY_COPY_OBJECTS)[number]; + fields: readonly string[]; + plan: (row: Record) => Record | null; + }> = [ + { object: 'sys_audit_log', fields: ['id', 'object_name', 'new_value', 'old_value', 'updated_at'], plan: planMailLogBodyAuditRowPatch }, + { object: 'sys_activity', fields: ['id', 'object_name', 'metadata', 'updated_at'], plan: planMailLogBodyActivityRowPatch }, + ]; + + for (const { object, fields, plan } of tables) { + const stats = (report.byObject[object] = { scanned: 0, rewritten: 0 }); + // Keyset paging on `id`: a rewritten row keeps its id and is never read + // twice in one run. + let after: string | undefined; + for (;;) { + const copies: Record = { object_name: { $in: [...MAIL_LOG_OBJECTS] } }; + const where = after === undefined ? copies : { $and: [copies, { id: { $gt: after } }] }; + let rows: Record[]; + try { + rows = asArray( + await engine.find( + object, + { where, fields: [...fields], orderBy: [{ field: 'id', order: 'asc' }], limit: pageSize }, + { context: SYSTEM_CTX }, + ), + ); + } catch (e) { + // A table this run could not read is NOT a clean table: counted, so the + // run says so (and the door exits non-zero). + report.failures += 1; + logger.warn( + `[mail-log-body-migration] could not read ${object} — its remaining rows were NOT examined: ` + + `${(e as Error)?.message ?? String(e)}`, + ); + break; + } + if (rows.length === 0) break; + + for (const row of rows) { + report.scanned += 1; + stats.scanned += 1; + const patch = plan(row); + if (!patch) continue; + report.rewritten += 1; + stats.rewritten += 1; + if (!apply) continue; + const id = row.id; + if (typeof id !== 'string' || id === '') { + report.failures += 1; + logger.warn(`[mail-log-body-migration] ${object} row has no id — cannot rewrite it`); + continue; + } + try { + // A single-id write, as the platform: a system caller is exempt from + // the static-readonly strip every audit/activity field carries. + // `preserveAudit` with the stored `updated_at` keeps the row's own + // timestamp (the shape `comment-parent-pair-backfill.ts` takes): + // masking a copy is not an edit of the ledger row. + await engine.update( + object, + { ...patch, id, ...(row.updated_at !== undefined && row.updated_at !== null ? { updated_at: row.updated_at } : {}) }, + { context: SYSTEM_PRESERVING_CTX }, + ); + } catch (e) { + report.failures += 1; + logger.warn( + `[mail-log-body-migration] failed to rewrite ${object}/${id}: ${(e as Error)?.message ?? String(e)}`, + ); + } + } + + const last = rows[rows.length - 1]?.id; + if (rows.length < pageSize || typeof last !== 'string' || last === '') break; + after = last; + } + } + + logger.info?.( + `[mail-log-body-migration] ${apply ? 'masked' : 'would mask'} the mail body in ${report.rewritten} of ` + + `${report.scanned} audit/activity cop(ies) of a mail-log row` + + (report.failures > 0 ? ` (${report.failures} failed — re-run to finish)` : ''), + ); + return report; +} diff --git a/packages/qa/dogfood/test/audit-mail-bodies.dogfood.test.ts b/packages/qa/dogfood/test/audit-mail-bodies.dogfood.test.ts new file mode 100644 index 00000000000..048092f5b95 --- /dev/null +++ b/packages/qa/dogfood/test/audit-mail-bodies.dogfood.test.ts @@ -0,0 +1,225 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. +// test-budget: 20 — spawns the real producer once per single-cause fixture: the real `os` binary, three runs (dry run, apply, re-run) over one 18.0 fixture copy, each a full CLI boot; measured 12.84 s wall on a shared agent box, re-measure on CI +// +// [#24693] `os migrate audit-mail-bodies` over an 18.0 database: the mail-log +// bodies the 18.0 audit writer copied in clear into `sys_audit_log` and +// `sys_activity` are masked at rest, and nothing else moves. +// +// ## What was measured +// +// The 18.0 fixture (`upgrade-harness/fixture-18.0/`) holds `sys_audit_log` and +// `sys_activity` copies of its `sys_email` rows — the verification and +// invitation mail its sign-ups and its invitation sent — with `body_text` / +// `body_html` in clear: the writer masked them only from #24625 on, and only on +// new writes. Those bodies carry the sign-in and invitation links. +// +// ## What is pinned, on a fresh copy of the fixture +// +// 0. The fixture holds clear copies on both tables (a vacuous fixture is no +// pin). +// 1. The dry run counts them and writes nothing: every row of both tables is +// byte-identical afterwards. +// 2. `--apply` leaves 0 copies holding a body in clear. A rewritten row +// differs from the fixture in its copy column alone, and in that column +// only in the body keys, which now read the mask; every other row of +// both tables is byte-identical. +// 3. A second `--apply` masks nothing and changes no byte. +// +// The command runs as an operator runs it: the real binary, in the showcase's +// directory, over the database file — not a booted server, so the ownership +// boot guard's verdict on an 18.0 database is no part of it. +// +// ⛔ No body, link or token value appears in a title, an assertion message or a +// log line: every comparison is reported as a count or a boolean. +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { createRequire } from 'node:module'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { isDeepStrictEqual } from 'node:util'; +import { DatabaseSync } from 'node:sqlite'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { SECRET_MASK } from '@objectstack/spec/data'; +import { loadSqlIntoSqlite } from './upgrade-harness/sqlite-text-dump.js'; + +const FIXTURE = new URL('./upgrade-harness/fixture-18.0/', import.meta.url); +const fixtureSql = readFileSync(new URL('database.sql', FIXTURE), 'utf8'); +const SHOWCASE_DIR = fileURLToPath(new URL('../../../../examples/app-showcase/', import.meta.url)); +const CLI_ROOT = dirname(createRequire(import.meta.url).resolve('@objectstack/cli/package.json')); +const CLI_BIN = join(CLI_ROOT, 'bin', 'run.js'); + +const MAIL_LOGS = ['sys_email', 'sys_identity_email']; +const BODY_FIELDS = ['body_text', 'body_html'] as const; + +type Row = Record; +/** Every row of `sys_audit_log` and every `sys_activity` shard, keyed `table/id`. */ +type Snapshot = Map; + +function snapshot(file: string): Snapshot { + const db = new DatabaseSync(file, { readOnly: true }); + try { + const tables = (db + .prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND (name = 'sys_audit_log' OR name LIKE 'sys_activity__%') ORDER BY name") + .all() as Array<{ name: string }>).map((r) => r.name); + const out: Snapshot = new Map(); + for (const table of tables) { + for (const row of db.prepare(`SELECT * FROM "${table}" ORDER BY rowid`).all() as Row[]) out.set(`${table}/${String(row.id)}`, { ...row }); + } + return out; + } finally { + db.close(); + } +} + +const isAudit = (key: string): boolean => key.startsWith('sys_audit_log/'); +const copyColumnsOf = (key: string): string[] => (isAudit(key) ? ['new_value', 'old_value'] : ['metadata']); +const parse = (v: unknown): unknown => { + if (typeof v !== 'string' || v === '') return v; + try { + return JSON.parse(v); + } catch { + return v; + } +}; + +/** The snapshot sides of one copy row: `new_value` / `old_value`, or the two halves of the activity pair. */ +function sidesOf(key: string, row: Row): unknown[] { + if (isAudit(key)) return [parse(row.new_value), parse(row.old_value)]; + const pair = parse(row.metadata) as { old?: unknown; new?: unknown } | null; + return pair && typeof pair === 'object' ? [pair.old, pair.new] : []; +} + +const holdsClearBody = (side: unknown): boolean => + !!side && typeof side === 'object' && + BODY_FIELDS.some((f) => f in (side as Row) && (side as Row)[f] != null && (side as Row)[f] !== SECRET_MASK); + +/** Copy rows of a mail log holding a body in clear on any side, per table family. */ +function clearCopies(snap: Snapshot): { audit: number; activity: number } { + const out = { audit: 0, activity: 0 }; + for (const [key, row] of snap) { + if (!MAIL_LOGS.includes(String(row.object_name))) continue; + if (sidesOf(key, row).some(holdsClearBody)) out[isAudit(key) ? 'audit' : 'activity'] += 1; + } + return out; +} + +/** `side` with each body key it carries masked as the writer masks it — the expected rewritten copy. */ +function masked(side: unknown): unknown { + if (!side || typeof side !== 'object' || Array.isArray(side)) return side; + const out: Row = { ...(side as Row) }; + for (const f of BODY_FIELDS) if (f in out) out[f] = out[f] == null ? null : SECRET_MASK; + return out; +} + +/** The copy column as the fixture holds it, with each side's body masked and every other key kept. */ +function expectedCopyColumn(key: string, value: unknown): unknown { + const parsed = parse(value); + if (isAudit(key)) return masked(parsed); + const pair = parsed as Row; + return { ...pair, ...('old' in pair ? { old: masked(pair.old) } : {}), ...('new' in pair ? { new: masked(pair.new) } : {}) }; +} + +function runCommand(file: string, args: string[]): { status: number | null; report: Record | undefined; out: string } { + const res = spawnSync(process.execPath, [CLI_BIN, 'migrate', 'audit-mail-bodies', '--database-url', `file:${file}`, '--json', ...args], { + cwd: SHOWCASE_DIR, + encoding: 'utf8', + timeout: 300_000, + env: { ...process.env, NO_COLOR: '1' }, + }); + let report: Record | undefined; + try { + report = (JSON.parse(res.stdout) as { report?: Record }).report; + } catch { + report = undefined; + } + // stdout is the counts-only report; stderr is the boot's own log. Neither is + // printed on success, and on a failure only its tail is named. + return { status: res.status, report, out: `${res.stdout.slice(-2000)}\n${res.stderr.slice(-2000)}` }; +} + +describe('[#24693] os migrate audit-mail-bodies over an 18.0 database', () => { + const dir = mkdtempSync(join(tmpdir(), 'dogfood-audit-mail-bodies-')); + const file = join(dir, 'fixture.db'); + let fixture: Snapshot; + let fixtureClear: { audit: number; activity: number }; + + beforeAll(() => { + loadSqlIntoSqlite(fixtureSql, file); + // The fixture travels as a dump of its TABLES, so it carries no view; the + // 18.0 server that wrote it left `sys_activity` as the read view over its + // rotation shards (driver-sql `ensureRotation`), and every boot rebuilds + // it. The dry run boots read-only and creates nothing, so the view is put + // back as the server left it before anything reads the copy. + const db = new DatabaseSync(file); + try { + const shards = (db.prepare("SELECT name FROM sqlite_master WHERE type = 'table' AND name LIKE 'sys_activity__r%' ORDER BY name DESC").all() as Array<{ name: string }>).map((r) => r.name); + db.exec(`CREATE VIEW "sys_activity" AS ${shards.map((s) => `SELECT * FROM "${s}"`).join(' UNION ALL ')}`); + } finally { + db.close(); + } + fixture = snapshot(file); + fixtureClear = clearCopies(fixture); + }); + + afterAll(() => { + rmSync(dir, { recursive: true, force: true }); + }); + + it('0. the 18.0 fixture holds mail-log copies with a clear body on both tables', () => { + expect(fixtureClear.audit, 'sys_audit_log copies holding a clear body').toBeGreaterThan(0); + expect(fixtureClear.activity, 'sys_activity copies holding a clear body').toBeGreaterThan(0); + }); + + it('1. the dry run counts them and writes nothing', () => { + const dry = runCommand(file, []); + expect(dry.status, dry.out).toBe(0); + expect(dry.report?.apply).toBe(false); + expect(dry.report?.byObject?.sys_audit_log?.rewritten).toBe(fixtureClear.audit); + expect(dry.report?.byObject?.sys_activity?.rewritten).toBe(fixtureClear.activity); + expect(isDeepStrictEqual(snapshot(file), fixture), 'every audit/activity row is byte-identical after the dry run').toBe(true); + }, 300_000); + + it('2. --apply leaves 0 clear copies, and moves no byte but the masked bodies', () => { + const run = runCommand(file, ['--apply', '--yes']); + expect(run.status, run.out).toBe(0); + expect(run.report?.failures).toBe(0); + expect(run.report?.rewritten).toBe(fixtureClear.audit + fixtureClear.activity); + + const after = snapshot(file); + expect(clearCopies(after), 'copies holding a clear body after the cleanup').toEqual({ audit: 0, activity: 0 }); + expect([...after.keys()], 'no row added or removed').toEqual([...fixture.keys()]); + + let rewritten = 0; + for (const [key, before] of fixture) { + const now = after.get(key)!; + const copyColumns = copyColumnsOf(key); + const touched = copyColumns.some((c) => now[c] !== before[c]); + if (touched) rewritten += 1; + // Every column but the copy column, byte for byte. + for (const column of Object.keys(before)) { + if (touched && copyColumns.includes(column)) continue; + expect(now[column] === before[column], `${key}.${column} is byte-identical`).toBe(true); + } + if (!touched) continue; + expect(MAIL_LOGS.includes(String(before.object_name)), `${key} is a copy of a mail-log row`).toBe(true); + // In the copy column, only the body keys moved, to the mask. + for (const column of copyColumns) { + expect( + isDeepStrictEqual(parse(now[column]), expectedCopyColumn(key, before[column])), + `${key}.${column} differs from the fixture only in its masked body keys`, + ).toBe(true); + } + } + expect(rewritten).toBe(fixtureClear.audit + fixtureClear.activity); + }, 300_000); + + it('3. a second --apply masks nothing and changes no byte', () => { + const before = snapshot(file); + const again = runCommand(file, ['--apply', '--yes']); + expect(again.status, again.out).toBe(0); + expect(again.report?.rewritten).toBe(0); + expect(again.report?.failures).toBe(0); + expect(isDeepStrictEqual(snapshot(file), before), 'every audit/activity row is byte-identical after the re-run').toBe(true); + }, 300_000); +});