Skip to content

Commit 0f17ba7

Browse files
committed
wip(cli): a declared archive this boot does not connect reads not-connected
Claude-Session: https://claude.ai/code/session_01WQWCrorWxW5zDh4n6jCEF5 Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4fb9afa commit 0f17ba7

4 files changed

Lines changed: 146 additions & 39 deletions

File tree

‎.changeset/24899-archived-mail-bodies.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@ Clause-②: yes (widening)
1111

1212
- **What was left in clear.** `sys_audit_log` declares `lifecycle.archive: { after: '90d', to: 'archive', keep: '7y' }`. Past the hot window, the lifecycle Archiver copies its rows to the datasource named `archive` as they are, then deletes them hot. A `sys_email` / `sys_identity_email` copy archived before the audit writer masked the mail body still holds `body_text` / `body_html` in clear there, password-reset, verification, magic-link and invitation links included, for as long as the archive keeps it. The engine never routes `sys_audit_log` to the archive, so the hot run could not reach those rows.
1313
- **What changed.** After the hot run, the command reads the archive target each copy table declares (`sys_audit_log` only; `sys_activity` declares none) and runs the same mask, in the same dry-run or `--apply` mode, through that datasource's driver. Each archived row keeps every other column, `updated_at` included. The run is idempotent and its report is counts only. A hot copy masked first is archived masked, so one run covers both the copies already archived and the ones still to be.
14-
- **Which archives it reaches.** The command now connects the datasources the stack config declares, as `os serve` does. An archive datasource is declared with `autoConnect: true`, since nothing routes objects to it. If the archive is instead a runtime datasource created in Setup → Datasources, which `os serve` connects and this command does not, the run reports it `not-connected` and exits `1` rather than reporting a clean archive. Declare it in the config's `datasources` and re-run.
15-
- **New in the `--json` document:** an `archive` array, one entry per declared target: `{ datasource, objects, status, report? }`, with `status` one of `examined`, `none`, `not-connected` or `unknown`. The `report` key and its shape are unchanged.
14+
- **Which archives it reaches.** The command now connects the managed datasources the stack config declares, through the same connect service `os serve` uses. An archive datasource is declared with `autoConnect: true`, since nothing routes objects to it. It is NOT examined, and the run reports `not-connected` and exits `1` rather than reporting a clean archive, in two cases. In the first, the archive is declared but this boot did not connect it; the entry carries the connect status and reason. In the second, the archive is a runtime datasource created in Setup → Datasources, which `os serve` connects and this command does not; declare it in the config's `datasources` and re-run.
15+
- **What `none` means.** No datasource of that name is declared and connected on this boot, none was attempted, and no active runtime datasource row names one. That is all it checks. An archive a host registers in code outside the stack config is not reached and reads `none`. To mask it, declare it in the stack config's `datasources` with `autoConnect: true`.
16+
- **The command's boot can now fail on a declared datasource.** A bound, mapped or boot-critical declared datasource that cannot connect now fails the command's boot with exit `1`, as it fails `os serve`'s. `OS_ALLOW_DRIVER_CONNECT_FAILURE` is the same escape.
17+
- **Run it with the server stopped**, or re-run it once and read 0 to mask in both halves. A live server's Archiver can move a row between the hot run and the archive run.
18+
- **New in the `--json` document:** an `archive` array, one entry per declared target: `{ datasource, objects, status, report?, connection? }`, with `status` one of `examined`, `none`, `not-connected` or `unknown`. The `report` key and its shape are unchanged.
1619
- **New exports from `@objectstack/plugin-audit`:** `migrateArchivedMailLogBodyCopies(driver, logger, options)`, the runner over a datasource's driver, and an `objects` option on both runners, which limits a run to the named copy tables. Types `MailLogBodyArchiveDriver`, `MailLogBodyCopyObject` and `MailLogBodyMigrationOptions`.

‎content/docs/deployment/cli.mdx‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1501,9 +1501,15 @@ value.
15011501
15021502
**Idempotent**: a masked body is the mask, so a second run masks nothing, and
15031503
re-running is how an interrupted run is finished. A failed read or write exits
1504-
`1`. The copies age out with each table's retention (14 days for activity, 90
1505-
for the audit log, unless the deployment lengthens it); run it once after
1506-
upgrading rather than wait for that.
1504+
`1`. Activity copies age out with its 14-day retention. Audit-log copies do not
1505+
age out: with no archive datasource they are retained hot, and with one they
1506+
are moved there after 90 days and kept for seven years. Run it once after
1507+
upgrading.
1508+
1509+
**Run it with the server stopped**, or re-run it once and read 0 to mask in
1510+
both halves. A live server's Archiver sweeps on its own schedule, and a row it
1511+
moves between the hot run and the archive run can land in the archive
1512+
unmasked.
15071513
15081514
**The archive datasource too.** `sys_audit_log` declares
15091515
`lifecycle.archive: { after: '90d', to: 'archive', keep: '7y' }`: past the hot
@@ -1516,9 +1522,9 @@ every other column, `updated_at` included. A hot copy masked first is archived
15161522
masked, so one run covers both the copies already archived and the ones still
15171523
to be. `sys_activity` declares no archive: its copies are only ever hot.
15181524
1519-
The command connects the datasources the stack config declares, as `os serve`
1520-
does, so it reaches an archive declared there. A managed datasource nothing
1521-
routes to is connected only with `autoConnect: true`, which is how an archive
1525+
The command connects the managed datasources the stack config declares through
1526+
the same connect service `os serve` uses. A managed datasource nothing routes
1527+
to is connected only with `autoConnect: true`, which is how an archive
15221528
datasource is declared:
15231529
15241530
```ts
@@ -1533,12 +1539,17 @@ The `--json` document carries one `archive` entry per declared target, with a
15331539
| `status` | Meaning | Exit |
15341540
| :--- | :--- | :--- |
15351541
| `examined` | Connected and masked (or counted); `report` holds the counts | `0`, or `1` on a failed read or write |
1536-
| `none` | No such datasource, and no runtime datasource row of that name: the Archiver retains every row hot, so nothing was archived | `0` |
1537-
| `not-connected` | The database holds an active runtime datasource of that name (one created in Setup → Datasources). `os serve` connects it; this command does not, so the archive was **not** examined | `1` |
1542+
| `none` | No datasource of that name is declared and connected on this boot, none was attempted, and no active runtime datasource row names one. That is all it checked | `0` |
1543+
| `not-connected` | The archive was **not** examined. Either the stack config declares it and this boot did not connect it (`connection` carries the connect status and reason), or the database holds an active runtime datasource of that name (one created in Setup → Datasources), which `os serve` connects and this command does not | `1` |
15381544
| `unknown` | Not connected, and the datasource rows could not be read | `1` |
15391545
1540-
On `not-connected`, declare the archive datasource in the stack config's
1541-
`datasources` (with `autoConnect: true`), then re-run.
1546+
On `not-connected` for a runtime datasource, declare the archive datasource in
1547+
the stack config's `datasources` (with `autoConnect: true`), then re-run. For a
1548+
declared one, make it reachable from where the command runs, then re-run.
1549+
1550+
An archive that a host registers in code outside the stack config (a plugin
1551+
that publishes the driver itself) is not reached and reads `none`. To mask it,
1552+
declare it in the stack config's `datasources` with `autoConnect: true`.
15421553
15431554
#### `os migrate organization-ownership`
15441555

‎packages/cli/src/commands/migrate/audit-mail-bodies.archive.integration.test.ts‎

Lines changed: 51 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2-
// test-budget: 70 — spawns the real producer once per single-cause fixture: two served-shape boots that run the real Archiver, and five spawns of the source CLI (dry run, --apply, re-run over the archive fixture; --apply on a project with no archive; a dry run with a runtime archive row); 54.7 s, the slower of two local runs on a shared container, x 1.25, no CI reading yet
2+
// test-budget: 70 — spawns the real producer once per single-cause fixture: two served-shape boots that run the real Archiver, and six spawns of the source CLI (dry run, --apply, re-run over the archive fixture; --apply on a project with no archive; a dry run with a runtime archive row; --apply with a declared archive that cannot connect); 54.7 s, the slower of two local runs on a shared container, x 1.25, no CI reading yet
33

44
/**
55
* [#24899] `os migrate audit-mail-bodies` masks the mail-log bodies the
@@ -40,6 +40,10 @@
4040
* connects and this command's boot does not) is NOT reported clean: the
4141
* archive reads `not-connected` and the command exits 1. With neither a
4242
* connected datasource nor such a row it reads `none` and exits 0.
43+
* 7. A DECLARED `autoConnect: true` archive this boot cannot connect is NOT
44+
* reported clean either: gate (c) is not fail-fast, so the boot degrades
45+
* and leaves no driver, and the archive reads `not-connected`, carrying
46+
* the connect service's verdict, and the command exits 1.
4347
*
4448
* ⛔ The bodies are sentinels, never a link or a token, and every comparison is
4549
* reported as a count or a boolean.
@@ -51,6 +55,7 @@ import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs';
5155
import { tmpdir } from 'node:os';
5256
import { dirname, join, resolve } from 'node:path';
5357
import { fileURLToPath } from 'node:url';
58+
import { isDeepStrictEqual } from 'node:util';
5459
import Database from 'better-sqlite3';
5560

5661
const HERE = dirname(fileURLToPath(import.meta.url));
@@ -167,6 +172,11 @@ let hotAfterApply: Record<string, unknown>[];
167172
let archivedAfterAging: Record<string, unknown>[];
168173
let runtimeRowRun: Run;
169174
let noArchiveRun: Run;
175+
let unreachableDir: string;
176+
let unreachableRun: Run;
177+
let archivedAfterDry: Record<string, unknown>[];
178+
let hotBeforeDry: Record<string, unknown>[];
179+
let hotAfterDry: Record<string, unknown>[];
170180

171181
function sweep(mode: 'seed' | 'age'): { source: string; archived: number | null } {
172182
const r = spawnSync(process.execPath, ['--input-type=module', '-e', SWEEP_CHILD], {
@@ -243,7 +253,10 @@ beforeAll(async () => {
243253
seedDetail = sweep('seed');
244254
archivedBefore = auditRows(archiveDb);
245255

256+
hotBeforeDry = auditRows(mainDb);
246257
runs.dry = await runCommand(dir, mainDb, ['--json']);
258+
archivedAfterDry = auditRows(archiveDb);
259+
hotAfterDry = auditRows(mainDb);
247260
runs.apply = await runCommand(dir, mainDb, ['--apply', '--yes', '--json']);
248261
archivedAfterApply = auditRows(archiveDb);
249262
hotAfterApply = auditRows(mainDb);
@@ -267,14 +280,27 @@ beforeAll(async () => {
267280
db.close();
268281
}
269282
runtimeRowRun = await runCommand(runtimeRowDir, runtimeRowDb, ['--json']);
283+
284+
// A declared archive whose SQLite file cannot be opened: its directory is a
285+
// regular file, so the connect fails on every attempt.
286+
unreachableDir = mkdtempSync(join(tmpdir(), 'os-24899-unreachable-'));
287+
writeFileSync(join(unreachableDir, 'blocker'), 'not a directory');
288+
writeArtifact(unreachableDir, [
289+
{ name: 'archive', driver: 'sqlite', config: { filename: join(unreachableDir, 'blocker', 'archive.db') }, autoConnect: true },
290+
]);
291+
unreachableRun = await runCommand(unreachableDir, join(unreachableDir, 'main.db'), ['--apply', '--yes', '--json']);
270292
}, HOOK_TIMEOUT_MS);
271293

272294
afterAll(() => {
273295
if (dir) rmSync(dir, { recursive: true, force: true });
274296
if (runtimeRowDir) rmSync(runtimeRowDir, { recursive: true, force: true });
297+
if (unreachableDir) rmSync(unreachableDir, { recursive: true, force: true });
275298
});
276299

277300
const doc = (run: Run): any => JSON.parse(run.stdout);
301+
/** Two row sets hold the same rows, column for column — a boolean, so a failure prints no row. */
302+
const sameRows = (a: Record<string, unknown>[], b: Record<string, unknown>[]): boolean =>
303+
a.length === b.length && a.every((row, i) => isDeepStrictEqual(row, b[i]));
278304
const archiveOf = (run: Run): any => doc(run).archive.find((a: { datasource: string }) => a.datasource === 'archive');
279305

280306
describe('[#24899] os migrate audit-mail-bodies over an archive datasource', () => {
@@ -293,8 +319,10 @@ describe('[#24899] os migrate audit-mail-bodies over an archive datasource', ()
293319
objects: ['sys_audit_log'],
294320
report: { apply: false, failures: 0, byObject: { sys_audit_log: { scanned: 2, rewritten: 2 } } },
295321
});
296-
// Written only by --apply below: the dry run left the archive as the Archiver did.
297-
expect(clear(archivedBefore)).toBe(2);
322+
// Both stores, read after the dry run: neither moved.
323+
expect(sameRows(archivedAfterDry, archivedBefore)).toBe(true);
324+
expect(sameRows(hotAfterDry, hotBeforeDry)).toBe(true);
325+
expect(clear(archivedAfterDry)).toBe(2);
298326
});
299327

300328
it('2. --apply leaves 0 clear copies in either store, and every other archived row is untouched', () => {
@@ -304,14 +332,15 @@ describe('[#24899] os migrate audit-mail-bodies over an archive datasource', ()
304332
expect(clear(archivedAfterApply)).toBe(0);
305333
expect(clear(hotAfterApply)).toBe(0);
306334
const before = new Map(archivedBefore.map((r) => [r.id, r]));
307-
for (const row of archivedAfterApply) {
308-
const original = before.get(row.id)!;
309-
for (const key of Object.keys(original)) {
310-
if (key === 'new_value' || key === 'old_value') continue;
311-
expect(row[key], `${String(row.id)}.${key}`).toEqual(original[key]);
312-
}
313-
}
314-
expect(archivedAfterApply.find((r) => r.id === 'aud_old_3')).toEqual(before.get('aud_old_3'));
335+
const movedColumns = archivedAfterApply.flatMap((row) =>
336+
Object.keys(before.get(row.id)!)
337+
.filter((key) => key !== 'new_value' && key !== 'old_value')
338+
.filter((key) => !isDeepStrictEqual(row[key], before.get(row.id)![key]))
339+
.map((key) => `${String(row.id)}.${key}`),
340+
);
341+
// Names only: an id and a column, never a value.
342+
expect(movedColumns).toEqual([]);
343+
expect(isDeepStrictEqual(archivedAfterApply.find((r) => r.id === 'aud_old_3'), before.get('aud_old_3'))).toBe(true);
315344
const masked = JSON.parse(String(archivedAfterApply.find((r) => r.id === 'aud_old_1')!.new_value));
316345
expect([masked.body_text === MASK, masked.body_html === MASK, masked.subject === 'Subject kept']).toEqual([true, true, true]);
317346
});
@@ -321,7 +350,7 @@ describe('[#24899] os migrate audit-mail-bodies over an archive datasource', ()
321350
expect(run.code, run.stderr.slice(-2000)).toBe(0);
322351
expect(doc(run).report.rewritten).toBe(0);
323352
expect(archiveOf(run).report).toMatchObject({ scanned: 2, rewritten: 0, failures: 0 });
324-
expect(archivedAfterAgain).toEqual(archivedAfterApply);
353+
expect(sameRows(archivedAfterAgain, archivedAfterApply)).toBe(true);
325354
});
326355

327356
it('4. the copy that was hot when the command ran is archived masked', () => {
@@ -332,7 +361,7 @@ describe('[#24899] os migrate audit-mail-bodies over an archive datasource', ()
332361
});
333362

334363
it('5. no run prints a body', () => {
335-
for (const run of [runs.dry, runs.apply, runs.again, runtimeRowRun, noArchiveRun]) {
364+
for (const run of [runs.dry, runs.apply, runs.again, runtimeRowRun, noArchiveRun, unreachableRun]) {
336365
expect(`${run.stdout}\n${run.stderr}`.includes(SENTINEL)).toBe(false);
337366
}
338367
});
@@ -343,4 +372,13 @@ describe('[#24899] os migrate audit-mail-bodies over an archive datasource', ()
343372
expect(runtimeRowRun.code).toBe(1);
344373
expect(archiveOf(runtimeRowRun)).toEqual({ datasource: 'archive', objects: ['sys_audit_log'], status: 'not-connected' });
345374
});
375+
376+
it('7. a declared archive this boot cannot connect is NOT reported clean', () => {
377+
expect(unreachableRun.code).toBe(1);
378+
const entry = archiveOf(unreachableRun);
379+
expect(entry).toMatchObject({ datasource: 'archive', objects: ['sys_audit_log'], status: 'not-connected' });
380+
expect(typeof entry.connection?.status).toBe('string');
381+
expect(entry.connection.status === 'connected' || entry.connection.status === 'already-registered').toBe(false);
382+
expect(entry.report).toBeUndefined();
383+
});
346384
});

0 commit comments

Comments
 (0)