Skip to content

Commit 6d9419b

Browse files
fix(service-datasource): a runtime managed datasource syncs its bound objects' tables on create and after a restart (#27748)
<!-- claude-projects-attribution: {"github_login":"os-elon-musk"} --> _Requested by **Elon** · [project thread](https://claude.ai/code/project/chan_01WedDzV26nniwxv7jC88B9D?thread=cmsg_01WedDzV26nniwxv7jC88B9D1H8LKUgLcNior93UM4jPpL)_ domain:server#3 · os-elon-musk · session_016WV4czbMpdApCUbUuNyHt7 Fixes #27701 ## What changes - `registerPool` (`packages/services/service-datasource/src/datasource-admin-plugin.ts`) now opens a runtime datasource's pool with `objects`: the names of the objects bound to it in the engine registry. The connect's existing sync loop gives them their tables. This covers the admin create, the boot rehydration, and a peer replica's convergence on a new datasource, which all go through `registerPool`. - ⛔ The objects are passed only when the datasource is managed. Its mode is read with `resolveDatasourceSchemaMode`, the same read the driver factory uses. An `external` or `validate-only` datasource passes none and never gets DDL (triage ruling `6506923793`). - The registry read that `listObjectsBoundTo` already did is lifted into one local helper, `objectsBoundTo`, which both readers call. `listObjectsBoundTo`'s answer is byte-for-byte the same. - Neither `datasource-connection-service.ts` (open PR #27715) nor `objectql/src/engine.ts` is touched. ## Why A code object bound to a Setup-created managed datasource never got its table. Boot's schema sync skipped it, because the driver did not exist yet. `registerPool` then called `connect(record, { context })` with no `objects`, so the sync loop had nothing to sync. The datasource answered `ok`, and every read of the object was a 500 (Postgres `42P01`), the same after a restart. ## Verification New pin: `src/__tests__/runtime-datasource-bound-object-storage.test.ts`. It uses the real `DatasourceAdminServicePlugin` through `init()` and `start()`, a real `ObjectQL` engine on the `objectql` and `data` slots, and the real default driver factory over SQLite files. | Case | `origin/main` plugin | this PR | |---|---|---| | on create, the bound object's table exists and the object reads | red | green | | after a restart, the rehydrated pool creates a missing table and the object reads | red | green | | control: an object bound to another datasource is not synced | red | green | | ⛔ `schemaMode: external`: the pool opens, nothing is synced | green | green | | ⛔ `schemaMode: validate-only`: the pool opens, nothing is synced | green | green | Ablations, each restored afterwards: - Dropping the `objects` argument turns the 3 positive cases red. - Forcing the managed gate open turns both negative cases red (`syncObjectSchema` called 2 times). Local readings at `6f773675`: - `@objectstack/service-datasource` `test`: 62 files, 1016/1016 passed. `typecheck`: exit 0. - `dispatch-gates --commands`: 59 families derived, 58 exit 0. `pnpm check:dual-build-cjs-loads` exited 3, so it is NOT MEASURED (it needs a whole-tree build; CI runs it). - `dispatch-gates --ran`: 59 accounted for. - `pnpm lint`: exit 0. Clause-②: no Metadata-protocol: none ## Acceptance notes - **Out of scope, measured (class a).** An edit that rebuilds the pool still syncs nothing. That covers a changed `config`, credential, `schemaMode` or `active` (`updateDatasource` → `reregisterPool` → `DatasourceConnectionService.reconnect`). `reconnect` takes no `objects`, and its inner `connect(record, { context })` is in `datasource-connection-service.ts`. - Measured with the same rig: create `warehouse` on `a.db`, then update its `config` to `b.db`. The result is `hasTable('wh_note') === false`, and the read is refused by the database. - `reach:` the admin update door (`PATCH /api/v1/datasources/:name` → `updateDatasource`). The fix belongs in the connection service, serial behind #27715. - A federated object (`external` set) on a managed datasource would go through `syncObjectSchema`'s DDL-free `registerExternalObject` branch. The admin door already refuses that binding at write time (#25277). ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. (not touched) - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes. - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. - [x] 6. `packages/spec` touched: n/a. - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English. - [x] The *Before Submitting* list in CONTRIBUTING.md is done. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_016WV4czbMpdApCUbUuNyHt7 Co-authored-by: Claude <noreply@anthropic.com>
1 parent 79d78b4 commit 6d9419b

3 files changed

Lines changed: 256 additions & 18 deletions

File tree

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
---
2+
'@objectstack/service-datasource': patch
3+
---
4+
5+
fix(service-datasource): a code object bound to a Setup-created managed datasource gets its table when the datasource's pool opens, on create and after a restart (#27701)
6+
7+
Clause-②: no
8+
Metadata-protocol: none
9+
10+
An author declares an object with `datasource: 'warehouse'`, and an admin creates `warehouse` in Setup → Datasources. Boot's schema sync skipped the object, because no `warehouse` driver existed yet, and the pool the admin door opened synced nothing either. The datasource answered `ok`, the table was never created, and every read of the object was a 500 (Postgres `42P01`), the same after every restart.
11+
12+
- **Now.** Opening a runtime datasource's pool (the admin create, the boot rehydration, and a peer replica's convergence on a new datasource) syncs the schema of the objects already bound to it, as boot does for a declared datasource. A table that is missing is created; a table that exists is left as the sync leaves it. One object's refused sync is logged at `error` and does not fail the connect, as on the declared path.
13+
- ⛔ **Only a managed datasource.** A datasource whose `schemaMode` is `external` or `validate-only` syncs nothing for its objects, so it never gets DDL. Its mode is read as the driver factory reads it.
14+
- **Not changed.** An edit that rebuilds the pool (a changed `config`, credential, `schemaMode` or `active`) still syncs nothing.
Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#27701] A code object bound to a runtime (Setup-created) managed datasource
5+
* gets its table when the datasource's pool opens: on create, and on the boot
6+
* rehydration after a restart.
7+
*
8+
* ## The defect this pins closed
9+
*
10+
* An author declares `wh_note` with `datasource: 'warehouse'`, and an admin
11+
* creates `warehouse` in Setup → Datasources. Boot's schema sync skipped the
12+
* object, because no `warehouse` driver existed yet. The plugin's
13+
* `registerPool` then opened the pool with `connect(record, { context })` and
14+
* no `objects`, so the connect's sync loop had nothing to sync. The datasource
15+
* answered `ok`, the table was never created, and every read of the object was
16+
* a 500 (Postgres `42P01`, SQLite `no such table`), the same after a restart.
17+
*
18+
* ## What each case pins
19+
*
20+
* - on create, the bound managed object's table exists and the object reads;
21+
* - on the restart's rehydration, a table that is missing (the state an
22+
* install created before this fix is in) is created, and the object reads;
23+
* - ⛔ an `external` or `validate-only` datasource gets no schema sync at all:
24+
* its pool opens, and nothing is synced for the objects bound to it.
25+
*
26+
* The composition is the real one: the real `DatasourceAdminServicePlugin`
27+
* through `init()` and `start()`, a real `ObjectQL` engine on the `'objectql'`
28+
* and `'data'` slots (one engine, as `ObjectQLPlugin` registers it), and the
29+
* real default driver factory opening SQLite files. The engine's `sys_metadata`
30+
* door is left off the `'data'` slot, so the plugin takes its in-memory branch:
31+
* durability across a restart is the subject of `datasource-admin-plugin.test.ts`
32+
* ("runtime datasource durability"). Here the restart's stored row arrives in
33+
* the metadata service, where that restore leaves it. The code object is put
34+
* in the engine registry before the plugin starts, as `AppPlugin.init()` does
35+
* through the `'manifest'` service in Phase 1.
36+
*/
37+
38+
import { afterEach, describe, expect, it, vi } from 'vitest';
39+
import { mkdtempSync, rmSync } from 'node:fs';
40+
import { tmpdir } from 'node:os';
41+
import { join } from 'node:path';
42+
import { ObjectQL } from '@objectstack/objectql';
43+
import type { IDataDriver } from '@objectstack/spec/contracts';
44+
import type { IDatasourceAdminService } from '../contracts/index.js';
45+
import { DatasourceAdminServicePlugin } from '../datasource-admin-plugin.js';
46+
import { createDefaultDatasourceDriverFactory } from '../default-datasource-driver-factory.js';
47+
48+
const teardowns: Array<() => Promise<void> | void> = [];
49+
afterEach(async () => {
50+
for (const t of teardowns.splice(0).reverse()) await t();
51+
});
52+
53+
type Knex = {
54+
schema: { hasTable(n: string): Promise<boolean>; dropTableIfExists(n: string): Promise<void> };
55+
};
56+
57+
/** The author's code object, bound by name to a datasource the stack does not declare. */
58+
const boundTo = (name: string, datasource: string, extra: Record<string, unknown> = {}) => ({
59+
name,
60+
label: name,
61+
datasource,
62+
fields: { title: { type: 'text', label: 'Title' } },
63+
...extra,
64+
});
65+
66+
/** A fresh directory for this case's SQLite files. */
67+
function scratch(): string {
68+
const dir = mkdtempSync(join(tmpdir(), 'ds-27701-'));
69+
teardowns.push(() => rmSync(dir, { recursive: true, force: true }));
70+
return dir;
71+
}
72+
73+
/**
74+
* One boot: an engine holding the code objects, then the plugin's `init()` and
75+
* `start()`. `stored` is what the restart's restore leaves in the metadata
76+
* service.
77+
*/
78+
async function boot(opts: { objects: Array<Record<string, unknown>>; stored?: Array<Record<string, unknown>> }) {
79+
const ql = new ObjectQL();
80+
for (const object of opts.objects) (ql as any).registry.registerObject(object as never, 'com.acme.warehouse');
81+
const synced = vi.spyOn(ql, 'syncObjectSchema');
82+
83+
const records = new Map<string, Map<string, unknown>>([
84+
['datasource', new Map((opts.stored ?? []).map((r) => [r.name as string, r]))],
85+
]);
86+
const metadata = {
87+
get: async (t: string, n: string) => records.get(t)?.get(n),
88+
list: async (t: string) => [...(records.get(t)?.values() ?? [])],
89+
register: async (t: string, n: string, d: unknown) => {
90+
if (!records.has(t)) records.set(t, new Map());
91+
records.get(t)!.set(n, d);
92+
},
93+
unregister: async (t: string, n: string) => void records.get(t)?.delete(n),
94+
};
95+
const sysMetadataDoor = new Set(['find', 'findOne', 'insert', 'update', 'delete']);
96+
const data = new Proxy(ql, {
97+
get: (target, key, receiver) => (sysMetadataDoor.has(key as string) ? undefined : Reflect.get(target, key, receiver)),
98+
});
99+
100+
const services = new Map<string, unknown>([
101+
['objectql', ql],
102+
['data', data],
103+
['metadata', metadata],
104+
]);
105+
const ctx = {
106+
getService: (n: string) => {
107+
if (!services.has(n)) throw new Error(`no service: ${n}`);
108+
return services.get(n);
109+
},
110+
registerService: (n: string, s: unknown) => void services.set(n, s),
111+
trigger: async () => {},
112+
logger: { info() {}, warn() {}, error() {}, debug() {} },
113+
};
114+
115+
const plugin = new DatasourceAdminServicePlugin({ driverFactory: createDefaultDatasourceDriverFactory({}) });
116+
await plugin.init(ctx as never);
117+
await plugin.start(ctx as never);
118+
teardowns.push(async () => {
119+
for (const driver of (ql as any).drivers?.values?.() ?? []) await (driver as IDataDriver).disconnect?.();
120+
});
121+
122+
const knexOf = (datasource: string) => (ql.getDriverByName(datasource) as unknown as { getKnex(): Knex }).getKnex();
123+
const read = (object: string) => ql.find(object, { context: { isSystem: true } } as never);
124+
return { ql, synced, knexOf, read, admin: services.get('datasource-admin') as IDatasourceAdminService };
125+
}
126+
127+
describe('a code object bound to a runtime managed datasource gets its table', () => {
128+
it('on create: the table exists and the object reads', async () => {
129+
const dir = scratch();
130+
const { admin, knexOf, read } = await boot({ objects: [boundTo('wh_note', 'warehouse')] });
131+
132+
const created = await admin.createDatasource({
133+
name: 'warehouse',
134+
driver: 'sqlite',
135+
config: { filename: join(dir, 'warehouse.db') },
136+
});
137+
138+
expect(created.status).toBe('ok');
139+
expect(await knexOf('warehouse').schema.hasTable('wh_note')).toBe(true);
140+
await expect(read('wh_note')).resolves.toEqual([]);
141+
});
142+
143+
it('after a restart: the rehydrated pool creates a table that is missing, and the object reads', async () => {
144+
const dir = scratch();
145+
const record = {
146+
name: 'warehouse',
147+
driver: 'sqlite',
148+
origin: 'runtime',
149+
active: true,
150+
config: { filename: join(dir, 'warehouse.db') },
151+
};
152+
const first = await boot({ objects: [boundTo('wh_note', 'warehouse')] });
153+
await first.admin.createDatasource({ name: record.name, driver: record.driver, config: record.config });
154+
// The state an install that created the datasource before this fix is in.
155+
await first.knexOf('warehouse').schema.dropTableIfExists('wh_note');
156+
await (first.ql.getDriverByName('warehouse') as IDataDriver).disconnect?.();
157+
158+
const restarted = await boot({ objects: [boundTo('wh_note', 'warehouse')], stored: [record] });
159+
160+
expect(await restarted.knexOf('warehouse').schema.hasTable('wh_note')).toBe(true);
161+
await expect(restarted.read('wh_note')).resolves.toEqual([]);
162+
});
163+
164+
it('control: an object bound to another datasource is not synced on this one', async () => {
165+
const dir = scratch();
166+
const { admin, synced } = await boot({
167+
objects: [boundTo('wh_note', 'warehouse'), boundTo('crm_note', 'crm')],
168+
});
169+
170+
await admin.createDatasource({ name: 'warehouse', driver: 'sqlite', config: { filename: join(dir, 'warehouse.db') } });
171+
172+
expect(synced.mock.calls.map(([name]) => name)).toEqual(['wh_note']);
173+
});
174+
});
175+
176+
describe('⛔ an external or federated datasource never gets DDL', () => {
177+
it.each(['external', 'validate-only'] as const)(
178+
'schemaMode %s: the pool opens, and no object bound to it is synced',
179+
async (schemaMode) => {
180+
const dir = scratch();
181+
const { admin, synced, knexOf } = await boot({
182+
objects: [
183+
boundTo('lake_note', 'lake'),
184+
boundTo('lake_orders', 'lake', { external: { remoteName: 'orders' } }),
185+
],
186+
});
187+
188+
const created = await admin.createDatasource({
189+
name: 'lake',
190+
driver: 'sqlite',
191+
schemaMode,
192+
external: { validation: { onMismatch: 'warn' } },
193+
config: { filename: join(dir, 'lake.db') },
194+
} as never);
195+
196+
expect(created.status).toBe('ok');
197+
expect(synced).not.toHaveBeenCalled();
198+
expect(await knexOf('lake').schema.hasTable('lake_note')).toBe(false);
199+
},
200+
);
201+
});

‎packages/services/service-datasource/src/datasource-admin-plugin.ts‎

Lines changed: 41 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import {
2020
type ConnectionEngineLike,
2121
} from './datasource-connection-service.js';
2222
import { datasourceConnectivityChanged } from './datasource-connectivity-change.js';
23+
import { resolveDatasourceSchemaMode } from './datasource-schema-mode.js';
2324
import type { DatasourceConnectPolicy } from './contracts/connect-policy.js';
2425
import type { Logger } from './logger.js';
2526
import { resolveTursoUrl } from './turso-driver-config.js';
@@ -402,6 +403,26 @@ export class DatasourceAdminServicePlugin implements Plugin {
402403
});
403404
ctx.registerService('datasource-connection', this.connection);
404405

406+
// [ADR-0015 §4.2 · #25277] The objects bound to a datasource, read from
407+
// the engine's object registry (the `objectql` service's `registry`), the
408+
// registry the metadata save door writes through to: an object saved
409+
// through `PUT /api/v1/meta/object/:name` or imported from a remote table
410+
// is in it the moment it is saved, and boot restores the stored rows into
411+
// it. ⛔ Not the metadata service's `listObjects()`, which holds a copy of
412+
// this registry taken once at boot and never sees a runtime save. A host
413+
// with no engine registry serves no objects, so none can be bound.
414+
//
415+
// Resolved where it is used, never at `init()` (AGENTS.md, "Startup
416+
// registry reads"). Two readers: the judgement of a datasource write
417+
// (`listObjectsBoundTo`) and the pool's connect (`registerPool`, #27701).
418+
const objectsBoundTo = (datasource: string): Array<{ name: unknown; external: unknown }> => {
419+
const registry = safeGetService<Partial<Pick<IObjectQLEngine, 'registry'>>>(ctx, 'objectql')?.registry;
420+
if (typeof registry?.getAllObjects !== 'function') return [];
421+
return (registry.getAllObjects() as Array<{ name?: unknown; datasource?: unknown; external?: unknown }>)
422+
.filter((o) => o?.datasource === datasource)
423+
.map((o) => ({ name: o.name, external: o.external }));
424+
};
425+
405426
const config: DatasourceAdminServiceConfig = {
406427
probe: (input) => this.probe(factory(), input),
407428

@@ -503,24 +524,9 @@ export class DatasourceAdminServicePlugin implements Plugin {
503524
},
504525

505526
// [ADR-0015 §4.2 · #25277] The objects a datasource write is judged
506-
// against. Read from the engine's object registry (the `objectql`
507-
// service's `registry`), the registry the metadata save door writes
508-
// through to: an object saved through `PUT /api/v1/meta/object/:name` or
509-
// imported from a remote table is in it the moment it is saved, and boot
510-
// restores the stored rows into it. ⛔ Not the metadata service's
511-
// `listObjects()`, which holds a copy of this registry taken once at boot
512-
// and never sees a runtime save. A host with no engine registry serves
513-
// no objects, so none can be bound.
514-
//
515-
// Resolved where it is used, never at `init()` (AGENTS.md, "Startup
516-
// registry reads").
517-
listObjectsBoundTo: async (datasource) => {
518-
const registry = safeGetService<Partial<Pick<IObjectQLEngine, 'registry'>>>(ctx, 'objectql')?.registry;
519-
if (typeof registry?.getAllObjects !== 'function') return [];
520-
return (registry.getAllObjects() as Array<{ name?: unknown; datasource?: unknown; external?: unknown }>)
521-
.filter((o) => o?.datasource === datasource)
522-
.map((o) => ({ name: o.name, external: o.external }));
523-
},
527+
// against, read from the engine's object registry (`objectsBoundTo`
528+
// above says why that registry, and why per call).
529+
listObjectsBoundTo: async (datasource) => objectsBoundTo(datasource),
524530

525531
// Hot pool (de)registration converges on the shared
526532
// DatasourceConnectionService (ADR-0062 D1) — one connect path for code-
@@ -529,9 +535,26 @@ export class DatasourceAdminServicePlugin implements Plugin {
529535
// opens the connection, and registers the live driver + datasource def.
530536
// Runtime-admin connects always degrade-with-warning on failure (never
531537
// fail-fast), preserving the pre-ADR-0062 admin behavior.
538+
//
539+
// [#27701] The connect carries the objects already bound to a MANAGED
540+
// datasource, so its sync loop gives them their tables, as boot's
541+
// `connectDeclared` does for a declared datasource. Boot's schema sync
542+
// ran before this driver existed and skipped them; without this a code
543+
// object bound to a Setup-created datasource had no table, on create and
544+
// after every restart, and every read of it was a 500. ⛔ A datasource
545+
// whose schema ObjectStack does not own (`external`, `validate-only`)
546+
// passes none: it never gets DDL. Its mode is read the way the driver
547+
// factory reads it, so the gate and the driver agree on what it is.
532548
registerPool: async (record) => {
533549
this.livePoolRecords.set(record.name, record);
550+
const managed = (resolveDatasourceSchemaMode({ ...record, config: record.config ?? {} }) ?? 'managed') === 'managed';
551+
const objects = managed
552+
? objectsBoundTo(record.name)
553+
.map((o) => o.name)
554+
.filter((n): n is string => typeof n === 'string')
555+
: [];
534556
await this.connection?.connect(record, {
557+
objects,
535558
context: { origin: record.origin ?? 'runtime', trigger: 'runtime-admin' },
536559
});
537560
},

0 commit comments

Comments
 (0)