fix(service-automation): an installed package's connectors materialize again after a restart (#28591)

.changeset/28591-rehydrated-package-connectors.md

@@ -0,0 +1,12 @@
+---
+'@objectstack/service-automation': patch
+---
+
+fix(service-automation): a connector an installed package declares is live again after the runtime restarts
+
+Clause-②: no
+Metadata-protocol: none
+
+After a restart, the install-local rehydrate registers every installed package again, but in its `kernel:ready` handler. That is after the automation service's boot reconcile in `start()`, and the rehydrate announces no reload. The package was listed by `GET /api/v1/marketplace/install-local` and its flows bound, but its provider-bound connector instances were absent from `GET /api/v1/automation/connectors` until some unrelated metadata reload. Every flow step that called them failed at dispatch.
+
+The automation service now reads the connector declaration once more at `kernel:bootstrapped`, which fires after every `kernel:ready` handler has settled, whatever the plugin order. It reconciles only when that declaration differs from the one the boot reconcile read. A boot that registers no connector after `start()` runs no second reconcile. This reconcile uses reload semantics: a configuration fault in a connector registered during `kernel:ready` (an unknown provider, an invalid `providerConfig`) is logged at `error` and that entry is skipped. It does not fail the boot. This is the same treatment the entry gets when it is installed into a running server.

content/docs/automation/connectors.mdx

@@ -873,6 +873,7 @@ retry):
 | A `connector_action` input value whose shape the parameter's style has no encoding for (an array of objects for `form`, an object for a `content`-typed parameter) | **The step fails before dispatch** naming the parameter, and nothing is sent |
 | A `connector_action` input that, once evaluated, breaks the action's `inputSchema` (a missing `required` parameter, a wrong `type`) | **The step fails before dispatch** with `VALIDATION_ERROR`, naming each parameter path, and nothing is sent; a `fault` edge or `try_catch` can route it. Supply the parameter or send the declared type: text is never converted, so cast it in a CEL envelope (`int()`, `double()`, `bool()`). `GET /api/v1/automation/connectors` lists the schema; `os validate` does not judge the input against the `inputSchema` yet |
 | A bad entry arrives via **runtime reload** (a Studio publish, a direct active save through `PUT /api/v1/meta/connector/:name`, or a delete, made on any replica — peers reconcile on [cluster lane 2](/docs/kernel/cluster#62-the-cluster-invalidation-family); or a dev reload) | Logged and **skipped**, never crashing the live server; a changed instance's old connector keeps serving until its replacement materializes |
+| A bad entry belongs to a package **installed into the runtime** (`POST /api/v1/marketplace/install-local`), at the install or at a restart that registers the package again | Logged and **skipped**, as on a runtime reload: the boot completes and the package's other instances materialize |
 
 ## Catalog descriptors: entries without a `provider`
 

packages/qa/dogfood/test/install-local-restart-connectors.dogfood.test.ts

@@ -0,0 +1,185 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+//
+// GOLDEN REGRESSION — a connector an install-local package declares is live
+// again after the runtime restarts on the same install ledger.
+//
+// ## What was measured before the fix
+//
+// Two showcase boots over one install-ledger directory. The first boot
+// installed an inline manifest declaring a provider-bound `openapi` connector
+// whose `providerConfig.spec` is an INLINE document, plus one flow. The
+// connector was listed `ready` on `GET /automation/connectors`. After the
+// second boot, `GET /marketplace/install-local` listed the package and the
+// engine served its flow (`GET /automation/NAME` 200), but the connector was
+// absent from `GET /automation/connectors`.
+//
+// Flows survive because the engine binds them in its own `kernel:ready`
+// handler, from the protocol's flow view, and the install-local rehydrate
+// registers the ledger's packages in an earlier `kernel:ready` handler.
+// Declared connectors are reconciled only in `start()`, before
+// `kernel:ready`, and again on `metadata:reloaded` — and the rehydrate
+// announced no reload, so nothing read the rehydrated connector entries.
+//
+// ## What this file pins
+//
+//   1. before the restart (precondition): the install answers 200 and the
+//      connector is listed `ready` with the document's one operation;
+//   2. after the restart: the package is listed by the install-local GET;
+//   3. after the restart: the connector is listed `ready` with the same
+//      operation;
+//   4. after the restart: the package's flow is served by the engine;
+//   5. after the restart: a connector the HOST declares (the showcase's
+//      `showcase_status_openapi`) is still listed (control).
+//
+// Boots fixture stacks of its own (the install-local plugin mounted), so it
+// stays out of `SHARED_SHOWCASE`.
+
+import { mkdtempSync, rmSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import { type VerifyStack } from '@objectstack/verify';
+import { MarketplaceInstallLocalPlugin } from '@objectstack/cloud-connection';
+import { bootShowcase } from './showcase-boot.js';
+
+const ROUTE = '/marketplace/install-local';
+const PACKAGE_ID = 'com.dogfood.restartconn';
+const CONNECTOR = 'rstconn_fixture';
+const FLOW = 'rstconn_manual_probe';
+const HOST_CONNECTOR = 'showcase_status_openapi';
+/** A one-operation OpenAPI 3 document; nothing is dispatched, so its server answers nothing. */
+const DOCUMENT = {
+  openapi: '3.0.3',
+  info: { title: 'Restart connector fixture', version: '1.0.0' },
+  servers: [{ url: 'http://127.0.0.1:9/' }],
+  paths: { '/v1/things': { get: { operationId: 'GetThings', responses: { '200': { description: 'ok' } } } } },
+};
+
+const INSTALL_BODY = {
+  manifest: {
+    id: PACKAGE_ID,
+    namespace: 'rstconn',
+    name: 'Restart connector probe',
+    version: '0.1.0',
+    type: 'app',
+    connectors: [
+      { name: CONNECTOR, label: 'Fixture', type: 'api', provider: 'openapi', providerConfig: { spec: DOCUMENT } },
+    ],
+    flows: [
+      {
+        name: FLOW,
+        label: 'Restart probe (run by name)',
+        type: 'autolaunched',
+        status: 'active',
+        nodes: [
+          { id: 'start', type: 'start', label: 'Start' },
+          { id: 'end', type: 'end', label: 'End' },
+        ],
+        edges: [{ id: 'e1', source: 'start', target: 'end' }],
+      },
+    ],
+  },
+};
+
+type Answer = { status: number; body: any };
+
+async function json(res: Response): Promise<Answer> {
+  return { status: res.status, body: await res.json().catch(() => null) };
+}
+
+const listed = (answer: Answer, name: string): any =>
+  (answer.body?.data?.connectors ?? []).find((c: any) => c?.name === name);
+
+const names = (answer: Answer): unknown => answer.body?.data?.connectors?.map((c: any) => c?.name);
+
+/** Read the connector listing until `name` is in it, or the window closes. */
+async function readConnectors(stack: VerifyStack, token: string, name: string): Promise<Answer> {
+  let answer: Answer;
+  const deadline = Date.now() + 10_000;
+  do {
+    answer = await json(await stack.apiAs(token, 'GET', '/automation/connectors'));
+    if (listed(answer, name)) break;
+    await new Promise((r) => setTimeout(r, 100));
+  } while (Date.now() < deadline);
+  return answer;
+}
+
+describe('dogfood: an install-local package\'s connector is live again after a restart on the same ledger', () => {
+  let storageDir: string;
+  let install: Answer;
+  let before: Answer;
+  let packages: Answer;
+  let after: Answer;
+  let flowAfter: Answer;
+  let bootError: unknown;
+
+  beforeAll(async () => {
+    storageDir = mkdtempSync(join(tmpdir(), 'dogfood-install-local-restart-conn-'));
+    const first = await bootShowcase({
+      extraPlugins: [new MarketplaceInstallLocalPlugin({ controlPlaneUrl: 'off', storageDir })],
+    });
+    try {
+      const token = await first.signIn();
+      install = await json(await first.apiAs(token, 'POST', ROUTE, INSTALL_BODY));
+      before = await readConnectors(first, token, CONNECTOR);
+    } finally {
+      await first.stop();
+    }
+
+    let second: VerifyStack | undefined;
+    try {
+      second = await bootShowcase({
+        extraPlugins: [new MarketplaceInstallLocalPlugin({ controlPlaneUrl: 'off', storageDir })],
+      });
+    } catch (err) {
+      bootError = err;
+      return;
+    }
+    try {
+      const token = await second.signIn();
+      packages = await json(await second.apiAs(token, 'GET', ROUTE));
+      after = await readConnectors(second, token, CONNECTOR);
+      flowAfter = await json(await second.apiAs(token, 'GET', `/automation/${FLOW}`));
+    } finally {
+      await second.stop();
+    }
+  }, 300_000);
+
+  afterAll(() => {
+    if (storageDir) rmSync(storageDir, { recursive: true, force: true });
+  });
+
+  it('before the restart, the install lists the connector ready (precondition)', () => {
+    expect(install.status, JSON.stringify(install.body)).toBe(200);
+    expect(install.body?.data?.hotLoaded).toBe(true);
+    const entry = listed(before, CONNECTOR);
+    expect(entry, JSON.stringify(names(before))).toBeDefined();
+    expect(entry.state).toBe('ready');
+    expect(entry.actions.map((a: any) => a.key)).toEqual(['GetThings']);
+  });
+
+  it('the second boot over the same ledger succeeds and lists the package', () => {
+    expect(bootError).toBeUndefined();
+    expect(packages.status, JSON.stringify(packages.body)).toBe(200);
+    const item = (packages.body?.data?.items ?? []).find((i: any) => i?.manifestId === PACKAGE_ID);
+    expect(item, JSON.stringify(packages.body)).toBeDefined();
+    expect(item.notLoaded).toBeUndefined();
+  });
+
+  it('after the restart, the package\'s connector is listed ready with the document\'s operation', () => {
+    expect(after.status, JSON.stringify(after.body)).toBe(200);
+    const entry = listed(after, CONNECTOR);
+    expect(entry, JSON.stringify(names(after))).toBeDefined();
+    expect(entry.state).toBe('ready');
+    expect(entry.actions.map((a: any) => a.key)).toEqual(['GetThings']);
+  });
+
+  it('after the restart, the engine serves the package\'s flow', () => {
+    expect(flowAfter.status, JSON.stringify(flowAfter.body)).toBe(200);
+  });
+
+  it('after the restart, a connector the host declares is still listed (control)', () => {
+    expect(listed(after, HOST_CONNECTOR), JSON.stringify(names(after))).toBeDefined();
+  });
+});

packages/services/service-automation/src/connector-boot-registered-reconcile.test.ts

@@ -0,0 +1,164 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+//
+// [#28591] A provider-bound connector entry a plugin registers during
+// `kernel:ready` — after the boot reconcile in `start()` — is materialized
+// before the boot completes.
+//
+// The install-local rehydrate is the producer this covers: it re-registers every
+// installed package in its own `kernel:ready` handler and announces no reload.
+// Before the fix, the boot reconcile in `start()` was the only one a boot ran,
+// so a rehydrated package's connector stayed absent from `GET /connectors`
+// until some unrelated `metadata:reloaded`.
+//
+// Pinned here:
+//   1. an entry registered during `kernel:ready` is listed and dispatchable once
+//      `bootstrap()` resolves, and the host entry is not materialized again;
+//   2. a boot that registers nothing after `start()` runs ONE reconcile (control);
+//   3. a configuration fault in an entry registered during `kernel:ready` is
+//      logged and that entry skipped — the boot completes and the rest of the
+//      batch is materialized (reload semantics, as on install).
+
+import { describe, it, expect, vi, afterEach } from 'vitest';
+import { LiteKernel, ObjectLogger } from '@objectstack/core';
+import type { Connector, ConnectorProviderContext, ConnectorProviderFactory } from '@objectstack/spec/integration';
+import { AutomationServicePlugin } from './plugin.js';
+import type { AutomationEngine } from './engine.js';
+
+afterEach(() => {
+    vi.restoreAllMocks();
+});
+
+/** A provider-bound declarative entry, as `registerApp` stores it. */
+function providerConnector(name: string, provider = 'fake') {
+    return { name, label: name, type: 'api', provider, providerConfig: {} };
+}
+
+/** A factory that records each materialization and returns a one-action connector. */
+function makeProvider() {
+    const calls: ConnectorProviderContext[] = [];
+    const factory: ConnectorProviderFactory = (ctx) => {
+        calls.push(ctx);
+        return {
+            def: {
+                name: ctx.name,
+                label: ctx.label,
+                type: 'api',
+                authentication: { type: 'none' },
+                actions: [{ key: 'ping', label: 'Ping' }],
+            } as unknown as Connector,
+            handlers: { ping: async () => ({ ok: true }) },
+        };
+    };
+    return { factory, calls };
+}
+
+/**
+ * Boot the automation plugin over a registry holding `atStart`, with a harness
+ * that appends `atReady` to the registry in its `kernel:ready` handler — the
+ * shape of the install-local rehydrate.
+ */
+async function boot(atStart: unknown[], atReady: unknown[], factory: ConnectorProviderFactory) {
+    const declared = [...atStart];
+    const kernel = new LiteKernel({ logger: { level: 'silent' } } as never);
+    kernel.use(new AutomationServicePlugin());
+    kernel.use({
+        name: 'test.rehydrate-harness',
+        type: 'standard' as const,
+        version: '1.0.0',
+        dependencies: ['com.objectstack.service-automation'],
+        async init(ctx: any) {
+            ctx.registerService('objectql', {
+                registry: { listItems: (type: string) => (type === 'connector' ? declared : []) },
+            });
+            ctx.getService('automation').registerConnectorProvider('fake', factory);
+        },
+        async start(ctx: any) {
+            ctx.hook('kernel:ready', async () => {
+                declared.push(...atReady);
+            });
+        },
+    } as never);
+    await kernel.bootstrap();
+    return { kernel, engine: kernel.getService('automation') as AutomationEngine };
+}
+
+const reconcileSpy = () =>
+    vi.spyOn(AutomationServicePlugin.prototype as any, 'reconcileDeclaredConnectors');
+
+describe('[#28591] connectors registered during kernel:ready are materialized before the boot completes', () => {
+    it('lists and dispatches an entry registered during kernel:ready, without re-materializing the host entry', async () => {
+        const { factory, calls } = makeProvider();
+        const { kernel, engine } = await boot([providerConnector('host_conn')], [providerConnector('installed_conn')], factory);
+
+        expect(engine.getRegisteredConnectors()).toEqual(expect.arrayContaining(['host_conn', 'installed_conn']));
+        expect(engine.getConnectorOrigin('installed_conn')).toBe('declarative');
+        expect(calls.map((c) => c.name)).toEqual(['host_conn', 'installed_conn']);
+
+        engine.registerFlow('call_installed', {
+            name: 'call_installed',
+            label: 'Call installed',
+            type: 'autolaunched',
+            variables: [{ name: 'call.ok', type: 'boolean', isOutput: true }],
+            nodes: [
+                { id: 'start', type: 'start', label: 'Start' },
+                {
+                    id: 'call',
+                    type: 'connector_action',
+                    label: 'Ping',
+                    connectorConfig: { connectorId: 'installed_conn', actionId: 'ping', input: {} },
+                },
+                { id: 'end', type: 'end', label: 'End' },
+            ],
+            edges: [
+                { id: 'e1', source: 'start', target: 'call' },
+                { id: 'e2', source: 'call', target: 'end' },
+            ],
+        } as never);
+        const result = await engine.execute('call_installed');
+        expect(result.success).toBe(true);
+        expect(result.output).toMatchObject({ 'call.ok': true });
+
+        await kernel.shutdown();
+    });
+
+    it('runs one reconcile on a boot that registers nothing after start() (control)', async () => {
+        const spy = reconcileSpy();
+        const { factory, calls } = makeProvider();
+        const { kernel, engine } = await boot([providerConnector('host_conn')], [], factory);
+
+        expect(engine.getRegisteredConnectors()).toContain('host_conn');
+        expect(calls).toHaveLength(1);
+        expect(spy).toHaveBeenCalledTimes(1);
+        expect(spy.mock.calls[0]?.[1]).toEqual({ fatal: true });
+
+        await kernel.shutdown();
+    });
+
+    it('runs a second, non-fatal reconcile when kernel:ready registered an entry', async () => {
+        const spy = reconcileSpy();
+        const { factory } = makeProvider();
+        const { kernel } = await boot([providerConnector('host_conn')], [providerConnector('installed_conn')], factory);
+
+        expect(spy.mock.calls.map((c) => c[1])).toEqual([{ fatal: true }, { fatal: false }]);
+
+        await kernel.shutdown();
+    });
+
+    it('logs and skips a configuration fault in an entry registered during kernel:ready; the boot completes', async () => {
+        const errors = vi.spyOn(ObjectLogger.prototype, 'error');
+        const { factory } = makeProvider();
+        const { kernel, engine } = await boot(
+            [providerConnector('host_conn')],
+            [providerConnector('broken_conn', 'no_such_provider'), providerConnector('installed_conn')],
+            factory,
+        );
+
+        const registered = engine.getRegisteredConnectors();
+        expect(registered).toEqual(expect.arrayContaining(['host_conn', 'installed_conn']));
+        expect(registered).not.toContain('broken_conn');
+        const messages = errors.mock.calls.map((c) => String(c[0]));
+        expect(messages.some((m) => m.includes('broken_conn') && m.includes('no_such_provider')), messages.join('\n')).toBe(true);
+
+        await kernel.shutdown();
+    });
+});