Skip to content

fix(service-automation): an installed package's connectors materialize again after a restart (#28591) - #28639

Merged
objectstack-fleet[bot] merged 7 commits into
mainfrom
claude/issue-28591-rehydrate-connectors
Nov 1, 2026
Merged

objectstack-fleet[bot] merged 7 commits into
mainfrom
claude/issue-28591-rehydrate-connectors

Conversation

@objectstack-fleet

Copy link
Copy Markdown
Contributor

Fixes #28591
Clause-②: no
Metadata-protocol: none

维护者速读

  • 改了什么:automation 服务在 kernel:bootstrapped 再读一次连接器声明;只有当它与 start() 里那次启动 reconcile 读到的不同,才以非致命方式再 reconcile 一次。重启后,已安装包的连接器重新上线。
  • 为什么改:重启后 install-local 在自己的 kernel:ready 里才重新注册已安装的包,晚于 automation 在 start() 里唯一一次 reconcile,且不广播 reload。包还在、流程还能绑定,但连接器从 GET /api/v1/automation/connectors 消失,调用它的流程在派发时失败。
  • 风险与回滚:只有在 start() 之后又有插件注册了连接器时,启动才多一次 reconcile;普通启动不多跑。这次 reconcile 用重载语义:配置错的条目记 ERROR 并跳过,不让启动失败。回滚就是 revert 本 PR,行为退回到「重启后已安装包的连接器缺席」。
  • 先看:packages/services/service-automation/src/plugin.ts(reconcileConnectorsRegisteredDuringBoot 和它的 kernel:bootstrapped 钩子,以及 declaredInstanceFingerprint);packages/services/service-automation/src/connector-boot-registered-reconcile.test.ts;packages/qa/dogfood/test/install-local-restart-connectors.dogfood.test.ts。
  • 两分钟验证:pnpm --filter @objectstack/dogfood exec vitest run --maxWorkers=2 test/install-local-restart-connectors.dogfood.test.ts,预期 Tests 5 passed (5);同一命令在 main 上是 1 failed(重启后连接器不在列表里)。
  • 需要你做的:无,席位落地。

What changes

Measured first, at BASE d94dd3a6. Two bootShowcase boots over one install-ledger directory. The first boot ran POST /api/v1/marketplace/install-local with an inline manifest that declares a provider-bound openapi connector with an INLINE document, and one run-by-name flow. Then the second boot read the listings.

  • GET /automation/connectors before the restart listed the connector ready.
  • After the restart, GET /marketplace/install-local listed the package, and GET /automation/rstconn_manual_probe answered 200, so the package's flows survive a restart.
  • The connector was absent: ["showcase_status_api","showcase_status_openapi","showcase_mcp_tools","rest","slack"].

How flows learn about rehydrated packages, and why connectors did not. The boot log of the second boot shows the order:

  1. [Automation] materialized 3 provider-bound connector instance(s): the boot reconcile in start().
  2. [MarketplaceInstallLocal] rehydrated com.dogfood.restartconn@0.1.0: the install-local kernel:ready handler.
  3. [Automation] Bound 33 flow(s) from the protocol at kernel:ready: 32 on the first boot. This is automation's own kernel:ready flow sync, and it reads the protocol's view after the rehydrate.

Flows therefore learn by reading in a consumer boot hook that runs after the producer's. Connectors had no such read. They are reconciled only in start() and on metadata:reloaded, and the rehydrate announces no reload.

The fix takes the same road for connectors: a consumer-side boot read, region-local to service-automation.

  • AutomationServicePlugin.start() registers a kernel:bootstrapped hook. The kernel fires that hook after every kernel:ready handler has settled, and its docs name it for "reconcile work that consumes data produced by a later-starting plugin's kernel:ready handler". So the hook does not depend on plugin order. The flow sync's kernel:ready read does depend on it. In the unit test the producer's kernel:ready handler runs after automation's, and the new hook still materializes the connector.
  • reconcileConnectorsRegisteredDuringBoot reads the registry declaration (post: false, no sys_metadata read). It compares that declaration with the declaredInstanceFingerprint of the declaration the last reconcile read: the sorted name and connectorInstanceSignature of each enabled, provider-bound entry. It runs a fatal: false reconcile only when the two differ. A boot with no installs runs no second reconcile. The unit control shows exactly one reconcileDeclaredConnectors call, { fatal: true }. In the dogfood log, the first boot prints one materialized … instance(s) line.
  • Reload semantics. A configuration fault in a connector registered during kernel:ready is logged at error and that entry is skipped, as on install. The boot completes. A reconcile run that throws anyway is reported once at warn: a connector that is not live is a functional degradation.
  • Unchanged instances are left alone: the host connectors are not re-materialized on the second boot, because the signatures are unchanged.

Why not candidate (a), one metadata:reloaded announced by the rehydrate. It would fire inside kernel:ready, ahead of handlers that have not run yet. It would also re-run every reload consumer on every boot with installs: ObjectQL's hook and action re-sync and schema sync, the flow re-sync, i18n, and security invalidation. Flows do not need that. The announceHotInstall docblock records the rehydrate's choice not to announce. Option (b) changes one consumer and leaves marketplace-install-local-plugin.ts untouched.

  • content/docs/automation/connectors.mdx, "Failure modes at a glance": one row. A bad entry of an installed package is logged and skipped at the install and at a restart. Before, the table read "hard boot error" for every configuration fault outside a runtime reload.
  • .changeset/28591-rehydrated-package-connectors.md: @objectstack/service-automation patch.

No packages/spec change.

Tests

  • packages/qa/dogfood/test/install-local-restart-connectors.dogfood.test.ts (new, permanent). It runs two boots over one ledger and asserts five things: the pre-restart precondition; the package listed after the restart and not marked notLoaded; the connector ready with its GetThings action after the restart; the flow served by the engine after the restart; and the host's showcase_status_openapi still listed (control).
  • packages/services/service-automation/src/connector-boot-registered-reconcile.test.ts (new, 4 cases):
    • an entry registered in a kernel:ready handler is listed after bootstrap() and dispatched end-to-end through a flow connector_action, and the host entry's factory runs once;
    • control: a boot that registers nothing after start() runs ONE reconcile;
    • such a boot runs [{fatal:true},{fatal:false}];
    • an unknown-provider entry in the batch is logged at error and skipped, the boot completes, and its sibling materializes.

Runs at head 84e3f0ab, after the second merge of origin/main:

Restart dogfood case: BASE red, fix green
  • BASE d94dd3a6 (the fix not in the tree, service-automation built from BASE): pnpm --filter @objectstack/dogfood exec vitest run --maxWorkers=2 test/install-local-restart-connectors.dogfood.test.ts. Result: Tests 1 failed | 4 passed (5). The failure was after the restart, the package's connector is listed ready … with AssertionError: ["showcase_status_api","showcase_status_openapi","showcase_mcp_tools","rest","slack"]: expected undefined to be defined. The flow and host-connector cases were green, so flows survive a restart.
  • Fix 63d2ffde, then 84e3f0ab after rebuilding: same command, Tests 5 passed (5). On the second boot the log prints rehydrated com.dogfood.restartconn@0.1.0, then Bound 33 flow(s) … at kernel:ready, then materialized connector instance 'rstconn_fixture' via provider 'openapi' (1 action(s)), all before Bootstrap complete. The first boot (empty ledger) prints a single materialized 3 provider-bound connector instance(s) before Bootstrap complete: no second reconcile.
Unit suite, typecheck, and the one ablation
  • pnpm --filter @objectstack/service-automation test (the whole suite, not file-selected): Test Files 251 passed (251), Tests 3177 passed (3177) at 84e3f0ab. It was 251 and 3173 at c5d3ce33.
  • pnpm --filter @objectstack/service-automation typecheck (tsc plus check:test-typecheck): exit 0. pnpm --filter @objectstack/dogfood typecheck: exit 0. With tsc --listFiles, each program holds its new test file (1 hit each).
  • Ablation, once, at committed HEAD bcbb7ad8 (plugin.ts blob e344a65d). node scripts/ablation-replace.mjs swapped the anchor await this.reconcileConnectorsRegisteredDuringBoot(ctx); for void ctx; /* ABLATED_28591 */. The anchor count went 1 to 0 and the blob went e344a65d to 05a767c8. The unit file imports ./plugin.js from source, so no build sits in the path. Result: Tests 3 failed | 1 passed (4). All three cases that need the second reconcile went red, and the plain-boot control stayed green. Restored: blob equals HEAD e344a65d and git diff HEAD is empty.

Acceptance notes

  • "Actions dispatchable" is proven in the unit layer: a flow connector_action runs against the rehydrated-shape entry. The dogfood case asserts the listed ready state and the action key. Its document's server is 127.0.0.1:9, so a real dispatch there would only measure an unreachable upstream.
  • Flows of an installed package survive a restart in the composition measured here, because the install-local kernel:ready handler ran before automation's. That order is not declared anywhere. The connector half now uses kernel:bootstrapped and does not depend on it. Whether os serve's composition orders the two the same way was not measured.

Verification

  • node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack, with no paths: 5-path change set against merge base a9c6eac6f, 91 commands. Each was run at 84e3f0ab with its exit code recorded before any pipe, and all 91 exited 0. --ran: 91 derived, 91 run, 0 NOT-MEASURED, 0 UNRUN.
  • check:skill-examples and check:dual-build-cjs-loads read the dists of 7 packages outside this diff. They were built (turbo, 75/75 tasks) before the run.
  • check-platform-checklist is not owed: no export was renamed or deleted. test:repo is declared to CI: no repo-project test was touched.
  • Control-byte self-scan of the five changed files: grep -naP exit 1 (no match).

Generated by Claude Code

…kage's connector

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
…ing kernel:ready, at kernel:bootstrapped

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
…nnectors registered during kernel:ready

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
…ped at install and at a restart; changeset

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
…ped at install and at a restart

Claude-Session: https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U
Co-authored-by: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added size/m documentation Improvements or additions to documentation tests tooling labels Nov 1, 2026
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/service-automation, touching 5 documentable anchor(s).

2 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/automation/flows.mdx (via AutomationServicePlugin (symbol, a top-level class))
  • content/docs/kernel/cluster.mdx (via AutomationServicePlugin (symbol, a top-level class))

⛔ 1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17/17-6.mdx (via AutomationServicePlugin (symbol, a top-level class))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 60 of 214 client-bound route-ledger rows — the other 154 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 154: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 99 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
  • a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 10 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 28ef35e0e3799e1305e82c2d12f0515ca6bf519c → packageMentionDocs.

Which tree this was computed on

This run read content/docs from ab42fa612464d2a1a09e78c8ab2c62d73c23255c — the merge of head 84e3f0ab42caded742dda94aa315d7148dc815da into base 28ef35e0e3799e1305e82c2d12f0515ca6bf519c, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin ab42fa612464d2a1a09e78c8ab2c62d73c23255c && git checkout ab42fa612464d2a1a09e78c8ab2c62d73c23255c
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 28ef35e0e3799e1305e82c2d12f0515ca6bf519c 84e3f0ab42caded742dda94aa315d7148dc815da && git checkout -B drift-repro 28ef35e0e3799e1305e82c2d12f0515ca6bf519c && git merge --no-ff 84e3f0ab42caded742dda94aa315d7148dc815da

node scripts/docs-audit/affected-docs.mjs --json 28ef35e0e3799e1305e82c2d12f0515ca6bf519c

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 28ef35e0e3799e1305e82c2d12f0515ca6bf519c → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@objectstack-fleet

Copy link
Copy Markdown
Contributor Author

epic:#27293#1 · sunhaolin · session_017xRBcVBKAFk1ntXSjvPt7U

席位核对

ACCEPT at head 84e3f0ab · Implemented-by: claude/issue-28591-rehydrate-connectors · Reviewed-by: this seat (lane ACCEPT at TIER_DEFAULT). Clause-②: no, Metadata-protocol: none. No packages/spec file is touched and no accept set changes, so no CONTRACT_REVIEW_TIER record is owed.

Check Result
Shape Draft, base main. Fixes #28591 is the only closing keyword. One @objectstack/service-automation patch changeset.
Scope 5 files, +434/−0. Not governed. Region-local to service-automation/src/plugin.ts: a kernel:bootstrapped hook re-reads the declaration and runs one fatal: false reconcile, only when the fingerprint of the enabled provider-bound entries differs from the last reconcile's. marketplace-install-local-plugin.ts is untouched.
Measurement first At d94dd3a6, after a restart the package was listed and its flow served, but its connector was absent from GET /automation/connectors. Flows survive a restart through automation's own kernel:ready sync; the fix takes the same consumer-side route for connectors, at a hook that does not depend on plugin order.
Evidence Permanent dogfood case install-local-restart-connectors.dogfood.test.ts: 1 failed at BASE, 5 passed after. Unit connector-boot-registered-reconcile.test.ts: 4 cases, among them a plain boot with exactly one reconcile (control) and an unknown-provider entry that is logged and skipped with the boot completing. One ablation turns exactly the 3 second-reconcile cases red. The whole service-automation suite passes: 251 files, 3177 tests.
Gates CI on the head: 50 check-runs, 38 success, 12 skipped, none red. check-expected-skips --pr 28639: OK. Mergeable clean.
Accepted deviation One row added to content/docs/automation/connectors.mdx (Failure modes), outside the claimed surface: that table said every configuration fault outside a reload is a hard boot error, and that is no longer true for an installed package's entry at a restart.

Noted, not filed (agreed): the flows of an installed package survive a restart only because install-local's kernel:ready handler runs before automation's flow sync, an order nothing declares. This is an inference from reading the code; no reach was measured under os serve.
Order with PR #28587 (in the merge queue, same file, different region): whichever lands second goes through the queue against the new base.

Landing: not governed. The seat lands it now (pr_ready + automerge_enable).

@objectstack-fleet
objectstack-fleet Bot marked this pull request as ready for review November 1, 2026 10:42
@objectstack-fleet
objectstack-fleet Bot enabled auto-merge November 1, 2026 10:42
@objectstack-fleet
objectstack-fleet Bot added this pull request to the merge queue Nov 1, 2026
Merged via the queue into main with commit 6b9c1ca Nov 1, 2026
51 checks passed
@objectstack-fleet
objectstack-fleet Bot deleted the claude/issue-28591-rehydrate-connectors branch November 1, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/m tests tooling

Projects

None yet

1 participant