feat(connectors): degrade + retry declarative instances on unreachable upstream (#3017) (#3049)

A provider factory can throw ConnectorUpstreamUnavailableError (spec) to mark an operational fault; the reconcile degrades that instance (visible husk with state/degradedReason, or old connector keeps serving on changed-config failures) and retries with exponential backoff plus on every metadata:reloaded. Configuration faults stay fatal at boot. The mcp provider classifies connect/tools-list failures as unavailable.

Author: os-zhuang

.changeset/adr-0097-mcp-nonfatal-boot.md

@@ -0,0 +1,35 @@
+---
+'@objectstack/spec': minor
+'@objectstack/service-automation': minor
+'@objectstack/connector-mcp': minor
+---
+
+feat(connectors): degrade + retry declarative instances whose upstream is unreachable (#3017)
+
+ADR-0097 kept every declarative-connector materialization failure fatal at
+boot. That is right for configuration faults (unknown provider, invalid
+`providerConfig`, unresolvable `credentialRef`, name conflict) but wrong for
+*operational* ones: a `provider: 'mcp'` instance must contact its MCP server
+(`tools/list`) to materialize, and a transient network blip aborted the whole
+app boot.
+
+- **spec**: a provider factory can now throw
+  `ConnectorUpstreamUnavailableError` (code `CONNECTOR_UPSTREAM_UNAVAILABLE`,
+  structural guard `isConnectorUpstreamUnavailable`) to mark a failure as
+  "upstream temporarily unreachable — degrade and retry" instead of fatal.
+- **service-automation**: the reconcile degrades such an instance in both boot
+  and reload modes: it registers an action-less husk (`state: 'degraded'` +
+  `degradedReason` on the `GET /connectors` descriptor) so the instance is
+  visible instead of silently missing — or, on a changed-config
+  re-materialization, keeps the old connector serving. A `connector_action`
+  against a degraded instance fails with the reason and a "retries
+  automatically" pointer. Degraded instances retry on an exponential backoff
+  (5s → 5min, reset by config edits) and on every `metadata:reloaded`
+  reconcile; recovery swaps the husk for the live connector atomically.
+  Reconcile runs (boot / reload / retry timer) are now serialized.
+- **connector-mcp**: the `mcp` provider classifies connect / `tools/list`
+  failures as upstream-unavailable; transport-shape validation stays a plain
+  (fatal) throw.
+
+Configuration faults remain loud boot failures — the carve-out is only for the
+unavailable marker.

docs/adr/0097-declarative-connector-instances.md

@@ -132,8 +132,44 @@ and skipped**, never crashing the live server; a changed instance's old connecto
 keeps serving until its replacement materializes successfully. Boot keeps the
 "fail loudly" contract.
 
+### Upstream availability: degrade, don't die (follow-up, landed — #3017)
+
+The fail-loud contract deliberately distinguishes **configuration** faults from
+**operational** faults. Unknown provider, invalid `providerConfig`, unresolvable
+`credentialRef`, name conflict — authoring mistakes — stay fatal at boot. But a
+provider whose factory must contact an upstream to materialize (the `mcp`
+provider connects and calls `tools/list`) can fail because that upstream is
+*temporarily unreachable*; aborting the whole app boot for one integration's
+network blip turns a degraded connector into a total outage.
+
+A factory signals the operational case by throwing an error carrying the
+`CONNECTOR_UPSTREAM_UNAVAILABLE` code (`ConnectorUpstreamUnavailableError` in
+`@objectstack/spec/integration`; the materializer's check is structural, not
+`instanceof`). The reconcile then — in both boot and reload modes — **degrades**
+that one instance instead of failing the run:
+
+- With no live connector under the name, an action-less husk is registered
+  (`state: 'degraded'` + `degradedReason` on the descriptor, `status: 'error'`
+  on the def) so `GET /connectors` shows the instance honestly instead of it
+  silently missing, and a `connector_action` dispatch fails with the reason and
+  "the platform retries automatically" — never the generic wiring hint.
+- On a changed-config re-materialization, the previous live connector keeps
+  serving untouched (the same guarantee every other reload failure gives).
+- Degraded instances are retried on an exponential backoff (5s doubling to a
+  5-minute ceiling; a config edit resets it) and immediately by any reconcile
+  (`metadata:reloaded`). Recovery replaces the husk atomically via
+  `registerConnector`; an instance deleted while degraded is dropped, husk and
+  pending retry included.
+
+The `mcp` provider classifies connect / `tools/list` failures as unavailable;
+transport-shape validation stays a plain (fatal) throw. Lazy connect-on-first-
+dispatch was rejected: MCP actions ARE the server's `tools/list`, so a
+connector materialized without connecting would register a zero-action def —
+exactly the plausible-but-dead shape this ADR exists to kill.
+
 ### Deliberate scope boundaries
 
 - **`providerConfig.spec` (openapi)** accepts an inline document, an http(s) URL, **or a package-relative file path** (#3016 follow-up). The connector still owns no filesystem access: the automation service injects a `loadPackageFile` capability into `ConnectorProviderContext` that resolves the ref against the declaring stack/package root (`packageRoot`, CLI default: the `objectstack.config.ts` directory) and **confines reads to that root** — absolute and `..`-escaping paths are rejected. Read/parse failures follow the reconcile policy above: fatal at boot, skipped on reload.
 - **MCP credentials** ride the transport (ADR-0024); for an http transport a resolved `auth` is folded into the request headers.
-- **MCP at boot** connects during materialization, so an unreachable server fails boot; a fail-soft "optional" marker for boot-time materialization is a possible future refinement (runtime reloads are already soft).
+- **A live `provider: 'mcp'` showcase demo** remains deferred: materialization needs a reachable MCP server, and the natural in-repo target (`@objectstack/mcp`, the platform's own MCP endpoint) is not listening until after the automation plugin's `start()` — the demo would deterministically boot degraded and heal seconds later, making the dogfood CI gate timing-sensitive. It should land together with an in-repo MCP fixture server (or an explicit boot-ordering story for self-connection).
+- **The openapi provider's remote-URL spec fetch** still throws plain on network failure (fatal at boot). It can adopt the same `CONNECTOR_UPSTREAM_UNAVAILABLE` classification once operators ask for it — the mechanism is provider-agnostic.

packages/connectors/connector-mcp/src/mcp-provider.test.ts

@@ -6,6 +6,7 @@
 
 import { describe, it, expect } from 'vitest';
 import type { ConnectorProviderContext } from '@objectstack/spec/integration';
+import { isConnectorUpstreamUnavailable } from '@objectstack/spec/integration';
 import type { McpClientLike, McpToolDescriptor, McpTransport } from './mcp-connector.js';
 import { createMcpProviderFactory, MCP_PROVIDER_KEY } from './mcp-provider.js';
 
@@ -82,3 +83,50 @@ describe('mcp provider factory (ADR-0097)', () => {
         ).rejects.toThrow(/kind must be 'stdio' or 'http'/);
     });
 });
+
+// ── #3017 — fault classification: config stays fatal, upstream degrades ─────
+
+describe('mcp provider fault classification (#3017)', () => {
+    const stdio = { transport: { kind: 'stdio', command: 'my-mcp' } };
+
+    it('classifies a connect failure as upstream-unavailable (retryable), keeping the cause', async () => {
+        const boom = new Error('connect ECONNREFUSED 127.0.0.1:9999');
+        const clientFactory = async (): Promise<McpClientLike> => { throw boom; };
+        const factory = createMcpProviderFactory({ clientFactory });
+
+        const err = await factory(ctx({ providerConfig: stdio })).then(
+            () => { throw new Error('expected rejection'); },
+            (e: unknown) => e,
+        );
+        expect(isConnectorUpstreamUnavailable(err)).toBe(true);
+        expect((err as Error).message).toMatch(/'github' could not reach its MCP server/);
+        expect((err as Error).message).toContain('ECONNREFUSED');
+        expect((err as { cause?: unknown }).cause).toBe(boom);
+    });
+
+    it('classifies a tools/list failure as upstream-unavailable and closes the client', async () => {
+        let closed = false;
+        const clientFactory = async (): Promise<McpClientLike> => ({
+            listTools: async () => { throw new Error('request timed out'); },
+            callTool: async () => ({}),
+            close: async () => { closed = true; },
+        });
+        const factory = createMcpProviderFactory({ clientFactory });
+
+        const err = await factory(ctx({ providerConfig: stdio })).then(
+            () => { throw new Error('expected rejection'); },
+            (e: unknown) => e,
+        );
+        expect(isConnectorUpstreamUnavailable(err)).toBe(true);
+        expect(closed).toBe(true); // discovery failure must not leak the connection
+    });
+
+    it('keeps transport-shape faults plain — configuration errors stay fatal at boot', async () => {
+        const factory = createMcpProviderFactory();
+        const err = await factory(ctx({ providerConfig: {} })).then(
+            () => { throw new Error('expected rejection'); },
+            (e: unknown) => e,
+        );
+        expect(isConnectorUpstreamUnavailable(err)).toBe(false);
+    });
+});

packages/connectors/connector-mcp/src/mcp-provider.ts

@@ -1,6 +1,7 @@
 // Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
 
 import type { ConnectorProviderFactory, ResolvedConnectorAuth } from '@objectstack/spec/integration';
+import { ConnectorUpstreamUnavailableError } from '@objectstack/spec/integration';
 import { createMcpConnector, type McpConnectorOptions, type McpTransport } from './mcp-connector.js';
 
 /**
@@ -103,9 +104,13 @@ function normalizeTransport(
  * {@link createMcpConnector} builds for a hand-wired MCP connector — one action
  * per tool, dispatched to the server's `tools/call`.
  *
- * The connection is opened at materialization; an unreachable server or invalid
- * transport therefore fails boot loudly (ADR-0097 fail-loud contract). Prefer a
- * fail-soft plugin instantiation for an *optional* server.
+ * The connection is opened at materialization. Faults are classified (#3017):
+ * an invalid transport shape is a *configuration* fault and throws plain —
+ * fatal at boot per the ADR-0097 fail-loud contract — while a connect /
+ * `tools/list` failure (server down, refused, timed out) is an *operational*
+ * fault and throws {@link ConnectorUpstreamUnavailableError}, which the
+ * materializer turns into a degraded instance that is retried with backoff
+ * instead of aborting the whole app boot.
  */
 export function createMcpProviderFactory(deps: McpProviderDeps = {}): ConnectorProviderFactory {
   return async (ctx) => {
@@ -116,14 +121,26 @@ export function createMcpProviderFactory(deps: McpProviderDeps = {}): ConnectorP
       : undefined;
     const include = includeList ? (toolName: string) => includeList.includes(toolName) : undefined;
 
-    const bundle = await createMcpConnector({
-      name: ctx.name,
-      label: ctx.label,
-      description: ctx.description,
-      transport,
-      include,
-      clientFactory: deps.clientFactory,
-    });
+    let bundle;
+    try {
+      bundle = await createMcpConnector({
+        name: ctx.name,
+        label: ctx.label,
+        description: ctx.description,
+        transport,
+        include,
+        clientFactory: deps.clientFactory,
+      });
+    } catch (err) {
+      // Everything past transport validation is talking to the server (connect,
+      // handshake, tools/list) — operational, hence retryable. A credential the
+      // server rejects also lands here: indistinguishable from the outside, and
+      // retrying it is loud (logged per attempt), never silent.
+      throw new ConnectorUpstreamUnavailableError(
+        `connector-mcp provider: connector '${ctx.name}' could not reach its MCP server: ${(err as Error).message}`,
+        { cause: err },
+      );
+    }
     return { def: bundle.def, handlers: bundle.handlers, close: bundle.close };
   };
 }

packages/services/service-automation/src/builtin/connector-nodes.ts

@@ -57,6 +57,18 @@ export function registerConnectorNodes(engine: AutomationEngine, ctx: PluginCont
 
             const handler = engine.resolveConnectorAction(cfg.connectorId, cfg.actionId);
             if (!handler) {
+                // A degraded declarative instance (#3017) is registered but has no
+                // handlers — say WHY dispatch fails and that recovery is automatic,
+                // instead of the generic wiring hint below.
+                const degraded = engine.getConnectorDegradedReason(cfg.connectorId);
+                if (degraded) {
+                    return {
+                        success: false,
+                        error:
+                            `connector_action '${node.id}': connector '${cfg.connectorId}' is degraded — ${degraded}. `
+                            + `Dispatch is unavailable until its upstream recovers; the platform retries automatically (#3017).`,
+                    };
+                }
                 return {
                     success: false,
                     error: