Skip to content

Commit 7735d1d

Browse files
feat(service-automation, runtime, spec): connector credential store and open-tier credentialRef chain (#27237) (#27344)
domain:server#3 · os-bill · session_01Utp6CXc5hequf76uJfzrdK <!-- claude-projects-attribution: {"github_login":"os-bill"} --> _Requested by **Bill** · [project thread](https://claude.ai/code/project/chan_016iMXWSZDzqWRWvWzdNo8wg?thread=cmsg_016iMXWSZDzqWRWvWzdNo8wgKURKuRGF1ewEVVZKx7U1i4)_ Fixes #27237 Clause-②: yes (widening) Metadata-protocol: none Skills-docs: updated here — skills/objectstack-automation/references/integration-recipes.md, content/docs/automation/connectors.mdx, packages/services/service-automation/README.md Ruling-ref: #27230 ruling A (comment 6468327468) and S1; #27237 Q3 ruling (comment 6470087293); seat surface call 6469928674 ADR-0087: n/a — nothing an author can write changed ## 维护者速读 - 改了什么:新增环境级连接器凭据存储 `sys_connector_credential` 和三条 REST 路由。`credentialRef` 按"环境变量 → 凭据存储 → 注入的解析器"的顺序解析;凭据缺失或被上游拒绝时,只让该实例降级,不再让启动失败。 - 为什么改:托管环境的管理员拿不到服务器环境变量,所以带连接器的应用装上后,凭据永远配不进去(#27230 裁定 A)。 - 风险与回滚:这是行为变化。缺失的 `credentialRef` 和 MCP 的 401/403 都不再让启动失败(见 ADR-0097 修订注记)。rest/openapi 任意一次调用返回 401/403 就会让实例降级,资源级的 403 也算。被拒的值若是门里存的,实例停住,直到重设、改声明或重启;若来自环境变量或 vault,就按退避重试。所以门里存的 token 遇到一次非凭据类的 403(比如限流),实例会停住,直到重新 PUT 或重启(triage 的升级项);env/vault 的值则不会。rejected 只有重设或 MCP 握手证明后才清除,rest/openapi 的 2xx 调用不算证明(端点可能根本不查凭据)。已知矛盾:env/vault 的值被拒后,实例按退避恢复并正常服务,列表仍显示 rejected,直到被证明、来源变化或重启。回滚就是 revert 本 PR,存储表留空无害。 - 先看:`service-automation` 的 `connector-credentials.ts`,以及 `plugin.ts` 里 reconcile、`holdConnectorForRejectedCredential`、`watchForCredentialRejection` 的改动;`runtime` 的 `domains/automation.ts` 里新加的两道门。 - 两分钟验证:`pnpm --filter @objectstack/service-automation exec vitest run src/connector-credential-door.test.ts`,共 22 项,跑的是真实 SQLite:缺失→降级,PUT→就绪且不重启,环境变量优先,rejected 跨重启保留,env/vault 被拒后恢复,并发写入,walled 姿态,值不外泄(包括 URL 编码和 base64 形式)。 - 需要你做的:本 PR 改了 `docs/adr`(Tier H),要等你批准;spec 新增导出和 `ConnectorMaterialization.credentialVerified` 还要过合约层评审。 ## What changes **The store.** `sys_connector_credential` is a new engine-owned object in `@objectstack/service-automation`. It is environment-level (`systemFields.tenant: false`, S1), has private access and no generic data door (`apiEnabled: false`). It holds one row per `credentialRef`: `value` is a `Field.secret` (the #7799 seam, so there is no second secret mechanism), plus `revision`, `connector_name`, `package_id`, `status` (`set` / `rejected`), `status_reason`, `last_verified_at`, and `set_at` / `set_by`, which only the door writes. Writes run under a system context that carries the caller's user id, so the audit stamps record who changed it. **The chain.** A `credentialRef` resolves from an environment variable of the same name, which always wins. Next comes the stored row, dereferenced through `resolveSecretField`. Last comes the host's injected `credentialResolver` (`connector-credentials.ts`). The value goes only to the provider factory. Nothing else gets it. Under the `group` and `isolated` tenancy postures the chain has no stored arm at all (S1): a row is not read, and the reference resolves from the environment or the host's resolver only. The posture is read from the `tenancy` service the way the rest plugin reads it: only "not registered" is absorbed, and a posture that cannot be read makes the rows unreadable for that reconcile (and the door's answer 503), never "no wall". **Reconcile on change.** Each instance's signature now includes where its reference resolves from (`env`, `object:<revision>`, `vault`, `missing`). A `PUT` or `DELETE` bumps the revision and runs the reconcile, so exactly the instances naming that reference re-materialize, with no restart. Door writes run on the reconcile queue, each with its reconcile, so two writes in one process never interleave and the instance ends on the value the store ends on. **Missing → degraded.** A reference no source answers for registers the instance degraded, with a reason that names the reference and the route to set it. It used to be a boot error. A host resolver that throws stays fatal, as before. **Rejected → degraded.** This follows the Q3 ruling on this card. `@objectstack/spec/integration` gains `ConnectorCredentialRejectedError` / `CONNECTOR_CREDENTIAL_REJECTED` / `isConnectorCredentialRejected`, the twin of the upstream-unavailable marker, with one ledger row (door `none`). `connector-mcp` throws it on a 401/403 at connect, handshake or `tools/list`. For `rest` and `openapi`, the handlers of a credentialed instance are wrapped: the first call that answers 401/403 still returns its answer, and then the instance is degraded. The reason quotes the upstream with the value scrubbed (`[credential withheld]`) in each spelling an upstream may echo: as sent, percent-encoded, base64, and, for a basic credential, the base64 `user:value` pair of its header. What follows depends on where the value came from: - A value stored through the door is marked `rejected` on its row, and the instance is held: no timer retry, since the same value would be refused again. Setting it again, a change of the declaration, or a restart re-attempts it. An instance with no `credentialRef` (its credential is written in the declaration) is held the same way. - A value from an environment variable or the host's resolver can change where the door cannot see it, so its instance is retried on the #3017 backoff (5s → 5min), and the listing reports it `rejected` until a provider proves it, it stops resolving from that source, or the process restarts. - `rejected` is cleared only by proof or a new value. A re-materialization proves nothing (`rest` and `openapi` first send the value on a call), so the mark survives it and a restart. It clears when the provider answers `credentialVerified` at materialization (new optional member of `ConnectorMaterialization`; `connector-mcp` sets it when its http handshake carried the credential), or when the value is set again. A call answered `2xx` is not proof, since the endpoint may not check the credential (a public path, an openapi operation with `security: []`): it only stamps `last_verified_at` on a row still at `set`. **Routes.** `GET /api/v1/automation/connectors/credential-refs`, plus `PUT` and `DELETE …/credential-refs/:ref`. All three require `manage_metadata`. Under `group` and `isolated` they refuse every caller, the platform operator included, with a 403 that names the posture; the service refuses its own writes there too. The query set is closed (none) on all three verbs, and the `PUT` body is the closed `{ value }`. A value starting with `secret:` is refused 400: ObjectQL stores such a value as given, as a handle to another `sys_secret` row. A first write another replica makes at the same moment meets the unique index on `ref` and answers 409. The routes answer with ADR-0112 standard codes only (400 / 403 / 404 / 409 / 503, and 501 when the service lacks the optional members), so no new wire code. Each route has a row in the route ledger. `IAutomationService` gains three optional members for them. **Install-local.** The install answer carries `credentialRefs`: each reference the package's connectors name, with the instances naming it, its status (`unknown` when the listing cannot be read) and its source. It also carries a `credentialNote` naming the ones still to set. An unset reference never refuses the install. Under `group` and `isolated`, where the credential routes refuse writes, the note names the environment variable or the host's resolver instead of the `PUT` route. For a package that names no credential, the answer's keys are unchanged. **Docs.** The `credentialRef` section of `content/docs/automation/connectors.mdx` is rewritten, along with the MCP paragraph, the failure table and the README option row. ADR-0097 gets an "Amended by" line, a note under §6 and a note under "Upstream availability" (seat call Q2). The automation skill's integration recipes no longer say an unset reference is a boot error. The tenant-audit census (page and counts ledger, six new write call sites), the system-context census (row 57b, the door's two `isSystem` reads) and the platform-object tenancy census are regenerated. The CLI's ADR-0131 D10 organization-ownership inventory gets a `sys_connector_credential` row (`column-drop`, as the flow credential store has). There are six changesets, one per published package: spec, service-automation, runtime, connector-mcp, cloud-connection and cli. Spec lines, all widening: `contracts/automation-service.ts` gets `ConnectorCredentialRef`, `ConnectorCredentialSetRequest`, `ConnectorCredentialClearRequest` and three optional `IAutomationService` members. `integration/connector-provider-errors.ts` gets `CONNECTOR_CREDENTIAL_REJECTED`, `ConnectorCredentialRejectedError` and `isConnectorCredentialRejected`. `integration/connector-provider.ts` gets the optional `ConnectorMaterialization.credentialVerified`. `api/error-code-ledger.zod.ts` gets one row. `system/constants/platform-object-names.ts` gets `sys_connector_credential`. Description text only, after the contract review: the `connectors` field in `stack.zod.ts`, three comments in `shared/connector-auth.zod.ts`, and the `auth` describe in `integration/connector.zod.ts`. Regenerated: `api-surface/{contracts,integration}.json`, `export-origins/{contracts,integration}.json` and five reference pages, `references/api/package-api-assembled.mdx` and `references/integration/connector.mdx` among them. Nothing was removed. ## Seat review, round 1 Every item from the round-1 review is fixed in this PR: 1. **must-fix: env / vault no longer held.** Only a door-stored value, or one written in the declaration, holds its instance. An environment or vault value falls back to the backoff, keeping its count across re-materializations, and the listing still shows the refusal. Pinned: an env value refused on a call (403) says "retried with backoff", arms the retry timer, re-materializes on the retry and serves again; a vault value refused at materialization recovers after a rotation. 2. **`rejected` survives re-materialization.** `markMaterialized` is gone. `markVerified` runs only on proof (`credentialVerified`). Pinned: the row stays `rejected` across a reconcile and a restart, and a provider proving at materialization clears it on restart. (Round 2 narrowed the call path further; see below.) 3. **`secret:` refused at the PUT door**, 400 `VALIDATION_ERROR`, with nothing written and the value not echoed. 4. **Door writes serialized** on the reconcile queue with their reconcile; `DUPLICATE_RECORD` maps to 409 `RESOURCE_CONFLICT`; `markRejected` and `markVerified` run on the same queue and check the revision. Pinned: two concurrent PUTs end at revision 2 with the instance on the second value; two stores inserting the first row at once give one success and one 409. 5. **No crypto provider pinned:** 503 `SERVICE_UNAVAILABLE`, no row written, the instance still degraded. 6. **Walled posture enforced in the chain** (not just reworded). Pinned for `group` and `isolated`: a row stored earlier is not read, the listing says `missing` / `writable: false`, set and clear refuse 403 naming the posture, and an environment variable still resolves. 7. **ADR-0097 amendment** now says a restart lifts the hold, that only a stored or declared credential is held, and that `rejected` holds across re-materialization until proof. Nits taken: `updatedAt` / `updatedBy` now come from the door-written `set_at` / `set_by`, so marking a row moves neither (pinned); the stale "MUST fail boot loudly … unresolvable credentialRef" comment is rewritten; the DELETE query refusal is pinned. The runtime door now also passes the service's own 403 through. The `connector-mcp` rejection message no longer says "is not retried". ## Seat review, round 2 **A. A `2xx` call no longer clears `rejected`** (option (a)). A call answered `2xx` now runs `markCallAccepted`, which stamps `last_verified_at` on a row still at `set` and does nothing else; `rejected` → `set` happens only through `credentialVerified` or a new `PUT`. The same rule holds for a refused environment or vault value: a `2xx` call leaves the listing's refusal and the backoff count alone. Pinned: a `set` row gets `last_verified_at` after a `2xx` call; a `rejected` row stays `rejected` (reason kept, no `last_verified_at`) after a `2xx` call, and a re-set clears it; an environment value's refusal stays listed while the instance serves again. Ablated twice, each red and restored: the call path running the proof path (two tests red: the env refusal cleared, the row back to `set`), and `markCallAccepted` stamping nothing (the `set` row's `last_verified_at` missing). The "verified" prose is aligned in `automation-service.ts`, `connector-provider.ts`, `sys-connector-credential.object.ts`, ADR-0097, `connectors.mdx` and two changesets. Known follow-ups, not changed here (B–E): - **B. Replicas are not coordinated beyond the unique insert.** Two replicas updating an existing row both write revision N+1 (last write wins), and a `markRejected` / `markVerified` on one replica can stamp a value another replica has just set. - **C. Status is per reference, acceptance per instance.** When a `rest` instance and an `mcp` (http) instance share a reference, the mcp handshake's proof clears a row the rest upstream refused. The rest instance's own `degradedReason` in the listing still shows its refusal. - **D. A restart leaves the held wording.** After a restart the instance is live again, but `status_reason` keeps "…until … set again or the server restarts" until the value is set again or proven. - **E. A credential written in the declaration is held.** An instance with no `credentialRef` is held after a refusal until its declaration changes or a restart. That is intended, and the ADR-0097 note says so. ## Contract review on `bda699c2` Triage's contract review (comment 6487646414) was FAIL with one must-fix. The must-fix and all six should-fixes are taken on this head. - **Must-fix: the `connectors` description.** `stack.zod.ts` said an unresolvable `credentialRef` is a hard boot error. The `.describe()` and its JSDoc now say an unknown provider or a name conflict is a boot error, while a `credentialRef` nothing resolves, or one the upstream refuses, degrades that instance until the credential is set. The JSON schema and `package-api-assembled.mdx` are regenerated from it. - **1. "Resolved at boot."** The three comments in `shared/connector-auth.zod.ts` and the describe in `integration/connector.zod.ts` now say the credential is resolved when the instance materializes. The same regeneration carries them. - **2. The walled 403 in the contract.** `setConnectorCredential`'s refusal list in `contracts/automation-service.ts` now names the 403 `PERMISSION_DENIED` under `group` / `isolated`, and the refusals it was also missing: the `secret:` 400, the 409 for a racing first write, and the 503 for an unreadable posture. `clearConnectorCredential` names the 403 too. - **3. Retry wording in `connector-provider-errors.ts`.** The docblock now says the instance waits for a change instead of a timer only when the value is a stored one or written in the declaration. A value from an environment variable or the host's resolver is retried on the backoff. - **4. Eval prose.** Case 2 of `skills/objectstack-automation/evals/integrations-declared.json` no longer expects an unset reference to fail the boot: it expects the reference to degrade the instance instead. The file sits at its token ceiling (1194 / 1194), so the sentence names no route. - **5. Encoded spellings withheld.** `withoutCredential`, `withoutCredentialThrown` and `upstreamRefusalMessage` now withhold every spelling from the new `credentialForms(value, username?)`: the value as sent, `encodeURIComponent(value)`, its base64, and for basic auth the base64 of `user:value`. Base64 is matched without its padding, and a spelling shorter than four characters is skipped. The provider factory's catch and the call watcher pass the instance's basic username; the store's write error withholds the value's own spellings. Two tests pin it: the list of spellings, and a basic-auth instance whose upstream echoes the encoded `Authorization` header and a percent-encoded query key, where no spelling reaches the degraded reason, the row's `status_reason` or a log record. - **6. The walled install note.** When the automation listing says none of the unset references is writable, as under the walled postures, the install-local `credentialNote` names the environment variable or the host's resolver and says the environment does not store connector credentials through the credential door. It no longer names the `PUT` route. One test pins it. Ablated, each red and then restored: `credentialForms` returning the raw value only (both encoded-spelling tests red, with the encoded header and URL reaching the reason); the `user:value` pair dropped (both red); and the install note ignoring the posture (the walled test red, with the note naming the `PUT` route). Kept as is, by the seat's call: - **Known contradiction: an environment or vault refusal stays listed after the instance recovers.** A value from an environment variable or the host's resolver that the upstream refused is retried on the backoff, and once it is accepted the instance serves again. The listing still reports the reference `rejected`, so `status: rejected` sits beside a ready instance, until a provider proves the value, the reference stops resolving from that source, or the server restarts. This follows from round 2's rule that a `2xx` call is not proof, and it is not changed here. - **Triage's escalation: a stored token is held on a 403 that is not about the credential.** A value stored through the door that meets a 403 for another reason, such as a rate limit or a resource-level denial, is marked `rejected`, and its instance is held until the value is `PUT` again or the server restarts. Unlike an environment or vault value, it does not recover on its own. ## Tests - `service-automation/src/connector-credential-door.test.ts` runs on real ObjectQL + SQLite with a reversible fake cipher (22 tests). It covers: missing → degraded, naming the ref; set → ready without a restart; set again → re-materialized with the new value; clear → degraded. It covers the 404 / 400 / 404 refusals, which write nothing. It checks the value appears in no log record (the kernel logger at `debug`, stdout+stderr+console captured), the listing, the descriptors, an engine read, or the audit mask set. It checks env wins with `source: env` and a 409 on `PUT`, and the order env → row → vault. Rejected at materialization (the mcp shape) and on the first call (401 and 403) both give degraded + `rejected` + scrubbed + held across a reconcile, and a re-set recovers. The round-1, round-2 and contract-review tests are listed in their sections above. - `runtime/src/domains/automation-connector-credential-door.test.ts` (17 tests) checks: 401 anonymous; 403 without `manage_metadata` on all three verbs; the walled 403 naming the posture for an admin and for the platform operator under `group` and `isolated`; admission under `single` and with no tenancy service; a closed query on GET, PUT and DELETE; a closed body that never echoes; refusal code mapping (403 included); a decoded `/` in a ref; 501 without the members. - `cloud-connection/src/marketplace-install-local-credential-refs.test.ts` (4 tests) and `connector-rest/src/connector-rest-plugin.test.ts`, the latter on the real rest provider: a 401 on a call degrades the instance and scrubs the echoed token. - `cli/src/utils/organization-ownership-inventory.test.ts` holds the new inventory row. CI's Test Core (4/8) went red at `70a67e29` because the row was missing; reproduced locally with the row stashed (1 failed), and green with it (26 passed). - Updated to the ruled behaviour: `connector-materialization.test.ts` (missing ref → degraded), `connector-fail-cause.test.ts` (a throwing host resolver stays fatal), `connector-mcp` `mcp-provider.test.ts` / `mcp-http-discovery-faults.test.ts` (401/403 → the marker, whose message no longer says "is not retried"; a declarative instance with no `credentialRef` degrades and is not retried), and the two object-list equalities. - Ablation: every negative pin was put back to the forbidden behaviour and watched go red, then restored. First round: the set log carried the value; a listing entry carried the resolved value; the descriptors spread `def.authentication`; `value` became `Field.text` (masked read and audit mask); `withoutCredential` became a no-op; the unknown-key 400 echoed the body. Round 1, eleven more, each red and restored: env refusals held; materialization resetting `rejected`; the `secret:` check removed; the `secret:` refusal echoing the value; the door write taken off the queue; the `DUPLICATE_RECORD` mapping removed; the 409 echoing the value; the no-crypto mapping removed; the 503 echoing the value; the store kept in the walled chain; and the DELETE query check skipped. Round 2, two more, and the contract review, three more (see their sections above). ## Acceptance notes - **403 is read as a refused credential on rest/openapi, as ruled.** An upstream that answers 403 for a resource-level denial or a rate limit (GitHub's secondary limit is a 403) degrades the instance. For a stored value the row is marked `rejected` and the instance is held until the value is `PUT` again or the server restarts (triage's escalation, above); for an environment or vault value the instance retries on the backoff, and the listing keeps it `rejected` (the known contradiction, above). - **Several replicas.** A write reconciles the replica that took it. Another replica learns of it on its next reconcile (a degraded instance's retry timer, a metadata reload) or at restart. A replica whose instance is live with the old value keeps it until then. ObjectQL offers no public conditional update, so replicas are not coordinated beyond the unique index on the first insert (409): later writes to an existing row are last-writer-wins, and a status mark on one replica can land on a value another just set (follow-up B above). - **Vault presence.** The `GET` listing and the install answer still read each vault-sourced value to tell whether it is present, and drop it on the spot. Left as is: the host's `credentialResolver` contract has no presence probe, and adding one would widen the plugin options for a read that happens only on those two answers. - The api trigger's `secretRef` still resolves through the injected resolver alone (env by default), as before. - **Left for other owners.** The QA checklist item `integration-system.connector-auth-kind-application` (`docs/qa/platform-checklist/areas/integration-system.json`) still expects an unset `credentialRef` to fail the boot, and cites a test name that no longer exists; it needs a revision from the QA lane. ADR-0097's reload paragraph still lists an unresolvable `credentialRef` among the entries that are "logged and skipped". The amendment notes govern it, and the original paragraph is left as written. - The vault arm needed nothing in objectstack-ai/cloud: it is the existing `credentialResolver` option. - Premise 5 (hosted runtimes ship the provider plugins) is out of scope and was not measured. I read only that the CLI depends on the three provider packages. ## Verification Final head `07b65dd4`, with `origin/main` merged at `82760e4e`. - **Gates.** `node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack --ran <record>` reports 133 derived, 133 run, 0 NOT-MEASURED, 0 UNRUN, and every exit code is 0. The whole list ran on `aff9d4f4`, the head before the last commit. One gate was red there: `check-skills-token-ratchet`, because the realigned eval sentence put `integrations-declared.json` 34 tokens over its ceiling. The last commit shortens that one sentence and changes nothing else, and the nine skill gates, the ratchet among them, ran again on the final head with exit 0. Nothing was waived. - **Fixed on the way.** First round: `check:adr-symbol-anchors` (the auth builder keeps the name `resolveInstanceAuth` that ADR-0143 anchors), `check-tenant-audit-census`, `check:doc-authoring`, `check:route-ledger-census` (105 to 108), `check:system-context-census` (row 57b) and `check-skills-token-ratchet`. Round 2: after `origin/main` was merged, the tenant-audit census (one more write site, from `markCallAccepted`) and the platform-object tenancy census (main had added `sys_trusted_key`) were regenerated, with the census page's prose figures restated. CI's "TypeScript Type Check" red at `d68bc8d9` was its `typecheck-workspace` lane `cancelled` by the next push, not a type error. - **Tests,** on `aff9d4f4`. service-automation 2790 passed. runtime 8630 passed, 19 skipped. connector-mcp 46, connector-rest 27, connector-openapi 41, cloud-connection 751. spec 23764 passed, 1 todo. cli 6567 passed, 9 skipped. `tsc --noEmit` is clean for service-automation, runtime, connector-mcp and cloud-connection, and the spec build, with its declaration emit, passes. - **Lint.** `pnpm lint` (eslint over the whole repo) exits 0 on the final head. ## Human Minimum - [x] 1. Schemas start as Zod; `packages/spec` holds no runtime logic. - [x] 2. Every name I added uses the casing for its kind. - [x] 3. No formatter ran; `pnpm lint` passes. - [x] 4. One issue, one branch off an up-to-date `main`, one pull request. - [x] 5. A change that publishes carries a changeset, and both declaration lines are filled in here and in the changeset. - [x] 6. `packages/spec` touched: its generated artifacts are regenerated and committed. - [x] 7. No hand edits to `content/docs/references/`, `packages/*/CHANGELOG.md` or `content/docs/releases/`. - [x] 8. Title, body and comments are in English. - [x] The *Before Submitting* list in CONTRIBUTING.md is done. --- _Generated by [Claude Code](https://claude.ai/code/session_01Utp6CaqVRa7ow6CjE1JwaG)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent aef631b commit 7735d1d

54 files changed

Lines changed: 3818 additions & 212 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'@objectstack/cli': patch
3+
---
4+
5+
fix(cli): the organization-ownership inventory has a row for `sys_connector_credential`
6+
7+
Clause-②: no
8+
Metadata-protocol: none
9+
10+
The ADR-0131 D10 inventory now has a row for the new environment-level connector credential store (`systemFields.tenant: false`). Its fate is `column-drop`, the same fate the flow credential store has: the upgrade plan treats the table as deployment-level state with no organization column, so a database carrying it is not refused as an uninventoried platform table.
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
'@objectstack/runtime': patch
3+
---
4+
5+
feat(runtime): `GET`/`PUT`/`DELETE /api/v1/automation/connectors/credential-refs` — list the connector credentials an environment's connectors name, and set or clear one
6+
7+
Clause-②: yes (widening)
8+
Metadata-protocol: none
9+
10+
Three new routes serve the automation service's connector credential store (#27237). `GET …/credential-refs` lists every `credentialRef` an installed connector names, plus stored references nothing names any more. Each entry has its `status` (`set` / `missing` / `rejected`), its `source` (`env` / `object` / `vault` / null) and the instances naming it, and never a value. `PUT …/credential-refs/:ref` with the closed body `{ "value": "…" }` stores a value, and the instances naming the reference re-materialize without a restart. `DELETE …/credential-refs/:ref` removes it.
11+
12+
All three require the `manage_metadata` capability. Under the `group` and `isolated` tenancy postures they refuse every caller with a 403 that names the posture, because connector credentials are environment-level. They take no query parameters. The service's refusals keep their standard codes: 400 `VALIDATION_ERROR` (an empty value, or one starting with `secret:`), 403 `PERMISSION_DENIED` (the service's own walled refusal), 404 `RESOURCE_NOT_FOUND` for an undeclared reference, 409 `RESOURCE_CONFLICT` for one an environment variable supplies or a first write another replica made at the same moment, and 503 `SERVICE_UNAVAILABLE` with no crypto provider. A mounted service without these optional members answers 501.
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
'@objectstack/spec': patch
3+
---
4+
5+
feat(spec): `ConnectorCredentialRejectedError` and the connector credential members of `IAutomationService`
6+
7+
Clause-②: yes (widening)
8+
Metadata-protocol: none
9+
10+
Three additions for the connector credential store (#27237):
11+
12+
- `@objectstack/spec/integration` exports `CONNECTOR_CREDENTIAL_REJECTED`, `ConnectorCredentialRejectedError` (with `status`: 401 or 403) and the structural check `isConnectorCredentialRejected`. They sit beside the upstream-unavailable marker. A connector provider factory throws it when the upstream refuses the credential it sent, and the materializer then degrades the instance instead of failing the boot. The code is registered in the error-code ledger under `@objectstack/spec`.
13+
- `ConnectorMaterialization` gains the optional `credentialVerified`. A provider factory sets it when the upstream accepted the instance's credential while materializing (the MCP handshake over http); the automation service then records the credential as verified, and a stored one marked `rejected` returns to `set`. A provider that first sends the credential on a call leaves it unset; a call answered 2xx is not taken as proof.
14+
- `IAutomationService` gains three optional members, `listConnectorCredentialRefs`, `setConnectorCredential` and `clearConnectorCredential`, with their types `ConnectorCredentialRef`, `ConnectorCredentialSetRequest` and `ConnectorCredentialClearRequest`. A listing entry never carries a value.
15+
16+
`PLATFORM_OBJECT_NAMES['service-automation']` lists the new `sys_connector_credential` object.
17+
18+
The descriptions now match the new behaviour. The stack's `connectors` field says a `credentialRef` nothing resolves, or one the upstream refuses, degrades that instance until the credential is set, where it said such a reference failed the boot. The auth and connector schemas say a credential is resolved when the instance materializes, not at boot.
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
'@objectstack/service-automation': patch
3+
---
4+
5+
feat(service-automation): an administrator sets a declarative connector's credential in the environment's credential store, and a missing one degrades its instance instead of failing the boot
6+
7+
Clause-②: yes (widening)
8+
Metadata-protocol: none
9+
10+
A provider-bound `connectors:` entry names its secret with `auth.credentialRef`. Before, the open tier resolved that name from the server's environment variables only, and a name with no value failed the boot. On a hosted environment, where the administrator who installs an app has no server environment, a connector-bearing app could never be credentialed.
11+
12+
The reference now resolves through three sources, in order: an environment variable of the same name (it always wins), then the new `sys_connector_credential` store, then the host's `credentialResolver` if one is injected. The store keeps one row per reference with its value in a `Field.secret` column: encrypted at rest by the host crypto provider and masked on every read. It is written through `setConnectorCredential` / `clearConnectorCredential` and listed by `listConnectorCredentialRefs` on the `automation` service, which the REST routes `GET`/`PUT`/`DELETE /api/v1/automation/connectors/credential-refs` serve.
13+
14+
- A reference no source answers for degrades the instance naming it, with a reason that names the reference and how to set it. Before, it failed the boot. A `credentialResolver` that throws is still fatal at boot.
15+
- Setting or clearing a credential reconciles at once: every instance naming the reference re-materializes, or is torn down and degraded, with no restart.
16+
- A credential the upstream refuses (HTTP 401 / 403) degrades the instance with the upstream's message, never the value. An `mcp` instance is refused when it connects; a `rest` or `openapi` instance on the first call that answers 401 / 403, which still returns its answer. A stored credential is marked `rejected`, and its instance is not retried until the credential is set again, the declaration changes, or the server restarts; the mark itself stays until the value is set again or an `mcp` handshake proves it. A call answered 2xx does not clear it, since the endpoint may not check the credential; it only stamps the last-verified time of a credential still `set`. A credential from an environment variable or the host's resolver is retried on the usual backoff instead, since it can change where the service cannot see it, and the listing reports it `rejected` until a provider proves it or the server restarts.
17+
- Under the `group` and `isolated` tenancy postures the store is not read, and writes to it are refused with a 403 naming the posture.
18+
- A value starting with `secret:` is refused with a 400: the data engine would store it as a handle to another stored secret. Writes run one at a time with their reconcile, so the instances end on the value the store ends on; a first write another replica makes at the same moment answers 409.
19+
- The value never appears in a log record, the listing, `GET /connectors`, an error body or a read of the object. A provider message that echoes it carries `[credential withheld]` instead, and so does one that echoes it percent-encoded or base64-encoded, including the base64 `user:value` pair of a basic credential.
20+
21+
A deployment that relied on the boot failing for an unset reference now boots, with that instance degraded and named on `GET /api/v1/automation/connectors`.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'@objectstack/cloud-connection': patch
3+
---
4+
5+
feat(cloud-connection): the install-local answer lists the connector credentials the installed package names, and which are not set yet
6+
7+
Clause-②: yes (widening)
8+
Metadata-protocol: none
9+
10+
`POST /api/v1/marketplace/install-local` answers with `credentialRefs` when the package's connectors name any `auth.credentialRef`. Each entry has the reference, the instances naming it, its `status` from the automation service's credential listing (`set` / `missing` / `rejected`, or `unknown` when that listing cannot be read) and its `source`, never a value. When any of them is not set, a `credentialNote` names them and says how to set them. An unset reference never refuses the install: the instance naming it installs degraded and materializes once the credential is set (#27237). Under the `group` and `isolated` tenancy postures, where the credential routes refuse writes, the note names the environment variable or the host's resolver instead of the `PUT` route. The answer's keys are unchanged for a package whose connectors name no credential.
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
---
2+
'@objectstack/connector-mcp': patch
3+
---
4+
5+
fix(connector-mcp): a credential the MCP server rejects is thrown as `ConnectorCredentialRejectedError`, so the instance is degraded instead of failing the boot
6+
7+
Clause-②: yes (widening)
8+
Metadata-protocol: none
9+
10+
An HTTP 401 / 403 on connect, handshake or `tools/list` used to throw a plain `Error`, which failed the boot. The provider now throws `ConnectorCredentialRejectedError` from `@objectstack/spec/integration`, with the status and the SDK error as `cause`; its message no longer says the fault "is not retried", since whether it is now depends on where the credential came from. The automation service degrades the instance naming the refusal and marks a stored credential `rejected` (#27237). Every other connect or `tools/list` failure is unchanged.
11+
12+
An http instance whose handshake carried its credential (any auth but `none`, and not an api-key sent as a query parameter) now answers `credentialVerified: true`, so a stored credential the server accepted is recorded as verified.

‎content/docs/automation/connectors.mdx‎

Lines changed: 95 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,10 @@ At boot, the automation service resolves each provider-bound entry:
7676
three in `plugins:` by default, paired with the `automation` capability
7777
that performs the materialization.
7878
2. **The factory validates `providerConfig`** and the automation service
79-
resolves `auth.credentialRef` to its secret.
79+
resolves `auth.credentialRef` to its secret (see
80+
[What `credentialRef` resolves through](#what-credentialref-resolves-through)).
81+
A reference that is not set yet does not stop the boot: that one instance
82+
registers degraded until an administrator sets it.
8083
3. **The result is registered** on the connector registry — indistinguishable
8184
from a hand-written connector. Its **actions are derived from the
8285
upstream** (the OpenAPI document's operations, the MCP server's
@@ -224,8 +227,9 @@ paginated tool list is followed through every page (`nextCursor`), up to 100
224227
pages; a server still paging after that fails discovery instead of yielding a
225228
truncated tool list. A server that is unreachable or failing degrades the
226229
instance, which is retried. A credential the server rejects (HTTP 401 / 403 on
227-
connect, handshake or `tools/list`) is a configuration fault instead: it fails
228-
the boot and is not retried — fix the credential and reload.
230+
connect, handshake or `tools/list`) is a configuration fault instead: the
231+
instance is degraded naming the refusal, as described under *A credential the
232+
upstream refuses* below.
229233

230234
<Callout type="warn">
231235
**Declarative stdio transports are denied by default.** Materializing one
@@ -275,18 +279,94 @@ Variant notes:
275279
- **`basic`** — the `username` stays in metadata (it is not a secret); only
276280
the password resolves through `credentialRef`.
277281

278-
**What `credentialRef` resolves through.** In the open tier the ref is the
279-
name of an **environment variable** (`BILLING_API_TOKEN` above means
280-
`process.env.BILLING_API_TOKEN` on the server). A ref that resolves to nothing
281-
is a **hard boot error** — an app must not silently run with a dead connector.
282-
The enterprise tier swaps in a vault/KMS-backed resolver
283-
(`AutomationServicePluginOptions.credentialResolver`) without changing what
284-
you author: the entry still just names a ref.
282+
### What `credentialRef` resolves through
283+
284+
The metadata author names a credential; the administrator who installs the app
285+
holds it. The automation service asks three sources, in order, and the first
286+
that answers wins:
287+
288+
1. **An environment variable of the same name** on the server
289+
(`BILLING_API_TOKEN` above means `process.env.BILLING_API_TOKEN`). A
290+
self-hosted operator keeps this path, and it always wins.
291+
2. **The connector credential store**, `sys_connector_credential`: one row per
292+
reference, its value encrypted at rest by the host's crypto provider and
293+
masked on every read. An administrator fills it through the API below, with
294+
no access to the server environment, which is the hosted case.
295+
3. **The host's resolver**, when it injects one
296+
(`AutomationServicePluginOptions.credentialResolver`, such as a vault or KMS
297+
client). You author the same entry either way: it still just names a ref.
298+
299+
A reference no source answers for **does not fail the boot**. An app installs
300+
first and gets its credentials after, so that one instance registers
301+
**degraded**: it is listed on `GET /api/v1/automation/connectors` with a reason
302+
that names the reference, and a dispatch to it fails with the same reason.
303+
304+
#### Setting a credential
305+
306+
Three routes, each requiring the `manage_metadata` capability:
307+
308+
```http
309+
GET /api/v1/automation/connectors/credential-refs
310+
PUT /api/v1/automation/connectors/credential-refs/BILLING_API_TOKEN { "value": "…" }
311+
DELETE /api/v1/automation/connectors/credential-refs/BILLING_API_TOKEN
312+
```
313+
314+
- **`GET`** lists every reference an installed connector names, plus every
315+
stored one that nothing names any more. Each entry has a `status` (`set`,
316+
`missing` or `rejected`), a `source` (`env`, `object`, `vault` or `null`),
317+
whether this door can write it, and the instances naming it with their state.
318+
A stored one also says when and by whom it was last set (`updatedAt`,
319+
`updatedBy`) and when it was last seen working (`lastVerifiedAt`). It never
320+
includes a value.
321+
- **`PUT`** stores the value, then reconciles. Every instance naming the
322+
reference re-materializes with it **without a restart**, and the answer is
323+
the reference's entry after that. The route refuses, before writing anything:
324+
a reference no installed connector declares (`404`), one an environment
325+
variable already supplies (`409`, because the variable wins and a stored value
326+
would never be read), an empty value, and a value starting with `secret:`
327+
(`400`: the store would read it as a handle to another stored secret). With no
328+
crypto provider it answers `503`, since a credential is only ever stored
329+
encrypted. Writes on one server run one at a time, each with its reconcile, so
330+
the instances always end on the value the store ends on.
331+
- **`DELETE`** removes the stored value. The instances naming the reference
332+
are torn down and registered degraded again.
333+
334+
Installing a package reports the same facts for it: the install-local answer
335+
(`POST /api/v1/marketplace/install-local`, which `os package install` calls)
336+
lists the `credentialRefs` the package's connectors name, each with its status,
337+
plus a `credentialNote` naming the ones still to set.
338+
339+
**A credential the upstream refuses** (HTTP `401` / `403`) degrades the
340+
instance, with a reason quoting the upstream's message, never the value. An
341+
`mcp` instance finds this out when it connects. A `rest` or `openapi` instance
342+
finds out on the first call answering `401` / `403`; that call returns its
343+
answer as usual, and the instance is degraded after it. What happens next
344+
depends on where the value came from:
345+
346+
- **A value stored here** is marked `rejected`, and its instance is not
347+
retried until the credential is set again, its declaration changes, or the
348+
server restarts. The mark outlives a restart: it clears only when the value
349+
is set again or an `mcp` handshake that carries it succeeds. A `rest` /
350+
`openapi` call answered `2xx` does not clear it, because the endpoint may not
351+
check the credential at all.
352+
- **A value from an environment variable or the host's resolver** can change
353+
where this door cannot see it (a rotation, or a `403` rate limit that lifts),
354+
so its instance is retried on the same 5s→5min backoff as an unreachable
355+
upstream. The listing reports it `rejected` until an `mcp` handshake proves
356+
it or the server restarts, even while the instance is serving again.
357+
358+
Credentials are **environment-level**: one value per reference for the whole
359+
environment, read by whichever instance names it. On a multi-organization
360+
environment (the `group` or `isolated` tenancy posture) the store is not used
361+
at all. The three routes refuse with a message naming the posture, and a
362+
reference resolves from the environment variable or the host's resolver only,
363+
so a row stored before the posture changed is not read.
285364

286365
The same resolver serves an `api` flow's start-node `config.secretRef`, the
287-
name of its inbound hook secret. That one is read at every post rather than at
288-
boot, so a name with no value refuses each post with `503` instead of failing
289-
the boot ([Flows](/docs/automation/flows)).
366+
name of its inbound hook secret. That one goes through the injected resolver
367+
alone (environment variables by default) and is read at every post rather than
368+
at boot, so a name with no value refuses each post with `503`
369+
([Flows](/docs/automation/flows)).
290370

291371
Inlining a secret is rejected when the stack is validated. Writing the runtime
292372
shape on a provider-bound instance —
@@ -359,7 +439,8 @@ retry):
359439
|:---|:---|
360440
| `provider` names no installed factory | **Hard boot error** naming the entry, the provider, and the plugin that supplies it — add the executor to `plugins:` |
361441
| Invalid `providerConfig` (missing `baseUrl` / `spec` / `transport`, wrong shape) | **Hard boot error** from the provider factory |
362-
| `credentialRef` resolves to nothing | **Hard boot error** — set the env var (open tier) or fix the vault ref |
442+
| `credentialRef` resolves to nothing | **Not fatal**: the instance registers **degraded**, naming the reference — set it (`PUT …/credential-refs/:ref`), and it materializes without a restart |
443+
| The upstream refuses the credential (`401` / `403`) | **Not fatal**: the instance registers **degraded**. A stored credential is marked `rejected` and waits until it is set again (or a restart); an environment or vault credential is retried on the backoff |
363444
| Instance `name` collides with a plugin-registered connector | **Hard boot error** — there is no silent precedence between the two worlds |
364445
| stdio transport without host opt-in | **Hard boot error** with the `declarativeStdio` fix inline — a security rejection is never retried |
365446
| MCP server unreachable / `tools/list` fails; remote spec URL unreachable or transiently failing (408/429/5xx) | **Not fatal**: the instance registers **degraded** (visible on `GET /api/v1/automation/connectors` with the reason), dispatches fail with that reason, and the platform retries on a 5s→5min backoff — recovery is automatic |

0 commit comments

Comments
 (0)