Repository navigation
Commit 262ab67
feat(connector-openapi): an offline normalize-openapi script — Swagger 2.0 to 3.0, YAML to JSON, $ref bundled and inlined, absolute server, sha256 and an include list (#27436)
Fixes #27373
Clause-②: no
Metadata-protocol: none
## 维护者速读
- **改了什么**:`connector-openapi` 包内新增离线脚本 `normalize-openapi`(只在仓库里,不进 npm
包),把厂商的 Swagger 2.0 / YAML / 多文件 OpenAPI 文档转成 `openapi` provider
直接能加载的单个 3.x JSON,并打印写入字节的 sha256 和按模式生成的 `include` 列表。
- **为什么改**:上架流水线和应用作者拿到的厂商文档多数是 2.0、YAML 或满是 `$ref`,运行时按设计拒收 2.0、并静默丢掉
`$ref` 形式的参数/请求体/响应;没有这一步,每个作者都得手工转换,而且会悄悄转错。
- **风险与回滚**:运行时零改动(`src/` 与发布的 `dist/` 字节不变),只新增两个 devDependency(Scalar
的 `@scalar/json-magic`、`@scalar/openapi-upgrader`,均 MIT);回滚即 revert 本
PR。
- **先看**:① `scripts/normalize-openapi.mts` 的 `Dereferencer.follow` /
`stub`(在递归处与 `--max-depth` 处把 schema 引用换成带说明的类型桩,这是有损的取舍);②
`fillEmptyConsumes`(空 `consumes` 默认拒收);③ `package.json` 的依赖与 `typecheck`
行。
- **两分钟验证**:`pnpm --filter @objectstack/connector-openapi exec vitest
run scripts/normalize-openapi.test.ts`,预期 `Tests 13 passed (13)`。
- **需要你做的**:无,席位落地。
## What changes
-
**`packages/connectors/connector-openapi/scripts/normalize-openapi.mts`**
(new): one script, a small CLI and an exported `normalizeOpenApi()`:
`pnpm --filter @objectstack/connector-openapi normalize-openapi --in
FILE_OR_URL --out FILE.json [--include-pattern REGEX]... [--max-depth N]
[--server-url URL] [--empty-consumes MEDIA_TYPE]`.
1. reads a file or an http(s) URL, JSON or YAML;
2. bundles every external `$ref` (sibling files of a file document,
confined to its directory; URLs only for a URL document);
3. converts Swagger 2.0 to OpenAPI 3.0; a 3.x document keeps its
version;
4. inlines every `$ref` reachable from `paths` / `webhooks`, then drops
the `components` sections only references used (`securitySchemes` stay,
they are referenced by name);
5. makes `servers[0].url` absolute (server variables take their
defaults; a relative URL resolves against the source URL, or
`--server-url`; a relative URL in a file document is refused until
`--server-url` is given);
6. with `--include-pattern`, emits the `providerConfig.include` list:
every operation whose action key, or whose `METHOD /path`, matches. The
key is computed with the provider's own `operationActionKey` (imported
from `src/openapi-connector.ts`, not re-derived), so the list and the
provider can never disagree;
7. writes `JSON.stringify(doc, null, 2)` plus a newline and prints a
JSON summary whose `sha256` is the hash of exactly those bytes, with the
cut counts and `refsLeft`.
Exit codes: 0 written, 1 refused (message names the remedy), 2 usage.
- **Where inlining stops, and why it has to.** "No `$ref` left" and
"every schema fully inlined" cannot both hold for Stripe: measured on
`spec3.json` (3.0.0, 431 paths, 612 operations, 117 back-edges among
component schemas), every one of the 612 operations expands past
2,000,000 JSON nodes when only true recursion is cut. So a SCHEMA
reference becomes a stub when it is recursive, or deeper than
`--max-depth` named-schema hops (default 2); a `$ref` inside a vendor
`x-` extension value is stubbed too, since its target is dropped with
`components`. The stub keeps the target's `type`, `nullable`, `title`,
`description` and says which reference it replaced and why. Structural
references (parameters, responses, request bodies, headers, path items)
are always inlined. Literal data (`example`, a schema's `default` /
`enum` / `const` / `examples`, Example Objects) is copied, never walked;
a PROPERTY named `default` is still a schema.
- **The one Swagger 2.0 repair, refused by default.** An operation-level
`consumes: []` clears the global list, so a faithful conversion gives
that request body no media type and therefore no schema: the body would
vanish from the action silently. DocuSign's eSignature document does
this on 187 of its 414 operations. The script refuses and names the
operations; `--empty-consumes application/json` fills them, and the
summary counts them.
- **`packages/connectors/connector-openapi/tsconfig.scripts.json`**
(new) puts `scripts/**` in a tsc program, named by the package's
`typecheck` script (precedent `packages/spec/tsconfig.scripts.json`);
without it the new directory would be TypeScript no program reads
(`check:type-check-coverage` SOURCES_COVERED / TESTS_COVERED). This file
is outside the claim's declared file surface (one package-root config);
it is the only such path.
- **`package.json`**: `normalize-openapi` script entry, the `typecheck`
line, two devDependencies. **`pnpm-lock.yaml`**: regenerated by `pnpm
install --lockfile-only` (38 added lines, nothing else moved).
- **`content/docs/automation/connectors.mdx`**: one paragraph and one
command where the page says "convert it to OpenAPI 3.x offline".
- ⛔ `openapi-provider.ts` and `openapi-connector.ts` are untouched; the
provider still refuses every non-3.x document.
### The new third-party dependencies (read from the npm registry
2026-10-29)
| Package | Role | Licence | Version | Last release | Weekly downloads |
Direct deps |
|:---|:---|:---|:---|:---|:---|:---|
| `@scalar/openapi-upgrader` | Swagger 2.0 → OpenAPI 3.0 (`upgrade(doc,
'3.0')`) | MIT | 0.4.6 | 2026-10-23 | 5,508,440 (order 10^6) | 2
(`@scalar/helpers`, `@scalar/openapi-types`) |
| `@scalar/json-magic` | bundle external `$ref` (files, URLs, YAML/JSON
parse) | MIT | 0.17.0 | 2026-10-23 | 7,140,252 (order 10^6) | 4
(`@scalar/helpers`, `pathe`, `undici`, `yaml`) |
Both live in the `scalar/scalar` monorepo and release roughly weekly (38
and 74 published versions). Both are pre-1.0, so the caret ranges pin
the minor. The lockfile adds four packages in all (`@scalar/helpers`
0.16.4 and `@scalar/openapi-types` 0.9.8 transitively; `pathe`,
`undici`, `yaml` were already in the tree). Weekly counts come from the
registry search endpoint (`registry.npmjs.org/-/v1/search`), because
`api.npmjs.org` is refused by this container's egress proxy.
Why two packages and not one: `@scalar/openapi-parser` (MIT, 0.29.18,
2026-10-28) would be the one-package option, but its own `upgrade()`
targets 3.1 only (the card's scope is 2.0 → 3.0), its `load()` is
deprecated in favour of `@scalar/json-magic/bundle`, and it depends on
both of these packages plus a validator (ajv). So the two direct
dependencies are a strictly smaller tree. Alternatives read and
rejected: `swagger2openapi` 7.0.8 (BSD-3-Clause, last release
2021-07-07), `@apidevtools/swagger-parser` 13.1.0 (MIT; parses and
dereferences, does not convert 2.0), `@redocly/openapi-core` 2.72.0
(MIT; bundles, does not convert 2.0, 13 direct deps). The inlining
itself is this script's own code: no library inlines a recursive
document into finite JSON, which is the cut policy above.
## Tests
`scripts/normalize-openapi.test.ts`, 13 cases, no network (a URL input
is served by an injected fetch). Every normalized output is loaded
through the REAL provider (`createOpenApiProviderFactory`, file-path
`spec` via `loadPackageFile`), and "no `$ref`" is asserted on both the
document and the materialized connector definition.
- Fixtures (hand-written, trimmed, not copies of the vendor files):
`scripts/fixtures/swagger2-esignature-shape.json` (Swagger 2.0, `host` /
`basePath`, body parameter, an operation-level `consumes: []`, a
recursive definition) and `scripts/fixtures/openapi3-billing-shape.yaml`
plus `billing-shared.yaml` (OpenAPI 3.0 YAML across two files, customer
↔ subscription recursion, a vendor `x-expansionResources` extension
holding `$ref`s, shared `responses.default` and parameter, a property
named `default`).
- Pinned: the 2.0 refusal names the operation; conversion keeps the body
schema and path parameter in the action; recursive stub; external file
bundled; structural refs inlined; `default` as property vs literal;
extension stub; action keys equal the provider's; the generated
`include` list materializes exactly those actions; zero-match and
invalid patterns refused; `--max-depth 0` stubs; relative server
resolved against a source URL, variables filled, refused from a file
without `--server-url`; non-OpenAPI and HTTP 404 refused; the CLI's
printed sha256 equals the hash of the file it wrote; exit codes 1 and 2
write nothing.
- One-time ablation (not kept): removing the `inSchema &&` guard on the
schema-literal keys (so `responses.default` is read as literal data)
turned 2 of 13 red ("bundles the external file…", "inlines up to
--max-depth…"); `scripts/ablation-replace.mjs` reported anchor 1 → 0,
blob `613cae347a0e` → `0cd0bc79a1fd`, restored to the HEAD blob with
`git diff HEAD` empty.
## Acceptance notes
- **Acceptance 1, real documents, run once locally** (not in CI, no
network in tests). Stripe `stripe/openapi` `openapi/spec3.json`
(8,317,513 bytes, 3.0.0) → 63,317,655 bytes, `refsLeft` 0, cuts 754
recursive / 29,831 max-depth / 8,164 extension; the provider
materializes 612 actions with 0 `$ref` in the definition. The same
document by URL with `--include-pattern '^GET /v1/customers'` gives the
byte-identical output (same sha256) and an `include` list of 22 keys;
the provider with that `include` materializes exactly 22 actions.
DocuSign `docusign/OpenAPI-Specifications`
`esignature.rest.swagger-v2.1.json` (4,846,375 bytes, Swagger 2.0) is
refused without `--empty-consumes` (187 operations named); with
`--empty-consumes application/json` → OpenAPI 3.0.4, 52,884,345 bytes,
`refsLeft` 0, server `https://www.docusign.net/restapi`; the provider
materializes 414 actions, 0 `$ref`, 192 with a body input.
- **Acceptance 2**: the printed sha256 equals `sha256sum` of the written
file for both real outputs (Stripe `0ea3c37a…b2`, DocuSign
`ceee5238…37`), and the CLI test pins it.
- **Acceptance 3**: no changeset. The script does not ship: `files` is
`dist`, `README.md`, `CHANGELOG.md`; README and CHANGELOG are untouched;
after `pnpm --filter @objectstack/connector-openapi build`,
`normalizeOpenApi` / `normalize-openapi` occur 0 times in `dist/*` while
the positive control `createOpenApiProviderFactory` occurs 4 times in
`dist/index.js`. Labelled `skip-changeset`.
- **Size vs `--max-depth`** (whole document, no include): depth 0 →
Stripe 6.1 MB / DocuSign 1.6 MB; 1 → 14.8 / 4.9 MB; 2 (default) → 63.3 /
52.9 MB; 3 → 361 / 160 MB. The default is a trade-off the reviewer may
want to change; it is one constant (`DEFAULT_MAX_DEPTH`).
- **Reachability**: the script is reachable only from a checkout of this
repository (it is not in the npm tarball and has no `bin`). A pipeline
in another repository cannot run it from npm; publishing it (a `bin`,
the two packages moved to `dependencies`) would widen the published
surface and is left to a ruling.
- Not done here, by scope: a 3.x document's path-item-level `parameters`
are not hoisted into operations (the 2.0 upgrader hoists them for 2.0
input). The generator does not read path-item-level parameters at all,
so such a parameter is missing from the action's `inputSchema`; that is
a runtime defect, reported separately rather than worked around here.
- `discriminator.mapping` values are strings, not `$ref` keys; after
`components` is dropped they name schemas no longer in the document. The
generator never reads `discriminator`.
- Upstream: `@scalar/openapi-upgrader` lets the global `produces` win
over an operation-level one (`document.produces ??
operationItem.produces`); harmless for both real documents, noted only.
## Verification
All local readings at head `f9a4f408`.
<details>
<summary>Package test and typecheck (head f9a4f40)</summary>
```text
$ bash scripts/pm/os-verify-lock.sh -c "pnpm --filter @objectstack/connector-openapi test && pnpm --filter @objectstack/connector-openapi typecheck"
Test Files 5 passed (5)
Tests 63 passed (63)
> tsc --noEmit && tsc --noEmit -p tsconfig.scripts.json && pnpm check:test-typecheck
check:test-typecheck: OK — @objectstack/connector-openapi's test layer compiles under packages/connectors/connector-openapi/tsconfig.test.json
os-verify-lock: VERDICT command-exit 0
$ tsc --noEmit -p tsconfig.scripts.json --listFiles | grep connector-openapi/scripts/
.../scripts/normalize-openapi.mts
.../scripts/normalize-openapi.test.ts
```
</details>
<details>
<summary>Derived gate families</summary>
`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` derived 89 commands for this diff; each was
run with its exit code captured before any pipe. 86 exited 0. 3 exited 3
(PREREQUISITE NOT MET, nothing measured, none reads these paths):
`check-plugin-teardown-shape.mjs --self-test` (shallow clone),
`@objectstack/spec check:skill-examples` (client packages unbuilt),
`check:dual-build-cjs-loads` (workspace unbuilt). `--ran`
reconciliation: "89 derived famil(ies) accounted for — 86 run, 3
NOT-MEASURED (3 DERIVED from a recorded exit 3)". The full rerun at the
final head is in the report comment on #27373.
</details>
<details>
<summary>Real-document runs (local, once)</summary>
```text
$ pnpm -s normalize-openapi --in stripe-spec3.json --out stripe.json
{ "bytes": 63317655, "sha256": "0ea3c37a0854ad2646a156404b1ef86cca57403855e3803cd4630601eb6fc7b2",
"sourceVersion": "3.0.0", "openapi": "3.0.0", "operations": 612, "maxDepth": 2,
"cuts": { "recursive": 754, "maxDepth": 29831, "extension": 8164 }, "refsLeft": 0, "filledConsumes": 0 }
provider: actions 612, handlers 612, $ref in def 0
$ pnpm -s normalize-openapi --in esignature.rest.swagger-v2.1.json --out docusign.json
normalize-openapi: 187 Swagger 2.0 operation(s) take a request body but declare an empty 'consumes' list ... (exit 1)
$ pnpm -s normalize-openapi --in esignature.rest.swagger-v2.1.json --out docusign.json --empty-consumes application/json
{ "bytes": 52884345, "sha256": "ceee523822c266fc1de6e5433915f6e4385753d2b239ce8b94f48cafe7d93d37",
"sourceVersion": "2.0", "openapi": "3.0.4", "operations": 414,
"cuts": { "recursive": 30, "maxDepth": 68434, "extension": 0 }, "refsLeft": 0, "filledConsumes": 187 }
provider: actions 414, handlers 414, $ref in def 0, actions with body input 192
$ sha256sum docusign.json stripe.json
ceee523822c266fc1de6e5433915f6e4385753d2b239ce8b94f48cafe7d93d37 docusign.json
0ea3c37a0854ad2646a156404b1ef86cca57403855e3803cd4630601eb6fc7b2 stripe.json
```
</details>
---
_Generated by [Claude
Code](https://claude.ai/code/session_017xRBcVBKAFk1ntXSjvPt7U)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 12a1aba commit 262ab67
9 files changed
Lines changed: 1228 additions & 1 deletion
File tree
- content/docs/automation
- packages/connectors/connector-openapi
- scripts
- fixtures
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
171 | 171 | | |
172 | 172 | | |
173 | 173 | | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
174 | 193 | | |
175 | 194 | | |
176 | 195 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | | - | |
| 28 | + | |
28 | 29 | | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
| 36 | + | |
| 37 | + | |
35 | 38 | | |
36 | 39 | | |
37 | 40 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
0 commit comments