Skip to content

Commit b77bb37

Browse files
feat(spec,core,rest): import column applyOn create-only, and matchFields ordered key sets (#25664)
Fixes #25577 Fixes #25578 Clause-②: yes (widening) Metadata-protocol: added Ruling-ref: 6377846440 Ruling-ref: 6377845536 One import-contract PR for two ruled cards, contract and runner together (spec first in each commit), so no release carries either key without the runner that honours it. One commit per card: `f21019ef` (key sets, the second card) lands first because `applyOn`'s update patch keeps the key set the row was matched by; `fb327b11` (`applyOn`, the first card) builds on it; `b89b3b24` merges `origin/main` (no regeneration was owed after it: `check:generated` 14/14 up to date). ## Rulings implemented - First card, ruling 6377846440, letter B, verbatim: 「**B** = `FieldMappingEntry.applyOn: 'create' | 'always'`(默认 `always`):标 `create` 的列只在新建行时写,已有行不动;一次 `upsert` 即幂等(主数据列刷新,状态列不冲回)。」 No new `writeMode` value. - Second card, ruling 6377845536, letter A, verbatim: 「**A** = 匹配键接受有序多组(`matchFields` 接受 `string[][]`,单组写法照旧有效),每行用第一组非空的键;哪组都空就报行错误,不再静默跳过;`AMBIGUOUS_MATCH` 规则不变。」 ## What changed **`packages/spec` (`src/api/export.zod.ts`)** - `FieldMappingEntry.applyOn: z.enum(['create', 'always']).default('always')`. - `ImportRequest.matchFields` is `string[]` (unchanged) or `string[][]` (non-empty list of non-empty sets). - A refinement refuses `skipBlankMatchKey: true` together with key sets (path `skipBlankMatchKey`), mirroring the door. - New export `isMatchKeySets(matchFields)`, the one predicate the spec refinement, the REST door and the runner share. - Regenerated: `api-surface/`, `export-origins/`, `authorable-surface/`, `authorable-defaults/`, `content/docs/references/api/export.mdx`. **`packages/core` (`src/utils/import-runner.ts`, the runner both import doors call)** - `RunImportOptions.matchFields: string[] | string[][]`, plus `createOnlyFields?: string[]`. - Each row uses the first key set whose fields are all non-blank on it, and only that set: a miss on it is `none` (upsert creates), never a fall-through to the next set. - Key sets: a row blank on every set is `{ ok: false, action: 'failed', code: 'BLANK_MATCH_KEY' }`, on the dry run and the commit alike. One key set keeps its own blank handling exactly (upsert creates, update and `skipBlankMatchKey` skip with `BLANK_MATCH_KEY`). - An update (and its dry-run preview, and its undo snapshot) writes the row minus its create-only columns, keeping the fields of the key set the row was matched by. **`packages/rest` (`src/import-prepare.ts`)** - Reads `matchFields` in both spellings. One key set is read exactly as before (non-string names dropped). Key sets are read strictly: an empty set or name, a non-string name, or names mixed with sets refuse with `400 INVALID_REQUEST`. - `skipBlankMatchKey: true` with key sets refuses with `400 INVALID_REQUEST`. - Collects the targets of inline entries with `applyOn: 'create'` into `createOnlyFields`; an `applyOn` outside `create` / `always` refuses with `400 INVALID_REQUEST`. **Docs:** `content/docs/data-modeling/import-mappings.mdx` gains the two sections, the new refusals and the failed `BLANK_MATCH_KEY` row. ## Decisions taken within the rulings 1. **`skipBlankMatchKey` with key sets is refused (400), not honoured.** Honouring it would skip a row blank on every set, which is the 「静默跳过」 the ruling retires; ignoring it would accept a flag the runtime does not keep. One key set keeps `skipBlankMatchKey` as today. 2. **The blank-row rule follows the spelling.** `[["credit_code"]]` (key-set form, one set) reports a blank row as a failed `BLANK_MATCH_KEY`; `["credit_code"]` keeps today's create/skip. Applying the row error to the single-set spelling would change its meaning, which 「单组写法照旧有效」 rules out. 3. **The row error reuses `BLANK_MATCH_KEY`** with `ok: false` / `action: 'failed'`, so no new error code enters the ledger; the skipped and the failed rows are told apart by `ok` / `action`. 4. **`applyOn: 'create'` on a match-key column:** the column is read for matching on every row; on update, the fields of the key set the row was matched by stay in the patch (the value equals the one the row was found by), so the update patch is never empty because of `applyOn`. With key sets, a create-only column that is not in the row's matched set is left out of that row's update. ## Tests All on `b89b3b24` (the final head) unless noted. - `packages/rest/src/import-apply-on-match-key-sets.test.ts` (new): 16/16, through the real `POST /data/:object/import` route on ObjectQL + sqlite. It covers: an upsert run twice keeps a create-only column and refreshes an `always` column (and creates a new row with it); update mode never writes it, and the dry run agrees; a create-only match-key column still matches; the `always` / absent control; `applyOn: 'insert'` gives 400 `INVALID_REQUEST`. For key sets: a mixed sheet matches set 1 and set 2 in one import and is idempotent on re-run; no fall-through from a non-blank first set; a row blank on every set is a failed `BLANK_MATCH_KEY` on the dry run and the commit; `AMBIGUOUS_MATCH` per set; `skipBlankMatchKey` with key sets gives 400 `INVALID_REQUEST`; four malformed key-set shapes give 400 `INVALID_REQUEST`; and a control showing one key set's blank handling unchanged. A last test exercises both keys in one sheet. - `packages/spec/src/api/export.test.ts`: 50/50 (applyOn default, accept and refuse; key sets accept and refuse; `skipBlankMatchKey` refinement; typed SDK request). - Package suites: `@objectstack/spec` 715 files / 21721 tests passed; `@objectstack/core` 118 / 2757 passed; `@objectstack/rest` 321 files / 6416 passed, 326 skipped (taken on `fb327b11`; the merge brought no change to these three packages' sources except the spec approval schema, rebuilt and regenerated). - Typecheck: `@objectstack/spec`, `@objectstack/core`, `@objectstack/rest`, and the other `matchFields` / `runImport` consumers `@objectstack/client`, `@objectstack/service-automation`, `@objectstack/plugin-auth` (downstream of the changed contract): all exit 0. - Dogfood import tests (`import-reference-exposure`, `import-user-field-target`): 2 files / 16 tests passed. - `packages/lint`: has no import-mapping rule (no file judges `matchFields` or mapping entries), so it is not touched. **Ablation** (the doors read the runner from `@objectstack/core`'s `dist/`; each leg: `scripts/ablation-replace.mjs` mutation, core rebuild, `ablation-dist-preflight.mjs` marker present in 2 built files, run, restore with blob equal to HEAD, rebuild, marker absent): - `applyOn`: `updatePatch` keeps every column. 3 tests red (upsert run twice; update mode; both keys in one sheet), 13 green. The first attempt at this leg did not run: the mutation left `keySet` unused, so the core build failed (exit 90) before any test; it was rerun with a mutation that still reads `keySet`. - Key sets: the blank-on-every-set branch disabled. 1 test red (row error on dry run and commit), 15 green. - After restore: 16/16 green. ## Gates `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` on `b89b3b24` derived 116 families, all run locally, each exit captured before any pipe. Reconciliation: `--ran` reports 116 derived, 116 run, 0 NOT-MEASURED, 0 UNRUN. Three families first refused a prerequisite (exit 3) and were rerun green once it was met: `check-plugin-teardown-shape --self-test` (shallow clone; the pinned fixture commit was fetched), `spec check:skill-examples` (built `client-react`), `check:dual-build-cjs-loads` (built the dogfood closure). `check:dev-prereqs` (local-only) exit 0. `test:repo` was not run locally; it belongs to CI. ## Acceptance notes - `ExportImportTemplateSchema.mappings` also uses `FieldMappingEntry`, so a template entry can now carry `applyOn`. No runtime code reads export/import templates (`git grep ExportImportTemplate` outside `packages/spec` finds nothing), so this is the same as its existing `transform` / `required` keys. Noted, not filed (no consumer). - The named `mapping` artifact (`mappingName`) has its own `fieldMapping` pipeline and an `upsertKey: string[]`. Neither ruling names it, so it gets neither key: `applyOn` is an inline-entry key, and `upsertKey` stays one key set. A request's `matchFields` key sets still override an artifact's `upsertKey` as before. - 「hotsrm 发版后拆掉三处字段声明绕行」 / 「拆掉两遍导入的绕行」 is hotsrm-side work, not part of this PR. --- _Generated by [Claude Code](https://claude.ai/code/session_013Zj6EHJ5eped9wNcSM4jvh)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 1fbd9de commit b77bb37

12 files changed

Lines changed: 627 additions & 33 deletions

File tree

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
'@objectstack/spec': minor
3+
'@objectstack/core': minor
4+
'@objectstack/rest': minor
5+
---
6+
7+
feat(spec,core,rest): an import column can be written only when its row is created (`applyOn`), and `matchFields` takes ordered key sets
8+
9+
Clause-②: yes (widening)
10+
Metadata-protocol: added
11+
12+
Two additions to the import contract (`POST /data/:object/import` and the async import job), both honoured by the import runner:
13+
14+
- **`FieldMappingEntry.applyOn: 'create' | 'always'`** (default `always`, today's behaviour). An inline mapping entry marked `create` is written when the import creates the row and left out when an `upsert` updates an existing one; an `update` import never writes it. One upsert can then be re-run safely: master-data columns refresh, while a column the app has changed since the first run (a status, a flag) keeps its current value. A create-only column is still read for matching, and the key a row was found by stays in that row's update. An `applyOn` value outside `create` / `always` refuses the request with `400 INVALID_REQUEST`.
15+
- **`matchFields` accepts ordered key sets** (`string[][]`) beside one key set (`string[]`, unchanged). Each row is matched by the first set whose fields are all non-blank on it, and only by that set, so one sheet can identify domestic suppliers by `credit_code` and overseas ones by `country` + `registration_no`. A row blank on every set fails with `BLANK_MATCH_KEY` (`ok: false`, `action: 'failed'`) instead of being created or skipped; `skipBlankMatchKey: true` with key sets is refused with `400 INVALID_REQUEST`, as are malformed key sets (an empty set or name, or names mixed with sets). A set that matches more than one record fails the row with `AMBIGUOUS_MATCH`, as before.
16+
17+
```json
18+
{
19+
"writeMode": "upsert",
20+
"matchFields": [["credit_code"], ["country", "registration_no"]],
21+
"mapping": [
22+
{ "sourceField": "Code", "targetField": "credit_code" },
23+
{ "sourceField": "Country", "targetField": "country" },
24+
{ "sourceField": "Reg. no.", "targetField": "registration_no" },
25+
{ "sourceField": "Name", "targetField": "name" },
26+
{ "sourceField": "Status", "targetField": "status", "applyOn": "create" }
27+
]
28+
}
29+
```
30+
31+
`@objectstack/spec` also exports `isMatchKeySets(matchFields)`, which tells the two `matchFields` spellings apart. `RunImportOptions` in `@objectstack/core` gains `createOnlyFields` and accepts both `matchFields` spellings.

‎content/docs/data-modeling/import-mappings.mdx‎

Lines changed: 43 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -198,14 +198,53 @@ sending both is `400 CONFLICTING_MAPPING`.
198198
| | inline `mapping` | `mappingName` |
199199
|:---|:---|:---|
200200
| Shape | A flat `{ "<source column>": "<field>" }` rename (or a `sourceField`/`targetField` array) | A registered `mapping` artifact |
201-
| Transforms | Per entry of the array form: `transform` (`trim`, `uppercase`, `lowercase`; `none` and `lookup` copy the cell), `defaultValue` for a blank cell, `required`. The flat record is a rename only | The `fieldMapping` pipeline |
201+
| Transforms | Per entry of the array form: `transform` (`trim`, `uppercase`, `lowercase`; `none` and `lookup` copy the cell), `defaultValue` for a blank cell, `required`, `applyOn`. The flat record is a rename only | The `fieldMapping` pipeline |
202202
| Unmapped columns | **Pass through** to the write path under their own header | **Dropped.** The artifact is a strict projection: only mapped targets survive |
203203
| Lives | In the one request | In a package or in metadata, reusable |
204204

205205
That projection difference is the one to keep in mind: a file from an external system
206206
routinely carries columns that must not reach the write path, and the artifact path
207207
guarantees they do not.
208208

209+
### Columns written only when a row is created: `applyOn`
210+
211+
An entry of the array form may declare `applyOn: 'create'` (the default is `'always'`).
212+
Its column is written when the import creates the row and left out when an `upsert`
213+
updates an existing one; an `update` import never writes it. That makes one upsert
214+
safe to re-run: master-data columns refresh, while a column the app has moved on since
215+
the first run (a status, a flag) keeps its current value.
216+
217+
```json
218+
{
219+
"writeMode": "upsert", "matchFields": ["credit_code"],
220+
"mapping": [
221+
{ "sourceField": "Code", "targetField": "credit_code" },
222+
{ "sourceField": "Name", "targetField": "name" },
223+
{ "sourceField": "Status", "targetField": "status", "applyOn": "create" }
224+
]
225+
}
226+
```
227+
228+
A create-only column is still read for matching: the match key a row was found by stays
229+
in that row's update, with the value it was found by.
230+
231+
### Rows identified in different ways: ordered key sets
232+
233+
`matchFields` takes one key set (`["email"]`) or an ordered list of key sets, for a sheet
234+
whose rows are identified in different ways:
235+
236+
```json
237+
{ "writeMode": "upsert", "matchFields": [["credit_code"], ["country", "registration_no"]] }
238+
```
239+
240+
Each row is matched by the **first** set whose fields are all non-blank on that row, and
241+
only by that set: a domestic row with a credit code is matched on it (and created when
242+
nothing matches), an overseas row with a blank credit code is matched on country and
243+
registration number. A row blank on every set fails with `BLANK_MATCH_KEY` instead of
244+
being created or skipped, so `skipBlankMatchKey` cannot be combined with key sets. A set
245+
that matches more than one record fails the row with `AMBIGUOUS_MATCH`, as one key set
246+
does.
247+
209248
---
210249

211250
## Rejections before any row is read
@@ -221,7 +260,7 @@ These are whole-request failures — nothing is written, and there is no per-row
221260
| 400 | `UNSUPPORTED_TRANSFORM` | Some artifact entry declares `javascript`, or a transform name the pipeline does not implement; or an inline entry declares `date_format` (it carries no format to apply) or a name outside `none`, `lookup`, `trim`, `uppercase`, `lowercase`. |
222261
| 400 | `MISSING_REQUIRED_FIELD` | An inline entry is marked `required: true` and some row leaves its column blank after its `defaultValue`. The error names the column and the rows; it applies to the dry run and the commit alike. |
223262
| 400 | `CONFLICTING_MAPPING` | Both `mappingName` and an inline `mapping` were supplied. |
224-
| 400 | `INVALID_REQUEST` | No recognizable payload, or `writeMode` resolved to update/upsert with no `matchFields` from either the request or the artifact. |
263+
| 400 | `INVALID_REQUEST` | No recognizable payload; `writeMode` resolved to update/upsert with no `matchFields` from either the request or the artifact; malformed key sets in `matchFields` (an empty set or name, or names mixed with sets); `skipBlankMatchKey: true` with key sets; or an inline entry's `applyOn` outside `create` / `always`. |
225264
| 413 | `PAYLOAD_TOO_LARGE` | More than 5,000 rows on the synchronous route. |
226265

227266
---
@@ -236,7 +275,8 @@ fails does **not** stop the import.
236275
| `created` | `true` | — | Inserted. |
237276
| `updated` | `true` | — | Matched an existing record and updated it. |
238277
| `skipped` | `true` | `NO_MATCH` | `writeMode: 'update'` and the match fields matched nothing. Nothing is created. |
239-
| `skipped` | `true` | `BLANK_MATCH_KEY` | The row's match fields are blank. Upsert creates such a row by default; `update` skips it, and `skipBlankMatchKey: true` skips it in either mode. |
278+
| `skipped` | `true` | `BLANK_MATCH_KEY` | One key set in `matchFields`, and a field of it is blank on the row. Upsert creates such a row by default; `update` skips it, and `skipBlankMatchKey: true` skips it in either mode. |
279+
| `failed` | `false` | `BLANK_MATCH_KEY` | Ordered key sets in `matchFields`, and every set has a blank field on the row. Nothing is written for that row. |
240280
| `failed` | `false` | `AMBIGUOUS_MATCH` | The match fields matched **more than one** record. Nothing is written for that row. |
241281
| `failed` | `false` | a field error code | A cell could not be coerced, or the engine rejected the record. `field` names the column and `error` carries the message. |
242282

0 commit comments

Comments
 (0)