Merge remote-tracking branch 'origin/main' into claude/issue-21044-cube-measure-type-table

Author: claude

.changeset/20821-plan-deferred-ddl-reads.md

@@ -0,0 +1,29 @@
+---
+'@objectstack/driver-sql': patch
+---
+
+fix(driver-sql): `os migrate plan` on a database that does not exist yet no longer prints `DATABASE_ERROR` for the tables whose DDL it deferred (#20821)
+
+`os migrate plan` (and the boot of `os migrate apply`) runs with the SQL driver's DDL deferred: the driver records every table as pending `create_table` and creates none of them. The same boot then reads `sys_metadata`, `sys_metadata_activation` and `sys_migration`. On a new database those tables do not exist yet, so each read was refused, and each refusal printed a line like this on the driver's warn channel (stderr by default):
+
+```text
+[sql-driver] DATABASE_ERROR — the backend refused a read on 'sys_metadata' (SQLITE_ERROR) ... no such table: sys_metadata
+```
+
+Nothing was wrong: every reader already answers from the refusal, and the plan lists the same tables as pending creates. A dry run on a new database printed six of these lines.
+
+The driver now sends such a refusal to the logger's `debug` channel instead of `warn`, when all three hold:
+
+- this driver has DDL deferred;
+- the refused statement targets a table whose DDL this driver deferred;
+- the shared `isMissingTableError` predicate from `@objectstack/types` recognises the refusal as that table being missing.
+
+The default logger has no `debug`, so the line is not printed. The refusal is still thrown to the caller with the same envelope (`DATABASE_ERROR`, status 500), and the plan's output is unchanged.
+
+What still warns:
+
+- every other refusal on a deferred driver, such as a malformed statement on a table that exists;
+- a missing table that the driver did not defer, such as a table nothing in the boot declares;
+- every refusal once the deferred DDL has been applied, or on a driver that never deferred any.
+
+There is nothing to migrate.

.changeset/20890-dataset-dimension-json-stored-refused.md

@@ -0,0 +1,17 @@
+---
+"@objectstack/lint": minor
+---
+
+fix(lint)!: `os validate`, `os build` and `os lint` refuse a dataset dimension over a JSON-stored field, the group key the analytics door already refuses at query time
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) a refusal of a grouping TARGET at authoring time: a dataset dimensions entry whose field resolves to a declared structured-JSON field (json, composite, repeater, record, location, address, vector) or multi-value field (multiselect, checkboxes, tags, or a select, radio, lookup, user, file or image declared multiple: true). It is the authoring leg of the analytics door that already refuses every query grouping by such a column with 400 INVALID_FIELD, and that door's own changeset declared this category for this surface. No authorable key, spelling, export or stored shape moves: DatasetSchema keeps parsing every dimension, no stored row is read or rewritten, and which scalar part of a document, or which member of a list, an author meant to group on is not something a ledger entry can rewrite. The other categories are closed on facts: the package publishes (not unpublished); no ADR-0087 id covers a grouping target, and dataset-measure-aggregate-field-type-refused says in its own reason that a field used as a DIMENSION is untouched (not already-registered); and the change is a rule verdict, not a declaration (not runtime-interface-only or type-surface-only). -->
+
+**BREAKING**: metadata that passed `os validate`, `os build` and `os lint` can now fail, and so can a runtime dataset save (Studio, REST `/meta`, MCP), which runs the same rule. A dataset dimension is a group key, and the analytics door refuses a query that groups by a JSON-stored column with `400 INVALID_FIELD` before any SQL is built, so such a dimension could be declared but never served. The new rule `dimension-json-stored-field-refused` (gating, `error`) refuses it where the author writes it. It ships as `minor` under the launch-window convention for accept-set narrowings. No export is removed. The package entry exports the new id as `DIMENSION_JSON_STORED_FIELD_REFUSED`, beside `MEASURE_AGGREGATE_FIELD_TYPE_REFUSED`, and the `rule` member of `DatasetMeasureAggregateFinding` gains it.
+
+**What is refused.** A dimension whose `field` resolves, on the dataset's object or across its join chain, to a field declared with a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`) or a multi-value declaration (`multiselect`, `checkboxes`, `tags`, or a `select`, `radio`, `lookup`, `user`, `file` or `image` declared `multiple: true`). The two classes are `@objectstack/spec/data`'s `STRUCTURED_JSON_TYPES` and `isMultiValueField`, the predicates the analytics door reads.
+
+**What an author sees now.** The finding names the dataset, the dimension, the field, the object that declares it and its declaration, and says the analytics door refuses every query that groups by it. It names the route: group by a field that stores one scalar value, storing the part of the document you group on in a field of its own; for a multi-value field, filter by one member with `$contains` in a record query, one query per member. It is located at `datasets[N].dimensions[M].field`, name-keyed on the runtime wire.
+
+**Unchanged.** A dimension over any other field, a single-value `select` or `lookup` included; a dimension whose field does not resolve (`dataset-field-unknown` reports that) or declares no type; a dataset over an object this stack does not define; measures, filters and every other position. A cube dimension (`analyticsCubes`) is not judged: no authoring rule reads cubes.

.changeset/20890-dataset-distinct-multiple-refused.md

@@ -0,0 +1,17 @@
+---
+"@objectstack/lint": minor
+---
+
+fix(lint)!: a dataset `count_distinct` measure over a field declared `multiple: true` is refused by `measure-aggregate-field-type-refused`, as the compile leg and the engine already refuse it
+
+Clause-②: yes (narrowing)
+
+<!-- adr-0087: not-required (already-registered dataset-measure-aggregate-field-type-refused) the registered entry carries this family's hand-migration, an aggregate the field accepts with count as the one that stays for a JSON-stored field, and the count_distinct narrowing over the JSON-stored types already rides it. A select, radio, lookup, user, file or image field declared multiple: true is the one JSON-stored shape the per-type table cannot see; the dataset compile leg and the engine's count_distinct door refuse it beside the table's row, and this is the authoring leg of that same pair, so it adds no surface of its own. -->
+
+**BREAKING**: metadata that passed `os validate`, `os build` and `os lint` can now fail, and so can a runtime dataset save, which runs the same rule. `measure-aggregate-field-type-refused` reads the field's declaration, not its type alone: `count_distinct` over a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true` is refused, because that field is a list stored as JSON and no two backends compare such values for equality alike. The dataset compile leg already answers the pair `400 DATASET_INVALID`, and the engine's `count_distinct` door answers it `400 INVALID_FIELD`. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+**What an author sees now.** The finding names the measure, the field, the object and the declaration with its flag (`select` with `multiple: true`), and says the aggregate accepts its row's types, none of them with `multiple: true`. The hint names the aggregates the declaration does accept, read from the same predicate: `count`.
+
+**What to write instead.** `count` over the field, or `count_distinct` over a field that stores one scalar value. To count the records holding one member, filter by it with `$contains` in a record query.
+
+**Unchanged.** `count_distinct` over the same types without the flag; `count` over any field; every other aggregate, whose rows accept no multi-capable type and whose verdicts therefore do not move; and the skips the rule already had.

.changeset/20892-package-visibility-default-org.md

@@ -0,0 +1,22 @@
+---
+'@objectstack/spec': minor
+---
+
+feat(spec): `PackageSchema.visibility` defaults to `org` (was `private`), the create-time default every publish path already produced
+
+Clause-②: yes
+
+`PackageSchema` (`@objectstack/spec/marketplace`) filled an omitted `visibility` with
+`private`, but no path that creates a package ever reached that value: the cloud control
+plane gives a new package `org` when the create request omits the key, and
+`os package publish` used to send `org` itself. The declared default now matches what the
+runtime does: `org`, which makes a package published from one environment installable in
+the owner organization's other environments.
+
+- **What changes:** `PackageSchema.parse(row)` on a row with no `visibility` now returns
+  `visibility: 'org'`. A row that names `private`, `org` or `marketplace` is read exactly
+  as before, and any other value is still refused.
+- **What does not change:** the accepted values, and `CreatePackageRequestSchema.visibility`,
+  which stays optional with no default. A create request that omits the key reaches the
+  control plane without it, and the control plane's default applies.
+- **If you relied on the old default:** pass `visibility: 'private'` explicitly.

.changeset/20892-plugin-publish-visibility-when-asked.md

@@ -0,0 +1,25 @@
+---
+'@objectstack/cli': patch
+---
+
+fix(cli): `os plugin publish` sends `visibility` only when `--visibility` is passed, so re-publishing no longer moves a `marketplace` plugin to `private`
+
+The `--visibility` flag of `os plugin publish` defaulted to `private`, and the CLI always
+sent it in the package upsert (`POST /api/v1/cloud/packages`). That upsert is also the
+re-publish path, and the control plane updates an existing package's visibility whenever
+the request carries one. So re-publishing a new version of a `marketplace` plugin without
+repeating `--visibility marketplace` silently set it to `private`. `os package publish`
+already works this way; both commands now behave the same.
+
+The flag no longer has a default, and an omitted flag is an omitted key:
+
+- **Re-publish without the flag:** the package keeps its current visibility.
+- **First publish without the flag:** the control plane applies its own default (`org` on
+  ObjectStack Cloud), the default `PackageSchema.visibility` declares. Before this change
+  `os plugin publish` sent `private` here; pass `--visibility private` to keep that.
+- **With the flag:** unchanged. `--visibility private|org|marketplace` is sent and applied.
+
+The publish summary gains a `Visibility` line showing the control plane's answer. When the
+control plane does not report it and no flag was given, the line says
+`not reported by the control plane`. The "Re-run with --submit" hint now follows that
+answer too, so it also fires when a `marketplace` plugin is re-published without the flag.

.changeset/20901-step18-rationale-form-view-subform-columns.md

@@ -0,0 +1,10 @@
+---
+'@objectstack/spec': patch
+---
+
+fix(spec): protocol 18's migration rationale now covers the form view's inline grid columns and the identity-only currency `scale` refusal (#20901)
+
+**`@objectstack/spec`**
+
+- **`MIGRATIONS_BY_MAJOR[18].rationale` gains one fragment, `form-view-subform-columns-closed`.** It is the paragraph `os migrate meta --step` shows for the protocol 17 → 18 hop. The new sentences say that a form view's `subforms[].columns` now takes the strict `InlineGridColumnSchema` a relationship field's `inlineColumns` takes, that the conversion `form-view-subform-columns-canonicalized` respells a `{ field }` column as `{ name }` in stored rows and assembled artifacts while an author writing `field` is refused, and that `defineStack` refuses `scale` on a column that declares no `type` when its `name` is a `currency` field of a child object declared in the same stack (`inline-grid-column-identity-only-currency-scale-refused`).
+- Text only: no schema, conversion or migration entry changes, and `conversionIds` and `semantic` for step 18 are unchanged.

.changeset/20914-release-sum-row.md

@@ -0,0 +1,30 @@
+---
+"@objectstack/objectql": minor
+"@objectstack/spec": patch
+---
+
+fix(objectql)!: the engine's `aggregate` judges the `sum` row of the aggregate × field-type table too, so `sum` over a type the table refuses answers `INVALID_FIELD` / 400 on every driver instead of `0` in memory and on SQLite and a 500 on PostgreSQL
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (already-registered dataset-measure-aggregate-field-type-refused) the pairs this change refuses are exactly the pairs the sum row of AGGREGATE_FIELD_TYPE_COMPATIBILITY already refuses, and the table is not edited: that id's prescription covers sum / avg over every field class the table refuses (widened to them by a later change that registered against it), with its routes (count, an aggregate the type accepts, or a numeric field for a quantity stored otherwise). This change lets the engine door ask the one row it had skipped; it refuses a query shape, not a stored one, and no authorable key, export or stored row moves. -->
+
+**BREAKING** (`@objectstack/objectql`): this narrows what `aggregate` accepts, on every driver and for every caller that reaches the engine — the REST query door, a flow or hook, a roll-up summary's recompute, and the analytics strategy that lowers a cube query onto `engine.aggregate`. Shipped as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
+
+This completes the change of this same release that made the engine's `aggregate` ask the table for `min`, `max` and `avg`, and that held the `sum` row back. Its paragraph "Not judged yet: `sum`" is superseded: this entry is the later word, and the door now asks every row of the table.
+
+FROM → TO, per aggregation `{ function: 'sum', field }` naming a declared field:
+
+- `sum` over a type outside `number`, `currency`, `rating`, `slider`, `progress`, `summary`, `boolean` and `toggle` — a `percent` (a rate does not add), the temporal types (`date`, `datetime`, `time`), the string family (`text`, `email`, `url`, `phone`, …), the option and reference types (`select`, `radio`, `lookup`, `master_detail`, `tree`, `user`), `autonumber`, the file family, the structured-JSON types (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), the multi-option types (`multiselect`, `checkboxes`, `tags`) and `formula` — and over any `select`, `radio`, `lookup`, `user`, `file` or `image` declared `multiple: true`: FROM whatever the driver answered TO `400 INVALID_FIELD`. Measured through `engine.aggregate` over two rows: a `json`, `text`, `select` or `tags` field summed to `0` in memory and on SQLite and answered 500 `DATABASE_ERROR` on PostgreSQL 16 (`function sum(json) does not exist`); a `datetime` field summed to `0` in memory, to the years added on SQLite and a 500 on PostgreSQL; a `formula` field summed to `0` in memory and was already refused `400 INVALID_FIELD` by both SQL drivers, which have no column for it; a `percent` field added the rates on all three.
+
+**What an author sees now.** `400 INVALID_FIELD`, naming the position (`aggregations[0].field`), what the function does and the field with its declaration (`sums 'meta', a declared json field — a structured-JSON value`), saying the query was not run, and naming the types `sum` accepts, read off the table, inside the first 500 characters the REST door keeps. The thrown error carries `field`, `fields` (every offending aggregation), `object` and `param` (`aggregations`).
+
+**What to write instead.** Sum a field of a type `sum` accepts: `number`, `currency`, `rating`, `slider`, `progress`, `summary`, `boolean` or `toggle`. A rate stored as a `percent` is averaged (`avg` accepts it), or the quantity it is a rate of is summed. A value computed by a `formula` is stored in a numeric field of its own when it must be summed on the server. A question that was counting in disguise is `count`.
+
+**Who is affected.** A caller that asked `sum` of such a field on the in-memory driver or SQLite and read the `0` as a real total, and a caller that summed a `percent` field on any driver. On PostgreSQL the other measured pairs were already refused (a 500, or a 400 for a `formula`), and on SQLite so was a `formula`. Metadata that lowers onto `engine.aggregate` takes the same verdict at run time: a roll-up summary (`summaryOperations`) whose `sum` names such a child field records a failed recompute, a grouped list view's server-side header summary is refused, and a chart or metric component's `sum` over such a field is refused. No example app authors such a pair. One published stack authors a `sum` list-column summary over a `formula` field; that summary is computed client-side and does not reach `engine.aggregate`.
+
+**Unchanged.** Every pair the table accepts, `sum` over the eight types above included; `count` over any field; an aggregation that names no field; an undeclared name or a relationship path, which this door does not judge (the REST door answers an unknown name `INVALID_FIELD` before the engine is reached); a field whose declared type is outside `FieldType`.
+
+**A correction to the earlier entry of this release.** It listed the multi-capable types declared `multiple: true` as `select`, `lookup`, `user`, `file` or `image`; the list is `select`, `radio`, `lookup`, `user`, `file` and `image` (`MULTI_CAPABLE_TYPES`). A `radio` declared `multiple: true` was refused by `min` / `max` / `avg` there all the same, by its type's own row, and it is refused by `sum` here.
+
+`@objectstack/spec`: the TSDoc of `AGGREGATE_FIELD_TYPE_COMPATIBILITY` and `isAggregateCompatibleWithFieldType` states that the engine's `aggregate` door asks every row of the table, where it said "the other rows" while one was held, and names `radio` among the multi-capable types. The table and the predicate are unchanged.

.changeset/20919-core-import-runner.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/core': minor
+---
+
+feat(core): the bulk-import runner, its row coercion, the mapping apply and the field-meta map now live in `@objectstack/core`, beside `bulkWrite` (#20919)
+
+`runImport` (with `sanitizeRowError` and its option/result types), the cell
+coercion (`coerceRow`, `coerceFieldValue`, `parseDateCell`, `parseNumberCell`,
+`parseBooleanCell`, `matchOption`, `splitMulti`, `isBlank`), the `mapping`
+artifact pipeline (`applyMappingToRows`, `refuseUnknownMappingTargets`,
+`MappingArtifactLike`, `MappingFailure`, `ApplyMappingOptions`) and the field
+metadata map (`buildFieldMetaMap`, `ExportFieldMeta`) are exported from
+`@objectstack/core`. They moved here unchanged from `@objectstack/rest` so the
+connector sync executor in `@objectstack/service-automation` writes through the
+same runner as the HTTP import door without depending on the HTTP layer. Nothing
+to change for consumers: `@objectstack/rest` re-exports every name it exported
+before.

.changeset/20919-rest-re-exports.md

@@ -0,0 +1,11 @@
+---
+'@objectstack/rest': patch
+---
+
+refactor(rest): the import runner, coercion, mapping apply, field-meta map and error classification moved to `@objectstack/core` / `@objectstack/types`; `rest` re-exports them (#20919)
+
+`runImport`, `coerceRow`, `buildFieldMetaMap` and their types (from the package
+index), and `mapDataError` with its sibling classification exports, are now
+re-exported from their new homes — byte-identical code, the same names, the same
+behaviour at both import routes and at `plugin-auth`'s identity import. Nothing to
+change for consumers.