Merge remote-tracking branch 'origin/main' into claude/issue-19519-picklist-runtime

Author: claude

.changeset/20873-aggregation-filter-array-membership.md

@@ -0,0 +1,24 @@
+---
+'@objectstack/objectql': patch
+---
+
+fix(objectql): a per-aggregation `filter` with `$contains` / `$notContains` on a multi-valued field counts the rows the same `where` finds (#20873)
+
+Clause-②: no
+
+`engine.aggregate({ aggregations: [{ …, filter }] })` — and so `POST /api/v1/data/:object/query`
+with a per-aggregation `filter` — evaluates that filter in the engine, not in the driver. Its
+`$contains` arm failed every value that was not a string, so on a `multiple: true` lookup,
+`multiselect`, `checkboxes` or `tags` field a stored array never matched:
+`{ owners: { $contains: 'u1' } }` counted 0 on every driver where the same condition as a `where`
+found 2 rows, and `$notContains` counted every row, the rows holding the member included.
+
+On a declared JSON-stored field (a multi-valued field, or a structured-JSON type) both operators
+now ask MEMBERSHIP, the reading `FILTER_OPERATORS`' `$contains` docblock declares and `where` gives
+on every SQL dialect: `'u1'` is a member of `['u1', 'u2']` and not of `['u10']`, and a member stored
+as a number or boolean is named by its text (`'1'` finds `[1, 2]`). `$notContains` is the exact
+complement, and a row with no value still satisfies it. A scalar text field keeps the substring
+test, unchanged, and so does `having`.
+
+No query that was refused now answers, and none that answered is refused: only the count of a
+per-aggregation `filter` on a multi-valued field moves, to the `where` count.

.changeset/20887-analytics-nested-relation-engine-answer.md

@@ -17,7 +17,7 @@ Clause-②: yes (narrowing)
 - At a measure's own `filter` the form is refused with `400 INVALID_FILTER`, as the engine refuses it at an aggregation's own `filter`: put the condition in the query's `where`.
 - `POST /api/v1/analytics/sql` refuses a `where` carrying the form with `400 INVALID_FILTER`: no statement it could print reproduces a read of the related object as the caller. The query itself is answered by `POST /api/v1/analytics/query`.
 
-**Why.** Measured on the base over one fixture with the real security layer (a related field the caller may not read, a related row scope, 1,001 matching related records). The native-SQL strategy flattened the form to a dotted member and joined the related table: through a dataset that `include`d the relationship it answered rows for a condition on a field the caller cannot read, answered a match past the engine's cap, and counted a measure filter carrying the form; without the declared join it named a table that does not exist (500), and a multi-valued relation was refused. The engine-aggregate strategy refused the form as a cross-object filter (400). The engine serves the form since the nested-relation filter landed in `where`.
+**Why.** Measured on the base before the field-level gate (#20917) and the relationship-path admission (#20933) landed, over one fixture with the real security layer (a related field the caller may not read, a related row scope, 1,001 matching related records). The native-SQL strategy flattened the form to a dotted member and joined the related table: through a dataset that `include`d the relationship it answered rows for a condition on a field the caller cannot read, answered a match past the engine's cap, and counted a measure filter carrying the form; without the declared join it named a table that does not exist (500), and a multi-valued relation was refused. The engine-aggregate strategy refused the form as a cross-object filter (400). The engine serves the form since the nested-relation filter landed in `where`.
 
 **How.** The native-SQL strategy declines a query in which the form appears in the `where`, the dataset's own `filter` or a requested measure's `filter`, so the query runs on the engine-aggregate path, which hands the form to the engine as written.
 

.changeset/20896-template-import-door.md

@@ -0,0 +1,23 @@
+---
+'@objectstack/rest': patch
+---
+
+fix(rest): the import template (`GET /api/v1/data/:object/export?template=true`) is gated by the import door's permissions, not the export's
+
+Clause-②: no
+
+The template carries no records — only the columns the caller may write, one
+example row and an instructions sheet — so it answers to whoever may import, and
+the export permission (`allowExport`) neither admits nor refuses it:
+
+- A caller with the create permission on the object gets the template, with or
+  without `allowExport`.
+- A caller without the create permission gets `403 PERMISSION_DENIED`, with or
+  without `allowExport`.
+- An object whose `enable.apiMethods` exposes neither `create` nor `update`
+  answers `405 OBJECT_API_METHOD_NOT_ALLOWED`, as `POST /api/v1/data/:object/import`
+  does. An object that exposes `create` without `list` serves the template.
+- Without `template=true` the export is unchanged: the same two export checks
+  and the same bytes.
+
+To let a role download the template, grant it create on the object.

.changeset/20897-exists-non-boolean-refused.md

@@ -0,0 +1,22 @@
+---
+'@objectstack/driver-memory': minor
+'@objectstack/driver-mongodb': minor
+---
+
+fix(driver-memory, driver-mongodb)!: a non-boolean `$exists` comparand is refused with `INVALID_FILTER` / 400, as `$null`'s is, instead of selecting the rows with no value (#20897)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (already-registered filter-query-face-comparands-refused-at-save) this narrows the query faces to the rule that registered entry already records: its reason states that every query face refuses a non-boolean $null / $exists flag, and its replacement is this change's whole migration (a flag is the boolean itself; $exists true is "has a value", false "has no value"). This change makes that statement true on the two drivers that did not yet refuse. No authorable key, spelling, export or published type moves, and no stored row is read or rewritten; a stored filter carrying such a flag is already refused when it is saved, by that entry. -->
+
+**BREAKING**: this narrows what the in-memory driver and the MongoDB driver accept in a filter. A `$exists` comparand that is not a boolean (a string, a number, `null`, `undefined`, an object) is now refused with `INVALID_FILTER` / 400, where these two drivers used to answer it. It ships as `minor` under the launch-window convention for accept-set narrowings. No export or published type changes.
+
+`FieldOperatorsSchema` declares `$exists` as a boolean, and `driver-sql`, `driver-sqlite-wasm` and both Turso transports already refused any other comparand. The in-memory driver and the MongoDB driver did not: they read `$exists` as `value === true`, so every other value asked for the rows with NO value. `{ stage: { $exists: "yes" } }` and `{ stage: { $exists: 1 } }` returned the rows without a stage, the opposite of what was written. `0`, `null` and the string `"false"` landed on that same side by the same default, not because anything read them. The in-memory driver's analytics face read the same flag by truthiness and answered the valued rows for the same filter, so that driver gave two different answers.
+
+**What an author sees now.** `400 INVALID_FILTER` with `driver-sql`'s message, beginning `Operator "$exists" on field "FIELD" requires a boolean comparand (true or false).` and naming the position (`filter.stage.$exists`). On the in-memory driver the refusal covers `find`, `findOne`, `count`, `aggregate`, `updateMany`, `deleteMany` and the analytics face (`query()` and `generateSql()`). There, an `undefined` or object comparand is refused first by that face's comparand-type check, also `INVALID_FILTER` / 400, in its own words. A refused write changes nothing.
+
+**What to write instead.** Write the boolean itself. `"$exists": true` matches rows whose field has a value, and `"$exists": false` matches rows whose field has none.
+
+**Who is affected.** A caller that sent a non-boolean `$exists` to `InMemoryDriver` or `MongoDBDriver` (a test suite, a local or embedded deployment, a flow or hook calling the engine in-process) and read the answer as a real one. On `SqlDriver` the same filter was already a 400.
+
+**Unchanged.** `$exists: true` and `$exists: false` answer exactly as before. The aggregation `filter` and `having` positions, which the engine evaluates itself after the driver, are not changed by this entry.

.changeset/20912-analytics-multi-value-distinct-refused.md

@@ -0,0 +1,27 @@
+---
+"@objectstack/service-analytics": minor
+---
+
+fix(service-analytics)!: a grouped dimension on a multi-value field and a `count_distinct` measure over a JSON-stored field are refused with `INVALID_FIELD` / 400 at the analytics door, before any SQL is built; a dataset `count_distinct` measure over a field declared `multiple: true` is refused at compile time
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (runtime-interface-only packages/services/service-analytics/src/dataset-compiler.ts#DatasetCompileOptions) a published options interface of `compileDataset`, whose one probe member is renamed and now answers the field's declaration instead of its type. It is not a Zod schema, a `packages/spec` declaration or an object definition, it is not a projection of a schema, and no metadata surface references it, so `objectstack migrate meta` has nothing to rewrite; the TypeScript host that passed the old member is told by its compiler at its own call site. The other halves refuse a QUERY shape at the analytics door (a grouping or distinct-count target) and one aggregate x declaration pair at the dataset compile door: no authorable key, spelling or stored row moves (`CubeSchema`, `DatasetSchema` and the analytics query body keep parsing every member), and which scalar a caller meant to group on or count is not something a ledger entry can rewrite. The other categories are closed on facts: the package publishes (not `unpublished`); no ADR-0087 id covers an analytics grouping or distinct-count target or this interface, and this diff adds none (not `registered` / `already-registered`); and the interface was concretely typed at the merge base, not erased (not `type-surface-only`). -->
+
+**BREAKING**: this narrows what the analytics query doors accept, in two positions, and what the dataset compiler accepts, in one. It holds on `POST /api/v1/analytics/query`, on its dry run `POST /api/v1/analytics/sql` and on `POST /api/v1/analytics/dataset/query`, on every driver and on both strategies. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+- A cube or dataset dimension whose column is a **multi-value** field, when it groups the result (a `dimensions` entry, or a `timeDimensions` entry with a `granularity`): an inherently multi option type (`multiselect`, `checkboxes`, `tags`), or a `select`, `radio`, `lookup`, `user`, `file` or `image` field declared `multiple: true`. The same types without the flag are served.
+- A `measures` entry that resolves to a `count_distinct` measure whose column is **JSON-stored**: a structured-JSON type (`json`, `composite`, `repeater`, `record`, `location`, `address`, `vector`), an inherently multi option type, or a multi-capable field declared `multiple: true`. An authored cube measure, a suffix-inferred one (`tags_count_distinct`) and a compiled dataset's are judged alike; a `count` measure is not judged.
+- A dataset measure that pairs `count_distinct` with a base-object field declared `multiple: true` is refused `400 DATASET_INVALID` when the dataset compiles, at registration and on the request door, beside the type row that already refused `tags`.
+
+The column is judged where it is declared: on the cube's object, or, for a dotted path, on the object the cube's declared join names.
+
+**What an author sees now.** `400 INVALID_FIELD`, naming the member as the request wrote it, the column, the object, the declaration (`select with multiple: true`), saying the query was not run, and naming the route. For a multi-value field the route is a record query on the declaring object filtered by one member with `$contains`, one query per member. For a structured-JSON field it is to store the scalar part in a field of its own. The thrown error carries `member`, `param` (`dimensions`, `timeDimensions` or `measures`), `cube`, `field` and `object`. The dataset compile refusal names the measure, the field and its declaration with `multiple: true`.
+
+**Why a refusal.** The engine's aggregate door already refuses both shapes, and the native-SQL strategy compiled its own statement and never reached it. Measured through this service as `AnalyticsServicePlugin` composes it over a real engine: a dimension on a `tags`, `multiselect` or `multiple: true` select answered 200 with one group per serialized array on SQLite and 500 `DATABASE_ERROR` on PostgreSQL 16; an inferred `count_distinct` over a `json`, `tags` or `multiple: true` select field answered 2, 3 and 2 on SQLite and 500 on PostgreSQL; a dataset `count_distinct` over the `multiple: true` select registered, then answered 2 on SQLite and 500 on PostgreSQL. The engine-aggregate strategy answered 400 for every one of these, under the engine's position (`groupBy[0]`, `aggregations[0].field`) rather than the member the caller wrote. The predicates are `@objectstack/spec/data`'s, the ones the engine's doors read: `isMultiValueField`, and the aggregate × field-type table's `count_distinct` row.
+
+**Your fix (a host calling `compileDataset` directly).** `DatasetCompileOptions` no longer has `declaredFieldType`. Pass `declaredValueShape` instead: the same read of the field's metadata, answering `{ type, multiple }` (the type, and `multiple === true`) rather than the type alone, or `undefined` when nothing answers. A host that passes neither compiles exactly as before, with no aggregate × field-type refusal at all. `AnalyticsService` and `AnalyticsServicePlugin` wire it themselves from `sourceFieldMeta`.
+
+**Who is affected.** A dashboard, report or caller that grouped by a multi-value field, or counted a JSON-stored field distinct, through the native-SQL strategy on SQLite and read the serialized arrays or the text-compared count as real answers. On PostgreSQL the same queries were already a 500. A dataset that pairs `count_distinct` with a `multiple: true` field no longer registers.
+
+**Unchanged.** A dimension or `count_distinct` on a scalar-stored field, a single-value `select` or `lookup` included; `count` over any field; a member naming a column the object does not have, which keeps its existing `INVALID_FIELD` answer first; a dotted path the cube declares no join for; a measure whose `sql` is an expression; and a host that wires no `sourceFieldMeta`, where the declaration cannot be read.

.changeset/20917-analytics-field-permission-gate.md

@@ -8,7 +8,7 @@ Clause-②: yes (narrowing)
 
 <!-- adr-0087: not-required (no-migration-prescription) No authorable key, export or stored shape is removed or renamed. The change refuses analytics queries that read a field the caller's field-level permissions hide, which the engine already refuses on the data API and on the ObjectQL strategy, so there is nothing for `objectstack migrate meta` to rewrite. The one public-surface addition is a new optional service hook. -->
 
-**BREAKING for analytics queries on a SQL deployment that read a field the caller may not read.**
+**BREAKING for analytics queries that read a field the caller may not read: on a SQL deployment, and on `POST /api/v1/analytics/sql` whichever strategy serves the cube.**
 
 **What changed.** `POST /api/v1/analytics/query`, `POST /api/v1/analytics/sql`
 and `POST /api/v1/analytics/dataset/query` now judge every field a query reads
@@ -19,7 +19,9 @@ filters. A member of an authored cube is judged by the field it resolves to,
 not by its name in the cube. A field the caller may not read answers
 `403 PERMISSION_DENIED`, in the words the engine uses for the same field. The
 native-SQL strategy, the one a SQL driver serves first, answered such queries;
-the ObjectQL strategy and the data API already refused them.
+the ObjectQL strategy already refused them on `POST /api/v1/analytics/query`
+and `POST /api/v1/analytics/dataset/query`, as the data API did, but printed
+the statement on `POST /api/v1/analytics/sql`.
 
 **What is not affected.** A query that reads only fields the caller may read
 answers as before. A system context, and a caller with no permission sets, are

.changeset/20933-analytics-relationship-path-admission.md

@@ -8,7 +8,7 @@ Clause-②: no (narrowing)
 
 <!-- adr-0087: not-required (no-migration-prescription) No authorable key, export or stored shape is removed or renamed. The change refuses, or row-scopes, what the native-SQL strategy read from an object reached through an undeclared relationship path, the way a declared join to the same object already was; there is nothing for `objectstack migrate meta` to rewrite. -->
 
-**BREAKING for analytics queries on a SQL deployment that read a related object through a relationship path the cube does not declare.**
+**BREAKING for analytics queries that read a related object through a relationship path the cube does not declare: on a SQL deployment, and on `POST /api/v1/analytics/sql` whichever strategy serves the cube.**
 
 **What changed.** The analytics door admits and row-scopes one object set
 before either strategy runs. It held the cube's base object and the joins the
@@ -40,7 +40,9 @@ deployment with no security service applies no object-level check, as on the
 data API.
 
 **Refusals that change form.** On the ObjectQL strategy a related object the
-caller may not read was already refused; it now answers the analytics door's
+caller may not read was already refused on `POST /api/v1/analytics/query` and
+`POST /api/v1/analytics/dataset/query`, though `POST /api/v1/analytics/sql`
+printed the statement; on those two doors it now answers the analytics door's
 refusal rather than the engine's, the same one a declared join gets. A filter,
 a time window or a two-hop path through such an object moves from
 `400 INVALID_FIELD` to that `403`. A relationship path whose relationship name