Repository navigation
Commit bb2eccf
fix(service-analytics)!: the analytics door refuses a multi-value grouping and a JSON-stored count_distinct, judged on the declaration, as the engine does (#21019)
Fixes #20912
Clause-②: no (narrowing)
## What changes
The analytics door refuses what the engine's aggregate door refuses,
ahead of both strategies, naming the member the caller wrote:
- a **grouped dimension on a multi-value field** (`multiselect`,
`checkboxes`, `tags`, or a `select` / `radio` / `lookup` / `user` /
`file` / `image` declared `multiple: true`), `INVALID_FIELD` / 400;
- a **`count_distinct` measure over a JSON-stored field** (the
compatibility table's `count_distinct` row refuses the type, or the
declaration is a list), `INVALID_FIELD` / 400. An authored cube measure,
a suffix-inferred `FIELD_count_distinct` and a compiled dataset's are
judged alike;
- and the dataset compiler's measure door **reads the declaration**: a
dataset `count_distinct` over a base-object field declared `multiple:
true` is refused `DATASET_INVALID` / 400 at compile time (registration
and the request door), beside the type row that already refused `tags`.
The predicates are `@objectstack/spec/data`'s, the ones the engine's two
doors read: `isMultiValueField`, `STRUCTURED_JSON_TYPES`, and
`isAggregateCompatibleWithFieldType('count_distinct', …)`. No copy of
the rule. Both doors receive the declaration `{ type, multiple }`
(`ValueShapeFieldDef`, the shape `declaredValueShape` already builds),
never the type alone: a type-only door would refuse `tags` and serve a
`select` with `multiple: true`, which is the same JSON column.
Files:
- `structured-json-dimension-door.ts` (the door #20807 landed): the
multi-value class beside the structured-JSON one for grouped members,
and a second walk over `query.measures` for `count_distinct`. The #20807
structured-JSON words are byte-identical.
- `dataset-compiler.ts`: `assertAggregateFieldTypeCompatible` asks
`isMultiValueField` beside the table's row, for `count_distinct` only
(the `sum` / `avg` / `min` / `max` rows accept no multi-capable type,
and `count` reads no value). `DatasetCompileOptions.declaredFieldType`
becomes `declaredValueShape`, answering `{ type, multiple }`.
- `analytics-service.ts`: exactly the two call sites the seat's surface
decision names. The lambda `assertDimensionsGroupScalarColumns` hands
the door, and `compile()`'s probe handed to `compileDataset`, each pass
`{ type, multiple }` instead of `?.type`. Nothing else in that file
changes.
## The three rows, measured before and after
Through `AnalyticsService` as `AnalyticsServicePlugin` composes it over
a real ObjectQL engine and `SqlDriver`, on SQLite (better-sqlite3) and a
live PostgreSQL 16.13 started for this run. Four rows: `status` a, a, b,
b (single-value select, the control); `tags_f` (tags) [a,b], [a,b], [b],
[ab]; `ms` (multiselect) and `multi_sel` (select, `multiple: true`)
[a,b], [a,b], [b], [b]; `meta` (json) two equal documents of four.
Before is `origin/main` `8d329f02`; after is this branch at
`d53f230cdb`.
| input | strategy | before, SQLite | before, PostgreSQL 16.13 | after,
both dialects |
|:--|:--|:--|:--|:--|
| row 1: cube dimension `tags_f` / `ms` / `multi_sel` | NativeSQL | 200,
one group per serialized array | 500 `DATABASE_ERROR` | 400
`INVALID_FIELD`, member named |
| row 1: same | ObjectQL | 400 `INVALID_FIELD`, engine's `groupBy[0]` |
same | 400 `INVALID_FIELD`, member named |
| row 1: dataset dimension over `tags_f` / `multi_sel` | NativeSQL |
200, one group per serialized array | 500 | 400 `INVALID_FIELD`, dataset
dimension named |
| row 2: `meta_count_distinct` / `tags_f_count_distinct` /
`multi_sel_count_distinct`, ad-hoc and authored cube | NativeSQL | 200:
2 / 3 / 2 | 500 | 400 `INVALID_FIELD`, measure named |
| row 2: same | ObjectQL | 400 `INVALID_FIELD`, engine's
`aggregations[0].field` | same | 400 `INVALID_FIELD`, measure named |
| row 3: dataset `count_distinct` over `multi_sel` | NativeSQL |
registers, then 200, 2 | registers, then 500 | 400 `DATASET_INVALID` at
registration |
| row 3: same | ObjectQL | registers, then 400 `INVALID_FIELD` at query
time | same | 400 `DATASET_INVALID` at registration |
| controls: dimension `status`, `status_count_distinct`, dataset
`count_distinct` over `status` | both | 200 | 200 | 200, unchanged |
On the ObjectQL face the code and status were already right; what
changes there is that the refusal names the member the caller wrote,
which is the fold-in the card offers.
### The words
They follow the engine's doors (verdict, "The query was NOT run.",
route, reason) in the member-named form #20807 used. One deliberate
difference in the multi-value **route**: it names a **record query** on
the declaring object with `where { "FIELD": { "$contains": VALUE } }`,
not this service's own `where`. Measured on the same base, the analytics
native strategy compiles `$contains` over a JSON-stored column to
`LIKE`: `where { tags_f: { $contains: 'a' } }` answered 3 on SQLite
(`[ab]` counted as holding `a`; membership is 2) and 500 on PostgreSQL
(`json ~~ text`), while the ObjectQL face and the engine answered 2 on
both. A route through the analytics `where` would trade one wrong answer
for another. That native-path `$contains` defect is outside this card
and is reported for filing.
## Pins
Committed red first (`f88977bf4e`), then the fix (`2c1b093a66`):
-
`service-analytics/src/__tests__/multi-value-json-stored-door.test.ts`:
rows 1 and 2 on both strategy faces (every multi-value declaration, the
member as spelled, a bucketed time dimension, a dataset dimension and a
dataset measure over a declared join, the dry run), row 3 through
`compileDataset` and through the service, single-value controls, and two
GUARDs over every `FieldType` with and without `multiple: true` whose
expected verdict is computed from the spec predicates. Red before the
fix: 16 of 21 cases; the 5 controls green.
-
`service-analytics/src/__tests__/json-stored-door-live-drivers.test.ts`:
the card's three rows and a scalar control for each, over a real
`SqlDriver`, SQLite always and PostgreSQL where `OS_TEST_POSTGRES_URL`
is set. Red before: all three refusal cases on both dialects; controls
green.
- `rest/src/analytics-dataset-multi-value-door.test.ts`: `POST
/api/v1/analytics/dataset/query` with an inline dataset, rows 1 and 3
and their controls, SQLite and PostgreSQL. Red before: both refusal
cases on both dialects. That first run also caught a wrong control
expectation of mine (the dataset route answers the option labels, `A` 2
and `B` 2, which it did on the base too); it was corrected before the
pins commit.
- `dimension-structured-json-door.test.ts`: its GUARD now expects
`STRUCTURED_JSON_TYPES` or `isMultiValueField` (an inherently multi
option type is judged by the door now).
The cube face over HTTP (`/api/v1/analytics/query`) is served by
`@objectstack/runtime`'s dispatcher, outside this card's surface; its
rows are pinned at the service on both dialects.
## Ablation
Run from the committed fix, each leg through
`scripts/ablation-replace.mjs` (anchor hit 1 to 0, blob changed), with a
whole-script `trap` restoring by absolute path; each restore proven by
blob equality with `HEAD` and an empty `git diff HEAD`. Suites: the
three service pin files plus
`aggregate-nontemporal-measure-refusal.test.ts` (92 cases, PostgreSQL
cell on).
| leg | mutation | result |
|:--|:--|:--|
| M1a | door call site back to `fieldMeta(o, field)?.type` | 27 red |
| M1b | door call site drops `multiple` (type-only shape) | 16 red:
every `multiple: true` row; `tags` still refused |
| M2a | compile call site back to `?.type` | 33 red, the existing
compile-door pins included |
| M2b | compile call site drops `multiple` | 3 red: exactly row 3
(SQLite, PostgreSQL, compile unit) |
| M3 | door's measure walk removed | 9 red: row 2 |
| M4 | door's multi-value grouped class removed | 10 red: row 1 |
| M5 | compiler's declaration half removed | 4 red: row 3 |
| control | restored tree | 92 / 92 green |
The REST pin resolves `@objectstack/service-analytics` through `dist/`,
so one leg ran there: M1a, rebuilt, `ablation-dist-preflight` found the
marker in `dist/index.js` and `dist/index.cjs`, the REST pin went 2 red
(the dimension rows, SQLite and PostgreSQL), then restored, rebuilt,
preflight `--absent` and a clean tree, 8 / 8 green. The mutated build's
DTS step failed on the `?.type` revert (TS2345), so the type system also
refuses that call site.
## Tests and gates, at `d53f230cdb`
- `@objectstack/service-analytics`: `vitest run` 152 files, 3489 tests
passed (PostgreSQL cell on); `typecheck` clean.
- `@objectstack/rest`: the new pin 8 / 8 (SQLite and PostgreSQL);
`typecheck` clean (`check:test-typecheck` OK).
- `node scripts/pm/dispatch-gates.mjs --commands` derived 62 commands;
all 62 run under the shared lock, plus the 4 roster families it marks
(`check-changeset-fixed`, `check:authz-resolver`,
`check:error-code-casing`, `check:filter-alias-parity`). 64 exit 0.
`--ran` reconciliation: 62 derived, 60 run, 2 NOT MEASURED.
- `check-adr-0087-registration`: passes with `not-required
(runtime-interface-only …#DatasetCompileOptions)`.
`check-changeset-no-major`: no `major`.
## Acceptance notes
- **NOT MEASURED: `check:dual-build-cjs-loads` and
`check:type-check-debt`**, reason: both exit 3 PREREQUISITE NOT MET,
needing a whole-workspace build; this run built only the tested closure
(`@objectstack/rest^...`). CI builds the full tree.
- **NOT MEASURED: MySQL.** The refusals happen before any SQL is built,
so the dialect does not enter the verdict.
- **PostgreSQL cells are un-run in CI** (no step provisions
`OS_TEST_POSTGRES_URL` for these packages); the PostgreSQL 16.13
readings above are from a local server.
- **Downstream suites declared to CI:** `@objectstack/runtime`, the
dogfood suites and other consumers of the analytics doors were not run
locally. A `git grep` of `count_distinct` in `examples/`,
`packages/runtime/src`, `packages/qa` and `packages/rest/src` found one
analytics use, `region_count_distinct` over a `text` field, which is not
judged.
- **Published interface:** `DatasetCompileOptions.declaredFieldType` is
renamed `declaredValueShape` and returns `{ type, multiple }`. The
changeset says this under **BREAKING** with a "Your fix" paragraph for a
host calling `compileDataset` directly. Its ADR-0087 disposition is
`runtime-interface-only` naming that interface, which the gate verified
(no metadata surface). The fix paragraph is prose, in the same shape as
the landed `20648` changeset's.
- **Naming:** `assertNoStructuredJsonDimension` and
`structured-json-dimension-door.ts` now also judge multi-value grouping
and `count_distinct` measures. They are not renamed, because a rename
would change the call in `analytics-service.ts` beyond the two granted
lambdas. The module header says what it judges.
- **Comments beside the two call sites** in `analytics-service.ts` still
say "declared type". They are left as they are for the same reason.
- **Reported for filing, not fixed here:** (1) the analytics native
strategy compiles `$contains` over a JSON-stored column to `LIKE`,
against the membership reading `filter.zod.ts` declares (substring on
SQLite, 500 on PostgreSQL); (2) the dataset-measure lint rule
`measure-aggregate-field-type-refused` judges the type only, so it
passes `count_distinct` over a `multiple: true` select, which the
compile leg now refuses.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 78e3e3a commit bb2eccf
8 files changed
Lines changed: 1175 additions & 53 deletions
File tree
- .changeset
- packages
- rest/src
- services/service-analytics/src
- __tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | | - | |
| 24 | + | |
| 25 | + | |
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
246 | 246 | | |
247 | 247 | | |
248 | 248 | | |
249 | | - | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
250 | 254 | | |
251 | 255 | | |
252 | 256 | | |
| |||
255 | 259 | | |
256 | 260 | | |
257 | 261 | | |
258 | | - | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
259 | 265 | | |
260 | 266 | | |
261 | 267 | | |
| |||
0 commit comments