fix(objectql, service-analytics)!: a date bucket over a formula field is refused alike on every analytics face, instead of a driver error, one null group or a rendered statement

.changeset/28977-sql-dry-run-refuses-date-bucket-over-non-date.md

@@ -1,3 +1,3 @@
 ---
 '@objectstack/service-analytics': patch
 ---
@@ -12,7 +12,7 @@
 **BREAKING** accept-set narrowing on the `/analytics/sql` dry run (`@objectstack/service-analytics`). `/analytics/query` is unchanged.
 
 - **Before.** Measured on real SQLite on `main` `ab085a04`. `/analytics/query` refused a month bucket over a text, select or boolean column, a year bucket over a number and a day bucket over a `time` column with the engine's `INVALID_FIELD` / 400 at `groupBy.N.dateGranularity`. The dry run rendered `strftime('%Y-%m', region)` and its kin, on the direct and the cross-object path, and for a cube dimension's one declared granularity asked raw. At `America/New_York` it answered `NOT_IMPLEMENTED` / 501.
-- **Now.** The dry run judges the keys `execute()` hands the engine (its cross-object path's when the selection routes there) with the reading the draft preview already used (#28852), now `dataset-date-bucket-door.ts`. It runs after the dry run's `dateRange` read and its column-asked-twice refusal, as `query()` refuses, and gives the engine's refusal: the same message, `object`, `param`, `path`, `field` and `fields`. A field with no declared type, or a `formula`, is not judged, as in the engine.
+- **Now.** The dry run judges the keys `execute()` hands the engine (its cross-object path's when the selection routes there) with the reading the draft preview already used (#28852), now `dataset-date-bucket-door.ts`. It runs after the dry run's `dateRange` read and its column-asked-twice refusal, as `query()` refuses, and gives the engine's refusal: the same message, `object`, `param`, `path`, `field` and `fields`. A field with no declared type is not judged, as in the engine; a `formula` is refused with the engine's virtual-field refusal on every face (#29167).
 - **Unchanged.** A date or datetime bucket renders byte for byte, the statement the query echoes.
 - **Who sees it.** A caller of the dry run for a selection the query refuses: the query's refusal, instead of SQL for a statement that never runs.
 

.changeset/29167-formula-date-bucket-refused-alike.md

@@ -0,0 +1,20 @@
+---
+'@objectstack/objectql': patch
+'@objectstack/service-analytics': patch
+---
+
+fix(objectql, service-analytics)!: a date bucket over a formula field is refused alike on every analytics face, instead of a driver error, one null group or a rendered statement
+
+Clause-②: no (narrowing) — `engine.aggregate` now refuses `INVALID_FIELD` / 400 at `groupBy.N.dateGranularity` a `dateGranularity` over a virtual (`formula`) field, whatever its `returnType`, before any driver is asked; `/analytics/sql` (`AnalyticsService.generateSql`) and the draft preview of `queryDataset` give the same refusal, field for field. An engine that aggregates in memory served one `null` group for it, the dry run rendered `strftime('%Y-%m', calc)`, and the preview served one `null` group. Nothing is widened: no export, public member, answer member, error code or status is added.
+Metadata-protocol: none
+
+<!-- adr-0087: not-required (no-migration-prescription) No key of any schema is added, removed, renamed or re-shaped, and no export or type of either package root changes: a query that bucketed a column the table does not store is refused before it reaches the driver, with an error code and status already on the wire. There is nothing for a migration to rewrite and no tombstone. -->
+
+**BREAKING** accept-set narrowing on `engine.aggregate` (`@objectstack/objectql`) and on the `/analytics/sql` dry run and the draft preview (`@objectstack/service-analytics`), for a date bucket over a `formula` field only.
+
+- **Before.** Measured on `main` `7788eedb`, a month bucket over a `formula` with `returnType: 'date'`. On a SQL driver, `/analytics/query` (both compositions) answered the driver's missing-column backstop: `INVALID_FIELD` / 400 with no `path` and a "run schema sync" prescription that cannot help. On an engine whose driver has no native `aggregate` (the in-memory aggregation), the query was served: one `null` group holding every row. The dry run rendered `strftime('%Y-%m', calc)`, and the draft preview served one `null` group holding the total.
+- **Now.** The engine's date-bucket door (`assertDateBucketTargetsAreTemporal`) judges a virtual field first, by the spec's `isVirtualSearchField` (`SEARCH_VIRTUAL_TYPES`), the predicate the sort, filter and search axes already refuse a formula by. It refuses with `INVALID_FIELD` / 400 at `groupBy.N.dateGranularity`, naming the field, its type and the granularity. The service's copy of that door (`dataset-date-bucket-door.ts`) refuses the same way, so the dry run and the preview give the query's refusal: the same message, `object`, `param`, `path`, `field` and `fields`.
+- **Unchanged.** A date or datetime bucket is served on every face, byte for byte. A plain `groupBy` over a formula (no `dateGranularity`) is not this change's.
+- **Who sees it.** A caller bucketing a formula field: a located refusal with a remedy, instead of a driver error, an empty-looking group or SQL for a statement that never runs.
+
+**The remedy.** Denormalise the value onto the object (a stored date or datetime field, written when the source changes) and bucket that.

packages/objectql/src/aggregate-query-shape-door.ts

@@ -46,7 +46,28 @@
  * day by. A `date` (`CALENDAR_DATE_TYPES`) and a `datetime` (`INSTANT_TYPES`)
  * pass. **Not judged:** a field the declaration cannot type — undeclared (the
  * ingress door answers an unknown one first; a system column the map does not
- * declare is bucketed as before), a `formula`, or a registry-less host.
+ * declare is bucketed as before), or a registry-less host.
+ *
+ * ## A virtual field (`formula`) is refused, on its storage ground [#29167]
+ *
+ * A `formula` names no stored value class (its value is the expression's), so
+ * the class reading above cannot judge it. It is refused on the ground the
+ * sort, filter, search and value-aggregate axes already refuse it on: it is
+ * VIRTUAL — computed on read, no column — judged by the spec's
+ * `isVirtualSearchField` (`SEARCH_VIRTUAL_TYPES`), never a local type list,
+ * and whatever its `returnType` says. Measured on `origin/main` `7788eedb`
+ * before this arm, a month bucket over a `formula` with `returnType: 'date'`:
+ *
+ * | caller | answer |
+ * |:--|:--|
+ * | a SQL driver | `INVALID_FIELD` / 400 from the driver's missing-column backstop: no `path`, and a "run schema sync" prescription that cannot help |
+ * | a driver with no native `aggregate` (the in-memory aggregation) | `200`, one `null` group holding every row |
+ *
+ * Both now answer {@link virtualDateBucketMessage}, `INVALID_FIELD` / 400 at
+ * `groupBy.N.dateGranularity`, with the remedy those axes give: store the
+ * value on a stored field and use that. `@objectstack/service-analytics`'s
+ * `dataset-date-bucket-door.ts` gives the same refusal on the two faces that
+ * reach no engine (the `/analytics/sql` dry run and the draft preview).
  *
  * ## The column names
  *
@@ -65,6 +86,7 @@
  */
 
 import { StandardErrorCode } from '@objectstack/spec/api';
+import { isVirtualSearchField } from '@objectstack/spec/data';
 import { declaredFieldClasses } from './having-filter.js';
 
 type ShapeRefusal = Error & {
@@ -87,10 +109,29 @@ function refusal(
   return err;
 }
 
+/**
+ * [#29167] The refusal of a date bucket over a VIRTUAL field (see the module
+ * header): the position, the field, its type and the granularity it was
+ * bucketed by, then the store-the-value remedy the sort and filter axes give.
+ * `@objectstack/service-analytics`'s `dataset-date-bucket-door.ts` carries the
+ * one copy, compared field for field by its tests.
+ */
+function virtualDateBucketMessage(
+  object: string, path: string, field: string, type: string, dateGranularity: string,
+): string {
+  return `aggregate('${object}'): ${path} buckets '${field}' by '${dateGranularity}', but '${field}' `
+    + `is a ${type} field — its value is computed on read and never stored, so no driver materialises a `
+    + `column to bucket. The query was NOT run. Denormalise the value onto '${object}' (a stored date or `
+    + 'datetime field, written when the source changes) and bucket that. A formula field is virtual: with '
+    + 'no column behind it the bucket reached the driver as a column that does not exist, and an engine '
+    + 'that aggregates in memory put every row into one null group while the answer looked served.';
+}
+
 /**
  * Refuse a `groupBy` entry whose `dateGranularity` buckets a field the object
- * declares neither a date nor a datetime — `INVALID_FIELD` / 400, naming the
- * position, the field and its declared type. See the module header.
+ * declares neither a date nor a datetime, or a virtual (`formula`) field —
+ * `INVALID_FIELD` / 400, naming the position, the field and its declared type.
+ * See the module header.
  */
 export function assertDateBucketTargetsAreTemporal(
   object: string,
@@ -105,11 +146,23 @@ export function assertDateBucketTargetsAreTemporal(
     if (entry === null || typeof entry !== 'object' || Array.isArray(entry)) continue;
     const { field, dateGranularity } = entry as { field?: unknown; dateGranularity?: unknown };
     if (dateGranularity == null || typeof field !== 'string') continue;
+    const path = `groupBy.${i}.dateGranularity`;
+    const declared = Object.prototype.hasOwnProperty.call(fields, field)
+      ? (fields as Record<string, { type?: unknown }>)[field]
+      : undefined;
+    // [#29167] A virtual field first: the class reading below has no class
+    // for it, so it would otherwise be skipped unjudged.
+    if (isVirtualSearchField(declared as never)) {
+      throw refusal(
+        'INVALID_FIELD',
+        virtualDateBucketMessage(object, path, field, String(declared?.type), String(dateGranularity)),
+        { field, fields: [field], object, param: 'groupBy', path },
+      );
+    }
     classes ??= declaredFieldClasses(fields);
     const cls = classes.get(field);
     if (cls === undefined || cls === 'date' || cls === 'datetime') continue;
     const type = String((fields as Record<string, { type?: unknown }>)[field]?.type);
-    const path = `groupBy.${i}.dateGranularity`;
     throw refusal(
       'INVALID_FIELD',
       `aggregate('${object}'): ${path} buckets '${field}' by '${String(dateGranularity)}', but '${field}' `

packages/objectql/src/engine-aggregate-query-shape-door.test.ts

@@ -39,6 +39,8 @@ const PROBE = {
     opened_on: { name: 'opened_on', type: 'date' },
     seen_at: { name: 'seen_at', type: 'datetime' },
     doubled: { name: 'doubled', type: 'formula', expression: 'amount * 2' },
+    // [#29167] A formula whose declared result IS a date: refused all the same.
+    opened_calc: { name: 'opened_calc', type: 'formula', expression: 'opened_on', returnType: 'date' },
   },
 };
 
@@ -106,6 +108,26 @@ describe('[#25818] the aggregate query shapes that answered a plausible wrong ta
     expect(reads, 'every refusal precedes the driver').toHaveLength(0);
   });
 
+  it('[#29167] refuses a dateGranularity over a virtual (formula) field, whatever its returnType: INVALID_FIELD / 400 at groupBy.N.dateGranularity — no read', async () => {
+    // Before this arm the in-memory aggregation served one null group here:
+    // the formula has no stored value for it to bucket.
+    for (const field of ['doubled', 'opened_calc']) {
+      const err = await refusalOf(aggregate({
+        groupBy: ['status', { field, dateGranularity: 'month' }],
+        aggregations: [COUNT],
+      }));
+      expect(envelopeOf(err), field).toEqual({ code: 'INVALID_FIELD', status: 400, httpStatus: 400 });
+      expect({ path: err?.path, param: err?.param, field: err?.field, fields: err?.fields, object: err?.object }, field)
+        .toEqual({ path: 'groupBy.1.dateGranularity', param: 'groupBy', field, fields: [field], object: OBJECT });
+      expect(err!.message, field).toContain(
+        `groupBy.1.dateGranularity buckets '${field}' by 'month', but '${field}' is a formula field — its value is computed on read and never stored`,
+      );
+      // The store-the-value remedy the sort and filter axes give a virtual field.
+      expect(err!.message, field).toContain(`Denormalise the value onto '${OBJECT}' (a stored date or datetime field, written when the source changes) and bucket that.`);
+    }
+    expect(reads, 'every refusal precedes the driver').toHaveLength(0);
+  });
+
   it('refuses a repeated aggregation alias: INVALID_QUERY / 400 at the second aggregations.N.alias — no read', async () => {
     const err = await refusalOf(aggregate({
       aggregations: [COUNT, { function: 'sum', field: 'amount', alias: 'n' }],
@@ -144,13 +166,13 @@ describe('[#25818] the aggregate query shapes that answered a plausible wrong ta
     expect(reads).toHaveLength(0);
   });
 
-  it('CONTROL a date or datetime bucket, a formula or undeclared bucket, and distinct columns are served', async () => {
+  it('CONTROL a date or datetime bucket, an undeclared bucket, and distinct columns are served', async () => {
     await expect(aggregate({ groupBy: [{ field: 'opened_on', dateGranularity: 'month' }], aggregations: [COUNT] }))
       .resolves.toEqual([{ opened_on: '2026-01', n: 1 }, { opened_on: '2026-02', n: 2 }]);
     await expect(aggregate({ groupBy: [{ field: 'seen_at', dateGranularity: 'month' }], aggregations: [COUNT] }))
       .resolves.toEqual([{ seen_at: '2026-01', n: 1 }, { seen_at: '2026-02', n: 2 }]);
-    // The declaration cannot type a formula or an undeclared column: no verdict.
-    await expect(aggregate({ groupBy: [{ field: 'doubled', dateGranularity: 'month' }], aggregations: [COUNT] })).resolves.toBeDefined();
+    // The declaration cannot type an undeclared column: no verdict. [#29167] A
+    // formula under a dateGranularity is refused (above), no longer a control.
     await expect(aggregate({ groupBy: [{ field: 'created_at', dateGranularity: 'month' }], aggregations: [COUNT] })).resolves.toBeDefined();
     // An alias moves the group column, so the field's own name is free.
     await expect(aggregate({ groupBy: [{ field: 'status', alias: 's' }], aggregations: [{ function: 'count', alias: 'status' }] }))
@@ -160,7 +182,7 @@ describe('[#25818] the aggregate query shapes that answered a plausible wrong ta
       groupBy: [{ field: 'opened_on', dateGranularity: 'year', alias: 'y' }, { field: 'opened_on', dateGranularity: 'month', alias: 'm' }],
       aggregations: [COUNT, { function: 'sum', field: 'amount', alias: 'total' }],
     })).resolves.toEqual([{ y: '2026', m: '2026-01', n: 1, total: 10 }, { y: '2026', m: '2026-02', n: 2, total: 50 }]);
-    expect(reads, "each served query read once").toHaveLength(6);
+    expect(reads, "each served query read once").toHaveLength(5);
   });
 
   it('the REST door answers the engine\'s refusal, not a 200 — through findData and the wire mapping', async () => {

packages/services/service-analytics/src/__tests__/objectql-echo-non-date-bucket.test.ts

@@ -22,6 +22,11 @@
  * | a cube dimension over `region` declaring one granularity, asked raw | the bucket |
  * | text `region` by month at `America/New_York` | `NOT_IMPLEMENTED` / 501 saying the query is served |
  *
+ * [#29167] …and a bucket over a `formula` (virtual: no stored column), which
+ * the dry run rendered as `strftime('%Y-%m', calc)` while the query refused it
+ * from the SQL driver's missing-column backstop: both now answer the engine's
+ * virtual-field refusal, field for field.
+ *
  * It now judges the keys `execute()` hands the engine with the service's one
  * reading of it (`dataset-date-bucket-door.ts`, #28852), after the `dateRange`
  * read and the column-asked-twice door as `query()` refuses, so it answers the
@@ -99,6 +104,8 @@ describe('[#28977] the /analytics/sql dry run refuses a date bucket over a colum
         closed_at: { name: 'closed_at', type: 'datetime' },
         closed_on: { name: 'closed_on', type: 'date' },
         account: { name: 'account', type: 'lookup', reference: ACCOUNT },
+        // [#29167] A formula the table stores no column for, whose result is a date.
+        calc: { name: 'calc', type: 'formula', expression: 'closed_on', returnType: 'date' },
       },
     } as never);
     await engine.syncSchemas();
@@ -155,6 +162,25 @@ describe('[#28977] the /analytics/sql dry run refuses a date bucket over a colum
     expect(asSql, 'the dry run vs the query').toEqual(asQuery);
   });
 
+  it.each<[string, Query, string]>([
+    ['by month', onObject({ timeDimensions: [td('calc', 'month')] }), 'groupBy.0.dateGranularity'],
+    ['after a datetime bucket', onObject({ timeDimensions: [td('closed_at', 'year'), td('calc', 'month')] }), 'groupBy.1.dateGranularity'],
+    ['at a non-UTC zone', onObject({ timeDimensions: [td('calc', 'month')], timezone: 'America/New_York' }), 'groupBy.0.dateGranularity'],
+  ])('[#29167] refuses a bucket over a formula (virtual: no stored column) %s with the query\'s refusal, where the dry run rendered strftime', async (_label, query, path) => {
+    const { asQuery, asSql } = await bothRefusals(query);
+    expect(asQuery).toEqual({
+      message:
+        `aggregate('${DEAL}'): ${path} buckets 'calc' by 'month', but 'calc' is a formula field — its value is computed `
+        + 'on read and never stored, so no driver materialises a column to bucket. The query was NOT run. Denormalise '
+        + `the value onto '${DEAL}' (a stored date or datetime field, written when the source changes) and bucket that. `
+        + 'A formula field is virtual: with no column behind it the bucket reached the driver as a column that does not '
+        + 'exist, and an engine that aggregates in memory put every row into one null group while the answer looked served.',
+      code: 'INVALID_FIELD', status: 400, httpStatus: 400,
+      object: DEAL, param: 'groupBy', path, field: 'calc', fields: ['calc'],
+    });
+    expect(asSql, 'the dry run vs the query').toEqual(asQuery);
+  });
+
   it.each<[string, Query, string]>([
     ["beside a cross-object 'account.name' (its FK column first)", onObject({ dimensions: ['account.name'], timeDimensions: [td('region', 'month')] }), 'groupBy.1.dateGranularity'],
     ['after a raw date column and a datetime bucket', onObject({ dimensions: ['closed_on'], timeDimensions: [td('closed_at', 'month'), td('region', 'month')] }), 'groupBy.2.dateGranularity'],