fix(objectql,metadata-protocol)!: the per-aggregation filter takes where's remaining doors — a bad date, an addDays numeric pair, an undeclared { $field } and an unknown key are refused; a Date bound compares as an instant

.changeset/20099-having-where-doors.md

@@ -1,3 +1,3 @@
 ---
 "@objectstack/objectql": minor
 ---
@@ -30,4 +30,4 @@
 
 Who is affected: `having` is a request-only key (`QuerySchema.having`, `EngineAggregateOptions.having`), and no metadata type stores it. Every `having` in this repository's docs and published skills is a scalar comparison against an aggregation alias (`{ order_count: { $gt: 5 } }` and the like), which answers exactly as before. Callers of `engine.aggregate` and of the REST aggregate query in a deployment were NOT measured.
 
-Not changed: scalars, `null` in the equality slot, `$in` / `$nin` lists, a two-bound `$between`, scalar ordering bounds, `{}`, `null` and an omitted `having`, on both paths. The `$like` / `$ilike` operators are still refused on `having` (they are staged out of `FILTER_OPERATORS`), and `$ne` with a list is still answered until the shared face judges it. A `having` key that names no column still keeps no group rather than being refused.
+Not changed: scalars, `null` in the equality slot, `$in` / `$nin` lists, a two-bound `$between`, scalar ordering bounds, `{}`, `null` and an omitted `having`, on both paths. The `$like` / `$ilike` operators are still refused on `having` (they are staged out of `FILTER_OPERATORS`), and `$ne` with a list is still answered until the shared face judges it. A `having` key that names no column is not judged by this change; since #20123 it is refused (`INVALID_FILTER` / 400) rather than keeping no group.

.changeset/20127-having-adddays-temporal-pair.md

@@ -1,3 +1,3 @@
 ---
 "@objectstack/objectql": minor
 ---
@@ -31,4 +31,4 @@
 
 Who is affected: `having` is a request-only key (`QuerySchema.having`, `EngineAggregateOptions.having`), and no metadata type stores it. No `having` in this repository's docs and published skills carries a `{ $field }` reference. Callers of `engine.aggregate` and of the REST aggregate query in a deployment were NOT measured.
 
-Not changed, measured identical before and after on both paths: a `date` / `date` pair and a `datetime` / `datetime` pair, with a positive or negative whole-day literal or with an offset read from a numeric column (`max` of a number, or a `count`); a `day` date bucket against a `date` column; and any `{ $field }` comparison WITHOUT `addDays`, including a numeric pair and a numeric column against a `date` one. A per-aggregation `filter` (`aggregations[i].filter`) is not judged by this rule: it reads the object's raw columns, and this change classifies only the aggregated row's.
+Not changed, measured identical before and after on both paths: a `date` / `date` pair and a `datetime` / `datetime` pair, with a positive or negative whole-day literal or with an offset read from a numeric column (`max` of a number, or a `count`); a `day` date bucket against a `date` column; and any `{ $field }` comparison WITHOUT `addDays`, including a numeric pair and a numeric column against a `date` one. A per-aggregation `filter` (`aggregations[i].filter`) is not judged by this change: it reads the object's raw columns, and this change classifies only the aggregated row's. Since #20148 it is judged by the same class rule, against the object's declared fields.

.changeset/20148-aggregation-filter-keys-rest.md

@@ -0,0 +1,24 @@
+---
+"@objectstack/metadata-protocol": minor
+---
+
+fix(metadata-protocol)!: the read door judges the keys inside each per-aggregation `filter` with the gate an explicit `where` meets — an unknown key is `INVALID_FIELD` / 400, not a count of zero (#20148)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) this change adds no transition to migrate. A per-aggregation filter key that names no field of the object never had a meaning: no row carries it, so the aggregation read "no value" in every row and counted zero, or every row under a negation, while the same key in `where` was already refused by this gate. There is no accepted spelling it can be mechanically rewritten to: which field the author meant is an authoring decision, and the refusal names the key and suggests the nearest declared field. The gate is the one `where` already meets, with the same verdicts. The request is not a stored document, so there is nothing for `objectstack migrate meta` to rewrite. The table below records the answer each shape had and has; it prescribes no rewrite. -->
+
+**BREAKING**: this narrows what `POST /data/:object/query` (`findData`) accepts in `aggregations[i].filter`. Each entry's filter now goes through the gate the explicit `where` meets, `assertFilterFieldsExist`: the same field set, the same verdicts, `INVALID_FIELD` / 400, with the entry's position as the parameter (`aggregations[1].filter`). The refusal is raised before the engine is called. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+Measured on the base through `POST /data/:object/query` on `driver-memory` and `driver-sql`, over six rows in three groups:
+
+| in `aggregations[i].filter` | before | now |
+|:--|:--|:--|
+| a key naming no field of the object (`{ nope: 1 }`), also under `$and` | `200`, that count 0 | `INVALID_FIELD` / 400, naming the key |
+| the same key under `$ne`, `$not`, or behind a `$or` branch that holds | `200`, every row counted | `INVALID_FIELD` / 400 |
+| an unknown key carrying an unknown operator (`{ nope: { $median: 1 } }`) | `INVALID_FILTER` / 400 from the engine | `INVALID_FIELD` / 400: the field gate runs first, as it does for the same key in `where` |
+| a dotted key on a scalar head, or a key naming a `formula` field | `INVALID_FIELD` / 400 from the engine's own filter seam | `INVALID_FIELD` / 400 from this gate, in the words `where` gets |
+
+Run after the entry checks and the aggregated-field check, so an entry the spec cannot read keeps its shape refusal and an unknown aggregated `field` keeps its own. A filter that is not a plain object names no key here and is left to the engine's shape gate.
+
+Not changed: a filter on declared keys, on `id`, `created_at` or `updated_at`, and a relation head in the nested-object form reach the engine with the filter untouched.

.changeset/20148-aggregation-filter-where-doors.md

@@ -0,0 +1,29 @@
+---
+"@objectstack/objectql": minor
+---
+
+fix(objectql)!: a per-aggregation `filter` (`aggregations[i].filter`) on `engine.aggregate` takes the doors `where` takes — the temporal-comparand door, a `{ $field }` referent that must be a declared field, and the `addDays` class rule — and a `Date` bound is compared as an instant, as it is in `where` (#20148)
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) this change adds no transition to migrate. Each refused input is one this position never evaluated as written: it answered a count by coercion (usually zero, every row under a negation), while the same condition as a `where` was already refused, by the engine's temporal-comparand door or by driver-sql's cross-field compiler. There is no accepted spelling a refused input can be mechanically rewritten to: which date, which declared field, or which pair of temporal fields the author meant is an authoring decision, and the refusal says what the position takes. A stored measure filter reaches this position through the analytics service, which lowers it; there is no stored document for `objectstack migrate meta` to rewrite. The table below records the answer each shape had and has; it prescribes no rewrite. -->
+
+**BREAKING**: this narrows what a per-aggregation `filter` accepts on `engine.aggregate`, and on the REST aggregate query (`POST /data/:object/query` with `aggregations`) that forwards it there. Every refusal below is `INVALID_FILTER` / 400, raised in the engine's per-aggregation loop before any driver is asked for a row, identically on an empty and on a populated table. It ships as `minor` under the launch-window convention for accept-set narrowings.
+
+Measured on the base through `engine.aggregate` and through `POST /data/:object/query`, on `driver-memory` and `driver-sql`, over six rows in three groups, with the same condition written as the call's `where` beside each:
+
+| in `aggregations[i].filter` | before | now |
+|:--|:--|:--|
+| a comparand a declared temporal field cannot read: `{ placed_on: { $gt: 'not-a-date' } }` on a `date`, the same as an `$in` member, a `$between` endpoint, in the implicit slot, behind a `$or` branch that holds or under `$not`; a preset name (`'last_30_days'`) on a `datetime`; `'noon'` on a `time` | counted as written: no row for the bad bound, only the readable members of the `$in`, every row for the `$between`, held-`$or` and `$not` shapes. The same condition as a `where` was refused | refused by the temporal-comparand door `where` takes, run unchanged on this position, in its words. A `{placeholder}` is stepped around and resolved, as there |
+| a `{ $field }` naming no declared field of the object (`{ amount: { $gt: { $field: 'nope' } } }`), a dotted referent, or an `addDays` offset column the object does not declare | no row counted; every row under `$ne`, `$not` or a held `$or`. `driver-sql` refused the same comparison in a `where` | refused |
+| `addDays` on a pair `FieldReferenceSchema.addDays` does not declare it for: two numeric fields (`{ amount: { $gt: { $field: 'cap', addDays: 1 } } }`), a numeric referent, a text or a time pair, a `date` against a `datetime`, or an offset read from a column that is not numeric | answered by the evaluator's epoch-millisecond coercion: no row on the fixture, 4 of 6 for the `date` / `datetime` pair. `driver-sql` refused the same pair in a `where` | refused |
+
+The two `{ $field }` rows are refused in the withholding posture `driver-sql` applies to the same comparison in a `where`: the message names the aggregation that carries the reference (`aggregations[1].filter`) and the rule, and withholds the fields, the operator and the specific reason. The engine logs the withheld diagnostic at `warn`, once per refusal. A referent is judged against the object's declared field map plus `id`, `created_at` and `updated_at`, the set the REST field gate reads; on a host whose registry holds no field map for the object, nothing is judged.
+
+These refusals run after the walker's own (an unknown operator, a malformed reference), so a filter carrying both gets the walker's refusal first.
+
+Not refused, but answering differently:
+
+- **A `Date` bound is compared as an instant.** `{ opened_at: { $gt: new Date('2026-02-01') } }` against the ISO text a `datetime` column holds counted no row: JS compared the `Date` with the string by coercion. `$ne` and `$nin` counted every row, and a `$between` of two `Date`s counted every row. The same bound in a `where` counted 4 of 6 on both drivers. The comparison now reads the pair through `@objectstack/spec/data`'s `utcInstantMs`, the lift `@objectstack/formula`'s evaluator applies, whenever one side is a `Date` and both sides denote an instant. Every `datetime` row and a UTC-midnight `Date` on a `date` field now count what the same bound counts in a `where`. `having` shares this comparison, so a `Date` bound in `having` keeps the groups its ISO spelling keeps on a `datetime` column; on a `date`-class column (`min` / `max` of a `date` field) a UTC-midnight `Date` follows the calendar-day reading a `where` gives (`{ $gte: new Date('2026-02-01') }` keeps the group whose max is `2026-02-01`, and so does `$eq`), which the ISO-instant text, compared as text, did not. A `Date` is not JSON, so this reaches in-process callers only. Two readings still differ from a `where`, because the per-row comparison holds no declaration: a `Date` carrying a time of day against a `date` field, which a `where` reads as that UTC calendar day, and a `Date` against a `time` field, which is not an instant and is left as before.
+
+Not changed, measured identical before and after on both drivers: every `where`, `groupBy` and `having` answer outside the `Date` shapes above, and a per-aggregation filter that uses implicit equality, ordering bounds, `$in`, `$between`, `$or`, `{}`, a `{ $field }` between two declared numeric fields, `addDays` between two `date` or two `datetime` fields (literal or read from a numeric column), and a reference to `created_at` or `id`.

packages/metadata-protocol/src/protocol.aggregation-filter-fields.test.ts

@@ -0,0 +1,139 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#20148] The KEYS inside a per-aggregation `filter`, at the read door.
+ *
+ * `findData` judged a `where`'s keys against the object's field set (#7534,
+ * {@link ObjectStackProtocolImplementation} `assertFilterFieldsExist`) and an
+ * aggregation's `function` / `alias` / `field` (#4254), but never the keys of
+ * the filter an aggregation carries. Measured on the base through
+ * `POST /api/v1/data/:object/query` on driver-memory and driver-sql:
+ *
+ * ```
+ * aggregations: [{ …, filter: { nope: 1 } }]     -> 200, that count 0
+ * aggregations: [{ …, filter: { nope: { $ne: 1 } } }] -> 200, EVERY row counted
+ * where: { nope: 1 }                              -> 400 INVALID_FIELD
+ * ```
+ *
+ * Now each aggregation's filter goes through the `where` gate itself: the same
+ * field set, the same `unknown` > `dotted` > virtual ladder, the same
+ * `INVALID_FIELD` / 400, the caller's position (`aggregations[1].filter`) as
+ * the parameter — and the engine is never reached. Every pin below asserts
+ * `code` AND `status` AND that `engine.aggregate` was not called; the
+ * preservation half asserts what the engine received.
+ */
+
+import { describe, it, expect, vi } from 'vitest';
+import { ObjectStackProtocolImplementation } from './protocol.js';
+
+const SCHEMA = {
+    name: 'invoice',
+    fields: {
+        name: { name: 'name', type: 'text' },
+        status: { name: 'status', type: 'text' },
+        amount: { name: 'amount', type: 'number' },
+        doubled: { name: 'doubled', type: 'formula', expression: 'amount * 2' },
+        owner_id: { name: 'owner_id', type: 'lookup', reference: 'sys_user' },
+    },
+};
+
+function makeProtocol() {
+    const find = vi.fn(async () => [] as unknown[]);
+    const aggregate = vi.fn(async () => [] as unknown[]);
+    const engine = {
+        registry: { getObject: (n: string) => (n === 'invoice' ? SCHEMA : undefined) },
+        find,
+        aggregate,
+        count: vi.fn(async () => 0),
+    };
+    return { p: new ObjectStackProtocolImplementation(engine as any), find, aggregate };
+}
+
+/** The second of two aggregations carries `filter`, so the position must name WHICH one. */
+function withFilter(filter: unknown) {
+    return {
+        aggregations: [
+            { function: 'count', alias: 'n' },
+            { function: 'count', alias: 'm', filter },
+        ],
+    };
+}
+
+async function refusalFor(query: Record<string, unknown>) {
+    const { p, find, aggregate } = makeProtocol();
+    let answered: unknown;
+    try {
+        answered = await p.findData({ object: 'invoice', query } as never);
+    } catch (e) {
+        expect(find, 'the engine was reached before the refusal').not.toHaveBeenCalled();
+        expect(aggregate, 'the engine was reached before the refusal').not.toHaveBeenCalled();
+        return e as Error & { status?: number; code?: string; param?: string; field?: string; fields?: string[] };
+    }
+    throw new Error(`${JSON.stringify(query)} was ACCEPTED (answered ${JSON.stringify(answered)}) instead of refused`);
+}
+
+describe('[#20148] findData — the keys inside a per-aggregation filter meet the where gate', () => {
+    const REFUSED: ReadonlyArray<readonly [string, Record<string, unknown>, string]> = [
+        ['an unknown key (the card\'s row 4)', { nope: 1 }, 'nope'],
+        ['an unknown key under $ne', { nope: { $ne: 1 } }, 'nope'],
+        ['an unknown key under $and', { $and: [{ amount: { $gt: 0 } }, { nope: 1 }] }, 'nope'],
+        ['an unknown key behind a $or branch that holds', { $or: [{ amount: { $gt: 0 } }, { nope: 1 }] }, 'nope'],
+        ['an unknown key under $not', { $not: { nope: 1 } }, 'nope'],
+        ['an unknown key beside an unknown operator', { nope: { $median: 1 } }, 'nope'],
+    ];
+
+    for (const [name, filter, key] of REFUSED) {
+        it(`${name}: 400 INVALID_FIELD at the aggregation's position, as the same key in where is`, async () => {
+            const err = await refusalFor(withFilter(filter));
+            expect(err.code).toBe('INVALID_FIELD');
+            expect(err.status).toBe(400);
+            expect(err.field).toBe(key);
+            expect(err.param).toBe('aggregations[1].filter');
+            // The twin: the same condition as the call's `where`.
+            const twin = await refusalFor({ where: filter, aggregations: [{ function: 'count', alias: 'n' }] });
+            expect(twin.code).toBe('INVALID_FIELD');
+            expect(twin.status).toBe(400);
+            expect(twin.field).toBe(key);
+        });
+    }
+
+    it('the gate\'s other two verdicts come with it — a dotted path and a virtual field, each as in where', async () => {
+        for (const [filter, field] of [[{ 'status.name': 'x' }, 'status.name'], [{ doubled: { $gt: 10 } }, 'doubled']] as const) {
+            const err = await refusalFor(withFilter(filter));
+            const twin = await refusalFor({ where: filter, aggregations: [{ function: 'count', alias: 'n' }] });
+            for (const e of [err, twin]) {
+                expect(e.code).toBe('INVALID_FIELD');
+                expect(e.status).toBe(400);
+                expect(e.field).toBe(field);
+            }
+            expect(err.param).toBe('aggregations[1].filter');
+        }
+    });
+
+    it('an unknown aggregated field keeps its own verdict ahead of a filter key', async () => {
+        const err = await refusalFor({
+            aggregations: [{ function: 'sum', field: 'amout', alias: 't', filter: { nope: 1 } }],
+        });
+        expect(err.code).toBe('INVALID_FIELD');
+        expect(err.param).toBe('aggregations');
+        expect(err.field).toBe('amout');
+    });
+
+    const ADMITTED: ReadonlyArray<readonly [string, unknown]> = [
+        ['declared keys, composed', { $or: [{ amount: { $gt: 100 } }, { status: 'paid' }] }],
+        ['the columns every row carries', { id: 'i1', created_at: { $ne: null }, updated_at: { $ne: null } }],
+        ['a relation head in the nested-object form', { owner_id: { $ne: null } }],
+        ['a filter that is not an object (left to the engine\'s shape gate)', 'status = paid'],
+        ['an array filter (left to the engine\'s shape gate)', [['amount', '>', 1]]],
+    ];
+
+    for (const [name, filter] of ADMITTED) {
+        it(`${name}: reaches engine.aggregate with the filter untouched`, async () => {
+            const { p, aggregate } = makeProtocol();
+            await p.findData({ object: 'invoice', query: withFilter(filter) } as never);
+            expect(aggregate).toHaveBeenCalledTimes(1);
+            const opts = (aggregate.mock.calls[0] as unknown[])[1] as { aggregations: Array<{ filter?: unknown }> };
+            expect(opts.aggregations[1].filter).toEqual(filter);
+        });
+    }
+});