Skip to content

Commit 6936d07

Browse files
Elon Muskclaude
andauthored
feat(objectql,spec): per-aggregation filter on engine.aggregate — drivers refuse instead of silently dropping (#10413 contract half) (#10771)
* feat(objectql,spec): honour per-aggregation filter on engine.aggregate; drivers refuse instead of dropping Fixes the contract half of the ObjectQL analytics silent-drop: aggregation entries gain an enforced per-aggregation filter (SQL FILTER (WHERE ...) semantics), lowered in memory by the engine for every driver; the four native driver aggregate faces refuse a directly-delivered filter with NOT_IMPLEMENTED/501; the StrategyContext.executeAggregate bridge carries the new key for #10413 phase 2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B4h3medzvhB9rpfoja9jcw * test(objectql): type the aggregate options in the new filter tests (TEST_DEBT ratchet stays at 355) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B4h3medzvhB9rpfoja9jcw --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent e072c97 commit 6936d07

20 files changed

Lines changed: 1055 additions & 25 deletions
Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
"@objectstack/spec": minor
3+
"@objectstack/objectql": minor
4+
"@objectstack/driver-sql": patch
5+
"@objectstack/driver-turso": patch
6+
"@objectstack/driver-mongodb": patch
7+
"@objectstack/driver-memory": patch
8+
---
9+
10+
`engine.aggregate` honours a per-aggregation `filter` (#10576, the contract
11+
half of #10413). `AggregationNodeSchema.filter` — declared since #4286 but
12+
marked experimental and enforced by nothing — is now live with SQL
13+
`FILTER (WHERE …)` semantics: the predicate narrows the SOURCE rows that one
14+
aggregation reads while sibling aggregations in the same call keep seeing
15+
every row of the group, so a measure-scoped filter (`stage: 'closed_won'`)
16+
can finally reach the engine instead of being silently dropped (the #10413
17+
wrong-numbers defect on the ObjectQL analytics path). The
18+
`StrategyContext.executeAggregate` bridge (`@objectstack/spec/contracts`)
19+
gains the same optional `filter` on its aggregation entries so analytics
20+
strategies can lower measure filters into it (#10413 phase 2 consumes this
21+
seam next).
22+
23+
Execution is the correct-first two-tier shape date bucketing and HAVING use:
24+
the engine lowers filtered aggregations in memory for every driver (unknown
25+
operators refuse loudly with `INVALID_FILTER`/400 naming the aggregation
26+
position; a group emptied by its filter answers the ruled empty-group values
27+
— count/sum 0, avg/min/max null). No driver compiles conditional aggregation
28+
natively today, so each native aggregate face (driver-sql — inherited by
29+
driver-sqlite-wasm and Turso local —, the Turso remote transport,
30+
driver-mongodb's pipeline builder, driver-memory's `performAggregation`)
31+
refuses a directly-delivered per-aggregation filter with
32+
`NOT_IMPLEMENTED`/501 instead of silently aggregating the unfiltered rows.
33+
Aggregations without a `filter` are byte-identically unchanged, including
34+
their native pushdown path.

‎content/docs/references/data/query.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ const result = AggregationFunction.parse(data);
4949
| **field** | `string` | optional | Field to aggregate (optional for COUNT(*)) |
5050
| **alias** | `string` | ✅ | Result column alias |
5151
| **distinct** | `never` | optional | [REMOVED] `query.aggregations[].distinct` was removed in @objectstack/spec 17 (#6815, ADR-0049) — exactly ONE of the six faces that read an aggregation honoured it. The objectql in-memory fallback deduplicated the values before applying the function, while `driver-sql`, `driver-turso`, `driver-mongodb`, `driver-memory` and the service-analytics SQL builder all ignored it — so `{ function: 'sum', field: 'amount', distinct: true }` answered a DEDUPLICATED sum when the engine fell back in memory and an ordinary sum on every SQL datasource: one query, two numbers, chosen by which backend happened to serve it. Both answers are plausible, so nothing surfaced the divergence. Delete the key. For a deduplicated COUNT the live spelling is the `count_distinct` aggregation function, which every SQL face compiles to `COUNT(DISTINCT field)` (#6409) and the in-memory fallback computes identically. `SUM(DISTINCT …)` / `AVG(DISTINCT …)` get no replacement: no backend ever computed them here, and a per-row measure that needs deduplicating is a modelling problem to fix in the data, not a flag on the read. |
52-
| **filter** | `any` | optional | [EXPERIMENTAL — not enforced] Per-aggregation filter (SQL FILTER (WHERE …)). Neither the SQL builders nor the in-memory fallback applies it (#4286); filter the whole query with `where` instead. |
52+
| **filter** | `any` | optional | Per-aggregation filter (SQL FILTER (WHERE …) semantics): narrows the source rows THIS aggregation reads, leaving sibling aggregations unfiltered. Enforced by engine.aggregate (#10576): lowered in memory for drivers without native conditional aggregation; a driver reached directly refuses rather than silently dropping it. |
5353

5454

5555
---

‎packages/drivers/driver-memory/src/filter-refusal.ts‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,36 @@ export function unsupportedFilterError(message: string): Error {
5454
return err;
5555
}
5656

57+
/**
58+
* [#10576] An aggregation entry carries a per-aggregation `filter`
59+
* (`AggregationNodeSchema.filter`, the contract half of #10413) — the twin of
60+
* `driver-sql`'s `unsupportedAggregationFilterError`, first sentence for first
61+
* sentence, and the same NOT_IMPLEMENTED/501 class for the same reason (#5907,
62+
* ADR-0112): the spec declares the key, this driver's `performAggregation`
63+
* evaluates no per-aggregation predicate, so it is a capability gap in the
64+
* backend rather than a mistake in the query. Building the evaluation here is
65+
* a capability investment this refusal deliberately is not (#5499 freeze).
66+
* Refused rather than silently aggregating the UNFILTERED rows — the #10413
67+
* defect. Unreachable through `engine.aggregate` (the engine lowers filtered
68+
* aggregations in memory for every driver); this fires only for a caller that
69+
* reaches the driver's own aggregation faces directly (`find()` with
70+
* aggregations, or `aggregate(AST)`).
71+
*/
72+
export function refusePerAggregationFilter(alias: string): never {
73+
const err = new Error(
74+
`Per-aggregation \`filter\` on "${alias}" is not supported by this backend (driver-memory). ` +
75+
`The query is spelled correctly and @objectstack/spec AggregationNodeSchema declares the key — ` +
76+
`this backend compiles no conditional-aggregate (SQL FILTER (WHERE …) / CASE WHEN) expression ` +
77+
`for it, so it is refused rather than silently aggregating the UNFILTERED rows (#10413), which ` +
78+
`is why it answers NOT_IMPLEMENTED/501 rather than a 400. \`engine.aggregate\` lowers filtered ` +
79+
`aggregations in memory for every driver without native support — route the query through the ` +
80+
`engine, or drop the \`filter\` key.`,
81+
) as Error & { code?: string; status?: number };
82+
err.code = StandardErrorCode.enum.NOT_IMPLEMENTED;
83+
err.status = 501;
84+
throw err;
85+
}
86+
5787
/**
5888
* [#5158] A `FilterArray` reached the driver unlowered.
5989
*
Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#10576] A per-aggregation `filter` (`AggregationNodeSchema.filter`, the
5+
* contract half of #10413) reaching this driver's own aggregation faces is
6+
* refused with the ADR-0112 envelope — never silently answered with the
7+
* UNFILTERED aggregate, which is the #10413 defect at the driver seam.
8+
*
9+
* This driver has TWO doors into `performAggregation` — `aggregate(AST)` (the
10+
* one objectql's engine uses) and `find()` with aggregations — so both are
11+
* pinned: a guard on one door alone re-opens the drop through the other. The
12+
* engine itself never pushes a filtered aggregation down (it lowers in
13+
* memory); the refusal exists for direct callers. Evaluating the predicate
14+
* here instead would be a capability build-out, which the #5499 family freeze
15+
* rules out — the refusal path is the sanctioned scope.
16+
*
17+
* Every case asserts `code` AND `status`, never merely "it threw" (#6144).
18+
*/
19+
20+
import { describe, it, expect, beforeEach } from 'vitest';
21+
import type { DriverQuery } from '@objectstack/spec/contracts';
22+
import { InMemoryDriver } from './memory-driver.js';
23+
24+
interface WireBearingError extends Error {
25+
code?: string;
26+
status?: number;
27+
}
28+
29+
const TABLE = 'deal';
30+
31+
describe('[#10576] InMemoryDriver refuses a per-aggregation filter it does not evaluate', () => {
32+
let driver: InMemoryDriver;
33+
34+
beforeEach(async () => {
35+
driver = new InMemoryDriver();
36+
await driver.connect();
37+
await driver.create(TABLE, { id: '1', stage: 'closed_won', amount: 500 });
38+
await driver.create(TABLE, { id: '2', stage: 'open', amount: 900 });
39+
});
40+
41+
const filteredQuery = (): DriverQuery => ({
42+
aggregations: [{ function: 'count', alias: 'won_count', filter: { stage: 'closed_won' } }],
43+
}) as unknown as DriverQuery;
44+
45+
it('the aggregate(AST) door answers NOT_IMPLEMENTED / 501, naming the aggregation and the engine lowering', async () => {
46+
let thrown: WireBearingError | undefined;
47+
try {
48+
await driver.aggregate(TABLE, filteredQuery());
49+
} catch (e) {
50+
thrown = e as WireBearingError;
51+
}
52+
expect(thrown, 'a filter this face does not evaluate must not be silently dropped').toBeDefined();
53+
expect(thrown!.code).toBe('NOT_IMPLEMENTED');
54+
expect(thrown!.status).toBe(501);
55+
expect(thrown!.message).toContain(
56+
'Per-aggregation `filter` on "won_count" is not supported by this backend (driver-memory).',
57+
);
58+
expect(thrown!.message).toContain('`engine.aggregate` lowers filtered aggregations in memory');
59+
});
60+
61+
it('the find()-with-aggregations door refuses identically — one guard covers both doors', async () => {
62+
let thrown: WireBearingError | undefined;
63+
try {
64+
await driver.find(TABLE, filteredQuery());
65+
} catch (e) {
66+
thrown = e as WireBearingError;
67+
}
68+
expect(thrown).toBeDefined();
69+
expect(thrown!.code).toBe('NOT_IMPLEMENTED');
70+
expect(thrown!.status).toBe(501);
71+
expect(thrown!.message).toContain('Per-aggregation `filter` on "won_count"');
72+
});
73+
74+
it('control: the same aggregation WITHOUT a filter still computes through both doors', async () => {
75+
const bare = { aggregations: [{ function: 'count', alias: 'n' }] } as unknown as DriverQuery;
76+
await expect(driver.aggregate(TABLE, bare)).resolves.toEqual([{ n: 2 }]);
77+
await expect(driver.find(TABLE, bare)).resolves.toEqual([{ n: 2 }]);
78+
});
79+
80+
it('control: an EMPTY filter object is vacuous (the where/having convention) and computes', async () => {
81+
const vacuous = {
82+
aggregations: [{ function: 'sum', field: 'amount', alias: 'total', filter: {} }],
83+
} as unknown as DriverQuery;
84+
await expect(driver.aggregate(TABLE, vacuous)).resolves.toEqual([{ total: 1400 }]);
85+
});
86+
});

‎packages/drivers/driver-memory/src/memory-driver.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ import {
2828
// [#7536] The `$like`/`$ilike` comparand refusals, beside their siblings.
2929
likePatternComparandError,
3030
danglingLikeEscapeError,
31+
// [#10576] The per-aggregation `filter` refusal — this driver evaluates none.
32+
refusePerAggregationFilter,
3133
} from './filter-refusal.js';
3234
import {
3335
coerceTemporalValue,
@@ -1159,6 +1161,19 @@ export class InMemoryDriver implements IDataDriver {
11591161

11601162
private performAggregation(records: any[], query: DriverQuery): any[] {
11611163
const { groupBy, aggregations } = query;
1164+
// [#10576] A per-aggregation `filter` this face does not evaluate is
1165+
// refused before any group is built — silently answering the UNFILTERED
1166+
// aggregate is the #10413 defect. Guarded HERE because both of this
1167+
// driver's aggregation doors (`find()` with aggregations and
1168+
// `aggregate(AST)`) funnel through this method. `{}` is the vacuous
1169+
// filter, same convention as `where` / `having`. See
1170+
// {@link refusePerAggregationFilter}.
1171+
for (const agg of aggregations ?? []) {
1172+
const f = (agg as { filter?: unknown }).filter;
1173+
if (f && typeof f === 'object' && Object.keys(f).length > 0) {
1174+
refusePerAggregationFilter((agg as any).alias ?? (agg as any).field ?? '(unaliased)');
1175+
}
1176+
}
11621177
const groups: Map<string, any[]> = new Map();
11631178

11641179
const normalizeGroupBy = (node: any): { field: string; alias: string } => {

‎packages/drivers/driver-mongodb/src/mongodb-aggregation-translation.test.ts‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -330,3 +330,40 @@ describe('the in-process pipeline evaluator discriminates', () => {
330330
expect(counted).toBe(1);
331331
});
332332
});
333+
334+
/**
335+
* [#10576] A per-aggregation `filter` (`AggregationNodeSchema.filter`, the
336+
* contract half of #10413) reaching this builder is refused with the ADR-0112
337+
* envelope — never silently accumulated over the UNFILTERED rows, which is the
338+
* #10413 defect at the driver seam. `AggregationInput.filter` has declared the
339+
* key since #6850 and nothing ever read it; a declared key a builder ignores
340+
* is exactly the silent drop this card closes. The engine never pushes a
341+
* filtered aggregation down (it lowers in memory) — the refusal exists for the
342+
* direct caller of this exported builder.
343+
*/
344+
describe('[#10576] per-aggregation filter refuses instead of silently dropping', () => {
345+
it('answers NOT_IMPLEMENTED / 501, naming the aggregation and the engine lowering', () => {
346+
let thrown: (Error & { code?: string; status?: number }) | undefined;
347+
try {
348+
buildAggregationPipeline({
349+
aggregations: [{ function: 'count', alias: 'won_count', filter: { stage: 'closed_won' } }],
350+
});
351+
} catch (err) {
352+
thrown = err as Error & { code?: string; status?: number };
353+
}
354+
expect(thrown, 'a filter this builder cannot lower must not be silently dropped').toBeDefined();
355+
expect(thrown!.code).toBe('NOT_IMPLEMENTED');
356+
expect(thrown!.status).toBe(501);
357+
expect(thrown!.message).toContain(
358+
'Per-aggregation `filter` on "won_count" is not supported by this backend (driver-mongodb).',
359+
);
360+
expect(thrown!.message).toContain('`engine.aggregate` lowers filtered aggregations in memory');
361+
});
362+
363+
it('control: an EMPTY filter object is vacuous (the where/having convention) and still lowers', () => {
364+
const pipeline = buildAggregationPipeline({
365+
aggregations: [{ function: 'count', alias: 'n', filter: {} }],
366+
});
367+
expect(pipeline[0]).toHaveProperty('$group');
368+
});
369+
});

‎packages/drivers/driver-mongodb/src/mongodb-aggregation.ts‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -303,6 +303,33 @@ function refuseDateBucketedGroupBy(granularity: string): never {
303303
throw err;
304304
}
305305

306+
/**
307+
* [#10576] An aggregation entry carries a per-aggregation `filter`
308+
* (`AggregationNodeSchema.filter`, the contract half of #10413) — the twin of
309+
* `driver-sql`'s `unsupportedAggregationFilterError`, first sentence for first
310+
* sentence, and the same NOT_IMPLEMENTED/501 class for the same reason (#5907,
311+
* ADR-0112): the spec declares the key, this builder emits no conditional
312+
* accumulator for it, so it is a capability gap in the backend rather than a
313+
* mistake in the query. Refused rather than silently accumulating the
314+
* UNFILTERED rows — the #10413 defect. Unreachable through `engine.aggregate`
315+
* (the engine lowers filtered aggregations in memory for every driver); this
316+
* fires only for a caller that drives the builder or driver directly.
317+
*/
318+
function refusePerAggregationFilter(alias: string): never {
319+
const err = new Error(
320+
`Per-aggregation \`filter\` on "${alias}" is not supported by this backend (driver-mongodb). ` +
321+
`The query is spelled correctly and @objectstack/spec AggregationNodeSchema declares the key — ` +
322+
`this backend compiles no conditional-aggregate (SQL FILTER (WHERE …) / CASE WHEN) expression ` +
323+
`for it, so it is refused rather than silently aggregating the UNFILTERED rows (#10413), which ` +
324+
`is why it answers NOT_IMPLEMENTED/501 rather than a 400. \`engine.aggregate\` lowers filtered ` +
325+
`aggregations in memory for every driver without native support — route the query through the ` +
326+
`engine, or drop the \`filter\` key.`,
327+
) as Error & { code?: string; status?: number };
328+
err.code = StandardErrorCode.enum.NOT_IMPLEMENTED;
329+
err.status = 501;
330+
throw err;
331+
}
332+
306333
/**
307334
* [#6850] A `groupBy` entry that is neither half of the declared union.
308335
*
@@ -386,6 +413,17 @@ export function buildAggregationPipeline(opts: {
386413

387414
// Build accumulators from aggregation descriptors
388415
for (const agg of opts.aggregations) {
416+
// [#10576] A per-aggregation `filter` (`AggregationNodeSchema.filter`,
417+
// the contract half of #10413) has no lowering in this builder — a
418+
// `$cond`-wrapped accumulator would be one, but building it is a
419+
// capability investment this refusal deliberately is not. Refused before
420+
// a pipeline exists rather than silently accumulating the UNFILTERED
421+
// rows (the #10413 defect). Unreachable through `engine.aggregate`,
422+
// which lowers filtered aggregations in memory for every driver; `{}` is
423+
// the vacuous filter, same convention as `where` / `having`.
424+
if (agg.filter && typeof agg.filter === 'object' && Object.keys(agg.filter).length > 0) {
425+
refusePerAggregationFilter(agg.alias ?? agg.field ?? '(unaliased)');
426+
}
389427
groupAccumulators[agg.alias] = buildAccumulator(agg);
390428
}
391429

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#10576] A per-aggregation `filter` (`AggregationNodeSchema.filter`, the
5+
* contract half of #10413) reaching this compiler DIRECTLY is refused with a
6+
* wire identity — never silently dropped.
7+
*
8+
* Before #10576 this driver's `aggregate()` never read `agg.filter`: the
9+
* statement it built aggregated EVERY row and reported success, which is the
10+
* #10413 defect ("won deals" counting every opportunity) at the driver seam.
11+
* The engine now lowers filtered aggregations in memory and never pushes one
12+
* down here — so the only caller that can arrive with the key is a direct
13+
* one, and the honest answers are exactly two: compile a conditional
14+
* aggregate, or refuse loudly. This backend refuses (NOT_IMPLEMENTED/501, the
15+
* #5907 class for "declared by the spec, not compiled by this face").
16+
*
17+
* Every case asserts `code` AND `status`, never merely "it threw" (#6144): a
18+
* bare-throw assertion is green before and after the envelope exists.
19+
*/
20+
21+
import { describe, it, expect, beforeEach } from 'vitest';
22+
import { SqlDriver } from './index.js';
23+
import type { QueryAST } from '@objectstack/spec/data';
24+
25+
interface WireBearingError extends Error {
26+
code?: string;
27+
status?: number;
28+
}
29+
30+
describe('[#10576] SqlDriver refuses a per-aggregation filter it cannot compile', () => {
31+
let driver: SqlDriver;
32+
33+
beforeEach(async () => {
34+
driver = new SqlDriver({
35+
client: 'better-sqlite3',
36+
connection: { filename: ':memory:' },
37+
useNullAsDefault: true,
38+
});
39+
await driver.initObjects([
40+
{
41+
name: 'deal',
42+
fields: {
43+
id: { type: 'text', name: 'id' },
44+
stage: { type: 'text', name: 'stage' },
45+
amount: { type: 'number', name: 'amount' },
46+
},
47+
} as any,
48+
]);
49+
await driver.create('deal', { id: '1', stage: 'closed_won', amount: 500 });
50+
await driver.create('deal', { id: '2', stage: 'open', amount: 900 });
51+
});
52+
53+
it('refuses NOT_IMPLEMENTED/501, naming the aggregation and the engine lowering', async () => {
54+
const ast = {
55+
object: 'deal',
56+
aggregations: [
57+
{ function: 'count', alias: 'won_count', filter: { stage: 'closed_won' } },
58+
],
59+
} as unknown as QueryAST;
60+
61+
let thrown: WireBearingError | undefined;
62+
try {
63+
await driver.aggregate('deal', ast);
64+
} catch (e) {
65+
thrown = e as WireBearingError;
66+
}
67+
68+
expect(thrown).toBeDefined();
69+
expect(thrown!.code).toBe('NOT_IMPLEMENTED');
70+
expect(thrown!.status).toBe(501);
71+
expect(thrown!.message).toContain(
72+
'Per-aggregation `filter` on "won_count" is not supported by this backend (driver-sql).',
73+
);
74+
// The remedy is named: the engine's in-memory lowering serves this query.
75+
expect(thrown!.message).toContain('`engine.aggregate` lowers filtered aggregations in memory');
76+
});
77+
78+
it('control: the same aggregation WITHOUT a filter still computes (only the refusal was added)', async () => {
79+
const rows = await driver.aggregate('deal', {
80+
object: 'deal',
81+
aggregations: [{ function: 'count', alias: 'n' }],
82+
} as unknown as QueryAST);
83+
expect(rows).toEqual([{ n: 2 }]);
84+
});
85+
86+
it('control: an EMPTY filter object is vacuous (the where/having convention) and computes', async () => {
87+
const rows = await driver.aggregate('deal', {
88+
object: 'deal',
89+
aggregations: [{ function: 'sum', field: 'amount', alias: 'total', filter: {} }],
90+
} as unknown as QueryAST);
91+
expect(rows).toEqual([{ total: 1400 }]);
92+
});
93+
});

0 commit comments

Comments
 (0)