Skip to content

Commit 9e5c309

Browse files
committed
test(objectql,rest,core,driver-sql,driver-memory): pin the card's rows, the family and the temporal conformance kit through a per-aggregation filter and having; changesets
Claude-Session: https://claude.ai/code/session_01Bvd69VPa6puiNzzPUroDBx Co-authored-by: Claude <noreply@anthropic.com>
1 parent 8e1c0cf commit 9e5c309

7 files changed

Lines changed: 807 additions & 1 deletion

‎.changeset/20148-aggregation-filter-where-doors.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,6 @@ These refusals run after the walker's own (an unknown operator, a malformed refe
2424

2525
Not refused, but answering differently:
2626

27-
- **A `Date` bound is compared as an instant.** `{ opened_at: { $gt: new Date('2026-02-01') } }` against the ISO text a `datetime` column holds counted no row: JS compared the `Date` with the string by coercion. `$ne` and `$nin` counted every row, and a `$between` of two `Date`s counted every row. The same bound in a `where` counted 4 of 6 on both drivers. The comparison now reads the pair through `@objectstack/spec/data`'s `utcInstantMs`, the lift `@objectstack/formula`'s evaluator applies, whenever one side is a `Date` and both sides denote an instant. Every `datetime` row and a UTC-midnight `Date` on a `date` field now count what the same bound counts in a `where`. `having` shares this comparison, so a `Date` bound in `having` keeps the groups its ISO spelling keeps on a `datetime` column; on a `date`-class column (`min` / `max` of a `date` field) a UTC-midnight `Date` follows the calendar-day reading a `where` gives (`{ $gte: new Date('2026-02-01') }` keeps the group whose max is `2026-02-01`, and so does `$eq`), which the ISO-instant text, compared as text, did not. A `Date` is not JSON, so this reaches in-process callers only. Two readings still differ from a `where`, because the per-row comparison holds no declaration: a `Date` carrying a time of day against a `date` field, which a `where` reads as that UTC calendar day, and a `Date` against a `time` field, which is not an instant and is left as before.
27+
- **A `Date` bound is compared as an instant.** `{ opened_at: { $gt: new Date('2026-02-01') } }` against the ISO text a `datetime` column holds counted no row: JS compared the `Date` with the string by coercion. `$ne` and `$nin` counted every row, and a `$between` of two `Date`s counted every row. The same bound in a `where` counted 4 of 6 on both drivers. The comparison now reads the pair through `@objectstack/spec/data`'s `utcInstantMs`, the lift `@objectstack/formula`'s evaluator applies, whenever one side is a `Date` and both sides denote an instant. Every `datetime` row and a UTC-midnight `Date` on a `date` field now count what the same bound counts in a `where`. `having` shares this comparison, so a `Date` bound in `having` keeps the groups its ISO spelling keeps on a `datetime` column; on a `date`-class column (`min` / `max` of a `date` field) a UTC-midnight `Date` follows the calendar-day reading a `where` gives (`{ $gte: new Date('2026-02-01') }` keeps the group whose max is `2026-02-01`, and so does `$eq`), which the ISO-instant text, compared as text, did not. A `Date` is not JSON, so this reaches in-process callers only. Two readings still differed from a `where` in this change alone, because the per-row comparison held no declaration (#20176 closes both in the same release, reading every temporal comparand at this position by the column's storage rule): a `Date` carrying a time of day against a `date` field, which a `where` reads as that UTC calendar day, and a `Date` against a `time` field, which is not an instant and is left as before.
2828

2929
Not changed, measured identical before and after on both drivers: every `where`, `groupBy` and `having` answer outside the `Date` shapes above, and a per-aggregation filter that uses implicit equality, ordering bounds, `$in`, `$between`, `$or`, `{}`, a `{ $field }` between two declared numeric fields, `addDays` between two `date` or two `datetime` fields (literal or read from a numeric column), and a reference to `created_at` or `id`.
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
"@objectstack/core": minor
3+
"@objectstack/objectql": minor
4+
"@objectstack/driver-sql": patch
5+
"@objectstack/driver-memory": patch
6+
---
7+
8+
fix(objectql,core): a per-aggregation `filter` and `having` on `engine.aggregate` read a temporal comparand by the column's storage rule, the rule `where` already applies — one function, `temporalStorageForm`, now exported by `@objectstack/core` and shared by both drivers (#20176)
9+
10+
A per-aggregation `filter` (`aggregations[i].filter`) and `having` are evaluated by the engine itself, over the rows (or aggregated rows) a driver returns. Both compared a temporal comparand exactly as written, while the same condition as a `where` is put into the column's storage form by the driver first. So they counted differently. Measured through `engine.aggregate` and through `POST /data/:object/query`, on `driver-memory` and `driver-sql`, over six rows:
11+
12+
| in `aggregations[i].filter` (or `having`) | before | now, and the `where` twin |
13+
|:--|:--|:--|
14+
| an ISO instant on a `date` field, `{ placed_on: { $gte: '2026-02-01T00:00:00.000Z' } }` | 1 | 3 |
15+
| the same instant under `$eq` | 0 | 2 |
16+
| a bare day as the upper bound of a `datetime`, `{ opened_at: { $lte: '2026-02-01' } }`, or as a `$between` max | 2 | 3 |
17+
| an epoch-millisecond bound on a `datetime` | 0 | 3 |
18+
| a `Date` carrying a time of day on a `date` field, `$gte` / `$lt` / `$eq` (in-process only) | 1 / 5 / 0 | 3 / 3 / 2 |
19+
| a `Date` on a `time` field (in-process only) | 0 | 3 |
20+
| `having` on `max` of a `date` field with an ISO-instant bound | kept one group | keeps the two groups whose day is on or after it |
21+
22+
The same holds for `$ne`, `$in` / `$nin` members, `$between` endpoints, implicit equality, an offset instant (`'…T18:00:00+08:00'`), an epoch-millisecond string, a zone-naive `'2026-02-01T10:00'`, and a short wall clock (`'11:00'`) or an ISO instant on a `time` field. On a `having` column, the class comes from the query, as the `addDays` rule already reads it: `min` / `max` take the class of the field they read, a `groupBy` projection takes its field's, and a `day` date bucket is a `date`.
23+
24+
What the rule does, now in one place:
25+
26+
- A comparand, and the row's value, are put into the column's storage form: canonical UTC ISO text for `datetime`, `YYYY-MM-DD` for `date`, and `HH:MM:SS` (`.fff` only when non-zero) for `time`.
27+
- A bare `YYYY-MM-DD` used as the upper bound of a `datetime` (`$lte`, a `$between` max) means that whole day, as it does in a `where` (ADR-0053 D-D). On a `date` or `time` column it is not widened.
28+
- A value the rule cannot read is compared as written, and so is every non-temporal column, presence tests (`$exists`, `$null`), the text operators and a `{ $field }` reference.
29+
- An object whose declared fields the engine cannot see keeps the previous comparison.
30+
31+
`@objectstack/core` exports the rule as `temporalStorageForm(value, kind)`, `kind` being `'datetime' | 'date' | 'time'`. `driver-sql` (`canonicalUtcDatetime`, `toDateOnly`, `canonicalTimeOfDay`) and `driver-memory` (`coerceTemporalValue`) each carried a copy of it; both now call it. The copies agreed on every shape measured when they were lifted, so no `where`, write or read answer of either driver changes. MySQL still binds a `datetime` in its own literal spelling.
32+
33+
`@objectstack/objectql`'s `applyInMemoryAggregation(rows, ast, timezone?, fields?)` takes the object's declared field map as an optional fourth argument, and a per-aggregation `filter` reads a temporal comparand by the rule only when it is given. Called without it, the function answers as before.
34+
35+
Not changed, measured identical before and after on both drivers: every `where` answer, every refusal a per-aggregation `filter` or `having` gives, and every per-aggregation `filter` and `having` cell whose column is not temporal.
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
//
3+
// [#20176] `temporalStorageForm` — the ONE storage rule a temporal comparand is
4+
// put in before it is compared: by `driver-sql` and `driver-memory` on `where`
5+
// (and on write), and by `@objectstack/objectql` on a per-aggregation `filter`
6+
// and `having`. Each driver carried a word-for-word copy until it was lifted
7+
// here; the drivers' own suites pin that their `where` coercion IS this
8+
// function (`memory-temporal-storage-form.test.ts`,
9+
// `sql-driver-temporal-storage-form.test.ts`). This file pins the rule itself.
10+
11+
import { describe, it, expect } from 'vitest';
12+
import { temporalStorageForm } from './temporal-storage-form.js';
13+
14+
describe('temporalStorageForm — datetime: canonical UTC ISO text', () => {
15+
const cases: ReadonlyArray<readonly [string, unknown, unknown]> = [
16+
['a Date', new Date('2026-02-01T10:00:00.000Z'), '2026-02-01T10:00:00.000Z'],
17+
['epoch milliseconds', 1769940000000, '2026-02-01T10:00:00.000Z'],
18+
['an epoch-millisecond string, padded', ' 1769940000000 ', '2026-02-01T10:00:00.000Z'],
19+
['a negative epoch string', '-86400000', '1969-12-31T00:00:00.000Z'],
20+
['a bare day — midnight UTC, never the host zone', '2026-02-01', '2026-02-01T00:00:00.000Z'],
21+
['a zone-naive wall clock — read AS UTC', '2026-02-01 09:00', '2026-02-01T09:00:00.000Z'],
22+
['a zone-naive T spelling with a long fraction', '2026-02-01T09:00:00.123456', '2026-02-01T09:00:00.123Z'],
23+
['an ISO instant without milliseconds', '2026-02-01T10:00:00Z', '2026-02-01T10:00:00.000Z'],
24+
['an offset instant — re-spelled in Z', '2026-02-01T18:00:00+08:00', '2026-02-01T10:00:00.000Z'],
25+
];
26+
for (const [name, input, expected] of cases) {
27+
it(name, () => expect(temporalStorageForm(input, 'datetime')).toBe(expected));
28+
}
29+
});
30+
31+
describe('temporalStorageForm — date: the UTC calendar day', () => {
32+
const cases: ReadonlyArray<readonly [string, unknown, unknown]> = [
33+
['an ISO instant — its leading day', '2026-02-01T00:00:00.000Z', '2026-02-01'],
34+
['a Date carrying a time of day — its UTC day', new Date('2026-02-01T23:30:00.000Z'), '2026-02-01'],
35+
['a bare day, padded', ' 2026-02-01 ', '2026-02-01'],
36+
['a zone-naive timestamp', '2026-02-01 23:59', '2026-02-01'],
37+
];
38+
for (const [name, input, expected] of cases) {
39+
it(name, () => expect(temporalStorageForm(input, 'date')).toBe(expected));
40+
}
41+
42+
it('epoch milliseconds are not a calendar day — returned as they came', () => {
43+
expect(temporalStorageForm(1769940000000, 'date')).toBe(1769940000000);
44+
});
45+
});
46+
47+
describe('temporalStorageForm — time: the UTC wall clock, .fff only when non-zero', () => {
48+
const cases: ReadonlyArray<readonly [string, unknown, unknown]> = [
49+
['a short wall clock', '09:00', '09:00:00'],
50+
['a zero-millisecond suffix is trimmed', '09:00:00.000', '09:00:00'],
51+
['a fraction is kept to milliseconds', '09:00:00.5', '09:00:00.500'],
52+
['a Date — its UTC time of day', new Date('2026-02-01T11:00:00.000Z'), '11:00:00'],
53+
['an ISO instant — its UTC time of day', '2026-02-01T11:00:00.250Z', '11:00:00.250'],
54+
['a bare day — its midnight', '2026-02-01', '00:00:00'],
55+
];
56+
for (const [name, input, expected] of cases) {
57+
it(name, () => expect(temporalStorageForm(input, 'time')).toBe(expected));
58+
}
59+
});
60+
61+
describe('temporalStorageForm — total: what the rule cannot read comes back unchanged', () => {
62+
const invalid = new Date('nope');
63+
const untouched: ReadonlyArray<readonly [string, unknown]> = [
64+
['null', null],
65+
['undefined', undefined],
66+
['the empty string', ''],
67+
['whitespace', ' '],
68+
['junk', 'not-a-date'],
69+
['a preset name', 'last_30_days'],
70+
['a placeholder', '{today}'],
71+
['a boolean', true],
72+
['an Invalid Date', invalid],
73+
['NaN', Number.NaN],
74+
];
75+
for (const kind of ['datetime', 'date', 'time'] as const) {
76+
for (const [name, input] of untouched) {
77+
it(`${kind}: ${name}`, () => expect(temporalStorageForm(input, kind)).toBe(input));
78+
}
79+
}
80+
81+
it('time: an out-of-range wall clock is not wrapped', () => {
82+
expect(temporalStorageForm('25:00', 'time')).toBe('25:00');
83+
});
84+
85+
it('a list is NOT mapped — a caller comparing a list maps its members', () => {
86+
const list = ['2026-02-01', '2026-02-02'];
87+
expect(temporalStorageForm(list, 'datetime')).toBe(list);
88+
});
89+
});
Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#20176] This driver's `where` reads a temporal comparand by the ONE storage
5+
* rule — `@objectstack/core`'s `temporalStorageForm` — that `driver-sql` and the
6+
* engine's per-aggregation `filter` / `having` read too.
7+
*
8+
* Two halves:
9+
*
10+
* 1. `coerceTemporalValue` IS that rule, shape for shape, with a list mapped
11+
* member by member. This file used to carry its own copy
12+
* (`storageDatetimeValue` / `storageDateValue` / `storageTimeValue`); the
13+
* copy and `driver-sql`'s agreed on every shape measured when the rule was
14+
* lifted, so the re-pointing moved no answer.
15+
* 2. The `where` answers the engine's per-aggregation `filter` is now held to —
16+
* the #20176 card's rows on its fixture. `@objectstack/objectql`'s
17+
* `engine-aggregate-temporal-storage-rule.test.ts` asserts the same numbers
18+
* for the per-aggregation position; this is the control half, on this
19+
* driver (`driver-sql`'s twin is the REST suite
20+
* `aggregation-filter-temporal-storage-rule.test.ts`).
21+
*/
22+
23+
import { describe, it, expect, beforeAll } from 'vitest';
24+
import { temporalStorageForm } from '@objectstack/core';
25+
import { coerceTemporalValue } from './memory-temporal.js';
26+
import { InMemoryDriver } from './memory-driver.js';
27+
28+
const SHAPES: ReadonlyArray<readonly [string, () => unknown]> = [
29+
['Date', () => new Date('2026-02-01T10:00:00.000Z')],
30+
['Invalid Date', () => new Date('nope')],
31+
['epoch ms', () => 1769940000000],
32+
['epoch ms string', () => '1769940000000'],
33+
['bare day', () => '2026-02-01'],
34+
['zone-naive', () => '2026-02-01 09:00'],
35+
['ISO instant', () => '2026-02-01T10:00:00Z'],
36+
['offset instant', () => '2026-02-01T18:00:00+08:00'],
37+
['wall clock', () => '09:00'],
38+
['wall clock .5', () => '09:00:00.5'],
39+
['out-of-range wall clock', () => '25:00'],
40+
['junk', () => 'not-a-date'],
41+
['empty', () => ''],
42+
['null', () => null],
43+
['boolean', () => true],
44+
];
45+
46+
describe('[#20176] coerceTemporalValue is core\'s temporalStorageForm', () => {
47+
for (const kind of ['datetime', 'date', 'time'] as const) {
48+
for (const [name, make] of SHAPES) {
49+
it(`${kind}: ${name}`, () => {
50+
expect(coerceTemporalValue(make(), kind)).toStrictEqual(temporalStorageForm(make(), kind));
51+
});
52+
}
53+
it(`${kind}: a list comparand is mapped member by member`, () => {
54+
const list = SHAPES.map(([, make]) => make());
55+
expect(coerceTemporalValue(list, kind)).toStrictEqual(list.map((v) => temporalStorageForm(v, kind)));
56+
});
57+
}
58+
59+
it('a non-temporal field passes through untouched', () => {
60+
const value = '2026-02-01T10:00:00Z';
61+
expect(coerceTemporalValue(value, undefined)).toBe(value);
62+
});
63+
});
64+
65+
const OBJECT = 'ledger_order';
66+
const ROWS = [
67+
{ id: 'o1', placed_on: '2026-01-10', opened_at: '2026-01-01T10:00:00.000Z', slot: '09:00:00' },
68+
{ id: 'o2', placed_on: '2026-01-02', opened_at: '2026-01-02T10:00:00.000Z', slot: '10:30:00' },
69+
{ id: 'o3', placed_on: '2026-03-01', opened_at: '2026-02-01T10:00:00.000Z', slot: '11:00:00' },
70+
{ id: 'o4', placed_on: '2026-02-01', opened_at: '2026-02-05T10:00:00.000Z', slot: '12:00:00' },
71+
{ id: 'o5', placed_on: '2026-01-15', opened_at: '2026-02-06T10:00:00.000Z', slot: '13:00:00' },
72+
{ id: 'o6', placed_on: '2026-02-01', opened_at: '2026-03-01T10:00:00.000Z', slot: '14:00:00' },
73+
];
74+
75+
/** The #20176 card's rows, as a `where`, with the count this driver answers. */
76+
const CARD_WHERE_TWINS: ReadonlyArray<readonly [string, () => Record<string, unknown>, number]> = [
77+
['row 1 — an ISO instant $gte on a date field', () => ({ placed_on: { $gte: '2026-02-01T00:00:00.000Z' } }), 3],
78+
['row 2 — an ISO instant $eq on a date field', () => ({ placed_on: { $eq: '2026-02-01T00:00:00.000Z' } }), 2],
79+
['row 3 — a bare day as the $lte of a datetime', () => ({ opened_at: { $lte: '2026-02-01' } }), 3],
80+
['row 4 — a bare day as the $between max of a datetime', () => ({ opened_at: { $between: ['2026-01-01', '2026-02-01'] } }), 3],
81+
['row 5 — an epoch-ms bound on a datetime', () => ({ opened_at: { $gt: 1769940000000 } }), 3],
82+
['row 6 — a Date with a time of day, $gte on a date field', () => ({ placed_on: { $gte: new Date('2026-02-01T10:00:00.000Z') } }), 3],
83+
['row 6 — the same Date, $lt', () => ({ placed_on: { $lt: new Date('2026-02-01T10:00:00.000Z') } }), 3],
84+
['row 6 — the same Date, $eq', () => ({ placed_on: { $eq: new Date('2026-02-01T10:00:00.000Z') } }), 2],
85+
['row 7 — a Date on a time field', () => ({ slot: { $gt: new Date('2026-02-01T11:00:00.000Z') } }), 3],
86+
['control — a zone-naive datetime string', () => ({ opened_at: { $gt: '2026-02-01 09:00' } }), 4],
87+
];
88+
89+
describe('[#20176] the where twins of the card\'s rows, on this driver', () => {
90+
let driver: InMemoryDriver;
91+
92+
beforeAll(async () => {
93+
driver = new InMemoryDriver({});
94+
await driver.connect();
95+
await driver.syncSchema(OBJECT, {
96+
name: OBJECT,
97+
fields: { placed_on: { type: 'date' }, opened_at: { type: 'datetime' }, slot: { type: 'time' } },
98+
});
99+
for (const row of ROWS) await driver.create(OBJECT, row);
100+
});
101+
102+
for (const [name, where, expected] of CARD_WHERE_TWINS) {
103+
it(`${name}: ${expected} of 6`, async () => {
104+
const rows = await driver.find(OBJECT, { where: where() } as any);
105+
expect(rows).toHaveLength(expected);
106+
});
107+
}
108+
});

0 commit comments

Comments
 (0)