Skip to content

Commit 8538edf

Browse files
fix(objectql): the rows path adds sum / avg with compensated summation, as SQLite does (#20543)
Fixes #20489 Clause-②: no ## What changed The engine's rows path (`packages/objectql/src/in-memory-aggregation.ts`, `applyInMemoryAggregation`) now adds `sum` / `avg` with Kahan-Babuska-Neumaier compensation. That is the direction triage 5875711059 ruled: fix the one face the platform owns, so the rows path agrees with SQLite native on both SQLite paths and gives the more accurate answer. - One helper, `compensatedSum`, is a transcription of SQLite's `kahanBabuskaNeumaierStep` plus its finalizers' overflow guard: when the error term is non-finite, the plain running sum is returned. The `sum` arm and the `avg` arm both call it. Before, each arm had its own naive `reduce`. - `toNumber`, the null / non-numeric handling, the empty group (`sum` `0`, `avg` `null`) and the answer's type (a JS number) are untouched. - Out of scope, as the claim set it: driver-sql's native aggregate, PostgreSQL / MySQL arithmetic, driver-memory, and `packages/spec`. Head measured: `8ad4a9d4a`. It is a true merge of `origin/main` at `03b19d9cf`, on top of the two commits of this branch. ## H1: the defect, before and after Base `b2b6a0643`, SQLite 3.53.4 (better-sqlite3), `driver-sql`. One `number` column. The rows path is forced by a filtered sibling `count`. Both `engine.aggregate` and `POST /api/v1/data/:object/query` were read, and they gave the same values in every cell: | group | native `sum` / `avg` | rows path, base | rows path, `8ad4a9d4a` | |:--|:--|:--|:--| | `0.1, 0.2, 0.3` | `0.6` / `0.19999999999999998` | `0.6000000000000001` / `0.20000000000000004` | `0.6` / `0.19999999999999998` | | `1e16, 1, -1e16` | `1` / `0.3333333333333333` | `0` / `0` | `1` / `0.3333333333333333` | | `1e16, 0.5, -1e16` | `0.5` / `0.16666666666666666` | `0` / `0` | `0.5` / `0.16666666666666666` | | `0.1, 0.2` | `0.30000000000000004` / `0.15000000000000002` | same | same | | `1, 2, 3, 40, 500` | `546` / `109.2` | same | same | `having { s: { $eq: 0.6 } }`, engine and REST, and `{ a: { $eq: 0.19999999999999998 } }`, engine: - base: keeps the group on native and no group on rows; - after: keeps the group on both. The REST pin also holds the `a` case through REST, on both paths. H1 holds as the card stated it. ## H3: the compensated fold against SQLite native SQLite's native `sum` / `avg` was measured on three engines: better-sqlite3 3.53.4 (driver-sql), sql.js 3.49.1 (driver-sqlite-wasm) and @libsql/client 3.45.1 (driver-turso). better-sqlite3 and sql.js were read over a `REAL` and a `NUMERIC` column, libsql over a `REAL` column. The JS fold was run over the same doubles. The `2^53` row was read on better-sqlite3 alone. | fixture | naive (rows path before) | Neumaier (rows path after) | SQLite native, all three engines | |:--|:--|:--|:--| | `0.1, 0.2, 0.3` | `0.6000000000000001` / `0.20000000000000004` | `0.6` / `0.19999999999999998` | `0.6` / `0.19999999999999998` | | `1e16, 1, -1e16` | `0` / `0` | `1` / `0.3333333333333333` | `1` / `0.3333333333333333` | | `1e16, 0.5, -1e16` | `0` / `0` | `0.5` / `0.16666666666666666` | `0.5` / `0.16666666666666666` | | `1e20, 1, -1e20` | `0` / `0` | `1` / `0.3333333333333333` | `1` / `0.3333333333333333` | | `0.1, 0.2` (two-addend control) | `0.30000000000000004` / `0.15000000000000002` | the same | the same | | `1, 2, 3, 40, 500` (integers) | `546` / `109.2` | the same | the same | | `2^53, 1, 1` | `9007199254740992` | `9007199254740994` | `9007199254740994` | - **No fixture disagrees with SQLite.** A randomized cross-check agrees too: 5,000 groups of 3 to 22 values (cents, tenths, raw doubles, mixed sign and magnitude up to 1e19) against better-sqlite3. The Neumaier `sum` and `avg` matched SQLite on all 5,000 groups; the naive fold disagreed on 1,601 of them. - **SQLite's integer affinity.** Under `NUMERIC`, SQLite stores `1e16` as an INTEGER and sums it in exact int64. The answer is the same `1`. - **Integers beyond 2^53 move.** Integers are unchanged while the running total stays within 2^53. Beyond it, the compensated total is the exact total SQLite also answers, where the naive fold dropped the `1`s (last row). ## H2: census of rows folds of `sum` / `avg` in `packages/**` | site | own fold? | reading over `0.1, 0.2, 0.3`, head `8ad4a9d4a` | |:--|:--|:--| | objectql `applyInMemoryAggregation`, `sum` and `avg` arms | the fold this PR changes. H2 partly held: two naive `reduce` copies in one function, now one helper | `0.6` / `0.19999999999999998` | | per-aggregation `filter` | shares it: filters the bucket's rows, then the same arms | pinned in the unit test | | `having` | no fold: it reads the aggregated row whichever path produced it | inherits | | roll-up `summary` (`summary-aggregate.ts`), and service-analytics `ObjectQLStrategy` | no fold: both call `engine.aggregate` | inherit the engine's path | | driver-memory data face (`memory-driver.ts`, `sum` / `avg` arm) | **own naive fold** | `0.6000000000000001` / `0.20000000000000004` | | driver-memory analytics face (`memory-analytics.ts`, mingo `$sum` / `$avg`) | **own fold (mingo)** | `0.6000000000000001` / `0.20000000000000004` | | service-analytics draft preview (`preview-evaluator.ts`, `sum` / `avg` arms) | **own naive fold** | `0.6000000000000001` / `0.20000000000000004` | | service-analytics `dataset-executor.ts` `computeDerived` `sum` | not a rows fold: it adds measure columns within one row | n/a | | service-analytics `NativeSQLStrategy`, driver-mongodb `$sum` | database's own arithmetic | not measured | Three sites keep their own fold. They are measured and reported, not edited, as the dispatch requires. One reading needs stating plainly: - **driver-memory now splits by path.** `engine.aggregate` on driver-memory answers `0.6000000000000001` on its native path and `0.6` on the rows path. `having { s: { $eq: 0.6 } }` keeps the group on the rows path only. - **Before this PR, both of its paths were naive.** The split on that driver is new here; SQLite's split is gone. - It goes back to the seat as a family finding. The suggested route: hoist the fold to `@objectstack/core`, as `bucketDateKey` was, and adopt it on driver-memory's two faces and the preview. - The changeset states it. ## H4: what stays as it was (pinned) - `null` adds nothing to `sum` and is left out of the `avg` count. - A non-numeric cell reads as `0` and counts in `avg`. A numeric string reads as its number. A boolean reads as 0 / 1. - The empty group, with no rows or only nulls: `sum` `0`, `avg` `null`. - A non-finite total (`Infinity`, `-Infinity`, overflow, `NaN`) is `Object.is`-equal to the naive answer. - The answer's type is a number. ## H5: the residual **Residual, stated.** PostgreSQL and MySQL add `sum` / `avg` natively in double without compensation. That arithmetic is the database's own, and this PR does not wrap it. So over three or more fractions, their native path can still differ from the rows path in the last place. For `0.1 + 0.2 + 0.3`, #20387's measurement on live PostgreSQL 16.13 and MySQL 8.0.46 gave native `0.6000000000000001`; the rows path is now `0.6`. An exact `$eq` on a fractional sum compares doubles, and a cross-dialect last-place difference remains there: compare with a range. - No live PostgreSQL / MySQL cell was added. The cross-path pin is SQLite-only, as H5 asked, and the residual is stated rather than held as a failing pin. - **NOT MEASURED at this head: the PostgreSQL / MySQL readings.** This container has no MySQL. A PostgreSQL cluster would have needed a data directory outside the session's root-only scratchpad. - The rows path's own answer is dialect-independent. It is a pure function of the doubles `find()` returns, and #20387's pin reads those as `[0.1, 0.2]` on both servers. ## Tests **New pins** - `packages/objectql/src/in-memory-aggregation-compensated-sum.test.ts`: 11 cases, covering the H3 fixtures, groupBy plus a per-aggregation filter, and the H4 cases. - `packages/rest/src/rest-aggregate-compensated-sum.test.ts`: 4 cases on SQLite, through the engine and REST. It proves each path really ran (a spy on `driver.aggregate`), checks that `find()` reads back the doubles, pins native equal to rows for every group, and pins `having` `$eq` / `$in` on both paths. **Reverse verification, from the committed fix** - Tool: `scripts/ablation-replace.mjs` (wrap mode). It swapped `return Number.isFinite(c) ? s + c : s;` for `const ablation20489 = s; return ablation20489;`, which is the naive running sum. - Anchor count went 1 to 0, blob `ff8d2385a69f` to `89ea11d3bc18`. - Then `pnpm --filter @objectstack/objectql build`, and `ablation-dist-preflight.mjs @objectstack/objectql ablation20489` found the marker in 4 built files. - Mutated leg: objectql unit test 6 failed / 5 passed of 11; REST pin 2 failed / 2 passed of 4. The expected direction, red. - Restore leg: the restore was proven (blob equal to HEAD, `git diff HEAD` empty). - Then a rebuild, and `--absent` found the marker absent from all 14 built files, with a clean tree. - Unit 11/11 and REST 4/4. **Suites** | suite | head | result | |:--|:--|:--| | `pnpm --filter @objectstack/objectql test` | `db9d27230` | 334 files / 6651 tests passed | | `pnpm --filter @objectstack/rest test` | `db9d27230` | 224 files / 4242 passed, 43 skipped | | `typecheck` for both packages, `check:test-typecheck` included | `db9d27230` | OK; both new test files are in the `tsconfig.test.json` programs (`--listFilesOnly`) | | targeted re-run: objectql aggregation files 72/72, REST pin plus `rest-aggregate-numeric-having` 16 passed / 24 skipped | `8ad4a9d4a` | green | The merge brought driver-sql, cli, lint, platform-objects and spec changes, and none in objectql or rest. After it, the `@objectstack/rest^...` closure was rebuilt before the targeted re-run. **Gates** - `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` at `8ad4a9d4a` printed 63 commands. All 63 exit 0, and each exit code was written to a file before any pipe. - `check:dual-build-cjs-loads` and `check:type-check-debt` first exited 3 (PREREQUISITE NOT MET: no `dist/`). - They were re-run after `pnpm exec turbo run build --concurrency=2 --filter='./packages/*' --filter='./packages/*/*'` (71 tasks), and both passed. - The dist-reading gates were re-run over the full build too. - `--ran`: 63 derived, 63 run, 0 NOT-MEASURED, 0 UNRUN. - CI-only lanes, NOT MEASURED locally: the 5 path-scheduled CI jobs, the workspace type-check lanes and the wide-population families that `dispatch-gates` names. **Lint** - Repo-wide `pnpm lint` is CI's. - A narrowed run, measured: `eslint --no-inline-config --format json` over the three changed TS files counted 3 files, 0 errors and 0 warnings. - Population: `eslint.config.mjs` `files: ['**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}']`; the changeset is not a linted extension. - Invariance: the config enables no type-aware linting (no `parserOptions.project`), so this diff cannot move an untouched file's verdict. ## Acceptance notes All three notes from the first round were corrected in patch round 1, text only (below): - The `AGGREGATE_ACCUMULATION` residual comment in `packages/drivers/driver-sql/src/sql-driver.ts` ("as the rows path adds them", "on SQLite's native face alone", "every other face `0.6000000000000001`"). - The header sentence of `sql-driver-20387-aggregate-double-accumulation.test.ts` that called `rowsPathSum` the rows path's arithmetic. - The pending `.changeset/20387-aggregate-one-double.md` residual paragraph, as a DELIBERATE CORRECTION. Still standing, not edited (outside the admitted text): the `aggregate()` inline comment in `sql-driver.ts` ("so one query answers one number on every face", beside the `accumulatesInDouble` call). It was already an overclaim before this PR, since SQLite's native sum compensated. It points at `AGGREGATE_ACCUMULATION`, which now states the residual. Carrier: none. ## DELIBERATE CORRECTION **Note:** `.changeset/20387-aggregate-one-double.md`. It is pending on `main`, so it ships in the same release as this PR. **Sentence corrected:** the "Residual, stated." paragraph, and only that paragraph. It read: - "Double sums are added in row order, one after another, as the rows path adds them." - "So for a group of three or more fractions, SQLite's native answer can still differ from every other face in the last place (`0.1 + 0.2 + 0.3`: SQLite `0.6`, every other face `0.6000000000000001`). This was already true of SQLite's two paths before this change." **What changed under it:** this PR makes the engine's rows path add with compensated summation, as SQLite 3.43+ does. The rows path now answers `0.6`, not `0.6000000000000001`, so "every other face" and "SQLite's native face alone" are false once this PR lands. **It now reads:** PostgreSQL and MySQL add without compensation, while SQLite and, since #20489, the rows path compensate. So the PostgreSQL / MySQL native answer can differ from SQLite's and the rows path's in the last place (`0.1 + 0.2 + 0.3`: PostgreSQL / MySQL native `0.6000000000000001`, SQLite and the rows path `0.6`). Before #20489, SQLite's own two paths differed there too. Two addends cannot differ. **Every other sentence was checked against this PR and left unchanged:** - Line 16, "`11 / 9` answered `1.2222222222222222`, where every other face answers `1.2222222222222223`", stays true. Integer addends within 2^53 sum exactly under both folds. Measured through the built rows path at `1c4767883`: `applyInMemoryAggregation` over `1,1,1,1,1,1,1,2,2` answers `sum` `11` and `avg` `1.2222222222222223`, and over `1, 2, 2` answers `1.6666666666666667`. - The title and line 10, "SQLite and the engine's rows path add JS doubles", stay true: both still add doubles, now compensated. - The `0.1 + 0.2` statements (base lines 11-12, 38-40 and 48-50) are two addends, which no compensation moves. **Gate:** `check-empty-changeset` is red on exactly this one name, by design ("DELIBERATE CORRECTION ... do NOT restore it"). ⛔ No `skip-changeset`; the note is not restored. The at-tier review is the written confirmation, as the seat answer asks. ## Patch round 1 New head: `1c4767883`. One push, carrying a true merge of `origin/main` at `288611e3e` (spec-only commits) as `925368f00`, then one text-only commit. Seat answer 5881988201 ruled the open question A. **Edits:** 1. `.changeset/20387-aggregate-one-double.md`: the DELIBERATE CORRECTION above. 2. `packages/drivers/driver-sql/src/sql-driver.ts`: the `AGGREGATE_ACCUMULATION` residual comment now states PostgreSQL / MySQL native against SQLite and the rows path. 3. `packages/drivers/driver-sql/src/sql-driver-20387-aggregate-double-accumulation.test.ts`: the header sentence says the expected values are `find()`'s rows added in row order, which is the rows path's compensated sum too over this file's fixtures (two addends, and integers within 2^53). "a compensated sum ... fail" is removed, since it was never true for two addends. The one-line docs of `rowsPathSum` / `rowsPathAvg` make the same statement, because the header links them. 4. `.changeset/20489-rows-path-compensated-sum.md`: it no longer says it "replaces" the driver-sql residual. It says the driver-sql entry states the same residual, and it keeps the driver-memory split and the PostgreSQL / MySQL residual. **Comment-only proof for items 2 and 3:** every changed line under `packages/drivers/driver-sql/` is a comment line (`git diff` filtered to non-comment lines is empty). **No `@objectstack/driver-sql` line added to this PR's changeset. Why none is needed:** - Check Changeset counts the changesets a PR adds, not the packages it touches. - All 69 publishable packages are one `fixed` group in `.changeset/config.json`, so the `@objectstack/objectql` patch releases driver-sql in lockstep anyway. - The edited comment does not ship. `@objectstack/driver-sql`'s built `dist/index.{js,mjs,d.ts,d.mts}` carry 0 hits for "Kahan", "20489" or "every other face". The positive control, `AGGREGATE_ACCUMULATION`, is found in those same four files. **Checks at `1c4767883`, each exit code recorded before any pipe:** | check | exit | reading | |:--|:--|:--| | `node scripts/check-empty-changeset.mjs --base origin/main` | 1 | red on exactly one name, `.changeset/20387-aggregate-one-double.md` (one `::error` line), by design; the empty-frontmatter half is green | | `node scripts/check-changeset-fixed.mjs` | 0 | `fixed` group in sync with 69 public packages | | `node scripts/check-changeset-no-major.mjs --base origin/main` | 0 | no `major` bump | | `node scripts/check-adr-0087-registration.mjs --base origin/main` | 0 | no declared-breaking changeset | | `pnpm check:nul-bytes` | 0 | green | | `node scripts/check-issue-citations.mjs` | 0 | 3 citations resolve | | `pnpm --filter @objectstack/driver-sql typecheck` | 0 | the test file is in its program (`--listFilesOnly`) | | `vitest run src/sql-driver-20387-aggregate-double-accumulation.test.ts` (driver-sql) | 0 | 5 passed, 2 skipped (the PostgreSQL / MySQL cells, no server here) | | `pnpm check:driver-conformance`, `check:object-def-param-keys`, `check:tenant-chokepoint` | 0, 0, 0 | the three families `dispatch-gates` newly derives for the driver-sql paths | | objectql aggregation tests, and the REST compensated-sum pin | 0 | 72/72 and 4/4 | The other 63 derived families were run at `8ad4a9d4a` (above). This round changes only text on top of a spec-only merge, and CI re-runs them. --- _Generated by [Claude Code](https://claude.ai/code/session_01N8TPEsoJxPsdSdNKGnNGEN)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 7e5246d commit 8538edf

7 files changed

Lines changed: 433 additions & 16 deletions

‎.changeset/20387-aggregate-one-double.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -39,11 +39,12 @@ the rows path add exact decimals and round once. It was rejected because SQLite'
3939
adds the stored doubles (`0.30000000000000004`), so the rows path would then disagree with SQLite
4040
for exactly `0.1 + 0.2`.
4141

42-
**Residual, stated.** Double sums are added in row order, one after another, as the rows path
43-
adds them. SQLite 3.43 and later adds with compensated summation. So for a group of three or
44-
more fractions, SQLite's native answer can still differ from every other face in the last
45-
place (`0.1 + 0.2 + 0.3`: SQLite `0.6`, every other face `0.6000000000000001`). This was
46-
already true of SQLite's two paths before this change. Two addends cannot differ.
42+
**Residual, stated.** On PostgreSQL and MySQL the double sums are added in row order, one after
43+
another, without compensation. SQLite 3.43 and later adds with compensated summation, and since
44+
#20489 so does the engine's rows path. So for a group of three or more fractions, the PostgreSQL
45+
and MySQL native answer can still differ from SQLite's and the rows path's in the last place
46+
(`0.1 + 0.2 + 0.3`: PostgreSQL / MySQL native `0.6000000000000001`, SQLite and the rows path
47+
`0.6`). Before #20489, SQLite's own two paths differed there too. Two addends cannot differ.
4748

4849
A consumer that compared `sum` / `avg` over a fractional column with a decimal literal on
4950
PostgreSQL or MySQL (`$eq: 0.3`) now gets the answer SQLite and the rows path already gave:
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
'@objectstack/objectql': patch
3+
---
4+
5+
fix(objectql): the engine's rows path adds `sum` / `avg` with compensated summation, as SQLite does
6+
7+
Clause-②: no
8+
9+
`engine.aggregate` answers a `sum` / `avg` on one of two paths: the driver's own aggregate, or
10+
the rows path (`applyInMemoryAggregation`), which aggregates `find()` rows in JavaScript and is
11+
taken for a per-aggregation `filter`, a non-UTC date bucket, or a driver without native
12+
aggregation. SQLite 3.43 and later adds with Kahan-Babuska-Neumaier compensation; the rows path
13+
added naively. So on SQLite one query answered two doubles depending on the path. A `number`
14+
column holding `0.1`, `0.2` and `0.3` in one group:
15+
16+
| | `sum` | `avg` | `having { s: { $eq: 0.6 } }` |
17+
|:--|:--|:--|:--|
18+
| SQLite native | `0.6` | `0.19999999999999998` | keeps the group |
19+
| rows path, before | `0.6000000000000001` | `0.20000000000000004` | keeps no group |
20+
| rows path, after | `0.6` | `0.19999999999999998` | keeps the group |
21+
22+
The rows path now adds with the same compensation, transcribed from SQLite's own, so on SQLite
23+
both paths answer the same double, through `engine.aggregate` and
24+
`POST /api/v1/data/:object/query` alike. It is also the more accurate sum: `1e16 + 1 - 1e16` is
25+
`1`, where the naive fold answered `0`.
26+
27+
Unchanged: two addends (the compensated `a + b` is the naive one, so `0.1 + 0.2` is still
28+
`0.30000000000000004`), integers whose running total stays within 2^53, a non-finite total,
29+
`null` and non-numeric cells, and the empty group (`sum` `0`, `avg` `null`). The answer is still a
30+
JS number.
31+
32+
**Residual, stated.** PostgreSQL and MySQL add `sum` / `avg` natively in double without
33+
compensation, and that arithmetic is the database's own. So over three or more fractions their
34+
native path can still differ from the rows path in the last place (`0.1 + 0.2 + 0.3`: native
35+
`0.6000000000000001`, rows path `0.6`). The `@objectstack/driver-sql` entry for the double
36+
accumulation states the same residual: the difference is no longer SQLite's native path against
37+
every other face; it is PostgreSQL / MySQL native against SQLite and the rows path. The
38+
in-memory driver (`@objectstack/driver-memory`) still adds naively in its own `aggregate`, so on
39+
that driver the two paths can now differ in the same last place. An exact `$eq` on a fractional
40+
sum compares doubles: compare with a range.

‎packages/drivers/driver-sql/src/sql-driver-20387-aggregate-double-accumulation.test.ts‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,11 @@
2525
* `having` is evaluated by the engine, over the value this door answers
2626
* (`having-filter.ts`: `$eq` compares the value with `==`), so the value pinned
2727
* here is what decides `having` `$eq` / `$in` on every face. The expected
28-
* values are computed from `find()`'s own rows with the rows path's arithmetic
28+
* values are computed from `find()`'s own rows, added in row order
2929
* ({@link rowsPathSum}), plus the literal the triage named, so a decimal
30-
* answer, a compensated sum or a string each fail.
30+
* answer or a string each fail. Over these fixtures — two addends, and
31+
* integer-valued columns within 2^53 — that sum is also the rows path's, which
32+
* adds with compensation since #20489: the two folds cannot differ there.
3133
*
3234
* What stays as it was, pinned too: `count` / `min` / `max`, and a `sum` over
3335
* an integer-valued column, which keeps the database's exact total (above 2^53
@@ -49,10 +51,10 @@ function toNumber(v: unknown): number {
4951
return Number.isFinite(n) ? n : 0;
5052
}
5153

52-
/** The rows path's `sum`: `values.reduce((a, b) => a + toNumber(b), 0)`, in row order. */
54+
/** `values.reduce((a, b) => a + toNumber(b), 0)`, in row order: the rows path's `sum` over this file's fixtures (see the header). */
5355
const rowsPathSum = (values: readonly unknown[]) => values.reduce<number>((a, b) => a + toNumber(b), 0);
5456

55-
/** The rows path's `avg`: the non-null values' `sum`, divided by how many there are. */
57+
/** The non-null values' {@link rowsPathSum}, divided by how many there are: the rows path's `avg` over this file's fixtures. */
5658
function rowsPathAvg(values: readonly unknown[]): number | null {
5759
const defined = values.filter((v) => v != null);
5860
return defined.length === 0 ? null : rowsPathSum(defined) / defined.length;

‎packages/drivers/driver-sql/src/sql-driver.ts‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1594,11 +1594,13 @@ const AGGREGATE_ANSWER_KIND: Readonly<Record<AggregationFunction, 'number' | 'co
15941594
* would widen the binary32 value (`0.1` → `0.10000000149011612`) where the
15951595
* client reads `0.1`. MySQL's `CAST(… AS DOUBLE)` needs 8.0.17 or later.
15961596
*
1597-
* ⚠️ Residual, stated: the double sums are added in scan order, one after
1598-
* another, as the rows path adds them. SQLite (3.43+) adds with compensated
1599-
* (Kahan-Babuska-Neumaier) summation, so a group of three or more fractions can
1600-
* still differ in the last place on SQLite's native face alone (`0.1 + 0.2 +
1601-
* 0.3`: SQLite `0.6`, every other face `0.6000000000000001`). Two addends
1597+
* ⚠️ Residual, stated: on PostgreSQL and MySQL the double sums are added in
1598+
* scan order, one after another, without compensation. SQLite (3.43+) adds with
1599+
* compensated (Kahan-Babuska-Neumaier) summation, and since #20489 so does the
1600+
* engine's rows path (`in-memory-aggregation.ts`, `compensatedSum`), so a group
1601+
* of three or more fractions can still differ in the last place between the
1602+
* PostgreSQL / MySQL native faces and those two (`0.1 + 0.2 + 0.3`: PostgreSQL /
1603+
* MySQL `0.6000000000000001`, SQLite and the rows path `0.6`). Two addends
16021604
* cannot differ, which is why the pin is `0.1 + 0.2`.
16031605
*
16041606
* A `Record` over `AggregationFunction` for the same reason as
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#20489] The rows path's `sum` / `avg` add with Kahan-Babuska-Neumaier
5+
* compensation (`in-memory-aggregation.ts`, `compensatedSum`), the summation
6+
* SQLite 3.43+ uses for its own `sum` / `avg`.
7+
*
8+
* The expected values below are SQLite's own answers, measured with
9+
* better-sqlite3 (SQLite 3.53.4), sql.js (3.49.1) and @libsql/client (3.45.1)
10+
* over a `REAL` and a `NUMERIC` column holding the same values. All three
11+
* engines agreed on every fixture. The naive fold the rows path used before is
12+
* shown beside each one:
13+
*
14+
* | fixture | naive `sum` / `avg` (before) | SQLite native = compensated (after) |
15+
* |:--|:--|:--|
16+
* | `0.1, 0.2, 0.3` | `0.6000000000000001` / `0.20000000000000004` | `0.6` / `0.19999999999999998` |
17+
* | `1e16, 1, -1e16` | `0` / `0` | `1` / `0.3333333333333333` |
18+
* | `1e16, 0.5, -1e16` | `0` / `0` | `0.5` / `0.16666666666666666` |
19+
* | `0.1, 0.2` (two addends) | `0.30000000000000004` / `0.15000000000000002` | the same |
20+
* | `1, 2, 3, 40, 500` (integers) | `546` / `109.2` | the same |
21+
*
22+
* `engine.aggregate` and REST on SQLite are pinned in `packages/rest`
23+
* (`rest-aggregate-compensated-sum.test.ts`): the rows path and the native
24+
* path answer the same double there.
25+
*/
26+
27+
import { describe, it, expect } from 'vitest';
28+
import { applyInMemoryAggregation } from './in-memory-aggregation.js';
29+
30+
/** The fold the rows path used before #20489: in order, one addition at a time. */
31+
const naiveSum = (xs: readonly number[]) => xs.reduce((a, b) => a + b, 0);
32+
33+
/** `sum` and `avg` of `w` over `values`, one group, through the rows path. */
34+
function sumAvg(values: readonly unknown[]): { s: unknown; a: unknown } {
35+
const rows = values.map((w, i) => ({ id: `r${i}`, w }));
36+
const [out] = applyInMemoryAggregation(rows, {
37+
aggregations: [
38+
{ function: 'sum', field: 'w', alias: 's' },
39+
{ function: 'avg', field: 'w', alias: 'a' },
40+
],
41+
});
42+
return out as { s: unknown; a: unknown };
43+
}
44+
45+
describe('[#20489] rows path — sum / avg add with compensation, as SQLite does', () => {
46+
it("the card's fixture: 0.1 + 0.2 + 0.3 answers SQLite's 0.6, and avg its 0.19999999999999998", () => {
47+
const values = [0.1, 0.2, 0.3];
48+
// The fixture discriminates: the naive fold answers another double.
49+
expect(naiveSum(values)).toBe(0.6000000000000001);
50+
expect(sumAvg(values)).toStrictEqual({ s: 0.6, a: 0.19999999999999998 });
51+
// So `having { s: { $eq: 0.6 } }` now keeps the group on this path too.
52+
expect(sumAvg(values).s === 0.6).toBe(true);
53+
});
54+
55+
it('a mixed-sign set with large cancellation keeps the small addend', () => {
56+
expect(naiveSum([1e16, 1, -1e16])).toBe(0);
57+
expect(sumAvg([1e16, 1, -1e16])).toStrictEqual({ s: 1, a: 1 / 3 });
58+
expect(naiveSum([1e16, 0.5, -1e16])).toBe(0);
59+
expect(sumAvg([1e16, 0.5, -1e16])).toStrictEqual({ s: 0.5, a: 0.5 / 3 });
60+
});
61+
62+
it('two addends are unchanged: the compensated a + b is the naive one', () => {
63+
for (const pair of [[0.1, 0.2], [0.7, 0.1], [1e16, 1], [-0.3, 0.1]]) {
64+
const n = naiveSum(pair);
65+
expect(sumAvg(pair), `${pair}`).toStrictEqual({ s: n, a: n / 2 });
66+
}
67+
expect(sumAvg([0.1, 0.2])).toStrictEqual({ s: 0.30000000000000004, a: 0.15000000000000002 });
68+
});
69+
70+
it('integers whose partial sums stay within 2^53 are unchanged, and stay integers', () => {
71+
const values = [1, 2, 3, 40, 500];
72+
const { s, a } = sumAvg(values);
73+
expect(s).toBe(naiveSum(values));
74+
expect(s).toBe(546);
75+
expect(Number.isInteger(s)).toBe(true);
76+
expect(a).toBe(109.2);
77+
expect(sumAvg([-7, 3, 12, 0, 9_000_000_000])).toStrictEqual({ s: 9_000_000_008, a: 9_000_000_008 / 5 });
78+
});
79+
80+
it('above 2^53 the compensated total is the exact one SQLite answers, where the naive fold lost the 1s', () => {
81+
// SQLite 3.53.4 answers 9007199254740994 over a REAL and over a NUMERIC column.
82+
expect(naiveSum([2 ** 53, 1, 1])).toBe(2 ** 53);
83+
expect(sumAvg([2 ** 53, 1, 1])).toStrictEqual({ s: 9007199254740994, a: 3002399751580331.5 });
84+
});
85+
86+
it('under groupBy and a per-aggregation filter, every bucket takes the same fold', () => {
87+
const rows = [
88+
{ g: 'x', k: 'in', w: 0.1 },
89+
{ g: 'x', k: 'in', w: 0.2 },
90+
{ g: 'x', k: 'out', w: 100 },
91+
{ g: 'x', k: 'in', w: 0.3 },
92+
{ g: 'y', k: 'in', w: 1e16 },
93+
{ g: 'y', k: 'in', w: 1 },
94+
{ g: 'y', k: 'in', w: -1e16 },
95+
];
96+
const out = applyInMemoryAggregation(rows, {
97+
groupBy: ['g'],
98+
aggregations: [
99+
{ function: 'sum', field: 'w', alias: 's', filter: { k: 'in' } },
100+
{ function: 'avg', field: 'w', alias: 'a', filter: { k: 'in' } },
101+
{ function: 'sum', field: 'w', alias: 'all' },
102+
],
103+
} as any).sort((p, q) => String(p.g).localeCompare(String(q.g)));
104+
expect(out).toStrictEqual([
105+
{ g: 'x', s: 0.6, a: 0.19999999999999998, all: 100.6 },
106+
{ g: 'y', s: 1, a: 1 / 3, all: 1 },
107+
]);
108+
});
109+
});
110+
111+
describe('[#20489] rows path — what the compensated fold leaves as it was', () => {
112+
it('null: sum adds nothing for it, avg leaves it out of the count', () => {
113+
expect(sumAvg([0.1, null, 0.2, undefined, 0.3])).toStrictEqual({ s: 0.6, a: 0.19999999999999998 });
114+
expect(sumAvg([10, null, 20])).toStrictEqual({ s: 30, a: 15 });
115+
});
116+
117+
it('a non-numeric cell reads as 0 in both, and counts in avg; a numeric string reads as its number', () => {
118+
expect(sumAvg([10, 'abc', 20])).toStrictEqual({ s: 30, a: 10 });
119+
expect(sumAvg(['0.1', '0.2', '0.3'])).toStrictEqual({ s: 0.6, a: 0.19999999999999998 });
120+
expect(sumAvg([true, false, true])).toStrictEqual({ s: 2, a: 2 / 3 });
121+
});
122+
123+
it('an empty group: sum 0, avg null — with no rows, and with only nulls', () => {
124+
expect(sumAvg([])).toStrictEqual({ s: 0, a: null });
125+
expect(sumAvg([null, undefined])).toStrictEqual({ s: 0, a: null });
126+
});
127+
128+
it('a non-finite total is the naive one, as SQLite returns its running sum when the error term overflows', () => {
129+
for (const values of [
130+
[Infinity, 1, 2],
131+
[1, -Infinity, 0.3],
132+
[1e308, 1e308, -1e308],
133+
[Infinity, -Infinity, 1],
134+
[NaN, 0.1, 0.2],
135+
]) {
136+
const n = naiveSum(values);
137+
const { s, a } = sumAvg(values);
138+
expect(Object.is(s, n), `sum of ${values}: ${s} vs naive ${n}`).toBe(true);
139+
expect(Object.is(a, n / values.length), `avg of ${values}`).toBe(true);
140+
}
141+
});
142+
143+
it('the answer is a JS number on every arm, never a string', () => {
144+
const { s, a } = sumAvg(['0.1', 0.2, '0.3']);
145+
expect(typeof s).toBe('number');
146+
expect(typeof a).toBe('number');
147+
});
148+
});

‎packages/objectql/src/in-memory-aggregation.ts‎

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -228,12 +228,15 @@ function aggregateBucket(
228228
case 'count_distinct':
229229
out[alias] = new Set(values.filter((v) => v != null)).size;
230230
break;
231+
// [#20489] Both arms add through ONE compensated fold
232+
// ({@link compensatedSum}) — the summation SQLite's own `sum` / `avg`
233+
// use — so the rows path and SQLite's native path answer the same double.
231234
case 'sum':
232-
out[alias] = values.reduce((a, b) => a + toNumber(b), 0);
235+
out[alias] = compensatedSum(values.map(toNumber));
233236
break;
234237
case 'avg': {
235238
const nums = values.filter((v) => v != null).map(toNumber);
236-
out[alias] = nums.length === 0 ? null : nums.reduce((a, b) => a + b, 0) / nums.length;
239+
out[alias] = nums.length === 0 ? null : compensatedSum(nums) / nums.length;
237240
break;
238241
}
239242
// [#11152] `min`/`max` read a BOOLEAN as the number it is worth (0/1) —
@@ -284,6 +287,44 @@ function toNumber(v: any): number {
284287
return Number.isFinite(n) ? n : 0;
285288
}
286289

290+
/**
291+
* [#20489] The sum of `nums`, added in order with Kahan-Babuska-Neumaier
292+
* compensation — the summation SQLite (3.43 and later) uses for its own `sum`
293+
* and `avg`, transcribed from its `kahanBabuskaNeumaierStep` and the
294+
* finalizers' overflow guard.
295+
*
296+
* Why: this fold used to add naively (`reduce((a, b) => a + b, 0)`), so one
297+
* query answered two doubles on SQLite depending on the path `engine.aggregate`
298+
* took. A `number` column holding `0.1`, `0.2` and `0.3` summed to `0.6`
299+
* natively and to `0.6000000000000001` here (`avg` `0.19999999999999998`
300+
* against `0.20000000000000004`), and `having { s: { $eq: 0.6 } }` kept the
301+
* group on the native path alone. Compensated, the two paths agree, and the
302+
* answer is the more accurate one (`1e16 + 1 - 1e16` is `1`, not `0`).
303+
*
304+
* What does not move: two addends (the compensated `a + b` IS the naive one),
305+
* integers whose partial sums stay within 2^53 (every addition is exact), and
306+
* a non-finite total. `s` below is exactly the naive running sum; once it
307+
* overflows or meets a NaN, the error term is non-finite and the naive answer
308+
* is returned as it was, which is SQLite's rule too.
309+
*
310+
* ⚠️ Residual, stated: PostgreSQL and MySQL add their doubles natively without
311+
* compensation, so over three or more fractions their native path can still
312+
* differ from this one in the last place. So can the folds that do not call
313+
* this one: `driver-memory`'s `aggregate` and analytics faces, and
314+
* `service-analytics`' draft preview (`preview-evaluator.ts`). An exact `$eq`
315+
* on a fractional sum compares doubles; compare with a range.
316+
*/
317+
function compensatedSum(nums: readonly number[]): number {
318+
let s = 0;
319+
let c = 0;
320+
for (const r of nums) {
321+
const t = s + r;
322+
c += Math.abs(s) > Math.abs(r) ? (s - t) + r : (r - t) + s;
323+
s = t;
324+
}
325+
return Number.isFinite(c) ? s + c : s;
326+
}
327+
287328
/**
288329
* Bucket a date-like value into an ISO-formatted period label. Weeks start
289330
* Monday and use ISO week numbering.

0 commit comments

Comments
 (0)