Skip to content

Commit e827c23

Browse files
committed
fix(core,driver-sql,service-analytics): the analytics native-SQL path answers measures declared number as numbers (#20889)
On PostgreSQL, NativeSQLStrategy returned the SQL client's rows unshaped, so count / count_distinct / sum / avg and numeric min / max came back as strings while fields[] declared number. driver-sql's own aggregate() has presented those answers since #20335, through a private table and presenter. - core: utils/aggregate-answer.ts receives AGGREGATE_ANSWER_KIND (its docblock moved byte-identical) and the 'number' presenter, presentAsNumber (the body of presentReadValue's 'number' arm). Exported from the root beside compensatedSum. - driver-sql: the definition becomes an import; presentReadValue's 'number' arm calls presentAsNumber. No behaviour change. - service-analytics: NativeSQLStrategy.execute presents each measure column keyed on its declared aggregate function, and min / max only over a column declared numeric (declaredFieldType). Expression measures are untouched. Claude-Session: https://claude.ai/code/session_01XY5uCwTjZj7884yYtyur4H Co-authored-by: Claude <noreply@anthropic.com>
1 parent 719b987 commit e827c23

5 files changed

Lines changed: 223 additions & 67 deletions

File tree

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
'@objectstack/core': minor
3+
'@objectstack/driver-sql': patch
4+
'@objectstack/service-analytics': patch
5+
---
6+
7+
fix: the analytics native-SQL path answers a measure its response declares `number` as a number on every dialect, presented by the one rule `driver-sql`'s `aggregate()` applies, which `@objectstack/core` now exports as `AGGREGATE_ANSWER_KIND` and `presentAsNumber` (#20889)
8+
9+
Clause-②: yes (widening)
10+
11+
**New exports.** `@objectstack/core` exports two names, moved here unchanged
12+
from `@objectstack/driver-sql`, which now imports them instead of keeping them
13+
private:
14+
15+
- `AGGREGATE_ANSWER_KIND`: what each declared aggregate function answers.
16+
`count`, `count_distinct`, `sum` and `avg` answer `'number'`; `min` and `max`
17+
answer `'column'`, a value of the aggregated column.
18+
- `presentAsNumber(value)`: the `'number'` presentation. A string `Number()`
19+
reads as a number becomes that number. Any other value is returned as given:
20+
a number, `null`, a boolean, empty or blank text, or text that reads as NaN.
21+
22+
**What changed.** On PostgreSQL, `POST /api/v1/analytics/query` and
23+
`POST /api/v1/analytics/dataset/query` answered through `NativeSQLStrategy`
24+
returned count, count_distinct, sum, avg, and min / max over a numeric column
25+
as strings, such as `count: "2"` and
26+
`sum: "500.000000000000000000000000000000"`, while `fields[]` declared
27+
`number`. A dataset's `row_count` did the same, and a measure-scoped count
28+
mixed `"1"` with the number `0` in one column. SQLite answered numbers. The
29+
strategy now presents each measure column by its declared aggregate function,
30+
through the same table and presenter as `SqlDriver.aggregate()`. `min` / `max`
31+
are presented only when their column is declared numeric, so `max` over a text
32+
column, every dimension, and expression measures keep the value the database
33+
returned.
34+
35+
**Precision.** The answer is one JS number, the policy `driver-sql`'s
36+
`aggregate()` already applies. A total that needs more digits than a double
37+
holds, such as `9007199254740993`, answers the nearest double
38+
(`9007199254740992`), which is also what SQLite and the engine path answer.
39+
40+
**What did not move.** `@objectstack/driver-sql`'s behaviour is unchanged: its
41+
`aggregate()` reads the same table, and its read presenter calls the same
42+
function. The answers on SQLite are byte-identical. The arithmetic of the
43+
analytics native statement did not change either. On PostgreSQL its `sum` and
44+
`avg` still add exact decimals, so `0.1 + 0.2` answers `0.3` where the engine
45+
path answers `0.30000000000000004`.

‎packages/core/src/index.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,9 @@ export * from './utils/datetime.js';
5656
// runtime dependencies, and a copy per face is how one `sum` came to answer
5757
// two doubles.
5858
export * from './utils/compensated-sum.js';
59+
// [#20889] What an aggregate ANSWERS and its `'number'` presenter, moved from
60+
// `driver-sql` so the analytics native-SQL face presents with the same rule.
61+
export * from './utils/aggregate-answer.js';
5962

6063
// Export the shared batched-write helper (framework#2678)
6164
export * from './utils/bulk-write.js';
Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
2+
3+
/**
4+
* [#20889] What an aggregate ANSWERS, and the `'number'` presenter that gives a
5+
* count or a total its type — defined once, for every face that hands a SQL
6+
* client's aggregate row to a caller.
7+
*
8+
* ## Why it lives here
9+
*
10+
* Two faces run an aggregate statement on a SQL client and must present its
11+
* answer the same way:
12+
*
13+
* - `@objectstack/driver-sql`'s native `aggregate()` (#20335), where this rule
14+
* was written: `aggregate()` reads {@link AGGREGATE_ANSWER_KIND} to choose
15+
* the result columns that take the `'number'` presentation, and
16+
* `presentReadValue`'s `'number'` arm is {@link presentAsNumber};
17+
* - `@objectstack/service-analytics`' native-SQL face
18+
* (`NativeSQLStrategy.execute`), which runs its own statement through the
19+
* host's raw-SQL bridge — `engine.execute`, or an embedder's own
20+
* `executeRawSql` — and so never passes through the driver's `aggregate()`.
21+
* On PostgreSQL it answered `count: "2"` under a `fields[]` that declared
22+
* `number`.
23+
*
24+
* `service-analytics` holds `driver-sql` as a development dependency only — it
25+
* is multi-driver, and its native path also runs over an embedder's raw SQL —
26+
* and this is the package both already stand on, the reason `compensatedSum`
27+
* lives here too. A second transcription is how a face comes to answer its own
28+
* type again.
29+
*
30+
* The table and its docblock below moved here from `driver-sql` unchanged, so
31+
* they speak of `SqlDriver.aggregate` and `formatOutput` in that driver's
32+
* voice.
33+
*/
34+
35+
import type { AggregationFunction } from '@objectstack/spec/data';
36+
37+
/**
38+
* [#20335] What each declared aggregate function ANSWERS — a derived `number`,
39+
* or a value OF the aggregated column — and therefore which read presentation
40+
* {@link SqlDriver.aggregate} gives its result column.
41+
*
42+
* - `'number'` — `count`, `count_distinct`, `sum`, `avg`. A count or a total is
43+
* a number whatever the column held, and it is presented as one (`'number'`,
44+
* the presenter `formatOutput` applies to a numeric field on a `find()` row).
45+
* - `'column'` — `min`, `max`. The answer is one of the column's own values, so
46+
* it takes that column's presentation ({@link SqlDriver.readPresentationKind}),
47+
* exactly as before this table existed.
48+
*
49+
* Why the `'number'` half needs presenting at all: the SQL client hands a
50+
* result back as the wire type of the SQL expression, not as the platform's
51+
* value type. Measured on live PostgreSQL 16.13 and MySQL 8.0.46 through this
52+
* driver's own connections: node-postgres parses `bigint` (OID 20 — `count`,
53+
* and `sum` over an integer column) and `numeric` (OID 1700 — `sum` / `avg`
54+
* over the exact-decimal numeric family, `avg` over an integer column) to
55+
* STRINGS (`"2"`, `"500.000000000000000000000000000000"`), and mysql2 does the
56+
* same for `DECIMAL` (`SUM` / `AVG`; its `COUNT` arrives as a number). The
57+
* engine's rows path (`objectql`'s `in-memory-aggregation.ts`) and SQLite answer
58+
* numbers for the same query, so `having { n: { $in: [2] } }` kept c1, c2 on
59+
* those and no group on PostgreSQL's native path.
60+
*
61+
* Keyed on the function the query ASKED for, never on whether a value looks
62+
* numeric, and deliberately not gated by dialect: the presenter only rewrites a
63+
* STRING, so a client that already answers a number (better-sqlite3, mysql2's
64+
* `COUNT`) passes through untouched — measured byte-identical on SQLite — and
65+
* no list of "string-answering dialects" exists to drift.
66+
*
67+
* ## The precision policy — one JS number, the loss declared
68+
*
69+
* The answer is `Number(text)`: an IEEE-754 double, on every dialect. A `sum` /
70+
* `avg` over the exact-decimal column (`numeric(65,30)` / `DECIMAL(65,30)`)
71+
* whose value needs more than a double's ~15-17 significant digits, or an
72+
* integer at or above 2^53, is ROUNDED to the nearest double — declared, not
73+
* silent: it is the same bound `formatOutput` already puts on a `find()` read of
74+
* that column (#16318, `valueSchemaFor`'s `z.number().finite()`, ADR-0104 D1),
75+
* and the bound the rows path has always had (`toNumber` sums JS doubles). A
76+
* value-dependent type — a number when it fits, a string when it does not — was
77+
* rejected: it would reopen this defect for exactly the large totals, where a
78+
* `having` `$in` or a chart silently stops matching. Only a string `Number()`
79+
* reads as NaN (PostgreSQL's `numeric` `'NaN'`) is left as written, the
80+
* presenter's existing rule.
81+
*
82+
* A `Record` over `AggregationFunction` on purpose: a function that joins the
83+
* declared vocabulary without an answer here fails `tsc` rather than reaching a
84+
* caller unpresented.
85+
*/
86+
export const AGGREGATE_ANSWER_KIND: Readonly<Record<AggregationFunction, 'number' | 'column'>> = {
87+
count: 'number',
88+
count_distinct: 'number',
89+
sum: 'number',
90+
avg: 'number',
91+
min: 'column',
92+
max: 'column',
93+
};
94+
95+
/**
96+
* [#20335, #20889] The `'number'` presentation: a STRING `Number()` reads as a
97+
* number becomes that number; every other value — a number, `null`, a boolean,
98+
* empty or blank text, text `Number()` reads as NaN (PostgreSQL's `numeric`
99+
* `'NaN'`) — is returned as given.
100+
*
101+
* It is the body of `driver-sql`'s `presentReadValue` `'number'` arm, moved
102+
* here unchanged: the presenter `formatOutput` applies to a declared numeric
103+
* column on a `find()` row (#16318), and the one {@link AGGREGATE_ANSWER_KIND}
104+
* names for a count or a total. The answer is one JS double — the precision
105+
* policy that table's docblock states.
106+
*
107+
* ⛔ Ask it by a column's DECLARED meaning — the aggregate function the query
108+
* asked for, or a column declared numeric — never because a value looks
109+
* numeric: a text column holding `'007'` is text.
110+
*/
111+
export function presentAsNumber(value: unknown): unknown {
112+
// Only strings are repaired, exactly as in `formatOutput`: a fresh
113+
// REAL/INTEGER column already yields a number, and genuinely
114+
// non-numeric legacy junk is left intact rather than turned into NaN.
115+
if (typeof value === 'string' && value.trim() !== '') {
116+
const n = Number(value);
117+
if (!Number.isNaN(n)) return n;
118+
}
119+
return value;
120+
}

‎packages/drivers/driver-sql/src/sql-driver.ts‎

Lines changed: 14 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ import { parseAutonumberFormat, renderAutonumber, resolveAutonumberFormat, readA
2121
// "the protocol has no such function" refusal cannot drift from what
2222
// `AggregationNodeSchema.function` actually admits.
2323
import { AggregationFunction, emptyGroupValueFor } from '@objectstack/spec/data';
24+
// [#20335, #20889] What each aggregate function ANSWERS, and the `'number'`
25+
// presenter its counts and totals take — defined once in core, for this
26+
// driver's `aggregate()` and the analytics native-SQL face alike.
27+
import { AGGREGATE_ANSWER_KIND, presentAsNumber } from '@objectstack/core';
2428
import { STRUCTURED_JSON_TYPES, FILE_REFERENCE_TYPES, MULTI_OPTION_TYPES, NUMERIC_VALUE_TYPES, isMultiValueField } from '@objectstack/spec/data';
2529
// [#16318] The per-field-type physical representation of the NUMERIC family.
2630
// `os generate migration` reads the SAME table, in both of its formats — that
@@ -1530,63 +1534,11 @@ const SQL_AGGREGATE_FUNCTIONS: ReadonlyMap<string, SqlAggregateLowering> = new M
15301534
['count_distinct', { sql: 'count', distinct: true }],
15311535
]);
15321536

1533-
/**
1534-
* [#20335] What each declared aggregate function ANSWERS — a derived `number`,
1535-
* or a value OF the aggregated column — and therefore which read presentation
1536-
* {@link SqlDriver.aggregate} gives its result column.
1537-
*
1538-
* - `'number'` — `count`, `count_distinct`, `sum`, `avg`. A count or a total is
1539-
* a number whatever the column held, and it is presented as one (`'number'`,
1540-
* the presenter `formatOutput` applies to a numeric field on a `find()` row).
1541-
* - `'column'` — `min`, `max`. The answer is one of the column's own values, so
1542-
* it takes that column's presentation ({@link SqlDriver.readPresentationKind}),
1543-
* exactly as before this table existed.
1544-
*
1545-
* Why the `'number'` half needs presenting at all: the SQL client hands a
1546-
* result back as the wire type of the SQL expression, not as the platform's
1547-
* value type. Measured on live PostgreSQL 16.13 and MySQL 8.0.46 through this
1548-
* driver's own connections: node-postgres parses `bigint` (OID 20 — `count`,
1549-
* and `sum` over an integer column) and `numeric` (OID 1700 — `sum` / `avg`
1550-
* over the exact-decimal numeric family, `avg` over an integer column) to
1551-
* STRINGS (`"2"`, `"500.000000000000000000000000000000"`), and mysql2 does the
1552-
* same for `DECIMAL` (`SUM` / `AVG`; its `COUNT` arrives as a number). The
1553-
* engine's rows path (`objectql`'s `in-memory-aggregation.ts`) and SQLite answer
1554-
* numbers for the same query, so `having { n: { $in: [2] } }` kept c1, c2 on
1555-
* those and no group on PostgreSQL's native path.
1556-
*
1557-
* Keyed on the function the query ASKED for, never on whether a value looks
1558-
* numeric, and deliberately not gated by dialect: the presenter only rewrites a
1559-
* STRING, so a client that already answers a number (better-sqlite3, mysql2's
1560-
* `COUNT`) passes through untouched — measured byte-identical on SQLite — and
1561-
* no list of "string-answering dialects" exists to drift.
1562-
*
1563-
* ## The precision policy — one JS number, the loss declared
1564-
*
1565-
* The answer is `Number(text)`: an IEEE-754 double, on every dialect. A `sum` /
1566-
* `avg` over the exact-decimal column (`numeric(65,30)` / `DECIMAL(65,30)`)
1567-
* whose value needs more than a double's ~15-17 significant digits, or an
1568-
* integer at or above 2^53, is ROUNDED to the nearest double — declared, not
1569-
* silent: it is the same bound `formatOutput` already puts on a `find()` read of
1570-
* that column (#16318, `valueSchemaFor`'s `z.number().finite()`, ADR-0104 D1),
1571-
* and the bound the rows path has always had (`toNumber` sums JS doubles). A
1572-
* value-dependent type — a number when it fits, a string when it does not — was
1573-
* rejected: it would reopen this defect for exactly the large totals, where a
1574-
* `having` `$in` or a chart silently stops matching. Only a string `Number()`
1575-
* reads as NaN (PostgreSQL's `numeric` `'NaN'`) is left as written, the
1576-
* presenter's existing rule.
1577-
*
1578-
* A `Record` over `AggregationFunction` on purpose: a function that joins the
1579-
* declared vocabulary without an answer here fails `tsc` rather than reaching a
1580-
* caller unpresented.
1581-
*/
1582-
const AGGREGATE_ANSWER_KIND: Readonly<Record<AggregationFunction, 'number' | 'column'>> = {
1583-
count: 'number',
1584-
count_distinct: 'number',
1585-
sum: 'number',
1586-
avg: 'number',
1587-
min: 'column',
1588-
max: 'column',
1589-
};
1537+
// [#20335, #20889] `AGGREGATE_ANSWER_KIND` — what each declared aggregate
1538+
// function ANSWERS, and the one-double precision policy for that answer — lives
1539+
// in `@objectstack/core` (`utils/aggregate-answer.ts`), imported above with its
1540+
// `'number'` presenter, so the analytics native-SQL face presents a count or a
1541+
// total with this driver's own rule.
15901542

15911543
/**
15921544
* [#20387] What each declared aggregate function ACCUMULATES IN on PostgreSQL
@@ -15839,16 +15791,11 @@ export class SqlDriver implements IDataDriver {
1583915791
return presentAuditTimestampOutput(value);
1584015792
case 'boolean':
1584115793
return Boolean(value);
15842-
case 'number': {
15843-
// Only strings are repaired, exactly as in `formatOutput`: a fresh
15844-
// REAL/INTEGER column already yields a number, and genuinely
15845-
// non-numeric legacy junk is left intact rather than turned into NaN.
15846-
if (typeof value === 'string' && value.trim() !== '') {
15847-
const n = Number(value);
15848-
if (!Number.isNaN(n)) return n;
15849-
}
15850-
return value;
15851-
}
15794+
case 'number':
15795+
// [#20889] The presenter lives in `@objectstack/core`, beside
15796+
// `AGGREGATE_ANSWER_KIND`, so the analytics native-SQL face presents
15797+
// a count or a total with this same function.
15798+
return presentAsNumber(value);
1585215799
}
1585315800
}
1585415801

‎packages/services/service-analytics/src/strategies/native-sql-strategy.ts‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import type { AnalyticsQuery, AnalyticsResult } from '@objectstack/spec/contracts';
44
import type { Cube } from '@objectstack/spec/data';
5+
import { NUMERIC_VALUE_TYPES, type AggregationFunction } from '@objectstack/spec/data';
56
import type { AnalyticsStrategy, StrategyContext, DatasetScopedStrategyContext } from './types.js';
67
import {
78
declaredDatetimeLowering,
@@ -21,6 +22,9 @@ import { datasetInvalidError, invalidMemberError } from '../dataset-refusal.js';
2122
import { type LikeShape } from '../like-pattern.js';
2223
import { textMatchPredicateSql, sqlDialectFor } from '../text-match-sql.js';
2324
import { nextUtcCalendarDay, resolveAnalyticsDateRangeString, isUnboundedAbove } from '@objectstack/core';
25+
// [#20889] What each aggregate function ANSWERS, and the `'number'` presenter —
26+
// the rule `driver-sql`'s own `aggregate()` applies, defined once in core.
27+
import { AGGREGATE_ANSWER_KIND, presentAsNumber } from '@objectstack/core';
2428
import { explicitDateRangeWindow } from '../date-range-array-arm.js';
2529

2630
/**
@@ -467,6 +471,43 @@ export class NativeSQLStrategy implements AnalyticsStrategy {
467471

468472
const rows = await ctx.executeRawSql!(objectName, sql, params);
469473

474+
// [#20889] A measure column `fields[]` declares `number` answers a number,
475+
// on every dialect. The SQL client hands an aggregate back as the wire type
476+
// of its expression: node-postgres parses `bigint` (`count`, `sum` over an
477+
// integer column) and `numeric` (`sum` / `avg` over the exact-decimal
478+
// column, `avg` over an integer one, `min` / `max` over a decimal one) to
479+
// STRINGS, mysql2 does the same for `DECIMAL`, and better-sqlite3 answers
480+
// numbers. `driver-sql`'s own `aggregate()` presents those answers
481+
// (#20335); this statement reaches the client through the host's raw-SQL
482+
// bridge instead, so it is presented here, with the same table and the same
483+
// presenter (`@objectstack/core`) — never a second coercion.
484+
//
485+
// Keyed on each measure's DECLARED aggregate function, never on whether a
486+
// value looks numeric: `count` / `count_distinct` / `sum` / `avg` answer a
487+
// number whatever the column held; `min` / `max` answer a value OF the
488+
// column, so they are presented only when that column is declared numeric
489+
// (`driver-sql`'s `readPresentationKind` rule), asked through
490+
// `declaredFieldType` — a host that cannot answer, or a relationship-path
491+
// column, leaves the value as the client gave it. Expression metric types
492+
// (`number` / `string` / `boolean`) are the author's SQL and stay as they
493+
// are. Rows are presented in place, as the driver presents its own.
494+
const declaredType = (ctx as DatasetScopedStrategyContext).declaredFieldType;
495+
const numberMeasures = (query.measures ?? []).filter((member) => {
496+
const measure = this.lookupMember(cube, member, 'measure');
497+
if (!measure?.type || !Object.prototype.hasOwnProperty.call(AGGREGATE_ANSWER_KIND, measure.type)) return false;
498+
if (AGGREGATE_ANSWER_KIND[measure.type as AggregationFunction] === 'number') return true;
499+
const sourceType = typeof declaredType === 'function' ? declaredType.call(ctx, objectName, measure.sql) : undefined;
500+
return sourceType !== undefined && NUMERIC_VALUE_TYPES.has(sourceType);
501+
});
502+
if (numberMeasures.length > 0 && Array.isArray(rows)) {
503+
for (const row of rows) {
504+
if (!row || typeof row !== 'object') continue;
505+
for (const member of numberMeasures) {
506+
if (row[member] !== undefined) row[member] = presentAsNumber(row[member]);
507+
}
508+
}
509+
}
510+
470511
// Build field metadata
471512
const fields = this.buildFieldMeta(query, cube);
472513

0 commit comments

Comments
 (0)