Skip to content

Commit 1065660

Browse files
committed
test(driver-memory,driver-sql): pin $contains membership on every memory face; one seam; docblock and sweep
- memory-20874-contains-membership.test.ts: driver-sql's #17590 fixture row for row, plus the stored shapes SQLite decides (scalar, object, nested array, null/boolean/fractional members), through find() in both spellings, count(), the analytics query and its SQL echo executed on sql.js, and the nested-relation lowering's driver input. - driver-sql #17590 suite: a multi-valued lookup column and the u1/u10 case. - driver-memory: one public seam, filterContainsTest; the analytics echo reads its member set off the test the $match exit runs. - spec: the $contains docblock's stale driver-memory status and dead tracker link are replaced with the measured status. - the rest suite header and the pending #20802 changeset no longer state the in-memory gap. Claude-Session: https://claude.ai/code/session_01Ujdtvqs7ree7WyQmEDwEnG Co-authored-by: Claude <noreply@anthropic.com>
1 parent 07a0e7a commit 1065660

8 files changed

Lines changed: 483 additions & 102 deletions

‎.changeset/20802-nested-relation-filter-served.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,4 +25,4 @@ Measured through `POST /api/v1/data/:object/query` on SQLite and PostgreSQL 16 (
2525
| `{ $not: { owner: { region: "NA" } } }` | `INVALID_FILTER` / 400 | `d2`, `d4` |
2626
| `{ owner: { region: "APAC" } }` (no owner matches) | `INVALID_FILTER` / 400 | no rows |
2727

28-
On the in-memory driver, a multi-valued relation's `$contains` still matches a stored id by substring per element, so there an id that is a substring of another stored id (`u1` inside `u10`) also matches; SQLite and PostgreSQL match the element.
28+
SQLite, PostgreSQL and the in-memory driver match the element of a multi-valued relation, so an id that is a substring of another stored id (`u1` inside `u10`) does not match it.
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
"@objectstack/driver-memory": minor
3+
---
4+
5+
fix(driver-memory): `$contains` / `$notContains` on a multi-valued or JSON-stored field answer by membership, as the SQL drivers do
6+
7+
Clause-②: no
8+
9+
On a field whose declaration makes it JSON-stored (`multiple: true` on a `lookup`, `user`, `select`, `radio`, `file` or `image` field, a `multiselect`, `checkboxes` or `tags` field, or a structured type such as `json`), the in-memory driver now answers `{ field: { $contains: v } }` by whole-element membership: some element of the stored array equals `v`. It used to match each element by substring, so `u1` matched a row storing `['u10']` and `'red'` matched a row storing `['redwood']`. A number member answered nothing: `{ nums: { $contains: '1' } }` missed `[1, 2]`. `$notContains` is the exact complement, and a row with no value still satisfies it. A scalar text column keeps the case-exact substring test.
10+
11+
`driver-sql` gives the same answer on SQLite, PostgreSQL and MySQL; the two drivers were measured over the same fixture. The answer holds on every face of this driver:
12+
13+
- `find()` and `count()`, in both filter spellings;
14+
- the nested-relation filter on a multi-valued relation, which the engine lowers to one `$contains` per related id;
15+
- `MemoryAnalyticsService`'s query, and its SQL echo, which now renders SQLite's `json_each` membership construct for such a column.
16+
17+
The comparand is still a string. A number or boolean member is named by its text: `'1'` matches the stored number `1` (and `'1.50'` the number `1.5`), `'true'` matches the boolean `true`, and `'null'` matches a `null` member. A field the driver holds no declaration for, such as a field on an object never passed through `syncSchema`, keeps the substring reading.
18+
19+
New: `InMemoryDriver.filterContainsTest(object, field, value)` returns the one test every face above lowers `$contains` to. It is a narrow seam for the analytics face, beside `filterSubstringPattern` and `filterComparandStorageForm`. The added public method is why this entry is `minor`.
20+
21+
**If your tests relied on the old answer:** on the in-memory driver, a filter that matched an id by prefix or a tag by substring now returns only the member rows. That is what SQL already returned in production. Write `$contains` with the whole member value.

‎packages/drivers/driver-memory/src/memory-20874-contains-membership.test.ts‎

Lines changed: 339 additions & 0 deletions
Large diffs are not rendered by default.

‎packages/drivers/driver-memory/src/memory-analytics.ts‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ import {
2020
// call: `isFilterAST` gates the shape, `parseFilterAST` lowers it. See
2121
// {@link lowerWhereFilterArray}. ⛔ No second FilterArray parser in this face.
2222
import { isFilterAST, parseFilterAST, VALID_AST_OPERATORS } from '@objectstack/spec/data';
23-
import type { InMemoryDriver } from './memory-driver.js';
23+
import type { InMemoryDriver, MemoryContainsTest } from './memory-driver.js';
2424
import {
2525
Logger,
2626
createLogger,
@@ -238,9 +238,9 @@ interface MongoPredicateInput {
238238
/** The operands as authored. For operands that are not comparands. */
239239
readonly raw: readonly unknown[];
240240
/**
241-
* The whole predicate `$contains` (or, `negate`d, `$notContains`) becomes on
242-
* this member's column, built by the DRIVER's own rule
243-
* (`filterContainsPredicate`) rather than re-derived here.
241+
* The test `$contains` asks of this member's column, built by the DRIVER's
242+
* own rule (`filterContainsTest`) rather than re-derived here — `$notContains`
243+
* wraps it in `$not`, as the live query path does.
244244
*
245245
* [#7723] Case-EXACT, because that rule is: `filterSubstringPattern` carried
246246
* an `i` flag until #7723 took it off, putting the `$contains` family on the
@@ -254,7 +254,7 @@ interface MongoPredicateInput {
254254
* the predicate rather than a piece of it is what makes the membership reading
255255
* reach this face with the same edit, the lesson #7723 records above.
256256
*/
257-
readonly containment: (value: unknown, negate: boolean) => Record<string, unknown>;
257+
readonly containment: (value: unknown) => MemoryContainsTest;
258258
/**
259259
* [#6520] A comparand as an ASCII-case-insensitive literal-substring pattern —
260260
* `$icontains`' fold, which is NOT {@link substring}'s.
@@ -345,7 +345,7 @@ const CUBE_OPERATOR_TO_MONGO_PREDICATE: Readonly<Record<CubeOperator, MongoPredi
345345
notIn: ({ comparands }) => ({ $nin: [...comparands] }),
346346
// A pattern, not a comparand: `raw`, and the driver's own rule — membership on
347347
// a declared JSON-stored field, the substring everywhere else (#20874).
348-
contains: ({ raw, containment }) => containment(raw[0], false),
348+
contains: ({ raw, containment }) => containment(raw[0]),
349349
// [#6520] The case-INSENSITIVE twin, folding ASCII and nothing else. It takes
350350
// `asciiSubstring`, not `substring`: the neighbour above folds Unicode, so
351351
// reusing it here would answer `CAFÉ` for `café` on this face while the SQL
@@ -355,7 +355,7 @@ const CUBE_OPERATOR_TO_MONGO_PREDICATE: Readonly<Record<CubeOperator, MongoPredi
355355
// negation has to wrap a pattern, which is exactly what the live query path
356356
// builds for `$notContains` (`memory-driver.ts` `normalizeFieldOperators`) —
357357
// and, since #20874, the SAME test it builds, membership or substring.
358-
notContains: ({ raw, containment }) => containment(raw[0], true),
358+
notContains: ({ raw, containment }) => ({ $not: containment(raw[0]) }),
359359
// [#13195] A presence flag, not a comparand — and "present" means HAS A
360360
// VALUE (`!= null`), never key presence: #5298 leg 3 / #5369, landed in PR
361361
// #5962, ruled onto this face 2026-08-30. It used to emit `{$exists: <bool>}`
@@ -398,9 +398,9 @@ interface SqlPredicateInput {
398398
readonly globSubstring: (value: unknown) => string;
399399
/**
400400
* [#20874] The stored members a `$contains` comparand names on this column,
401-
* or `null` when the column asks the SUBSTRING question — the DRIVER's answer
402-
* (`filterContainsMembers`), the same one its `$match` twin executes. See
403-
* {@link sqliteMembershipPredicate}.
401+
* or `null` when the column asks the SUBSTRING question — read off the
402+
* DRIVER's test (`filterContainsTest`), the one its `$match` twin executes.
403+
* See {@link sqliteMembershipPredicate}.
404404
*/
405405
readonly members: (value: unknown) => readonly unknown[] | null;
406406
}
@@ -1665,7 +1665,7 @@ export class MemoryAnalyticsService implements IAnalyticsService {
16651665
const predicate = this.mongoPredicateBuilder(filter.operator)({
16661666
comparands: filter.values.map(storageForm),
16671667
raw: filter.values,
1668-
containment: (value, negate) => this.driver.filterContainsPredicate(table, fieldPath, value, negate),
1668+
containment: (value) => this.driver.filterContainsTest(table, fieldPath, value),
16691669
// [#6520] `$icontains`' fold, from the spec's shared definition rather
16701670
// than from the driver's Unicode-folding `filterSubstringPattern`.
16711671
asciiSubstring: (value) => new RegExp(asciiCaseInsensitiveRegexSource(String(value))),
@@ -1737,7 +1737,12 @@ export class MemoryAnalyticsService implements IAnalyticsService {
17371737
raw: entry.values,
17381738
literal: (value) => this.toSqlLiteral(value),
17391739
globSubstring: (value) => this.toSqlLiteral(globSubstringPattern(value)),
1740-
members: (value) => this.driver.filterContainsMembers(table, fieldPath, value),
1740+
members: (value) => {
1741+
// [#20874] Read off the very test the `$match` exit runs, so the echo
1742+
// and the chart can never name two different member sets.
1743+
const test = this.driver.filterContainsTest(table, fieldPath, value);
1744+
return '$elemMatch' in test ? test.$elemMatch.$in : null;
1745+
},
17411746
}));
17421747
}
17431748
return clauses;

‎packages/drivers/driver-memory/src/memory-driver.ts‎

Lines changed: 55 additions & 67 deletions
Original file line numberDiff line numberDiff line change
@@ -363,10 +363,10 @@ const JSON_NUMBER_TEXT = /^-?(?:0|[1-9][0-9]*)(?:\.[0-9]+)?(?:[eE][+-]?[0-9]+)?$
363363
* compares a stored element against, which is what lets the analytics face's
364364
* SQLite echo name the same set (`memory-analytics.ts`).
365365
*
366-
* @see InMemoryDriver.filterContainsMembers — where the population decides
366+
* @see InMemoryDriver.filterContainsTest — where the population decides
367367
* whether a comparand is read this way at all.
368368
*/
369-
export function containsMemberCandidates(value: unknown): Array<string | number | boolean | null> {
369+
function containsMemberCandidates(value: unknown): ContainsMember[] {
370370
const text = String(value);
371371
if (text === 'true') return [text, true];
372372
if (text === 'false') return [text, false];
@@ -378,6 +378,18 @@ export function containsMemberCandidates(value: unknown): Array<string | number
378378
return [text];
379379
}
380380

381+
/** [#20874] A stored member a `$contains` comparand can name. */
382+
type ContainsMember = string | number | boolean | null;
383+
384+
/**
385+
* [#20874] The un-negated test `$contains` lowers to on this driver —
386+
* membership on a declared JSON-stored field, the substring pattern elsewhere.
387+
* See {@link InMemoryDriver.filterContainsTest}.
388+
*/
389+
export type MemoryContainsTest =
390+
| { $elemMatch: { $in: ContainsMember[]; $not: { $type: 'array' } } }
391+
| { $regex: RegExp };
392+
381393
/**
382394
* [#20444] Each field's declared value shape — the `type` and `multiple` slice
383395
* `expandEmptyOperator` reads — for the fields that declare a `type`. The RAW
@@ -1483,10 +1495,10 @@ export class InMemoryDriver implements IDataDriver {
14831495
// was always literal, and that half was never the defect.
14841496
//
14851497
// [#20874] `contains` / `not_contains` take the ONE test the `$`-spelling
1486-
// takes ({@link containsTest}): membership on a declared JSON-stored
1498+
// takes ({@link filterContainsTest}): membership on a declared JSON-stored
14871499
// field, the case-exact substring everywhere else.
14881500
case 'contains':
1489-
return { [field]: this.containsTest(object, field, value) };
1501+
return { [field]: this.filterContainsTest(object, field, value) };
14901502
// [#7536] `like` / `ilike` are NOT `contains`, and sharing this arm with
14911503
// it was the memory-face twin of the wire defect #7536 closed: the
14921504
// comparand was regex-ESCAPED (so a caller's `%` matched a literal percent
@@ -1508,7 +1520,7 @@ export class InMemoryDriver implements IDataDriver {
15081520
};
15091521
}
15101522
case 'notcontains': case 'not_contains':
1511-
return { [field]: { $not: this.containsTest(object, field, value) } };
1523+
return { [field]: { $not: this.filterContainsTest(object, field, value) } };
15121524
case 'startswith': case 'starts_with':
15131525
return { [field]: { $regex: new RegExp(`^${this.escapeRegex(value)}`) } };
15141526
case 'endswith': case 'ends_with':
@@ -1693,7 +1705,7 @@ export class InMemoryDriver implements IDataDriver {
16931705
* refused — the vocabulary itself is enforced one level up (#5324).
16941706
* [#20874] `object` is carried for `$contains` / `$notContains` alone: which
16951707
* question they ask is the FIELD's declared storage shape
1696-
* ({@link containsTest}), and `field` names it only together with `object`.
1708+
* ({@link filterContainsTest}), and `field` names it only together with `object`.
16971709
*/
16981710
private normalizeFieldOperators(
16991711
ops: Record<string, any>,
@@ -1728,19 +1740,19 @@ export class InMemoryDriver implements IDataDriver {
17281740
// Unicode-folding `i` flag is gone.
17291741
//
17301742
// [#20874] `$contains` / `$notContains` ask the ONE test
1731-
// {@link containsTest} builds: MEMBERSHIP on a declared JSON-stored
1743+
// {@link filterContainsTest} builds: MEMBERSHIP on a declared JSON-stored
17321744
// field, the substring pattern everywhere else. The substring test
17331745
// still joins `regexConditions`, so it composes with `$startsWith` /
17341746
// `$endsWith` exactly as before; the membership test lowers to
17351747
// `$elemMatch`, a key no other operator writes.
17361748
case '$contains': {
1737-
const test = this.containsTest(object, field, val);
1749+
const test = this.filterContainsTest(object, field, val);
17381750
if ('$elemMatch' in test) put('$elemMatch', test.$elemMatch);
17391751
else regexConditions.push(test);
17401752
break;
17411753
}
17421754
case '$notContains':
1743-
put('$not', this.containsTest(object, field, val));
1755+
put('$not', this.filterContainsTest(object, field, val));
17441756
break;
17451757
case '$startsWith':
17461758
regexConditions.push({ $regex: new RegExp(`^${this.escapeRegex(val)}`) });
@@ -2303,7 +2315,7 @@ export class InMemoryDriver implements IDataDriver {
23032315
*
23042316
* [#20874] On a scalar column. A declared JSON-stored field asks MEMBERSHIP
23052317
* instead, so the analytics face now takes the whole predicate
2306-
* ({@link filterContainsPredicate}), which uses this for the substring half.
2318+
* ({@link filterContainsTest}), which uses this for the substring half.
23072319
*
23082320
* Same reasoning as {@link filterComparandStorageForm} one method up, on the
23092321
* other half of what a `contains` predicate needs. This driver's rule is
@@ -2375,71 +2387,47 @@ export class InMemoryDriver implements IDataDriver {
23752387
}
23762388

23772389
/**
2378-
* [#20874] The stored members a `$contains` / `$notContains` comparand names
2379-
* on `field` of `object` ({@link containsMemberCandidates}), or `null` when
2380-
* the field asks the SUBSTRING question instead
2381-
* ({@link isJsonStoredField}).
2382-
*
2383-
* Public for the analytics face's SQL echo, which has to RENDER the same
2384-
* question in SQLite's language — the same reason
2385-
* {@link filterComparandStorageForm} and {@link filterSubstringPattern} are
2386-
* public: a face that re-derived the population or the member reading would
2387-
* be a second place for the two faces to drift apart (#5374).
2388-
*/
2389-
filterContainsMembers(object: string | undefined, field: string, value: unknown): Array<string | number | boolean | null> | null {
2390-
return this.isJsonStoredField(object, field) ? containsMemberCandidates(value) : null;
2391-
}
2392-
2393-
/**
2394-
* [#20874] The mingo predicate `$contains` becomes on `field` of `object`, or
2395-
* — `negate`d — `$notContains`: the whole field predicate, for the analytics
2396-
* (cube) face, built by the one rule the live query path uses
2397-
* ({@link containsTest}).
2398-
*
2399-
* Returned whole, not as a pattern, because the two readings do not share a
2400-
* shape: the substring reading is a `$regex`, the membership reading an
2401-
* `$elemMatch`. The analytics face used to take {@link filterSubstringPattern}
2402-
* and wrap it in `$regex` itself, which is how it inherited the per-element
2403-
* substring answer on a stored array along with the query path.
2404-
*/
2405-
filterContainsPredicate(object: string | undefined, field: string, value: unknown, negate: boolean): Record<string, unknown> {
2406-
const test = this.containsTest(object, field, value);
2407-
return negate ? { $not: test } : test;
2408-
}
2409-
2410-
/**
2411-
* [#20874] The one un-negated test behind every spelling of `$contains` on
2412-
* this package — the `$`-spelling and the AST spelling of the query path, and
2413-
* the analytics face through {@link filterContainsPredicate}.
2390+
* [#20874] The one un-negated test `$contains` asks of `field` on `object` —
2391+
* behind every spelling of the operator on this package: the `$`-spelling and
2392+
* the AST spelling of the query path, and the analytics face, which wraps it
2393+
* in `$not` for `$notContains` exactly as the query path does and renders its
2394+
* SQL echo from the members it names.
24142395
*
2415-
* - **A declared JSON-stored field → MEMBERSHIP**: some element of the stored
2416-
* array IS one of the comparand's candidates. `$elemMatch` is array-only by
2417-
* construction, so a stored scalar, an object or `null` has no member — the
2418-
* answer `driver-sql` gives on every dialect (its constructs are array-only
2419-
* too). The `$not: { $type: 'array' }` clause keeps a NESTED array out: mingo
2420-
* applies `$in` through an element that is itself an array, so `[['u1']]`
2421-
* would otherwise answer `'u1'`, where SQLite compares the element's JSON
2422-
* text `["u1"]` and does not. Measured on mingo 7.2 over one fixture: with
2423-
* the clause, this and SQLite answer the same rows on every shape, `[null]`
2424-
* and `[[null]]` included.
2425-
* - **Anything else → the case-exact literal SUBSTRING** (#6682), unchanged.
2396+
* - **A declared JSON-stored field ({@link isJsonStoredField}) → MEMBERSHIP**:
2397+
* `{ $elemMatch: { $in: members, … } }` — some element of the stored array
2398+
* IS one of the members the comparand names ({@link containsMemberCandidates}).
2399+
* `$elemMatch` is array-only by construction, so a stored scalar, an object
2400+
* or `null` has no member — the answer `driver-sql` gives on every dialect
2401+
* (its constructs are array-only too). The `$not: { $type: 'array' }` clause
2402+
* keeps a NESTED array out: mingo applies `$in` through an element that is
2403+
* itself an array, so `[['u1']]` would otherwise answer `'u1'`, where SQLite
2404+
* compares the element's JSON text `["u1"]` and does not. Measured on mingo
2405+
* 7.2 against `driver-sql`/SQLite over one fixture: with the clause the two
2406+
* answer the same rows on every shape, `[null]` and `[[null]]` included.
2407+
* - **Anything else → `{ $regex }`, the case-exact literal SUBSTRING** (#6682),
2408+
* {@link filterSubstringPattern} unchanged.
24262409
*
2427-
* The negation composes with the NULL rule rather than replacing it: `$not`
2410+
* Negated, it composes with the NULL rule rather than replacing it: `$not`
24282411
* over either test admits a row whose field is null or missing (#5298), which
24292412
* is `driver-sql`'s `col IS NULL OR NOT (…)`.
24302413
*
2414+
* Public for the analytics face, for the reason {@link filterComparandStorageForm}
2415+
* and {@link filterSubstringPattern} are: a face that re-derived the population
2416+
* or the member reading would be a second place for the two faces to drift
2417+
* apart (#5374). Returned whole rather than as a pattern because the two
2418+
* readings do not share a shape — the analytics face used to take
2419+
* {@link filterSubstringPattern} and wrap it in a `$regex` of its own, which is
2420+
* how it inherited the per-element substring answer along with the query path.
2421+
*
24312422
* ⚠️ On a field this driver holds no declaration for, the substring pattern
24322423
* still reaches mingo, which applies a `$regex` to each element of an array
2433-
* value. That residue is the population's deliberate `false` above, the same
2434-
* one `driver-sql` keeps for a table it was never told about.
2424+
* value — the population's deliberate `false`, the same one `driver-sql` keeps
2425+
* for a table it was never told about.
24352426
*/
2436-
private containsTest(
2437-
object: string | undefined,
2438-
field: string,
2439-
value: unknown,
2440-
): { $elemMatch: Record<string, unknown> } | { $regex: RegExp } {
2441-
const members = this.filterContainsMembers(object, field, value);
2442-
if (members) return { $elemMatch: { $in: members, $not: { $type: 'array' } } };
2427+
filterContainsTest(object: string | undefined, field: string, value: unknown): MemoryContainsTest {
2428+
if (this.isJsonStoredField(object, field)) {
2429+
return { $elemMatch: { $in: containsMemberCandidates(value), $not: { $type: 'array' } } };
2430+
}
24432431
return { $regex: this.filterSubstringPattern(value) };
24442432
}
24452433

0 commit comments

Comments
 (0)