diff --git a/.changeset/engine-internal-field-evaluation-gate.md b/.changeset/engine-internal-field-evaluation-gate.md new file mode 100644 index 00000000000..2c7abd45421 --- /dev/null +++ b/.changeset/engine-internal-field-evaluation-gate.md @@ -0,0 +1,30 @@ +--- +'@objectstack/spec': patch +'@objectstack/objectql': patch +'@objectstack/metadata-protocol': patch +'@objectstack/runtime': patch +'@objectstack/mcp': patch +--- + +fix(objectql)!: the engine refuses filtering, sorting, grouping and aggregating on an `internal: true` field for every caller but a system context + +Clause-②: yes (narrowing) +Metadata-protocol: none + + +Skills-docs: updated here — `content/docs/permissions/system-context.mdx` (row 18e: the engine's read path) and `content/docs/api/error-catalog.mdx` (`INVALID_FIELD`) state who is refused and who is served; no file under `skills/**` teaches filtering on an `internal: true` field, so no skill line changes. + +**BREAKING** for one shape only: server code that reads the engine as a user (not as a system context) and filters, sorts, groups or aggregates on a field declared `internal: true`. + +- FROM: the engine ran such a read. +- TO: the engine refuses it with the data door's `400 INVALID_FIELD`, naming the object, the field and the position, before anything is read. A read made as a system context (`isSystem: true`) is served as before, and the flagged column is still never returned on a row. + +What to do: run a server-side lookup that matches on a flagged field as a system context, or match on another field. + +- **`@objectstack/objectql`**: the `internal: true` evaluate refusal is defined once, in the engine. `find`, `findOne`, `count` and `aggregate` apply it to every caller but a system context, in every position the data door judges: a filter (including a relation hop and a `$field` comparand), a sort, a group key, an aggregation operand and an aggregation's own filter, and the same positions inside `expand`. `ObjectQL#assertNoInternalFieldEvaluated` answers the same question for any caller, for a door that judges before it reads. +- **`@objectstack/spec`**: new types on the `objectql` slot contract: `QueryEvaluationGate`, `InternalFieldEvaluationQuery` and `InternalFieldEvaluationOptions`. `IObjectQLEngine` extends `QueryEvaluationGate`. +- **`@objectstack/metadata-protocol`**: the data door asks the engine's refusal instead of keeping its own copy. Its answers are unchanged. +- **`@objectstack/runtime`**: the HTTP MCP data bridge asks the engine's refusal directly. Its answers are the data door's. +- **`@objectstack/mcp`**: the stdio data bridge's own copy of the refusal is retired in favour of the engine's. Its answers are the data door's. + +The level is patch rather than the minor a new spec type would take: the changeset gate refuses `minor` beside `Metadata-protocol: none`, and nothing an author writes is refused. diff --git a/content/docs/api/error-catalog.mdx b/content/docs/api/error-catalog.mdx index 469fc7d2421..7ea410f0d5e 100644 --- a/content/docs/api/error-catalog.mdx +++ b/content/docs/api/error-catalog.mdx @@ -158,7 +158,10 @@ when a read names it in a position that would evaluate its stored value: a filte aggregation's own `filter`), a sort, a `groupBy`, an `aggregations[].field`, a `searchFields` entry, or an `expand` entry's own filter or sort at any depth. Such a field is never returned on a generic read, so it cannot be queried by either — the -refusal names the object that declares it. Query by another field. +refusal names the object that declares it. Query by another field. The engine +itself gives the same refusal on its own reads for every caller but a system +context; server code that matches on such a field runs its lookup as a system +context. A non-blank `search` is refused (`param: 'search'`), not answered with every row, when the object's resolved search set is empty: no `searchableFields` entry names a stored field, and the default search scans none (`internal: true` fields never diff --git a/content/docs/permissions/system-context.mdx b/content/docs/permissions/system-context.mdx index ca25bd2c3c3..144fc551e8c 100644 --- a/content/docs/permissions/system-context.mdx +++ b/content/docs/permissions/system-context.mdx @@ -123,6 +123,7 @@ bugs — it is the protection or the side effect that silently does not happen. | 18b | Settings namespace read scope not applied | service-settings | Get: a read of `sys_setting`, `sys_setting_audit` or `sys_platform_setting` across every namespace, whatever each namespace's `readPermission` names — the settings service's own reads of its stores take this path. Lose: the per-namespace read gate the generic data API applies to every other caller, the same rule the settings routes enforce | `settingsReadDoorMiddleware` | | 18c | **Credential-column filter / sort refusal not applied** — a `where`, `orderBy`, per-aggregation `filter` or `expand` entry naming a `password` / `secret` column is evaluated | objectql | Get: the server's own match on a stored credential value, a lookup that never returns it. Every other caller, administrators included, is refused `400 INVALID_FIELD` for anything but a has-a-value check (ADR-0100). Lose: nothing a wire caller reaches — `isSystem` is never set on an inbound request | `rejectCredentialEvaluation` | | 18d | **Stored-metadata type read gate not applied** — a read of `sys_metadata` / `sys_metadata_history` is not narrowed by the capability its row's type requires | objectql | Get: every row of the family on `find` / `findOne` / `count` / `aggregate` — the metadata loader, the `/meta` doors' own store reads, boot (a read with no execution context is not narrowed either). Lose: the narrowing that serves a stored row of a capability-gated type (`datasource`, `external_catalog`) only to holders of the capability the `/meta` type gate names for it | `storedMetadataTypeReadFilter` | +| 18e | **`internal: true` evaluate refusal not applied on the engine's read path** — a filter, sort, group key, aggregation operand, per-aggregation `filter`, nested-relation condition or `expand` entry naming a field declared `internal: true` is evaluated | objectql | Get: the server's own match on a flagged column (the API-key, share-link, SCIM-credential and approval-token lookups, the better-auth adapter), which still never returns the value. Every other caller of `find` / `findOne` / `count` / `aggregate`, administrators included, is refused `400 INVALID_FIELD`; the data door asks the same judgement for every caller, a system context included. Lose: nothing a wire caller reaches — `isSystem` is never set on an inbound request | `rejectInternalFieldEvaluation` | ### 2. Write pipeline and data integrity @@ -544,6 +545,7 @@ and CI refuses one that differs from what the tree produces. | `packages/objectql/src/engine.ts#insert` | 3 | 21, 22 | | `packages/objectql/src/engine.ts#isSystem` | 1 | 64 | | `packages/objectql/src/engine.ts#update` | 4 | 19, 20 | +| `packages/objectql/src/internal-field-evaluation-guard.ts#rejectInternalFieldEvaluation` | 1 | 18e | | `packages/objectql/src/stored-metadata-type-read-gate.ts#storedMetadataTypeReadFilter` | 1 | 18d | | `packages/observability/src/perf-timing.ts#isPerfDisclosurePrincipal` | 1 | 13 | | `packages/plugins/organizations/src/organizations-plugin.ts#start` | 1 | 62 | diff --git a/packages/core/src/security/second-object-read-exposure.pin.test.ts b/packages/core/src/security/second-object-read-exposure.pin.test.ts index 164e77446fe..d38927c3a8e 100644 --- a/packages/core/src/security/second-object-read-exposure.pin.test.ts +++ b/packages/core/src/security/second-object-read-exposure.pin.test.ts @@ -218,6 +218,16 @@ const CALLERS: Record = { "serves nothing of the target — the expanded read itself is the engine's `expandRelatedRecords`. Pinned by " + '`packages/objectql/src/credential-evaluation-guard.test.ts` (the expand refusals assert the driver was never asked)', }, + // The `internal: true` evaluate refusal's expand half (QueryEvaluationGate). + 'packages/objectql/src/internal-field-evaluation-guard.ts': { + kind: 'not-a-served-read', + reason: + "names the object an `expand` entry the caller wrote targets, only to read that object's DECLARED `internal: true` " + + 'fields from the registry and refuse (`INVALID_FIELD` / 400) a filter or sort key on one; it reads no record and ' + + "serves nothing of the target. A door's own exposure question runs first at each hop, through the gate's " + + '`beforeRelationTarget` hook (the data door hands it `refuseUnservedRelationTarget`). Pinned by ' + + '`packages/objectql/src/internal-field-evaluation-guard.test.ts` (the refusals assert the driver was never asked)', + }, // [#25522 cut 10] The write-path reference check, moved out of engine.ts. 'packages/objectql/src/engine-referential-integrity.ts': { kind: 'not-a-served-read', diff --git a/packages/mcp/src/stdio-data-bridge.internal-field-evaluation.test.ts b/packages/mcp/src/stdio-data-bridge.internal-field-evaluation.test.ts index 6d9b47c3627..5e74146cbab 100644 --- a/packages/mcp/src/stdio-data-bridge.internal-field-evaluation.test.ts +++ b/packages/mcp/src/stdio-data-bridge.internal-field-evaluation.test.ts @@ -1,20 +1,20 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. /** - * The stdio data bridge asks the data door's internal-field gate — the - * `protocol` service's `assertNoInternalFieldEvaluated` — before its engine - * reads anything, on `query` (the call's `where` and `orderBy`) and on - * `aggregate` (`where`, `groupBy`, `aggregations`), and a refusal it throws - * reaches the caller as thrown: the tool renders its `code`, `status` and - * sentence, and the engine is never asked. + * The stdio data bridge leaves the `internal: true` evaluate refusal to the + * ENGINE: it hands the engine the caller's query under the caller's principal + * on `query` (`where`, `orderBy`) and on `aggregate` (`where`, `groupBy`, + * `aggregations`), never asks a `protocol` service for it (a host may register + * none, or a protocol other than the shipped one), and a refusal the engine + * throws reaches the caller as thrown: the tool renders its `code`, `status` + * and sentence. * - * The gate itself is the protocol's and is pinned there - * (`@objectstack/metadata-protocol`, the public-face cases beside the data - * door's position enumeration), and the whole path on a real stack in - * `@objectstack/dogfood`. This file pins the bridge's half: that it asks, with - * what, before what — and that a host whose protocol has no such face (or that - * registers no protocol) is served exactly as before. Refusals are asserted on - * the envelope, never on a bare throw. + * The refusal itself is the engine's one definition and is pinned there + * (`@objectstack/objectql`, `internal-field-evaluation-guard.test.ts`, which + * also boots a kernel with no protocol service), and the whole path on a real + * stack, with and without a protocol service, in `@objectstack/dogfood`. This + * file pins the bridge's half. Refusals are asserted on the envelope, never on + * a bare throw. */ import { describe, it, expect, vi } from 'vitest'; @@ -23,40 +23,41 @@ import { Client } from '@modelcontextprotocol/sdk/client/index.js'; import { InMemoryTransport } from '@modelcontextprotocol/sdk/inMemory.js'; import type { ExecutionContext } from '@objectstack/spec/kernel'; import type { IDataEngine, IMetadataService } from '@objectstack/spec/contracts'; -import { createStdioDataBridge, type StdioProtocolObjectReads } from './stdio-data-bridge.js'; +import { createStdioDataBridge } from './stdio-data-bridge.js'; import { registerObjectTools } from './mcp-http-tools.js'; const OBJECT = 'ifx_vault'; const FLAGGED = 'pin'; const DEF = { name: OBJECT, fields: { name: { type: 'text' }, [FLAGGED]: { type: 'text' } } }; -/** The door's refusal as the protocol throws it (fields only; the sentence is the protocol's to pin). */ -function doorRefusal(param: string) { - return Object.assign(new Error(`Cannot evaluate '${OBJECT}' by '${FLAGGED}' (${param}): the query was not run.`), { +/** The engine's refusal (fields only; the sentence is the engine's to pin). */ +function engineRefusal(param: string) { + return Object.assign(new Error(`Cannot filter '${OBJECT}' by '${FLAGGED}' (${param}): the query was not run.`), { code: 'INVALID_FIELD', status: 400, httpStatus: 400, object: OBJECT, field: FLAGGED, fields: [FLAGGED], param, }); } -/** A stand-in for the protocol's gate: refuses any query whose JSON names the flagged field. */ -function gate() { - return vi.fn((_object: string, query: Record) => { - if (JSON.stringify(query).includes(`"${FLAGGED}"`)) throw doorRefusal('filter'); - }); -} +const PRINCIPAL = { userId: 'u1', isSystem: false } as unknown as ExecutionContext; -function makeBridge(protocol?: StdioProtocolObjectReads) { +/** A stand-in engine that refuses any query whose JSON names the flagged field, as the real one does. */ +function makeBridge(withProtocolFace = false) { + const refuses = (query: unknown) => { + if (JSON.stringify(query).includes(`"${FLAGGED}"`)) throw engineRefusal('filter'); + }; const engine = { - find: vi.fn(async () => [{ id: 'r1', name: 'alpha' }]), - aggregate: vi.fn(async () => [{ n: 1 }]), + find: vi.fn(async (_object: string, query: unknown) => { refuses(query); return [{ id: 'r1', name: 'alpha' }]; }), + aggregate: vi.fn(async (_object: string, query: unknown) => { refuses(query); return [{ n: 1 }]; }), }; const metadataService = { getObject: vi.fn(async (name: string) => (name === OBJECT ? DEF : undefined)) }; + // A protocol that still carries a face of that name: the bridge must not ask it. + const protocolFace = vi.fn(); const bridge = createStdioDataBridge({ engine: engine as unknown as IDataEngine, metadataService: metadataService as unknown as IMetadataService, - ...(protocol ? { protocol: () => protocol } : {}), - resolvePrincipal: async () => ({ userId: 'u1', isSystem: false }) as unknown as ExecutionContext, + ...(withProtocolFace ? { protocol: () => ({ assertNoInternalFieldEvaluated: protocolFace }) as never } : {}), + resolvePrincipal: async () => PRINCIPAL, }); - return { bridge, engine }; + return { bridge, engine, protocolFace }; } const refusalOf = async (run: () => Promise) => { @@ -66,24 +67,22 @@ const refusalOf = async (run: () => Promise) => { }; const REFUSED = { code: 'INVALID_FIELD', status: 400, field: FLAGGED, object: OBJECT }; -describe('the stdio bridge asks the protocol\'s internal-field gate before the engine', () => { - it('query: the gate is asked with the call\'s where and orderBy, and its refusal stops the read', async () => { - const assertNoInternalFieldEvaluated = gate(); - const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated }); +describe('the stdio bridge leaves the internal-field refusal to the engine', () => { + it('query: the engine gets the call\'s where and orderBy under the principal, and its refusal reaches the caller', async () => { + const { bridge, engine } = makeBridge(); const where = { [FLAGGED]: { $startsWith: 'x' } }; const orderBy = [{ field: 'name', order: 'asc' as const }]; expect(await refusalOf(() => bridge.query(OBJECT, { where, orderBy, limit: 5 }))).toEqual(REFUSED); - expect(assertNoInternalFieldEvaluated).toHaveBeenCalledWith(OBJECT, { where, orderBy }); - expect(engine.find).not.toHaveBeenCalled(); + expect(engine.find).toHaveBeenCalledTimes(1); + expect(engine.find.mock.calls[0]![1]).toMatchObject({ where, orderBy }); + expect((engine.find.mock.calls[0] as unknown[])[2]).toEqual({ context: PRINCIPAL }); const sorted = [{ field: FLAGGED, order: 'desc' as const }]; expect(await refusalOf(() => bridge.query(OBJECT, { orderBy: sorted }))).toEqual(REFUSED); - expect(engine.find).not.toHaveBeenCalled(); }); - it('aggregate: the gate is asked with where, groupBy and aggregations, and its refusal stops the aggregate', async () => { - const assertNoInternalFieldEvaluated = gate(); - const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated }); + it('aggregate: the engine gets where, groupBy and aggregations under the principal, and its refusal reaches the caller', async () => { + const { bridge, engine } = makeBridge(); const bodies = [ { where: { [FLAGGED]: 'x' }, aggregations: [{ function: 'count' as const, alias: 'n' }] }, { groupBy: [FLAGGED], aggregations: [{ function: 'count' as const, alias: 'n' }] }, @@ -92,39 +91,31 @@ describe('the stdio bridge asks the protocol\'s internal-field gate before the e ]; for (const body of bodies) { expect(await refusalOf(() => bridge.aggregate!(OBJECT, body))).toEqual(REFUSED); - expect(assertNoInternalFieldEvaluated).toHaveBeenLastCalledWith(OBJECT, { - where: (body as { where?: unknown }).where, - groupBy: (body as { groupBy?: unknown }).groupBy, - aggregations: body.aggregations, - }); + expect(engine.aggregate.mock.lastCall![1]).toMatchObject({ ...body, context: PRINCIPAL }); } - expect(engine.aggregate).not.toHaveBeenCalled(); }); - it('CONTROL — a query the gate passes reaches the engine unchanged, on both verbs', async () => { - const assertNoInternalFieldEvaluated = gate(); - const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated }); - const where = { name: 'alpha' }; - await bridge.query(OBJECT, { where, orderBy: [{ field: 'name', order: 'asc' }] }); + it('never asks a protocol service for it, even one carrying a face of that name', async () => { + const { bridge, engine, protocolFace } = makeBridge(true); + expect(await refusalOf(() => bridge.query(OBJECT, { where: { [FLAGGED]: 'x' } }))).toEqual(REFUSED); + expect(await refusalOf(() => bridge.aggregate!(OBJECT, { groupBy: [FLAGGED], aggregations: [{ function: 'count', alias: 'n' }] }))).toEqual(REFUSED); + expect(protocolFace).not.toHaveBeenCalled(); expect(engine.find).toHaveBeenCalledTimes(1); - expect((engine.find.mock.calls[0] as unknown[])[1]).toMatchObject({ where, orderBy: [{ field: 'name', order: 'asc' }] }); - await bridge.aggregate!(OBJECT, { where, groupBy: ['name'], aggregations: [{ function: 'count', alias: 'n' }] }); expect(engine.aggregate).toHaveBeenCalledTimes(1); - expect(assertNoInternalFieldEvaluated).toHaveBeenCalledTimes(2); }); - it('CONTROL — no protocol, or one without the face: served as before (no door to agree with)', async () => { - for (const protocol of [undefined, {} as StdioProtocolObjectReads]) { - const { bridge, engine } = makeBridge(protocol); - await bridge.query(OBJECT, { where: { [FLAGGED]: 'x' } }); - await bridge.aggregate!(OBJECT, { where: { [FLAGGED]: 'x' }, aggregations: [{ function: 'count', alias: 'n' }] }); - expect(engine.find).toHaveBeenCalledTimes(1); - expect(engine.aggregate).toHaveBeenCalledTimes(1); - } + it('CONTROL — an ordinary-field query is served by the engine, on both verbs', async () => { + const { bridge, engine } = makeBridge(); + const where = { name: 'alpha' }; + const rows = await bridge.query(OBJECT, { where, orderBy: [{ field: 'name', order: 'asc' }] }); + expect(engine.find).toHaveBeenCalledTimes(1); + expect(JSON.stringify(rows)).toContain('alpha'); + await bridge.aggregate!(OBJECT, { where, groupBy: ['name'], aggregations: [{ function: 'count', alias: 'n' }] }); + expect(engine.aggregate).toHaveBeenCalledTimes(1); }); - it('query_records and aggregate_records render the gate\'s refusal as its envelope', async () => { - const { bridge, engine } = makeBridge({ assertNoInternalFieldEvaluated: gate() }); + it('query_records and aggregate_records render the engine\'s refusal as its envelope', async () => { + const { bridge } = makeBridge(); const server = new McpServer({ name: 'pin', version: '0' }); registerObjectTools(server, bridge); const [clientSide, serverSide] = InMemoryTransport.createLinkedPair(); @@ -140,10 +131,8 @@ describe('the stdio bridge asks the protocol\'s internal-field gate before the e expect(res.isError, name).toBe(true); const { error } = JSON.parse(res.content[0]!.text) as { error: { code: unknown; status: unknown; message: string } }; expect({ code: error.code, status: error.status }).toEqual({ code: 'INVALID_FIELD', status: 400 }); - expect(error.message).toBe(doorRefusal('filter').message); + expect(error.message).toBe(engineRefusal('filter').message); } - expect(engine.find).not.toHaveBeenCalled(); - expect(engine.aggregate).not.toHaveBeenCalled(); } finally { await client.close(); } diff --git a/packages/mcp/src/stdio-data-bridge.ts b/packages/mcp/src/stdio-data-bridge.ts index 6395c8db39b..d74fec244d3 100644 --- a/packages/mcp/src/stdio-data-bridge.ts +++ b/packages/mcp/src/stdio-data-bridge.ts @@ -93,16 +93,18 @@ * * ⚠️ A THIRD limb was security as well: the EVALUATE positions of a field * declared `internal: true`. The engine omits such a field from every row, but - * it still evaluates one named in a filter or a sort, and the protocol's data - * door is where those positions are refused — for every caller, administrators - * included, before the engine is asked. This engine-only bridge evaluated them. - * `query` and `aggregate` now ask that door's ONE gate of the `protocol` - * service before the engine ({@link refuseInternalFieldEvaluation}), so a - * filter, a sort, a group key or an aggregation over such a field answers the - * door's `400 INVALID_FIELD`, word for word, on this transport as on REST and - * on the HTTP bridge. ⛔ Never a copy of that gate here: a second definition is - * the fork this note exists to close. What remains of the divergence above is - * genuinely not security. + * it would still evaluate one named in a filter or a sort. Those positions are + * refused by the ENGINE itself now, on its own read path, for every caller but + * a system context (`QueryEvaluationGate`, `@objectstack/objectql`'s + * `internal-field-evaluation-guard.ts`) — the one definition the protocol's + * data door asks too. This transport's principal is never a system context + * (it is admitted from an API key alone), so `query` and `aggregate` hand the + * engine the caller's query and a filter, a sort, a group key or an + * aggregation over such a field answers the door's `400 INVALID_FIELD`, word + * for word, on this transport as on REST and on the HTTP bridge — whether or + * not the host registers a `protocol` service. ⛔ Never a copy of that gate + * here: a second definition is the fork this note exists to close. What + * remains of the divergence above is genuinely not security. */ import { omitInternalFieldsFromWriteResponse, relationConditionSites } from '@objectstack/core'; @@ -164,14 +166,6 @@ export interface StdioProtocolObjectReads { getMetaItem?(request: { type: string; name: string }): Promise; /** The by-name read without the serving decorations. */ getMetaItemForExecution?(request: { type: string; name: string }): Promise; - /** - * The data door's evaluate refusal for a field declared `internal: true`: - * throws its `400 INVALID_FIELD` when `query` filters, sorts, groups or - * aggregates on one, and returns otherwise. Not a read, but asked of the - * same service at the same moment, so it rides this slot - * ({@link refuseInternalFieldEvaluation}). - */ - assertNoInternalFieldEvaluated?(object: string, query: Record): void; } /** [#25138] Where this transport reads an object's definition from. */ @@ -572,38 +566,6 @@ async function enforceRelationTargetExposure( } } -/** - * The data door's evaluate refusal for a field declared `internal: true`, - * asked of the `protocol` service before the engine reads anything: the - * protocol's own `assertNoInternalFieldEvaluated`, the gate `findData` asks - * for REST and for the HTTP bridge's `query_records` — ⛔ never a copy of it - * here (route-ownership rule 1). It throws that door's `400 INVALID_FIELD`, - * naming the object, the field and the position in the door's own sentence, - * for a filter (`where`, each aggregation's own `filter`, a nested-relation - * condition judged on its target), a sort, a group key or an aggregation - * operand on such a field — for every principal this transport admits. A - * query over ordinary fields passes untouched. - * - * Asked whatever the caller's context, as the door asks it: the engine's - * privileged lookups that legitimately match on such a field call the engine - * directly and never reach this bridge. - * - * Structural absence answers nothing: on a host that registers no protocol, - * or one whose protocol has no such face, there is no data door whose verdict - * this transport could match, and the call proceeds to the engine as before. - * An object the protocol's registry does not hold is likewise left to the - * engine's existence answer, as the door leaves it. - */ -function refuseInternalFieldEvaluation( - reads: StdioObjectReads, - object: string, - query: Record, -): void { - const protocol = protocolOf(reads); - if (typeof protocol?.assertNoInternalFieldEvaluated !== 'function') return; - protocol.assertNoInternalFieldEvaluated(object, query); -} - // --------------------------------------------------------------------------- // [#21207] The stored-metadata-body family, at this engine-only reader // --------------------------------------------------------------------------- @@ -1021,9 +983,9 @@ export function createStdioDataBridge(deps: StdioDataBridgeDeps): McpDataBridge // [#21207] …and the same for a stored content-hash column. const hashRefusal = storedMetadataHashRefusal(object, { where: opts?.where, orderBy: opts?.orderBy }); if (hashRefusal) throw hashRefusal; - // …and a filter or sort on an `internal: true` field, by the data door's - // own gate, after the stored-metadata refusals as the door orders them. - refuseInternalFieldEvaluation(reads, object, { where: opts?.where, orderBy: opts?.orderBy }); + // A filter or sort on an `internal: true` field is refused by the engine + // itself (see the module header), after the refusals above, as the + // data door orders them. const read = storedMetadataBodyReadFields(object, opts?.fields); const query: Record = {}; if (opts?.where) query.where = opts.where; @@ -1178,14 +1140,9 @@ export function createStdioDataBridge(deps: StdioDataBridgeDeps): McpDataBridge aggregations: opts?.aggregations, }); if (hashRefusal) throw hashRefusal; - // …and a filter (`where` or an aggregation's own), a group key or an - // aggregation operand on an `internal: true` field, by the data door's - // own gate ({@link refuseInternalFieldEvaluation}). - refuseInternalFieldEvaluation(reads, object, { - where: opts?.where, - groupBy: opts?.groupBy, - aggregations: opts?.aggregations, - }); + // A filter (`where` or an aggregation's own), a group key or an + // aggregation operand on an `internal: true` field is refused by the + // engine itself (see the module header). // No casts: `McpDataBridge.aggregate` declares the engine's own // `EngineAggregateOptions` slices since #8032, so the honest call // compiles — the two `as unknown as` casts this line used to carry diff --git a/packages/metadata-protocol/src/host-engine.ts b/packages/metadata-protocol/src/host-engine.ts index 06f4ecae8f2..8e582a6b6c5 100644 --- a/packages/metadata-protocol/src/host-engine.ts +++ b/packages/metadata-protocol/src/host-engine.ts @@ -32,6 +32,17 @@ export interface MetadataHostEngine extends IDataEngine { * cannot lose atomicity silently fails closed (see `batchData`, ADR-0119 D4). */ transaction?: IObjectQLEngine['transaction']; + /** + * The engine's `internal: true` evaluate refusal (`QueryEvaluationGate` on the + * `objectql` slot contract), which the data door asks before the engine reads + * anything (`assertNoInternalFieldEvaluated` on the protocol). Typed off the + * slot contract for the reason `transaction` is; optional HERE for the same + * reason too. Degrading means judging no field at the door: the flag is the + * engine's to honour, and an engine without the gate does not withhold the + * column from its rows either. The door's relation-target exposure question + * does not degrade: it is asked at every hop either way. + */ + assertNoInternalFieldEvaluated?: IObjectQLEngine['assertNoInternalFieldEvaluated']; // Protocol accesses additional engine members structurally; keep it permissive // for this relocation (behavior unchanged — the concrete engine is injected). [key: string]: any; diff --git a/packages/metadata-protocol/src/protocol-query-validation.ts b/packages/metadata-protocol/src/protocol-query-validation.ts index 3028c46e833..471290bebed 100644 --- a/packages/metadata-protocol/src/protocol-query-validation.ts +++ b/packages/metadata-protocol/src/protocol-query-validation.ts @@ -7,8 +7,11 @@ * every field a query names exists, can be sorted, projected, expanded, * searched, grouped or aggregated (`assert*FieldsExist` and their siblings), * and no internal field is evaluated or served (`assertNoInternalFieldEvaluated`, - * the `refuseInternal*` walks, `servedExpand`) — moved out of `protocol.ts` - * whole. + * `servedExpand`) — moved out of `protocol.ts` whole. `assertNoInternalFieldEvaluated` + * has since become a delegation: the walk it held is the engine's + * (`QueryEvaluationGate`, `@objectstack/objectql`'s + * `internal-field-evaluation-guard.ts`), and only this door's relation-target + * exposure question is still handed in from here. * * A MOVE, not a redesign. Each function below is the body of the * `ObjectStackProtocolImplementation` method of the same name as it stood, @@ -26,8 +29,8 @@ * body that reads no member takes no host and is called directly. A method is * handed over as a closure that calls it on the protocol at call time, so a * test double or spy set on the instance is still the one the body reaches, - * and the moved bodies that call one another (the `refuseInternal*` walks - * recurse) do so through the protocol, exactly as before. + * and the moved bodies that call one another (`servedExpand` recurses) do so + * through the protocol, exactly as before. * * The one field the bodies read, `engine`, goes as a GETTER, so each read * happens when the body makes it: tests replace it on a constructed protocol. @@ -43,7 +46,7 @@ */ import type { ObjectStackProtocolImplementation } from './protocol.js'; -import { FILTER_LOGICAL_KEYS, ODATA_SPELLING, QUERY_PARAM_NEAR_MISS, collectFilterReads, invalidSortError, isObjectOfDisabledPackage, isPlainRecord, nearMissKey, objectPackageDisabledError, quotedQueryValue, suggestFieldName } from './protocol.js'; +import { FILTER_LOGICAL_KEYS, ODATA_SPELLING, QUERY_PARAM_NEAR_MISS, invalidSortError, isObjectOfDisabledPackage, isPlainRecord, nearMissKey, objectPackageDisabledError, quotedQueryValue, suggestFieldName } from './protocol.js'; import { objectNotFoundError, relationConditionSites } from '@objectstack/core'; import { collectInternalWriteResponseFields } from './write-response-internal-fields.js'; import { AggregationFunction, DateGranularity, REFERENCE_VALUE_TYPES, SEARCHABLE_ENUM_TYPES, SEARCHABLE_TEXTUAL_TYPES, SEARCH_AUTO_EXCLUDED_FIELDS, apiExposureDenialReason, canServeApiOperation, classifyDottedFilterHead, effectiveOperationsArray, isCredentialFieldType, isFilterAST, isVirtualSearchField, parseFilterAST, referenceTargetOf, resolveEffectiveApiMethods, resolveSearchFieldResolution } from '@objectstack/spec/data'; @@ -233,10 +236,10 @@ function collectFilterFieldKeys( } /** - * [#22646] How deep the data door's `internal: true` walk follows nested - * relation conditions and `expand` levels — a backstop against a - * self-referential in-process value, well above anything the engine executes - * (one nested-relation hop; `expand` stops at its own depth cap). + * [#22661] How deep {@link servedExpand} follows `expand` levels — a backstop + * against a self-referential in-process value, well above anything the engine + * executes (`expand` stops at its own depth cap). The same depth the engine's + * `internal: true` walk stops at. */ const INTERNAL_FIELD_WALK_DEPTH = 8; @@ -278,11 +281,6 @@ const RELATION_CONDITION_TARGET_OPERATION = 'list'; export interface QueryValidationHost { assertFilterFieldsExist: ObjectStackProtocolImplementation['assertFilterFieldsExist']; readonly engine: ObjectStackProtocolImplementation['engine']; - groupByFieldNames: ObjectStackProtocolImplementation['groupByFieldNames']; - orderByFieldNames: ObjectStackProtocolImplementation['orderByFieldNames']; - refuseInternalInExpand: ObjectStackProtocolImplementation['refuseInternalInExpand']; - refuseInternalInFilter: ObjectStackProtocolImplementation['refuseInternalInFilter']; - refuseInternalNames: ObjectStackProtocolImplementation['refuseInternalNames']; refuseUnservedRelationTarget: ObjectStackProtocolImplementation['refuseUnservedRelationTarget']; resolveQueryFields: ObjectStackProtocolImplementation['resolveQueryFields']; servedExpand: ObjectStackProtocolImplementation['servedExpand']; @@ -1256,50 +1254,65 @@ export function assertAggregationFieldsExist( /** The body of `ObjectStackProtocolImplementation#assertNoInternalFieldEvaluated`; the method's docblock there is its contract. */ export function assertNoInternalFieldEvaluated( - this: Pick, + this: Pick, object: string, options: Record): void { - const gate = this.resolveQueryFields(object); - if (!gate) return; - const internal = new Set(collectInternalWriteResponseFields(gate.schema)); - // where + the `filter` alias a direct bag may still carry, each with its - // nested-relation conditions. - this.refuseInternalInFilter(object, options.where, 'filter', 0); - this.refuseInternalInFilter(object, options.filter, 'filter', 0); - this.refuseInternalNames(object, this.orderByFieldNames(options.orderBy), internal, 'sort'); - this.refuseInternalNames(object, this.groupByFieldNames(options.groupBy), internal, 'groupBy'); - if (Array.isArray(options.aggregations)) { - options.aggregations.forEach((agg: any, i: number) => { - const field = agg?.field; - if (typeof field === 'string' && field !== '*') { - this.refuseInternalNames(object, [field], internal, `aggregations[${i}].field`); - } - this.refuseInternalInFilter(object, agg?.filter, `aggregations[${i}].filter`, 0); - }); + const engine = this.engine; + if (typeof engine?.assertNoInternalFieldEvaluated !== 'function') { + // No gate to hand the hop question to: this door's own exposure + // question is still asked, at every hop, before the engine reads. The + // `internal: true` judgement is the engine's alone and is not copied. + refuseUnservedRelationTargetsIn.call(this, object, options); + return; } - // The whole expand tree, every level against its own target. - this.refuseInternalInExpand(object, options.expand, 'expand', 0); + engine.assertNoInternalFieldEvaluated(object, options, { + beforeRelationTarget: (target, param) => this.refuseUnservedRelationTarget(target, param), + }); } -/** The body of `ObjectStackProtocolImplementation#refuseInternalInFilter`; the method's docblock there is its contract. */ -export function refuseInternalInFilter( - this: Pick, - object: string, where: unknown, param: string, depth: number): void { - if (where == null || depth > INTERNAL_FIELD_WALK_DEPTH) return; - const gate = this.resolveQueryFields(object); - if (!gate) return; - const condition = isFilterAST(where) ? parseFilterAST(where) : where; - const internal = new Set(collectInternalWriteResponseFields(gate.schema)); - if (internal.size > 0) { - const reads = new Set(); - collectFilterReads(condition, reads); - this.refuseInternalNames(object, [...reads], internal, param); - } - for (const site of relationConditionSites(gate.fields, condition)) { - const position = `${param}.${site.field}`; - // [#22737] The target's exposure before anything of it is judged. - this.refuseUnservedRelationTarget(site.target, position); - this.refuseInternalInFilter(site.target, site.condition, position, depth + 1); +/** + * [#22737] The data door's relation-target exposure question, asked at every + * nested-relation condition of a read the door hands an engine that carries no + * `QueryEvaluationGate` (a test double, a metadata-only store). With the gate, + * the same question rides its `beforeRelationTarget` hook instead, at the same + * hops and in the same order. The positions are the ones that hook is asked + * at: the filter (`where`, or the `filter` alias), each aggregation's own + * `filter`, and each `expand` entry's filter, every level against its own + * target. Exposure only — it judges no field. + */ +function refuseUnservedRelationTargetsIn( + this: Pick, + object: string, options: Record): void { + const inFilter = (source: string, where: unknown, param: string, depth: number): void => { + if (where == null || depth > INTERNAL_FIELD_WALK_DEPTH) return; + const gate = this.resolveQueryFields(source); + if (!gate) return; + const condition = isFilterAST(where) ? parseFilterAST(where) : where; + for (const site of relationConditionSites(gate.fields, condition)) { + const position = `${param}.${site.field}`; + this.refuseUnservedRelationTarget(site.target, position); + inFilter(site.target, site.condition, position, depth + 1); + } + }; + const inExpand = (source: string, expand: unknown, prefix: string, depth: number): void => { + if (!isPlainRecord(expand) || depth > INTERNAL_FIELD_WALK_DEPTH) return; + const gate = this.resolveQueryFields(source); + if (!gate) return; + for (const [rel, entry] of Object.entries(expand)) { + if (!isPlainRecord(entry)) continue; + const def = gate.fields?.[rel]; + const target = def != null && REFERENCE_VALUE_TYPES.has(def.type) ? referenceTargetOf(def) : undefined; + if (!target) continue; + const path = `${prefix}['${rel}']`; + inFilter(target, entry.where, `${path}.filter`, 0); + inExpand(target, entry.expand, `${path}.expand`, depth + 1); + } + }; + inFilter(object, options.where, 'filter', 0); + inFilter(object, options.filter, 'filter', 0); + if (Array.isArray(options.aggregations)) { + options.aggregations.forEach((agg: any, i: number) => inFilter(object, agg?.filter, `aggregations[${i}].filter`, 0)); } + inExpand(object, options.expand, 'expand', 0); } /** The body of `ObjectStackProtocolImplementation#refuseUnservedRelationTarget`; the method's docblock there is its contract. */ @@ -1327,31 +1340,6 @@ export function refuseUnservedRelationTarget( throw err; } -/** The body of `ObjectStackProtocolImplementation#refuseInternalInExpand`; the method's docblock there is its contract. */ -export function refuseInternalInExpand( - this: Pick, - source: string, expand: unknown, prefix: string, depth: number): void { - if (!isPlainRecord(expand) || depth > INTERNAL_FIELD_WALK_DEPTH) return; - const gate = this.resolveQueryFields(source); - if (!gate) return; - for (const [rel, entry] of Object.entries(expand)) { - if (!isPlainRecord(entry)) continue; - const sourceField = gate.fields?.[rel]; - const target = sourceField != null && REFERENCE_VALUE_TYPES.has(sourceField.type) - ? referenceTargetOf(sourceField) - : undefined; - if (!target) continue; - const path = `${prefix}['${rel}']`; - this.refuseInternalInFilter(target, entry.where, `${path}.filter`, 0); - const targetGate = this.resolveQueryFields(target); - if (targetGate) { - const targetInternal = new Set(collectInternalWriteResponseFields(targetGate.schema)); - this.refuseInternalNames(target, this.orderByFieldNames(entry.orderBy), targetInternal, `${path}.sort`); - } - this.refuseInternalInExpand(target, entry.expand, `${path}.expand`, depth + 1); - } -} - /** The body of `ObjectStackProtocolImplementation#servedExpand`; the method's docblock there is its contract. */ export function servedExpand( this: Pick, @@ -1387,47 +1375,3 @@ export function servesExpansionTarget( return canServeApiOperation(schema?.enable, EXPANSION_TARGET_OPERATION); } -/** The body of `ObjectStackProtocolImplementation#orderByFieldNames`; the method's docblock there is its contract. */ -export function orderByFieldNames(orderBy: unknown): string[] { - if (!Array.isArray(orderBy)) return []; - return orderBy - .map((e) => (typeof e === 'string' ? e : (e as { field?: unknown })?.field)) - .filter((f): f is string => typeof f === 'string' && f.length > 0) - .map((f) => f.split('.')[0] as string); -} - -/** The body of `ObjectStackProtocolImplementation#groupByFieldNames`; the method's docblock there is its contract. */ -export function groupByFieldNames(groupBy: unknown): string[] { - if (!Array.isArray(groupBy)) return []; - return groupBy - .map((e) => (typeof e === 'string' ? e : (e as { field?: unknown })?.field)) - .filter((f): f is string => typeof f === 'string' && f.length > 0) - .map((f) => f.split('.')[0] as string); -} - -/** The body of `ObjectStackProtocolImplementation#refuseInternalNames`; the method's docblock there is its contract. */ -export function refuseInternalNames(object: string, names: readonly string[], internal: ReadonlySet, param: string): void { - const hit = names.filter((n) => internal.has(n)); - if (hit.length === 0) return; - const first = hit[0] as string; - const verb = param.endsWith('sort') ? 'sort' : param === 'groupBy' ? 'group' : param.includes('filter') ? 'filter' : 'query'; - const err: any = new Error( - `Cannot ${verb} '${object}' by '${first}' (${param}): the query was not run. The ${first} field ` - + 'is declared `internal: true`, so its value is never returned on a generic data exit — the engine ' - + 'omits it from every row. ' - + (verb === 'sort' - ? 'A sort on it orders by that withheld value' - : verb === 'group' - ? 'A group key would be the withheld value, served as the key' - : 'A filter on it confirms a guessed value by whether the row comes back') - + ', so it is refused rather than evaluated. Use another field.', - ); - err.code = 'INVALID_FIELD'; - err.status = 400; - err.httpStatus = 400; - err.field = first; - err.fields = hit; - err.object = object; - err.param = param; - throw err; -} diff --git a/packages/metadata-protocol/src/protocol.data-door-internal-field-delegation.test.ts b/packages/metadata-protocol/src/protocol.data-door-internal-field-delegation.test.ts new file mode 100644 index 00000000000..54f502b9720 --- /dev/null +++ b/packages/metadata-protocol/src/protocol.data-door-internal-field-delegation.test.ts @@ -0,0 +1,120 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The data door's `internal: true` evaluate refusal is the ENGINE's: `findData` + * and the public face (`assertNoInternalFieldEvaluated`) ask the engine's + * `QueryEvaluationGate` member before the engine reads anything, for every + * caller, and hand it this door's relation-target exposure question as the + * hook it runs at each hop. This file pins the door's half — that it asks, + * with what, before what, and what an engine without the member gets. The + * walk itself and the position enumeration are pinned beside the definition, + * in `@objectstack/objectql` (`internal-field-evaluation-guard.test.ts`, + * `protocol-data-door-internal-field-positions.test.ts`). + */ + +import { describe, expect, it, vi } from 'vitest'; +import type { ServiceObject } from '@objectstack/spec/data'; +import { assertEngineFindOnePredicate } from '@objectstack/metadata-core'; +import { ObjectStackProtocolImplementation } from './protocol.js'; + +const VAULT = { + name: 'dl_vault', + label: 'Vault', + fields: { + id: { name: 'id', label: 'ID', type: 'text' }, + name: { name: 'name', label: 'Name', type: 'text' }, + holder_id: { name: 'holder_id', label: 'Holder', type: 'lookup', reference: 'dl_holder' }, + }, +} as unknown as ServiceObject; + +/** The relation target, served nowhere on the API. */ +const HOLDER = { + name: 'dl_holder', + label: 'Holder', + enable: { apiEnabled: false }, + fields: { + id: { name: 'id', label: 'ID', type: 'text' }, + name: { name: 'name', label: 'Name', type: 'text' }, + }, +} as unknown as ServiceObject; + +const SCHEMAS: Record = { dl_vault: VAULT, dl_holder: HOLDER }; + +function refusalOf(param: string) { + return Object.assign(new Error(`Cannot filter 'dl_vault' by 'name' (${param}): the query was not run.`), { + code: 'INVALID_FIELD', status: 400, httpStatus: 400, object: 'dl_vault', field: 'name', param, + }); +} + +function makeProtocol(gate?: (object: string, query: any, options?: any) => void) { + const find = vi.fn(async () => [{ id: 'v1', name: 'alpha' }]); + const engine: any = { + registry: { getObject: (n: string) => SCHEMAS[n] }, + find, + findOne: vi.fn(async (object: string, query: any) => { assertEngineFindOnePredicate(object, query); return { id: 'v1' }; }), + count: vi.fn(async () => 1), + aggregate: vi.fn(async () => []), + ...(gate ? { assertNoInternalFieldEvaluated: vi.fn(gate) } : {}), + }; + return { p: new ObjectStackProtocolImplementation(engine), engine, find }; +} + +describe('the data door asks the engine\'s internal-field gate', () => { + it('findData asks it with the object and the query, before the engine reads anything; its refusal is the answer', async () => { + const { p, engine, find } = makeProtocol(() => { throw refusalOf('filter'); }); + let thrown: any; + try { await p.findData({ object: 'dl_vault', query: { where: { name: 'alpha' } } }); } catch (e) { thrown = e; } + expect(thrown).toMatchObject({ code: 'INVALID_FIELD', status: 400, object: 'dl_vault', param: 'filter' }); + expect(engine.assertNoInternalFieldEvaluated).toHaveBeenCalledTimes(1); + const [object, query, options] = engine.assertNoInternalFieldEvaluated.mock.calls[0]; + expect(object).toBe('dl_vault'); + expect(query).toMatchObject({ where: { name: 'alpha' } }); + expect(typeof options?.beforeRelationTarget).toBe('function'); + expect(find).not.toHaveBeenCalled(); + }); + + it('asks it for a system context too — the door takes no caller carve-out', async () => { + const { p, engine, find } = makeProtocol(() => { throw refusalOf('filter'); }); + await expect(p.findData({ object: 'dl_vault', query: { where: { name: 'alpha' } }, context: { isSystem: true } })) + .rejects.toMatchObject({ code: 'INVALID_FIELD', status: 400 }); + expect(engine.assertNoInternalFieldEvaluated).toHaveBeenCalledTimes(1); + expect(find).not.toHaveBeenCalled(); + }); + + it('the hook it hands over is this door\'s relation-target exposure question', () => { + const { p } = makeProtocol((_object, _query, options) => options?.beforeRelationTarget?.('dl_holder', 'filter.holder_id')); + let thrown: any; + try { p.assertNoInternalFieldEvaluated('dl_vault', { where: { holder_id: { name: 'Pat' } } }); } catch (e) { thrown = e; } + expect(thrown).toMatchObject({ code: 'OBJECT_API_DISABLED', status: 404, object: 'dl_holder', param: 'filter.holder_id' }); + }); + + it('CONTROL — a gate that passes leaves the read served by the engine', async () => { + const { p, engine, find } = makeProtocol(() => {}); + const res: any = await p.findData({ object: 'dl_vault', query: { where: { name: 'alpha' } } }); + expect(engine.assertNoInternalFieldEvaluated).toHaveBeenCalledTimes(1); + expect(find).toHaveBeenCalledTimes(1); + expect((res.records ?? res.data ?? res).length).toBe(1); + }); + + it('an engine without the member is judged on no field here — the flag is the engine\'s to honour', async () => { + const { p, find } = makeProtocol(); + expect(() => p.assertNoInternalFieldEvaluated('dl_vault', { where: { name: 'alpha' } })).not.toThrow(); + await p.findData({ object: 'dl_vault', query: { where: { name: 'alpha' } } }); + expect(find).toHaveBeenCalledTimes(1); + }); + + it('…but the door\'s exposure question is still asked at every hop on such an engine, before it reads', async () => { + const { p, find } = makeProtocol(); + for (const [query, param] of [ + [{ where: { holder_id: { name: 'Pat' } } }, 'filter.holder_id'], + [{ aggregations: [{ function: 'count', alias: 'n', filter: { holder_id: { name: 'Pat' } } }] }, 'aggregations[0].filter.holder_id'], + ] as const) { + let thrown: any; + try { p.assertNoInternalFieldEvaluated('dl_vault', query as any); } catch (e) { thrown = e; } + expect(thrown).toMatchObject({ code: 'OBJECT_API_DISABLED', status: 404, object: 'dl_holder', param }); + } + await expect(p.findData({ object: 'dl_vault', query: { where: { holder_id: { name: 'Pat' } } } })) + .rejects.toMatchObject({ code: 'OBJECT_API_DISABLED', status: 404 }); + expect(find).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index 7ffb87a253d..9775a689263 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -10,7 +10,7 @@ import { judgePermissionSetOwnerBinding, type TenancyPosture } from '@objectstac // `DiscoverySchema` "System Identity" field — never a literal again. import { resolveDiscoveryVersion } from './discovery-version.js'; import type { MetadataHostEngine } from './host-engine.js'; -import { collectInternalWriteResponseFields, omitInternalFieldsFromWriteResponse } from './write-response-internal-fields.js'; +import { omitInternalFieldsFromWriteResponse } from './write-response-internal-fields.js'; import { staleSummaryRefusalOrThrow, withEarnedAnswer, type SummaryRecomputeRefusal } from '@objectstack/types'; // [#17502] The served JSON Schema publishes what an author MAY write, so a // property no instance can satisfy — a `retiredKey()` tombstone, rendered @@ -281,7 +281,7 @@ import { // the same name below hands the body the members it reads (`MetaDiagnosticsHost`). import { getMetaDiagnostics } from './protocol-meta-diagnostics.js'; // [#25467 cut 3] The query validation (`assertObjectRegistered`, the `assert*FieldsExist` -// family, the `refuseInternal*` walks, `servedExpand`); the methods of the same names +// family, `refuseUnservedRelationTarget`, `servedExpand`); the methods of the same names // below hand each body the members it reads (`QueryValidationHost`). import { assertAggregationFieldsExist, @@ -294,11 +294,6 @@ import { assertQueryParamsAreFields, assertSearchFieldsAreSearchable, assertSortFieldsExist, - groupByFieldNames, - orderByFieldNames, - refuseInternalInExpand, - refuseInternalInFilter, - refuseInternalNames, refuseUnservedRelationTarget, resolveQueryFields, servedExpand, @@ -13638,67 +13633,30 @@ export class ObjectStackProtocolImplementation implements /** * [#22646] The data door's EVALUATE refusal for a field declared * `internal: true` — "the value is never returned on a generic data exit" - * (#21197), extended from the row position (where it is a withhold — the - * engine OMITS the column from every row) to every position that would - * reveal the stored value by EVALUATING it. A filter or sort on such a - * column is a confirmation oracle — a predicate matches the row only when - * the guess is right, an order leaks the comparative value — so it is - * REFUSED rather than withheld, the posture this door already takes for the - * stored-metadata body/hash family ({@link storedMetadataBodyPredicateRefusal}, - * {@link storedMetadataHashEvaluateRefusal}) and the analytics door for an - * internal member. `INVALID_FIELD` / 400, the family's envelope, located at - * the object, the field and the offending position. - * - * It binds EVERY caller of this generic door, administrators included - * (#21197): there is no caller carve-out, the same stance the stored-body - * family takes here. The engine's privileged internal consumers (the - * API-key verifier's `where: { key }`, the share-link / SCIM / approval-token - * lookups) call the engine directly and never reach this door, so the - * flag's one legitimate use — a `where` match the engine resolves without - * ever returning the value — is untouched, exactly as #7823 kept the engine's - * write results whole while the generic-data-path ingress stripped them. - * - * PUBLIC, and asked by the generic data mouths that hand a caller's query - * to the engine without passing through {@link findData}: the MCP stdio - * transport's `query_records` / `aggregate_records` and the HTTP MCP - * bridge's `aggregate_records` arm (`callData`'s `aggregate`). Each asks - * this method on the `protocol` service it already resolves, before the - * engine, so a filter, sort, group key or aggregation over an `internal` - * field gets this door's refusal, word for word, on every transport — - * one definition, never a copy beside it (route-ownership rule 1). The - * engine's privileged consumers above still never reach it. - * - * Positions, with the ONE flag reader ({@link collectInternalWriteResponseFields}, - * `@objectstack/core`'s cross-package twin of objectql's - * `collectInternalReadFields` — identical rule, imported because this package - * sits below objectql, as the write-response strip and the analytics door do): - * - * - `where` / `filter` and each `aggregations[i].filter` — every column a - * filter READS, by the stricter collector that also walks a cross-field - * `{ $field }` comparand ({@link collectFilterReads}). - * - `orderBy` — the fields a sort names. - * - `groupBy` and `aggregations[].field` — defence in depth; the engine's - * `rejectCredentialAggregation` refuses these for every caller with the - * same code, and this door answers first so the envelope is the door's. - * - a NESTED-RELATION condition inside any of those filters - * (`{ : { : … } }`), judged against - * the RELATED object's own internal fields: the engine lowers it by - * evaluating the condition on the related object - * (`ObjectQL.lowerRelationConditions`) and keeps the rows whose related - * record matched, so a related `internal` field there is the same - * confirmation oracle one hop away. Judged here, at the door, rather than - * in the engine's relation-site walk, so the engine's privileged callers - * keep the #7823 split (none of them reaches this door). The RELATED - * object's exposure is asked first, at the same hop - * ({@link refuseUnservedRelationTarget}, #22737). - * - every `expand` entry's own `where` / `orderBy`, at EVERY depth of the - * expand tree, judged against that level's TARGET object: the nested - * sub-read's row strip already withholds the column, but a nested filter - * is a presence oracle on the related record (the expanded object appears - * only when the guess matches), and a deeper level's sub-read is one more - * `engine.find` that answers the same way. Target OBJECT exposure is a - * different axis, judged before this walk ({@link servedExpand}, #22661): - * an entry whose target the API does not serve never reaches it. + * (#21197): a filter, sort, group key, aggregation operand, per-aggregation + * filter, nested-relation condition or `expand` entry's own filter/sort on + * such a column reveals the stored value by evaluating it, so it is REFUSED, + * `INVALID_FIELD` / 400, located at the object, the field and the position. + * + * The judgement is the ENGINE's, not this door's: the engine owns the flag + * (it omits the column from every row) and holds the one definition, + * `QueryEvaluationGate#assertNoInternalFieldEvaluated` on the `objectql` + * slot (`internal-field-evaluation-guard.ts` in `@objectstack/objectql`), + * which its own read path also applies to every non-system caller — so a + * host with no `protocol` service, or a protocol other than this one, still + * refuses. This door asks it, before the engine reads anything, for EVERY + * caller (administrators and system contexts included, the stored-body + * family's stance here), and hooks its own question in at each + * nested-relation hop: the RELATED object's API exposure + * ({@link refuseUnservedRelationTarget}, #22737) is asked before anything of + * that object is judged, so the order of refusals is the one this door has + * always given. ⛔ Never a copy of the walk here (route-ownership rule 1). + * + * An engine that does not carry the member (a test double, a metadata-only + * store) is judged on no field here: the flag is the engine's to honour, and + * an engine without the gate does not withhold the column either. The + * door's own exposure question is still asked at every hop on such an + * engine, by a walk that judges no field. * * Not judged here because the ingress already refuses them, measured: a * DOTTED sort or group-by path crossing a relation (the sort gate's @@ -13708,34 +13666,14 @@ export class ObjectStackProtocolImplementation implements * batch the expansion re-keys by id with no `limit`, so it changes no answer. */ assertNoInternalFieldEvaluated(object: string, options: Record): void { - // [#25467 cut 3] The body moved to `protocol-query-validation.ts` (see `QueryValidationHost`). + // [#25467 cut 3] The body lives in `protocol-query-validation.ts` (see `QueryValidationHost`). + const readEngine = () => this.engine; const host: ThisParameterType = { - groupByFieldNames: (...a) => this.groupByFieldNames(...a), - orderByFieldNames: (...a) => this.orderByFieldNames(...a), - refuseInternalInExpand: (...a) => this.refuseInternalInExpand(...a), - refuseInternalInFilter: (...a) => this.refuseInternalInFilter(...a), - refuseInternalNames: (...a) => this.refuseInternalNames(...a), - resolveQueryFields: (...a) => this.resolveQueryFields(...a), - }; - return assertNoInternalFieldEvaluated.call(host, object, options); - } - - /** - * One filter's internal-field judgement: every column it reads on `object` - * ({@link collectFilterReads}), then each nested-relation condition it holds, - * recursively, on the related object, once that object's exposure serves the - * condition ({@link refuseUnservedRelationTarget}). `param` names the position, extended - * by the relation field at each hop (`filter.owner_id`). - */ - private refuseInternalInFilter(object: string, where: unknown, param: string, depth: number): void { - // [#25467 cut 3] The body moved to `protocol-query-validation.ts` (see `QueryValidationHost`). - const host: ThisParameterType = { - refuseInternalInFilter: (...a) => this.refuseInternalInFilter(...a), - refuseInternalNames: (...a) => this.refuseInternalNames(...a), + get engine() { return readEngine(); }, refuseUnservedRelationTarget: (...a) => this.refuseUnservedRelationTarget(...a), resolveQueryFields: (...a) => this.resolveQueryFields(...a), }; - return refuseInternalInFilter.call(host, object, where, param, depth); + return assertNoInternalFieldEvaluated.call(host, object, options); } /** @@ -13761,23 +13699,6 @@ export class ObjectStackProtocolImplementation implements return refuseUnservedRelationTarget.call(host, target, param); } - /** - * Every level of an `expand` tree: each entry's own `where` (with its - * nested-relation conditions) and `orderBy`, judged against that entry's - * target object, then the entry's own `expand` with the target as source. - */ - private refuseInternalInExpand(source: string, expand: unknown, prefix: string, depth: number): void { - // [#25467 cut 3] The body moved to `protocol-query-validation.ts` (see `QueryValidationHost`). - const host: ThisParameterType = { - orderByFieldNames: (...a) => this.orderByFieldNames(...a), - refuseInternalInExpand: (...a) => this.refuseInternalInExpand(...a), - refuseInternalInFilter: (...a) => this.refuseInternalInFilter(...a), - refuseInternalNames: (...a) => this.refuseInternalNames(...a), - resolveQueryFields: (...a) => this.resolveQueryFields(...a), - }; - return refuseInternalInExpand.call(host, source, expand, prefix, depth); - } - /** * [#22661] The `$expand` tree this door hands the engine, without the * entries whose TARGET object's declared exposure does not serve the read. @@ -13832,24 +13753,6 @@ export class ObjectStackProtocolImplementation implements return servesExpansionTarget.call(host, target); } - /** Field heads an `orderBy` names (string or `{ field }`), dotted head only. */ - private orderByFieldNames(orderBy: unknown): string[] { - // [#25467 cut 3] The body moved to `protocol-query-validation.ts` (see `QueryValidationHost`). - return orderByFieldNames(orderBy); - } - - /** Field heads a `groupBy` names (string or `{ field }`), dotted head only. */ - private groupByFieldNames(groupBy: unknown): string[] { - // [#25467 cut 3] The body moved to `protocol-query-validation.ts` (see `QueryValidationHost`). - return groupByFieldNames(groupBy); - } - - /** Refuse when any of `names` is an internal field of `object`. */ - private refuseInternalNames(object: string, names: readonly string[], internal: ReadonlySet, param: string): void { - // [#25467 cut 3] The body moved to `protocol-query-validation.ts` (see `QueryValidationHost`). - return refuseInternalNames(object, names, internal, param); - } - async findData(request: { object: string, query?: any, context?: any }) { // [#3770] Existence first: an unregistered object is a 404 before any // query parameter is even parsed, so an unknown name can never be diff --git a/packages/objectql/src/engine-count-aggregate.ts b/packages/objectql/src/engine-count-aggregate.ts index 20718ca9ee8..d6e9c571eec 100644 --- a/packages/objectql/src/engine-count-aggregate.ts +++ b/packages/objectql/src/engine-count-aggregate.ts @@ -40,8 +40,9 @@ * this module imports `engine.ts`). Nothing here runs at module evaluation, * only declarations, and the helpers are read only when a body runs, after * both modules have evaluated. `engine.ts` imports this module right after - * `./secret-fields.js`, and the two other modules this one imports - * (`./secret-fields.js`, `./credential-evaluation-guard.js`) import nothing + * `./secret-fields.js`, and the three other modules this one imports + * (`./secret-fields.js`, `./credential-evaluation-guard.js`, + * `./internal-field-evaluation-guard.js`) import nothing * from `engine.ts`, so the evaluation order of every module is what it was; the * `@objectstack/spec/api` import that left `engine.ts` with * `rejectCredentialAggregation` was not that package's first load. The @@ -58,6 +59,7 @@ import { collectCredentialFields, } from './secret-fields.js'; import { rejectCredentialEvaluation } from './credential-evaluation-guard.js'; +import { rejectInternalFieldEvaluation } from './internal-field-evaluation-guard.js'; import { ENGINE_COUNT_OPTION_KEYS, ENGINE_WHERE_SLOTS, @@ -103,7 +105,8 @@ export async function count( query = foldEngineOptionAliases(object, 'count', query, ENGINE_WHERE_SLOTS); rejectUnknownEngineOptions(object, 'count', query, ENGINE_COUNT_OPTION_KEYS); query = lowerWhereFilterArray(object, 'count', query, this._registry.getObject(object), this.relatedSchemaOf); - // [ADR-0100] Same credential-column refusal as `find`, same position. + // [ADR-0100] Same internal-field and credential-column refusals as `find`, same position. + rejectInternalFieldEvaluation(object, query, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); rejectCredentialEvaluation(object, 'count', query, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); // [#22790] ADR-0061 `search` → the rows are searched before they are // counted, by the one expander `find` runs, at the same point in the diff --git a/packages/objectql/src/engine.ts b/packages/objectql/src/engine.ts index d80e4629227..14955090e15 100644 --- a/packages/objectql/src/engine.ts +++ b/packages/objectql/src/engine.ts @@ -166,7 +166,7 @@ import type { FlowFunctionEffect } from '@objectstack/spec/automation'; // contract is new (#5945). import type { IPubSub, IScopedContext, IScopedObjectRepository, IntrospectedSchema as SpecIntrospectedSchema } from '@objectstack/spec/contracts'; // [#20157] The judge-only filter-admission member's verdict and options. -import type { EngineFilterJudgement, EngineFilterJudgementOptions } from '@objectstack/spec/contracts'; +import type { EngineFilterJudgement, EngineFilterJudgementOptions, InternalFieldEvaluationOptions, InternalFieldEvaluationQuery } from '@objectstack/spec/contracts'; import { IDataDriver, IDataEngine, @@ -312,6 +312,7 @@ import { count, rejectCredentialAggregation } from './engine-count-aggregate.js' // ADR-0100: the credential-column guard's filter / sort half, beside the // aggregate half above — one collector, one envelope. import { rejectCredentialEvaluation } from './credential-evaluation-guard.js'; +import { assertNoInternalFieldEvaluated, rejectInternalFieldEvaluation } from './internal-field-evaluation-guard.js'; import { assertGroupByNamesNoJsonStoredField } from './group-by-structured-json-door.js'; import { assertProjectionNamesAField, refuseProjectionOfUnknownColumns } from './empty-projection-door.js'; import { assertAggregationFieldTypesAccepted } from './aggregate-field-type-door.js'; @@ -14441,6 +14442,10 @@ export class ObjectQL implements IObjectQLEngine { // for a non-system caller — judged on the caller's own `where` / `orderBy` // / `expand`, before the search term is expanded into `where` and before // the middlewares inject their server-authored filters. + // [internal: true] …and a flagged column is never evaluated for one either, + // judged first so the refusal is the data door's own (the door asks the + // same definition before the engine runs). + rejectInternalFieldEvaluation(object, ast, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); rejectCredentialEvaluation(object, 'find', ast, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); // Plan formula projection: rewrite ast.fields to drop virtual formula @@ -14767,7 +14772,8 @@ export class ObjectQL implements IObjectQLEngine { // [commit 74155c735] And the same default-filling as find(), for the same reason. const { context: _findOneContext, ...findOneQuery } = query ?? {}; const ast: QueryAST = { ...this.fillQueryAstDefaults(findOneQuery), object: objectName, limit: 1 }; - // [ADR-0100] Same credential-column refusal as `find`, same position. + // [ADR-0100] Same internal-field and credential-column refusals as `find`, same position. + rejectInternalFieldEvaluation(objectName, ast, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); rejectCredentialEvaluation(objectName, 'findOne', ast, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); // Plan formula projection (same as find): rewrite ast.fields so the driver @@ -21137,6 +21143,21 @@ export class ObjectQL implements IObjectQLEngine { return count.call(host, object, query, options); } + /** + * The `internal: true` evaluate refusal for EVERY caller — the + * `QueryEvaluationGate` member of the `objectql` slot contract. The body is + * `internal-field-evaluation-guard.ts`, the one definition, which this + * engine's own read path also applies to every non-system caller; a door + * asks this method and never restates it. + */ + assertNoInternalFieldEvaluated( + object: string, + query: InternalFieldEvaluationQuery, + options?: InternalFieldEvaluationOptions, + ): void { + assertNoInternalFieldEvaluated(object, query, this.relatedSchemaOf, options); + } + /** * Fail-closed guard (ADR-0100 / #3171 / #7922): refuse to aggregate over a * field whose value is withheld on the generic read path. Such fields reach @@ -21197,6 +21218,11 @@ export class ObjectQL implements IObjectQLEngine { // [ADR-0100] The filter half of the credential-column guard — `where` // and every `aggregations[i].filter` — before the search term joins // `where`; the aggregate half (`rejectCredentialAggregation`) follows. + // [internal: true] The flagged-column refusal goes first, over every + // position at once (`where`, `groupBy`, each aggregation's `field` and + // `filter`), so a non-system caller meets the data door's words here + // too rather than the aggregate guard's. + rejectInternalFieldEvaluation(object, query, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); rejectCredentialEvaluation(object, 'aggregate', query, mergeReadContext(query?.context, options?.context), this.relatedSchemaOf); // ADR-0061 `search` → the rows are searched BEFORE they are grouped, by // the one expander `find` runs, at the same point in the sequence (after diff --git a/packages/objectql/src/federated-injected-column-readers.test.ts b/packages/objectql/src/federated-injected-column-readers.test.ts index dc975b53587..faa92fb9576 100644 --- a/packages/objectql/src/federated-injected-column-readers.test.ts +++ b/packages/objectql/src/federated-injected-column-readers.test.ts @@ -260,6 +260,10 @@ const READERS: Record = { disposition: 'caller-predicate', why: "names the object an expand entry the caller wrote is judged against, by its declared fields; it reads no row", }, + 'internal-field-evaluation-guard.ts#judgeExpand :: referenceTargetOf()': { + disposition: 'caller-predicate', + why: "names the object an expand entry the caller wrote is judged against, by its declared fields; it reads no row", + }, 'record-title.ts#resolveRelatedTitleTarget :: referenceTargetOf()': { disposition: 'target-by-id', why: "resolves which object a related record's title is read from", diff --git a/packages/objectql/src/internal-field-evaluation-guard.test.ts b/packages/objectql/src/internal-field-evaluation-guard.test.ts new file mode 100644 index 00000000000..239d6696a2b --- /dev/null +++ b/packages/objectql/src/internal-field-evaluation-guard.test.ts @@ -0,0 +1,325 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The `internal: true` evaluate refusal on the ENGINE's own read path, against + * a REAL {@link ObjectQL} over a minimal recording driver, and on a real + * kernel that composes the engine plugin with no `protocol` service at all. + * + * Every refusal is the ADR-0112 `INVALID_FIELD` / 400 envelope, located at the + * object, the field and the position, and leaves the driver unasked. A system + * context is served (the server's own lookups match on such a column), every + * ordinary field is served, and the public face (`assertNoInternalFieldEvaluated`, + * the `QueryEvaluationGate` member) binds a system context too and runs a + * door's relation-target hook before it judges that hop. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { ObjectKernel } from '@objectstack/core'; +import type { EngineAggregateOptions, EngineQueryOptions } from '@objectstack/spec/data'; +import { ObjectQL } from './engine.js'; +import { ObjectQLPlugin } from './plugin.js'; + +function makeRecordingDriver() { + const rows = new Map[]>(); + const calls: string[] = []; + const matches = (row: Record, where: any): boolean => { + if (!where || typeof where !== 'object') return true; + for (const [k, v] of Object.entries(where)) { + if (k.startsWith('$')) continue; + if (v !== null && typeof v === 'object') continue; + if ((row[k] ?? null) !== (v ?? null)) return false; + } + return true; + }; + const driver: any = { + name: 'memory', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, + async execute() { return null; }, + async find(object: string, ast: any) { + calls.push(`find:${object}`); + const found = (rows.get(object) ?? []).filter((r) => matches(r, ast?.where)); + const page = typeof ast?.limit === 'number' ? found.slice(0, ast.limit) : found; + return page.map((r) => ({ ...r })); + }, + async findOne(object: string, ast: any) { + calls.push(`findOne:${object}`); + const r = (rows.get(object) ?? []).find((x) => matches(x, ast?.where)); + return r ? { ...r } : null; + }, + async count(object: string, ast: any) { + calls.push(`count:${object}`); + return (rows.get(object) ?? []).filter((r) => matches(r, ast?.where)).length; + }, + async aggregate(object: string) { + calls.push(`aggregate:${object}`); + return [{ n: (rows.get(object) ?? []).length }]; + }, + async create(object: string, data: Record) { + const list = rows.get(object) ?? []; + const row = { id: `r${list.length + 1}`, ...data }; + list.push(row); + rows.set(object, list); + return { ...row }; + }, + async update(object: string, id: string, data: Record) { + calls.push(`update:${object}`); + return { id, ...data }; + }, + async updateMany(object: string) { calls.push(`updateMany:${object}`); return 0; }, + async delete(object: string) { calls.push(`delete:${object}`); return true; }, + async deleteMany(object: string) { calls.push(`deleteMany:${object}`); return 0; }, + async upsert(_o: string, data: Record) { return data; }, + async bulkCreate() { return []; }, async bulkUpdate() { return []; }, async bulkDelete() {}, + async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { driver, rows, calls }; +} + +/** The related object: its own flagged column, reached through a lookup and an expand. */ +const holder = { + name: 'ifg_holder', label: 'Holder', + fields: { + id: { name: 'id', label: 'ID', type: 'text' as const }, + name: { name: 'name', label: 'Name', type: 'text' as const }, + holder_digest: { name: 'holder_digest', label: 'Digest', type: 'text' as const, internal: true }, + }, +}; + +/** The flagged column is a one-way digest in an ordinary `text` column, the shape the flag exists for. */ +const vault = { + name: 'ifg_vault', label: 'Vault', + fields: { + id: { name: 'id', label: 'ID', type: 'text' as const }, + name: { name: 'name', label: 'Name', type: 'text' as const }, + rank: { name: 'rank', label: 'Rank', type: 'number' as const }, + digest: { name: 'digest', label: 'Digest', type: 'text' as const, internal: true }, + holder: { name: 'holder', label: 'Holder', type: 'lookup' as const, reference: 'ifg_holder' }, + }, +}; + +const MEMBER: EngineQueryOptions['context'] = { userId: 'u1' }; +const ADMIN: EngineQueryOptions['context'] = { userId: 'u_admin', roles: ['admin'], isPlatformAdmin: true } as never; +const SYSTEM: EngineQueryOptions['context'] = { isSystem: true }; +const COUNT = { function: 'count' as const, alias: 'n' }; + +async function buildEngine() { + const engine = new ObjectQL(); + const recording = makeRecordingDriver(); + engine.registerDriver(recording.driver, true); + await engine.init(); + engine.registry.registerObject(holder as any); + engine.registry.registerObject(vault as any); + recording.rows.set('ifg_holder', [{ id: 'h1', name: 'Pat', holder_digest: 'hd1' }]); + recording.rows.set('ifg_vault', [{ id: 'v1', name: 'alpha', rank: 1, digest: 'd1', holder: 'h1' }]); + return { engine, ...recording }; +} + +/** The ADR-0112 envelope, asserted on the thrown error, never on a bare throw. */ +async function expectRefused(run: () => unknown, object: string, field: string, param: string) { + let thrown: any; + try { await run(); } catch (e) { thrown = e; } + expect(thrown, 'the call was refused').toBeDefined(); + expect(thrown.code).toBe('INVALID_FIELD'); + expect(thrown.status).toBe(400); + expect(thrown.httpStatus).toBe(400); + expect(thrown.object).toBe(object); + expect(thrown.field).toBe(field); + expect(thrown.param).toBe(param); + return thrown; +} + +describe('internal: true — the engine read path refuses every evaluate position for a non-system caller', () => { + let ctx: Awaited>; + beforeEach(async () => { ctx = await buildEngine(); }); + + const wheres: Array<[string, EngineQueryOptions['where']]> = [ + ['equality', { digest: 'd1' }], + ['$eq', { digest: { $eq: 'd1' } }], + ['$startsWith', { digest: { $startsWith: 'd' } }], + ['under $or', { $or: [{ name: '__none__' }, { digest: 'd1' }] }], + ['under $not', { $not: { digest: 'd1' } }], + ['a presence check (no carve-out, unlike the credential guard)', { digest: { $null: false } }], + ['a cross-field comparand', { name: { $eq: { $field: 'digest' } } }], + ['the array form', [['digest', '=', 'd1']]], + ]; + for (const [label, where] of wheres) { + for (const [persona, context] of Object.entries({ member: MEMBER, admin: ADMIN })) { + it(`find refuses a filter on it (${label}) — ${persona}, driver never asked`, async () => { + await expectRefused(() => ctx.engine.find('ifg_vault', { where, context }), 'ifg_vault', 'digest', 'filter'); + expect(ctx.calls).toEqual([]); + }); + } + } + + it('a context-less engine call is judged like any non-system caller', async () => { + await expectRefused(() => ctx.engine.find('ifg_vault', { where: { digest: 'd1' } }), 'ifg_vault', 'digest', 'filter'); + }); + + it('refuses a sort, on find and on findOne', async () => { + await expectRefused( + () => ctx.engine.find('ifg_vault', { orderBy: [{ field: 'digest', order: 'asc' }], context: MEMBER }), 'ifg_vault', 'digest', 'sort', + ); + await expectRefused( + () => ctx.engine.findOne('ifg_vault', { where: { name: 'alpha' }, orderBy: [{ field: 'digest', order: 'desc' }], context: MEMBER }), + 'ifg_vault', 'digest', 'sort', + ); + expect(ctx.calls).toEqual([]); + }); + + it('refuses on findOne and count', async () => { + await expectRefused(() => ctx.engine.findOne('ifg_vault', { where: { digest: 'd1' }, context: MEMBER }), 'ifg_vault', 'digest', 'filter'); + await expectRefused(() => ctx.engine.count('ifg_vault', { where: { digest: 'd1' }, context: MEMBER }), 'ifg_vault', 'digest', 'filter'); + expect(ctx.calls).toEqual([]); + }); + + it('refuses every aggregate position — where, group key, operand, per-aggregation filter — in the door\'s words', async () => { + const shapes: Array<[EngineAggregateOptions, string]> = [ + [{ where: { digest: 'd1' }, aggregations: [COUNT] }, 'filter'], + [{ groupBy: ['digest'], aggregations: [COUNT] }, 'groupBy'], + [{ groupBy: [{ field: 'digest' }] as never, aggregations: [COUNT] }, 'groupBy'], + [{ aggregations: [{ function: 'max', field: 'digest', alias: 'm' }] }, 'aggregations[0].field'], + [{ aggregations: [{ ...COUNT, filter: { digest: 'd1' } }] }, 'aggregations[0].filter'], + ]; + for (const [query, param] of shapes) { + const err = await expectRefused(() => ctx.engine.aggregate('ifg_vault', query, { context: MEMBER }), 'ifg_vault', 'digest', param); + // The door's sentence, not the aggregate guard's. + expect(err.message).toMatch(/is declared `internal: true`/); + } + expect(ctx.calls).toEqual([]); + }); + + it('refuses a nested-relation condition on the related object\'s flagged column, located at the hop', async () => { + await expectRefused( + () => ctx.engine.find('ifg_vault', { where: { holder: { holder_digest: 'hd1' } }, context: MEMBER }), + 'ifg_holder', 'holder_digest', 'filter.holder', + ); + expect(ctx.calls).toEqual([]); + }); + + it('refuses an expand entry\'s own filter and sort on the target\'s flagged column', async () => { + await expectRefused( + () => ctx.engine.find('ifg_vault', { expand: { holder: { object: 'ifg_holder', where: { holder_digest: 'hd1' } } }, context: MEMBER } satisfies EngineQueryOptions), + 'ifg_holder', 'holder_digest', "expand['holder'].filter", + ); + await expectRefused( + () => ctx.engine.find('ifg_vault', { expand: { holder: { object: 'ifg_holder', orderBy: [{ field: 'holder_digest' }] } }, context: MEMBER } satisfies EngineQueryOptions), + 'ifg_holder', 'holder_digest', "expand['holder'].sort", + ); + expect(ctx.calls).toEqual([]); + }); +}); + +describe('internal: true — what the engine path still serves', () => { + let ctx: Awaited>; + beforeEach(async () => { ctx = await buildEngine(); }); + + it('a system context is served unchanged — the server\'s own lookups match on a flagged column', async () => { + const rows = await ctx.engine.find('ifg_vault', { where: { digest: 'd1' }, context: SYSTEM }); + expect(rows.map((r: any) => r.id)).toEqual(['v1']); + // Matched on, and still never returned. + expect(rows[0]).not.toHaveProperty('digest'); + await expect(ctx.engine.findOne('ifg_vault', { where: { digest: 'd1' }, context: SYSTEM })).resolves.toMatchObject({ id: 'v1' }); + await expect(ctx.engine.count('ifg_vault', { where: { digest: 'd1' }, context: SYSTEM })).resolves.toBe(1); + await expect(ctx.engine.find('ifg_vault', { orderBy: [{ field: 'digest', order: 'asc' }], context: SYSTEM })).resolves.toHaveLength(1); + expect(ctx.calls).toEqual(['find:ifg_vault', 'findOne:ifg_vault', 'count:ifg_vault', 'find:ifg_vault']); + }); + + it('every ordinary field filters, sorts, counts and aggregates for a member', async () => { + const rows = await ctx.engine.find('ifg_vault', { + where: { name: 'alpha', holder: 'h1' }, orderBy: [{ field: 'name', order: 'asc' }], context: MEMBER, + }); + expect(rows.map((r: any) => r.id)).toEqual(['v1']); + expect(rows[0]).not.toHaveProperty('digest'); + await expect(ctx.engine.count('ifg_vault', { where: { name: 'alpha' }, context: MEMBER })).resolves.toBe(1); + await expect(ctx.engine.aggregate('ifg_vault', { + groupBy: ['name'], aggregations: [{ function: 'max', field: 'rank', alias: 'm', filter: { name: 'alpha' } }], + }, { context: MEMBER })).resolves.toBeInstanceOf(Array); + // Reached the driver (natively or through the engine's in-memory fold). + expect(ctx.calls.length).toBeGreaterThan(2); + }); +}); + +describe('internal: true — the public face (QueryEvaluationGate) binds every caller and runs a door\'s hop hook first', () => { + let ctx: Awaited>; + beforeEach(async () => { ctx = await buildEngine(); }); + + it('refuses for every caller — the face takes no context, so a system caller at a door is judged too', () => { + let thrown: any; + try { ctx.engine.assertNoInternalFieldEvaluated('ifg_vault', { where: { digest: 'd1' } }); } catch (e) { thrown = e; } + expect(thrown).toMatchObject({ code: 'INVALID_FIELD', status: 400, object: 'ifg_vault', field: 'digest', param: 'filter' }); + }); + + it('answers exactly what the engine path answers, message included', async () => { + const shapes: Array> = [ + { where: { digest: { $startsWith: 'd' } } }, + { orderBy: [{ field: 'digest', order: 'asc' }] }, + { where: { holder: { holder_digest: 'hd1' } } }, + ]; + for (const query of shapes) { + let face: any; + try { ctx.engine.assertNoInternalFieldEvaluated('ifg_vault', query); } catch (e) { face = e; } + let path: any; + try { await ctx.engine.find('ifg_vault', { ...query, context: MEMBER } as EngineQueryOptions); } catch (e) { path = e; } + const envelope = (e: any) => ({ code: e?.code, status: e?.status, object: e?.object, field: e?.field, param: e?.param, message: e?.message }); + expect(face).toBeDefined(); + expect(envelope(face)).toEqual(envelope(path)); + } + }); + + it('calls beforeRelationTarget at each hop before judging it, and lets what it throws win', () => { + const seen: Array<[string, string]> = []; + ctx.engine.assertNoInternalFieldEvaluated('ifg_vault', { where: { holder: { name: 'Pat' } } }, { + beforeRelationTarget: (target, param) => { seen.push([target, param]); }, + }); + expect(seen).toEqual([['ifg_holder', 'filter.holder']]); + + const hop = Object.assign(new Error('not exposed'), { code: 'OBJECT_API_DISABLED', status: 404 }); + expect(() => ctx.engine.assertNoInternalFieldEvaluated('ifg_vault', { where: { holder: { holder_digest: 'hd1' } } }, { + beforeRelationTarget: () => { throw hop; }, + })).toThrow(hop); + }); + + it('CONTROL — ordinary fields in every position pass the face', () => { + expect(() => ctx.engine.assertNoInternalFieldEvaluated('ifg_vault', { + where: { name: { $startsWith: 'a' }, holder: { name: 'Pat' } }, + orderBy: [{ field: 'name', order: 'asc' }], + groupBy: ['name'], + aggregations: [{ function: 'max', field: 'rank', alias: 'm', filter: { name: 'alpha' } }], + expand: { holder: { object: 'ifg_holder', where: { name: 'Pat' }, orderBy: [{ field: 'name' }] } }, + })).not.toThrow(); + }); +}); + +describe('internal: true — a kernel that registers no protocol service still refuses', () => { + let kernel: ObjectKernel; + beforeEach(() => { + kernel = new ObjectKernel({ logger: { level: 'silent' }, gracefulShutdown: false }); + }); + afterEach(async () => { + if (kernel.getState() === 'running') await kernel.shutdown(); + }); + + it('registerProtocol: false without MetadataProtocolPlugin: the objectql slot refuses a member and serves the system context', async () => { + await kernel.use(new ObjectQLPlugin({ registerProtocol: false })); + await kernel.bootstrap(); + expect(() => kernel.getService('protocol')).toThrow(); + + const engine = kernel.getService('objectql'); + const recording = makeRecordingDriver(); + engine.registerDriver(recording.driver, true); + engine.registry.registerObject(holder as any); + engine.registry.registerObject(vault as any); + recording.rows.set('ifg_vault', [{ id: 'v1', name: 'alpha', rank: 1, digest: 'd1', holder: 'h1' }]); + + await expectRefused( + () => engine.find('ifg_vault', { where: { digest: { $startsWith: 'd' } }, context: MEMBER }), 'ifg_vault', 'digest', 'filter', + ); + await expectRefused( + () => engine.aggregate('ifg_vault', { groupBy: ['digest'], aggregations: [COUNT] }, { context: MEMBER }), 'ifg_vault', 'digest', 'groupBy', + ); + expect(recording.calls).toEqual([]); + await expect(engine.find('ifg_vault', { where: { digest: 'd1' }, context: SYSTEM })).resolves.toHaveLength(1); + await expect(engine.find('ifg_vault', { where: { name: 'alpha' }, context: MEMBER })).resolves.toHaveLength(1); + }); +}); diff --git a/packages/objectql/src/internal-field-evaluation-guard.ts b/packages/objectql/src/internal-field-evaluation-guard.ts new file mode 100644 index 00000000000..a5773c2941f --- /dev/null +++ b/packages/objectql/src/internal-field-evaluation-guard.ts @@ -0,0 +1,272 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The `internal: true` flag's EVALUATE refusal — the ONE definition + * (route-ownership rule 1). The engine owns it because the engine owns the + * flag: `internal: true` means "the value is never returned on a generic data + * exit", and the engine is what omits such a column from every row + * ({@link collectInternalReadFields}, the one flag reader). A filter or sort on + * such a column is a confirmation oracle (a predicate matches the row only + * when the guess is right, an order leaks the comparative value) and a group + * key or aggregation operand IS the value, so those positions are REFUSED + * rather than evaluated: ADR-0112 `INVALID_FIELD` / 400, located at the + * object, the field and the position. + * + * ## Two ways in, one judgement + * + * - {@link rejectInternalFieldEvaluation} — the engine's own read path + * (`find`, `findOne`, `count`, `aggregate`), for every caller but a SYSTEM + * context. It is what makes the refusal independent of the `protocol` + * service: a host that registers none, or a protocol other than the + * shipped one, still refuses, on every transport that hands a caller's + * query to the engine (the MCP stdio bridge, `callData`'s fallbacks). + * - {@link assertNoInternalFieldEvaluated} — the same judgement for every + * caller, system included, served as `ObjectQL#assertNoInternalFieldEvaluated` + * (`QueryEvaluationGate` in `@objectstack/spec/contracts`). The data door + * asks it before the engine runs, with its relation-target exposure + * question hooked in at each hop, so its answer and its order are the ones + * it always gave; `callData`'s `aggregate` arm asks it for the system + * callers that arm admits. + * + * ## Who it binds on the engine path + * + * Every caller except `context.isSystem === true`, the credential guard's line + * ({@link rejectCredentialEvaluation}). The server's own consumers that match + * on a flagged column without ever returning it — the API-key verifier's + * `where: { key }`, the share-link, SCIM-credential and approval-token + * lookups, the better-auth adapter — all run as system, and the + * security/sharing middlewares and `beforeFind` hooks inject their filters + * after this judgement, so none of them is judged here. + * + * ## Positions + * + * - `where` (and a direct bag's `filter` alias) and each + * `aggregations[i].filter` — every column a filter READS: each key's head + * and every cross-field `{ $field }` comparand, at any depth; + * - a NESTED-RELATION condition inside any of those + * (`{ LOOKUP: { RELATED_FIELD: … } }`), judged against the RELATED object's + * own flagged fields at `PARAM.LOOKUP`; + * - `orderBy` (`sort`), `groupBy`, and `aggregations[i].field`; + * - every `expand` entry's own `where` / `orderBy`, at every depth, judged + * against that level's TARGET object. + * + * An object with no usable field map (none, an array, an empty one) is judged + * on nothing, and its relation sites are not found: there is no declaration + * to read a flag from. + */ + +import { StandardErrorCode } from '@objectstack/spec/api'; +import { REFERENCE_VALUE_TYPES, isFilterAST, parseFilterAST, referenceTargetOf } from '@objectstack/spec/data'; +import type { InternalFieldEvaluationOptions, InternalFieldEvaluationQuery } from '@objectstack/spec/contracts'; +import { relationConditionSites } from '@objectstack/core'; +import { collectInternalReadFields } from './secret-fields.js'; + +/** A backstop against a self-referential in-process value. */ +const INTERNAL_FIELD_WALK_DEPTH = 8; + +/** The slice of an execution context the engine path reads. */ +interface InternalFieldEvaluationContext { + isSystem?: unknown; +} + +/** How the walk reads an object's declaration by name. */ +type SchemaOf = (name: string) => unknown; + +interface ObjectGate { + fields: Record; + internal: ReadonlySet; +} + +function isPlainRecord(value: unknown): value is Record { + return value !== null && typeof value === 'object' && !Array.isArray(value); +} + +function headSegment(name: string): string { + const dot = name.indexOf('.'); + return dot === -1 ? name : name.slice(0, dot); +} + +function gateOf(schemaOf: SchemaOf, object: string): ObjectGate | null { + const schema = schemaOf(object) as { fields?: unknown } | undefined | null; + const fields = schema?.fields; + if (!fields || typeof fields !== 'object' || Array.isArray(fields)) return null; + if (Object.keys(fields).length === 0) return null; + return { fields: fields as Record, internal: new Set(collectInternalReadFields(schema as never)) }; +} + +/** + * Every column a filter READS: each condition key's head and every `{ $field }` + * comparand's head, at any depth, combinators and operator bags descended. + */ +function collectFilterReads(root: unknown, out: Set): void { + const seenAsCondition = new WeakSet(); + const seenAsComparand = new WeakSet(); + const pending: Array<{ node: unknown; condition: boolean }> = [{ node: root, condition: true }]; + while (pending.length > 0) { + const { node, condition } = pending.pop() as { node: unknown; condition: boolean }; + if (node === null || typeof node !== 'object') continue; + const seen = condition ? seenAsCondition : seenAsComparand; + if (seen.has(node)) continue; + seen.add(node); + if (Array.isArray(node)) { + for (const member of node) pending.push({ node: member, condition }); + continue; + } + const record = node as Record; + if (!condition && typeof record.$field === 'string') out.add(headSegment(record.$field)); + for (const [key, value] of Object.entries(record)) { + if (condition && !key.startsWith('$')) { + out.add(headSegment(key)); + pending.push({ node: value, condition: false }); + } else { + pending.push({ node: value, condition }); + } + } + } +} + +/** The fields an `orderBy` names (string or `{ field }` entries), head segment only. */ +function orderByFieldNames(orderBy: unknown): string[] { + if (!Array.isArray(orderBy)) return []; + return orderBy + .map((e) => (typeof e === 'string' ? e : (e as { field?: unknown })?.field)) + .filter((f): f is string => typeof f === 'string' && f.length > 0) + .map((f) => headSegment(f)); +} + +/** The fields a `groupBy` names (string or `{ field }` bucket entries), head segment only. */ +function groupByFieldNames(groupBy: unknown): string[] { + if (!Array.isArray(groupBy)) return []; + return groupBy + .map((e) => (typeof e === 'string' ? e : (e as { field?: unknown })?.field)) + .filter((f): f is string => typeof f === 'string' && f.length > 0) + .map((f) => headSegment(f)); +} + +/** Throw the refusal for the first of `names` that `internal` holds, or return. */ +function refuseInternalNames(object: string, names: readonly string[], internal: ReadonlySet, param: string): void { + const hit = names.filter((n) => internal.has(n)); + if (hit.length === 0) return; + const first = hit[0] as string; + const verb = param.endsWith('sort') ? 'sort' : param === 'groupBy' ? 'group' : param.includes('filter') ? 'filter' : 'query'; + const err = new Error( + `Cannot ${verb} '${object}' by '${first}' (${param}): the query was not run. The ${first} field ` + + 'is declared `internal: true`, so its value is never returned on a generic data exit — the engine ' + + 'omits it from every row. ' + + (verb === 'sort' + ? 'A sort on it orders by that withheld value' + : verb === 'group' + ? 'A group key would be the withheld value, served as the key' + : 'A filter on it confirms a guessed value by whether the row comes back') + + ', so it is refused rather than evaluated. Use another field.', + ) as Error & { + code?: string; status?: number; httpStatus?: number; field?: string; fields?: string[]; object?: string; param?: string; + }; + // ADR-0112 envelope: `status` for the HTTP doors, `httpStatus` for a + // consumer holding the thrown error. + err.code = StandardErrorCode.enum.INVALID_FIELD; + err.status = 400; + err.httpStatus = 400; + err.field = first; + err.fields = hit; + err.object = object; + err.param = param; + throw err; +} + +function judgeFilter( + object: string, + where: unknown, + param: string, + depth: number, + schemaOf: SchemaOf, + options: InternalFieldEvaluationOptions | undefined, +): void { + if (where == null || depth > INTERNAL_FIELD_WALK_DEPTH) return; + const gate = gateOf(schemaOf, object); + if (!gate) return; + const condition = isFilterAST(where) ? parseFilterAST(where) : where; + if (gate.internal.size > 0) { + const reads = new Set(); + collectFilterReads(condition, reads); + refuseInternalNames(object, [...reads], gate.internal, param); + } + for (const site of relationConditionSites(gate.fields, condition)) { + const position = `${param}.${site.field}`; + options?.beforeRelationTarget?.(site.target, position); + judgeFilter(site.target, site.condition, position, depth + 1, schemaOf, options); + } +} + +function judgeExpand( + source: string, + expand: unknown, + prefix: string, + depth: number, + schemaOf: SchemaOf, + options: InternalFieldEvaluationOptions | undefined, +): void { + if (!isPlainRecord(expand) || depth > INTERNAL_FIELD_WALK_DEPTH) return; + const gate = gateOf(schemaOf, source); + if (!gate) return; + for (const [rel, entry] of Object.entries(expand)) { + if (!isPlainRecord(entry)) continue; + const sourceField = gate.fields[rel]; + const target = sourceField != null && REFERENCE_VALUE_TYPES.has(sourceField.type) + ? referenceTargetOf(sourceField) + : undefined; + if (!target) continue; + const path = `${prefix}['${rel}']`; + judgeFilter(target, entry.where, `${path}.filter`, 0, schemaOf, options); + const targetGate = gateOf(schemaOf, target); + if (targetGate) refuseInternalNames(target, orderByFieldNames(entry.orderBy), targetGate.internal, `${path}.sort`); + judgeExpand(target, entry.expand, `${path}.expand`, depth + 1, schemaOf, options); + } +} + +/** + * Refuse a read whose filter, sort, group key, aggregation operand, + * per-aggregation filter or `expand` tree evaluates a field declared + * `internal: true` — for EVERY caller. See the module header for positions. + */ +export function assertNoInternalFieldEvaluated( + object: string, + query: InternalFieldEvaluationQuery | undefined, + schemaOf: SchemaOf, + options?: InternalFieldEvaluationOptions, +): void { + if (!query) return; + const gate = gateOf(schemaOf, object); + if (!gate) return; + judgeFilter(object, query.where, 'filter', 0, schemaOf, options); + judgeFilter(object, query.filter, 'filter', 0, schemaOf, options); + refuseInternalNames(object, orderByFieldNames(query.orderBy), gate.internal, 'sort'); + refuseInternalNames(object, groupByFieldNames(query.groupBy), gate.internal, 'groupBy'); + if (Array.isArray(query.aggregations)) { + query.aggregations.forEach((agg: unknown, i: number) => { + const field = isPlainRecord(agg) ? agg.field : undefined; + if (typeof field === 'string' && field !== '*') { + refuseInternalNames(object, [field], gate.internal, `aggregations[${i}].field`); + } + judgeFilter(object, isPlainRecord(agg) ? agg.filter : undefined, `aggregations[${i}].filter`, 0, schemaOf, options); + }); + } + judgeExpand(object, query.expand, 'expand', 0, schemaOf, options); +} + +/** + * The engine read path's call: {@link assertNoInternalFieldEvaluated} for every + * caller but a system context. Called by `find`, `findOne`, `count` and + * `aggregate` right after the `where` array-form lowering, before the + * credential guard, any `search` expansion, token resolution, middleware or + * driver. + */ +export function rejectInternalFieldEvaluation( + object: string, + query: InternalFieldEvaluationQuery | undefined, + context: InternalFieldEvaluationContext | undefined, + schemaOf: SchemaOf, +): void { + if (!query || context?.isSystem === true) return; + assertNoInternalFieldEvaluated(object, query, schemaOf); +} diff --git a/packages/objectql/src/internal-fields.test.ts b/packages/objectql/src/internal-fields.test.ts index 5b11b83239c..e5864cd2a12 100644 --- a/packages/objectql/src/internal-fields.test.ts +++ b/packages/objectql/src/internal-fields.test.ts @@ -288,13 +288,17 @@ describe('#7728: the `internal` field flag omits a value from the generic data p }); it('keeps the field usable as a WHERE filter — the verifier lookup', async () => { - // `resolveApiKeyPrincipal` does exactly this: match the at-rest hash, - // then read `expires_at`/`user_id`/`scopes` off the row. If the flag were - // implemented by dropping the column from the QUERY instead of from the - // response, this returns nothing and authentication breaks platform-wide. + // `resolveApiKeyPrincipal` does exactly this, as a system context: match + // the at-rest hash, then read `expires_at`/`user_id`/`scopes` off the + // row. If the flag were implemented by dropping the column from the + // QUERY instead of from the response, this returns nothing and + // authentication breaks platform-wide. Any other caller is refused the + // filter (`internal-field-evaluation-guard.test.ts`); the system context + // is the server's own lookup. const created = await seed(); + const SYSTEM = { isSystem: true } as any; - const found = await ctx.engine.find('itest_api_key', { where: { key: HASH, revoked: false }, limit: 1 }); + const found = await ctx.engine.find('itest_api_key', { where: { key: HASH, revoked: false }, limit: 1, context: SYSTEM }); expect(found).toHaveLength(1); expect((found[0] as any).id).toBe(created.id); // …and the matched row still hands back the columns the verifier reads, @@ -303,15 +307,16 @@ describe('#7728: the `internal` field flag omits a value from the generic data p expect(Object.keys(found[0] as any)).not.toContain('key'); // A wrong hash still misses — the filter is real, not ignored. - expect(await ctx.engine.find('itest_api_key', { where: { key: 'sha256:wrong' } })).toHaveLength(0); + expect(await ctx.engine.find('itest_api_key', { where: { key: 'sha256:wrong' }, context: SYSTEM })).toHaveLength(0); }); it('survives repeated reads — the strip is not cumulative on storage', async () => { await seed(); + const SYSTEM = { isSystem: true } as any; for (let i = 0; i < 3; i++) { - await ctx.engine.find('itest_api_key', { where: { key: HASH } }); + await ctx.engine.find('itest_api_key', { where: { key: HASH }, context: SYSTEM }); } - const found = await ctx.engine.find('itest_api_key', { where: { key: HASH } }); + const found = await ctx.engine.find('itest_api_key', { where: { key: HASH }, context: SYSTEM }); expect(found).toHaveLength(1); }); }); diff --git a/packages/metadata-protocol/src/protocol.data-door-internal-field-positions.test.ts b/packages/objectql/src/protocol-data-door-internal-field-positions.test.ts similarity index 94% rename from packages/metadata-protocol/src/protocol.data-door-internal-field-positions.test.ts rename to packages/objectql/src/protocol-data-door-internal-field-positions.test.ts index db1917256ae..4d9829d9675 100644 --- a/packages/metadata-protocol/src/protocol.data-door-internal-field-positions.test.ts +++ b/packages/objectql/src/protocol-data-door-internal-field-positions.test.ts @@ -21,12 +21,20 @@ * * The position enumeration is the load-bearing shape: a position added later * that does not pass through this door turns a case here red. + * + * The walk is the ENGINE's (`internal-field-evaluation-guard.ts`, the + * `QueryEvaluationGate` member the door asks), so this file lives beside it: + * the protocol double's engine carries that real gate over the same schemas, + * and every case below is the door and the engine's one definition together. + * (`@objectstack/metadata-protocol` cannot import this package even as a dev + * dependency — see `plugin.step2.test.ts`.) */ import { describe, expect, it, vi } from 'vitest'; import type { ServiceObject } from '@objectstack/spec/data'; import { assertEngineFindOnePredicate } from '@objectstack/metadata-core'; -import { ObjectStackProtocolImplementation } from './protocol.js'; +import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; +import { assertNoInternalFieldEvaluated } from './internal-field-evaluation-guard.js'; const VAULT: ServiceObject = { name: 'pv_vault', @@ -78,7 +86,13 @@ function makeProtocol() { }); const aggregate = vi.fn(async () => [{ n: 1 }]); const count = vi.fn(async () => (ROWS.pv_vault?.length ?? 0)); - const engine: any = { registry: { getObject: (n: string) => SCHEMAS[n] }, find, findOne, count, aggregate }; + const engine: any = { + registry: { getObject: (n: string) => SCHEMAS[n] }, + find, findOne, count, aggregate, + // The engine's one definition, over the same schemas the registry serves. + assertNoInternalFieldEvaluated: (object: string, query: any, options?: any) => + assertNoInternalFieldEvaluated(object, query, (n) => SCHEMAS[n], options), + }; return { p: new ObjectStackProtocolImplementation(engine), find, aggregate, count }; } diff --git a/packages/qa/dogfood/test/mcp-internal-field-evaluate-positions.dogfood.test.ts b/packages/qa/dogfood/test/mcp-internal-field-evaluate-positions.dogfood.test.ts index 1a13acd04e2..2b03ca23299 100644 --- a/packages/qa/dogfood/test/mcp-internal-field-evaluate-positions.dogfood.test.ts +++ b/packages/qa/dogfood/test/mcp-internal-field-evaluate-positions.dogfood.test.ts @@ -10,16 +10,21 @@ * * ## What is pinned * - * The data door refuses those positions for every caller - * (`assertNoInternalFieldEvaluated`, `@objectstack/metadata-protocol`), before - * the engine is asked: the engine omits such a field from every row, but it - * still evaluates one named in a predicate or an order. Three MCP mouths hand a - * caller's query to the engine without passing through `findData`, where the - * door asks its gate — the stdio transport's `query_records` and - * `aggregate_records`, and the HTTP bridge's `aggregate_records` — and each now - * asks that same gate of the `protocol` service. The HTTP bridge's - * `query_records` reaches `findData` itself; it is driven here too, as the - * parity row the other three are held to. + * The refusal is the ENGINE's one definition (`QueryEvaluationGate`, + * `@objectstack/objectql`): the engine omits such a field from every row, and + * its own read path refuses to evaluate one for every non-system caller. The + * data door asks the same member before the engine runs, for every caller. + * Three MCP mouths hand a caller's query to the engine without passing through + * `findData` — the stdio transport's `query_records` and `aggregate_records`, + * and the HTTP bridge's `aggregate_records` — and each meets that one + * definition (stdio on the engine's path, the HTTP arm by asking the engine's + * member). The HTTP bridge's `query_records` reaches `findData` itself; it is + * driven here too, as the parity row the other three are held to. + * + * The stdio transport is driven a second time on a host that registers NO + * `protocol` service (the plugin's context answers that slot with "not + * registered", as a kernel composed with `registerProtocol: false` and no + * `MetadataProtocolPlugin` does): the refusal must not depend on that service. * * Every refusal is compared with what REST answers for the same body — * `code`, `status` and `message`, read from REST in this run, never a sentence @@ -46,9 +51,8 @@ * process cannot give a test its stdin), and `start()`'s claim of the real * stdin/stdout is the one call stubbed. * - * The negative control — every stdio row and the HTTP aggregate rows served, - * or answered in the engine's words, with the two MCP-side gate calls removed — - * is a one-time ablation cited in the PR, not a permanent case. + * The negative control — the stdio rows served with the engine's read-path + * call removed — is a one-time ablation cited in the PR, not a permanent case. */ import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; @@ -185,6 +189,7 @@ describe('MCP data tools refuse an internal field in every evaluate position, as let stack: VerifyStack; const sessionTokens: Record = {}; const stdio: Record = {}; + const stdioWithoutProtocol: Record = {}; const started: MCPServerPlugin[] = []; const originalKey = process.env.OS_MCP_STDIO_API_KEY; @@ -211,11 +216,16 @@ describe('MCP data tools refuse an internal field in every evaluate position, as * `autoStart` over this kernel's own services. Registrations stay local, so * the showcase's own MCP plugin is untouched. */ - async function stdioSession(apiKey: string): Promise { + async function stdioSession(apiKey: string, opts: { withoutProtocol?: boolean } = {}): Promise { const local = new Map(); const host = { registerService: (name: string, service: unknown) => { local.set(name, service); }, - getService: (name: string): unknown => (local.has(name) ? local.get(name) : stack.kernel.getService(name)), + getService: (name: string): unknown => { + if (local.has(name)) return local.get(name); + // A host that registers no protocol: the kernel's own "not registered" answer. + if (opts.withoutProtocol && name === 'protocol') throw new Error(`Service 'protocol' not found`); + return stack.kernel.getService(name); + }, replaceService: (name: string, service: unknown) => { local.set(name, service); }, hook: () => {}, trigger: async () => {}, @@ -301,7 +311,9 @@ describe('MCP data tools refuse an internal field in every evaluate position, as await ql.insert('sys_user_permission_set', { user_id: memberId, permission_set: set.name }, SYS); for (const persona of ['admin', 'member'] as const) { - stdio[persona] = await stdioSession(await mintKey(sessionTokens[persona]!, `mcp-internal-field-${persona}`)); + const key = await mintKey(sessionTokens[persona]!, `mcp-internal-field-${persona}`); + stdio[persona] = await stdioSession(key); + stdioWithoutProtocol[persona] = await stdioSession(key, { withoutProtocol: true }); } }, 240_000); @@ -315,6 +327,7 @@ describe('MCP data tools refuse an internal field in every evaluate position, as const transports = { stdio: (persona: string) => stdio[persona]!, + 'stdio, no protocol service': (persona: string) => stdioWithoutProtocol[persona]!, http: (persona: string) => httpCall(sessionTokens[persona]!), } as const; diff --git a/packages/runtime/src/action-execution-calldata-aggregate-internal-field.test.ts b/packages/runtime/src/action-execution-calldata-aggregate-internal-field.test.ts index 6c7d451cc5e..b4559438b49 100644 --- a/packages/runtime/src/action-execution-calldata-aggregate-internal-field.test.ts +++ b/packages/runtime/src/action-execution-calldata-aggregate-internal-field.test.ts @@ -1,23 +1,26 @@ // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. /** - * `callData`'s `aggregate` arm — the HTTP MCP bridge's `aggregate_records` — - * hands a caller's `where`, `groupBy` and `aggregations` to the ObjectQL engine - * without passing through the protocol's `findData`, where the data door - * refuses to EVALUATE a field declared `internal: true`. So the arm asks the - * door's own gate (`assertNoInternalFieldEvaluated`) of the `protocol` service - * before the engine aggregates, and throws what the door throws. + * `callData` — the HTTP MCP bridge's data mouth — refuses to EVALUATE a field + * declared `internal: true` exactly as the data door does, whether or not the + * host registers a `protocol` service. * - * Driven against the REAL `ObjectStackProtocolImplementation` registered as the - * `protocol` service, and asserted against it: for each position the refusal is - * the one `findData` itself answers for the same body — code, status, object, - * field, position and message — and the engine never aggregates. Every caller - * meets it, a system context included, as at the door. CONTROLS: an ordinary - * field in every position reaches the engine unchanged, and a host that - * registers no protocol has no door to agree with and is served as before. + * The refusal is the ENGINE's one definition (`QueryEvaluationGate` on the + * `objectql` slot, which the engine's own read path also applies to every + * non-system caller). The `aggregate` arm never reaches `findData`, so it asks + * that member of the engine itself, for every caller it admits, a system + * context included; the `query` arm's fallback on a host with no protocol does + * the same before `ql.find`. + * + * Driven against a REAL `ObjectQL` over a recording driver, and compared with + * the REAL `ObjectStackProtocolImplementation`'s `findData` over the same + * engine: for each position the refusal is the door's — code, status, object, + * field, position and message — and the driver is never asked. CONTROLS: an + * ordinary field in every position is served, with and without a protocol. */ -import { describe, expect, it, vi } from 'vitest'; +import { beforeEach, describe, expect, it } from 'vitest'; +import { ObjectQL } from '@objectstack/objectql'; import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol'; import { callData, type ActionExecutionDeps } from './action-execution.js'; import type { HttpProtocolContext } from './http-dispatcher.js'; @@ -26,85 +29,146 @@ const REQ = { request: {} } as HttpProtocolContext; const MEMBER = { userId: 'ai_member', isSystem: false, positions: [], permissions: [] } as any; const ADMIN = { userId: 'ai_admin', isSystem: false, roles: ['admin'], isPlatformAdmin: true, positions: [], permissions: [] } as any; const SYSTEM = { isSystem: true } as any; +const PERSONAS = { member: MEMBER, admin: ADMIN, system: SYSTEM } as const; const F = 'pin'; -const SCHEMAS: Record = { - ai_vault: { - name: 'ai_vault', - label: 'Vault', - fields: { - id: { name: 'id', label: 'ID', type: 'text' }, - name: { name: 'name', label: 'Name', type: 'text' }, - rank: { name: 'rank', label: 'Rank', type: 'number' }, - [F]: { name: F, label: 'Pin', type: 'text', internal: true }, - }, +const VAULT = { + name: 'ai_vault', + label: 'Vault', + fields: { + id: { name: 'id', label: 'ID', type: 'text' }, + name: { name: 'name', label: 'Name', type: 'text' }, + rank: { name: 'rank', label: 'Rank', type: 'number' }, + [F]: { name: F, label: 'Pin', type: 'text', internal: true }, }, }; -function makeHarness(opts: { withProtocol?: boolean } = {}) { - const aggregate = vi.fn(async () => [{ n: 1 }]); - const find = vi.fn(async () => [{ id: 'v1', name: 'alpha', rank: 1 }]); - const engine: any = { registry: { getObject: (n: string) => SCHEMAS[n] }, find, aggregate }; - const protocol = new ObjectStackProtocolImplementation(engine); +function makeRecordingDriver() { + const rows: Record[] = [{ id: 'v1', name: 'alpha', rank: 1, [F]: 'p1' }]; + const calls: string[] = []; + const driver: any = { + name: 'memory', version: '0.0.0', supports: {}, + async connect() {}, async disconnect() {}, async checkHealth() { return true; }, + async execute() { return null; }, + async find(object: string) { calls.push(`find:${object}`); return rows.map((r) => ({ ...r })); }, + async findOne(object: string) { calls.push(`findOne:${object}`); return { ...rows[0] }; }, + async count(object: string) { calls.push(`count:${object}`); return rows.length; }, + async aggregate(object: string) { calls.push(`aggregate:${object}`); return [{ n: rows.length }]; }, + async create(_o: string, data: Record) { return { id: 'v2', ...data }; }, + async update(_o: string, id: string, data: Record) { return { id, ...data }; }, + async updateMany() { return 0; }, async delete() { return true; }, async deleteMany() { return 0; }, + async upsert(_o: string, data: Record) { return data; }, + async bulkCreate() { return []; }, async bulkUpdate() { return []; }, async bulkDelete() {}, + async beginTransaction() { return { commit: async () => {}, rollback: async () => {} }; }, + async commit() {}, async rollback() {}, + }; + return { driver, calls }; +} + +async function makeHarness(opts: { withProtocol: boolean }) { + const engine = new ObjectQL(); + const recording = makeRecordingDriver(); + engine.registerDriver(recording.driver, true); + await engine.init(); + engine.registry.registerObject(VAULT as any); + // The door over the same engine — the reference every answer is compared with. + const door = new ObjectStackProtocolImplementation(engine as any); const services: Record = { - metadata: { getObject: async (n: string) => SCHEMAS[n] }, + metadata: { getObject: async (n: string) => (n === VAULT.name ? VAULT : undefined) }, objectql: engine, - ...(opts.withProtocol === false ? {} : { protocol }), + ...(opts.withProtocol ? { protocol: door } : {}), }; const deps: ActionExecutionDeps = { resolveService: (async (_ctx: HttpProtocolContext, name: string) => services[name]) as any, getObjectQL: async () => engine, }; - return { deps, aggregate, protocol }; + return { deps, door, calls: recording.calls }; } const envelope = (e: any) => ({ code: e?.code, status: e?.status, object: e?.object, field: e?.field, param: e?.param, message: e?.message }); +async function doorAnswer(door: ObjectStackProtocolImplementation, query: Record) { + const err = await door.findData({ object: 'ai_vault', query: { ...query } as any }).then(() => undefined, (e) => e); + expect(envelope(err)).toMatchObject({ code: 'INVALID_FIELD', status: 400, object: 'ai_vault', field: F }); + return err; +} + const COUNT = { function: 'count', alias: 'n' }; -const POSITIONS: Array<[string, { where?: unknown; groupBy?: unknown[]; aggregations: unknown[] }]> = [ +const AGGREGATE_POSITIONS: Array<[string, { where?: unknown; groupBy?: unknown[]; aggregations: unknown[] }]> = [ ['where', { where: { [F]: 'x' }, aggregations: [COUNT] }], ['where ($startsWith)', { where: { [F]: { $startsWith: 'x' } }, aggregations: [COUNT] }], ['group key', { groupBy: [F], aggregations: [COUNT] }], ['aggregation operand', { aggregations: [{ function: 'max', field: F, alias: 'm' }] }], ["aggregation's own filter", { aggregations: [{ ...COUNT, filter: { [F]: 'x' } }] }], ]; +const QUERY_POSITIONS: Array<[string, Record]> = [ + ['filter ($eq)', { where: { [F]: { $eq: 'x' } } }], + ['filter ($startsWith)', { where: { [F]: { $startsWith: 'x' } } }], + ['sort', { orderBy: [{ field: F, order: 'asc' }] }], +]; + +for (const withProtocol of [true, false]) { + const host = withProtocol ? 'a host with a protocol service' : 'a host that registers no protocol service'; -describe('callData aggregate asks the data door\'s internal-field gate before the engine aggregates', () => { - for (const [label, body] of POSITIONS) { - for (const [persona, context] of Object.entries({ member: MEMBER, admin: ADMIN, system: SYSTEM })) { - it(`${label} — ${persona}: findData's own refusal, and the engine never aggregates`, async () => { - const h = makeHarness(); - const door = await h.protocol.findData({ object: 'ai_vault', query: { ...body } as any }).then(() => undefined, (e) => e); - expect(envelope(door)).toMatchObject({ code: 'INVALID_FIELD', status: 400, object: 'ai_vault', field: F }); + describe(`callData refuses an internal field in every evaluate position — ${host}`, () => { + for (const [label, body] of AGGREGATE_POSITIONS) { + for (const [persona, context] of Object.entries(PERSONAS)) { + it(`aggregate ${label} — ${persona}: the door's refusal, and the driver is never asked`, async () => { + const h = await makeHarness({ withProtocol }); + const door = await doorAnswer(h.door, body); + const err = await callData(h.deps, REQ, 'aggregate', { object: 'ai_vault', ...body }, undefined, undefined, context) + .then(() => undefined, (e) => e); + expect(err, 'the aggregate arm evaluated a position the door refuses').toBeDefined(); + expect(envelope(err)).toEqual(envelope(door)); + expect(h.calls).toEqual([]); + }); + } + } - const err = await callData(h.deps, REQ, 'aggregate', { object: 'ai_vault', ...body }, undefined, undefined, context) - .then(() => undefined, (e) => e); - expect(err, 'the aggregate arm evaluated a position the door refuses').toBeDefined(); - expect(envelope(err)).toEqual(envelope(door)); - expect(h.aggregate).not.toHaveBeenCalled(); - }); + for (const [label, query] of QUERY_POSITIONS) { + for (const [persona, context] of Object.entries(PERSONAS)) { + it(`query ${label} — ${persona}: the door's refusal, and the driver is never asked`, async () => { + const h = await makeHarness({ withProtocol }); + const door = await doorAnswer(h.door, query); + const err = await callData(h.deps, REQ, 'query', { object: 'ai_vault', query }, undefined, undefined, context) + .then(() => undefined, (e) => e); + expect(err, 'the query arm evaluated a position the door refuses').toBeDefined(); + expect(envelope(err)).toEqual(envelope(door)); + expect(h.calls).toEqual([]); + }); + } } - } - it('CONTROL — ordinary fields in every position reach the engine unchanged', async () => { - const h = makeHarness(); - const body = { - where: { name: 'alpha' }, - groupBy: ['name'], - aggregations: [{ function: 'max', field: 'rank', alias: 'm', filter: { name: 'alpha' } }], - }; - const res = await callData(h.deps, REQ, 'aggregate', { object: 'ai_vault', ...body }, undefined, undefined, MEMBER); - expect(res).toEqual({ object: 'ai_vault', rows: [{ n: 1 }] }); - expect(h.aggregate).toHaveBeenCalledTimes(1); - expect(h.aggregate).toHaveBeenCalledWith('ai_vault', expect.objectContaining(body)); + it('CONTROL — ordinary fields in every position are served', async () => { + const h = await makeHarness({ withProtocol }); + const res: any = await callData(h.deps, REQ, 'aggregate', { + object: 'ai_vault', + where: { name: 'alpha' }, + groupBy: ['name'], + aggregations: [{ function: 'max', field: 'rank', alias: 'm', filter: { name: 'alpha' } }], + }, undefined, undefined, MEMBER); + expect(res).toMatchObject({ object: 'ai_vault' }); + expect(Array.isArray(res.rows)).toBe(true); + const listed: any = await callData(h.deps, REQ, 'query', { + object: 'ai_vault', query: { where: { name: 'alpha' }, orderBy: [{ field: 'name', order: 'asc' }] }, + }, undefined, undefined, MEMBER); + expect(listed.records.map((r: any) => r.id)).toEqual(['v1']); + // Served, and the flagged column still never on a row. + expect(listed.records[0]).not.toHaveProperty(F); + expect(h.calls.length).toBeGreaterThan(0); + }); }); +} + +describe('callData — the system context is refused at the door, and the engine path serves it', () => { + let h: Awaited>; + beforeEach(async () => { h = await makeHarness({ withProtocol: false }); }); - it('CONTROL — a host that registers no protocol has no door to agree with, and is served as before', async () => { - const h = makeHarness({ withProtocol: false }); - const res = await callData( - h.deps, REQ, 'aggregate', { object: 'ai_vault', where: { [F]: 'x' }, aggregations: [COUNT] }, undefined, undefined, MEMBER, - ); - expect(res).toEqual({ object: 'ai_vault', rows: [{ n: 1 }] }); - expect(h.aggregate).toHaveBeenCalledTimes(1); + it('the server\'s own engine read (no door in between) still matches on the flagged column', async () => { + const engine = await h.deps.getObjectQL(REQ); + if (!engine) throw new Error('the harness serves an engine'); + const rows = await engine.find('ai_vault', { where: { [F]: 'p1' }, context: SYSTEM }); + expect(rows.map((r: any) => r.id)).toEqual(['v1']); + expect(rows[0]).not.toHaveProperty(F); }); }); diff --git a/packages/runtime/src/action-execution.ts b/packages/runtime/src/action-execution.ts index f1f67521ba8..c4f84dea7ec 100644 --- a/packages/runtime/src/action-execution.ts +++ b/packages/runtime/src/action-execution.ts @@ -22,7 +22,7 @@ import { templateSlotSource } from '@objectstack/spec/automation'; import { EngineQueryOptionsSchema, DATA_ACTION_TO_API_OPERATION } from '@objectstack/spec/data'; import type { ExecutionContext } from '@objectstack/spec/kernel'; import { isStoredMetadataBodyObject } from '@objectstack/spec/kernel'; -import type { IHttpServer, IObjectQLEngine, ServiceSlotContract, ServiceSlotContracts } from '@objectstack/spec/contracts'; +import type { IHttpServer, IObjectQLEngine, InternalFieldEvaluationQuery, QueryEvaluationGate, ServiceSlotContract, ServiceSlotContracts } from '@objectstack/spec/contracts'; import { standardErrorCodeForHttpStatus, type MetadataProtocol } from '@objectstack/spec/api'; // [#15942 / #16293] The confirmation member's SPELLING is the contract, so it // is imported, never hand-spelled: the door that refuses and the client that @@ -558,6 +558,19 @@ export async function refuseUnexposedDataAction(deps: ActionExecutionDeps, if (credentialRefusal) throw credentialRefusal; } +/** + * Ask `engine` the `internal: true` evaluate refusal (`QueryEvaluationGate` + * on the `objectql` slot contract, `IObjectQLEngine`) for `object` and + * `query`, for every caller — or nothing, when the value handed in is not an + * engine that carries it (a test double, a foreign engine, the raw driver the + * MCP multi-env path threads through), which then answers for itself. + */ +function assertNoInternalFieldEvaluatedBy(engine: unknown, object: string, query: InternalFieldEvaluationQuery): void { + const gate = (engine as Partial | null | undefined)?.assertNoInternalFieldEvaluated; + if (typeof gate !== 'function') return; + gate.call(engine, object, query); +} + export async function callData(deps: ActionExecutionDeps, requestContext: HttpProtocolContext, action: string, @@ -846,6 +859,11 @@ export async function callData(deps: ActionExecutionDeps, if (exposureGated) { await refuseUnservedRelationTargets(deps, requestContext, ql, params.object, bag.where, scopeId); } + // …and the `internal: true` evaluate refusal for every caller, as + // the protocol path asks it inside `findData` — the engine's one + // definition (see the `aggregate` arm). A raw driver on the MCP + // multi-env path carries no such member and is not an engine. + assertNoInternalFieldEvaluatedBy(ql, params.object, bag); let all = await ql.find(params.object, findOpts(bag)); if (!Array.isArray(all) && all && (all as any).value) all = (all as any).value; if (!all) all = []; @@ -880,22 +898,23 @@ export async function callData(deps: ActionExecutionDeps, if (exposureGated) { await refuseUnservedRelationTargets(deps, requestContext, engine, params.object, params.where, scopeId); } - // The data door's evaluate refusal for a field declared - // `internal: true` — a filter (`where`, an aggregation's own - // `filter`), a group key or an aggregation operand on it. The query - // arm meets it inside `findData`; this arm never reaches `findData`, - // so it asks the protocol's ONE gate itself, before the engine, and - // throws the door's `400 INVALID_FIELD` word for word. ⛔ Never a - // copy of the gate here (route-ownership rule 1). Every caller, as - // at the door; a host with no protocol has no door to agree with - // and is served as before. - if (protocol && typeof protocol.assertNoInternalFieldEvaluated === 'function') { - protocol.assertNoInternalFieldEvaluated(params.object, { - where: params.where, - groupBy: params.groupBy, - aggregations: params.aggregations, - }); - } + // The `internal: true` evaluate refusal — a filter (`where`, an + // aggregation's own `filter`), a group key or an aggregation operand + // on such a field — asked of the ENGINE's one definition + // (`QueryEvaluationGate` on the `objectql` slot contract), the one + // the data door asks too, so the refusal is the door's word for + // word whether or not this host registers a `protocol` service. + // `engine.aggregate` applies it itself to every non-system caller; + // it is asked here as well so a system context this arm admits is + // refused like every caller at the door. ⛔ Never a copy of the + // walk here (route-ownership rule 1). The probe tolerates a test + // double or foreign engine without the member, which then answers + // for itself. + assertNoInternalFieldEvaluatedBy(engine, params.object, { + where: params.where, + groupBy: params.groupBy, + aggregations: params.aggregations, + }); const rows = await engine.aggregate( params.object, { diff --git a/packages/spec/api-surface/contracts.json b/packages/spec/api-surface/contracts.json index 2242855c52e..a51e4d4a294 100644 --- a/packages/spec/api-surface/contracts.json +++ b/packages/spec/api-surface/contracts.json @@ -191,6 +191,8 @@ "InboxQuery (interface)", "InstallPackageInput (interface)", "InstallPackageResult (interface)", + "InternalFieldEvaluationOptions (interface)", + "InternalFieldEvaluationQuery (interface)", "IntrospectedColumn (interface)", "IntrospectedIndex (interface)", "IntrospectedSchema (interface)", @@ -256,6 +258,7 @@ "PubSubHandler (type)", "PubSubMessage (interface)", "PublishOptions (interface)", + "QueryEvaluationGate (interface)", "QueueBackoffPolicy (interface)", "QueueExpansionPurpose (type)", "QueueHandler (type)", diff --git a/packages/spec/export-origins/contracts.json b/packages/spec/export-origins/contracts.json index cea404c877f..2c3310765ca 100644 --- a/packages/spec/export-origins/contracts.json +++ b/packages/spec/export-origins/contracts.json @@ -191,6 +191,8 @@ "InboxQuery": "src/contracts/notification-service.ts#InboxQuery (interface)", "InstallPackageInput": "src/contracts/package-service.ts#InstallPackageInput (interface)", "InstallPackageResult": "src/contracts/package-service.ts#InstallPackageResult (interface)", + "InternalFieldEvaluationOptions": "src/contracts/objectql-engine.ts#InternalFieldEvaluationOptions (interface)", + "InternalFieldEvaluationQuery": "src/contracts/objectql-engine.ts#InternalFieldEvaluationQuery (interface)", "IntrospectedColumn": "src/contracts/schema-diff-service.ts#IntrospectedColumn (interface)", "IntrospectedIndex": "src/contracts/schema-diff-service.ts#IntrospectedIndex (interface)", "IntrospectedSchema": "src/contracts/schema-diff-service.ts#IntrospectedSchema (interface)", @@ -256,6 +258,7 @@ "PubSubHandler": "src/contracts/cluster-service.ts#PubSubHandler (type)", "PubSubMessage": "src/contracts/cluster-service.ts#PubSubMessage (interface)", "PublishOptions": "src/contracts/cluster-service.ts#PublishOptions (interface)", + "QueryEvaluationGate": "src/contracts/objectql-engine.ts#QueryEvaluationGate (interface)", "QueueBackoffPolicy": "src/contracts/queue-service.ts#QueueBackoffPolicy (interface)", "QueueExpansionPurpose": "src/contracts/ownership-queue-service.ts#QueueExpansionPurpose (type)", "QueueHandler": "src/contracts/queue-service.ts#QueueHandler (type)", diff --git a/packages/spec/src/contracts/objectql-engine.ts b/packages/spec/src/contracts/objectql-engine.ts index d4850732e9e..713d41ed007 100644 --- a/packages/spec/src/contracts/objectql-engine.ts +++ b/packages/spec/src/contracts/objectql-engine.ts @@ -183,6 +183,65 @@ export interface EngineTransactionInfo { owned: boolean; } +/** + * The slice of a read {@link QueryEvaluationGate.assertNoInternalFieldEvaluated} + * judges. Every member is optional and is read as the generic data door hands + * it over: `where` (or its `filter` alias), `orderBy`, `groupBy`, + * `aggregations` and the `expand` tree. + */ +export interface InternalFieldEvaluationQuery { + where?: unknown; + filter?: unknown; + orderBy?: unknown; + groupBy?: unknown; + aggregations?: unknown; + expand?: unknown; +} + +/** + * Options for {@link QueryEvaluationGate.assertNoInternalFieldEvaluated}. + */ +export interface InternalFieldEvaluationOptions { + /** + * Called at every nested-relation condition the walk reaches, with the + * related object and the condition's position, BEFORE anything of the + * related object is judged. A door uses it to ask its own question of + * that hop first (the data door asks the related object's API exposure), + * so the first refusal a caller meets is the same whichever door it used. + * Whatever it throws propagates. + */ + beforeRelationTarget?: (target: string, param: string) => void; +} + +/** + * The engine's refusal to EVALUATE a field declared `internal: true`. + * + * Such a field is withheld from every generic data exit: the engine omits it + * from every row. A filter, sort, group key, aggregation operand or + * per-aggregation filter on it would still reveal the stored value by + * evaluating it, so the engine refuses those positions before any driver is + * asked: ADR-0112 `INVALID_FIELD` / 400, located at the object, the field and + * the position (`filter`, `sort`, `groupBy`, `aggregations[i].field`, + * `aggregations[i].filter`, `expand['REL'].filter`, `expand['REL'].sort`, a + * nested-relation condition at `filter.LOOKUP`). + * + * The engine applies it on its own read path (`find`, `findOne`, `count`, + * `aggregate`) for every caller but a system context, so a host that + * registers no `protocol` service, or a protocol other than the shipped one, + * still refuses. This member is the same judgement asked directly, for every + * caller, by a door that wants to answer before the engine runs (the data + * door, and a data path a system context can reach). It is ONE definition: + * a door asks it and never restates it. + */ +export interface QueryEvaluationGate { + /** Throw the refusal described on {@link QueryEvaluationGate}, or return. */ + assertNoInternalFieldEvaluated( + object: string, + query: InternalFieldEvaluationQuery, + options?: InternalFieldEvaluationOptions, + ): void; +} + /** * Options for {@link IObjectQLEngine.judgeFilter} (#20157). */ @@ -236,7 +295,7 @@ export type EngineFilterJudgement = * (#28001, that shape again). Their callers must work with an engine * that lacks them, so the optional mark states that requirement in the type. */ -export interface IObjectQLEngine extends IDataEngine { +export interface IObjectQLEngine extends IDataEngine, QueryEvaluationGate { // ── Schema access ──────────────────────────────────────────────────── /** * The registered schema for an object, or `undefined` — the write diff --git a/scripts/engine-double-contract.pinned.json b/scripts/engine-double-contract.pinned.json index 68632249ff5..3cf447d41ac 100644 --- a/scripts/engine-double-contract.pinned.json +++ b/scripts/engine-double-contract.pinned.json @@ -637,7 +637,7 @@ "pinned": 1 }, { - "file": "packages/metadata-protocol/src/protocol.data-door-internal-field-positions.test.ts", + "file": "packages/metadata-protocol/src/protocol.data-door-internal-field-delegation.test.ts", "verb": "findOne", "pinned": 1 }, @@ -2746,6 +2746,11 @@ "verb": "update", "pinned": 1 }, + { + "file": "packages/objectql/src/protocol-data-door-internal-field-positions.test.ts", + "verb": "findOne", + "pinned": 1 + }, { "file": "packages/objectql/src/protocol-delete-object-registry-heal.test.ts", "verb": "delete",