fix(lint,objectql)!: a formula reading a field its lookup target does not declare is refused at save, and no longer nulls its sibling formulas

.changeset/24410-lint-formula-hop-undeclared-field.md

@@ -0,0 +1,19 @@
+---
+'@objectstack/lint': minor
+---
+
+fix(lint)!: a formula that reads, through a reference, a field the reference's target does not declare is refused where it is written
+
+Clause-②: no (narrowing)
+
+<!-- adr-0087: not-required (no-migration-prescription) No spec key, stored shape or authorable spelling is added, removed or renamed. The field-formula pass now refuses one expression shape it accepted before: a one-hop read of a field the reference's target does not declare, which the runtime never computed a value for (the formula read null on every row). Nothing stored changes, so there is nothing for `objectstack migrate meta` to rewrite. -->
+
+**BREAKING** (an authoring narrowing), shipped as `minor` under the launch-window convention for breaking changes.
+
+The field-formula pass judges the field a formula reads one hop through a reference against the target object's fields: the fields it declares, the system columns the platform adds to it, and `id`. `os validate`, `os build` and the object save door (`PUT /api/v1/meta/object/:name`) run the same pass, under the existing `EXPRESSION_INVALID` rule.
+
+- FROM: `record.book.name`, where `book` is a lookup to an object that declares no `name`, saved with a 200 and read `null` on every row. TO: refused (`422 INVALID_METADATA` at the save door). The message names the field and the target: "unknown field `name` on `<target>`".
+- `record.book.title`, where the target declares `title`, passes as before.
+- A target the stack does not carry is not judged. At the save door the stack carries the live registry's objects in the written package's closure, so a target from a package it depends on is judged.
+
+What to do. Read a field the target declares, or declare the field on the target object. Do not declare it on the object that carries the formula. At `os build` and `os validate` this pass does not see a target field that an `objectExtensions` entry contributes, while the save door judges the merged object.

.changeset/24410-objectql-formula-hop-sibling-isolation.md

@@ -0,0 +1,17 @@
+---
+'@objectstack/objectql': patch
+---
+
+fix(objectql): a formula reading a field its reference's target does not declare no longer makes the other formulas reading through the same reference read null
+
+Clause-②: no
+
+<!-- adr-0087: not-required (no-migration-prescription) Nothing an author writes, and nothing stored, is added, removed or renamed. A valid formula that read null beside a misspelt sibling now reads its value; no spec key, export, option, object definition or stored shape changes, so there is nothing for `objectstack migrate meta` to rewrite. -->
+
+The formula read loads, once per page, every column the object's formulas read through each reference. When one formula named a field the target does not declare, the whole reference was answered as undeclared. Every formula reading through it then read `null` and reported the other formula's field.
+
+- The undeclared fields are left out of that shared read and judged for each formula on its own.
+- A formula that names one reads `null`, with a fault report naming only its own undeclared fields.
+- A formula that reads only declared fields through the same reference gets the value it gets when read alone (`?fields=`).
+
+What to do. Nothing. A formula that read `null` this way now reads its value. To find the formula at fault, re-save the object: the save door refuses the misspelt read and names the field.

packages/core/src/security/second-object-read-exposure.pin.test.ts

@@ -201,6 +201,11 @@ const CALLERS: Record<string, Classification> = {
     kind: 'not-a-served-read',
     reason: 'authoring-time static analysis of metadata; reads no record',
   },
+  // [#24410] A formula's one-hop read judged against its target's declared fields.
+  'packages/lint/src/validate-expressions.ts': {
+    kind: 'not-a-served-read',
+    reason: 'authoring-time static analysis of metadata; reads no record',
+  },
   'packages/metadata-protocol/src/seed-loader.ts': {
     kind: 'not-a-served-read',
     reason: "resolves a seed's references at install, under the system identity; serves nothing to a caller",

packages/lint/src/runtime-gate.object-formula-writes.test.ts

@@ -117,6 +117,46 @@ describe('#22019 — the object door dispatches the build\'s expression rule', (
   });
 });
 
+describe('#24410 — the object door refuses a formula hop to a field the target does not declare', () => {
+  /** The lookup's target, delivered by ANOTHER package the written one depends on. */
+  const book = {
+    name: 'fx_book', label: 'Book', sharingModel: 'private', _packageId: 'com.example.library',
+    fields: { title: { type: 'text', label: 'Title' } },
+  };
+  const loan = (expression: string) => ({
+    name: 'fx_loan',
+    label: 'Loan',
+    sharingModel: 'private',
+    fields: {
+      book: { type: 'lookup', label: 'Book', reference: 'fx_book' },
+      summary: { type: 'formula', label: 'Summary', expression },
+    },
+  });
+  const gateLoan = (expression: string) => runRuntimeAuthoringRules({
+    type: 'object',
+    item: loan(expression),
+    context: { objects: [book] },
+    packageScope: { packageId: 'com.example.loans', dependencies: ['com.example.library'] },
+  });
+  const LOAN_WHERE = "object 'fx_loan' · field 'summary' expression";
+
+  it('⭐ LIT — the read is REFUSED under EXPRESSION_INVALID, naming the field and the target from the other package', () => {
+    const result = gateLoan('record.book.name');
+
+    const errs = expressionFindings(result.errors);
+    expect(errs, dump(result)).toHaveLength(1);
+    expect(errs[0]).toMatchObject({ severity: 'error', where: LOAN_WHERE, path: LOAN_WHERE });
+    expect(errs[0]!.message).toContain('unknown field `name` on `fx_book`');
+  });
+
+  it('⭐ CONTROL — a read of a field the target declares publishes clean', () => {
+    const result = gateLoan('record.book.title');
+
+    expect(expressionFindings(result.errors), dump(result)).toEqual([]);
+    expect(expressionFindings(result.advisories), dump(result)).toEqual([]);
+  });
+});
+
 describe('#22019 / #22032 — every pass over the object\'s body judges on this door, and no pass over another collection', () => {
   /**
    * One body carrying a fault in EVERY pass over the object's own body — the

packages/lint/src/validate-expressions.formula-one-hop.test.ts

@@ -87,6 +87,34 @@ describe('a formula expression reads one hop through a reference (#24197)', () =
     expect(errors[0].message).toContain('`record.reader` is read BOTH through the relationship');
     expect(errors[0].message).toContain("`record.reader.?<related field>.orValue('…')`");
   });
+
+  it('[#24410] refuses a read of a field the target does not declare, naming the field and the target', () => {
+    const errors = errorsOf("record.book.name + ' · ' + record.reader.name");
+    expect(errors).toHaveLength(1);
+    expect(errors[0].message).toContain('unknown field `name` on `hop_book`');
+    expect(errors[0].message).toContain('through the reference `record.book`');
+    expect(errors[0].message).toContain("declare 'name' on 'hop_book' — not on 'hop_loan'");
+    expect(errors[0].source).toBe("record.book.name + ' · ' + record.reader.name");
+  });
+
+  it('[#24410] admits the primary key and a column the platform injects on the target', () => {
+    expect(errorsOf('record.book.id')).toEqual([]);
+    expect(errorsOf('string(record.book.created_at) + record.book.title')).toEqual([]);
+  });
+
+  it('[#24410] does not judge a target the stack does not carry', () => {
+    const stray = {
+      name: 'hop_loan',
+      label: 'Loan',
+      fields: {
+        book: { type: 'lookup', label: 'Book', reference: 'hop_elsewhere' },
+        title: { type: 'formula', label: 'Title', expression: 'record.book.name' },
+      },
+    };
+    const errors = validateStackExpressions(specValid({ objects: [book, shelf, reader, stray] }))
+      .filter((i) => i.where === WHERE && (i.severity ?? 'error') === 'error');
+    expect(errors).toEqual([]);
+  });
 });
 
 /**

packages/lint/src/validate-expressions.ts

@@ -115,7 +115,7 @@ import type { FlowNodeParsed, FlowEdgeParsed } from '@objectstack/spec/automatio
 // hand-written notion of "blank" here — that drift is what #15662 built the
 // shared refusal to prevent.
 import { EvaluatedExpressionInputSchema, EVALUATED_EXPRESSION_SOURCE_REQUIRED } from '@objectstack/spec/shared';
-import { referenceCarrierOf, REFERENCE_VALUE_TYPES } from '@objectstack/spec/data';
+import { referenceCarrierOf, referenceTargetOf, REFERENCE_VALUE_TYPES } from '@objectstack/spec/data';
 import { isStoredMetadataBodyObject } from '@objectstack/spec/kernel';
 // [#22394] The acting user's declared members — the contract half of the option
 // verdict's member allowlist for `current_user` and its ADR-0068 aliases. Read
@@ -1768,6 +1768,20 @@ function storedMetadataBodyHopMessage(
     + 'stored id, if the id is what you need.';
 }
 
+/**
+ * [#24410] The refusal of a formula's one-hop read of a field the reference's
+ * target does not declare. It opens in the shared validator's `unknown field`
+ * words, so the two first-segment and second-segment verdicts read as one
+ * family, and names the read, the target and where the field would have to be
+ * declared — never on the object carrying the formula.
+ */
+function undeclaredHopFieldMessage(objectName: string, field: string, target: string, column: string): string {
+  return `unknown field \`${column}\` on \`${target}\`: this formula reads it through the reference `
+    + `${celMemberPath('record', field)}, and '${target}' declares no field '${column}', so the formula reads `
+    + `null on every row. Read a field '${target}' declares, or declare '${column}' on '${target}' — not on `
+    + `'${objectName}', the object carrying this formula.`;
+}
+
 /**
  * [#17495] Why a structural condition whose SHAPE is fine is still not
  * authorable: its source is blank. The author-time twin of the gate #17322 put
@@ -1948,6 +1962,45 @@ export function runStackExpressionPasses(stack: AnyRec, options: StackExpression
     }
   };
 
+  /**
+   * [#24410] Refuse a formula's one-hop read of a field the reference's target
+   * does not declare (`record.book.name` where the book declares no `name`).
+   * ObjectQL's one-hop loader reads only declared columns of the target, so as
+   * written the formula reads null on every row; this says so where it is
+   * written, at the build and at the object save door alike.
+   *
+   * Judged against the target's field index — its declared fields and the
+   * system columns the platform injects on it, the set the registry carries —
+   * and the primary key, which no field map carries and the loader reads. A
+   * target outside the stack (or the door's snapshot, which carries the live
+   * registry's objects, another package's included) has no index here and is
+   * not judged; a hop into the stored-metadata-body family is refused by
+   * {@link refuseStoredMetadataBodyHop} and not judged twice.
+   */
+  const refuseUndeclaredHopField = (where: string, raw: unknown, objectName: string): void => {
+    const source = celSourceOf(raw);
+    if (!source) return;
+    const traversals = analyzeRelationshipTraversals(source, 'record')?.traversals;
+    if (!traversals || traversals.size === 0) return;
+    for (const obj of objects) {
+      if (obj.name !== objectName) continue;
+      for (const [field, def] of fieldEntries(obj)) {
+        const columns = traversals.get(field);
+        const target = columns ? referenceTargetOf(def)?.trim() : undefined;
+        if (!columns || !target || isStoredMetadataBodyObject(target) || !fieldIndex.has(target)) continue;
+        for (const column of [...columns].sort()) {
+          if (column === 'id' || fieldIndex.get(target)?.includes(column)) continue;
+          issues.push({
+            where,
+            message: undeclaredHopFieldMessage(objectName, field, target, column),
+            source,
+            severity: 'error',
+          });
+        }
+      }
+    }
+  };
+
   // [#8116] object name → the injected anchors registered with NO storage
   // behind them (non-empty only for ADR-0015 `external` objects). The other
   // half of {@link buildFieldIndex}'s #5378 widening: the anchors ARE
@@ -2848,6 +2901,8 @@ export function runStackExpressionPasses(stack: AnyRec, options: StackExpression
         for (const e of res.errors) issues.push({ where: fieldWhere, message: e.message, source: e.source, severity: 'error' });
         for (const w of res.warnings) issues.push({ where: fieldWhere, message: w.message, source: w.source, severity: 'warning' });
         if (objectName) refuseStoredMetadataBodyHop(fieldWhere, f.expression, objectName, 'formula');
+        // [#24410] And a hop to a field the target does not declare.
+        if (objectName) refuseUndeclaredHopField(fieldWhere, f.expression, objectName);
         // [#8116] Formulas read the same record binding the predicates do, so
         // a `record.<anchor>` read inside one degrades identically.
         warnUnprovisionedAnchors(fieldWhere, f.expression, objectName);

packages/objectql/src/engine-formula-one-hop.test.ts

@@ -289,6 +289,24 @@ describe('#24197 — a formula field reads one hop through a reference', () => {
       expect(row?.title).toBeNull();
       expect(faultsOn('lib_loan', 'title')[0].message).toContain("'lib_book' declares no such field");
     });
+
+    it('[#24410] an undeclared related field nulls only its own formula: a sibling through the same reference is served as it is alone', async () => {
+      registerLoan(TITLE, { bad: { type: 'formula', expression: 'record.book.isbn' } });
+      const alone = await engine.find('lib_loan', { fields: ['title'], limit: 4, context: READER });
+      d.calls.length = 0;
+      const rows = await engine.find('lib_loan', { limit: 4, context: READER });
+      expect(rows.map((r) => r.title)).toEqual(alone.map((r) => r.title));
+      expect(rows.map((r) => r.title)).toEqual(['Dune · Ann', 'Emma · Ann', 'Ulysses · Bo', 'Dune · Bo']);
+      expect(rows.map((r) => r.bad)).toEqual([null, null, null, null]);
+      // The undeclared column never enters the shared read; ONE book read still serves the page.
+      expect(relatedReads('lib_book')).toBe(1);
+      expect([...d.calls.find((c) => c.object === 'lib_book')!.ast.fields].sort()).toEqual(['id', 'title']);
+      // Each formula's own report: the bad one names its column, the sibling reports nothing.
+      expect(faultsOn('lib_loan', 'title')).toEqual([]);
+      const fault = faultsOn('lib_loan', 'bad');
+      expect(fault).toHaveLength(1);
+      expect(fault[0].message).toContain("the formula reads 'isbn' through `book` (object 'lib_book')");
+    });
   });
 
   describe('one hop is the depth', () => {

packages/objectql/src/engine.ts

@@ -2651,8 +2651,36 @@ type FormulaFaultSink = (field: string, error: EvalError) => void;
  * ({@link ObjectQL.resolveFormulaRelated}), or — when this read serves no hop
  * at all — the sentence saying why, which every traversing formula of the read
  * then reports.
+ *
+ * [#24410] `undeclared` is decided per FORMULA, never per reference: the
+ * related fields a reference's target does not declare, which the shared read
+ * never asks for. A formula naming one of them reads `null` with the
+ * `undeclared-field` reason; a sibling reading only declared fields through the
+ * same reference is served from the shared read exactly as it is alone.
+ */
+type FormulaRelated = {
+  readonly bindings: (row: Record<string, unknown>) => RelatedRecordBinding | undefined;
+  readonly undeclared: ReadonlyMap<string, { readonly object: string; readonly fields: ReadonlySet<string> }>;
+} | string;
+
+/**
+ * [#24410] The named related fields a reference's target does not DECLARE —
+ * the one reading both seams that serve a hop make of it: the shared loader
+ * ({@link ObjectQL.resolveOneHopRelated}) and the formula read, which judges
+ * them per formula ({@link ObjectQL.resolveFormulaRelated}).
+ *
+ * [#8215] The PRIMARY KEY counts as declared, though no field map carries it:
+ * `record.<fk>.id` is the repair the traversal conflict refusal prescribes. An
+ * unknown target (`declared` absent) declares nothing to judge against, so
+ * nothing is undeclared.
  */
-type FormulaRelated = ((row: Record<string, unknown>) => RelatedRecordBinding | undefined) | string;
+function undeclaredRelatedFields(
+  declared: Record<string, unknown> | undefined,
+  named: Iterable<string>,
+): string[] {
+  if (!declared) return [];
+  return [...named].filter((n) => n !== SystemFieldName.ID && !(n in declared));
+}
 
 /** [#24197] Why a hop was not served where no reader supplied one. */
 const FORMULA_HOP_NOT_SERVED =
@@ -2687,8 +2715,22 @@ function formulaHopScope(
   hops: ReadonlyMap<string, ReadonlySet<string>>,
   related: FormulaRelated | undefined,
 ): { ok: true; record: Record<string, unknown>; empty: string[] } | { ok: false; reason: string } {
-  if (typeof related !== 'function') return { ok: false, reason: related ?? FORMULA_HOP_NOT_SERVED };
-  const binding = related(rec);
+  if (related === undefined || typeof related === 'string') {
+    return { ok: false, reason: related ?? FORMULA_HOP_NOT_SERVED };
+  }
+  // [#24410] This formula's own undeclared reads first, before any row: they
+  // are a fault of the expression, whatever the record holds.
+  for (const [field, named] of hops) {
+    const missing = related.undeclared.get(field);
+    const own = missing ? [...named].filter((n) => missing.fields.has(n)) : [];
+    if (missing && own.length > 0) {
+      return {
+        ok: false,
+        reason: formulaHopRefusal(field, { object: missing.object, unavailable: 'undeclared-field', undeclaredFields: own }, named),
+      };
+    }
+  }
+  const binding = related.bindings(rec);
   let copy: Record<string, unknown> | undefined;
   const empty: string[] = [];
   for (const [field, named] of hops) {
@@ -6264,8 +6306,30 @@ export class ObjectQL implements IObjectQLEngine {
         + 'formula resolves ONE hop: a formula read this way does not read through a reference of its own '
         + '(a second hop). Store the value on this object as a field of its own to read it from there';
     }
-    const lookup = await this.resolveOneHopRelated(schema, wanted, rows, context, 'caller');
-    return (row) => lookup(row);
+    // [#24410] The read set per reference is the UNION of every formula's
+    // columns, and the loader answers a whole reference `undeclared-field` when
+    // any column of it is undeclared — so one formula's misspelt column nulled
+    // every sibling reading through the same reference. The undeclared columns
+    // are taken out of the shared read here and judged per formula
+    // ({@link formulaHopScope}); the loader reads the declared rest.
+    const fields = (schema?.fields ?? {}) as Record<string, unknown>;
+    const undeclared = new Map<string, { object: string; fields: ReadonlySet<string> }>();
+    for (const [fk, named] of wanted) {
+      const target = referenceTargetOf(fields[fk]);
+      // The loader's own verdicts stand for these: no target, or a hop into
+      // the stored-metadata-body family, which is never read.
+      if (!target || isStoredMetadataBodyObject(target)) continue;
+      const declared = (this._registry.getObject(target) as { fields?: Record<string, unknown> } | undefined)?.fields;
+      const missing = undeclaredRelatedFields(declared, named);
+      if (missing.length === 0) continue;
+      undeclared.set(fk, { object: target, fields: new Set(missing) });
+      for (const name of missing) named.delete(name);
+      if (named.size === 0) wanted.delete(fk);
+    }
+    const lookup = wanted.size > 0
+      ? await this.resolveOneHopRelated(schema, wanted, rows, context, 'caller')
+      : () => undefined;
+    return { bindings: (row) => lookup(row), undeclared };
   }
 
   /**
@@ -10279,9 +10343,7 @@ export class ObjectQL implements IObjectQLEngine {
       //
       // ⛔ This widens the READ SET by nothing: `id` is already unconditionally
       // in the projection below, because the by-id map is keyed on it.
-      const declaredHere = (n: string): boolean =>
-        n === SystemFieldName.ID || !!declared && n in declared;
-      const undeclared = declared ? named.filter((n) => !declaredHere(n)) : [];
+      const undeclared = undeclaredRelatedFields(declared, named);
       if (undeclared.length > 0) {
         resolved.set(fk, { object: target, byId: new Map(), undeclared });
         continue;