fix(objectql): a cleared optional single-value select or lookup stores null, so a second unique clear is no longer refused 409 (#29277)

.changeset/29176-optional-unique-blank-null.md

@@ -1,3 +1,3 @@
 ---
 '@objectstack/objectql': patch
 ---
@@ -10,6 +10,6 @@
 A cleared text input is sent as `''`, and the write path stored it as sent. Every unique constraint the drivers build is NULL-distinct, so `null` never collides, but `''` is a value: the first record whose SKU (tax id, employee number, …) was cleared saved, and the second was refused "A record with this value already exists" although neither held a value. Measured on SQLite and the memory driver: two `''` rows collide, two `null` rows do not.
 
 - **Now.** The engine's write door, which `insert`, `insertMany`, `update` (by id and by predicate) and the dry run (`validate`) apply before anything else reads the payload, stores exactly `''` as `null` on a column that is text-like (`text`, `textarea`, `email`, `url`, `phone`, `markdown`, `html`, `richtext`, `code`, `color`, `signature`, `qrcode`), not `required`, and covered by a unique constraint: the field's own `unique`, or membership in a declared `indexes[]` entry carrying `unique`. Every REST, batch, upsert and import door reaches the engine through those methods.
-- **Unchanged.** A `required` field given `''` is refused `required` exactly as before, also when it declares a `defaultValue`. A real duplicate is still `409 UNIQUE_VIOLATION`. Whitespace is a value and is stored as sent. `password` and `secret` keep their `''` refusal. A column no unique constraint covers, and `select` / `lookup` columns, keep storing `''`.
+- **Unchanged.** A `required` field given `''` is refused `required` exactly as before, also when it declares a `defaultValue`. A real duplicate is still `409 UNIQUE_VIOLATION`. Whitespace is a value and is stored as sent. `password` and `secret` keep their `''` refusal. A text column no unique constraint covers keeps storing `''`. (Optional single-value `select` / `lookup` columns store a cleared `''` as `null` too, unique or not: a separate change in this release.)
 - **Defaults.** An optional unique field that declares a `defaultValue` now takes it on insert when sent `''`, exactly as when sent `null`.
 - **Rows already stored.** Existing `''` values are not migrated. They keep matching a `field = ''` filter; a newly cleared row matches `$null: true` instead. The `is_empty` filter (`$empty`) matches both, as before. A newly cleared row no longer collides with a stored `''`.

.changeset/29277-single-value-blank-null.md

@@ -0,0 +1,20 @@
+---
+'@objectstack/objectql': patch
+'@objectstack/core': patch
+'@objectstack/metadata-protocol': patch
+---
+
+fix(objectql): a cleared optional single-value `select` or `lookup` stores `null`, so a second record cleared on a unique one is no longer refused `409 UNIQUE_VIOLATION`
+
+Clause-②: yes (widening)
+Metadata-protocol: none
+
+A cleared `select`, `lookup` or single file field sent as `''` was stored as sent. `''` is neither an option code, a record id nor a file id, so it held no value, yet it was one to the store and to the `is_empty` filter: on a unique column the second record cleared was refused "A record with this value already exists", and on every column the cleared record did not match `is_empty` (`$empty`), whose ruled table reads only `null` as empty on these types.
+
+- **Now.** The engine's write door (`insert`, `insertMany`, `update` by id and by predicate, and the `validate` dry run; every REST, batch, upsert and import door reaches it) stores exactly `''` as `null` on every optional column the `$empty` table puts on its `null_only` row whose value is one string: the option types (`select`, `radio`), the reference types (`lookup`, `master_detail`, `user`, `tree`) and the file references (`image`, `file`, `avatar`, `video`, `audio`). Unique or not. Object- and array-valued types (`json`, `composite`, `location`, …) and the engine-owned `formula` / `autonumber` keep what they store.
+- **Read shape.** Such a field cleared from now on reads back `null`, not `''`. It matches `is_empty` / `$empty: true`, `$null: true` and `field = null`, and no longer matches `field = ''`.
+- **Unchanged.** The `$empty` table itself. A `required` field given `''` is refused `required` as before, also when it declares a `defaultValue`. Whitespace is a value. A `multiple: true` option or reference column keeps what it stores. A real duplicate is still `409 UNIQUE_VIOLATION`.
+- **Defaults.** An optional field that declares a `defaultValue` now takes it on insert when sent `''`, exactly as when sent `null`.
+- **Rows already stored.** Existing `''` values are not migrated by this change; they keep answering as a value (`$empty: false`, `field = ''`) until a data step cleans them.
+- **Seed replay.** A seed that writes `''` on such a column, or on any column whose blank the door already stored as `null` (an optional unique text column, a number / boolean / date column), is compared in that stored form, so replaying it is no longer a change on every boot. Before, the replay compared the seed's `''` with the stored `null` and rewrote the row each time (bumping `updated_at`).
+- **New export.** `@objectstack/core` exports the rule itself, `blankAsNullColumns` and `blankStorageForm` (with `ENGINE_LIFECYCLE_COLUMNS`), which the write door and the seed loader both read.

packages/core/src/index.ts

@@ -169,6 +169,14 @@ export * from './utils/multi-value-storage-form.js';
 // engine at runtime.
 export * from './utils/required-emptiness.js';
 
+// [#29277] …and the blank-as-null rule: which columns of an object store a
+// blank string as `null`, and which blanks (the #20308, #29176 and #29277
+// arms). `@objectstack/objectql`'s write door applies it, and
+// `@objectstack/metadata-protocol`'s seed replay compares a seed value in the
+// form it stores — one rule, here because those two do not depend on each
+// other.
+export * from './utils/blank-storage-form.js';
+
 // [#21007] …and the refusal a scalar comparison gets on a field stored as a
 // JSON column: the operator set and the words. `driver-sql` refuses it on
 // `where`, and `@objectstack/objectql` on the per-aggregation `filter` it

packages/core/src/utils/blank-storage-form.test.ts

@@ -0,0 +1,76 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+import { describe, it, expect } from 'vitest';
+import { blankAsNullColumns, blankStorageForm, ENGINE_LIFECYCLE_COLUMNS } from './blank-storage-form.js';
+
+const schema = {
+  fields: {
+    qty: { type: 'number', required: true },
+    due: { type: 'date' },
+    sku: { type: 'text', unique: true },
+    code: { type: 'text' },
+    idx_a: { type: 'text' },
+    pw: { type: 'password', unique: true },
+    stage: { type: 'select', options: [{ value: 'a' }] },
+    stage_req: { type: 'select', required: true, options: [{ value: 'a' }] },
+    tags: { type: 'select', multiple: true, options: [{ value: 'a' }] },
+    ref: { type: 'lookup' },
+    pic: { type: 'image' },
+    blob: { type: 'json' },
+    num: { type: 'autonumber' },
+    created_by: { type: 'user' },
+    created_at: { type: 'datetime' },
+  },
+  indexes: [{ fields: ['idx_a'], unique: true }, { fields: ['code'] }],
+};
+
+describe('blankAsNullColumns: the three arms of the write door\'s blank-as-null rule', () => {
+  const cols = blankAsNullColumns(schema);
+
+  it('[#20308] a typed column takes any blank, required and lifecycle included', () => {
+    expect(cols.get('qty')).toBe('blank');
+    expect(cols.get('due')).toBe('blank');
+    expect(cols.get('created_at')).toBe('blank');
+  });
+
+  it('[#29176] exactly "" on an optional text column a unique constraint covers', () => {
+    expect(cols.get('sku')).toBe('empty');
+    expect(cols.get('idx_a')).toBe('empty');
+    expect(cols.has('code')).toBe(false);
+    expect(cols.has('pw')).toBe(false);
+  });
+
+  it('[#29277] exactly "" on an optional one-string null_only column, unique or not', () => {
+    expect(cols.get('stage')).toBe('empty');
+    expect(cols.get('ref')).toBe('empty');
+    expect(cols.get('pic')).toBe('empty');
+    for (const out of ['stage_req', 'tags', 'blob', 'num', 'created_by']) expect(cols.has(out), out).toBe(false);
+  });
+
+  it('lifecycle names are the five engine-owned columns', () => {
+    expect([...ENGINE_LIFECYCLE_COLUMNS]).toEqual(['id', 'created_at', 'created_by', 'updated_at', 'updated_by']);
+  });
+
+  it('an unresolvable definition declares none', () => {
+    expect(blankAsNullColumns(undefined).size).toBe(0);
+    expect(blankAsNullColumns({}).size).toBe(0);
+  });
+});
+
+describe('blankStorageForm', () => {
+  it('"blank" nulls "" and whitespace; "empty" nulls only ""; no reach keeps everything', () => {
+    expect(blankStorageForm('', 'blank')).toBeNull();
+    expect(blankStorageForm('  ', 'blank')).toBeNull();
+    expect(blankStorageForm('', 'empty')).toBeNull();
+    expect(blankStorageForm('  ', 'empty')).toBe('  ');
+    expect(blankStorageForm('', undefined)).toBe('');
+  });
+
+  it('a non-blank value or a non-string comes back as it is', () => {
+    const obj = { a: 1 };
+    for (const v of ['a', 0, false, null, undefined, obj]) {
+      expect(blankStorageForm(v, 'blank')).toBe(v);
+      expect(blankStorageForm(v, 'empty')).toBe(v);
+    }
+  });
+});

packages/core/src/utils/blank-storage-form.ts

@@ -0,0 +1,174 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * THE blank-as-null rule — which columns of an object store a blank string as
+ * `null`, and which blanks — read by the write door that stores it and by the
+ * seed loader's replay, which must compare a seed value in the form the door
+ * stores it.
+ *
+ * ## Two faces, one rule
+ *
+ * - `@objectstack/objectql`'s write door (`normalizeBlankTypedValues`) applies
+ *   it to every insert and update payload before anything else reads it.
+ * - `@objectstack/metadata-protocol`'s seed loader asks it in `isNoOpReplay`
+ *   (the `blankAsNull` replay comparand): a seed writing `''` on such a column
+ *   lands as `null`, and compared as written it read the row as changed and
+ *   rewrote it on every boot (#29277; the #20308 and #29176 arms churned the
+ *   same way before it).
+ *
+ * The two packages do not depend on each other, and a copy each is how a
+ * replay comes to disagree with the store. So the rule lives here, beside
+ * `multiValueStorageForm` and `temporalStorageForm`, which are shared for the
+ * same reason. The type classes and the `$empty` table it reads are the
+ * spec's, never re-listed.
+ *
+ * ## The rule — three arms
+ *
+ * - [#20308] `'blank'`: a column whose type is non-string-typed
+ *   (`NON_TEXT_STORED_VALUE_TYPES`: numeric, boolean, `date`, `datetime`,
+ *   `time`) stores any blank (`isMissing`: `''` or whitespace) as `null` —
+ *   required or not, lifecycle or not: a blank is never a value of its type.
+ * - [#29176] `'empty'`: exactly `''` on an optional text-like column
+ *   (`STRING_VALUE_TYPES` minus `password` / `secret`) a unique constraint
+ *   covers — its own `unique`, or a unique `indexes[]` entry, both read through
+ *   `isUniqueDeclared`, the drivers' own predicate.
+ * - [#29277] `'empty'`: exactly `''` on an optional column on the `$empty`
+ *   table's `null_only` row (`expandEmptyOperator`) whose value is ONE string
+ *   ({@link ONE_STRING_STORED_TYPES}), unique or not (maintainer ruling B on
+ *   #29346, record 6579813296).
+ *
+ * The two `'empty'` arms never reach a `required: true` column (there `''` is
+ * refused `required`, and must not first become a `null` a `defaultValue`
+ * would fill) nor a lifecycle column the engine owns
+ * ({@link ENGINE_LIFECYCLE_COLUMNS}), and never whitespace (a value on both
+ * rows of the `$empty` table they touch).
+ */
+
+import {
+  NON_TEXT_STORED_VALUE_TYPES,
+  STRING_VALUE_TYPES,
+  SINGLE_OPTION_TYPES,
+  REFERENCE_VALUE_TYPES,
+  FILE_REFERENCE_TYPES,
+  expandEmptyOperator,
+  isUniqueDeclared,
+  type ValueShapeFieldDef,
+} from '@objectstack/spec/data';
+import { isMissing } from './required-emptiness.js';
+
+/**
+ * Lifecycle columns the engine always owns and the client never supplies,
+ * skipped by NAME because they are not author-declared business fields. The
+ * write door's validator skips the same names (its `SKIP_FIELDS` is this list).
+ */
+export const ENGINE_LIFECYCLE_COLUMNS: readonly string[] = Object.freeze([
+  'id', 'created_at', 'created_by', 'updated_at', 'updated_by',
+]);
+const LIFECYCLE: ReadonlySet<string> = new Set(ENGINE_LIFECYCLE_COLUMNS);
+
+/**
+ * [#29176] The text-like types of the unique arm: the spec's
+ * `STRING_VALUE_TYPES` — exactly the "text-like" row of the `$empty` table,
+ * where `''` and `null` already answer as one — minus the two credential
+ * types. ⛔ `password` and `secret` stay out: a maintainer ruling refuses `''`
+ * on them (`EmptyCredentialWriteError`) while `null` means "clear", so
+ * rewriting the blank would turn that refusal into a silent clear.
+ */
+const UNIQUE_BLANK_TEXT_TYPES: ReadonlySet<string> = new Set(
+  [...STRING_VALUE_TYPES].filter((t) => t !== 'password' && t !== 'secret'),
+);
+
+/**
+ * [#29277] The storage-shape half of the `null_only` arm: the spec's value
+ * classes whose single value IS one string — an option code
+ * (`SINGLE_OPTION_TYPES`), a record id (`REFERENCE_VALUE_TYPES`), an opaque
+ * `sys_file` id (`FILE_REFERENCE_TYPES`; SQL stores it JSON-encoded, still one
+ * string to every reader). Read as the spec's constants, never re-listed. The
+ * spec has no predicate for "stores one string", so this set is stated here,
+ * and `record-validator.single-value-blank.test.ts` (`@objectstack/objectql`)
+ * ties it to the table: every type on the `null_only` row that is not typed
+ * must be in this set or in one of the two classes left out below, so a type
+ * that joins the row unclassified turns that pin red.
+ *
+ * Left out, each measured on SQLite through the engine:
+ *
+ *  - `STRUCTURED_JSON_TYPES` (`json`, `composite`, `record`, `location`,
+ *    `address`, `repeater`, `vector`): the value is an object or an array, so
+ *    `''` is not a blank of its shape. What empty means for them is a
+ *    separate question.
+ *  - `COMPUTED_VALUE_TYPES` (`formula`, `autonumber`; `summary` is typed): the
+ *    engine owns the value — a formula is never stored, an autonumber is
+ *    issued on insert whatever the caller sent, `''` included.
+ */
+const ONE_STRING_STORED_TYPES: ReadonlySet<string> = new Set([
+  ...SINGLE_OPTION_TYPES,
+  ...REFERENCE_VALUE_TYPES,
+  ...FILE_REFERENCE_TYPES,
+]);
+
+/**
+ * Which blanks a column stores as `null`: `'blank'` — any blank string
+ * (`isMissing`); `'empty'` — exactly `''`.
+ */
+export type BlankAsNullReach = 'blank' | 'empty';
+
+/** The field shape the rule reads. */
+export interface BlankAsNullFieldDef extends ValueShapeFieldDef {
+  required?: boolean;
+  unique?: unknown;
+}
+
+/**
+ * The columns of one object whose blank the write door stores as `null`, each
+ * with its reach. A column not in the map keeps every blank as written.
+ */
+export function blankAsNullColumns(
+  objectSchema: { fields?: Record<string, BlankAsNullFieldDef | undefined>; indexes?: unknown } | undefined | null,
+): ReadonlyMap<string, BlankAsNullReach> {
+  const out = new Map<string, BlankAsNullReach>();
+  const fields = objectSchema?.fields;
+  if (!fields || typeof fields !== 'object') return out;
+  let covered: Set<string> | undefined;
+  const isCovered = (name: string) => (covered ??= uniqueCoveredColumns(fields, objectSchema?.indexes)).has(name);
+  for (const [name, def] of Object.entries(fields)) {
+    if (!def || typeof def.type !== 'string') continue;
+    if (NON_TEXT_STORED_VALUE_TYPES.has(def.type)) {
+      out.set(name, 'blank');
+      continue;
+    }
+    if (LIFECYCLE.has(name) || def.required === true) continue;
+    const empty = UNIQUE_BLANK_TEXT_TYPES.has(def.type)
+      ? isCovered(name)
+      : expandEmptyOperator(def).arm === 'null_only' && ONE_STRING_STORED_TYPES.has(def.type);
+    if (empty) out.set(name, 'empty');
+  }
+  return out;
+}
+
+/**
+ * The form a value written to a column of the given reach is stored in:
+ * `null` when the value is a blank that reach takes, else the value itself
+ * (the same reference). `reach` `undefined` — a column not in
+ * {@link blankAsNullColumns} — keeps every value.
+ */
+export function blankStorageForm(value: unknown, reach: BlankAsNullReach | undefined): unknown {
+  if (reach === undefined || typeof value !== 'string') return value;
+  if (reach === 'empty') return value === '' ? null : value;
+  return isMissing(value) ? null : value;
+}
+
+function uniqueCoveredColumns(fields: Record<string, BlankAsNullFieldDef | undefined>, indexes: unknown): Set<string> {
+  const covered = new Set<string>();
+  for (const [name, def] of Object.entries(fields)) {
+    if (def && isUniqueDeclared(def.unique)) covered.add(name);
+  }
+  if (Array.isArray(indexes)) {
+    for (const idx of indexes) {
+      if (!idx || typeof idx !== 'object' || Array.isArray(idx)) continue;
+      const { unique, fields: members } = idx as { unique?: unknown; fields?: unknown };
+      if (!isUniqueDeclared(unique) || !Array.isArray(members)) continue;
+      for (const name of members) if (typeof name === 'string') covered.add(name);
+    }
+  }
+  return covered;
+}