Merge remote-tracking branch 'origin/main' into claude/issue-21009-json-column-text-operators

Author: claude

.changeset/21059-layered-code-null-unshipped.md

@@ -0,0 +1,13 @@
+---
+'@objectstack/metadata-protocol': patch
+---
+
+fix(metadata-protocol): the layered read's code layer is empty for an item no package ships, whether or not a stored copy of it has been loaded into the registry (#21059)
+
+Clause-②: no
+
+The layered read, `GET /api/v1/meta/:type/:name/layers`, reports an item's code layer as the packaged definition, and as empty when no package ships the item and it exists only as a stored customization. For an item no package ships, it answered that correctly only until the stored copy had been loaded into the in-memory registry, for example by a restart's startup load or by a list read. After that, the code layer answered the stored copy, and the lock and provenance flags were derived from it. So the same read of the same item gave two answers, depending on what had been loaded.
+
+The code layer now skips a stored copy the registry holds, using the tenant-authorship mark the startup load already puts on every stored copy it registers. An item no package ships has an empty code layer before and after the load, and its flags come from the stored layer both times. This includes a stored copy whose own content claims a package's provenance: a claim is not a packaged definition. The deprecated layers flag on the by-name read, `GET /api/v1/meta/:type/:name`, answers the same.
+
+Packaged items keep their packaged code layer. An item registered at runtime with no package keeps its code layer, as before. The stored rows are not changed.

.changeset/21096-liveness-hint-never-note.md

@@ -0,0 +1,7 @@
+---
+'@objectstack/lint': patch
+---
+
+fix(lint): a liveness finding's fix text is the row's `authorHint`, else the verdict's default hint, and never the row's internal ledger `note`, for every row class. Before this, a row that opted in with `authorWarn`, and an `experimental` row, with no `authorHint` printed its `note` as the fix text: on `app-showcase`, the two `liveness-planned-property` findings for `externalSharingModel` printed a 728-character maintainer note that cites a tracker id. They now print "Keep it — a consumer is being built against this property; it has no runtime effect yet." No rule id, message, severity or exit code changes, and a row that has an `authorHint` prints it exactly as before (#21096)
+
+Clause-②: no

.changeset/21113-turso-remote-autonumber.md

@@ -0,0 +1,42 @@
+---
+'@objectstack/driver-turso': minor
+'@objectstack/driver-sql': patch
+---
+
+feat(driver-turso): the remote transport issues `auto_number` values (#21113)
+
+Clause-②: yes (widening)
+
+A `create()`, `bulkCreate()` or `upsert()` on the Turso REMOTE transport that
+leaves an `autonumber` field empty (`undefined`, `null` or `''`) now gets a
+generated value. It used to be refused with `NOT_IMPLEMENTED` / 501, so on a
+hosted tenant database — which is on this transport — no object declaring an
+`auto_number` field could get a new record at all. Nothing is declared anew in
+the spec or in the package exports; `supports.autonumber` stays `true` and is
+now honoured.
+
+- The value comes from the same persistent `_objectstack_sequences` counter the
+  local and embedded-replica transports use, rendered by the same format rules
+  (`autonumberFormat` / `format`, organization scope, date and `{field}`
+  tokens), bootstrapped from the table's highest existing value by the same
+  reading, and re-seeded the same way after rows land above the counter by a
+  seed replay or import. A remote driver and an embedded replica of one
+  database draw from one counter row.
+- The counter moves in one statement over the connection (`UPDATE … RETURNING`,
+  or on a cold counter `INSERT … ON CONFLICT (key_hash) DO UPDATE … RETURNING`),
+  so writers in different processes never draw the same number.
+- An `upsert()` that merges into an existing row keeps the number already in
+  the row; a row that carries its own number is written unchanged.
+- A `_objectstack_sequences` table in the pre-`key_hash` shape is refused in
+  remote mode with `DATABASE_ERROR` / 500 and the remedy in the message (open
+  the database once through the local or embedded-replica transport, which
+  migrates it); remote mode does not migrate it and does not key by the legacy
+  rule.
+- `RemoteTransport.upsert()` takes an optional fifth argument naming columns
+  that are written on insert and left alone on merge.
+
+`@objectstack/driver-sql`: the sequence rules a second transport shares are
+now `protected` members of `SqlDriver` (`resolveSequenceTenantId`,
+`defineSequencesTable`, `maxAutonumberCounter`, `escapeLikePrefix`,
+`sequencesTableName`, `autoNumberCollisionRetries`). No export is added and no
+behaviour changes on any dialect.

.changeset/21147-s3-presign-no-checksum.md

@@ -0,0 +1,31 @@
+---
+'@objectstack/service-storage': patch
+---
+
+fix(service-storage): presigned S3 uploads no longer bake the CRC32 of an empty body into the signed URL (#21147)
+
+Clause-②: no
+
+`S3StorageAdapter.getPresignedUpload()` handed the browser a URL carrying
+`x-amz-sdk-checksum-algorithm=CRC32&x-amz-checksum-crc32=AAAAAA==`. `AAAAAA==`
+is the CRC32 of an empty body: the AWS SDK's default (`WHEN_SUPPORTED`) stamps a
+checksum into every `PutObjectCommand` it prepares, and a presign prepares the
+command before any byte exists. A store that enforces a query-signed checksum
+against the uploaded body refuses every presigned browser upload with a checksum
+mismatch, while server-side `upload()` keeps working. Hosted R2 does not enforce
+it, which is why this stayed quiet; self-hosted deployments on other
+S3-compatible stores may.
+
+The adapter's `S3Client` now sets `requestChecksumCalculation` and
+`responseChecksumValidation` to `WHEN_REQUIRED`. None of the commands the adapter
+issues is checksum-required, so the presigned PUT URL carries no `x-amz-checksum-*`
+or `x-amz-sdk-checksum-algorithm` parameter, and a presigned GET URL no longer
+carries `x-amz-checksum-mode=ENABLED`.
+
+Two server-side effects of the same setting, measured against a loopback server:
+`upload()` and multipart `uploadChunk()` no longer send the client-computed
+`x-amz-checksum-crc32` header (the SigV4 `x-amz-content-sha256` payload hash is
+unchanged), and `download()` no longer asks the store for, or validates, a
+response checksum. Both were added only by the SDK's default change; the older
+behaviour is restored. SDKs older than the flexible-checksum release ignore the
+two options.

packages/drivers/driver-sql/src/sql-driver.ts

@@ -7347,6 +7347,56 @@ export class SqlDriver implements IDataDriver {
       .digest('hex');
   }
 
+  // ── The sequence semantics a second face shares, stated once ──────────────
+  //
+  // `TursoDriver`'s REMOTE face issues record numbers over `@libsql/client`
+  // rather than through this class's Knex transaction, and an embedded-replica
+  // face and a remote face can point at ONE database. Two faces computing a
+  // different counter key, a different tenant bucket or a different data-table
+  // bootstrap would hand out colliding numbers on that database, so each rule
+  // below is a `protected` member that face CALLS — never a copy it keeps. The
+  // members are small on purpose: they are the whole of what decides which
+  // counter a value is drawn from and where a cold counter starts.
+
+  /** The sequence-counter table's name, for a face that spells its own statements against it. */
+  protected get sequencesTableName(): string {
+    return SEQUENCES_TABLE;
+  }
+
+  /** How many times a write re-seeds and retries on a provable autonumber collision (#5495). */
+  protected get autoNumberCollisionRetries(): number {
+    return AUTONUMBER_COLLISION_RETRIES;
+  }
+
+  /**
+   * The tenant bucket a counter row is keyed by: the row's tenant when the
+   * object HAS a tenant column and the row carries one, else the platform
+   * bucket `GLOBAL_TENANT`. One spelling for the issue path, the re-seed path
+   * and the counter identity, which used to carry it three times.
+   */
+  protected resolveSequenceTenantId(tenantField: string | null, tenantId: string | null): string {
+    return tenantField && tenantId ? String(tenantId) : GLOBAL_TENANT;
+  }
+
+  /**
+   * The column definitions of the `key_hash`-keyed sequences table — the ONE
+   * place the shape is spelled. {@link createSequencesTable} runs it through a
+   * live connection; a face with no Knex connection compiles the same builder
+   * to text (`knex.schema.createTable(name, t => this.defineSequencesTable(t)).toSQL()`)
+   * and sends that, so both faces create byte-for-byte the same table.
+   */
+  protected defineSequencesTable(t: Knex.CreateTableBuilder): void {
+    t.string('key_hash', 64).notNullable().primary();
+    t.string('object').notNullable();
+    t.string('tenant_id').notNullable();
+    t.string('field').notNullable();
+    // Non-indexed, so it is free of the PK length limit — a long `{plan_no}`
+    // composite scope fits. 1024 is far above any realistic rendered prefix.
+    t.string('scope', 1024).notNullable().defaultTo('');
+    t.bigInteger('last_value').notNullable().defaultTo(0);
+    t.timestamp('updated_at').defaultTo(this.knex.fn.now());
+  }
+
   /**
    * Create the current `key_hash`-keyed sequences table shape. `runner` is the
    * connection the DDL runs on (a fresh pooled connection by default, or the
@@ -7356,17 +7406,7 @@ export class SqlDriver implements IDataDriver {
     table: string,
     runner: Knex | Knex.Transaction = this.knex,
   ): Promise<void> {
-    await runner.schema.createTable(table, (t) => {
-      t.string('key_hash', 64).notNullable().primary();
-      t.string('object').notNullable();
-      t.string('tenant_id').notNullable();
-      t.string('field').notNullable();
-      // Non-indexed, so it is free of the PK length limit — a long `{plan_no}`
-      // composite scope fits. 1024 is far above any realistic rendered prefix.
-      t.string('scope', 1024).notNullable().defaultTo('');
-      t.bigInteger('last_value').notNullable().defaultTo(0);
-      t.timestamp('updated_at').defaultTo(this.knex.fn.now());
-    });
+    await runner.schema.createTable(table, (t) => this.defineSequencesTable(t));
   }
 
   /**
@@ -7497,16 +7537,34 @@ export class SqlDriver implements IDataDriver {
     tenantId: string | null,
     suffix = '',
   ): Promise<number> {
-    const escapedPrefix = prefix.replace(/([\\%_])/g, '\\$1');
-    let builder = queryRunner(tableName).select(field).where(field, 'like', `${escapedPrefix}%`).whereNotNull(field);
+    let builder = queryRunner(tableName).select(field).where(field, 'like', `${this.escapeLikePrefix(prefix)}%`).whereNotNull(field);
     if (tenantField && tenantId !== null) {
       builder = builder.where(tenantField, tenantId);
     }
     const rows = await builder;
+    return this.maxAutonumberCounter((rows as any[]).map((r) => (r as any)[field]), prefix, suffix);
+  }
+
+  /**
+   * The rendered prefix as a `LIKE` anchor: `\`, `%` and `_` escaped with a
+   * backslash, so a prefix is matched literally. The predicate's pre-filter
+   * only — {@link maxAutonumberCounter} re-checks the prefix per row.
+   */
+  protected escapeLikePrefix(prefix: string): string {
+    return prefix.replace(/([\\%_])/g, '\\$1');
+  }
+
+  /**
+   * The bootstrap reading itself — the highest counter among the stored values
+   * of one counter's partition, under the anchored / unanchored rules the
+   * {@link scanMaxNumericTail} docblock states. Separated from the statement
+   * that fetches the values so a face that fetches them through another
+   * transport reads them by this one rule rather than by a copy of it.
+   */
+  protected maxAutonumberCounter(values: Iterable<unknown>, prefix: string, suffix: string): number {
     let maxN = 0;
     const anchored = prefix !== '' || suffix !== '';
-    for (const r of rows as any[]) {
-      const v: string = (r as any)[field];
+    for (const v of values) {
       if (typeof v !== 'string') continue;
       let n: number;
       if (anchored) {
@@ -7647,7 +7705,7 @@ export class SqlDriver implements IDataDriver {
     // locking on a second one (SQLite pool max=1). `initObjects` normally warms
     // this up front, making the call a no-op — this only bites the lazy path.
     await this.ensureSequencesTable(parentTrx);
-    const resolvedTenantId = tenantField && tenantId ? String(tenantId) : GLOBAL_TENANT;
+    const resolvedTenantId = this.resolveSequenceTenantId(tenantField, tenantId);
     if (scope !== '' && !this.sequencesHasKeyHash) {
       // The legacy sequences table could not be migrated to the key_hash shape,
       // so it cannot represent per-scope counters. Fail with a clear, actionable
@@ -7907,8 +7965,7 @@ export class SqlDriver implements IDataDriver {
    */
   protected async resyncSequenceToDataMax(reservation: AutoNumberReservation): Promise<void> {
     await this.ensureSequencesTable();
-    const resolvedTenantId =
-      reservation.tenantField && reservation.tenantId ? String(reservation.tenantId) : GLOBAL_TENANT;
+    const resolvedTenantId = this.resolveSequenceTenantId(reservation.tenantField, reservation.tenantId);
     const key = this.sequencesHasKeyHash
       ? { key_hash: this.sequenceKeyHash(reservation.tableName, resolvedTenantId, reservation.field, reservation.scope) }
       : { object: reservation.tableName, tenant_id: resolvedTenantId, field: reservation.field };
@@ -7942,8 +7999,7 @@ export class SqlDriver implements IDataDriver {
    * that drew from it. See {@link bulkCreate}.
    */
   protected autoNumberCounterKey(reservation: AutoNumberReservation): string {
-    const resolvedTenantId =
-      reservation.tenantField && reservation.tenantId ? String(reservation.tenantId) : GLOBAL_TENANT;
+    const resolvedTenantId = this.resolveSequenceTenantId(reservation.tenantField, reservation.tenantId);
     return this.sequenceKeyHash(reservation.tableName, resolvedTenantId, reservation.field, reservation.scope);
   }
 

packages/drivers/driver-turso/README.md

@@ -130,7 +130,6 @@ table lists the ones that answer differently from local mode.
 | Operation | Refused call | Use instead |
 |:---|:---|:---|
 | Transactions | `beginTransaction()`, `commit()`, `rollback()` (and their deprecated aliases `commitTransaction()` and `rollbackTransaction()`), and `options.transaction` passed to any `RemoteTransport` method in the tree above, to `aggregate()` or to `syncSchemasBatch()` | The local or embedded-replica transport, which run Knex transactions and honour `options.transaction` |
-| Record numbers | `create()`, `bulkCreate()`, and an `upsert()` with no `id`, `_id` or `conflictKeys`, when a row leaves an `autonumber` field empty (`undefined`, `null` or `''`) | The local or embedded-replica transport, which generate record numbers, or a value you supply, which is written unchanged |
 | Deferring schema DDL | `setDeferredDdl(true)`, which `os migrate plan` calls. `setDeferredDdl(false)` is accepted | Run the command against a local SQLite copy of the database (a `file:` URL) |
 | Schema drift detection | `detectManagedDrift()` | `os migrate plan` against a local SQLite copy of the database (a `file:` URL) |
 | Planning the ADR-0104 media column move | `planMediaColumnMove()`, the column step of `os migrate files-to-references` | The local or embedded-replica transport, which plan it |
@@ -147,18 +146,30 @@ table lists the ones that answer differently from local mode.
   reads that and does not call `beginTransaction()`: it runs the callback with
   no transaction and logs a warning once per datasource, or, called with
   `require: true`, throws `TransactionUnsupportedError` before the callback runs.
-- **Record numbers.** Remote mode never generates one. The check uses the
-  `autonumber` fields this driver recorded when it synced the object's schema
-  (`syncSchema`, `syncSchemasBatch` or `initObjects`). An `upsert()` that
-  carries an `id`, `_id` or `conflictKeys` is not refused, because it may merge
-  into an existing row. If it inserts instead, the row is written with the field
-  empty and the driver logs a warning naming the row.
 - **Aggregation.** `aggregate()` called on the driver directly also refuses,
   with the same code, a `groupBy` entry that has a `dateGranularity`, and an
   `aggregations` entry with a non-empty `filter`. `engine.aggregate()` never
   sends either one to this driver: it fetches the rows and computes both in
   memory. The local transport also refuses a per-aggregation `filter`.
 
+#### Record numbers in remote mode
+
+An `autonumber` field a row leaves empty (`undefined`, `null` or `''`) is
+filled in remote mode as it is in local and embedded-replica mode: `create()`,
+`bulkCreate()` and `upsert()` issue the next value from the persistent
+`_objectstack_sequences` counter, rendered with the field's `autonumberFormat`
+(or `format`) and scoped per organization, per date token and per `{field}`
+token exactly as `SqlDriver` scopes it. The counter moves in one statement over
+the connection (`UPDATE … RETURNING`; on a cold counter, a bootstrap from the
+table's highest existing value followed by `INSERT … ON CONFLICT DO UPDATE …
+RETURNING`), so two processes writing to one database never draw the same
+number. A row that already carries a value keeps it (a seed replay or an
+import); an `upsert()` that merges into an existing row keeps the number
+already in the row. A `_objectstack_sequences` table in the legacy shape
+(created by a local-mode driver older than the `key_hash` column) is refused
+with `code: 'DATABASE_ERROR'` / `status: 500` and the remedy in the message;
+remote mode does not migrate it.
+
 Answered in remote mode, differently from local mode:
 
 - **`distinct()`** runs a `SELECT DISTINCT` on the remote database and answers

packages/drivers/driver-turso/package.json

@@ -47,6 +47,7 @@
   "devDependencies": {
     "@types/node": "^26.6.3",
     "better-sqlite3": "^13.0.3",
+    "tsx": "^4.23.12",
     "typescript": "^6.0.3",
     "vitest": "^4.1.11"
   },

packages/drivers/driver-turso/src/remote-transport.ts

@@ -1595,11 +1595,21 @@ export class RemoteTransport {
     return rows[0] ?? null;
   }
 
+  /**
+   * `insertOnlyColumns` — columns written on the INSERT leg and left alone on
+   * the MERGE leg, so a value the caller (or the driver, on its behalf) put on
+   * the row for a NEW row never overwrites the one an existing row already
+   * holds. The transport is told WHICH columns, not WHY: the driver decides
+   * from the schema it holds and this class does not (the same division
+   * `setFilterColumnSql` and `setTenantFieldResolver` draw). A column named
+   * here that is also a merge key is simply a merge key.
+   */
   async upsert(
     object: string,
     data: Record<string, unknown>,
     conflictKeys?: string[],
     table: string = object,
+    insertOnlyColumns: readonly string[] = [],
   ): Promise<Record<string, unknown>> {
     await this.ensureConnected();
 
@@ -1618,7 +1628,7 @@ export class RemoteTransport {
     const mergeKeys = conflictKeys && conflictKeys.length > 0 ? conflictKeys : ['id'];
 
     // Build ON CONFLICT ... DO UPDATE SET
-    const updateCols = columns.filter((c) => !mergeKeys.includes(c));
+    const updateCols = columns.filter((c) => !mergeKeys.includes(c) && !insertOnlyColumns.includes(c));
     const updateClauses = updateCols.map((col) => `"${col}" = excluded."${col}"`).join(', ');
 
     let sql = `INSERT INTO ${this.tableSql(table)} (${columns.map((c) => `"${c}"`).join(', ')}) VALUES (${placeholders})`;