fix(rest)!: the CSV export writes a text cell in a form its consumer reads as text, and the CSV import reverses it (#25078)

.changeset/25078-csv-text-cell.md

@@ -0,0 +1,21 @@
+---
+'@objectstack/rest': patch
+---
+
+fix(rest)!: the CSV export writes a text cell its consumer would interpret in a form that consumer reads as text, and the CSV import reverses exactly that
+
+Clause-②: no (narrowing) — the CSV export serves different bytes for some text cells, and the CSV import parse reads a cell of the same shape differently; nothing is widened.
+Metadata-protocol: none (a REST export's bytes and a REST import's parse; nothing a metadata author writes changes)
+
+<!-- adr-0087: not-required (no-migration-prescription) No metadata moves: no spec key, authorable spelling, object definition or stored shape is added, removed, renamed or re-shaped, and no exported declaration is removed or narrowed, so there is nothing for `objectstack migrate meta` to rewrite. What changes is the bytes one REST export format serves for some text cells and how the CSV import parse reads a cell of that same shape; stored rows are not touched. The other categories are closed on facts: the package publishes (not unpublished); no ADR-0087 id is named or touched (not registered or already-registered); and nothing exported is removed or narrowed (not runtime-interface-only, type-surface-only or runtime-export-retired). -->
+
+**BREAKING** (the CSV export's bytes change for some text cells, and the CSV import reads a user-authored cell of the same shape differently), shipped as `patch` because the metadata protocol does not move.
+
+The CSV export (`GET /api/v1/data/:object/export?format=csv`) wrote every text cell exactly as stored, so a cell whose leading character the program the file is opened in interprets was read there as something other than text. The rule below is defined once, in `csvTextCellNeedsMark` (`packages/rest/src/export-format.ts`), and both directions read it.
+
+- **The export.** A CSV text cell whose first character is one of a small fixed set that its consumer interprets is written with one apostrophe (U+0027) in front of its content, before the usual RFC-4180 quoting. A text cell is a string in a column whose field is not declared numeric, boolean, date or datetime, a header label, or a reference field's display title; with no field metadata (the raw path) every string cell is one. A cell made only of digits, spaces and the characters a signed number or a phone number is written with is never marked, so phone numbers and signed amounts are byte-identical.
+- **The import.** `parseCsvToRows` removes exactly one leading apostrophe from a cell, header row included, when the rest of the cell is one the export would mark. It serves the sync import (`POST /api/v1/data/:object/import`), the async import job (`POST /api/v1/data/:object/import/jobs`) and plugin-auth's admin user import (`POST /api/v1/auth/admin/import-users`), with an inline mapping or a named one. So a CSV exported and imported back returns the stored value, with the one exception in the next point.
+- **A cell that begins with an apostrophe** followed by a cell the export would mark now imports without that apostrophe; the file cannot tell it from an exported one. That holds for a user-authored cell, and for a stored text value of that shape, which the export writes unchanged, so it comes back without its leading apostrophe. A cell beginning with an apostrophe followed by anything else imports unchanged.
+- **Measured.** A value beginning with TAB or CR is marked; the import removes the mark, and the default `trimWhitespace: true` then trims the leading TAB or CR exactly as it did before. With `trimWhitespace: false` it returns exactly.
+- **Unchanged.** Every other cell; numeric, boolean, date and datetime cells; the xlsx and json export formats; json and xlsx import payloads.
+- **What to do.** Nothing for a file that goes back through these import doors. A program that reads the CSV export some other way and needs the stored text applies the same reversal: remove one leading apostrophe from a cell whose rest the export would mark, by the rule in this repository's `packages/rest/src/export-format.ts`.

packages/plugins/plugin-auth/src/admin-import-users.test.ts

@@ -5,6 +5,7 @@ import { readFileSync } from 'node:fs';
 import { dirname, resolve } from 'node:path';
 import { fileURLToPath } from 'node:url';
 import { assertEngineUpdateDispatch } from '@objectstack/objectql';
+import { RestServer } from '@objectstack/rest';
 import { runAdminImportUsers, IMPORT_USERS_MAX_ROWS, type IdentityImportDeps } from './admin-import-users.js';
 import { settleImportResetDelivery } from './account-invitation-marker.js';
 import type { AdminActor } from './admin-user-endpoints.js';
@@ -719,6 +720,75 @@ describe('runAdminImportUsers — CSV payloads', () => {
   });
 });
 
+/**
+ * [#25078] Export → admin user import round trip, through the CSV text-cell rule.
+ *
+ * `@objectstack/rest`'s CSV export writes a text cell its consumer would
+ * interpret in a form that consumer reads as text, and its CSV import parse
+ * (`parseCsvToRows`, reached through `prepareImportRequest`) reverses exactly
+ * that. This door parses a `csv` body through that same `prepareImportRequest`,
+ * so a user list exported as CSV and imported back here must hand better-auth's
+ * `createUser` every stored name exactly.
+ *
+ * The file comes from the REAL export route of `@objectstack/rest`'s
+ * `RestServer`, over a protocol double serving a two-field `sys_user` schema and
+ * the rows below. The rule is pinned at its readers in `@objectstack/rest`
+ * (`csv-text-cell.test.ts`), and its other CSV doors in that package's
+ * `csv-text-cell-round-trip.test.ts`.
+ */
+describe('runAdminImportUsers — CSV export → import round trip (#25078)', () => {
+  /** Plain words after a trigger character: marked by the export. */
+  const MARKED = ['=Total', '+Name', '-Note', '@acme'];
+  /** Ordinary business values, and a user-authored name that begins with the mark itself. */
+  const UNMARKED = ['+86 138 0013 8000', '+1 555 010 0000', '-120.50', 'Acme Ltd', '120.50', "'Quoted"];
+  const NAMES = [...MARKED, ...UNMARKED];
+
+  async function exportUsersCsv(rows: Array<Record<string, string>>): Promise<string> {
+    const noop = () => {};
+    const server = { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} };
+    const schema = { name: 'sys_user', fields: { email: { name: 'email', type: 'email' }, name: { name: 'name', type: 'text' } } };
+    const protocol = {
+      getMetaItem: vi.fn(async () => ({ type: 'object', name: 'sys_user', item: schema })),
+      findData: vi.fn(async () => ({ object: 'sys_user', records: rows, total: rows.length })),
+    };
+    const rest = new RestServer(server as any, protocol as any, { api: { requireAuth: false } } as any);
+    (rest as any).resolveExecCtx = async () => ({ userId: ACTOR.id });
+    rest.registerRoutes();
+    const route: any = rest.getRoutes().find(
+      (r: any) => r.method === 'GET' && r.path === '/api/v1/data/:object/export',
+    );
+    expect(route).toBeDefined();
+    const chunks: string[] = [];
+    const res: any = {
+      write: (s: unknown) => { chunks.push(String(s)); return true; },
+      end: () => {},
+      header: () => res,
+      status: () => res,
+      json: () => res,
+    };
+    await route.handler({ params: { object: 'sys_user' }, query: { format: 'csv' } }, res);
+    return chunks.join('');
+  }
+
+  it('hands createUser every exported name exactly', async () => {
+    const rows = NAMES.map((name, i) => ({ email: `user${i + 1}@example.com`, name }));
+    const csv = await exportUsersCsv(rows);
+
+    // The file carried the marks, so the round trip below is not a pass-through.
+    expect(csv.split('\r\n')[0]).toBe('email,name');
+    for (const [i, v] of MARKED.entries()) expect(csv).toContain(`user${i + 1}@example.com,'${v}\r\n`);
+    expect(csv).toContain(',+86 138 0013 8000\r\n');
+    expect(csv).toContain(",'Quoted\r\n");
+
+    const m = makeDeps();
+    const res = await runAdminImportUsers(m.deps, makeRequest({ passwordPolicy: 'temporary', csv }), ACTOR);
+    expect(res.status).toBe(200);
+    expect((res.body.data as any).summary.created).toBe(NAMES.length);
+    const byEmail = new Map(m.createUser.mock.calls.map((c: any[]) => [c[0].body.email, c[0].body.name]));
+    expect(rows.map((r) => byEmail.get(r.email))).toEqual(NAMES);
+  });
+});
+
 /**
  * [#17422] The IMPLEMENTOR half of #16952's contract.
  *

packages/rest/src/csv-text-cell-round-trip.test.ts

@@ -0,0 +1,228 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * [#25078] Export → import round trip through each REST door that parses CSV.
+ *
+ * The CSV export writes a text cell its consumer would interpret in a form that
+ * consumer reads as text; the CSV import parse reverses exactly that. The rule
+ * is pinned at its readers in `csv-text-cell.test.ts`. This file drives the
+ * REAL routes over a REAL `ObjectQL` engine with a better-sqlite3 `:memory:`
+ * driver: rows are stored in `note`, exported as CSV through
+ * `GET /data/note/export`, and that file is imported into `note_copy`'s text
+ * fields through
+ *
+ *   - the sync door, `POST /data/note_copy/import`, with an inline mapping;
+ *   - the sync door with a NAMED mapping, whose rows keep their source headers
+ *     until core's `applyMappingToRows` — the reversal must not depend on a
+ *     target field's type;
+ *   - the async door, `POST /data/note_copy/import/jobs`.
+ *
+ * Every stored value comes back exactly. Each case also asserts that the file
+ * it imported carried the marks, so the round trip cannot pass by the export
+ * never having marked anything. plugin-auth's admin user import is pinned in
+ * its own package (`admin-import-users.test.ts`).
+ *
+ * Control characters are written as escapes in this file, never as raw bytes.
+ */
+
+import { describe, it, expect, beforeEach, afterEach } from 'vitest';
+import { ObjectQL } from '@objectstack/objectql';
+import { SqlDriver } from '@objectstack/driver-sql';
+import { ObjectStackProtocolImplementation } from '@objectstack/metadata-protocol';
+import { SysMetadataAuditObject, SysMetadataCommitObject, SysMetadataHistoryObject, SysMetadataObject } from '@objectstack/metadata-core';
+import { SysImportJob } from '@objectstack/platform-objects/audit';
+import { RestServer } from './rest-server';
+import { installDefaultOrganization } from './default-organization.testkit.js';
+
+const liveEngines: ObjectQL[] = [];
+afterEach(async () => {
+  while (liveEngines.length) {
+    try { await liveEngines.pop()?.destroy(); } catch { /* noop */ }
+  }
+});
+
+/**
+ * Text fields only, so every value lands in a text field. `memo`'s label starts
+ * with a trigger character, so the header row carries a marked label too.
+ */
+function noteObject(name: string) {
+  return {
+    name,
+    label: name,
+    systemFields: false,
+    fields: {
+      id: { name: 'id', type: 'text' as const, primaryKey: true, label: 'ID' },
+      body: { name: 'body', type: 'text' as const, label: 'Body' },
+      memo: { name: 'memo', type: 'text' as const, label: '-Memo' },
+    },
+  };
+}
+
+/** Plain words after a trigger character: marked by the export. */
+const MARKED = ['=Total', '+Name', '-Note', '@acme'];
+/** Ordinary business values, and a user-authored cell that begins with the mark itself. */
+const UNMARKED = ['+86 138 0013 8000', '+1 555 010 0000', '-120.50', 'Acme Ltd', '120.50', "'Quoted"];
+const VALUES = [...MARKED, ...UNMARKED];
+
+const INLINE_MAPPING = { ID: 'id', Body: 'body', '-Memo': 'memo' };
+
+function createMockServer() {
+  const noop = () => {};
+  return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} };
+}
+
+function makeRes() {
+  const chunks: string[] = [];
+  const res: any = {
+    write: (s: string) => { chunks.push(typeof s === 'string' ? s : String(s)); return true; },
+    end: () => {},
+    header: () => res,
+    status: (code: number) => { res._status = code; return res; },
+    json: (body: any) => { res._json = body; return res; },
+  };
+  return { res, text: () => chunks.join('') };
+}
+
+async function boot(values: string[]) {
+  const engine = new ObjectQL();
+  liveEngines.push(engine);
+  engine.registerDriver(new SqlDriver({ client: 'better-sqlite3', connection: { filename: ':memory:' }, useNullAsDefault: true }), true);
+  await engine.init();
+  engine.registry.registerObject(noteObject('note') as any);
+  engine.registry.registerObject(noteObject('note_copy') as any);
+  engine.registry.registerObject(SysImportJob as any);
+  await engine.syncSchemas();
+  // A real `single` install's Default Organization: the system writes below
+  // derive it (ADR-0131 D9), as a fresh NOT NULL tenant column requires (D1).
+  await installDefaultOrganization(engine);
+  for (let i = 0; i < values.length; i++) {
+    await engine.insert('note', { id: `r${i + 1}`, body: values[i], memo: values[i] });
+  }
+  for (const o of [SysMetadataObject, SysMetadataHistoryObject, SysMetadataAuditObject, SysMetadataCommitObject]) {
+    if (!engine.registry.getObject(o.name)) engine.registry.registerObject(o as any);
+  }
+  engine.registry.registerItem('mapping', {
+    name: 'note_copy_feed',
+    label: 'Note copy feed',
+    sourceFormat: 'csv',
+    targetObject: 'note_copy',
+    fieldMapping: [
+      { source: 'ID', target: 'id', transform: 'none' },
+      { source: 'Body', target: 'body', transform: 'none' },
+      { source: '-Memo', target: 'memo', transform: 'none' },
+    ],
+  } as any, 'name');
+  const protocol = new ObjectStackProtocolImplementation(engine as any);
+  const rest = new RestServer(createMockServer() as any, protocol as any, { api: { requireAuth: false } } as any);
+  (rest as any).resolveExecCtx = async () => ({ userId: 'test-user' });
+  rest.registerRoutes();
+  const routes = rest.getRoutes();
+  const find = (method: string, path: string) => routes.find((r: any) => r.method === method && r.path === path)!;
+  return {
+    engine,
+    exportRoute: find('GET', '/api/v1/data/:object/export'),
+    importRoute: find('POST', '/api/v1/data/:object/import'),
+    jobCreate: find('POST', '/api/v1/data/:object/import/jobs'),
+    jobProgress: find('GET', '/api/v1/data/import/jobs/:jobId'),
+  };
+}
+
+type Booted = Awaited<ReturnType<typeof boot>>;
+
+async function exportCsv(ctx: Booted): Promise<string> {
+  const { res, text } = makeRes();
+  await ctx.exportRoute.handler({ params: { object: 'note' }, query: { format: 'csv' } } as any, res);
+  return text();
+}
+
+async function importSync(ctx: Booted, body: Record<string, unknown>): Promise<any> {
+  const { res } = makeRes();
+  await ctx.importRoute.handler({ params: { object: 'note_copy' }, body } as any, res);
+  return res._json;
+}
+
+async function importAsync(ctx: Booted, body: Record<string, unknown>): Promise<any> {
+  const { res } = makeRes();
+  await ctx.jobCreate.handler({ params: { object: 'note_copy' }, body } as any, res);
+  expect(res._status).toBe(201);
+  const jobId = res._json.jobId;
+  for (let i = 0; i < 200; i++) {
+    const p = makeRes();
+    await ctx.jobProgress.handler({ params: { jobId }, query: {} } as any, p.res);
+    const status = p.res._json?.status;
+    if (status === 'succeeded' || status === 'failed' || status === 'cancelled') return p.res._json;
+    await new Promise((r) => setTimeout(r, 5));
+  }
+  throw new Error(`import job ${jobId} did not finish`);
+}
+
+async function storedCopies(ctx: Booted, count: number): Promise<Array<{ body: unknown; memo: unknown }>> {
+  const out: Array<{ body: unknown; memo: unknown }> = [];
+  for (let i = 1; i <= count; i++) {
+    const row: any = await ctx.engine.findOne('note_copy', { where: { id: `r${i}` } });
+    out.push({ body: row?.body, memo: row?.memo });
+  }
+  return out;
+}
+
+/** The file carried the marks: the round trip below is not a pass-through. */
+function expectMarkedFile(csv: string): void {
+  expect(csv.split('\r\n')[0]).toBe("ID,Body,'-Memo");
+  for (const v of MARKED) expect(csv).toContain(`,'${v},'${v}\r\n`);
+  for (const v of ['+86 138 0013 8000', '-120.50', 'Acme Ltd']) expect(csv).toContain(`,${v},${v}\r\n`);
+}
+
+describe('CSV export → import round trip, every value returns exactly (#25078)', () => {
+  let ctx: Booted;
+  beforeEach(async () => {
+    ctx = await boot(VALUES);
+  });
+
+  it('sync door, inline mapping', async () => {
+    const csv = await exportCsv(ctx);
+    expectMarkedFile(csv);
+    const res = await importSync(ctx, { format: 'csv', csv, mapping: INLINE_MAPPING });
+    expect(res).toMatchObject({ total: VALUES.length, ok: VALUES.length, errors: 0, created: VALUES.length });
+    expect(await storedCopies(ctx, VALUES.length)).toEqual(VALUES.map((v) => ({ body: v, memo: v })));
+  });
+
+  it('sync door, named mapping (reversal runs before the mapping renames the columns)', async () => {
+    const csv = await exportCsv(ctx);
+    expectMarkedFile(csv);
+    const res = await importSync(ctx, { format: 'csv', csv, mappingName: 'note_copy_feed' });
+    expect(res).toMatchObject({ total: VALUES.length, ok: VALUES.length, errors: 0, created: VALUES.length });
+    expect(await storedCopies(ctx, VALUES.length)).toEqual(VALUES.map((v) => ({ body: v, memo: v })));
+  });
+
+  it('async import-job door', async () => {
+    const csv = await exportCsv(ctx);
+    expectMarkedFile(csv);
+    const done = await importAsync(ctx, { format: 'csv', csv, mapping: INLINE_MAPPING });
+    expect(done).toMatchObject({ status: 'succeeded', total: VALUES.length, created: VALUES.length, errors: 0 });
+    expect(await storedCopies(ctx, VALUES.length)).toEqual(VALUES.map((v) => ({ body: v, memo: v })));
+  });
+});
+
+describe('CSV round trip of a value that begins with TAB or CR (#25078)', () => {
+  const LEADING_WHITESPACE = ['\tTotal', '\rTotal'];
+  let ctx: Booted;
+  beforeEach(async () => {
+    ctx = await boot(LEADING_WHITESPACE);
+  });
+
+  it('the file carries the mark, and `trimWhitespace: false` returns the stored value exactly', async () => {
+    const csv = await exportCsv(ctx);
+    expect(csv).toContain("r1,'\tTotal,'\tTotal\r\n");
+    expect(csv).toContain(`r2,"'\rTotal","'\rTotal"\r\n`);
+    const res = await importSync(ctx, { format: 'csv', csv, mapping: INLINE_MAPPING, trimWhitespace: false });
+    expect(res).toMatchObject({ ok: 2, errors: 0 });
+    expect(await storedCopies(ctx, 2)).toEqual(LEADING_WHITESPACE.map((v) => ({ body: v, memo: v })));
+  });
+
+  it("under the default `trimWhitespace`, core's text trim still removes the leading TAB / CR after the reversal", async () => {
+    const csv = await exportCsv(ctx);
+    const res = await importSync(ctx, { format: 'csv', csv, mapping: INLINE_MAPPING });
+    expect(res).toMatchObject({ ok: 2, errors: 0 });
+    expect(await storedCopies(ctx, 2)).toEqual([{ body: 'Total', memo: 'Total' }, { body: 'Total', memo: 'Total' }]);
+  });
+});