Skip to content

feat: add many() for batch ID generation - #45

Merged
suhailopensource merged 1 commit into
suhailopensource:mainfrom
NspxMiguel:feat/batch-many
Oct 11, 2026
Merged

suhailopensource merged 1 commit into
suhailopensource:mainfrom
NspxMiguel:feat/batch-many

Conversation

@NspxMiguel

Copy link
Copy Markdown
Contributor

Closes #36

id.many(prefix, count), sortableId.many(prefix, count), and .many on the generators returned by createId() / createSortableId().

Handling the traps from the issue:

  1. IdGenerator is unchanged. New BatchIdGenerator<S> = IdGenerator<S> & { many } is the return type of createId / createSortableId. A test assigns a plain function to IdGenerator and npm run typecheck covers it.
  2. 65,536-byte quota: bytes are requested in chunks of at most 65,536. A test wraps the provider and asserts no call exceeds that.
  3. Cap: count must be an integer from 0 to 10,000 (MAX_BATCH), otherwise RangeError. 0 returns [].
  4. Unbiased sampling: same mask + rejection sampling as randomIndices, no modulo.
  5. Sortable stays strictly increasing: many calls the single generator in a loop, so monotonic overflow into the next millisecond is handled by the existing code. A test uses a 2-char alphabet and a frozen clock to force the overflow.

Benchmark

10,000 IDs, median of 15 runs after warm-up, Apple Silicon, same machine:

Array.from many speed-up
Node 26.7, id (size 24) 10.5 ms 2.0 ms 5.3x
Node 26.7, id (size 8) 8.4 ms 0.9 ms 8.9x
Node 26.7, sortableId 3.6 ms 3.4 ms 1.05x
Bun 1.3.14, id (size 24) 2.3 ms 1.3 ms 1.7x
Bun 1.3.14, sortableId 2.7 ms 2.7 ms 1.0x

So the speed-up holds for plain IDs, and is much larger on Node, where each node:crypto call is expensive. It does not hold for sortable IDs: the random part is only drawn when the millisecond changes, so there is nothing to batch, and many there is a convenience over a loop (documented in the README). Not measured: browsers or Deno.

CHANGELOG and README updated. npm run verify passes.

@suhailopensource suhailopensource mentioned this pull request Oct 11, 2026
8 tasks done
@suhailopensource

Copy link
Copy Markdown
Owner

Thanks a lot for this, Miguel! It's a careful PR, and walking through each trap in the description made it easy to review. I checked it locally. Here is what I found.

What's good:

  • IdGenerator is untouched and BatchIdGenerator is new. A plain function still type-checks as IdGenerator. When I added many to IdGenerator, npm run typecheck failed on your test, so that test does its job.
  • id.many("user", 3) is typed as `user_${string}`[], and assigning it to the wrong prefix type is an error.
  • Bytes come in chunks. The biggest single request was 65,536 bytes for both id.many("user", 10000) and createId({ size: 4096 }).many("x", 100), on the CJS and Node ESM builds, and in Deno and Bun.
  • Sampling stays unbiased. A chi-square check over 10,000 IDs landed where it should for 62-, 36-, 33- and 32-character alphabets.
  • Sortable batches stay strictly increasing, including the overflow into the next millisecond. The next single call also sorts after the batch.
  • The single-ID path hasn't changed. With a fixed random source, id, createId, template and sortableId give byte-for-byte the same output as main.
  • Your benchmark holds up here too (Node 22): about 3-4x for id, about 9x for size 8, and about 1x for sortableId. Thanks for being upfront about the sortable case.
  • npm run verify passes on Node 18, 22 and 24, and the built files work on Node 14 and 16.

Three small changes to the tests before merge. The code itself is fine. These are spots where a future bug would slip through with every test still green. I tried each bug on purpose to check.

1. Test the rejection sampling. If I replace bytes[i] & mask + if (index < radix) with bytes[i] % radix, every test still passes. This test catches it. With a 3-letter alphabet, byte 3 is out of range and must be skipped. Modulo would turn it into "a" and give "k_abab":

  it("skips out-of-range bytes instead of wrapping them", () => {
    setBytesProvider((length) =>
      Uint8Array.from({ length }, (_, i) => (i % 2 === 0 ? 3 : 1)),
    );
    expect(createId({ alphabet: "abc", size: 4 }).many("k", 2)).toEqual([
      "k_bbbb",
      "k_bbbb",
    ]);
  });

2. Test that each ID gets its own slice. If randomStrings read indices[n + i] instead of indices[cursor++], each ID would just be the previous one shifted by one character. Every test still passes with that bug. This test catches it:

  it("gives each id its own slice of the random bytes", () => {
    let next = 0;
    setBytesProvider((length) =>
      Uint8Array.from({ length }, () => next++ & 0xff),
    );
    expect(createId({ size: 4, alphabet: "abcdefgh" }).many("u", 2)).toEqual([
      "u_abcd",
      "u_efgh",
    ]);
  });

3. Make the count tests check our own error message. new Array(-1) and new Array(1.5) also throw a built-in RangeError ("Invalid array length"), so the current tests pass even without assertValidCount. When I removed the check from sortableId.many, every test still passed, and sortableId.many("evt", 10001) returned 10,001 IDs. Suggested versions:

  it("rejects an invalid count", () => {
    for (const bad of [-1, 1.5, Number.NaN, Infinity, MAX_BATCH + 1]) {
      const call = () => id.many("user", bad);
      expect(call).toThrow(RangeError);
      expect(call).toThrow(/`count`/);
    }
  });
  it("validates count and prefix", () => {
    for (const bad of [-1, 1.5, MAX_BATCH + 1]) {
      expect(() => sortableId.many("evt", bad)).toThrow(/`count`/);
    }
    expect(() => sortableId.many("", 0)).toThrow(TypeError);
  });

The "" with a count of 0 is on purpose. With count 0 the single generator never runs, so only many's own prefix check can catch it. Today, removing that check doesn't fail any test.

The new tests don't need a try/finally. The afterEach at the top of the file already resets the provider.

Small things (optional):

  • Could you move assertValidCount out of src/internal/prefix.ts into a new src/internal/count.ts? It isn't about prefixes, and it's the only reason this PR touches prefix.ts, which feat: add opt-in strictPrefix option #41 also changes. The imports in generate.ts and sortable.ts would become:
import { assertValidCount } from "../internal/count.js";
import { assertValidPrefix } from "../internal/prefix.js";
  • README: "draws the random bytes in one pass" → "draws the random bytes in a few large chunks". Big batches use several calls (4 for 10,000 default IDs).
  • CHANGELOG: "Sortable batches stay strictly increasing." → "Sortable batches stay strictly increasing (with the default monotonic: true)." With monotonic: false and a frozen clock they don't, same as a loop.

Heads-up about #41, since you wrote that one too. Whichever of the two lands second, the assertValidPrefix(prefix, separator) calls inside both many() functions need the strictPrefix argument as well. Git won't flag those lines, because they're outside the conflict. No test fails either, so createId({ strictPrefix: true }).many("us er", 2) would quietly return IDs. When you rebase, please pass the flag there and add a test for it. (I've also asked in #41 to make strict a required parameter, so TypeScript will point at those lines.)

I tried all of this locally and npm run verify still passes (134 tests, coverage 99.53% lines / 98.23% branches). Thanks again!

@NspxMiguel

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough review. Pushed: the rejection-sampling test, the per-id slice test, and count tests that match our own count message (including sortableId.many("", 0)); assertValidCount moved to src/internal/count.ts so this PR no longer touches prefix.ts; README says "a few large chunks"; the CHANGELOG notes monotonic: true. Rebased on current main; npm run verify passes (134 tests).

On #41: I'll pass strictPrefix into the two many() prefix checks and add a test once #41 lands, so it doesn't depend on whichever order they merge.

@suhailopensource

Copy link
Copy Markdown
Owner

Hi! #41 (strictPrefix) is merged, so this branch needs a rebase AND one small code change. Squash your two commits first: rebasing them one at a time conflicts in 5 files in commit 1 (including src/internal/prefix.ts, because assertValidCount lived there before you moved it to count.ts) and again in commit 2. After squashing there are only 4 conflicts:

git fetch upstream
git checkout feat/batch-many
git reset --soft $(git merge-base HEAD upstream/main)
git commit -m "feat: add many() for batch ID generation"
git rebase upstream/main

(If you'd rather not rewrite history, git merge upstream/main gives the same 4 conflicts.)

Resolve them like this:

  1. CHANGELOG.md: ONE ### Added under ## [Unreleased] with all 4 bullets in this order: entropy (feat: add entropy and collision-probability helpers #43), strictPrefix (feat: add opt-in strictPrefix option #41), your many(prefix, count) bullet, your BatchIdGenerator<S> bullet.
  2. README.md: keep both sections, ## Strict prefixes first, then your ## Bulk generation, then ## Install.
  3. src/generators/generate.ts: keep your declaration line (no return) with feat: add opt-in strictPrefix option #41's 3-argument call:
    function id<P extends string>(prefix: P): PrefixedId<P, S> {
    assertValidPrefix(prefix, separator, strictPrefix);
  4. src/generators/sortable.ts: same pattern:
    function sortableId<P extends string>(prefix: P): PrefixedId<P, S> {
    assertValidPrefix(prefix, separator, strictPrefix);

Now run npm run typecheck. It fails, and that is expected:

src/generators/generate.ts(51,5): error TS2554: Expected 3 arguments, but got 2.
src/generators/sortable.ts(174,5): error TS2554: Expected 3 arguments, but got 2.

These are the assertValidPrefix(prefix, separator) calls inside your two many() functions. Git doesn't flag them because they're not in a conflict hunk. Without the fix, createId({ strictPrefix: true }).many("bad id", 3) silently returns invalid IDs, and the sortable many("bad id", 0) returns [] instead of throwing. Change both to:
assertValidPrefix(prefix, separator, strictPrefix);
(Please pass the variable, not a literal, and don't make strict optional again. Either would hide the bug from the compiler.)

Then add this guard test at the end of test/many.test.ts:

describe("many() with strictPrefix", () => {
  it("rejects an invalid prefix like the single generator", () => {
    for (const gen of [
      createId({ strictPrefix: true }),
      createSortableId({ strictPrefix: true }),
    ]) {
      expect(() => gen.many("bad id", 3)).toThrow(
        /invalid character " " at index 3/,
      );
      expect(() => gen.many("bad id", 0)).toThrow(TypeError);
      expect(() => gen.many("a".repeat(64), 1)).toThrow(TypeError);
      expect(gen.many("ok", 3)).toHaveLength(3);
    }
  });
});

Optionally, add to your many() CHANGELOG bullet that it "checks the prefix exactly like a single call, including strictPrefix and even when count is 0".

git add -A
git rebase --continue
npm ci
npm run verify             # expect 10 test files / 158 tests, coverage above 99/97/100/99, publint + attw clean
git push --force-with-lease

id, sortableId and the generators from createId/createSortableId gain
many(prefix, count). Random bytes are drawn in chunks of at most 65,536
bytes (the getRandomValues quota), count is capped at 10,000, and the
existing rejection sampling is reused. BatchIdGenerator is a new type, so
IdGenerator stays unchanged.
@NspxMiguel

Copy link
Copy Markdown
Contributor Author

Thanks for the exact steps. Squashed, rebased onto main (with #41, #42, #43), and resolved the 4 conflicts as described: one ### Added in the order entropy / strictPrefix / many / BatchIdGenerator, README with ## Strict prefixes then ## Bulk generation, and the two declarations with the 3-argument call.

The typecheck error you predicted was real; both many() bodies now call assertValidPrefix(prefix, separator, strictPrefix) (variable, strict still required). Added the many() with strictPrefix guard test, and the CHANGELOG bullet says it checks the prefix exactly like a single call, including strictPrefix and a count of 0.

npm run verify: 10 test files, 158 tests, coverage 99.6 / 98.49 / 100 / 99.6, publint and attw clean.

@suhailopensource
suhailopensource merged commit 10c86dd into suhailopensource:main Oct 11, 2026
8 checks passed
@suhailopensource

Copy link
Copy Markdown
Owner

Merged, thank you! Verified locally: identical to the version I'd fully tested, 158 tests passing, and the strictPrefix guard holds. Great work across all five PRs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature] : batch generation — many IDs from one random-bytes call

2 participants