feat: isolate persistent Next.js caches by build

DESIGN.md

@@ -0,0 +1,3 @@
+# Design notes
+
+- [src/DESIGN.md](src/DESIGN.md) — immutable production filesystem-cache binding and read-only build seeds.

README.md

@@ -171,6 +171,10 @@ When enabled, the plugin will look for an existing `.next` directory and skip th
 
 Build the Next.js application and then exit (including shutting down Harper). Defaults to `false`.
 
+### `cacheDirectory: string`
+
+Root for the opt-in [versioned filesystem cache](#versioned-filesystem-cache), outside the component directory. Defaults to `<harper-root>/.nextjs-cache` for the standard `<harper-root>/components/<app>` layout. Set an absolute path for other layouts or a separate volume; relative paths resolve from the app directory and must still point outside it.
+
 ### `port: number`
 
 Specify a custom HTTP port for the Next.js server. Defaults to the Harper default port (`9926`).
@@ -190,6 +194,48 @@ The `files` option is now optional with plugins. This make configuration simpler
 Glob pattern specifying which files Harper should watch for changes. Example: `'/app/*'`.
 -->
 
+## Versioned filesystem cache
+
+To keep Next.js's incremental render and data caches on disk while isolating deployments, use `versionedCacheHandlerPath()` in `next.config.mjs` or `next.config.ts`:
+
+```js
+import { withHarper, versionedCacheHandlerPath } from '@harperfast/nextjs';
+
+export default withHarper({
+	cacheHandler: versionedCacheHandlerPath(import.meta.dirname),
+});
+```
+
+For CommonJS configs, use `require('@harperfast/nextjs')` and pass `__dirname` instead. This is opt-in and replaces the incremental `cacheHandler`; choose it or the Harper-backed handler below. It does not register the separate `cacheHandlers` interface used by `'use cache'`.
+
+Each production worker captures its build identity before serving. Runtime HTML, RSC, route-handler bodies, Pages Router JSON, and fetch-cache writes go to a directory outside the replaceable component tree. For `~/harper/components/my-app`, the default is:
+
+```text
+~/harper/.nextjs-cache/<app-path-hash>/<build-artifact-hash>/
+```
+
+The identity includes `BUILD_ID` and a digest of the build artifacts, so changing an artifact still separates its cache when `generateBuildId` returns the same ID. Startup reads those files asynchronously twice, before and after Next prepares; its I/O cost grows with build size. The build and runtime configuration must both select this handler with the default `.next` output directory; a mismatch fails startup. Cache contents survive worker restarts and are shared by workers of that app/build on the same filesystem. This is local disk storage; it does not replicate entries across cluster nodes or broadcast tag invalidations between workers or nodes. Next.js's installed filesystem-cache implementation supplies the cache formats and invalidation behavior.
+
+Set `cacheMaxMemorySize: 0` in every app sharing a physical Next.js installation, then restart the workers. Next.js's process-local LRU uses unqualified keys and, once another app initializes it, can also be used by an app configured with zero memory. For a single app using its own Next installation, you can omit that setting and keep Next's memory-cache default. This handler isolates disk caches, not that shared LRU. Disk caching remains enabled. Development and builds outside the Harper plugin retain Next.js's normal filesystem behavior.
+
+For a custom component layout or a separate cache volume, set the plugin's `cacheDirectory` in `config.yaml` to an absolute path outside the component directory:
+
+```yaml
+'@harperfast/nextjs':
+  package: '@harperfast/nextjs'
+  cacheDirectory: /var/cache/harper-nextjs
+```
+
+Build seeds remain read-only in `.next`; cold whole-entry misses can read them without combining their files with partial runtime entries. Once a runtime write takes ownership of a key, a persisted marker prevents invalidated or missing entries from falling back to an older seed; Next.js must regenerate them. Legacy `.next/server/route-cache` entries are ignored. Stop outgoing stock-cache workers before the first deployment using this handler: they can overwrite render seeds on Next.js 14/15/16.2 and fetch seeds on all supported versions. Subsequent rolling deployments need the outgoing workers already using this handler. Existing contaminated build seeds require a clean deployment.
+
+The plugin resolves existing cache-root symlinks at startup, rejects roots that point inside the component, and captures the resolved external path. Retargeting a configured alias afterward does not redirect an existing worker's cache writes.
+
+Persistence covers worker restarts and retains Next.js's unsynced filesystem writes; it does not add power-loss durability. Versioning applies when Harper's plugin serves production; standalone Next.js serving keeps stock caching.
+
+This isolates incremental-cache writes. It does not isolate arbitrary app-code reads after a directory swap, move image-optimization caches, or change the Harper-backed and `'use cache'` handlers. Build artifacts under `.next/server` and initial `.next/cache/fetch-cache` must contain regular files and directories.
+
+Namespaces are retained, including after dropping an app. Debug startup logs identify the bound cache directory. Plan disk capacity and remove unused namespaces while their workers are stopped. There is no automatic cache sweep: removing ownership markers while a worker is serving could revive older seeds after tag invalidation. The existing `HARPER_NEXTJS_SWEEP_OLD_BUILDS` option continues to control static-build cleanup only.
+
 ## Caching (Work In Progress)
 
 `@harperfast/nextjs` includes a Harper-backed cache handler for Next.js [Incremental Static Regeneration (ISR)](https://nextjs.org/docs/app/guides/incremental-static-regeneration), the [Data Cache (`fetch()`)](https://nextjs.org/docs/app/deep-dive/caching#data-cache), and [`unstable_cache`](https://nextjs.org/docs/app/api-reference/functions/unstable_cache). Cached entries live in Harper instead of the worker's local filesystem, so a cache write on one node is visible to every node in the cluster.

fixtures/next-16-versioned-cache/.npmrc

@@ -0,0 +1 @@
+package-lock=false

fixtures/next-16-versioned-cache/app/api/revalidate/route.js

@@ -0,0 +1,6 @@
+import { revalidatePath } from 'next/cache';
+
+export async function POST() {
+	revalidatePath('/', 'page');
+	return Response.json({ revalidated: true });
+}

fixtures/next-16-versioned-cache/app/layout.js

@@ -0,0 +1,3 @@
+export default function Layout({ children }) {
+	return <html><body>{children}</body></html>;
+}

fixtures/next-16-versioned-cache/app/page.js

@@ -0,0 +1,16 @@
+import { existsSync } from 'node:fs';
+import { writeFile } from 'node:fs/promises';
+import { join } from 'node:path';
+import { randomUUID } from 'node:crypto';
+import { release } from '../release.mjs';
+
+export const revalidate = 3600;
+
+export default async function Page() {
+	const gate = process.env.HARPER_NEXTJS_CACHE_GATE;
+	if (gate && existsSync(join(gate, 'hold'))) {
+		await writeFile(join(gate, 'entered'), release);
+		while (!existsSync(join(gate, 'release'))) await new Promise((resolve) => setTimeout(resolve, 25));
+	}
+	return <main><h1 data-release>{release}</h1><p data-nonce>{randomUUID()}</p></main>;
+}

fixtures/next-16-versioned-cache/config.yaml

@@ -0,0 +1,3 @@
+'@harperfast/nextjs':
+  package: '@harperfast/nextjs'
+  bundler: webpack

fixtures/next-16-versioned-cache/next.config.mjs

@@ -0,0 +1,9 @@
+import * as plugin from '@harperfast/nextjs';
+
+export default plugin.withHarper({
+	...(process.env.HARPER_NEXTJS_CACHE_BASELINE !== '1' && {
+		cacheHandler: plugin.versionedCacheHandlerPath(import.meta.dirname),
+	}),
+	cacheMaxMemorySize: 0,
+	generateBuildId: async () => 'deliberately-reused-build-id',
+});

fixtures/next-16-versioned-cache/package.json

@@ -0,0 +1,11 @@
+{
+	"name": "next-16-versioned-cache",
+	"private": true,
+	"type": "module",
+	"dependencies": {
+		"@harperfast/nextjs": "file:../../",
+		"next": "16.3.8",
+		"react": "^19",
+		"react-dom": "^19"
+	}
+}

fixtures/next-16-versioned-cache/release.mjs

@@ -0,0 +1 @@
+export const release = 'v1';

integrationTests/next-16-versioned-cache.pw.ts

@@ -0,0 +1,116 @@
+import { test, expect } from '@playwright/test';
+import { createHarperContext, startHarper, killHarper, teardownHarper, type StartedHarperTestContext } from '@harperfast/integration-testing';
+import { createRequire } from 'node:module';
+import { dirname, join } from 'node:path';
+import { mkdtemp, cp, mkdir, readFile, writeFile, readdir, rename, rm } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { execFile } from 'node:child_process';
+import { promisify } from 'node:util';
+
+const fixtureName = 'next-16-versioned-cache';
+const require = createRequire(import.meta.url);
+const exec = promisify(execFile);
+
+async function htmlFiles(directory: string): Promise<string[]> {
+	let entries;
+	try {
+		entries = await readdir(directory, { withFileTypes: true });
+	} catch (error) {
+		if ((error as NodeJS.ErrnoException).code === 'ENOENT') return [];
+		throw error;
+	}
+	const contents: string[] = [];
+	for (const entry of entries) {
+		const file = join(directory, entry.name);
+		if (entry.isDirectory()) contents.push(...await htmlFiles(file));
+		else if (entry.name.endsWith('.html')) contents.push(await readFile(file, 'utf8'));
+	}
+	return contents;
+}
+
+function nonce(html: string): string {
+	const match = html.match(/<p data-nonce="true">([^<]+)<\/p>/);
+	expect(match).not.toBeNull();
+	return match![1];
+}
+
+const loaderModes = [
+	['native', { lockdown: 'none', moduleLoader: 'none', dependencyLoader: 'native', allowedDirectory: 'any' }],
+	['default', undefined],
+] as const;
+
+for (const [mode, applications] of loaderModes) test(`an old regeneration cannot pollute the next release, and regenerated pages survive restarts (${mode} loader)`, async ({ request }) => {
+	test.setTimeout(300_000);
+	const gate = await mkdtemp(join(tmpdir(), 'next-cache-render-gate-'));
+	const dataRootDir = await mkdtemp(join(tmpdir(), 'next-cache-deployment-'));
+	const cacheDirectory = join(dataRootDir, 'cache-by-build');
+	const context = createHarperContext(fixtureName);
+	context.harper = { dataRootDir };
+	const options = {
+		harperBinPath: join(dirname(require.resolve('harper')), 'bin', 'harper.js'),
+		startupTimeoutMs: 120_000,
+		env: {
+			HARPER_NEXTJS_CACHE_GATE: gate,
+			...(process.env.HARPER_NEXTJS_CACHE_BASELINE && { HARPER_NEXTJS_CACHE_BASELINE: process.env.HARPER_NEXTJS_CACHE_BASELINE }),
+		},
+		config: { threads: { count: 1 }, ...(applications && { applications }) },
+	};
+	let started: StartedHarperTestContext | undefined;
+	try {
+		const app = join(dataRootDir, 'components', fixtureName);
+		await cp(join(import.meta.dirname, '..', 'fixtures', fixtureName), app, { recursive: true, dereference: true });
+		const configFile = join(app, 'config.yaml');
+		await writeFile(configFile, await readFile(configFile, 'utf8') + `  cacheDirectory: ${JSON.stringify(cacheDirectory)}\n`);
+		started = await startHarper(context, options);
+		const { httpURL } = started.harper;
+		const candidate = join(dataRootDir, 'candidate');
+		await cp(app, candidate, { recursive: true, dereference: true, filter: (file) => file !== join(app, '.next') });
+		await writeFile(join(candidate, 'release.mjs'), "export const release = 'v2';\n");
+		await writeFile(join(candidate, 'config.yaml'), await readFile(join(candidate, 'config.yaml'), 'utf8') + '  prebuilt: true\n');
+		await exec(process.execPath, [join(candidate, 'node_modules', 'next', 'dist', 'bin', 'next'), 'build', '--webpack'], {
+			cwd: candidate, env: { ...process.env, ...options.env }, timeout: 120_000, maxBuffer: 10 * 1024 * 1024,
+		});
+
+		const initial = await (await request.get(httpURL)).text();
+		expect(initial).toContain('<h1 data-release="true">v1</h1>');
+		await writeFile(join(gate, 'hold'), '');
+		expect((await request.post(`${httpURL}/api/revalidate`)).status()).toBe(200);
+		const held = request.get(httpURL);
+		await expect.poll(async () => readFile(join(gate, 'entered'), 'utf8').catch(() => '')).toBe('v1');
+		await mkdir(join(dataRootDir, 'aside'));
+		await rename(app, join(dataRootDir, 'aside', fixtureName));
+		await rename(candidate, app);
+		await writeFile(join(gate, 'release'), '');
+		const late = await (await held).text();
+		expect(late).toContain('<h1 data-release="true">v1</h1>');
+		const lateNonce = nonce(late);
+		expect(lateNonce).not.toBe(nonce(initial));
+		const baseline = process.env.HARPER_NEXTJS_CACHE_BASELINE === '1';
+		const writeRoot = baseline ? join(app, '.next', 'server', 'route-cache') : cacheDirectory;
+		await expect.poll(async () => (await htmlFiles(writeRoot)).some((html) => html.includes(lateNonce))).toBe(true);
+
+		await killHarper(started);
+		started = await startHarper(context, options);
+		const first = await (await request.get(httpURL)).text();
+		expect(first).toContain('<h1 data-release="true">v2</h1>');
+		expect(await htmlFiles(join(app, '.next', 'server', 'route-cache'))).toEqual([]);
+		const seedNonce = nonce(first);
+		expect((await request.post(`${httpURL}/api/revalidate`)).status()).toBe(200);
+		let regenerated = '';
+		await expect.poll(async () => {
+			regenerated = nonce(await (await request.get(httpURL)).text());
+			return regenerated;
+		}).not.toBe(seedNonce);
+		await expect.poll(async () => (await htmlFiles(cacheDirectory)).some((html) => html.includes(regenerated))).toBe(true);
+		await killHarper(started);
+		started = await startHarper(context, options);
+		const afterRestart = await request.get(httpURL);
+		expect(nonce(await afterRestart.text())).toBe(regenerated);
+		expect(afterRestart.headers()['x-nextjs-cache']).toBe('HIT');
+	} finally {
+		await writeFile(join(gate, 'release'), '').catch(() => {});
+		if (started) await teardownHarper(started);
+		await rm(dataRootDir, { recursive: true, force: true });
+		await rm(gate, { recursive: true, force: true });
+	}
+});