diff --git a/docs/15-costs-and-margins.md b/docs/15-costs-and-margins.md index 659e5c2..09ecc41 100644 --- a/docs/15-costs-and-margins.md +++ b/docs/15-costs-and-margins.md @@ -103,7 +103,7 @@ Prices go up when bitcoin falls. When it rises, do nothing until a line has sat ## Accounting coverage, checked Sept. 4, 2026 -The following inventory describes the host with ntig 0.2.0 and unmigrated +The following inventory describes the host with ntig 0.2.1 and unmigrated format-1 repositories. Installing the library does not migrate roots. Observing an expense does not authorize a new customer charge. `/fuel` and the Health tab expose aggregate tenant meters, not every operation the host pays for. @@ -149,6 +149,15 @@ storage. Neither receipt lookup nor checkpointing reclaims old bytes. Operation counts are host costs, not a new tenant charge; lower read costs alone do not establish a margin after metadata writes and retained storage. +Git HTTP requests reuse up to 2 MiB and 512 entries of immutable payload in a +session that ends with the request. Repeated validation and promotion can +reuse those bytes, while mutable roots and all writes still reach R2 and +its accounting adapter. New requests start fresh, oversized objects bypass +the cache, and Git validation still runs. Lower R2 read counts do not imply +lower CPU cost or a complete request-memory bound. Bulk initial checkpoint +construction avoids writing intermediate receipt-index nodes; it removes +no stored data and does not start migration automatically. + `graspBusy`, `graspControls` and the job round's `working` flag fence work within a live instance. Async handlers can interleave across awaits. Those flags are admission controls, not persisted crash recovery or authority to @@ -232,7 +241,7 @@ These are proposed operating boundaries, not new allowances or charges. ## Proposed maintenance contract -ntig 0.2.0 has no resumable optimization runner. No maintenance job, budget +ntig 0.2.1 has no resumable optimization runner. No maintenance job, budget setting, wake schedule, format migration or garbage collection is enabled by this contract. Existing ref cleanup, retention and alarms keep their current roles. An eventual integration has these boundaries: diff --git a/docs/22-grasp-01-git-hosting.md b/docs/22-grasp-01-git-hosting.md index b6358c6..870c222 100644 --- a/docs/22-grasp-01-git-hosting.md +++ b/docs/22-grasp-01-git-hosting.md @@ -79,6 +79,11 @@ new tip's objects arrive. ## Pull requests and purgatory +Promotion checks for relevant pending events before loading Git packs. An +empty queue or another repository's pending work does not trigger that read. +A retried Git write still completes any pending promotion left by an earlier +successful publication; expiry and authority are checked again after reads. + A pull request or update can name a Git ref as `refs/nostr/`. The event ID is checked as a 64-character lowercase hex value. The relay can hold the event and the Git data while it waits for @@ -125,12 +130,14 @@ The hard limits are: | Format 1 | 128 transactions | legacy WAL replay | | Format 2 | 128 unique packs, 2 MiB manifest | explicitly migrated repository metadata | -ntig 0.2.0 reads both formats, but installing it leaves existing roots alone +ntig 0.2.1 reads both formats, but installing it leaves existing roots alone and new repositories still use format 1. Bindws exposes no checkpoint action or scheduled migration. The backend's explicit `checkpoint()` upgrade is one-way: every reader and writer must support format 2 before an isolated repository is migrated. Versions 0.1.x cannot read the upgraded root. -Migration retains old data and receipts; it does not reclaim storage. +Migration retains old data and receipts; it does not reclaim storage. Bulk +construction publishes the final receipt-index nodes without storing each +intermediate index from the migration. Existing roots are unchanged. Format-2 ref advertisements read the root and manifest without downloading packs. They retain the same accepted HEAD and hidden PR rules. Full clone, @@ -143,6 +150,14 @@ hidden when cleanup cannot commit, while immutable WAL history keeps its old object bytes billed until the relay is torn down. The same retention behavior applies when a ref is cleaned after its 20-minute window. +Each Git HTTP request reuses immutable object bytes within its existing +authority fence. The read session retains at most 2 MiB of payload and 512 +entries; larger objects are read normally. Roots are never cached, every +write still passes through storage reservations, and each new request +rechecks authority. The session closes on return or error. This bounds the +cache, not total request memory, and does not skip Git validation or pending +promotion. + ## Information document and limits When GRASP is on and reads are open, NIP-11 includes: diff --git a/package-lock.json b/package-lock.json index 565b989..701cb31 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,7 +9,7 @@ "dependencies": { "@noble/hashes": "^2.4.0", "nostr-tools": "^2.25.1", - "ntig": "file:vendor/ntig-0.2.0.tgz" + "ntig": "file:vendor/ntig-0.2.1.tgz" }, "devDependencies": { "@cloudflare/vitest-pool-workers": "^0.22.0", @@ -2852,9 +2852,9 @@ "license": "MIT" }, "node_modules/ntig": { - "version": "0.2.0", - "resolved": "file:vendor/ntig-0.2.0.tgz", - "integrity": "sha512-t2/XOHMfFs4nom7ZHnvpaoOHc1ZZRyNThocoYhVWAr76t6UY33zBgngMy6tB3YafAdxfUp8nR7CjXMtKmljyHg==", + "version": "0.2.1", + "resolved": "file:vendor/ntig-0.2.1.tgz", + "integrity": "sha512-PcXiSReGSJJ87q0Nb71DQnzzd5RB+8YNkmYDE2XEWQQxblwuWJHx8cD0M0ser460sxPC4PPMKrkbIrZHKySFeA==", "license": "MIT", "dependencies": { "@scure/base": "2.4.0", diff --git a/package.json b/package.json index bc70f4b..b99d777 100644 --- a/package.json +++ b/package.json @@ -25,7 +25,7 @@ "dependencies": { "@noble/hashes": "^2.4.0", "nostr-tools": "^2.25.1", - "ntig": "file:vendor/ntig-0.2.0.tgz" + "ntig": "file:vendor/ntig-0.2.1.tgz" }, "devDependencies": { "@cloudflare/vitest-pool-workers": "^0.22.0", diff --git a/src/grasp.ts b/src/grasp.ts index e5d50ec..bd18836 100644 --- a/src/grasp.ts +++ b/src/grasp.ts @@ -67,17 +67,22 @@ function prTip(relay: Relay, repo: RepositoryAnnouncement, id: string): string | return /^[0-9a-f]{40}$/.test(tag(e, "c")) ? tag(e, "c") : false; } -// promote makes only verified, present Git tips visible to ordinary queries. +// promote checks pending work before loading Git packs, and rechecks authority +// after the read. Replays still finish promotion left pending by an earlier write. async function promote(relay: Relay, repo: RepositoryAnnouncement, wal: WalRepository) { + const requiredFor = (e: Event) => { + if (relay.settings.isEventHidden(e.id) || relay.settings.isEventBanned(e.id) || relay.settings.isBanned(e.pubkey)) return null; + const relevant = e.kind === KIND_REPO ? repository(relay, repo.owner, repo.identifier)?.id === e.id : e.kind === KIND_REPO_STATE ? tag(e, "d") === repo.identifier && repositoryState(relay, repo)?.id === e.id : eventRepository(relay, e)?.id === repo.id; + return relevant ? requiredObjects(relay, e) : null; + }; + const pending = () => relay.sql.exec<{ raw: string }>(`SELECT raw FROM events JOIN grasp_pending ON grasp_pending.id=events.id WHERE grasp_pending.until>? AND (events.expires=0 OR events.expires>?) LIMIT 1024`, now(), now()).toArray(); + if (!pending().some(({ raw }) => requiredFor(JSON.parse(raw) as Event) !== null)) return; const snapshot = await wal.load(); if (!snapshot.sequence) return; const objects = await readObjects(snapshot.packs); - const pending = relay.sql.exec<{ raw: string }>(`SELECT raw FROM events JOIN grasp_pending ON grasp_pending.id=events.id WHERE grasp_pending.until>? LIMIT 1024`, now()).toArray(); - for (const { raw } of pending) { + for (const { raw } of pending()) { const e = JSON.parse(raw) as Event; - const relevant = e.kind === KIND_REPO ? e.id === repo.id : e.kind === KIND_REPO_STATE ? tag(e, "d") === repo.identifier && repositoryState(relay, repo)?.id === e.id : eventRepository(relay, e)?.id === repo.id; - if (!relevant) continue; - const required = requiredObjects(relay, e); + const required = requiredFor(e); if (required && required.every((oid) => objects.has(oid))) { relay.sql.exec(`DELETE FROM grasp_pending WHERE id=?`, e.id); relay.broadcast(e); @@ -132,8 +137,10 @@ export async function grasp(relay: Relay, req: Request, url: URL): Promise true, observe: (event) => relay.meterBytes(event.requestBytes, event.responseBytes) }); - const response = await handler(req); + const response = await wal.withReadSession(async (scopedWal) => { + const handler = createGitHandler(authorizedRepository(relay, repo, scopedWal), { prefix: parsed.prefix, authorizePush: () => true, observe: (event) => relay.meterBytes(event.requestBytes, event.responseBytes) }); + return await handler(req); + }); const headers = new Headers(response.headers); for (const [key, value] of Object.entries(CORS)) headers.set(key, value); return new Response(response.body, { status: response.status, headers }); diff --git a/test/object/grasp-checkpoint.test.ts b/test/object/grasp-checkpoint.test.ts index 16d714c..cd94ee7 100644 --- a/test/object/grasp-checkpoint.test.ts +++ b/test/object/grasp-checkpoint.test.ts @@ -107,6 +107,7 @@ describe("GRASP checkpoint integration", () => { beforeBytes: measurements.before.bytes, afterBytes: measurements.after.bytes, retainedKeys: measurements.keyCount, + categories: measurements.categories, sequence: measurements.sequence, }); // The accounting adapter retains every immutable manifest/index/receipt @@ -128,6 +129,7 @@ describe("GRASP checkpoint integration", () => { const operations = { get: 0, getBytes: 0, put: 0, putBytes: 0, putAck: 0, head: 0, headBytes: 0 }; let writes = 0; let throwManifest = false; + let ownerCap = 0; let failedManifestKey = ""; let failedManifestBytes = 0; const previous = { ...operations }; @@ -172,7 +174,7 @@ describe("GRASP checkpoint integration", () => { media: bucket, sql: relay.sql, fuelStatus: () => ({ outOfFuel: false }), - settings: { limitsOf: () => ({ cap: 0 }) }, + settings: { limitsOf: () => ({ cap: ownerCap }) }, store: { authorBytes: () => 0 }, } as unknown as typeof relay; const repo = { id: "a".repeat(64), owner: pk(owner), identifier: "ambiguous", clone: [], relays: [], maintainers: [pk(owner)], private: false }; @@ -203,6 +205,17 @@ describe("GRASP checkpoint integration", () => { console.info("ambiguous checkpoint reservation", reserved); expect(reserved.n).toBeGreaterThan(0); expect(reserved.bytes).toBeGreaterThan(0); + ownerCap = relay.sql.exec<{ bytes: number }>("SELECT sum(size) AS bytes FROM grasp_objects WHERE owner=?", pk(owner)).one().bytes; + const beforeQuota = await wal.load(); + const putsBeforeQuota = operations.put; + await wal.withReadSession(async (scoped) => { + await expect(scoped.commit({ id: "ambiguous-initial", updates: [{ name: "refs/heads/main", old: null, new: fixture.commitID }], pack: fixture.pack })).resolves.toMatchObject({ sequence: 1, replayed: true }); + await expect(scoped.commit({ id: "over-quota", updates: [{ name: "refs/tags/quota", old: null, new: fixture.commitID }] })).rejects.toThrow("Repository owner's storage cap reached"); + }); + expect(operations.put).toBe(putsBeforeQuota); + expect((await wal.load()).sequence).toBe(beforeQuota.sequence); + expect((await wal.loadRefs()).refs["refs/tags/quota"]).toBeUndefined(); + expect(relay.sql.exec<{ bytes: number }>("SELECT sum(size) AS bytes FROM grasp_objects WHERE owner=?", pk(owner)).one().bytes).toBe(ownerCap); }); }); }); diff --git a/test/object/grasp.test.ts b/test/object/grasp.test.ts index 5962d81..f962eef 100644 --- a/test/object/grasp.test.ts +++ b/test/object/grasp.test.ts @@ -264,10 +264,10 @@ describe("GRASP", () => { expect((await c.ok(ev(owner, KIND_REPO_STATE, "", [["d", "correction"], ["HEAD", "ref: refs/heads/main"], ["refs/heads/main", first.commitID]]))).ok).toBe(true); c.ws.close(); const path = gitRepositoryPath(npubEncode(pk(owner)), "correction"); - expect(await (await receiveSettled(path, host, null, first.commitID, "refs/heads/main", first.pack)).text()).toContain("ok refs/heads/main"); + expect(await (await receiveSettled(path, host, null, first.commitID, "refs/heads/main", first.pack, "benchmark-initial")).text()).toContain("ok refs/heads/main"); const pr = ev(owner, KIND_GIT_PR, "", [["a", `30617:${pk(owner)}:correction`], ["c", next.commitID]]); const ref = `refs/nostr/${pr.id}`; - expect(await (await receiveSettled(path, host, null, first.commitID, ref)).text()).toContain(`ok ${ref}`); + expect(await (await receiveSettled(path, host, null, first.commitID, ref, undefined, "benchmark-unknown")).text()).toContain(`ok ${ref}`); const publisher = await WS.connect(host); expect((await publisher.ok(pr)).ok).toBe(true); publisher.ws.close(); @@ -315,6 +315,11 @@ describe("GRASP", () => { return snapshot.sequence; } finally { relay.graspBusy = false; } }); + // Pending work for another repository does not reload this one's packs. + const other = await WS.connect(host); + expect((await other.ok(repository(owner, host, "other"))).ok).toBe(true); + expect((await other.ok(ev(owner, KIND_REPO_STATE, "", [["d", "other"], ["HEAD", "ref: refs/heads/main"], ["refs/heads/main", first.commitID]]))).ok).toBe(true); + other.ws.close(); const retried = await runInDurableObject(stub, async (relay) => { const bucket = relay.media; const counters = { gets: 0, getBytes: 0, puts: 0, heads: 0 }; @@ -344,6 +349,8 @@ describe("GRASP", () => { }); expect(retried.body).toContain(`ok ${ref}`); expect(retried.counters.puts).toBe(0); + expect(retried.counters.gets).toBe(11); + expect(retried.counters.getBytes).toBeLessThan(47_529); console.info("checkpoint old PR correction HTTP retry", JSON.stringify({ sequence, ...retried.counters })); await runInDurableObject(stub, async (relay) => { const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "correction")!); @@ -353,6 +360,102 @@ describe("GRASP", () => { const body = await visible.text(); expect(body).toContain(`${next.commitID} ${ref}`); expect(body).toContain("symref=HEAD:refs/heads/main"); + // A new request must recheck authority even when its receipt is committed. + await runInDurableObject(stub, (relay) => relay.settings.setEvent(pr.id, "hide")); + const revoked = await receiveSettled(path, host, null, next.commitID, ref, next.pack, requestId); + expect(await revoked.text()).toContain(`ng ${ref}`); + await runInDurableObject(stub, async (relay) => { + const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "correction")!); + expect((await wal.load()).sequence).toBe(sequence); + }); + }); + + it.each(["expired", "moderated"])("an identical Git retry completes interrupted promotion and rechecks %s candidates", async (mode) => { + const host = `grasp-promotion-${mode}.bind.ws`; + const owner = generateSecretKey(); + const pack = await tinyRepository(); + await rpc(host, owner, "claim"); + await rpc(host, owner, "applypreset", "grasp"); + const announcement = repository(owner, host, "promotion"); + const state = ev(owner, KIND_REPO_STATE, "", [["d", "promotion"], ["HEAD", "ref: refs/heads/main"], ["refs/heads/main", pack.commitID]]); + const c = await WS.connect(host); + expect((await c.ok(announcement)).ok).toBe(true); + expect((await c.ok(state)).ok).toBe(true); + const pr = ev(owner, KIND_GIT_PR, "", [["a", `30617:${pk(owner)}:promotion`], ["c", pack.commitID]]); + expect((await c.ok(pr)).ok).toBe(true); + c.ws.close(); + const path = gitRepositoryPath(npubEncode(pk(owner)), "promotion"); + const requestId = "promotion-recovery"; + const stub = env.RELAY.getByName(`grasp-promotion-${mode}`); + await runInDurableObject(stub, async (relay) => { + const bucket = relay.media; + let published = false; + let failed = false; + const observed = new Proxy(bucket, { get(target, name) { + if (name === "get") return (key: string, options?: R2GetOptions) => { + if (published && key.endsWith("/root.json")) { failed = true; throw new Error("publication succeeded before promotion read failed"); } + return target.get(key, options); + }; + if (name === "put") return async (...args: unknown[]) => { + const result = await Reflect.apply(target.put, target, args); + if (String(args[0]).endsWith("/root.json") && result) published = true; + return result; + }; + const value = Reflect.get(target, name, target); + return typeof value === "function" ? value.bind(target) : value; + } }); + Object.defineProperty(relay, "media", { value: observed, configurable: true }); + try { + const response = await relay.fetch(new Request(`http://${host}${path}/git-receive-pack`, { + method: "POST", + headers: { "content-type": "application/x-git-receive-pack-request", "X-Git-Request-Id": requestId }, + body: concat(packet(`${"0".repeat(40)} ${pack.commitID} refs/heads/main\0report-status\n`), flush(), pack.pack), + })); + expect(await response.text()).not.toContain("ok refs/heads/main"); + expect(published).toBe(true); + expect(failed).toBe(true); + } finally { Reflect.deleteProperty(relay, "media"); } + const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "promotion")!); + expect((await wal.load()).sequence).toBe(1); + expect(relay.sql.exec("SELECT id FROM grasp_pending WHERE id IN (?,?)", announcement.id, state.id).toArray()).toHaveLength(2); + }); + const replay = await runInDurableObject(stub, async (relay) => { + const bucket = relay.media; + let roots = 0; + const observed = new Proxy(bucket, { get(target, name) { + if (name === "get") return (key: string, options?: R2GetOptions) => { + // The replay loads the WAL, then promotion loads it again. Change + // eligibility only after promotion's initial candidate check. + if (key.endsWith("/root.json") && ++roots === 2) { + if (mode === "expired") relay.sql.exec("UPDATE events SET expires=1 WHERE id=?", pr.id); + else relay.settings.setEvent(pr.id, "hide"); + } + return target.get(key, options); + }; + const value = Reflect.get(target, name, target); + return typeof value === "function" ? value.bind(target) : value; + } }); + Object.defineProperty(relay, "media", { value: observed, configurable: true }); + try { + const response = await relay.fetch(new Request(`http://${host}${path}/git-receive-pack`, { + method: "POST", + headers: { "content-type": "application/x-git-receive-pack-request", "X-Git-Request-Id": requestId }, + body: concat(packet(`${"0".repeat(40)} ${pack.commitID} refs/heads/main\0report-status\n`), flush(), pack.pack), + })); + expect(roots).toBeGreaterThanOrEqual(2); + return await response.text(); + } finally { Reflect.deleteProperty(relay, "media"); } + }); + expect(replay).toContain("ok refs/heads/main"); + await runInDurableObject(stub, async (relay) => { + const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "promotion")!); + expect((await wal.load()).sequence).toBe(1); + expect(relay.sql.exec("SELECT id FROM grasp_pending WHERE id IN (?,?)", announcement.id, state.id).toArray()).toEqual([]); + expect(relay.sql.exec("SELECT id FROM grasp_pending WHERE id=?", pr.id).toArray()).toHaveLength(1); + }); + const reader = await WS.connect(host); + expect((await reader.query({ ids: [announcement.id, state.id] })).events).toHaveLength(2); + reader.ws.close(); }); it("removes the public Git contract when reads become restricted", async () => { diff --git a/vendor/README.md b/vendor/README.md index 3e8a250..ae86f38 100644 --- a/vendor/README.md +++ b/vendor/README.md @@ -1,24 +1,25 @@ # Vendored dependencies -`ntig-0.2.0.tgz` is the Git engine and conditional R2 WAL used by GRASP. +`ntig-0.2.1.tgz` is the Git engine and conditional R2 WAL used by GRASP. The lockfile pins the archive integrity. `ntig-source.tar.gz` preserves the complete source and tests for the same build; neither artifact depends on a sibling checkout or a public package registry publication. | Fact | Value | |---|---| -| Source commit | `f6346eb54ce12f54b1c5a18fdd4d3e16220b4feb` | -| Package SHA-256 | `dcfba370518014a0dace1960e13782748965d2479d46d264fdbe0cf0745b153b` | -| Source archive SHA-256 | `ab5552da34bd2dbf3e751c794949d891a5fca8ec1c17ea8d5d845b1636009eff` | +| Source commit | `de3dc0b2a9151973d555658f3e75b0c5669426e8` | +| Package SHA-256 | `0f2ceab406cd40935ec96ba53a0a03835de2e8611792f4a0c137d596c81a2f05` | +| Source archive SHA-256 | `24d9b27856d034e88662168a156f7ff01302012e85809381ff98fb7f373021d1` | | Source branch | `main` in [FelineStateMachine/ntig](https://github.com/FelineStateMachine/ntig) | | License | MIT, included in both archives | +| Supplied package toolchain | Node 22.22.3, npm 10.9.8 | | Library build | `npm ci`, then `npm run build:library` in the extracted source | | Package build | `npm pack` in the extracted source | | Backend checks | `npm run check`, `npm run format:check` | Rebuilding the preserved source with Node 26.8.1 and npm 11.19.0 produces identical tar entries and metadata. The uncompressed tar SHA-256 is -`2c0acfc94c300c59ba73cac4c83185ea0b1b7f939cecf565cfc2f69eb4952fad`; +`f1ad28f92b7ac0ebad4487d3cdd6cabb1a7477f965e3ce883dae47c00a4f9804`; gzip bytes can differ across toolchains. The supplied package checksum above and the lockfile pin the exact installed archive. @@ -29,7 +30,7 @@ integrity after its checks and bindws's integration checks pass. New and existing repositories remain in format 1, whose 128-transaction limit also stops timed ref deletion. Bindws hides expired unknown PR refs -even when physical cleanup cannot advance. Installing 0.2.0 does not migrate +even when physical cleanup cannot advance. Installing 0.2.1 does not migrate roots. The source's `docs/CHECKPOINTS.md` describes the explicit one-way `checkpoint()` upgrade: every reader and writer must be upgraded first, and 0.1.x cannot read a format-2 root. Bindws exposes no migration endpoint diff --git a/vendor/ntig-0.2.0.tgz b/vendor/ntig-0.2.0.tgz deleted file mode 100644 index 672faa0..0000000 Binary files a/vendor/ntig-0.2.0.tgz and /dev/null differ diff --git a/vendor/ntig-0.2.1.tgz b/vendor/ntig-0.2.1.tgz new file mode 100644 index 0000000..7812b3e Binary files /dev/null and b/vendor/ntig-0.2.1.tgz differ diff --git a/vendor/ntig-source.tar.gz b/vendor/ntig-source.tar.gz index 7271375..5a8b1b4 100644 Binary files a/vendor/ntig-source.tar.gz and b/vendor/ntig-source.tar.gz differ