Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions docs/15-costs-and-margins.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ Prices go up when bitcoin falls. When it rises, do nothing until a line has sat

## Accounting coverage, checked Sept. 4, 2026

The following inventory describes the host with ntig 0.2.0 and unmigrated
The following inventory describes the host with ntig 0.2.1 and unmigrated
format-1 repositories. Installing the library does not migrate roots. Observing an
expense does not authorize a new customer charge. `/fuel` and the Health tab
expose aggregate tenant meters, not every operation the host pays for.
Expand Down Expand Up @@ -149,6 +149,15 @@ storage. Neither receipt lookup nor checkpointing reclaims old bytes.
Operation counts are host costs, not a new tenant charge; lower read costs
alone do not establish a margin after metadata writes and retained storage.

Git HTTP requests reuse up to 2 MiB and 512 entries of immutable payload in a
session that ends with the request. Repeated validation and promotion can
reuse those bytes, while mutable roots and all writes still reach R2 and
its accounting adapter. New requests start fresh, oversized objects bypass
the cache, and Git validation still runs. Lower R2 read counts do not imply
lower CPU cost or a complete request-memory bound. Bulk initial checkpoint
construction avoids writing intermediate receipt-index nodes; it removes
no stored data and does not start migration automatically.

`graspBusy`, `graspControls` and the job round's `working` flag fence work
within a live instance. Async handlers can interleave across awaits. Those
flags are admission controls, not persisted crash recovery or authority to
Expand Down Expand Up @@ -232,7 +241,7 @@ These are proposed operating boundaries, not new allowances or charges.

## Proposed maintenance contract

ntig 0.2.0 has no resumable optimization runner. No maintenance job, budget
ntig 0.2.1 has no resumable optimization runner. No maintenance job, budget
setting, wake schedule, format migration or garbage collection is enabled by
this contract. Existing ref cleanup, retention and alarms keep their current
roles. An eventual integration has these boundaries:
Expand Down
19 changes: 17 additions & 2 deletions docs/22-grasp-01-git-hosting.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,11 @@ new tip's objects arrive.

## Pull requests and purgatory

Promotion checks for relevant pending events before loading Git packs. An
empty queue or another repository's pending work does not trigger that read.
A retried Git write still completes any pending promotion left by an earlier
successful publication; expiry and authority are checked again after reads.

A pull request or update can name a Git ref as
`refs/nostr/<event-id>`. The event ID is checked as a 64-character lowercase
hex value. The relay can hold the event and the Git data while it waits for
Expand Down Expand Up @@ -125,12 +130,14 @@ The hard limits are:
| Format 1 | 128 transactions | legacy WAL replay |
| Format 2 | 128 unique packs, 2 MiB manifest | explicitly migrated repository metadata |

ntig 0.2.0 reads both formats, but installing it leaves existing roots alone
ntig 0.2.1 reads both formats, but installing it leaves existing roots alone
and new repositories still use format 1. Bindws exposes no checkpoint action
or scheduled migration. The backend's explicit `checkpoint()` upgrade is
one-way: every reader and writer must support format 2 before an isolated
repository is migrated. Versions 0.1.x cannot read the upgraded root.
Migration retains old data and receipts; it does not reclaim storage.
Migration retains old data and receipts; it does not reclaim storage. Bulk
construction publishes the final receipt-index nodes without storing each
intermediate index from the migration. Existing roots are unchanged.

Format-2 ref advertisements read the root and manifest without downloading
packs. They retain the same accepted HEAD and hidden PR rules. Full clone,
Expand All @@ -143,6 +150,14 @@ hidden when cleanup cannot commit, while immutable WAL history keeps its old
object bytes billed until the relay is torn down. The same retention behavior
applies when a ref is cleaned after its 20-minute window.

Each Git HTTP request reuses immutable object bytes within its existing
authority fence. The read session retains at most 2 MiB of payload and 512
entries; larger objects are read normally. Roots are never cached, every
write still passes through storage reservations, and each new request
rechecks authority. The session closes on return or error. This bounds the
cache, not total request memory, and does not skip Git validation or pending
promotion.

## Information document and limits

When GRASP is on and reads are open, NIP-11 includes:
Expand Down
8 changes: 4 additions & 4 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"dependencies": {
"@noble/hashes": "^2.4.0",
"nostr-tools": "^2.25.1",
"ntig": "file:vendor/ntig-0.2.0.tgz"
"ntig": "file:vendor/ntig-0.2.1.tgz"
},
"devDependencies": {
"@cloudflare/vitest-pool-workers": "^0.22.0",
Expand Down
23 changes: 15 additions & 8 deletions src/grasp.ts
Original file line number Diff line number Diff line change
Expand Up @@ -67,17 +67,22 @@ function prTip(relay: Relay, repo: RepositoryAnnouncement, id: string): string |
return /^[0-9a-f]{40}$/.test(tag(e, "c")) ? tag(e, "c") : false;
}

// promote makes only verified, present Git tips visible to ordinary queries.
// promote checks pending work before loading Git packs, and rechecks authority
// after the read. Replays still finish promotion left pending by an earlier write.
async function promote(relay: Relay, repo: RepositoryAnnouncement, wal: WalRepository) {
const requiredFor = (e: Event) => {
if (relay.settings.isEventHidden(e.id) || relay.settings.isEventBanned(e.id) || relay.settings.isBanned(e.pubkey)) return null;
const relevant = e.kind === KIND_REPO ? repository(relay, repo.owner, repo.identifier)?.id === e.id : e.kind === KIND_REPO_STATE ? tag(e, "d") === repo.identifier && repositoryState(relay, repo)?.id === e.id : eventRepository(relay, e)?.id === repo.id;
return relevant ? requiredObjects(relay, e) : null;
};
const pending = () => relay.sql.exec<{ raw: string }>(`SELECT raw FROM events JOIN grasp_pending ON grasp_pending.id=events.id WHERE grasp_pending.until>? AND (events.expires=0 OR events.expires>?) LIMIT 1024`, now(), now()).toArray();
if (!pending().some(({ raw }) => requiredFor(JSON.parse(raw) as Event) !== null)) return;
const snapshot = await wal.load();
if (!snapshot.sequence) return;
const objects = await readObjects(snapshot.packs);
const pending = relay.sql.exec<{ raw: string }>(`SELECT raw FROM events JOIN grasp_pending ON grasp_pending.id=events.id WHERE grasp_pending.until>? LIMIT 1024`, now()).toArray();
for (const { raw } of pending) {
for (const { raw } of pending()) {
const e = JSON.parse(raw) as Event;
const relevant = e.kind === KIND_REPO ? e.id === repo.id : e.kind === KIND_REPO_STATE ? tag(e, "d") === repo.identifier && repositoryState(relay, repo)?.id === e.id : eventRepository(relay, e)?.id === repo.id;
if (!relevant) continue;
const required = requiredObjects(relay, e);
const required = requiredFor(e);
if (required && required.every((oid) => objects.has(oid))) {
relay.sql.exec(`DELETE FROM grasp_pending WHERE id=?`, e.id);
relay.broadcast(e);
Expand Down Expand Up @@ -132,8 +137,10 @@ export async function grasp(relay: Relay, req: Request, url: URL): Promise<Respo
return new Response(req.method === "HEAD" ? null : html, { headers: { ...CORS, "content-type": "text/html; charset=utf-8", "cache-control": "no-store" } });
}
const wal = await gitRepository(relay, repo);
const handler = createGitHandler(authorizedRepository(relay, repo, wal), { prefix: parsed.prefix, authorizePush: () => true, observe: (event) => relay.meterBytes(event.requestBytes, event.responseBytes) });
const response = await handler(req);
const response = await wal.withReadSession(async (scopedWal) => {
const handler = createGitHandler(authorizedRepository(relay, repo, scopedWal), { prefix: parsed.prefix, authorizePush: () => true, observe: (event) => relay.meterBytes(event.requestBytes, event.responseBytes) });
return await handler(req);
});
const headers = new Headers(response.headers);
for (const [key, value] of Object.entries(CORS)) headers.set(key, value);
return new Response(response.body, { status: response.status, headers });
Expand Down
15 changes: 14 additions & 1 deletion test/object/grasp-checkpoint.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ describe("GRASP checkpoint integration", () => {
beforeBytes: measurements.before.bytes,
afterBytes: measurements.after.bytes,
retainedKeys: measurements.keyCount,
categories: measurements.categories,
sequence: measurements.sequence,
});
// The accounting adapter retains every immutable manifest/index/receipt
Expand All @@ -128,6 +129,7 @@ describe("GRASP checkpoint integration", () => {
const operations = { get: 0, getBytes: 0, put: 0, putBytes: 0, putAck: 0, head: 0, headBytes: 0 };
let writes = 0;
let throwManifest = false;
let ownerCap = 0;
let failedManifestKey = "";
let failedManifestBytes = 0;
const previous = { ...operations };
Expand Down Expand Up @@ -172,7 +174,7 @@ describe("GRASP checkpoint integration", () => {
media: bucket,
sql: relay.sql,
fuelStatus: () => ({ outOfFuel: false }),
settings: { limitsOf: () => ({ cap: 0 }) },
settings: { limitsOf: () => ({ cap: ownerCap }) },
store: { authorBytes: () => 0 },
} as unknown as typeof relay;
const repo = { id: "a".repeat(64), owner: pk(owner), identifier: "ambiguous", clone: [], relays: [], maintainers: [pk(owner)], private: false };
Expand Down Expand Up @@ -203,6 +205,17 @@ describe("GRASP checkpoint integration", () => {
console.info("ambiguous checkpoint reservation", reserved);
expect(reserved.n).toBeGreaterThan(0);
expect(reserved.bytes).toBeGreaterThan(0);
ownerCap = relay.sql.exec<{ bytes: number }>("SELECT sum(size) AS bytes FROM grasp_objects WHERE owner=?", pk(owner)).one().bytes;
const beforeQuota = await wal.load();
const putsBeforeQuota = operations.put;
await wal.withReadSession(async (scoped) => {
await expect(scoped.commit({ id: "ambiguous-initial", updates: [{ name: "refs/heads/main", old: null, new: fixture.commitID }], pack: fixture.pack })).resolves.toMatchObject({ sequence: 1, replayed: true });
await expect(scoped.commit({ id: "over-quota", updates: [{ name: "refs/tags/quota", old: null, new: fixture.commitID }] })).rejects.toThrow("Repository owner's storage cap reached");
});
expect(operations.put).toBe(putsBeforeQuota);
expect((await wal.load()).sequence).toBe(beforeQuota.sequence);
expect((await wal.loadRefs()).refs["refs/tags/quota"]).toBeUndefined();
expect(relay.sql.exec<{ bytes: number }>("SELECT sum(size) AS bytes FROM grasp_objects WHERE owner=?", pk(owner)).one().bytes).toBe(ownerCap);
});
});
});
107 changes: 105 additions & 2 deletions test/object/grasp.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -264,10 +264,10 @@ describe("GRASP", () => {
expect((await c.ok(ev(owner, KIND_REPO_STATE, "", [["d", "correction"], ["HEAD", "ref: refs/heads/main"], ["refs/heads/main", first.commitID]]))).ok).toBe(true);
c.ws.close();
const path = gitRepositoryPath(npubEncode(pk(owner)), "correction");
expect(await (await receiveSettled(path, host, null, first.commitID, "refs/heads/main", first.pack)).text()).toContain("ok refs/heads/main");
expect(await (await receiveSettled(path, host, null, first.commitID, "refs/heads/main", first.pack, "benchmark-initial")).text()).toContain("ok refs/heads/main");
const pr = ev(owner, KIND_GIT_PR, "", [["a", `30617:${pk(owner)}:correction`], ["c", next.commitID]]);
const ref = `refs/nostr/${pr.id}`;
expect(await (await receiveSettled(path, host, null, first.commitID, ref)).text()).toContain(`ok ${ref}`);
expect(await (await receiveSettled(path, host, null, first.commitID, ref, undefined, "benchmark-unknown")).text()).toContain(`ok ${ref}`);
const publisher = await WS.connect(host);
expect((await publisher.ok(pr)).ok).toBe(true);
publisher.ws.close();
Expand Down Expand Up @@ -315,6 +315,11 @@ describe("GRASP", () => {
return snapshot.sequence;
} finally { relay.graspBusy = false; }
});
// Pending work for another repository does not reload this one's packs.
const other = await WS.connect(host);
expect((await other.ok(repository(owner, host, "other"))).ok).toBe(true);
expect((await other.ok(ev(owner, KIND_REPO_STATE, "", [["d", "other"], ["HEAD", "ref: refs/heads/main"], ["refs/heads/main", first.commitID]]))).ok).toBe(true);
other.ws.close();
const retried = await runInDurableObject(stub, async (relay) => {
const bucket = relay.media;
const counters = { gets: 0, getBytes: 0, puts: 0, heads: 0 };
Expand Down Expand Up @@ -344,6 +349,8 @@ describe("GRASP", () => {
});
expect(retried.body).toContain(`ok ${ref}`);
expect(retried.counters.puts).toBe(0);
expect(retried.counters.gets).toBe(11);
expect(retried.counters.getBytes).toBeLessThan(47_529);
console.info("checkpoint old PR correction HTTP retry", JSON.stringify({ sequence, ...retried.counters }));
await runInDurableObject(stub, async (relay) => {
const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "correction")!);
Expand All @@ -353,6 +360,102 @@ describe("GRASP", () => {
const body = await visible.text();
expect(body).toContain(`${next.commitID} ${ref}`);
expect(body).toContain("symref=HEAD:refs/heads/main");
// A new request must recheck authority even when its receipt is committed.
await runInDurableObject(stub, (relay) => relay.settings.setEvent(pr.id, "hide"));
const revoked = await receiveSettled(path, host, null, next.commitID, ref, next.pack, requestId);
expect(await revoked.text()).toContain(`ng ${ref}`);
await runInDurableObject(stub, async (relay) => {
const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "correction")!);
expect((await wal.load()).sequence).toBe(sequence);
});
});

it.each(["expired", "moderated"])("an identical Git retry completes interrupted promotion and rechecks %s candidates", async (mode) => {
const host = `grasp-promotion-${mode}.bind.ws`;
const owner = generateSecretKey();
const pack = await tinyRepository();
await rpc(host, owner, "claim");
await rpc(host, owner, "applypreset", "grasp");
const announcement = repository(owner, host, "promotion");
const state = ev(owner, KIND_REPO_STATE, "", [["d", "promotion"], ["HEAD", "ref: refs/heads/main"], ["refs/heads/main", pack.commitID]]);
const c = await WS.connect(host);
expect((await c.ok(announcement)).ok).toBe(true);
expect((await c.ok(state)).ok).toBe(true);
const pr = ev(owner, KIND_GIT_PR, "", [["a", `30617:${pk(owner)}:promotion`], ["c", pack.commitID]]);
expect((await c.ok(pr)).ok).toBe(true);
c.ws.close();
const path = gitRepositoryPath(npubEncode(pk(owner)), "promotion");
const requestId = "promotion-recovery";
const stub = env.RELAY.getByName(`grasp-promotion-${mode}`);
await runInDurableObject(stub, async (relay) => {
const bucket = relay.media;
let published = false;
let failed = false;
const observed = new Proxy(bucket, { get(target, name) {
if (name === "get") return (key: string, options?: R2GetOptions) => {
if (published && key.endsWith("/root.json")) { failed = true; throw new Error("publication succeeded before promotion read failed"); }
return target.get(key, options);
};
if (name === "put") return async (...args: unknown[]) => {
const result = await Reflect.apply(target.put, target, args);
if (String(args[0]).endsWith("/root.json") && result) published = true;
return result;
};
const value = Reflect.get(target, name, target);
return typeof value === "function" ? value.bind(target) : value;
} });
Object.defineProperty(relay, "media", { value: observed, configurable: true });
try {
const response = await relay.fetch(new Request(`http://${host}${path}/git-receive-pack`, {
method: "POST",
headers: { "content-type": "application/x-git-receive-pack-request", "X-Git-Request-Id": requestId },
body: concat(packet(`${"0".repeat(40)} ${pack.commitID} refs/heads/main\0report-status\n`), flush(), pack.pack),
}));
expect(await response.text()).not.toContain("ok refs/heads/main");
expect(published).toBe(true);
expect(failed).toBe(true);
} finally { Reflect.deleteProperty(relay, "media"); }
const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "promotion")!);
expect((await wal.load()).sequence).toBe(1);
expect(relay.sql.exec("SELECT id FROM grasp_pending WHERE id IN (?,?)", announcement.id, state.id).toArray()).toHaveLength(2);
});
const replay = await runInDurableObject(stub, async (relay) => {
const bucket = relay.media;
let roots = 0;
const observed = new Proxy(bucket, { get(target, name) {
if (name === "get") return (key: string, options?: R2GetOptions) => {
// The replay loads the WAL, then promotion loads it again. Change
// eligibility only after promotion's initial candidate check.
if (key.endsWith("/root.json") && ++roots === 2) {
if (mode === "expired") relay.sql.exec("UPDATE events SET expires=1 WHERE id=?", pr.id);
else relay.settings.setEvent(pr.id, "hide");
}
return target.get(key, options);
};
const value = Reflect.get(target, name, target);
return typeof value === "function" ? value.bind(target) : value;
} });
Object.defineProperty(relay, "media", { value: observed, configurable: true });
try {
const response = await relay.fetch(new Request(`http://${host}${path}/git-receive-pack`, {
method: "POST",
headers: { "content-type": "application/x-git-receive-pack-request", "X-Git-Request-Id": requestId },
body: concat(packet(`${"0".repeat(40)} ${pack.commitID} refs/heads/main\0report-status\n`), flush(), pack.pack),
}));
expect(roots).toBeGreaterThanOrEqual(2);
return await response.text();
} finally { Reflect.deleteProperty(relay, "media"); }
});
expect(replay).toContain("ok refs/heads/main");
await runInDurableObject(stub, async (relay) => {
const wal = await gitRepository(relay, storedRepository(relay, pk(owner), "promotion")!);
expect((await wal.load()).sequence).toBe(1);
expect(relay.sql.exec("SELECT id FROM grasp_pending WHERE id IN (?,?)", announcement.id, state.id).toArray()).toEqual([]);
expect(relay.sql.exec("SELECT id FROM grasp_pending WHERE id=?", pr.id).toArray()).toHaveLength(1);
});
const reader = await WS.connect(host);
expect((await reader.query({ ids: [announcement.id, state.id] })).events).toHaveLength(2);
reader.ws.close();
});

it("removes the public Git contract when reads become restricted", async () => {
Expand Down
Loading