Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 15 additions & 2 deletions docs/15-costs-and-margins.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,8 @@ Prices go up when bitcoin falls. When it rises, do nothing until a line has sat

## Accounting coverage, checked Sept. 4, 2026

The following inventory describes the host with ntig 0.1.1. Observing an
The following inventory describes the host with ntig 0.2.0 and unmigrated
format-1 repositories. Installing the library does not migrate roots. Observing an
expense does not authorize a new customer charge. `/fuel` and the Health tab
expose aggregate tenant meters, not every operation the host pays for.

Expand Down Expand Up @@ -136,6 +137,18 @@ automatic reconciliation queue or orphan collector exists. Counting PUT
bytes again as stored bytes would double-charge existing media accounting.
Future temporary packs also need reservations while both copies exist.

For an explicitly migrated format-2 repository, ref advertisements read two
R2 objects, the root and manifest. This saves legacy history and pack reads
on that path; full loads and Git transfers still read all stored packs.
Each new commit also retains a manifest and receipt-index path nodes beside
its record and optional pack. Ref-only workloads can pass 128 transactions,
but their metadata and reservation rows continue to grow. All of these keys
use the same pre-PUT storage reservation. CAS-loser objects remain accounted
for, and an ambiguous PUT can leave a reservation larger than physical
storage. Neither receipt lookup nor checkpointing reclaims old bytes.
Operation counts are host costs, not a new tenant charge; lower read costs
alone do not establish a margin after metadata writes and retained storage.

`graspBusy`, `graspControls` and the job round's `working` flag fence work
within a live instance. Async handlers can interleave across awaits. Those
flags are admission controls, not persisted crash recovery or authority to
Expand Down Expand Up @@ -219,7 +232,7 @@ These are proposed operating boundaries, not new allowances or charges.

## Proposed maintenance contract

ntig 0.1.1 has no resumable optimization runner. No maintenance job, budget
ntig 0.2.0 has no resumable optimization runner. No maintenance job, budget
setting, wake schedule, format migration or garbage collection is enabled by
this contract. Existing ref cleanup, retention and alarms keep their current
roles. An eventual integration has these boundaries:
Expand Down
22 changes: 17 additions & 5 deletions docs/22-grasp-01-git-hosting.md
Original file line number Diff line number Diff line change
Expand Up @@ -121,11 +121,23 @@ The hard limits are:
|---|---:|---|
| Relay | 16 repositories, 320 MiB Git storage | accepted repositories and billed retained Git objects across the relay |
| Repository | 4 MiB per pack, 16 MiB packed history | one upload and the aggregate packed history read into memory |
| Repository | 128 transactions, 1,024 refs, 4,096 objects | WAL history, ref maps and verified object graphs |

Checkpoints, compaction, orphan collection, large-pack streaming and
production capacity measurements remain follow-up work. Once a repository
reaches its transaction limit, all further Git writes are refused, including
| Repository | 1,024 refs, 4,096 objects | ref maps and verified object graphs |
| Format 1 | 128 transactions | legacy WAL replay |
| Format 2 | 128 unique packs, 2 MiB manifest | explicitly migrated repository metadata |

ntig 0.2.0 reads both formats, but installing it leaves existing roots alone
and new repositories still use format 1. Bindws exposes no checkpoint action
or scheduled migration. The backend's explicit `checkpoint()` upgrade is
one-way: every reader and writer must support format 2 before an isolated
repository is migrated. Versions 0.1.x cannot read the upgraded root.
Migration retains old data and receipts; it does not reclaim storage.

Format-2 ref advertisements read the root and manifest without downloading
packs. They retain the same accepted HEAD and hidden PR rules. Full clone,
fetch and push still load the stored packs; selective object reads,
compaction, orphan collection, large-pack streaming and production capacity
measurements remain follow-up work. Once a format-1 repository reaches its
transaction limit, all further Git writes are refused, including
the timed cleanup transaction. An expired unknown PR ref therefore stays
hidden when cleanup cannot commit, while immutable WAL history keeps its old
object bytes billed until the relay is torn down. The same retention behavior
Expand Down
8 changes: 4 additions & 4 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
"dependencies": {
"@noble/hashes": "^2.4.0",
"nostr-tools": "^2.25.1",
"ntig": "file:vendor/ntig-0.1.1.tgz"
"ntig": "file:vendor/ntig-0.2.0.tgz"
},
"devDependencies": {
"@cloudflare/vitest-pool-workers": "^0.22.0",
Expand Down
1 change: 1 addition & 0 deletions src/grasp.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ function authorizedRepository(relay: Relay, repo: RepositoryAnnouncement, wal: W
});
return {
load: () => authorized.load(),
loadRefs: () => authorized.loadRefs ? authorized.loadRefs() : authorized.load(),
commit: async (request: CommitRequest) => {
const receipt = await authorized.commit(request);
for (const update of request.updates) {
Expand Down
2 changes: 1 addition & 1 deletion src/nip11.ts
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ export function nip11(relay: Relay, host: string) {
if (relay.settings.isLeased() && p.lease) doc.lease = { expires_at: p.lease.until, holder: p.lease.holder || undefined, claim_url: host ? "https://" + host + "/" : undefined };
if (featureOn(p, "grasp") && p.reads === "open") {
doc.supported_grasps = ["GRASP-01"];
doc.repo_acceptance_criteria = "Repository announcements name this service in clone and relays; the relay write rule, bans, proof of work and fuel apply. At most 16 repositories, 4 MiB per pack, 16 MiB packed history and 128 Git transactions per repository.";
doc.repo_acceptance_criteria = "Repository announcements name this service in clone and relays; the relay write rule, bans, proof of work and fuel apply. At most 16 repositories, 4 MiB per pack, 16 MiB packed history and 128 Git transactions per unmigrated format-1 repository. Explicitly migrated format-2 repositories retain at most 128 unique packs.";
if (p.writes !== "open" || p.blockedWords.length) doc.curation = "The relay's configured write rule and blocked topics apply to Nostr repository and collaboration events.";
}
if (featureOn(p, "sites")) doc.nsites = {
Expand Down
208 changes: 208 additions & 0 deletions test/object/grasp-checkpoint.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,208 @@
// Checkpoint migration is exercised through bindws' repository store. The
// fixture is isolated to one Durable Object and never touches a production key.
import { env, runInDurableObject } from "cloudflare:test";
import { describe, expect, it } from "vitest";
import { generateSecretKey } from "nostr-tools/pure";
import { npubEncode } from "nostr-tools/nip19";
import { encodePack, type WalRecord } from "ntig";
import { KIND_REPO } from "../../src/kinds.ts";
import { repository } from "../../src/grasp-state.ts";
import { gitRepository } from "../../src/grasp.ts";
import type { RepositoryAnnouncement } from "../../src/grasp-policy.ts";
import { ev, pk, rpc } from "../helpers/relay.ts";
import { WS } from "../helpers/ws.ts";

const encoder = new TextEncoder();
const oid = async (type: string, data: Uint8Array) => {
const header = encoder.encode(`${type} ${data.length}\0`);
const input = new Uint8Array(header.length + data.length);
input.set(header); input.set(data, header.length);
return Array.from(new Uint8Array(await crypto.subtle.digest("SHA-1", input)), (b) => b.toString(16).padStart(2, "0")).join("");
};

async function fixturePack() {
const blob = encoder.encode("checkpoint fixture\n");
const blobID = await oid("blob", blob);
const tree = new Uint8Array([...encoder.encode("100644 README\0"), ...Uint8Array.from(blobID.match(/../g)!, (part) => parseInt(part, 16))]);
const treeID = await oid("tree", tree);
const commit = encoder.encode(`tree ${treeID}\nauthor ntig <ntig@example.com> 1 +0000\ncommitter ntig <ntig@example.com> 1 +0000\n\ncheckpoint\n`);
const commitID = await oid("commit", commit);
return { commitID, pack: await encodePack([{ type: "blob", data: blob }, { type: "tree", data: tree }, { type: "commit", data: commit }]) };
}

const announcement = (sk: Uint8Array, host: string, identifier: string) => ev(sk, KIND_REPO, "", [
["d", identifier],
["clone", `https://${host}/${npubEncode(pk(sk))}/${identifier}.git`],
["relays", `wss://${host}`],
["maintainers", pk(sk)],
]);

describe("GRASP checkpoint integration", () => {
it("migrates an isolated legacy repository and retains indexed receipts", async () => {
const host = "grasp-checkpoint.bind.ws";
const owner = generateSecretKey();
const identifier = "checkpoint";
await rpc(host, owner, "claim");
await rpc(host, owner, "setpolicy", { features: { grasp: true } });
const connection = await WS.connect(host);
const event = announcement(owner, host, identifier);
expect((await connection.ok(event)).ok).toBe(true);
connection.ws.close();

const fixture = await fixturePack();
const measurements = await runInDurableObject(env.RELAY.getByName(host.split(".")[0]), async (relay) => {
const repo = repository(relay, pk(owner), identifier) as RepositoryAnnouncement | null;
expect(repo).not.toBeNull();
const wal = await gitRepository(relay, repo!);
const first = await wal.commit({ id: "checkpoint-initial", updates: [{ name: "refs/heads/main", old: null, new: fixture.commitID }], pack: fixture.pack });
const legacy: WalRecord[] = [];
legacy.push((await wal.lookupRecord!(first.id))!);
let prior = fixture.commitID;
for (let i = 1; i < 128; i++) {
const receipt = await wal.commit({ id: `checkpoint-legacy-${i}`, updates: [{ name: `refs/tags/legacy-${i}`, old: null, new: prior }] });
legacy.push((await wal.lookupRecord!(receipt.id))!);
}
expect((await wal.load()).sequence).toBe(128);
const before = relay.sql.exec<{ n: number; bytes: number }>("SELECT count(*) AS n, coalesce(sum(size),0) AS bytes FROM grasp_objects WHERE owner=?", pk(owner)).one();

const migration = await wal.checkpoint();
expect(migration).toEqual({ sequence: 128, changed: true });
expect(await wal.checkpoint()).toEqual({ sequence: 128, changed: false });
const migrated = await wal.load();
expect(migrated.checkpoint?.manifestHash).toMatch(/^[0-9a-f]{64}$/);
expect(migrated.records).toHaveLength(1);
expect(migrated.refs["refs/heads/main"]).toBe(fixture.commitID);
expect(await wal.lookupRecord!(legacy[0]!.id)).toMatchObject({ id: legacy[0]!.id, sequence: 1 });

// Format 2 allows more than the legacy 128-record ceiling. Ref-only
// commits exercise manifest/index growth without creating more packs.
for (let i = 128; i < 260; i++) {
await wal.commit({ id: `checkpoint-v2-${i}`, updates: [{ name: `refs/tags/v2-${i}`, old: null, new: prior }] });
}
const replay = await wal.commit({ id: first.id, updates: [{ name: "refs/heads/main", old: null, new: fixture.commitID }], pack: fixture.pack });
expect(replay).toMatchObject({ id: first.id, sequence: 1, replayed: true });
const rebuilt = await gitRepository(relay, repo!);
const cold = await rebuilt.load();
expect(cold.sequence).toBe(260);
expect(cold.refs["refs/tags/v2-259"]).toBe(fixture.commitID);
const advertised = await rebuilt.loadRefs!();
expect(advertised.sequence).toBe(260);
expect(advertised.refs["refs/heads/main"]).toBe(fixture.commitID);
expect(await rebuilt.lookupRecord!("checkpoint-legacy-127")).toMatchObject({ sequence: 128 });
expect(await rebuilt.lookupRecord!("missing-checkpoint-receipt")).toBeNull();

const after = relay.sql.exec<{ n: number; bytes: number }>("SELECT count(*) AS n, coalesce(sum(size),0) AS bytes FROM grasp_objects WHERE owner=?", pk(owner)).one();
const keys = relay.sql.exec<{ key: string; size: number }>("SELECT key, size FROM grasp_objects WHERE owner=? ORDER BY key", pk(owner)).toArray();
const categories = keys.reduce<Record<string, number>>((counts, { key }) => {
const category = key.split("/").at(-2) ?? key;
counts[category] = (counts[category] ?? 0) + 1;
return counts;
}, {});
return { before, after, keyCount: keys.length, categories, sequence: cold.sequence };
});

console.info("checkpoint storage measurement", {
beforeObjects: measurements.before.n,
afterObjects: measurements.after.n,
beforeBytes: measurements.before.bytes,
afterBytes: measurements.after.bytes,
retainedKeys: measurements.keyCount,
sequence: measurements.sequence,
});
// The accounting adapter retains every immutable manifest/index/receipt
// write. This is intentionally observational: no successful PUT sum is
// treated as net storage, and the test records the retained SQL totals.
expect(measurements.after.n).toBeGreaterThan(measurements.before.n);
expect(measurements.after.bytes).toBeGreaterThan(measurements.before.bytes);
expect(measurements.keyCount).toBeGreaterThan(128);
expect(measurements.sequence).toBe(260);
});

it("retains a reservation when an immutable checkpoint write is ambiguous", async () => {
const host = "grasp-checkpoint-ambiguous.bind.ws";
const owner = generateSecretKey();
await rpc(host, owner, "claim");
const fixture = await fixturePack();
await runInDurableObject(env.RELAY.getByName(host.split(".")[0]), async (relay) => {
const objects = new Map<string, { bytes: Uint8Array; etag: string }>();
const operations = { get: 0, getBytes: 0, put: 0, putBytes: 0, putAck: 0, head: 0, headBytes: 0 };
let writes = 0;
let throwManifest = false;
let failedManifestKey = "";
let failedManifestBytes = 0;
const previous = { ...operations };
const measure = (phase: string) => {
const delta = Object.fromEntries(Object.keys(operations).map((key) => [key, operations[key as keyof typeof operations] - previous[key as keyof typeof operations]]));
Object.assign(previous, operations);
console.info("checkpoint R2 operation measurement", { phase, ...delta });
};
const bucket = {
get: async (key: string) => {
operations.get++;
const item = objects.get(key);
if (item) operations.getBytes += item.bytes.length;
return item ? { size: item.bytes.length, etag: item.etag, arrayBuffer: async () => item.bytes.slice().buffer } : null;
},
put: async (key: string, value: ArrayBufferView, options: { onlyIf: { etagMatches?: string; etagDoesNotMatch?: string } }) => {
operations.put++;
operations.putBytes += value.byteLength;
const current = objects.get(key);
if (options.onlyIf.etagDoesNotMatch === "*" && current) return null;
if (options.onlyIf.etagMatches !== undefined && (!current || current.etag !== options.onlyIf.etagMatches)) return null;
const bytes = new Uint8Array(value.buffer, value.byteOffset, value.byteLength).slice();
const item = { bytes, etag: `fake-${++writes}` };
objects.set(key, item);
if (throwManifest && key.includes("/manifests/")) {
failedManifestKey = key;
failedManifestBytes = bytes.length;
throw new Error("lost manifest acknowledgement");
}
operations.putAck++;
return { etag: item.etag };
},
head: async (key: string) => {
operations.head++;
const item = objects.get(key);
if (item) operations.headBytes += item.bytes.length;
return item ? { size: item.bytes.length } : null;
},
};
const fakeRelay = {
slug: "checkpoint-ambiguous",
media: bucket,
sql: relay.sql,
fuelStatus: () => ({ outOfFuel: false }),
settings: { limitsOf: () => ({ cap: 0 }) },
store: { authorBytes: () => 0 },
} as unknown as typeof relay;
const repo = { id: "a".repeat(64), owner: pk(owner), identifier: "ambiguous", clone: [], relays: [], maintainers: [pk(owner)], private: false };
const wal = await gitRepository(fakeRelay, repo);
await wal.commit({ id: "ambiguous-initial", updates: [{ name: "refs/heads/main", old: null, new: fixture.commitID }], pack: fixture.pack });
measure("legacy commit");
await wal.checkpoint();
measure("format1 to format2 checkpoint migration");
await wal.commit({ id: "measured-v2-ref", updates: [{ name: "refs/tags/measured", old: null, new: fixture.commitID }] });
measure("format2 ref-only commit");
await wal.loadRefs();
measure("format2 loadRefs");
await wal.load();
measure("format2 full load");
await wal.lookupRecord("ambiguous-initial");
measure("format2 old receipt lookup");
throwManifest = true;
await expect(wal.commit({ id: "ambiguous-next", updates: [{ name: "refs/tags/ambiguous", old: null, new: fixture.commitID }] })).rejects.toThrow();
measure("ambiguous manifest PUT");
expect(failedManifestKey).toContain("/manifests/");
const retained = relay.sql.exec<{ size: number }>("SELECT size FROM grasp_objects WHERE key=?", failedManifestKey).one();
expect(retained.size).toBe(failedManifestBytes);
expect((await wal.load()).refs["refs/tags/ambiguous"]).toBeUndefined();
throwManifest = false;
await expect(wal.commit({ id: "ambiguous-next", updates: [{ name: "refs/tags/ambiguous", old: null, new: fixture.commitID }] })).resolves.toMatchObject({ replayed: false });
expect((await wal.load()).refs["refs/tags/ambiguous"]).toBe(fixture.commitID);
const reserved = relay.sql.exec<{ n: number; bytes: number }>("SELECT count(*) AS n, coalesce(sum(size),0) AS bytes FROM grasp_objects WHERE owner=? AND key LIKE '%/manifests/%'", pk(owner)).one();
console.info("ambiguous checkpoint reservation", reserved);
expect(reserved.n).toBeGreaterThan(0);
expect(reserved.bytes).toBeGreaterThan(0);
});
});
});
Loading