Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
376 changes: 40 additions & 336 deletions crates/forge-core/src/platform/mod.rs

Large diffs are not rendered by default.

25 changes: 11 additions & 14 deletions docs/BUILDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,20 +158,17 @@ Protocol 14 changes three things that a client has to handle:
Drive rejects. forge-web's `documentForCreate` builds documents through
`Document.fromObject`, which keeps bytes as bytes.

Protocol 13 under the 4.2 verifier has two paging gotchas, which both clients handle in their
complete readers (`query_all_documents` / `queryAllDocuments`):

- **Descending page after a cursor.** The 4.2 grovedb verifier rejects the proof a
protocol-13 node returns for one ("Proof op family does not match the query direction").
A descending read is paged ascending and reversed, which gives the same rows in the same
order.
- **Same-block ties.** A `start_after` cursor excludes the cursor's whole `$createdAt`, so
documents created in the same block as a page's last row, and sorting after it, are
skipped. When the index ends in `$createdAt`, the readers also query the boundary
timestamp with `==`. This needs the boundary row's `$createdAt`, which history-keeping
types (`packManifest`, `event`, `refUpdate`, `issue`, ...) do not return from a proved
query on protocol 13. forge-v2 runs only on protocol 14, where the readers bound the
cursor by document id.
The complete readers (`query_all_documents` / `queryAllDocuments`) page on the `startAfter`
cursor alone, in the requested order. On protocol 14 Drive lowers a `startAfter` off the
primary key as a `startAt` on the cursor's own index key, continues within that key by
document id and drops the cursor row (`DriveDocumentQuery::pads_cursor_page`). So documents
created in the same block as a page's last row are not skipped, and pages are not one row
short. Descending pages after a cursor prove and verify like ascending ones. The primary-key
path is not padded and needs no padding, because ids are unique. `indexOnly` types (`star`,
`watch`, `follow`) refuse a cursor outright, so their reads must stay within one page or page
by a range clause. Protocol 13 got
both wrong, and the readers used to work around it; forge-v2 needs protocol 14, so they no
longer do.

## Networks

Expand Down
4 changes: 2 additions & 2 deletions docs/design/platform-parity-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -236,7 +236,7 @@ CLI (audit of `origin/master`; every read proof-verified; `fetch_contract` is **

| Command | Queries | N+1 |
|---|---|---|
| `git clone` / `git fetch` | identity 1; resolve 2; `read_refs` = contract + `⌈Cfg/100⌉` + `2⌈U/100⌉` (`repo.rs:454-472`, `refs.rs:156-190`); default branch 2; manifests `1+⌈P/100⌉` (+ tie probes); members `2⌈M/100⌉`; per pack (8 in flight) copies in reader order, Platform fallback `⌈chunks/100⌉` | **fetch re-downloads every kind-0 pack whenever anything is new** (`helper.rs:242-316`); partial-clone promisor fetches start a new helper (each pays the whole prologue) |
| `git clone` / `git fetch` | identity 1; resolve 2; `read_refs` = contract + `⌈Cfg/100⌉` + `2⌈U/100⌉` (`repo.rs:454-472`, `refs.rs:156-190`); default branch 2; manifests `1+⌈P/100⌉`; members `2⌈M/100⌉`; per pack (8 in flight) copies in reader order, Platform fallback `⌈chunks/100⌉` | **fetch re-downloads every kind-0 pack whenever anything is new** (`helper.rs:242-316`); partial-clone promisor fetches start a new helper (each pays the whole prologue) |
| `git push` (N refs, C chunks) | `read_refs` ×3 (list, push, converge ×1–6, `helper.rs:368, 501-537`); precheck roles 4 + protected patterns 2; `read_pack_copies` 2; **C sequential chunk writes each awaiting a proof** (`repo.rs:695-715`; only `PlatformBackend::put` pipelines 8, `backends/platform.rs:260-273`); manifest 2; browse index: manifests again + members + locator (pipelined) + manifest; **per ref: contract + full config history + write** (`repo.rs:392-432`) | 0.5–0.7 chunks/s (D-910) is the sequential proof wait; config re-read per ref |
| `dg pr list` | session 3; one page `patch` (cap 100); members 3; **per PR: `patch_view` (target log 2, base ref tips 2 + config, contract ×3) + reviews 2 ≈ 9** (`pr.rs:286-300`, `collab/v2.rs:1028-1046`) | 915 requests / 117 s for 100 PRs (D-500) |
| `dg pr view` | ≈ 20 round trips, 7 of them `getDataContract` | |
Expand Down Expand Up @@ -282,7 +282,7 @@ CLI: `dg pr list` = one `DocumentQuery::with_sub_queries` (rs-sdk; changelog dev

### 3.4 Caches (the delta-cache invariant)

A type is delta-cacheable iff it is immutable **and** non-deletable (then a `$createdAt > cursor` read on a `(repoId, $createdAt)` index is exact): `refUpdate`, `protectedRefUpdate`, `config`, `packManifest`, `manifestPart`, `chunk`, `event`, `authorEvent`, `issue`/`patch` creation rows (the documents are mutable — cache by `$updatedAt`, re-read the page you show), `policy`, `repoKey`. Not cacheable without a TTL: `comment`, `review`, `release`, `label`, `checkRun`, `webhook`, `star`/`follow` (deletable). IDB stores: `refs`, `config`, `feed`, `manifests`, `fragments`, `objects` (LRU 64 MB), keyed by network + repoId, with the cursor = the last row's `$createdAt` (ties are re-read, `$createdAt >=` then dedupe by `$id`; the `$createdAt` tie probe already exists in `query.ts:253-263`). The CLI mirrors this in `~/.cache/dash-forge/<network>/<repoId>/*.jsonl` and in `.git/dash/`. Storage objects: HTTP `Cache-Control: immutable` on content-addressed keys is a bucket setting we document in the storage guide; the browser cache then serves fragments and ranges for free.
A type is delta-cacheable iff it is immutable **and** non-deletable (then a `$createdAt > cursor` read on a `(repoId, $createdAt)` index is exact): `refUpdate`, `protectedRefUpdate`, `config`, `packManifest`, `manifestPart`, `chunk`, `event`, `authorEvent`, `issue`/`patch` creation rows (the documents are mutable — cache by `$updatedAt`, re-read the page you show), `policy`, `repoKey`. Not cacheable without a TTL: `comment`, `review`, `release`, `label`, `checkRun`, `webhook`, `star`/`follow` (deletable). IDB stores: `refs`, `config`, `feed`, `manifests`, `fragments`, `objects` (LRU 64 MB), keyed by network + repoId, with the cursor = the last row's `$createdAt` (ties are re-read, `$createdAt >=` then dedupe by `$id`). The CLI mirrors this in `~/.cache/dash-forge/<network>/<repoId>/*.jsonl` and in `.git/dash/`. Storage objects: HTTP `Cache-Control: immutable` on content-addressed keys is a bucket setting we document in the storage guide; the browser cache then serves fragments and ranges for free.

Contracts: seed evo-sdk from the deployment file's contracts (`wasm-sdk` "let apps seed the contracts they already hold", #4746) so no page fetches `getDataContract`; forge-core memoises `fetch_contract` per process and persists it under `~/.cache/dash-forge/contracts/<id>.bin`, validated by `getDataContractsLatestVersions` (one request for all ids, #4739) once per hour.

Expand Down
3 changes: 1 addition & 2 deletions forge-web/lib/repo/chrome.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ vi.mock('../constants', async (importOriginal) => {
})

import type { ForgeIds } from '../deployments'
import { base64ToHex, bytesToBase64, hexToBase64, setPlatformVersion, type DocumentQuery } from '../sdk'
import { base64ToHex, bytesToBase64, hexToBase64, type DocumentQuery } from '../sdk'
import { CHROME_KEYSET_SPLITS } from './refs'
import { loadRepoHome } from '../view/repo-view'
import { cachedDpnsName, clearDpnsCache } from '../view/dpns'
Expand Down Expand Up @@ -184,7 +184,6 @@ function fakeSdk(
const REF: RepoRef = { forge: FORGE, repoId: REPO, ownerId: OWNER, name: 'demo', visibility: 'public' }

beforeEach(() => {
setPlatformVersion(14)
resetRepoTimelines()
clearDpnsCache()
invalidateMembers(REF, 'devnet')
Expand Down
6 changes: 2 additions & 4 deletions forge-web/lib/repo/chrome.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import type { EvoSDK } from '@dashevo/evo-sdk'
import { DEFAULT_NETWORK, NETWORKS, type Network } from '../constants'
import type { ForgeIds } from '../deployments'
import { compositeOf, countsAt, docsAt, queryComposite, type CompositeSub } from '../sdk/composite'
import { base64ToHex, cursorPadded, queryAllDocuments, shareInFlight, type PlainDocument, type WhereClause } from '../sdk'
import { base64ToHex, queryAllDocuments, shareInFlight, type PlainDocument, type WhereClause } from '../sdk'
import { DOC, type RepoRef } from './contract'
import { membersGeneration, membershipsFromDocs, seedMemberships } from './members'
import { noteForkOf } from './fork-parent'
Expand Down Expand Up @@ -382,9 +382,7 @@ async function readChrome(
const generation = generationOf(key)
const membersAtStart = membersGeneration()
const held = touch(key)
// Only on protocol 14 is a delta page continued safely: a `startAfter` cursor below it drops
// rows tied with the cursor, and the `>=` bound rules out the tie probe. Read whole there.
const known = cursorPadded() ? held?.settled : undefined
const known = held?.settled
const q = compositeOf(
{ dataContractId: forge.core, documentTypeName: DOC.repo, where: [['$ownerId', '==', ownerId], ['name', '==', name]] },
1,
Expand Down
6 changes: 2 additions & 4 deletions forge-web/lib/repo/list-budget.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@
*/

import { sha256 } from '@noble/hashes/sha2.js'
import { beforeAll, describe, expect, it, vi } from 'vitest'
import { describe, expect, it, vi } from 'vitest'

import { base58Encode } from '../auth/base58'
import type { ForgeIds } from '../deployments'
import { bytesToBase64, hexToBase64, setPlatformVersion } from '../sdk'
import { bytesToBase64, hexToBase64 } from '../sdk'
import type { RepoRef } from './contract'
import { mockSdk, newSeen, type Doc, type Seen, type Store } from './drive-mock'
import { queryIssues, type IssueListPage, type IssueSelection } from './issue-index'
Expand Down Expand Up @@ -202,8 +202,6 @@ async function settledIssues(r: Repo, q: IssueSelection): Promise<{ page: IssueL
return { page, loads }
}

beforeAll(() => setPlatformVersion(14))

describe('list request budget on a dash-sized repo (QW2-002, QW3-003)', () => {
for (const items of [5615, 7700, 15_000]) {
for (const state of ['merged', 'closed', 'all', 'unmerged'] as const) {
Expand Down
4 changes: 2 additions & 2 deletions forge-web/lib/repo/packs.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ describe('packed byteArray parsing (tips / supersedes)', () => {
]),
REPO,
)
// The read's row order is the reader's concern (it pages ascending and reverses); look
// the rows up by pack instead of by position.
// The read's row order is the reader's concern (it reads newest first); look the rows up
// by pack instead of by position.
const byPack = (b: number) => docs.find((d) => d.packHash === hashHex(b))
expect(byPack(0xaa)?.uris).toEqual(['https://m/p.pack'])
expect(byPack(0xaa)?.supersedes).toEqual([])
Expand Down
135 changes: 8 additions & 127 deletions forge-web/lib/repo/pagination.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,7 @@ import type { EvoSDK } from '@dashevo/evo-sdk'
import { sha256 } from '@noble/hashes/sha2.js'
import { describe, expect, it } from 'vitest'

import {
ascendingEquivalent,
bytesToBase64,
IncompleteReadError,
queryAllDocuments,
tieProbeAllowed,
} from '../sdk'
import { bytesToBase64, IncompleteReadError, queryAllDocuments } from '../sdk'
import { readConfigBundle } from './config'
import { DOC, type RepoRef } from './contract'
import { readEvents, readIssue, readReviews } from './issues'
Expand Down Expand Up @@ -187,19 +181,12 @@ describe('ref history across a page boundary', () => {

// The query SHAPE, not just the row count: the filter must be carried on every page,
// and page 2 must resume from the last `$id` of page 1 rather than re-reading page 1.
// Between them sits the boundary-tie read (`$createdAt ==` the last row's), which the
// tie-safe pager issues after every full page of a `(prefix, $createdAt)` read.
const refQueries = seen.filter((q) => q.documentTypeName === DOC.refUpdate)
expect(refQueries).toHaveLength(3)
expect(refQueries).toHaveLength(2)
expect(refQueries[0]?.where).toEqual([SCOPE, ['refNameHash', '==', REF_HASH_B64]])
expect(refQueries[0]?.startAfter).toBeUndefined()
expect(refQueries[1]?.where).toEqual([
SCOPE,
['refNameHash', '==', REF_HASH_B64],
['$createdAt', '==', expect.any(Number)],
])
expect(refQueries[2]?.where).toEqual([SCOPE, ['refNameHash', '==', REF_HASH_B64]])
expect(refQueries[2]?.startAfter).toBe(`u-${String(PAGE - 1).padStart(4, '0')}`)
expect(refQueries[1]?.where).toEqual([SCOPE, ['refNameHash', '==', REF_HASH_B64]])
expect(refQueries[1]?.startAfter).toBe(`u-${String(PAGE - 1).padStart(4, '0')}`)
})

it('resolves the tip to the newest push, not the last one on page 1', async () => {
Expand Down Expand Up @@ -413,116 +400,10 @@ describe('pack manifests across a page boundary', () => {
expect(read).toHaveLength(PAGE + 1)
expect(read[0]?.documentId).toBe(`m-${String(PAGE).padStart(4, '0')}`)
expect(read[read.length - 1]?.documentId).toBe('m-0000')
// No descending page after a cursor: the proof a protocol-13 node returns for one fails
// evo-sdk 4.2's verifier, so the newest-first read is paged ascending and reversed.
expect(seen.length).toBeGreaterThan(1)
expect(
seen.filter((q) => q.startAfter !== undefined && q.orderBy?.some(([, d]) => d === 'desc')),
).toEqual([])
})
})

describe('same-block ties at a page boundary (protocol 13)', () => {
/**
* PROTOCOL-13 cursor semantics: `startAfter` excludes every row whose `$createdAt` is <=
* the cursor's, so rows sharing the cursor's timestamp but sorting after it are skipped —
* the real Drive behaviour. An `==` on `$createdAt` is served normally.
*/
function protocol13Sdk(rows: Record<string, unknown>[]): EvoSDK {
return {
documents: {
query: (q: QueryLike): Promise<Map<string, unknown>> => {
let out = [...rows]
for (const [field, op, value] of q.where ?? []) {
if (op === '==') out = out.filter((d) => d[field] === value)
}
if (q.startAfter !== undefined) {
const t = rows.find((d) => d['$id'] === q.startAfter)?.['$createdAt'] as number
out = out.filter((d) => (d['$createdAt'] as number) > t)
}
out = out.slice(0, Math.min(q.limit ?? PAGE, PAGE))
return Promise.resolve(new Map(out.map((d) => [String(d['$id']), d])))
},
},
} as unknown as EvoSDK
}

/** 99 rows at distinct times, then `tied` rows in one block, then one more row. */
function straddlingTie(tied: number): Record<string, unknown>[] {
const rows: Record<string, unknown>[] = Array.from({ length: 99 }, (_, i) => ({
$id: `r-${String(i).padStart(4, '0')}`,
$createdAt: i,
}))
for (let i = 99; i < 99 + tied; i++) {
rows.push({ $id: `r-${String(i).padStart(4, '0')}`, $createdAt: 1_000 })
}
rows.push({ $id: `r-${String(99 + tied).padStart(4, '0')}`, $createdAt: 2_000 })
return rows
}

const q = { dataContractId: 'c', documentTypeName: 'issue', orderBy: [['$createdAt', 'asc']] as const }

it('recovers the rows the cursor alone would skip, once each and in order', async () => {
const rows = straddlingTie(3)
// The mock reproduces the gap: without the tie read, the two tied rows after row 100
// are lost (paging a range-filtered read skips the probe).
const lossy = await queryAllDocuments(protocol13Sdk(rows), {
...q,
where: [['$createdAt', '>', -1]] as never,
})
expect(lossy).toHaveLength(rows.length - 2)

const read = await queryAllDocuments(protocol13Sdk(rows), q)
expect(read.map((d) => d['$id'])).toEqual(rows.map((d) => d['$id']))
})

it('refuses a boundary tie of a full page rather than guessing', async () => {
await expect(queryAllDocuments(protocol13Sdk(straddlingTie(PAGE)), q)).rejects.toThrow(
IncompleteReadError,
)
})

it('probes only where $createdAt ends the index', () => {
const base = { dataContractId: 'c', documentTypeName: 'x' }
expect(tieProbeAllowed({ ...base, orderBy: [['$createdAt', 'asc']] })).toBe(true)
expect(
tieProbeAllowed({
...base,
where: [['$ownerId', '==', 'me']],
orderBy: [
['$ownerId', 'asc'],
['$createdAt', 'asc'],
],
}),
).toBe(true)
expect(tieProbeAllowed({ ...base, orderBy: [['seq', 'asc']] })).toBe(false)
expect(tieProbeAllowed({ ...base, orderBy: [['$createdAt', 'desc']] })).toBe(false)
expect(
tieProbeAllowed({ ...base, where: [['kind', '>', 1]], orderBy: [['$createdAt', 'asc']] }),
).toBe(false)
})
})

describe('ascendingEquivalent', () => {
const base = { dataContractId: 'c', documentTypeName: DOC.packManifest }
it('flips a descending order, leaves an ascending one alone', () => {
expect(ascendingEquivalent({ ...base, orderBy: [['$createdAt', 'desc']] })).toEqual([
['$createdAt', 'asc'],
])
expect(ascendingEquivalent({ ...base, orderBy: [['$createdAt', 'asc']] })).toBeNull()
expect(ascendingEquivalent(base)).toBeNull()
})
it('flips a mixed order only when its ascending fields are pinned by ==', () => {
const orderBy = [
['$ownerId', 'asc'],
['$createdAt', 'desc'],
] as const
expect(ascendingEquivalent({ ...base, orderBy, where: [['$ownerId', '==', 'me']] })).toEqual([
['$ownerId', 'asc'],
['$createdAt', 'asc'],
])
expect(ascendingEquivalent({ ...base, orderBy })).toBeNull()
expect(ascendingEquivalent({ ...base, orderBy, where: [['$ownerId', '>', 'me']] })).toBeNull()
// Paged newest first: page 2 continues after page 1's oldest row, in the same order.
expect(seen).toHaveLength(2)
expect(seen[1]?.orderBy).toEqual([['$createdAt', 'desc']])
expect(seen[1]?.startAfter).toBe('m-0001')
})
})

Expand Down
7 changes: 2 additions & 5 deletions forge-web/lib/repo/pull-index.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
*/

import { sha256 } from '@noble/hashes/sha2.js'
import { beforeAll, describe, expect, it } from 'vitest'
import { describe, expect, it } from 'vitest'

import { base58Encode } from '../auth/base58'
import type { ForgeIds } from '../deployments'
import { bytesToBase64, hexToBase64, setPlatformVersion, type DocumentQuery } from '../sdk'
import { bytesToBase64, hexToBase64, type DocumentQuery } from '../sdk'
import { invalidateRepoFeed } from './issues'
import { queryIssues } from './issue-index'
import { openPullsOnBranch, pullsLinking, queryPulls, type PullSelection } from './pull-index'
Expand Down Expand Up @@ -113,9 +113,6 @@ function fresh(n: number, opts?: { churn?: boolean }) {
const plainOf = (seen: Seen, type: string): DocumentQuery[] => seen.queries.filter((q) => q.documentTypeName === type)
const requests = (seen: Seen): number => seen.composites.length + seen.queries.length + seen.counts.length + seen.sums.length

// Protocol 14 (moutai, bonsia): a paged read continues past its cursor without a tie probe.
beforeAll(() => setPlatformVersion(14))

describe('pull index', () => {
it('lists the first page from one composite and one state sum, with the proved Open / Merged / Closed counts', async () => {
const { sdk, seen, repo } = fresh(203)
Expand Down
Loading
Loading