Ephemeral AI FS implementation plan

Field	Value
Status	M7 complete; overall plan in progress
Target	Version 0.1 integration candidate
Delivery style	Milestone exits with objective acceptance evidence
Database foundation	SQLite remains authoritative

Use the ready-to-paste implementation prompt to start an execution task that follows this plan.

1. Execution rules

Milestones are ordered by dependency. Work inside one milestone may proceed in parallel, but the next milestone does not become the integration baseline until the current milestone's acceptance criteria pass.

Each milestone exit records:

the exact commit;
completed checklist items;
commands and environments used;
machine-readable correctness and benchmark results;
known deviations with an owner and follow-up milestone; and
confirmation that the working tree and generated package artifacts are reproducible from a clean checkout.

Correctness, durability, integrity, and resource ceilings are hard gates. Latency cannot waive them. Tests are finite and iteration-based:

the mandatory smoke profile is capped at 60 seconds per target;
the default correctness and benchmark selection should finish within 10 minutes per target;
an optional load-10m profile is hard-capped at 10 minutes; and
10 GiB logical manifests and millions-of-rows jobs are extended, non-gating diagnostics.

Execution environments and external authority

The normative Durable Object runtime terms are defined in filesystem-api.md. M6 uses the credential-free faithful local Workers runtime for its complete conformance and 60-second smoke gates. M6 creates a deployable preview fixture but MUST NOT deploy it. M9 is the first milestone whose gate requires that reviewed fixture to run in a hosted Cloudflare preview.

The other external gates are equally explicit:

M7 requires a privileged Linux runner with a real mounted FUSE filesystem. A shim, mocked binding, or bridge-only test does not satisfy the real-FUSE acceptance items.
M8 may use the faithful local Durable Object runtime for its Node-to-Durable-Object correctness matrix; hosted replication evidence is deferred to M9/M10.
M9 requires the hosted Cloudflare preview, the real-FUSE reference runner, the isolated DOFS comparison, and publication of the selected release-candidate package set.
M10 requires the actual Computer integration target and the exact published package versions selected by M9.

Git pushes, hosted deployments, registry publication, changes outside the authorized workspace, and mutations of a Computer integration repository are external actions. They MUST NOT be inferred from a request to implement or validate a milestone. Each requires explicit user authorization at the point of execution. If a mandatory external gate lacks its environment, access, or authorization, the milestone remains incomplete and is reported as blocked; local evidence MUST NOT be relabeled as the missing external evidence.

2. Dependency map

M0 repository and test foundation
  -> M1 CAS, CDC, COW, patches, and manifests
      -> M2 SQLite storage and Node driver
          -> M3 filesystem namespace and I/O
              -> M4 branches and publication
                  -> M5 maintenance and recovery
                      -> M6 Cloudflare SQLite parity
                          -> M7 Node VFS
                          -> M8 replication
                              -> M9 release candidate
                                  -> M10 Computer integration

Milestones 7 and 8 may proceed in parallel after milestones 0 through 6 pass.

3. Milestone 0: Repository and test foundation

M0 objective

Create a buildable monorepo whose package and dependency boundaries prevent later implementations from bypassing the architecture.

M0 checklist

M0 acceptance criteria

A clean install builds every empty package without unpublished local state.
The dependency graph has zero cycles and matches the allowed direction.
Packed consumers can import only documented export paths.
Deep imports of CAS, CDC, COW, manifests, repositories, schema, and transactions fail.
The testkit can create, label, seed, and dispose a driver fixture through a recording implementation.
Fixture generation is byte-identical across two clean runs.
Documentation lint, link validation, build, and architecture checks pass.

4. Milestone 1: Content algorithms and binary formats

M1 objective

Implement the pure, host-independent CAS, CDC, COW, patch, and segmented manifest algorithms before adding persistence.

M1 checklist

M1 acceptance criteria

5. Milestone 2: Transactional SQLite storage and Node driver

M2 objective

Persist the content engine, revisions, staging, quotas, and schema through one transaction-only SQLite contract and a production-capable Node driver.

M2 checklist

M2 acceptance criteria

6. Milestone 3: Filesystem namespace, revisions, and I/O

M3 objective

Deliver the public filesystem facade with complete namespace semantics, revision commits, bounded range I/O, and snapshot streams on Node SQLite.

M3 checklist

M3 acceptance criteria

All portable namespace and I/O conformance cases pass on Node SQLite.
Links, rename, timestamps, revision history, and UTF-8 ordering survive reopen.
Every mutation is atomic under statement-level fault injection.
Reads return exact selected bytes and create no durable content state except a required bounded lease.
A cold range lookup does not enumerate the complete manifest.
A one-byte edit in files with both fewer and more than 16,384 manifest entries reads authenticated work bounded by manifest depth plus affected groups, or takes an explicitly reported safe fallback; stale/corrupt derived indexes cannot change the result.
Increasing a streamed fixture from 100 MiB to 1 GiB adds no more than one output chunk, one maximum CAS object, and one manifest node to managed memory high-water.
Cancellation, failure, and close release every lease and reservation.
The 60-second Node SQLite smoke profile passes.

7. Milestone 4: Branches and publication

M4 objective

Deliver durable private branches, deterministic conflicts, atomic publication, and exact replay without copying whole workspaces.

M4 status

Complete. All M4 checklist items and acceptance criteria below are verified. The final M4 suite passes 58/58 tests, including the 50-writer publication cases, fault-injected preparation and finalization, replay after physical reopen, stream and lease stability, COW cleanup, retention, and bounded publication preflight. The smaller branch benchmark passes 20/20 cells; see the repository README for the measured preparation and publication results.

M4 checklist

M4 acceptance criteria

Fifty independent writers publish into one valid parent chain.
Fifty same-inode writers produce one merge and 49 explicit conflicts.
A conflict changes neither main nor the active branch overlay.
Lost-response replay returns the original result after physical restart.
One operation ID can never publish another branch or generation.
Independent sibling publications succeed in either order with exact parent timestamps.
Hard-link aliases preserve inode identity and conflict as one node.
A branch stream retains its original bytes across later edit, materialization, publication or discard, collection, and restart.
Repeated same-page writes retain one current page plus only explicitly leased predecessor versions.

8. Milestone 5: Maintenance, recovery, and bounded scale

M5 objective

Make long-lived databases self-verifying and reclaimable without unbounded transactions, memory, WAL retention, or process-local indexes.

M5 status

Complete. All mandatory M5 checklist items and acceptance criteria below are verified. The default pnpm validate:m5:pre-evidence Node selection builds once, measures its benchmarks and 60-second smoke without competing I/O, and then runs isolated correctness groups concurrently without repeating predecessor maintenance tests; it enforces the ten-minute per-target ceiling directly. The final M5 suite passes 34/34 tests. Its physical-reopen fault matrix independently tests all 328 observed durable statement positions and all 150 committed batch positions, including root mutations between interruption and reopen. The mandatory scale fixture contains 100,001 namespace rows, 100,001 reachable objects, 100,002 manifest roots, and 100,001 manifest nodes, plus 300,003 durable snapshot and collection marks. It completes full verification, snapshotting, collection, exact usage recount, concurrent writing, WAL checkpointing, and physical reopen. Identical workloads at 10,240 and 100,000 fixture rows measured 9,274,632 and 9,315,643 bytes of managed-memory high-water, respectively, proving a 41,011-byte non-proportional difference under the 16 MiB asserted ceiling. Exact run-specific process peaks, WAL high-water, and maximum bounded-call duration are retained in the candidate's machine-readable exit evidence; the executable gates require absolute heap below 512 MiB, RSS below 768 MiB, WAL at or below the explicit 512 MiB limit, and every maintenance call below 5 seconds. The optional 10 GiB and millions-of-rows diagnostics remain intentionally deferred.

M5 checklist

M5 acceptance criteria

Collection never deletes any value reachable from a required root.
Reachable corruption stops the sweep and reports integrity failure.
Mark and sweep resume after every injected interruption.
Root additions reconcile without discarding completed mark work.
Bounded storage accounting holds no database-wide read transaction.
Managed memory does not grow with the mandatory 100,000-row fixture.
Metadata-only and blocked-checkpoint workloads enforce finite ceilings.
Quota failure leaves usage counters exact and maintenance able to progress.
All maintenance operations expose bounded progress and stable metrics.

9. Milestone 6: Cloudflare Durable Object SQLite parity

M6 objective

Run the same filesystem engine and portable conformance outcomes on Durable Object SQLite without importing DOFS or reimplementing filesystem logic.

M6 checklist

Implement the Cloudflare SQLite driver over Durable Object SQLite.
Map callback-scoped transactions to the runtime transaction facility.
Normalize rows, BLOBs, safe integers, constraints, busy errors, and corruption errors.
Report conservative BLOB, binding, physical quota, journal, durability, and runtime-memory capabilities.
Implement runtime restart and eviction test hooks.
Add the deployable preview fixture and exercise its exact Worker bundle, compatibility date, bindings, and SQLite migration in the faithful local Workers runtime without deploying it.
Run storage, filesystem, branch, maintenance, recovery, and resource suites through the shared testkit in the faithful local Workers runtime.
Add the credential-free, faithful-local 60-second Durable Object SQLite smoke profile.

M6 acceptance criteria

Every mandatory portable test from milestones 1 through 5 passes on both Node and Durable Object SQLite.
Adapter-specific setup does not change public results or error codes.
Runtime restart reconstructs all state from committed SQLite data.
The driver never uses an in-memory SQLite mirror or filesystem index.
The driver reports finite conservative resource capabilities.
The faithful-local 60-second Durable Object smoke profile passes without a Cloudflare account, credentials, network deployment, or remote resource.
pnpm test:m6 and pnpm validate:m6 are fully local and create no external state.

10. Milestone 7: Node VFS and real FUSE readiness

M7 objective

Provide the synchronous Node filesystem surface needed by Computer while keeping FUSE and process ownership outside Ephemeral AI FS.

M7 checklist

M7 acceptance criteria

Repeated reads on one handle reuse a pinned selection and return exact bytes.
Three sessions on one inode pass every commit order without lost updates.
Hidden staging never satisfies fsync or advances visible state.
Successful commit, close, restart, unmount, and remount preserve digest.
Large reads and writes allocate no whole-file buffer.
Sixty-four sessions remain inside pending-write and aggregate memory limits with backpressure.
The real-mounted-FUSE smoke profile completes within 60 seconds; a shim or mocked binding does not count.
Computer needs only handle forwarding and no filesystem semantics.

The implementation, shared conformance/fault suite, resource gates, and test-only real FUSE host are complete. The candidate-bound profile completed in 24,815 ms with all 9,056 operations, exact digests, three restarts, resumed collection, and zero active durable state. M7 is accepted and validate:accepted selects it.

11. Milestone 8: Replication

M8 objective

Replicate revisions, active private branches, manifests, CAS objects, and authority-owned results through a bounded host-neutral protocol without exposing tables or raw content mutation. Deliver the shared-runtime, branch-mounted, resumable contract required by the pinned Ephemeral AI Computer fork; production cutover remains M10.

M8 checklist

M8 acceptance criteria

12. Milestone 9: Version 0.1 integration candidate

M9 objective

Turn the implementation into reproducible packages with complete correctness, resource, performance, migration, and compatibility evidence.

M9 checklist

M9 acceptance criteria

Every normative MUST and MUST NOT maps to passing evidence.
There are zero digest mismatches, partial commits, lost updates, unsafe collections, leaked reservations, or usage-counter mismatches.
All mandatory gates complete without an elapsed-time soak.
Default correctness and benchmarks finish within the documented bounded profile on the reference targets.
Performance and resource result artifacts are checked in and reproducible.
Public exports match the approved API snapshot.
A clean consumer can open, use, close, reopen, and verify both drivers.
The hosted-preview artifact identifies the reviewed bundle, compatibility date, target environment, Durable Object migration, and non-secret capability limits.

13. Milestone 10: Ephemeral AI Computer integration

M10 objective

Make Ephemeral AI FS the default Computer filesystem while retaining DOFS only as an explicitly selected, isolated benchmark comparison.

M10 checklist

M10 acceptance criteria

The full path passes:

workspace.fs
  -> authenticated replication
  -> computerd
  -> real FUSE
  -> shell and Git
  -> pull
  -> branch publication
  -> restart and reconnect
  -> collection and verification

Computer's default path contains transport, execution, workspace, and user-facing integration logic but no filesystem persistence semantics.
The Ephemeral AI FS wiring remains within 100 net-new Computer production lines, excluding tests, benchmark harnesses, generated code, and the optional DOFS comparison adapter.
Omitted configuration selects Ephemeral AI FS; DOFS never becomes an automatic fallback.
Ephemeral AI FS and DOFS schemas, diagnostics, and databases remain isolated.
Durable Object restart, container restart, FUSE remount, branch reconnect, and final integrity verification pass.
The 60-second end-to-end smoke profile passes.

14. Milestone status template

Use this block when recording a milestone exit:

### Milestone N exit

- Commit:
- Date:
- Checklist complete: yes/no
- Correctness artifact:
- Benchmark artifact:
- Smoke duration and operation counts:
- Resource high-water:
- Known deviations:
- Approved to begin next milestone: yes/no

Uh oh!

FilesExpand file tree

implementation-plan.md

Latest commit

History