Skip to content

One block = one serial append and one fsync: accept a whole block of permanent entries as a unit #98

Description

@PaulSnow

From accumulate (issue-4205-lead), Paul 2026-09-26: "Building a block should not make writes while building the block. All the entries to be written to perm should be done in batch because all the entries are going to be written to a file in a serial block. That is one write."

Today

accumulate's executor already holds a block in memory and commits once. The adapter (pkg/database/keyvalue/bcdb/database.go writeThrough) then calls the store once per changed key and finally SealBlock. The permanent layer is spread across shard segments, so one block becomes many file writes and one fsync per shard. #84/PR 85 made the seal concurrent across shards (253 to 50 ms per boundary), but it is still many fsyncs per block, and the rewritable (dynamic) keys live in a separate layer.

Ask

An API that takes a whole block at once, write-once entries plus the block's rewritable keys, and makes it durable with:

  • one serial append of the permanent entries, in order, to one segment file;
  • the rewritable keys in the same block, indexed by position;
  • one fsync.
    The block is the durability unit. A block is either wholly on disk or not at all.

Also

  • The read window should count blocks, not commits. A writer that commits many small units between blocks (a node joining, a backfill) currently pushes data written one block ago out of the window, and ordinary reads then report it absent. accumulate patched two read sites for this (#4446, #4405 notes).
  • accumulate will pass its consensus checkpoint and certificates inside the same block unit, so they become exactly as durable as the block.

accumulate side: see the matching accumulate issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions