[suhailopensource]

Index storage manager

Why this is needed

After 0x09 a B+Tree node can be written to disk as bytes. What a file still
could not answer is the first question any reader asks after a restart: where
does the tree start? Nothing is in memory. A file holding millions of perfectly
formatted keys is useless if nothing says which page is the root.

This adds internal/pagestore, which owns the file: it hands pages out, takes
them back, grows the file, records the root, and guarantees that a crash at any
moment — a kill -9, a failed fsync, or a power cut — leaves a file that
reopens to the last committed state.

What is in the change

file	lines	what
internal/pagestore/pagestore.go	870	the store: open and recovery, allocation, free list, checkpoints, verification
internal/pagestore/describe.go	124	decoders for the meta page and free list, for humans
internal/pagestore/*_test.go	6,511	94 tests and 4 fuzzers: a fake disk that models the OS page cache, crash and fault injection, hostile files, an independent oracle, and a real kill -9
internal/page/page.go	+33	Grow(from, to), KindFreeList, PageCount clamped at 2³²
internal/fileio/fileio.go	+70	SyncDir, SyncParent and OpenExisting, so a file's name is durable and a missing file is an error
internal/fileio/fileio_test.go	230	the first tests for fileio: the directory-fsync error policy, symlink resolution, the hop limit
internal/page/grow_test.go	267	Grow, PageCount, and Read against a fake file that records every write
cmd/scutedb-demo/pagestore.go	326	the walkthrough behind make demo-pagestore, including twenty real kill -9s

pagestore imports core, fileio and page. Nothing imports it yet: the
B+Tree still lives in memory, and wiring it to this store is the next step.

The meta page

Page 0 is the only place a reader can look without being told where to look:

offset    bytes                      meaning
0..15     page header                id 0, kind meta
16..23    53 43 55 54 45 44 42 00    magic "SCUTEDB", NUL-terminated
24..27    00 00 00 01                format version
28..31    00 00 10 00                page size, 4096
32..35                               root page
36..39                               first free-list page
40..43                               free page count
44..47                               next page to hand out
48..51                               pages the file holds
52..55                               checkpoint generation

Page 0 also means "no page": a root of 0 is "no root yet", and a free-list chain
ends at 0. That is safe only because page 0 can never be a root or a free page.

How the code is organised

Opening.

function	does
Create(path)	makes a new store; refuses a file that already holds data, and says whether that data is a store
Open(path)	opens an existing store, or initialises a blank one; never creates a missing file; always fsyncs the directory that really holds it
openFile(f)	the shared path: a blank file is an interrupted create, anything else goes to load
isBlank(f)	true for an empty file, or one that is all zeros and no bigger than one chunk
create(pf)	writes the first chunk of zeros and the first meta page
load(pf)	validates everything, builds live state, then rewrites the meta it read and fsyncs it

Allocating and freeing.

function	does
Allocate(kind)	pops a free page, or hands out the next new one, growing the file by a chunk when needed
Free(id)	puts the page on the pending list; its bytes are not touched
SetRoot(id) / Root()	the root, which the next checkpoint records in the meta page
Read(id) / Write(p)	page I/O, refused for pages that are free, pending, or hold the free list

Committing.

function	does
Checkpoint()	makes the current state durable, in a fixed order (below)
Sync()	fsyncs written pages without committing them
Close()	checkpoints if anything changed, then closes
Discard()	closes without checkpointing — how the tests simulate a crash

Checking.

function	does
Verify()	reads the durable meta and free list back from disk, validates them, and compares them with the live state
Durable() / Meta()	the last committed meta, and the live view
FreePages() / PendingPages() / ListPages()	the three kinds of page that are not in use

The flow of a checkpoint

Checkpoint()
  checkpoint()
    1. pf.Sync()                    every page written since the last checkpoint
    2. pick free-list pages          from pages free in the DURABLE state, or by growing
                                     — never pending pages, never the current list pages
    3. write the list                old list pages + free pages + pending pages
    4. pf.Sync()                     the list is on disk
    5. pf.Write(encodeMeta(next))    the new root, list head, and counts
    6. pf.Sync()                     the commit point
    then, in memory only:
       free = everything just listed, pending = empty, list pages = the new ones

Nothing that the previous checkpoint references is overwritten before step 6
completes. If any fsync fails, the store is poisoned and refuses every further
call until the file is reopened.

The flow of Open

Open(path)
  fileio.OpenExisting(path)                            no O_CREATE: a missing file is an error
  openFile(f)
    isBlank(f)?          -> create(pf)                  an interrupted Create
    else                 -> load(pf)
                              decodeMeta               magic, then version, then page size,
                                                       then kind and id
                              checkMeta                counts are consistent with the file
                              readFreeList             follow the chain, checking every page
                              checkFreeSet             no duplicates, root not free
                              verifyLive               live state matches durable state
                              write meta + fsync       RECOVERY: what we read is now on disk
  fileio.SyncParent(path)                              the file's name is now on disk too,
                                                       in the directory the kernel really used

Why a freed page has to wait

The first version wrote a free-list link into each freed page, so the free list
was a chain through the free pages. Two crash tests broke it. Freeing the old root
after checkpointing a new one, then crashing, destroyed the last committed tree:
the meta page still pointed at the old root, but Free had already overwritten
it. And reusing a free page broke the chain the durable free list depended on.

The rule that fixes both: until a new checkpoint is durable, nothing the previous
one references may be touched. So Free only records the page as pending, and
the free list lives in dedicated pages taken from somewhere the durable state does
not look.

The page cache is not the disk

Most of what review found came from one confusion: a read returns what the
operating system holds, not necessarily what the drive holds.

trusted as durable	could really be on disk	fix
the meta page Open read	an older meta page	load rewrites and fsyncs it before handing out any page
the file's size	a shorter file	Grow(from, to) writes zeros for exactly the new pages, from the store's own count
the file's name	no directory entry at all	Open and Create fsync the directory that really holds the file, following symlinks

The first is the subtle one. If a checkpoint's final fsync fails, the new meta can
sit in the cache but not on disk. Without recovery, the next Open would reuse
pages the on-disk checkpoint still needs, and a later power cut would revert to
that checkpoint — now overwritten.

The directory row took three attempts. Fsyncing the directory named in the path's
text misses the real one behind a symlink. Resolving the path with
filepath.EvalSymlinks found the right place but built one long string that fails
past PATH_MAX, and a filepath.Dir "lexical parent" tidied away .. before
following symlinks. The version that holds splits off the last component without
cleaning, opens the prefix so the kernel resolves it, and follows only a
final-component symlink itself. fileio.SyncParent is that function.

A missing database is an error, not an empty database. Open used to create a
missing file, which is what turned any lost directory entry into silent data loss:
the next Open quietly returned an empty store. Now only Create makes files.

A consequence worth knowing: after a failed checkpoint the outcome is unknown.
If only the final fsync failed, the next Open may make the new state durable.
Reopen and read the root to find out.

Verifying it

A real kill -9, twenty times. The demo's child process loops forever —
allocate a root, stamp it with the checkpoint number it will become, free the old
root, checkpoint — and is killed at an arbitrary moment. All twenty reopen to a
consistent checkpoint whose root holds the matching stamp.

A fake disk for everything a demo cannot stage. memFile keeps three copies of
the truth — what reads see, what is dirty, and what is on the drive — and can
simulate a power cut that keeps any subset of dirty writes, a failed fsync that
behaves like Linux after 2018, a process restart that keeps the page cache, and
the cache later evicting pages that were never written.

A crash fuzzer. FuzzCrashAnywhereKeepsTheLastCheckpoint runs random
sequences of allocate, free, set-root, checkpoint, restart, crash, power cut,
failed checkpoint and failed sync. After every step it checks that every page the
last checkpoint referenced holds its original bytes and that every page is in
exactly one state. It ran 2,436,886 executions.

Independent review, round by round. Reviewers worked in private copies of the
repository and had to prove every finding with a failing test that a separate
skeptic then reproduced. Round two included a reviewer who wrote an independent
crash fuzzer from scratch; it found nothing. Round four found that round three's
own path fix had introduced two ways to make a valid file unopenable; round four's
generalist fuzzed allocation, checkpointing and recovery with new fault types for
eleven minutes and found nothing there.

bug	severity	found by
Free overwrote pages the last checkpoint referenced	data loss	crash tests, first draft
Open trusted a meta page that existed only in the cache	data loss	review
a failed Sync did not poison, so the next checkpoint committed lost pages	data loss	review
page ids could wrap past 2³² into the meta page	data loss	review
a root that was also a free-list page was accepted	corruption	review
a crash during Create left a file that could never be opened	unrecoverable	review
Verify never compared live state with the durable free list	blind spot	review
growth trusted the file size the cache reported	corruption	the fuzzer
the directory was fsynced only when Open created the file	data loss	review
running out of page ids reported "an fsync failed" and hid the cause	wrong error	review
the directory fsync followed the path's text, not its symlink	data loss	review
that fix built the resolved path as one string, failing past PATH_MAX	unopenable	review
that fix also tidied away .. before following symlinks	unopenable	review
a failed Allocate that had grown the file left the store reporting the bigger size	wrong state	fault-injection tests
a failed Write still marked the store as needing a checkpoint	wrong state	fault-injection tests

How strong the tests are. Coverage only says a line ran. Mutation testing is
stronger: a script breaks the code on purpose, one change at a time — 563 changes
in all — and checks that some test fails each time.

Breaking each fix by hand caught 28 of 28, but those were the spots I had just
fixed. The full sweep was less kind, and drove a round of new tests:

                         before hardening    after hardening
coverage, storage code   79.7%               93.5%
mutation score           68.9%               93.1%
breaks nobody caught     174                 39
tests                    181                 238

The new tests are: 81 hostile rewrites of a real store's bytes plus a fuzzer that
fed Open random bytes 20.7 million times; a fake file that fails or tears the
Nth read, write, sync or size call at 251 points inside every operation; an
independent crash fuzzer that decodes the simulated disk with its own decoder;
each of Open's two layers of checks tested on its own; and a real kill -9
test that runs in CI.

The sweep also exposed weak tests of my own: six symlink tests passed if Open
failed for any reason, and a Discard test was rescued by its own Close. Both
now check the exact cause. The 39 remaining survivors were each checked by hand:
17 make no observable difference, 15 are caught by a second layer of checks that
is tested separately, and 7 need an OS failure that cannot be produced on demand.

Known limits

• A single meta page assumes a 56-byte write is atomic within a sector. Two
  alternating meta pages with checksums remove that; checksums arrive in 0x13.
• A path whose final component is a symlink with a relative target cannot be opened
  if that target joined to the link's directory passes PATH_MAX (1024 on macOS);
  avoiding it needs openat, which Go's standard library lacks on macOS.
• A full disk with an empty free list cannot checkpoint, because the new free list
  needs a page nothing durable references.
• A directory fsync counts as done when the filesystem cannot do one at all
  (EINVAL, ENOTSUP), as SQLite does. Every other error fails.
• No file lock yet, so two processes can open one store; that is 0x0C.
• A checkpoint can write one empty trailing free-list page. It is valid, and it is
  recycled at the next checkpoint.
• Nothing is thread-safe yet, and the B+Tree does not use this store yet.

Verification

go build ./...     ok
go vet ./...       clean
gofmt -l .         0 files
go test ./...      ok, 7 packages
go test -race      clean

238 tests, 14 fuzz targets. Run the walkthrough with
make demo-pagestore.