Skip to content

Add internal/slots: fixed-size aligned record slots - #4

Merged
suhailopensource merged 1 commit into
mainfrom
step/0x05-alignment
Aug 31, 2026
Merged

suhailopensource merged 1 commit into
mainfrom
step/0x05-alignment

Conversation

@suhailopensource

Copy link
Copy Markdown
Member

Summary

Adds internal/slots: fixed-size, aligned record slots inside a page, so any
slot is reachable by arithmetic rather than by scanning.

func (l Layout) Offset(i int) int { return int(l.base) + i*int(l.slotSize) }

That line is the feature. Everything else in the package exists to make it safe
and to account for what it costs.

Background

page.Append copies bytes into a page and advances FreeStart. It has no way
to find item n again: records are unknown lengths at unknown positions, so
reaching the 200th means decoding the 199 before it.

Measured in the same 4096-byte page:

fixed slot, first        4-6 ns
fixed slot, 200th        4-5 ns      flat
packed record, first     6-7 ns
packed record, 200th     ~990 ns     about 200x slower

Ranges rather than single figures deliberately — see the methodology note under
Testing.

Design

Alignment is applied to the stride, not to each position

Align is called exactly twice, and neither call is per-slot:

base := Align(page.HeaderSize, align)   // where slot 0 starts
slot := Align(recordSize, align)        // how big every slot is

Aligning the step size once makes every subsequent position aligned for free:

base 16, slot 13:  16 ok, 29 misaligned, 42 misaligned, 55 misaligned ...
base 16, slot 16:  16 ok, 32 ok, 48 ok, 64 ok, 80 ok ...

Aligning positions individually would not work: Align(base + i*recordSize, 8)
can round two different slots to the same place, or make a slot overlap its
neighbour. An aligned start plus an aligned stride is what lets Offset stay
pure arithmetic.

Why align at all

Memory is fetched in words, not bytes, so an 8-byte value starting at offset 6
straddles two words and costs two fetches instead of one. The same shape appears
one scale up: a record straddling two 4096-byte pages costs two page reads.

Go's compiler already does this to every struct, which makes it easy to show:

bool, int64, bool, int64   ->  32 bytes
int64, int64, bool, bool   ->  24 bytes

Same fields, 8 bytes saved by ordering largest-first.

What it costs, exposed rather than hidden

record  slot  pad  per page  waste
1       8     7    510       87.2%
8       8     0    510        0.0%
13      16    3    255       18.7%
16      16    0    255        0.0%
17      24    7    170       29.1%

PadPerSlot(), Tail() and Waste() are public so a caller can see the bill
before choosing a record size. A 1-byte record padded to 8 throws away 87% of
the page, which is the honest limit of the design.

Layout is valid by construction

Layout's fields are unexported and NewLayout is the only constructor. This
is not stylistic — see the defects below. Validation happens once, at
construction; the per-access path checks only what can still vary: the slot
index, and whether the slot ends inside this page.

An earlier draft called a Validate() method on every Get/Put. That checked
constants in the hottest path in the package and cost 2.3x, which defeated the
purpose of the step. Making a bad Layout impossible to build is both faster and
stronger than detecting one repeatedly.

Padding is zeroed on write

Put zeroes the bytes between the record and the end of its slot. Without it,
padding retains whatever the page held before — fragments of deleted records,
readable by anything scanning raw bytes, and enough to make two logically
identical pages differ byte for byte. TestPutZeroesThePadding fills a slot
with 0xFF first and checks it comes back clean.

Defects found and fixed during review

Four, all in this change, found by probing rather than by re-reading.

A panic writing past the page. An inconsistent Layout made Put compute a
slice bound of [:8024] on a 4096-byte page. A panic takes the process down,
which is the worst available outcome for a storage engine.

A silent over-read. Get with RecordSize=64 and SlotSize=8 returned 64
bytes out of an 8-byte slot — reading into neighbouring slots — and reported
success.

Both had the same cause: exported fields let callers build an inconsistent
Layout, and Put/Get trusted it. Unexporting the fields turned this from a
runtime concern into a compile error; two of the probe tests no longer build,
which is the intended outcome.

NewLayout(-5, 8) succeeded, producing RecordSize = 65531 — a negative
int wrapping through uint16. That Layout then reproduced the panic above by a
different route. Now rejected explicitly.

Align(-5, 8) returned 8 and Padding(-5, 8) returned 13. Non-positive
input now passes through unchanged, so NewLayout's existing <= 0 check
catches it.

Testing

53 tests across the module. go vet, gofmt -l and go test -race ./...
clean.

New in this change:

  • TestLastSlotEndsInsideThePage — for every record size from 1 to 512, checks
    the last slot ends inside the page and that Tail() and Waste() are never
    negative
  • TestZeroLayoutErrorsRatherThanPanicking, TestShortPageErrorsRatherThanPanicking
  • TestNegativeOrZeroRecordSizeIsRejected, TestAlignIgnoresNonPositiveInput
  • FuzzPutGetNeverPanics — 15s clean; guards the panic above

Benchmark methodology

Three runs of each benchmark:

fixed slot, first     3.97 / 4.36 / 5.68 ns
fixed slot, 200th     4.22 / 4.44 / 4.51 ns
packed, first         5.99 / 6.51 / 6.77 ns
packed, 200th          987 /  991 /  991 ns

The fixed-slot figures vary by about 1.7 ns run to run, so first and 200th
overlap. That overlap is the result — access cost does not depend on the
index. Quoting two significant digits from this benchmark would be misleading,
so the README and demo quote ranges.

The hardening costs roughly 2 ns per access. Against 990 ns that is irrelevant,
and real access will go through a buffer pool with locking that dwarfs it. Not
optimised further: there is no workload to optimise for yet, and the measurement
is not precise enough to verify an improvement.

Not in scope

  • Variable-length records. Fixed slots require a fixed record size. Rows of
    varying length need a slot directory — offsets stored in the page rather than
    computed — which belongs with the heap file. Measured on three sample rows,
    fixed slots left 53% of the space empty, which is what justifies that work.
  • Deletion and free-slot reuse. Put overwrites; there is no free list and
    no tombstone.
  • Choosing the record size. A caller supplies it. Deriving it from a schema
    belongs to the tuple layer.
  • Cache-line alignment. 8 bytes matches word size, not the 64-byte cache
    line. Worth revisiting once something is hot enough to measure.

page.Append can store bytes but cannot find item n again, so reaching the 200th means decoding the 199 before it. Fixed-size slots make any slot reachable by one multiplication: about 4-5ns whichever slot, against ~990ns for the packed 200th. Align is applied to the stride rather than to each position, because an aligned start plus an aligned step size makes every later position aligned for free, which is what keeps Offset pure arithmetic. Layout's fields are unexported and NewLayout is the only constructor, so validation happens once at construction rather than on every access. Padding is zeroed on write so deleted fragments cannot survive in the gaps, and PadPerSlot/Tail/Waste expose the cost rather than hiding it.

Review of this change found four defects, all fixed with regression tests: Put could panic computing a slice bound past the end of the page, Get could silently read 64 bytes out of an 8-byte slot, NewLayout accepted a negative record size that wrapped through uint16 to 65531, and Align returned nonsense for non-positive input. The first two came from exported fields allowing an inconsistent Layout; unexporting them turned the bug class into a compile error.
@suhailopensource
suhailopensource merged commit 75f62a3 into main Aug 31, 2026
1 check passed
@suhailopensource
suhailopensource deleted the step/0x05-alignment branch August 31, 2026 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant