[suhailopensource]

Summary

Adds internal/codec, the layer that converts Go values to bytes and back.

It is split into two encoders with different goals, because a single encoding
cannot satisfy both:

file	goal	technique
value.go	minimise size	varint, zigzag, length prefixing
key.go	preserve sort order under bytes.Compare	fixed-width big-endian, sign-flipped
varint.go	shared primitive	LEB128 + zigzag

Background

internal/page can store bytes but nothing marks where one stored item ends:

p.Append([]byte("scute-db"))
p.Append([]byte("hello"))
→ "scute-dbhello"

Both items are unrecoverable. Separately, the B+Tree will compare keys using
bytes.Compare and nothing else — it has no type information — so the key
encoding is what determines whether range queries are correct.

Encoding is a file-format decision. Unlike a cache or an index implementation,
it cannot be changed after data exists without rewriting every file, so it is
settled before the tree is built rather than after.

Design

Why two encoders

The two requirements are in direct conflict.

Compactness wants the low bits first, so a small number can terminate early.
Sort-correctness wants the high bits first, so the first byte that differs is
the most significant one. Varint encoding demonstrates the conflict:

255 → FF 01
256 → 80 02        FF > 80, so 255 sorts above 256

Compact and order-preserving are mutually exclusive properties of a single
encoding, so the package provides both and the caller picks by context: index
keys use key.go, everything stored inside a record uses value.go.

Value encoding

Length prefixing (AppendBytes, AppendString) — length first, then
payload:

08 73 63 75 74 65 2D 64 62   05 68 65 6C 6C 6F
^^ 8 bytes                   ^^ 5 bytes

Chosen over delimiters because a delimiter requires escaping any occurrence in
the data, and because a length prefix allows skipping an item without reading
it — a prerequisite for the slot directory the heap file will need.

Varints (AppendUvarint) — 7 payload bits per byte plus a continuation
flag:

300 → AC 02
      AC = 1 0101100    continue, low group = 44
      02 = 0 0000010    terminate, next group = 2
      44 + 2 x 128 = 300

Most integers a storage engine writes are bookkeeping — lengths, counts, slot
indices — and are small. A fixed 8 bytes for values under 128 costs 7 wasted
bytes per occurrence; a row with three variable-length fields wastes 21 bytes,
which at scale is both disk and read amplification, since fewer rows fit in a
4 KB page.

Cost: the maximum uint64 takes 10 bytes rather than 8.

Zigzag (zigzag, unzigzag) — required because two's complement makes -1
all ones, which a varint encodes in the maximum 10 bytes. Zigzag maps signed to
unsigned so that magnitude, not sign, determines size:

signed:   -3  -2  -1   0   1   2   3
unsigned:  5   3   1   0   2   4   6

func zigzag(v int64) uint64 { return uint64(v<<1) ^ uint64(v>>63) }

v << 1 doubles the value and frees the low bit. v >> 63 is an arithmetic
shift on a signed integer, producing 0 for non-negative input and all-ones for
negative — a branchless sign mask. XOR with all-ones inverts, giving 2v for
non-negative and 2|v| - 1 for negative. -1 drops from 10 bytes to 1.

Chosen over a separate sign flag because the result is an ordinary uint64 and
reuses the existing varint encoder unchanged, with no branch in the hot path.

Key encoding

Big-endian (AppendKeyUint64) — bytes.Compare is left-to-right,
first-difference-wins, so the most significant byte must come first. Sorting
[1 2 255 256 300] by encoded bytes:

big-endian     [1 2 255 256 300]
little-endian  [256 1 2 300 255]
varint         [1 2 256 300 255]

Sign flip (AppendKeyInt64) — big-endian alone is insufficient for signed
values. Positives begin 00–7F, negatives begin 80–FF, so the entire
positive range sorts below the entire negative range:

sorted by raw two's complement:   [0 1 2 100  -100 -2 -1]

Each half is internally ordered; only the halves are transposed. XOR-ing the
sign bit swaps them:

uint64(v) ^ (1 << 63)

value   raw                       encoded
-1      FF FF FF FF FF FF FF FF   7F FF FF FF FF FF FF FF
 0      00 00 00 00 00 00 00 00   80 00 00 00 00 00 00 00
 1      00 00 00 00 00 00 00 01   80 00 00 00 00 00 00 01

Without this, a range scan over -2..2 starts at FF FF ... FE and ends at
00 00 ... 02 — the start is above the end, so the scan terminates immediately
and returns zero rows with no error. This is a silent-wrong-answer failure mode,
which is why it is fixed before any index exists rather than found later.

Floats (AppendKeyFloat64) — same approach with an extra case: invert the
sign bit if positive, invert all bits if negative, since negative floats are
ordered in reverse by their bit pattern.

This is the standard technique; CockroachDB, FoundationDB and TiKV encode keys
the same way.

Testing

Table tests cover specific values, boundaries and truncation. Three fuzz targets
cover the general claims:

• FuzzValueRoundTrip — encode/decode is lossless for all types, and the
  decoder consumes exactly the bytes the encoder produced
• FuzzKeyOrderingInt64 — for all a, b: numeric ordering equals
  bytes.Compare ordering. ~3.9M pairs in 8s, no counterexample
• FuzzDecoderNeverPanics — arbitrary input produces an error, never a panic.
  Required because corrupt pages will reach the decoder in production and must
  surface as errors rather than crashes

TestVarintMatchesStdlib asserts our varint output is byte-identical to
encoding/binary across boundary values.

--- PASS: TestVarintMatchesStdlib
--- PASS: TestValueRoundTrip
--- PASS: TestTruncatedInputIsAnError
--- PASS: TestKeyOrderingUint64
--- PASS: TestKeyOrderingInt64
--- PASS: TestKeyOrderingFloat64
--- PASS: TestKeyRoundTrip
--- PASS: TestVarintIsNotOrderPreserving
--- PASS: TestLittleEndianBreaksKeyOrdering

fuzz: elapsed: 8s, execs: 3903441 (467462/sec), new interesting: 1 (total: 7)

go vet, gofmt -l, and go test -race ./... are clean.

Behaviour change, via make demo-encode:

before:  "scute-dbhello"                                     unrecoverable
after:   08 73 63 75 74 65 2D 64 62 05 68 65 6C 6C 6F
         item 1: len=8 "scute-db"   item 2: len=5 "hello"

Size

varint.go       58
value.go        69
key.go          55
codec_test.go  180
fuzz_test.go    92
               ───
               454     182 implementation, 272 tests

Not in scope

• Record encoding. This layer encodes single values. Field ordering, schema
  and null bitmaps belong to the tuple layer and are not addressed here.
• Null representation. Nothing yet distinguishes an absent value from a zero
  value; that requires a null bitmap in the record header.
• Composite keys. AppendKeyString is a passthrough, which is correct for
  single-column keys but ambiguous for composites: ("a","bc") and ("ab","c")
  both encode to abc. An escaping scheme is needed before multi-column indexes
  land.
• Compression. No prefix or block compression.

Unrelated fix included

The README justified the 4096-byte page size as matching "the machine's
virtual-memory page size". That holds on x86-64 but not on Apple Silicon, which
uses 16 KB VM pages. The relevant alignment is the filesystem block size (4096
on APFS and ext4). Corrected.