[RFC] diff-hunks: a read/write cache of diff hunk coordinates

[mmontalbo]

Blame and git log --stat recover hunk coordinates by diffing blob
pairs: blame diffs each line's blob against its parent's, and --stat
diffs every changed file at every commit shown. The results are not
stored, so the same pairs are decompressed and recomputed on every run.

This series adds a cache of those coordinates under
$GIT_DIR/objects. It is a read/write cache: reading is on by default
(core.diffHunks), while writing is off by default and enabled per
run. A git diff or git log that produces a --stat fills the cache
as a side effect, but only when writing is enabled, so ordinary reads
never modify the repository. When the cache is absent, a lookup misses,
or an entry was recorded under different settings, the command computes
the diff as before. Using the cache does not change command output (a
malformed store may still emit a diagnostic on stderr).

Writing is enabled per invocation by GIT_DIFF_HUNKS_WRITE, or
persistently by the diffHunks.write config, with the environment
variable winning. A repository owner warms the cache by running the
diff-producing commands they care about with writing on, for example:

GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null

A --stat walk records the hunks at both zero context (what blame
reads) and the configured context (what --stat sums), so one warming
walk serves both readers.

The cache is two tiers: a consolidated base and a small overlay of the
pairs recorded since the last fold. A warm appends only to the overlay
and does not rewrite the base, so its cost is proportional to the pairs
it adds; a lookup checks both tiers, and because entries are
content-keyed either tier's hit is identical. git diff-hunks compact
folds the overlay into the base.

An entry is keyed by the two blobs diffed and the settings they were
diffed under: struct diff_hunks_settings, the subset of diff_options
that changes the hunks (xdl_opts and the context length). A lookup
made under other settings misses. Context is part of the key because a
zero context length triggers trim_common_tail, which can produce
different hunks than a nonzero context does; keying the two apart lets
one cache serve blame (zero context) and --stat (its context) from
the same pair. Identical hunk blocks are stored once and shared across
keys.

The writer fsyncs (a new diff-hunks core.fsync component) and
commits the file atomically; readers bounds-check every record but do
not re-verify the checksum on load, and git diff-hunks verify and
git fsck check it, as commit-graph and the multi-pack-index do.
git diff-hunks itself only inspects (verify) and manages (compact,
clear) the files.

Measured with hyperfine. The warm is a full cold build of the overlay,
GIT_DIFF_HUNKS_WRITE=1 git log --all --stat; compact folds it into the
base; the read rows compare the warmed store to core.diffHunks=false:

git.git (81,530 commits)
  warm log --all --stat     24.1 s   overlay 38 MB, verify 54 ms
  compact overlay -> base    0.24 s   base 38 MB
  log --stat -5000          1.91x    (1.45 s -> 0.76 s)
  blame diff.c              1.27x    (513 ms -> 405 ms)

linux (1,462,831 commits)
  warm log --all --stat      644 s   overlay 446 MB, verify 0.60 s
  compact overlay -> base    3.3 s   base 446 MB
  log --stat -5000          1.97x    (2.22 s -> 1.13 s)
  blame kernel/sched/core.c 1.56x   (1.40 s -> 0.89 s)

The log --stat speedup is ~1.9x at both scales; the blame speedup is
file-dependent under a first-parent warm (some files hit few pairs), as
the coverage note below explains.

• Patch 1: git fsck decouples its exit status from the error-class
  bitmask, so a class added past the eighth does not overflow the
  eight-bit exit status. No behavior change; preparation for patch 5.
• Patch 2: the two-tier on-disk format, the reader/writer library
  including the write gate (GIT_DIFF_HUNKS_WRITE / diffHunks.write),
  and the git diff-hunks command (verify, compact, clear).
  The command name intends to follow the git commit-graph convention,
  but it is possible diff-hunks could be confused with a command that
  is related to diffing some specific content instead. As such, other
  suggestions for names are highly welcome.
• Patch 3: builtin_diffstat() reads and records hunks; git diff,
  git log, git show, and git diff-tree with --stat, --numstat,
  or --shortstat consult the repository's store and, when writing is
  enabled, attach a writer.
• Patch 4: blame reads precomputed hunks.
• Patch 5: git fsck runs the store's verify; tests.

The main limitation is blame coverage. Blame hits are partial and
repo-dependent (git.git 53-57%, linux 74-88%) under a first-parent
--stat walk: a blame that follows a merge's second parent recomputes
those pairs. Because the cache is filled by whatever commands the owner
runs, coverage can be raised by warming with a walk that diffs those
pairs (for example log --all -m --stat), at the cost of a larger
cache. A repository that configures a non-default diff.algorithm
records under it while blame keys without it, so blame there is not
served until it adopts the algorithm.

A warm appends to the overlay and does not rewrite the base, so its cost
is proportional to the pairs it adds rather than to the whole cache.
git diff-hunks compact folds the overlay back into the base; it is a
separate O(cache) step, so that cost stays off the warm path. A
deployment that warms continuously would schedule compaction, for
example via a maintenance task, to bound the overlay size. The warm
figures above are a full cold build; an incremental warm is proportional
to the pairs it newly records.

Beyond this series, one thing worth thinking about is whether the store
could hold hunk content, not only coordinates. The coordinates serve
blame and --stat; a patch reader (git log -p, git show) needs the
changed lines, which the store does not hold. A content-addressed
payload of those lines beside the index might serve patch output without
re-diffing, though storing it for all of history is roughly the size of
the repository's diff text. A bounded variant, perhaps only recent
diffs evicted at compaction, could keep it small, and the two tiers seem
to accommodate that. This is a direction to consider, not something the
series pursues.

Another consumer the coordinate cache could serve is git log -L, which
maps line ranges through the hunks of the same blob-pair diffs. The
consult path blame uses (a small diff_hunks_emit() helper) is written to
be shared, so wiring it would be a follow-up, once line-log honors
diff.indentHeuristic so its diffs match a default warm.