[RFC] diff-hunks: a read/write cache of diff hunk coordinates

.gitignore

@@ -56,6 +56,7 @@
 /git-diagnose
 /git-diff
 /git-diff-files
+/git-diff-hunks
 /git-diff-index
 /git-diff-pairs
 /git-diff-tree

Documentation/Makefile

@@ -33,6 +33,7 @@ MAN5_TXT += gitattributes.adoc
 MAN5_TXT += gitformat-bundle.adoc
 MAN5_TXT += gitformat-chunk.adoc
 MAN5_TXT += gitformat-commit-graph.adoc
+MAN5_TXT += gitformat-diff-hunks.adoc
 MAN5_TXT += gitformat-index.adoc
 MAN5_TXT += gitformat-loose.adoc
 MAN5_TXT += gitformat-pack.adoc

Documentation/config.adoc

@@ -419,6 +419,8 @@ include::config/credential.adoc[]
 
 include::config/diff.adoc[]
 
+include::config/diff-hunks.adoc[]
+
 include::config/difftool.adoc[]
 
 include::config/extensions.adoc[]

Documentation/config/core.adoc

@@ -670,12 +670,13 @@ but risks losing recent work in the event of an unclean system shutdown.
 * `pack` hardens objects added to the repo in packfile form.
 * `pack-metadata` hardens packfile bitmaps and indexes.
 * `commit-graph` hardens the commit-graph file.
+* `diff-hunks` hardens the diff-hunks store.
 * `index` hardens the index when it is modified.
 * `objects` is an aggregate option that is equivalent to
   `loose-object,pack`.
 * `reference` hardens references modified in the repo.
 * `derived-metadata` is an aggregate option that is equivalent to
-  `pack-metadata,commit-graph`.
+  `pack-metadata,commit-graph,diff-hunks`.
 * `committed` is an aggregate option that is currently equivalent to
   `objects`. This mode sacrifices some performance to ensure that work
   that is committed to the repository with `git commit` or similar commands
@@ -750,6 +751,13 @@ core.commitGraph::
 	to parse the graph structure of commits. Defaults to true. See
 	linkgit:git-commit-graph[1] for more information.
 
+core.diffHunks::
+	If true, then Git will consult the diff-hunks store (if it
+	exists) to skip recomputing diff hunk coordinates in commands
+	such as `git log --stat` and linkgit:git-blame[1]. This controls
+	only reading; writing the store is controlled by `diffHunks.write`.
+	See linkgit:git-diff-hunks[1] for more information. Defaults to true.
+
 core.useReplaceRefs::
 	If set to `false`, behave as if the `--no-replace-objects`
 	option was given on the command line. See linkgit:git[1] and

Documentation/config/diff-hunks.adoc

@@ -0,0 +1,7 @@
+diffHunks.write::
+	If true, diff-producing commands (`git diff` and `git log` with a
+	`--stat`, `--numstat`, or `--shortstat` format) write the hunks
+	they compute to the diff-hunks store, filling it as a side effect.
+	The `GIT_DIFF_HUNKS_WRITE` environment variable overrides this for
+	a single invocation. Reading the store is controlled separately by
+	`core.diffHunks`. See linkgit:git-diff-hunks[1]. Defaults to false.

Documentation/git-diff-hunks.adoc

@@ -0,0 +1,134 @@
+git-diff-hunks(1)
+=================
+
+NAME
+----
+git-diff-hunks - Inspect and manage the precomputed diff hunk store
+
+SYNOPSIS
+--------
+[synopsis]
+git diff-hunks verify
+git diff-hunks compact
+git diff-hunks clear
+
+DESCRIPTION
+-----------
+
+The diff hunk store is a cache of diff hunk coordinates, so that commands
+which need them, such as linkgit:git-blame[1] and `git log` and `git diff`
+with the `--stat`, `--numstat`, and `--shortstat` formats, can skip
+decompressing blobs and running the diff algorithm.
+
+The store has two tiers under `$GIT_DIR/objects`: a consolidated base,
+`diff-hunks`, and a small overlay, `diff-hunks-overlay`, holding the pairs
+recorded since the last fold. Reading is enabled by default; writing is
+off by default. A `git diff`, `git log`, `git show`, or `git diff-tree`
+that produces one of the stat formats fills the store as a side effect,
+but only when writing is enabled for that run (see "WARMING THE STORE"
+below), so ordinary reads never modify the repository. A lookup checks
+both tiers; when neither has the pair, the store holds a different object
+hash, or the file is unreadable, the consumer falls back to computing the
+diff. A store only speeds up these commands; it never changes their
+output.
+
+`git diff-hunks` itself only inspects and manages the file. See
+linkgit:gitformat-diff-hunks[5] for the file format.
+
+WARMING THE STORE
+-----------------
+
+The store is filled by running ordinary commands with writing enabled.
+Turn writing on for a single invocation with the `GIT_DIFF_HUNKS_WRITE`
+environment variable, or persistently with the `diffHunks.write`
+configuration; the environment variable takes precedence. A repository
+owner warms the store by running the diff-producing commands they care
+about with writing on, for example:
+
+	GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
+
+A `--stat` walk records the hunks at both zero context (what
+linkgit:git-blame[1] reads) and the configured context (what `--stat`
+sums), so a single warming walk serves both. A warm appends only to the
+overlay tier and never rewrites the base, so its cost is proportional to
+the pairs it adds; a later warm adds newly computed pairs without
+discarding what earlier warms recorded.
+Run `git diff-hunks compact` to fold the overlay into the base once the
+overlay has grown.
+
+COMMANDS
+--------
+
+`verify`::
+	Check the integrity of the store: the trailing hash checksum, the
+	chunk table of contents, the sort order of the index, and the
+	bounds of every entry. Exits with non-zero status if the store is
+	corrupt. An absent store is valid.
+
+`compact`::
+	Fold the overlay tier into the base and remove the overlay, so
+	subsequent reads consult a single consolidated file. This rewrites
+	the base, so its cost is proportional to the store size; run it
+	periodically rather than after every warm.
+
+`clear`::
+	Remove the store files (base and overlay).
+
+CORRECTNESS
+-----------
+
+A stored result is interchangeable with a freshly computed one because an
+entry is keyed by everything that determines the diff:
+
+* the object IDs of the old and new blob, so a result is used only for
+  the exact contents it was computed from; and
+* the diff settings the hunks were computed under: the diff algorithm
+  and ignore flags (`xdl_opts`) and the context length. A lookup whose
+  settings differ from a stored entry misses. This is why, for
+  example, `blame -w` and `--diff-algorithm=<algorithm>` (including a
+  per-path `diff.<driver>.algorithm`) do not reuse entries recorded under
+  the default settings: they change `xdl_opts`. The context length is
+  part of the key because a zero context length triggers
+  `trim_common_tail`, which can produce a different but equally valid
+  set of hunks than a nonzero context; blame diffs at zero context and
+  `git log --stat` at its configured context, so they key apart and one
+  store serves both.
+
+Some options change the hunks in ways that are not part of the key, so
+they are excluded from the store in both directions: break detection
+(`-B`), `--ignore-matching-lines` (`-I`), `--anchored`, and
+`--ignore-blank-lines`. linkgit:git-blame[1] additionally does not
+consult the store for reverse blame, ignored revisions, or paths with a
+textconv driver.
+
+The store carries a trailing hash checksum, but readers do not
+re-checksum it on every load. As with the commit-graph and
+multi-pack-index, the writer fsyncs the file (honoring `core.fsync`) and
+commits it atomically, so a committed store is intact; every offset and
+count is still bounds-checked as it is read. The checksum is verified by
+`git diff-hunks verify` and by linkgit:git-fsck[1], not on the read path.
+
+CONFIGURATION
+-------------
+
+`core.diffHunks`::
+	Whether commands read the store. Defaults to true. See
+	linkgit:git-config[1].
+
+`diffHunks.write`::
+	Whether diff-producing commands write to the store. Defaults to
+	false. The `GIT_DIFF_HUNKS_WRITE` environment variable overrides it
+	for a single invocation. See linkgit:git-config[1].
+
+Writing the store honors the `core.fsync` configuration through the
+`diff-hunks` component; see linkgit:git-config[1].
+
+SEE ALSO
+--------
+linkgit:git-blame[1],
+linkgit:git-log[1],
+linkgit:gitformat-diff-hunks[5]
+
+GIT
+---
+Part of the linkgit:git[1] suite

Documentation/git-fsck.adoc

@@ -136,6 +136,9 @@ the hopes that somebody else has the object you have corrupted).
 If core.commitGraph is true, the commit-graph file will also be inspected
 using 'git commit-graph verify'. See linkgit:git-commit-graph[1].
 
+If core.diffHunks is true, the diff-hunks store will also be inspected
+using 'git diff-hunks verify'. See linkgit:git-diff-hunks[1].
+
 Extracted Diagnostics
 ---------------------
 

Documentation/gitformat-diff-hunks.adoc

@@ -0,0 +1,143 @@
+gitformat-diff-hunks(5)
+=======================
+
+NAME
+----
+gitformat-diff-hunks - Precomputed diff hunk store format
+
+SYNOPSIS
+--------
+[verse]
+$GIT_DIR/objects/diff-hunks
+$GIT_DIR/objects/diff-hunks-overlay
+
+DESCRIPTION
+-----------
+
+The diff-hunks store memoizes diff hunk coordinates so that commands
+that need them, such as `git log --stat` and linkgit:git-blame[1], can
+skip decompressing blobs and running the diff algorithm. See
+linkgit:git-diff-hunks[1] for how the store is filled and managed and the
+configuration that controls it.
+
+The store has two files under `$GIT_DIR/objects` in this format: a
+consolidated base, `diff-hunks`, and a small overlay, `diff-hunks-overlay`,
+of the pairs recorded since the last fold. Each is written in one pass and
+replaces its whole file atomically, so a reader sees either the old file
+or the complete new one. The layout below describes one such file.
+
+Entries are keyed by the object IDs of the blob pair that was diffed and
+by the diff settings the pair was diffed under. A blob pair fully
+determines the diff input, so an entry is valid regardless of which
+commits, branches, or index states the pair was encountered in, and
+identical diffs performed in different contexts share one entry.
+
+The diff settings recorded in each entry are the diff algorithm and
+ignore flags (`xdl_opts`) and the context length. Both change the
+resulting hunks, so they are part of the key: in particular a zero
+context length triggers `trim_common_tail`, which can pick a different
+but equally valid set of hunks than a nonzero context. A reader whose
+settings differ from an entry does not match it and falls back to
+computing the diff. Because the settings are per entry, one store can
+hold the hunks several consumers compute (for example blame at zero
+context and `git log --stat` at its context).
+
+FILE FORMAT
+-----------
+
+All multi-byte integers are stored in network byte order. The file is an
+8-byte header, the chunk table of contents and chunk data described in
+linkgit:gitformat-chunk[5], and a trailing checksum.
+
+HEADER
+~~~~~~
+
+- 4-byte signature: `DHPF` (diff-hunks precomputed format)
+- 1-byte version number: currently 1
+- 1-byte hash version: 1 for SHA-1, 2 for SHA-256. A store whose hash
+  function differs from the repository's is ignored.
+- 1-byte number of chunks
+- 1-byte reserved
+
+CHUNK LOOKUP
+~~~~~~~~~~~~
+
+A table of contents in the format of linkgit:gitformat-chunk[5], listing
+the offset of each chunk. Both chunks below are required; a file missing
+either is treated as corrupt.
+
+CHUNK DATA
+~~~~~~~~~~
+
+DHIX (index)::
+	A sorted sequence of fixed-size entries. Each entry is the old
+	blob object ID, the new blob object ID, a 4-byte `xdl_opts`
+	value, a 4-byte context length, and a 4-byte offset into the
+	DHDT chunk. Entries are sorted by old object ID, then new object
+	ID, then `xdl_opts`, then context, so lookups can use binary
+	search on the full key.
+
+DHDT (hunk data)::
+	For each index entry, at its offset: a 4-byte hunk count followed
+	by that many 16-byte hunk records. A hunk record is four 4-byte
+	values: old start, old count, new start, new count, as produced
+	by the xdiff hunk callback. Identical hunk blocks are stored once:
+	distinct index entries whose recorded hunks are byte-for-byte
+	equal point at the same offset, so keying the same result under
+	several settings costs a single block.
+
+TRAILER
+~~~~~~~
+
+A checksum of all preceding bytes, computed with the repository hash
+function.
+
+CORRECTNESS
+-----------
+
+Serving hunks from a valid store produces the same output as recomputing
+the diff. The diff of a blob pair is not unique: a zero context length
+triggers trim_common_tail, which can pick a different but equally valid
+set of hunks than a nonzero context does. Context is part of the key, so
+these two are stored apart.
+
+A pair is recorded as up to two entries. The zero-context entry holds the
+trimmed hunks, which git-blame reads directly (it diffs at zero context).
+The nonzero-context entry holds the untrimmed hunks, whose per-hunk line
+counts diffstat sums. One nonzero-context entry serves every context:
+context length changes how many unchanged lines a hunk shows, not the
+counts diffstat sums. Its coordinates are the untrimmed hunk boundaries,
+not those of any particular context, so a future consumer that needs a
+given context's boundaries recomputes them.
+
+A store that cannot be used is ignored, and the consumer falls back to
+computing the diff. Every offset and count read from the file is
+bounds-checked, so a store that is missing, truncated, of an unknown
+version, or of a different object hash does not change the diff output
+(a malformed store may still produce a diagnostic on stderr).
+
+The store is not re-checksummed on the read path. The writer fsyncs the
+file (honoring `core.fsync`) and commits it atomically, so a
+committed store is intact, the same trust model the commit-graph and
+multi-pack-index use. The trailing checksum is recomputed by
+`git diff-hunks verify` and linkgit:git-fsck[1] to detect corruption.
+
+The checksum detects corruption but does not prove who wrote the file. A
+reader trusts the coordinates in a store that passes its checks, so
+anything able to write a checksum-valid file at the store path can
+influence output, the same as it could by writing objects directly.
+
+LIMITATIONS
+-----------
+
+- Hunk counts, offsets, and line coordinates are 32-bit, capping a
+  single entry at roughly four billion hunks and the hunk data at 4 GiB.
+  A result whose coordinates cannot be represented is not recorded.
+- Context lengths that produce identical hunks (for example 3 and 5,
+  where neither trims) are keyed separately rather than folded together,
+  so a lookup at a context the store was not built for misses even though
+  the result would be the same.
+
+GIT
+---
+Part of the linkgit:git[1] suite

Documentation/meson.build

@@ -41,6 +41,7 @@ manpages = {
   'git-describe.adoc' : 1,
   'git-diagnose.adoc' : 1,
   'git-diff-files.adoc' : 1,
+  'git-diff-hunks.adoc' : 1,
   'git-diff-index.adoc' : 1,
   'git-diff-pairs.adoc' : 1,
   'git-difftool.adoc' : 1,
@@ -175,6 +176,7 @@ manpages = {
   'gitformat-bundle.adoc' : 5,
   'gitformat-chunk.adoc' : 5,
   'gitformat-commit-graph.adoc' : 5,
+  'gitformat-diff-hunks.adoc' : 5,
   'gitformat-index.adoc' : 5,
   'gitformat-loose.adoc' : 5,
   'gitformat-pack.adoc' : 5,

Makefile

@@ -1147,6 +1147,7 @@ LIB_OBJS += diffcore-order.o
 LIB_OBJS += diffcore-pickaxe.o
 LIB_OBJS += diffcore-rename.o
 LIB_OBJS += diffcore-rotate.o
+LIB_OBJS += diff-hunks.o
 LIB_OBJS += dir-iterator.o
 LIB_OBJS += dir.o
 LIB_OBJS += editor.o
@@ -1409,6 +1410,7 @@ BUILTIN_OBJS += builtin/credential.o
 BUILTIN_OBJS += builtin/describe.o
 BUILTIN_OBJS += builtin/diagnose.o
 BUILTIN_OBJS += builtin/diff-files.o
+BUILTIN_OBJS += builtin/diff-hunks.o
 BUILTIN_OBJS += builtin/diff-index.o
 BUILTIN_OBJS += builtin/diff-pairs.o
 BUILTIN_OBJS += builtin/diff-tree.o