RFC (superseded staging): hunk providers, content-first protocol ordering

.gitignore

@@ -56,6 +56,7 @@
 /git-diagnose
 /git-diff
 /git-diff-files
+/git-diff-hunks
 /git-diff-index
 /git-diff-pairs
 /git-diff-tree

Documentation/Makefile

@@ -33,6 +33,7 @@ MAN5_TXT += gitattributes.adoc
 MAN5_TXT += gitformat-bundle.adoc
 MAN5_TXT += gitformat-chunk.adoc
 MAN5_TXT += gitformat-commit-graph.adoc
+MAN5_TXT += gitformat-diff-hunks.adoc
 MAN5_TXT += gitformat-index.adoc
 MAN5_TXT += gitformat-loose.adoc
 MAN5_TXT += gitformat-pack.adoc

Documentation/config.adoc

@@ -417,6 +417,8 @@ include::config/credential.adoc[]
 
 include::config/diff.adoc[]
 
+include::config/diff-hunks.adoc[]
+
 include::config/difftool.adoc[]
 
 include::config/extensions.adoc[]

Documentation/config/core.adoc

@@ -670,12 +670,13 @@ but risks losing recent work in the event of an unclean system shutdown.
 * `pack` hardens objects added to the repo in packfile form.
 * `pack-metadata` hardens packfile bitmaps and indexes.
 * `commit-graph` hardens the commit-graph file.
+* `diff-hunks` hardens the diff-hunks store.
 * `index` hardens the index when it is modified.
 * `objects` is an aggregate option that is equivalent to
   `loose-object,pack`.
 * `reference` hardens references modified in the repo.
 * `derived-metadata` is an aggregate option that is equivalent to
-  `pack-metadata,commit-graph`.
+  `pack-metadata,commit-graph,diff-hunks`.
 * `committed` is an aggregate option that is currently equivalent to
   `objects`. This mode sacrifices some performance to ensure that work
   that is committed to the repository with `git commit` or similar commands
@@ -750,6 +751,13 @@ core.commitGraph::
 	to parse the graph structure of commits. Defaults to true. See
 	linkgit:git-commit-graph[1] for more information.
 
+core.diffHunks::
+	If true, then Git will consult the diff-hunks store (if it
+	exists) to skip recomputing diff hunk coordinates in commands
+	such as `git log --stat` and linkgit:git-blame[1]. This controls
+	only reading; writing the store is controlled by `diffHunks.write`.
+	See linkgit:git-diff-hunks[1] for more information. Defaults to true.
+
 core.useReplaceRefs::
 	If set to `false`, behave as if the `--no-replace-objects`
 	option was given on the command line. See linkgit:git[1] and

Documentation/config/diff-hunks.adoc

@@ -0,0 +1,7 @@
+diffHunks.write::
+	If true, diff-producing commands (`git diff` and `git log` with a
+	`--stat`, `--numstat`, or `--shortstat` format) write the hunks
+	they compute to the diff-hunks store, filling it as a side effect.
+	The `GIT_DIFF_HUNKS_WRITE` environment variable overrides this for
+	a single invocation. Reading the store is controlled separately by
+	`core.diffHunks`. See linkgit:git-diff-hunks[1]. Defaults to false.

Documentation/config/diff.adoc

@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.

Documentation/diff-algorithm-option.adoc

@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).

Documentation/diff-options.adoc

@@ -825,7 +825,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::

Documentation/git-diff-hunks.adoc

@@ -0,0 +1,125 @@
+git-diff-hunks(1)
+=================
+
+NAME
+----
+git-diff-hunks - Inspect and manage the precomputed diff hunk store
+
+SYNOPSIS
+--------
+[synopsis]
+git diff-hunks verify
+git diff-hunks clear
+
+DESCRIPTION
+-----------
+
+The diff hunk store is a cache of diff hunk coordinates, so that commands
+which need them, such as linkgit:git-blame[1] and `git log` and `git diff`
+with the `--stat`, `--numstat`, and `--shortstat` formats, can skip
+decompressing blobs and running the diff algorithm.
+
+The store is a single file, `$GIT_DIR/objects/diff-hunks`. Reading is
+enabled by default; writing is off by default. A `git diff`, `git log`,
+`git show`, or `git diff-tree` that produces one of the stat formats
+fills the store as a side effect, but only when writing is enabled for
+that run (see "WARMING THE STORE" below), so ordinary reads never
+modify the repository. When the store does not have the pair, holds a
+different object hash, or the file is unreadable, the consumer falls
+back to computing the diff. A store only speeds up these commands; it
+never changes their output.
+
+`git diff-hunks` itself only inspects and manages the file. See
+linkgit:gitformat-diff-hunks[5] for the file format.
+
+WARMING THE STORE
+-----------------
+
+The store is filled by running ordinary commands with writing enabled.
+Turn writing on for a single invocation with the `GIT_DIFF_HUNKS_WRITE`
+environment variable, or persistently with the `diffHunks.write`
+configuration; the environment variable takes precedence. A repository
+owner warms the store by running the diff-producing commands they care
+about with writing on, for example:
+
+	GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
+
+A `--stat` walk records one entry per blob pair, which serves both
+linkgit:git-blame[1] (it replays the coordinates) and the summary
+formats (they sum the counts), so a single warming walk serves both.
+A warm seeds from the existing store and rewrites the file with the
+newly computed pairs merged in, so a later warm adds to what earlier
+warms recorded rather than discarding it.
+
+COMMANDS
+--------
+
+`verify`::
+	Check the integrity of the store: the trailing hash checksum, the
+	chunk table of contents, the sort order of the index, and the
+	bounds of every entry. Exits with non-zero status if the store is
+	corrupt. An absent store is valid.
+
+`clear`::
+	Remove the store file.
+
+CORRECTNESS
+-----------
+
+A stored result is interchangeable with a freshly computed one because an
+entry is keyed by everything that determines the diff:
+
+* the object IDs of the old and new blob, so a result is used only for
+  the exact contents it was computed from; and
+* the diff algorithm and ignore flags (`xdl_opts`) the hunks were
+  computed under. A lookup whose `xdl_opts` differ from a stored entry
+  misses. This is why, for example, `blame -w` and
+  `--diff-algorithm=<algorithm>` (including a per-path
+  `diff.<driver>.algorithm`) do not reuse entries recorded under the
+  default settings: they change `xdl_opts`.
+
+The context length is not part of the key because only trim-stable
+pairs are recorded: pairs whose zero-context trimmed diff and untrimmed
+diff are identical, so one entry answers blame (zero context) and the
+summary formats (any context) alike. The rare pair where
+`trim_common_tail` picks a different but equally valid set of hunks is
+never recorded and is always computed.
+
+Some options change the hunks in ways that are not part of the key, so
+they are excluded from the store in both directions: break detection
+(`-B`), `--ignore-matching-lines` (`-I`), `--anchored`, and
+`--ignore-blank-lines`. linkgit:git-blame[1] additionally does not
+consult the store for reverse blame, ignored revisions, or paths with a
+textconv driver.
+
+The store carries a trailing hash checksum, but readers do not
+re-checksum it on every load. As with the commit-graph and
+multi-pack-index, the writer fsyncs the file (honoring `core.fsync`) and
+commits it atomically, so a committed store is intact; every offset and
+count is still bounds-checked as it is read. The checksum is verified by
+`git diff-hunks verify`, not on the read path.
+
+CONFIGURATION
+-------------
+
+`core.diffHunks`::
+	Whether commands read the store. Defaults to true. See
+	linkgit:git-config[1].
+
+`diffHunks.write`::
+	Whether diff-producing commands write to the store. Defaults to
+	false. The `GIT_DIFF_HUNKS_WRITE` environment variable overrides it
+	for a single invocation. See linkgit:git-config[1].
+
+Writing the store honors the `core.fsync` configuration through the
+`diff-hunks` component; see linkgit:git-config[1].
+
+SEE ALSO
+--------
+linkgit:git-blame[1],
+linkgit:git-log[1],
+linkgit:gitformat-diff-hunks[5]
+
+GIT
+---
+Part of the linkgit:git[1] suite