[RFC PATCH v7 0/10] diff: add provider interface and initial providers - #4
Draft
mmontalbo wants to merge 10 commits into
Draft
[RFC PATCH v7 0/10] diff: add provider interface and initial providers#4mmontalbo wants to merge 10 commits into
mmontalbo wants to merge 10 commits into
Conversation
mmontalbo
force-pushed
the
mm/hunk-providers-oid-first
branch
3 times, most recently
from
July 30, 2026 05:05
88eea49 to
942eff6
Compare
mmontalbo
changed the title
mmontalbo
changed the title
mmontalbo
force-pushed
the
mm/hunk-providers-oid-first
branch
from
July 30, 2026 05:13
942eff6 to
bdfdaec
Compare
mmontalbo
changed the title
mmontalbo
force-pushed
the
mm/hunk-providers-oid-first
branch
22 times, most recently
from
July 31, 2026 18:00
1b50341 to
0cafd69
Compare
mmontalbo
force-pushed
the
mm/hunk-providers-oid-first
branch
8 times, most recently
from
August 1, 2026 00:40
ad84c9f to
3a6939e
Compare
…ures The "Defining an external diff driver" section explains how to configure diff.<driver>.command but not how the driver relates to the rest of Git's diff machinery. In particular, the command only replaces the textual patch: word diff, function context, color, and the like cannot apply to its output, while the summary formats, blame, and git log -L do not run it at all and keep using the builtin diff. Spell this out so the scope of an external diff driver is clear. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
To learn which line ranges changed between two blobs, every consumer in the diff machinery loads both blobs and runs xdiff. There is no other way to supply that answer, even when it is known elsewhere: a cache may hold the ranges from the last time the pair was diffed, and a format-aware process may have its own idea of which lines changed. Either could answer from the blob object ids alone, but the loading and computing are hard-wired into each consumer, so such an answer has no place to enter. Introduce the hunk provider interface, diff-provider.h, between asking the question and computing the answer. A provider answers a request made of the pair's identity, its blob object ids and the parameters that determine the diff. A provider is either authoritative, so its answer may deliberately differ from the builtin diff, or not, so its answer must reproduce the builtin result exactly. Every answer served from identity passes diff_provider_check_hunk() before a consumer sees it: coordinates fit int32, hunks are ordered and non-overlapping, and the unchanged runs between them match on both sides. A failing answer is discarded and the pair falls through as unanswered. Providers are repository-lifecycle objects. Each repository owns a chain of them, built on first consultation and released from repo_clear(), so a submodule gets its own providers and no provider state outlives the repository it serves. The chain has a fixed composition, and each provider gates itself per request, passing when it does not apply. Chain order is the authority: the first answer wins. A provider may instead refuse a pair whose request is shaped by parameters its recording key cannot express. After a refusal, no later provider answers the pair from identity, and the consumer must not record what it computes for it. The last provider is the builtin computation, the only one that computes rather than answering from identity, so a walk given a fill callback always ends in an answer, refusal or not. The walk in diff-provider.c maps a provider's four dispositions (answer, pass, fail, refuse) onto the consumer-facing outcomes, and checks with BUG() that only the computing provider fails and that it passes on a walk with no fill callback. The implementor contract, the provider struct, its dispositions, and the shared check, lives in diff-provider-internal.h, as refs/refs-internal.h is to refs.h; consumers see only diff-provider.h. The consumer surface is two types. struct diff_provider_request names what is diffed and under which parameters; each later commit that consults on more state adds the field it keys on (the object ids and diff options, then the path). enum diff_provider_outcome flattens two dependent axes into four points: the response state (answered, unanswered, failed) and, only when unanswered, whether the caller may record what it computes. The record rule rides in the outcome, not a separate flag, so -Wswitch forces every consumer to place the no-record arm. A provider added later maps onto these values inside the walk, so consumer code is written once. diff_provider_emit_hunks() is the consumer entry: the caller states the request, a hunk callback, and a content-loading callback that reaches the terminal provider only when the ranges are computed. Blame's pass_blame_to_parent() is the first consumer, since it knows both blob ids before reading either blob; its loads move into the fill callback. With only the terminal provider registered, every request still computes, so behavior is unchanged. (Blame's -C/-M split detection diffs partial buffers with no blob identity and stays on xdi_diff().) Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Blame and "git log --stat" recover hunk coordinates by diffing blob pairs, and recompute them on every run. Add a cache of those coordinates at $GIT_DIR/objects/info/diff-hunks, beside the commit-graph, so a later run can look them up instead of decompressing the blobs and running xdiff again. The store is a single chunk-format file (see gitformat-chunk(5)): an 8-byte header, a DHIX index of fixed-size entries sorted by key, a DHDT segment of hunk records, and a trailing hash checksum. An entry is keyed by the two blob object ids and the xdl_opts the pair was diffed under, so a stored result is served only where that exact key recurs, independent of path. A zero-context diff trims unchanged lines from hunk edges and can pick a different but equally valid set of hunks than an untrimmed diff, so a recording caller stores a pair only when its trimmed and untrimmed diffs are identical; such an entry answers any consumer at any context, and the rare divergent pair is always computed. Identical hunk blocks are interned once and shared across keys. The library provides a reader (repo_diff_hunks_store and _replay, gated by core.diffHunks), loaded once and cached on the object database as the commit-graph is, and a writer that accumulates entries and flushes them in one atomic pass. An absent, corrupt, or disabled store reads as all misses. A record with no hunks is invalid too: replaying it would claim the pair equivalent, which the store never asserts, so it reads as a miss. Ordinary reads are diagnostic-free. Loading parses the chunk table through read_table_of_contents_quiet(), new in chunk-format, which prints nothing on a malformed table and takes the repository's hash algorithm rather than the_hash_algo, so the file is bounds-checked under the algorithm it is keyed by. The flush closes the repository's mmapped store and forgets that loading was attempted before committing the lockfile. A warming run that also reads may hold the file it is replacing mapped, and the rename must not land on a live mapping, which Windows refuses; a read after the flush then observes the committed file. commit-graph closes its graph before committing for the same reason. Writing is off by default, enabled per run by GIT_DIFF_HUNKS_WRITE or persistently by diffHunks.write, the environment winning. A writer seeds from the existing store, so a flush merges rather than replaces. The seed's checksum is verified first: a corrupt store is discarded, not rewritten with a fresh checksum verify could no longer catch. An entry that fails the shared diff_provider_check_hunk() or names no blob is dropped with a warning, since it would only ever read as a miss. A seed that discarded or dropped anything forces the flush even when the warming run computed nothing new. The writer fsyncs through a new diff-hunks core.fsync component. "git diff-hunks" inspects and manages the file: "verify" checks the checksum, chunk table, sort order, entry bounds, and every entry's hunk sequence against that shared check, so a store whose entries could only read as misses fails verify; "clear" removes the file. Later patches wire the readers and the writer into the diff and blame paths. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
mmontalbo
force-pushed
the
mm/hunk-providers-oid-first
branch
from
August 1, 2026 01:06
3a6939e to
4195250
Compare
mmontalbo
force-pushed
the
mm/hunk-providers-base
branch
from
August 1, 2026 02:46
f67c51d to
43bc512
Compare
The diff-hunks store has a writer, but nothing fills it. Teach
builtin_diffstat() to do so: on a warming run (a writer is attached), a
modified pair's stat is produced by collecting the pair's hunk
coordinates instead of emitting text, the counts are summed from those
hunks, and the pair is recorded. A run without a writer is unchanged,
and nothing reads the store yet; the read side arrives next.
The store records one context-free entry per pair, and only for a
trim-stable pair: one whose zero-context trimmed diff (what blame will
read) and untrimmed diff (whose counts a nonzero-context stat matches)
are identical. The warming path computes both and hands them to
diff_hunks_writer_record_stable(), new here, which records only when
they agree; a divergent pair is never recorded and every consumer
computes it. The warming run displays the counts it shows a store-less
run: the trimmed ones, since xdi_diff trims at zero context, while the
untrimmed counts serve only the stability comparison.
Not everything the stat path computes may be recorded.
--ignore-blank-lines is part of the key, but it coalesces hunks
differently between the text-emitting and coordinate-callback paths, so
a recorded entry would not match a store-less run's --stat. -I
patterns, --anchored, and break detection (-B) shape the diff outside
the key entirely; the guard for those three sits in this consumer for
now and moves into the store's own provider when it registers, next. A
"log -L" range-scoped stat is not the whole-pair diff the key describes,
so it does not record. Recording also requires both sides to be valid
regular files whose blobs the key can name: a working-tree side,
textconv output, or a gitlink has no usable id.
"git diff", "git log", "git show", and "git diff-tree" with the --stat,
--numstat, and --shortstat formats attach a writer when writing is
enabled and flush it when the traversal finishes, so a warming run such
as
GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
fills the cache as a side effect of the diff work the command already
does. Writing is controlled by diffHunks.write and GIT_DIFF_HUNKS_WRITE.
Add the write half of t4220:
- ordinary commands never create the store, and creation is gated off
by default, the environment overriding the config;
- a warming run builds a store that verifies, and a second refreshes it
in place;
- a warming run displays parity at zero context on a trim-divergent
pair, committed as a fixture (small synthetic pairs cannot diverge:
minimal diffs add and delete equal counts, and trimming preserves
that);
- binary and mode-only pairs do not break the writer;
- a corrupt store is discarded at seed;
- verify and clear run against the files a warming run builds.
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Teach builtin_diffstat() to consult the hunk provider interface through diff_provider_consult(), new here: the consult-only entry that answers without loading content or computing, so it never returns DIFF_PROVIDER_ERROR. On an answer, the summing callback accumulates the provided counts directly into the diffstat entry; the blobs were already loaded for the binary check, so an answer saves the diff run, not the content load (blame, taught next, skips its loads too). On an unanswered outcome it computes as before and, with a writer attached, records what it computed; on unanswered-no-record it computes without recording. The provider behind the consult is the diff-hunks store, registered in front of the terminal builtin computation. Its consult serves a recorded pair through diff_hunks_replay(), which validates the sequence before any hunk reaches the callback, so direct accumulation is safe. The request gains the pair's object ids and the diff options read by the exclusions below. A side whose bytes are not a stored blob, such as a working-tree file or a gitlink, has a NULL id; the store passes it by and the terminal provider computes it. diff_provider_emit_hunks() walks the same chain, so blame's requests follow these rules the moment blame supplies identity. The walk also insists, as a BUG check, that a request's diff options belong to the repository whose chain it walks. Each exclusion lives with the provider whose key cannot express it. -I patterns and --anchored shape the diff outside the store key, and break detection (-B) rescores the pair outside it; the store's consult maps all three to stop-no-record, so such a request is neither served nor recorded for any consumer. The consumer-side guard the recording commit carried for those three comes out here. The compile-time assert on xpparam_t's layout sits next to that decision, forcing an explicit keying decision whenever a diff parameter is added. The stat consumer keeps only the exclusion that is not about the key: --ignore-blank-lines is part of the key but coalesces hunks differently between the text-emitting and coordinate-callback paths, so the consumer returns before consulting. A "log -L" range-scoped stat neither reads nor records; the line-range filter computes it as before. "git diff", "git log", "git show", and "git diff-tree" with the --stat, --numstat, and --shortstat formats consult the interface. Reading is controlled by core.diffHunks. An answer is invisible in the output, so the store counts the pairs it serves and the consultations it cannot, and diff_hunks_read_stats() reports both; the stat path emits the hits as a trace2 "read-hits" datum for tests and tuning. The counters live on the store because only the store knows whether a consultation reached it, and none of its exclusion legs reaches the replay, so none counts as a miss. Extend t4220 with the read half: - output parity with and without the store, at several context lengths and both directions, and reversed pairs keying apart; - the consultation made visible through the read-hits datum, and the trim-divergent pair correct at every context; - the settings that must bypass the store doing so in both directions (-I, -B, --anchored, --ignore-blank-lines), asserted through the trace rather than output parity alone, which a coincidentally equal count could satisfy; - a driver-forced algorithm keying apart rather than bypassing: it is part of the key, so a read under it misses the default entries and a warm records under its own. A "log -L" range-scoped stat neither reads nor records. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Before diffing a target blob against a parent, offer the pair's identity to the hunk provider interface. Blame's requests have gone through diff_provider_emit_hunks() since the interface arrived, but carried no identity, so nothing could answer them. Now blame fills in the pair's blob object ids and its diff options, and the chain serves the pair from the store, keyed by the ids and the request's xdiff flags, before the terminal provider falls back to fill-and-compute. Blame diffs at zero context, which is not part of the key. An answer replays the recorded hunks through blame_chunk_cb without loading either blob; a request carrying -I patterns or anchors is outside the key and always computes. Blame withholds the identity where its diff is not the plain blob-pair diff the key describes: reverse blame, ignored revisions, textconv paths, and the working-tree or --contents pseudo-commit, whose blob is not a stored object. Those requests always compute. Whitespace and algorithm options such as -w instead change blame's xdl_opts, so the consult keys a different entry and misses a store warmed without them. Blame's default xdl_opts now come from DIFF_HUNKS_DEFAULT_XDL_OPTS, new here, which records the key-relevant defaults a diff_options-based consumer already carries (today the indent heuristic), so a default blame run and a default "log --stat" warming run share keys by construction. "--show-stats" reports how many pairs the store served and how many consultations it could not, read from diff_hunks_read_stats(); the store counts its own consultations, so blame keeps no tally. Extend t4220 with the blame side: - parity for plain, --porcelain, and --incremental output, and hit and miss accounting across warming runs; - the blame inputs that must bypass or miss the store: -w, indent heuristics, --reverse, textconv, -M/-C, and the --ignore-rev pass; - rename and merge handling, and --contents; - reading a truncated or corrupt store as absent, and a crafted zero-hunk record as a miss that verify flags. Add p4218, measuring the cost of a warming run and the read speedups. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
subprocess_start() and subprocess_stop() couple two concerns: managing a child process (setup, handshake, teardown) and managing a hashmap that indexes running processes by command string. The hashmap suits callers like convert.c where many files may share one filter process looked up by name, but callers that manage process membership under their own rules do not need the coupled operations. Extract subprocess_start_command() and subprocess_stop_command() so callers can reuse the child process setup and handshake machinery without the map operations. subprocess_start() and subprocess_stop() become thin wrappers that add hashmap operations on top. The diff process support added later in this series keeps its processes in a pool owned by a per-repository provider object, and an entry for a failed command must stay behind there so the command is not retried. That membership follows rules subprocess_start() and subprocess_stop() do not know. The pool therefore uses the _command variants for process lifecycle and manages its own map. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
subprocess_read_status() reads "status=<key>" packets up to a flush with packet_read_line_gently(), which is gentle only about EOF. A malformed length header still dies inside pkt-line, and an empty packet is indistinguishable from the flush that ends the section. A protocol violation in a status section therefore either kills the whole command or silently truncates the section. That posture fits the filter protocol's callers, which treat their process as required infrastructure; the diff process consult added later in this series treats its process as optional, and any protocol error must degrade to the builtin diff rather than abort the command. Add subprocess_read_status_gently(): the same status loop, reading through packet_read_with_status() with the gentle options, returning -1 on a truncated or malformed packet and on an empty packet where a status line or the terminating flush belongs. subprocess_read_status() and its callers are unchanged. The handshake has its gentle counterpart in 061a68e (sub-process: use gentle handshake to avoid die() on startup failure, 2026-06-01), which turned truncated handshake reads into error returns for every caller. This series' base includes that commit, so a process that dies during the handshake feeds the same non-fatal fallback as a status failure here, and an optional diff process degrades to the builtin diff on either kind of protocol error. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
Add the process field to struct userdiff_driver and teach the config parser to populate it from diff.<driver>.process. The field names a long-running hunk provider process. Nothing reads it yet: the consult, the protocol, and the documentation arrive with the next commit, which starts and pools processes keyed by this field's command string. Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
mmontalbo
force-pushed
the
mm/hunk-providers-oid-first
branch
2 times, most recently
from
August 1, 2026 17:35
c223063 to
391e3d9
Compare
mmontalbo
changed the title
The provider chain so far holds the diff-hunks store in front of the terminal builtin computation. Open it to external processes: a pair on a path whose driver configures diff.<driver>.process is answered by a long-running process speaking a pkt-line protocol (following the filter process protocol), registered at the head of the chain and consulted before the store and before any blob is loaded. The protocol starts with the smallest request that can carry an answer: object names alone. A request is the pathname and the pair's old-oid/new-oid, with no content. The process answers with hunk lines, with a zero-hunk success that asserts the blobs equivalent (trailing newlines included), or with status=need-content, on which the pair falls through to the builtin answer. This serves the two shapes that need no content pushed to them: a cache keyed on the blob pair, and a process that fetches the blobs itself (for example over "git cat-file --batch"). A pair whose side is not a stored blob carries a NULL id; the provider sends no request and passes it. Because Git holds no content for the exchange, the answer is used as sent: hunks are validated for order, overlap, lockstep alignment, and magnitude, then replayed without the normalization xdiff applies to diffs it computes itself. The magnitude bound is the blobs' sizes, read from the object database without loading content: a blob of N bytes holds at most N lines. Because the process's answer is authoritative, it outranks the store, and its head-of-chain position says so. A pair the process answers never reaches the store and is never recorded, so nothing it produces enters the store, which holds the builtin answer only. A request it does not answer, whether need-content, a missing capability, or a missing id, passes down the chain to the builtin answer, which is what the store serves, so the store may serve such a pair and a warming run may record it. Entries recorded before a process was configured are not purged; a pair the process answers ignores them, and "git diff-hunks clear" discards them. The provider gates itself per request. The driver is looked up by the old-side path, so a renamed file resolves to the same driver, and by the repository-relative path, so a diff.relative run from a subdirectory names the pair the same way. Options the process is never told about select no process: the whitespace-ignoring options, -I, --anchored, and an algorithm forced by option or configuration (blame routes its algorithm through xdl_opts, so --histogram is covered). The request gains its last field, the path; the consumers change only by filling it, and neither names the process. The provider's state is its repository's pool of running processes, keyed by the configured command, so drivers sharing a command share a process, a submodule speaks to its own, and releasing the provider (from repo_clear()) stops them. The pool owns a copy of each command string, so an entry outlives a config re-read. A command that fails stays as an entry that is not retried: its request and every later one pass, so the store may serve the path for the rest of the command. A protocol error in a response never kills the command. The response is read through a packet reader gentle about framing, so an error takes one path: a single warning, the process stopped and marked failed, and the builtin diff for the rest of the command. That covers garbage bytes, a truncated response, an empty packet, a bare status, and an unrecognized status. Semantically invalid coordinates cost only their pair: the response is drained, the pair is computed, and the process stays alive. A path the protocol cannot carry (an embedded newline, or one too long for a packet) falls back per path rather than costing the command its process. The handshake keeps one fatal check: a process that announces a capability Git did not request aborts the command, as the long-running filter protocol does. Consulting is allowed per command, following the allow_textconv precedent. "git diff", "git log" and "git show", and "git blame" set allow_diff_process; the plumbing diff commands and the interactive-patch machinery never set it, so scripted and staging output stays builtin. The options adjust the flag: - --no-ext-diff clears it and --ext-diff sets it; - --diff-process and --no-diff-process set and clear it alone, leaving external diff drivers as they were; - format-patch clears it unconditionally, so a generated patch applies for recipients without the process; - range-diff passes --no-ext-diff to the "git log" it compares. git blame and the summary formats consult the process. For blame, a pair reported equivalent emits no hunks, so the whole commit passes to its parent. In the stat formats such a pair sums to a zero-count entry, which the "nothing changed" rule omits, as under -w. The subprocess is long-running: one startup cost across a traversal, one round-trip per consulted pair. Answers travel in struct xdl_hunk, new in xdiff-interface.h, holding xdiff's 1-based coordinates; nothing feeds them back to xdiff, since only coordinate consumers consult. A content-carrying request is the natural extension: it would serve sides that are not stored blobs and processes that want content pushed to them, and bring patch output and log -L's range tracking to the same answer. As it stands, a process's answers show in blame and the summary formats while patch output stays builtin. t4080 exercises the protocol, the per-command gate, and the error paths: - each adversarial response shape warns and falls back to builtin, the request log proving which failures disable the process and which keep it alive (a malformed hunk line, coordinates past the blob size, a count overflowing strtol(), overlapping or misaligned hunks, an unrecognized status, a bare status, an empty packet, a mid-response crash, and raw garbage); - a capability-less process and status=abort degrade without noise, and a failed start warns once and returns the path to the store; - a trailing token on a hunk line is ignored, pinning field appendability; - positive consults for git diff, git show, and diff-tree under --ext-diff and --diff-process; textconv output and gitlink sides are never identified; a diff.relative run consults by the repo-relative path; - the equivalence answer is pinned from both consumers, and a warming run past a deferring process records the pair for a later read. Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
mmontalbo
force-pushed
the
mm/hunk-providers-oid-first
branch
from
August 4, 2026 02:31
391e3d9 to
0b865ee
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every in-process diff in Git reduces, at one point, to a single
question: given two blobs and the settings the diff runs under, which
line ranges changed? The answer is the diff's hunks: for each change,
the position and length of the range on the old side and on the new.
Each consumer asks in its own shape:
coordinates through xdiff's hunk callback;
content around them;
In every case the answer is computed the same way: load both blobs and
run xdiff. That is the only source, so nothing that already holds the
answer, or that would answer differently on purpose, can supply it
instead. Sometimes that is what we want, which is why patch-id and
format-patch stay on the builtin computation throughout: patch-id needs
identical hashes on every machine, and a format-patch must apply for
recipients who share none of the sender's configuration. Other times
another source would be useful.
This RFC sketches a direction. The unified series shows one interface
carrying two example providers and their interaction; it is not shaped
to merge as one topic. If the direction holds, the work returns as
separate reviewable series (see Roadmap). The two examples are
demonstrations, each an RFC on its own: diff..process, the RFC
cooking as mm/diff-process-hunks, lets a configured external process
answer with its own notion of which lines changed, and the diff-hunks
store, new in this thread, remembers what xdiff computed and serves it
back. One is authoritative and external, one a cache and in-process.
Three pieces:
A hunk provider interface (diff-provider.h) is the point of the
series. A provider is an alternate source for the answer: asked with
the pair's object ids and the diff settings, before any blob is
loaded, it may supply the hunks in place of the builtin computation.
A miss falls through to that computation, and every answer passes one
shared validity check first. The providers form a chain the
repository owns, built on first consultation and released in
repo_clear(), so provider state such as a running process never
outlives its repository. Chain order is the authority, and the
terminal provider is the builtin computation itself, so the interface
never exists without an implementor: patch 02 ships it answering every
request the way the consumers did before. A consumer states its
request in one struct and reads one set of outcomes (answered,
unanswered, or failed); it never names a provider, and a provider
added later maps onto those outcomes inside the interface, so consumer
code is written once. Because every diff now walks the chain even
with no store or process configured, the default path was measured
against the pre-series base and runs within noise (a 5000-commit
log --stat and a long-history blame, ratio 1.00 either way).
The diff-hunks store shows the non-authoritative side: an in-process
cache at $GIT_DIR/objects/info/diff-hunks that may only reproduce the
builtin diff, so serving from it never changes a command's output. It
is read by default and written only when a repository owner opts in,
warming it as a side effect of diff work the command already does:
A warmed store then serves the stat formats and blame from stored
coordinates instead of a fresh diff: on git.git a 5000-commit log
--stat runs about 1.9x faster, and blame reads the same entries
opportunistically (full numbers in [1]). Its format and keying, what
it may not serve, and how it handles corruption and staleness are in
git-diff-hunks(1), gitformat-diff-hunks(5), and [2]. The interface
point is small: a cache drops in as the provider that stands aside
wherever an authoritative one answers.
diff..process shows the authoritative side: an external
process, configured per driver, whose answers may deliberately differ
from the builtin diff and outrank the store. Git asks it for a pair by
object names alone, so it answers before any blob is read, which suits
a cache or a process that fetches the blobs itself. Consulting is
opt-in per command, following the allow_textconv precedent, and a pair
the process cannot answer falls back to the builtin diff. The
protocol, the per-command gate, how failures are handled, and the
versioning that lets it grow are in gitattributes(5) and footnotes [3]
and [4]. The interface point, again, is small: an external,
authoritative provider joins the same chain ahead of the cache, and
neither consumer learns it is there. A later content-carrying request
would extend it to the pairs and consumers this identity-only form
leaves on the builtin diff.
The series stops at the coordinates. A consumer that needs the changed
text, such as patch output, would have only its hunk selection replaced,
with xdiff still emitting content from the blobs; that machinery is the
content enrichment sketched in the Roadmap. Establishing the framework
on coordinates first keeps this series one design: the question, the
interface, and two providers answering by identity.
Shape of the series:
01 documentation: how external diff drivers relate to the
features layered on the diff
02 the provider interface: the request and outcome types, the
emit entry point, the shared validity check, and the
repository-owned chain with its terminal builtin provider
03 the store: on-disk format, library, and the diff-hunks command
04 recording: the stat walk computes, sums, and records
trim-stable pairs (writes gated off by default)
05 reading: the consult entry point and the store's registration
as a provider; the request gains the object ids and diff
options
06 blame reading through the interface's emit path
07-09 process preparation: sub-process lifecycle split, a gentle
status read for an optional process, and the
diff..process config
10 the process provider, oid-only, at the head of the chain, with
the per-command gate; the request gains the path
Roadmap:
This RFC asks whether the direction is right, not for these ten patches
to merge as one topic. If it holds, the work returns in reviewable
pieces:
the interface and the store (patches 01 through 06): a cache with
measured numbers and no external-process machinery
the process provider (patches 07 through 10) on the same interface
the content enrichment (the content-carrying request, patch output and
log -L consulting, and the xdiff machinery that feeds a provider's
hunks into emission) once the identity-keyed framework settles.
Several design questions are left for those series.
mm/diff-process-hunks in seen would be dropped in favor of this thread
and its split.
The series applies on the line-log topic (mm/line-log-limited-ops)
rebased onto current master. The topic rewrites the same
builtin_diffstat() region this series touches, and current master
includes 061a68e (sub-process: use gentle handshake to avoid die()
on startup failure), which this topic leans on: a process that dies
during the handshake degrades to the builtin diff like every other
failure. A trial merge against seen shows no interaction with other
topics beyond the mm/diff-process-hunks replacement above.
The base (line-log topic on current master) and the full series are
available at:
git fetch https://github.com/mmontalbo/git mm/line-log-stat-formats-followup
git fetch https://github.com/mmontalbo/git mm/hunk-providers-oid-first
Changes since v6:
This is a restructuring, not an incremental reroll, so a range-diff
against v6 is unreadable; the map of what changed:
diff-hunks store in as its in-process implementation (patches 02
through 06, new to this thread). It keeps only the identity-keyed
half of the external diff process protocol from mm/diff-process-hunks.
config return close to their v6 form as patches 01, 07, and 09. Patch
08 is new: a gentle status read so a protocol error in an optional
process degrades to the builtin diff instead of dying.
request, consulting through the interface, and carrying a per-command
gate (v6's bypass patch folds into it).
(patches 05, 06, and 10); their content legs, along with v6's xdiff
external-hunks machinery, content-carrying request, and line-log
consult, are withheld for the content enrichment.
Footnotes:
[1] Store numbers, measured with hyperfine against the same build with
core.diffHunks=false. The warm is a full cold build of the store;
the blame speedup is file-dependent (see the coverage limitation):
[2] The store is its own file because nothing existing is addressed by a
blob pair: notes attach to single objects, commit-graph chunks to
commits. One entry per pair, keyed by (old blob, new blob,
xdl_opts) and recorded only when the pair's trimmed and untrimmed
diffs agree, serves blame at zero context and the stat formats at
any -U (divergent pairs are 0.4-0.5% of a warm and always compute).
The writer fsyncs and commits atomically, and a reader bounds-checks
every record and treats an unparsable file as absent; the trailing
checksum is checked by git diff-hunks verify, not on every read, the
same read-time trust the commit-graph and multi-pack-index take.
There is deliberately no fsck integration, expiry, or background
maintenance: the store is derivable at any time, so the recovery
path is git diff-hunks clear and a re-warm. New commits make it
incomplete, not wrong; a later warm seeds from the file and pays
only for what is new.
[3] Consulting the process is allowed per command, like textconv: git
diff, git log and git show, and git blame consult it; the plumbing
diff commands do not unless --ext-diff or --diff-process is given,
and the interactive-patch machinery, format-patch, and range-diff
stay builtin. Options the process is never told about select no
process, and an object id is sent only when it names the exact bytes
diffed (a pair under an active object replacement is not sent). The
command comes from local configuration, as with filter.
.process: attributes select only a driver name, so cloning cannot
cause a process to run. gitattributes(5) has the full gate.
[4] The protocol is versioned and capability-negotiated, and extends
without breaking deployed processes: a process ignores request keys
it does not know, Git ignores trailing tokens on a hunk line so
fields can be appended, and new request forms arrive as capabilities
a process may decline. Announcing a capability Git did not request
aborts the command, the filter protocol's handshake rule. The
content-carrying request is the natural first extension; markers for
formatting-only changes and function or token boundaries are
candidates beyond it.