[PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets external tools provide hunks to git's diff and blame pipelines.

Over the past 18 years, git's diff pipeline accumulated many features that
operate on hunks: word diff, function context, color-moved, indent
heuristic, blame. External tools can replace the pipeline entirely
(diff.<driver>.command) or select among builtin algorithms
(diff.<driver>.algorithm), but there is no way for a tool to provide
line-change information into the pipeline. Tools that understand code
structure (tree-sitter parsers, format-aware analyzers, tools like
Difftastic and Mergiraf) must bypass git's pipeline and lose access to
everything downstream.

The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
capability negotiation, one tool invocation per git command. The tool
receives file pairs and returns hunk descriptors that git feeds into the
standard xdiff pipeline. All output features work normally.

Zero hunks with status=success means the tool considers the files
equivalent. git diff shows no output for the file, and git blame skips the
commit, attributing lines to earlier commits.

On error or tool crash, git falls back silently to the builtin diff
algorithm. The feature is opt-in via diff.<driver>.process and
.gitattributes; unconfigured files are unaffected.

The series includes git diff-process-normalize, a built-in tool that
compares files line by line ignoring whitespace (same logic as "git diff -w"
via xdiff_compare_lines):

[diff "cdiff"]
    process = git diff-process-normalize


A whitespace-only boolean flag could serve this specific case. The
subprocess protocol is more general, allowing any tool to participate
without further changes to git.

Michael Montalbo (5):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  diff: add long-running diff process via diff.<driver>.process
  blame: consult diff process for zero-hunk detection
  diff-process-normalize: add built-in whitespace normalizer

 Documentation/config/diff.adoc   |  10 +
 Documentation/gitattributes.adoc |  58 +++++
 Makefile                         |   2 +
 blame.c                          |  43 +++-
 builtin.h                        |   1 +
 builtin/diff-process-normalize.c | 143 ++++++++++
 diff-process.c                   | 203 +++++++++++++++
 diff-process.h                   |  28 ++
 diff.c                           |  25 ++
 git.c                            |   1 +
 t/t4080-diff-process.sh          | 430 +++++++++++++++++++++++++++++++
 userdiff.c                       |   7 +
 userdiff.h                       |   2 +
 xdiff-interface.c                |   7 +-
 xdiff/xdiff.h                    |  13 +
 xdiff/xdiffi.c                   |  98 ++++++-
 16 files changed, 1063 insertions(+), 8 deletions(-)
 create mode 100644 builtin/diff-process-normalize.c
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh


base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/2120
-- 
gitgitgadget

[PATCH 1/5] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 13 +++++++
 xdiff/xdiffi.c    | 98 +++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 114 insertions(+), 4 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index f043330f2a..9542c0bcc2 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..2ee6f1aae3 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based, matching unified diff convention.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +97,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm. */
+	const struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index 5455b4690d..7eca4ab4a1 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,108 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    const struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long total_old = 0, total_new = 0;
+
+	/*
+	 * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
+	 * them via xdl_cleanup_records().  The allocation is nrec + 2
+	 * elements; changed points one past the start (see xprepare.c).
+	 */
+	memset(xe->xdf1.changed - 1, 0,
+	       (xe->xdf1.nrec + 2) * sizeof(bool));
+	memset(xe->xdf2.changed - 1, 0,
+	       (xe->xdf2.nrec + 2) * sizeof(bool));
+
+	for (i = 0; i < nr_hunks; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0)
+			return -1;
+
+		/* Bounds check (1-based line numbers) */
+		if (h->old_count > 0 &&
+		    (h->old_start < 1 ||
+		     h->old_start + h->old_count - 1 > xe->xdf1.nrec))
+			return -1;
+		if (h->new_count > 0 &&
+		    (h->new_start < 1 ||
+		     h->new_start + h->new_count - 1 > xe->xdf2.nrec))
+			return -1;
+
+		/* Zero-count hunks: start must still be in [1, nrec+1] */
+		if (h->old_count == 0 &&
+		    (h->old_start < 1 || h->old_start > xe->xdf1.nrec + 1))
+			return -1;
+		if (h->new_count == 0 &&
+		    (h->new_start < 1 || h->new_start > xe->xdf2.nrec + 1))
+			return -1;
+
+		/* Ordering: no overlap with previous hunk */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+		total_old += h->old_count;
+		total_new += h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 */
+	if ((long)xe->xdf1.nrec - total_old !=
+	    (long)xe->xdf2.nrec - total_new)
+		return -1;
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
-
-		return -1;
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) < 0) {
+			/*
+			 * Invalid external hunks; fall back to the
+			 * builtin diff algorithm.  Re-runs
+			 * xdl_prepare_env() via xdl_do_diff().
+			 */
+			xdl_free_env(&xe);
+			if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
+				return -1;
+		}
+	} else {
+		if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
+			return -1;
 	}
+
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
-- 
gitgitgadget

[PATCH 2/5] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add a new per-driver configuration key that specifies the command
for a long-running diff process.

The name follows filter.<driver>.process: a long-running subprocess
that stays alive across files within a single git invocation.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index fe710a68bf..81c0bebcce 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH 3/5] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool receives file pairs and returns hunks describing which
lines changed.  Git feeds these hunks into the standard xdiff
pipeline, so all output features (word diff, function context,
color) work normally.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet.  On error, git falls back to the builtin diff algorithm.

Zero hunks with status=success means the tool considers the files
equivalent.  Git skips diff output for that file entirely.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   8 +
 Documentation/gitattributes.adoc |  40 ++++
 Makefile                         |   1 +
 diff-process.c                   | 203 +++++++++++++++++++
 diff-process.h                   |  28 +++
 diff.c                           |  25 +++
 t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
 7 files changed, 643 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..4ab5f60df6 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,14 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process.
+	The tool communicates via the pkt-line protocol and returns
+	hunks that are fed into Git's diff and blame pipelines.
+	If the tool returns zero hunks, the file is treated as
+	unchanged for both diff output and blame attribution.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f20041a323..cc724f8c63 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,46 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An external tool can provide content-aware line matching by
+setting `diff.<name>.process` to the command that runs
+the tool.  The tool is a long-running process that communicates via
+the pkt-line protocol (see
+linkgit:gitprotocol-long-running-process[5]).
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+The tool receives file pairs and returns hunk descriptors indicating
+which lines changed.  Git feeds these hunks into its standard diff
+pipeline, so all output features (word diff, function context,
+color) work normally.
+
+If the tool fails or returns an error, Git silently falls back to
+the builtin diff algorithm.  If the tool returns invalid hunks
+(out of bounds, overlapping), Git also falls back silently.
+
+The handshake negotiates `version=1` and `capability=hunks`.
+Per-file requests send `command=hunks` and `pathname=<path>`,
+followed by the old and new file content as packetized data.
+The tool responds with lines of the form
+`hunk <old_start> <old_count> <new_start> <new_count>`
+(1-based line numbers), a flush packet, and `status=success`.
+
+If the tool returns zero hunks with `status=success`, Git treats
+the file as having no changes and produces no diff output.
+
+Tools should ignore unknown keys in the per-file request to
+remain forward-compatible.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index cedc234173..22900368dd 100644
--- a/Makefile
+++ b/Makefile
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..7b0f0e1f7e
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,203 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain content-aware hunks.
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * Zero hunks with status=success means the tool considers the
+ * files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int subprocess_map_initialized;
+static struct hashmap subprocess_map;
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		(struct diff_subprocess *)subprocess;
+
+	/* Uses dying pkt-line variant, same as convert.c filters. */
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *find_or_start_process(const char *cmd)
+{
+	struct diff_subprocess *entry;
+
+	if (!subprocess_map_initialized) {
+		subprocess_map_initialized = 1;
+		hashmap_init(&subprocess_map, cmd2process_cmp, NULL, 0);
+	}
+
+	entry = (struct diff_subprocess *)
+		subprocess_find_entry(&subprocess_map, cmd);
+	if (entry)
+		return entry;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start(&subprocess_map, &entry->subprocess,
+			     cmd, start_diff_process_fn)) {
+		free(entry);
+		return NULL;
+	}
+
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret;
+
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	else
+		ret = 0;
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	hunk->old_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->old_count = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_count = strtol(line, &end, 10);
+	if (end == line || *end != '\0')
+		return -1;
+
+	return 0;
+}
+
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	if (!drv || !drv->process)
+		return -1;
+
+	backend = find_or_start_process(drv->process);
+	if (!backend)
+		return -1;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return -1;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto error;
+
+	if (strcmp(status.buf, "success")) {
+		if (!strcmp(status.buf, "abort"))
+			backend->supported_capabilities &= ~CAP_HUNKS;
+		goto error;
+	}
+
+	*hunks_out = hunks;
+	*nr_hunks_out = nr_hunks;
+	strbuf_release(&status);
+	return 0;
+
+error:
+	free(hunks);
+	strbuf_release(&status);
+	return -1;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..4c84951e02
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,28 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+struct userdiff_driver;
+struct xdl_hunk;
+
+/*
+ * Query a diff process for hunks describing the changes
+ * between old_buf and new_buf.
+ *
+ * The backend is a long-running subprocess configured via
+ * diff.<driver>.process.  It receives file content via
+ * pkt-line and returns hunks with 1-based line numbers.
+ *
+ * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
+ * array (caller must free) and returns 0.
+ *
+ * On failure, returns -1.  The caller should fall back to the
+ * builtin diff algorithm.
+ */
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 397e38b41c..c5e7c329b2 100644
--- a/diff.c
+++ b/diff.c
@@ -25,7 +25,9 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
+#include "trace2.h"
 #include "hashmap.h"
 #include "mem-pool.h"
 #include "merge-ll.h"
@@ -3991,6 +3993,7 @@ static void builtin_diff(const char *name_a,
 		xpparam_t xpp;
 		xdemitconf_t xecfg;
 		struct emit_callback ecbdata;
+		struct xdl_hunk *ext_hunks = NULL;
 		unsigned ws_rule;
 		const struct userdiff_funcname *pe;
 
@@ -4031,6 +4034,27 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (!o->ignore_driver_algorithm &&
+		    one->driver && one->driver->process) {
+			size_t ext_hunks_nr = 0;
+			if (!diff_process_get_hunks(
+				    one->driver, name_a,
+				    mf1.ptr, mf1.size,
+				    mf2.ptr, mf2.size,
+				    &ext_hunks, &ext_hunks_nr)) {
+				if (!ext_hunks_nr)
+					goto free_ab_and_return;
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = ext_hunks_nr;
+			} else {
+				trace2_data_string("diff",
+						   o->repo,
+						   "diff-process-fallback",
+						   name_a);
+			}
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4111,6 +4135,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(ext_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..6f49f4e66b
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,338 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+if test_have_prereq PYTHON
+then
+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+fi
+
+#
+# A single parametric diff process.
+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
+#
+# Modes:
+#   whole-file  - report all lines as changed (default)
+#   fixed-hunk  - always report hunk 5 2 5 2
+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
+#   zero-hunk   - return zero hunks (files considered equivalent)
+#   error       - return status=error for every request
+#   abort       - return status=abort for every request
+#   crash       - read one request then exit without responding
+#
+setup_backend () {
+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
+	import sys, os
+
+	def read_pkt():
+	    hdr = sys.stdin.buffer.read(4)
+	    if len(hdr) < 4: return None
+	    length = int(hdr, 16)
+	    if length == 0: return ""
+	    data = sys.stdin.buffer.read(length - 4)
+	    return data.decode().rstrip("\n")
+
+	def write_pkt(line):
+	    data = (line + "\n").encode()
+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    sys.stdout.buffer.flush()
+
+	def write_flush():
+	    sys.stdout.buffer.write(b"0000")
+	    sys.stdout.buffer.flush()
+
+	def read_content():
+	    chunks = []
+	    while True:
+	        hdr = sys.stdin.buffer.read(4)
+	        if len(hdr) < 4: break
+	        length = int(hdr, 16)
+	        if length == 0: break
+	        chunks.append(sys.stdin.buffer.read(length - 4))
+	    return b"".join(chunks)
+
+	mode = "whole-file"
+	logfile = None
+	for arg in sys.argv[1:]:
+	    if arg.startswith("--mode="):
+	        mode = arg[7:]
+	    elif arg.startswith("--log="):
+	        logfile = open(arg[6:], "a")
+
+	def log(msg):
+	    if logfile:
+	        logfile.write(msg + "\n")
+	        logfile.flush()
+
+	# Handshake
+	assert read_pkt() == "git-diff-client"
+	assert read_pkt() == "version=1"
+	read_pkt()
+	write_pkt("git-diff-server")
+	write_pkt("version=1")
+	write_flush()
+	while True:
+	    p = read_pkt()
+	    if p == "": break
+	write_pkt("capability=hunks")
+	write_flush()
+
+	log("ready")
+
+	while True:
+	    cmd = None
+	    pathname = None
+	    while True:
+	        p = read_pkt()
+	        if p is None: sys.exit(0)
+	        if p == "": break
+	        if p.startswith("command="): cmd = p.split("=",1)[1]
+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
+	    if cmd is None: sys.exit(0)
+	    old = read_content()
+	    new = read_content()
+	    log(f"command={cmd} pathname={pathname}")
+
+	    if mode == "error":
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	        continue
+
+	    if mode == "abort":
+	        write_flush()
+	        write_pkt("status=abort")
+	        write_flush()
+	        continue
+
+	    if mode == "crash":
+	        sys.exit(1)
+
+	    if cmd == "hunks":
+	        if mode == "fixed-hunk":
+	            write_pkt("hunk 5 2 5 2")
+	        elif mode == "bad-hunk":
+	            write_pkt("hunk 999 1 999 1")
+	        elif mode == "zero-hunk":
+	            pass
+	        else:
+	            ol = len(old.split(b"\n"))
+	            nl = len(new.split(b"\n"))
+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	        write_flush()
+	        write_pkt("status=success")
+	        write_flush()
+	    else:
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	PYEOF
+	write_script diff-process-backend <<-SHEOF
+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
+	SHEOF
+}
+
+BACKEND="./diff-process-backend"
+
+test_expect_success PYTHON 'setup' '
+	setup_backend &&
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+	git commit -m "initial"
+'
+
+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+	git commit -m "add boundary.c" &&
+
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	grep "^-OLD5" actual &&
+	grep "^-OLD6" actual &&
+	grep "^+NEW5" actual &&
+	grep "^+NEW6" actual &&
+	! grep "^-OLD9" actual &&
+	! grep "^-OLD10" actual &&
+	! grep "^+NEW9" actual &&
+	! grep "^+NEW10" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool error status' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual &&
+	# Fallback produces the full builtin diff (both change regions).
+	grep "^-OLD5" actual &&
+	grep "^+NEW5" actual &&
+	grep "^-OLD9" actual &&
+	grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	grep "command=hunks pathname=boundary.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual &&
+	grep "^-OLD5" actual &&
+	grep "^+NEW5" actual &&
+	grep "^-OLD9" actual &&
+	grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual &&
+	grep "^-OLD5" actual &&
+	grep "^+NEW5" actual &&
+	grep "^-OLD9" actual &&
+	grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process abort disables for session' '
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add abort1.c abort2.c &&
+	git commit -m "add abort files" &&
+
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- abort1.c abort2.c >actual &&
+	# Both files should still produce diff output via fallback.
+	grep "return 10" actual &&
+	grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool,
+	# so the log should have exactly one entry, not two.
+	grep "command=hunks" backend.log >matches &&
+	test_line_count = 1 matches
+'
+
+test_expect_success PYTHON 'diff process handles multiple files' '
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 1; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 2; }
+	EOF
+	git add multi1.c multi2.c &&
+	git commit -m "add multi files" &&
+
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 10; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- multi1.c multi2.c >actual &&
+	grep "return 10" actual &&
+	grep "return 20" actual &&
+	grep "pathname=multi1.c" backend.log &&
+	grep "pathname=multi2.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process with --word-diff' '
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+	git commit -m "add worddiff.c" &&
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND" \
+		diff --word-diff worddiff.c >actual &&
+	grep "\[-1;-\]" actual &&
+	grep "{+999;+}" actual
+'
+
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process works with git log -p' '
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+	git commit -m "add logtest.c" &&
+
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		log -1 -p -- logtest.c >actual &&
+	grep "return 2" actual &&
+	grep "command=hunks pathname=logtest.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add zerohunk.c &&
+	git commit -m "add zerohunk.c" &&
+
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		diff zerohunk.c >actual &&
+	test_must_be_empty actual
+'
+
+test_done
-- 
gitgitgadget

[PATCH 4/5] blame: consult diff process for zero-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns zero hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc |  3 +++
 blame.c                          | 43 +++++++++++++++++++++++++++++---
 t/t4080-diff-process.sh          | 32 ++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index cc724f8c63..7d66fa3aa1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -857,6 +857,9 @@ The tool responds with lines of the form
 
 If the tool returns zero hunks with `status=success`, Git treats
 the file as having no changes and produces no diff output.
+`git blame` also consults the diff process and skips commits
+where it reports zero hunks, attributing lines to earlier commits
+instead.
 
 Tools should ignore unknown keys in the per-file request to
 remain forward-compatible.
diff --git a/blame.c b/blame.c
index a3c49d132e..8a5f14db7a 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "userdiff.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -315,16 +317,47 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data,
+		      int xdl_opts, struct index_state *istate,
+		      const char *path)
 {
 	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
+	struct xdl_hunk *ext_hunks = NULL;
+	int ret;
 
 	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+
+	if (path && istate) {
+		struct userdiff_driver *drv;
+		drv = userdiff_find_by_path(istate, path);
+		if (drv && drv->process) {
+			size_t nr = 0;
+			if (!diff_process_get_hunks(drv, path,
+						    file_a->ptr, file_a->size,
+						    file_b->ptr, file_b->size,
+						    &ext_hunks, &nr)) {
+				if (!nr) {
+					/*
+					 * Zero hunks: the diff process
+					 * considers these files equivalent.
+					 * Skip so blame looks past this
+					 * commit.
+					 */
+					return 0;
+				}
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = nr;
+			}
+		}
+	}
+
+	ret = xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	free(ext_hunks);
+	return ret;
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1961,7 +1994,8 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts,
+		       sb->revs->diffopt.repo->index, target->path))
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
@@ -2114,7 +2148,8 @@ static void find_copy_in_blob(struct blame_scoreboard *sb,
 	 * file_p partially may match that image.
 	 */
 	memset(split, 0, sizeof(struct blame_entry [3]));
-	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts))
+	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts,
+		       NULL, NULL))
 		die("unable to generate diff (%s)",
 		    oid_to_hex(&parent->commit->object.oid));
 	/* remainder, if any, all match the preimage */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 6f49f4e66b..5ed644b786 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -335,4 +335,36 @@ test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
 	test_must_be_empty actual
 '
 
+test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without zero-hunk mode, blame attributes the change.
+	git blame blame.c >without &&
+	grep "$BLAME_COMMIT" without &&
+
+	# With zero-hunk mode, the process considers the files equivalent
+	# and blame skips the reformat commit.
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		blame blame.c >with &&
+	! grep "$BLAME_COMMIT" with
+'
+
+
 test_done
-- 
gitgitgadget

[PATCH 5/5] diff-process-normalize: add built-in whitespace normalizer

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add git diff-process-normalize, a built-in diff process that
detects whitespace-only changes.  It compares files line by line
using xdiff_compare_lines() with XDF_IGNORE_WHITESPACE (same
logic as "git diff -w").  If all lines match, it returns zero
hunks; otherwise it returns an error so git falls back to the
builtin diff algorithm.

    [diff "cdiff"]
        process = git diff-process-normalize

Update documentation to describe zero-hunk behavior for diff
and blame, and document the built-in normalize tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   2 +
 Documentation/gitattributes.adoc |  15 ++++
 Makefile                         |   1 +
 builtin.h                        |   1 +
 builtin/diff-process-normalize.c | 143 +++++++++++++++++++++++++++++++
 git.c                            |   1 +
 t/t4080-diff-process.sh          |  60 +++++++++++++
 7 files changed, 223 insertions(+)
 create mode 100644 builtin/diff-process-normalize.c

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 4ab5f60df6..475736c6ed 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -224,6 +224,8 @@ endif::git-diff[]
 	hunks that are fed into Git's diff and blame pipelines.
 	If the tool returns zero hunks, the file is treated as
 	unchanged for both diff output and blame attribution.
+	Git provides `git diff-process-normalize` as a built-in
+	tool that detects whitespace-only changes.
 	See linkgit:gitattributes[5] for details.
 
 `diff.indentHeuristic`::
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 7d66fa3aa1..3f1d7affd8 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -861,6 +861,21 @@ the file as having no changes and produces no diff output.
 where it reports zero hunks, attributing lines to earlier commits
 instead.
 
+Git ships with a built-in diff process, `git diff-process-normalize`,
+that detects whitespace-only changes.  Files whose only differences
+are whitespace produce zero hunks; files with non-whitespace changes
+fall back to the builtin diff algorithm.  To use it:
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = git diff-process-normalize
+----------------------------------------------------------------
+
+This is useful after running a code formatter: `git diff` shows
+no output for files that only had whitespace changes,
+`git blame` skips whitespace-only commits automatically without
+requiring a `.git-blame-ignore-revs` file.
+
 Tools should ignore unknown keys in the per-file request to
 remain forward-compatible.
 
diff --git a/Makefile b/Makefile
index 22900368dd..01acfaf7b8 100644
--- a/Makefile
+++ b/Makefile
@@ -1409,6 +1409,7 @@ BUILTIN_OBJS += builtin/diagnose.o
 BUILTIN_OBJS += builtin/diff-files.o
 BUILTIN_OBJS += builtin/diff-index.o
 BUILTIN_OBJS += builtin/diff-pairs.o
+BUILTIN_OBJS += builtin/diff-process-normalize.o
 BUILTIN_OBJS += builtin/diff-tree.o
 BUILTIN_OBJS += builtin/diff.o
 BUILTIN_OBJS += builtin/difftool.o
diff --git a/builtin.h b/builtin.h
index 235c51f30e..c713a0417f 100644
--- a/builtin.h
+++ b/builtin.h
@@ -178,6 +178,7 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repos
 int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);
+int cmd_diff_process_normalize(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff_tree(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_difftool(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_env__helper(int argc, const char **argv, const char *prefix, struct repository *repo);
diff --git a/builtin/diff-process-normalize.c b/builtin/diff-process-normalize.c
new file mode 100644
index 0000000000..1580f6b7d9
--- /dev/null
+++ b/builtin/diff-process-normalize.c
@@ -0,0 +1,143 @@
+/*
+ * Built-in diff process that returns zero hunks for files whose
+ * only differences are whitespace, and status=error otherwise.
+ * See diff-process.c for the protocol and gitattributes(5) for usage.
+ *
+ * Uses xdiff_compare_lines() with XDF_IGNORE_WHITESPACE to compare
+ * lines, giving the same whitespace handling as "git diff -w".
+ */
+
+#include "builtin.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff-interface.h"
+
+/*
+ * Read a single pkt-line.  Returns 1 for data, 0 for flush, -1 for EOF.
+ */
+static int read_pkt(int fd, struct strbuf *line)
+{
+	int len;
+	char *data;
+
+	if (packet_read_line_gently(fd, &len, &data) < 0)
+		return -1;
+	if (!data || !len)
+		return 0; /* flush */
+	strbuf_reset(line);
+	strbuf_add(line, data, len);
+	strbuf_rtrim(line);
+	return 1;
+}
+
+/*
+ * Read packetized content until a flush packet.
+ */
+static int read_content(int fd, struct strbuf *out)
+{
+	strbuf_reset(out);
+	if (read_packetized_to_strbuf(fd, out, PACKET_READ_GENTLE_ON_EOF) < 0)
+		return -1;
+	return 0;
+}
+
+/*
+ * Compare two buffers line by line using xdiff_compare_lines() with
+ * XDF_IGNORE_WHITESPACE (same logic as "git diff -w").
+ * Returns 1 if all lines match, 0 otherwise.
+ */
+static int whitespace_equivalent(const char *a, long size_a,
+				 const char *b, long size_b)
+{
+	const char *ea = a + size_a;
+	const char *eb = b + size_b;
+
+	while (a < ea && b < eb) {
+		const char *eol_a = memchr(a, '\n', ea - a);
+		const char *eol_b = memchr(b, '\n', eb - b);
+		long len_a = (eol_a ? eol_a : ea) - a;
+		long len_b = (eol_b ? eol_b : eb) - b;
+
+		if (!xdiff_compare_lines(a, len_a, b, len_b,
+					 XDF_IGNORE_WHITESPACE))
+			return 0;
+
+		a += len_a + (eol_a ? 1 : 0);
+		b += len_b + (eol_b ? 1 : 0);
+	}
+
+	/* Both sides must be exhausted */
+	return a >= ea && b >= eb;
+}
+
+int cmd_diff_process_normalize(int argc UNUSED, const char **argv UNUSED,
+			       const char *prefix UNUSED,
+			       struct repository *repo UNUSED)
+{
+	struct strbuf line = STRBUF_INIT;
+	struct strbuf old_content = STRBUF_INIT;
+	struct strbuf new_content = STRBUF_INIT;
+	int ret;
+
+	/* Handshake: read client greeting */
+	ret = read_pkt(0, &line);
+	if (ret <= 0 || strcmp(line.buf, "git-diff-client"))
+		return 1;
+	ret = read_pkt(0, &line);
+	if (ret <= 0 || strcmp(line.buf, "version=1"))
+		return 1;
+	read_pkt(0, &line); /* flush */
+
+	/* Send server greeting */
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Read client capabilities until flush */
+	while ((ret = read_pkt(0, &line)) > 0)
+		; /* consume */
+
+	/* Send our capabilities */
+	packet_write_fmt(1, "capability=hunks\n");
+	packet_flush(1);
+
+	/* Main loop: process file pairs */
+	for (;;) {
+		int have_command = 0;
+
+		/* Read request headers until flush */
+		while ((ret = read_pkt(0, &line)) > 0) {
+			if (starts_with(line.buf, "command="))
+				have_command = 1;
+		}
+		if (ret < 0)
+			break; /* EOF: client closed connection */
+		if (!have_command)
+			break;
+
+		/* Read old file content */
+		if (read_content(0, &old_content) < 0)
+			break;
+		/* Read new file content */
+		if (read_content(0, &new_content) < 0)
+			break;
+
+		if (whitespace_equivalent(old_content.buf, old_content.len,
+					  new_content.buf, new_content.len)) {
+			/* Whitespace-only differences */
+			packet_flush(1); /* zero hunks */
+			packet_write_fmt(1, "status=success\n");
+			packet_flush(1);
+		} else {
+			/* Non-whitespace differences: fall back */
+			packet_flush(1);
+			packet_write_fmt(1, "status=error\n");
+			packet_flush(1);
+		}
+	}
+
+	strbuf_release(&line);
+	strbuf_release(&old_content);
+	strbuf_release(&new_content);
+	return 0;
+}
diff --git a/git.c b/git.c
index 5a40eab8a2..6239240b02 100644
--- a/git.c
+++ b/git.c
@@ -568,6 +568,7 @@ static struct cmd_struct commands[] = {
 	{ "diff-files", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },
 	{ "diff-index", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },
 	{ "diff-pairs", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },
+	{ "diff-process-normalize", cmd_diff_process_normalize, NO_PARSEOPT },
 	{ "diff-tree", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },
 	{ "difftool", cmd_difftool, RUN_SETUP_GENTLY },
 	{ "fast-export", cmd_fast_export, RUN_SETUP },
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 5ed644b786..a6fa1df456 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -366,5 +366,65 @@ test_expect_success PYTHON 'blame skips commits with zero hunks from diff proces
 	! grep "$BLAME_COMMIT" with
 '
 
+NORMALIZE="git diff-process-normalize"
+
+test_expect_success 'diff-process-normalize setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+	test_commit normalize-base
+'
+
+test_expect_success 'diff-process-normalize suppresses whitespace-only changes' '
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add ws.c &&
+	git commit -m "add ws.c" &&
+
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+
+	git -c diff.cdiff.process="$NORMALIZE" \
+		diff ws.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success 'diff-process-normalize falls back on non-whitespace changes' '
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+
+	int added_function(void)
+	{
+	    return 99;
+	}
+	EOF
+
+	git -c diff.cdiff.process="$NORMALIZE" \
+		diff ws.c >actual &&
+	grep "added_function" actual
+'
+
+test_expect_success 'diff-process-normalize falls back on mixed whitespace and real changes' '
+	cat >ws.c <<-\EOF &&
+	int main(void)
+	{
+	        return 42;
+	}
+	EOF
+
+	git -c diff.cdiff.process="$NORMALIZE" \
+		diff ws.c >actual &&
+	grep "return 42" actual
+'
 
 test_done
-- 
gitgitgadget

Re: [PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> This series adds diff.<driver>.process, a long-running subprocess protocol
> that lets external tools provide hunks to git's diff and blame pipelines.
>
> Over the past 18 years, git's diff pipeline accumulated many features that
> operate on hunks: word diff, function context, color-moved, indent
> heuristic, blame. External tools can replace the pipeline entirely
> (diff.<driver>.command) or select among builtin algorithms
> (diff.<driver>.algorithm), but there is no way for a tool to provide
> line-change information into the pipeline. Tools that understand code
> structure (tree-sitter parsers, format-aware analyzers, tools like
> Difftastic and Mergiraf) must bypass git's pipeline and lose access to
> everything downstream.
>
> The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
> capability negotiation, one tool invocation per git command. The tool
> receives file pairs and returns hunk descriptors that git feeds into the
> standard xdiff pipeline. All output features work normally.
>
> Zero hunks with status=success means the tool considers the files
> equivalent. git diff shows no output for the file, and git blame skips the
> commit, attributing lines to earlier commits.
>
> On error or tool crash, git falls back silently to the builtin diff
> algorithm. The feature is opt-in via diff.<driver>.process and
> .gitattributes; unconfigured files are unaffected.
>
> The series includes git diff-process-normalize, a built-in tool that
> compares files line by line ignoring whitespace (same logic as "git diff -w"
> via xdiff_compare_lines):

Interesting.

If the goal is purely to normalize content before comparison
(e.g. stripping comments or canonicalizing formatting), we already
have the `textconv` mechanism.  While `textconv` is a "one-shot"
per-file process, it is significantly simpler.

I suspect, however, that the primary focus here is to allow external
tools to provide structural alignment (e.g. for AST- aware diffs
like Difftastic or Mergiraf) without losing the original content in
the display.  Unlike `textconv`, which transforms the text the user
sees, this protocol lets the display remain identical to the source
while using a custom engine for the line-matching logic.

If that is the intent, it should be stated more explicitly in the
documentation and commit messages.  The "whitespace-normalize"
demonstration in [PATCH 5/5] is misleading because it's exactly the
case where `textconv` would be sufficient.

I am afraid that the use of a long-running subprocess for every
diff/blame invocation adds significant complexity and overhead.  In
particular, wouldn't the `blame` implementation performs a
round-trip to the subprocess for every commit in the history?  Even
with a persistent process, the overhead of serializing and
deserializing the entire file content twice (old and new) for every
commit could be prohibitive for large files or deep histories.

So, I dunno.

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +/*
> + * Populate the changed[] arrays from externally supplied hunks,
> + * bypassing the diff algorithm.  Validates that hunks are in order,
> + * non-overlapping, and within bounds.
> + *
> + * Returns 0 on success, -1 on validation failure.
> + */
> +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
> +					    const struct xdl_hunk *hunks,
> +					    size_t nr_hunks)
> +{
> +	size_t i;
> +	long j, prev_old_end = 0, prev_new_end = 0;
> +	long total_old = 0, total_new = 0;
> +
> +	/*
> +	 * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
> +	 * them via xdl_cleanup_records().  The allocation is nrec + 2
> +	 * elements; changed points one past the start (see xprepare.c).
> +	 */
> +	memset(xe->xdf1.changed - 1, 0,
> +	       (xe->xdf1.nrec + 2) * sizeof(bool));
> +	memset(xe->xdf2.changed - 1, 0,
> +	       (xe->xdf2.nrec + 2) * sizeof(bool));

This, especially the starting offset of -1, looks horrible.  The
internal layout of xdfenv_t might happen to match the way the above
code expects, which is how xdl_prepare_ctx() may have give you, but
it somehow feels brittle.  I guess the assumption that changed[]
does not point at the beginning of the allocated area (e.g., it is a
no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
it cannot be helped.  Sigh.

>  int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
>  	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
>  	xdchange_t *xscr;
>  	xdfenv_t xe;
>  	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
>  
> -	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
> -
> -		return -1;
> +	if (xpp->external_hunks) {
> +		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
> +			return -1;
> +		if (xdl_populate_hunks_from_external(&xe,
> +						     xpp->external_hunks,
> +						     xpp->external_hunks_nr) < 0) {
> +			/*
> +			 * Invalid external hunks; fall back to the
> +			 * builtin diff algorithm.  Re-runs
> +			 * xdl_prepare_env() via xdl_do_diff().
> +			 */
> +			xdl_free_env(&xe);
> +			if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> +				return -1;

If the external tool keeps sending bogus hunks, silently falling
back to what we would have done if there weren't any external stuff
may be necessary to pleasantly keep using Git, but two and a half
short comments here.

 (1) "What we would have done" is exactly the same as what appears
     in the corresponding "else" block.  Can we make sure that we do
     not have to keep updating both copies in the future with some
     code rearrangement?

 (2) The writer of the external tool may want to see some trace of
     warning under certain flags when a failure of the tool forces
     the receiving end to fallback.

 (3) If the tool throws too many broken replies, perhaps we want to
     disable it automatically?

> +		}
> +	} else {
> +		if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> +			return -1;
>  	}
> +
>  	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
>  	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
>  	    xdl_build_script(&xe, &xscr) < 0) {

Re: [PATCH 0/5] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo]

On Thu, May 21, 2026 at 10:29 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > This series adds diff.<driver>.process, a long-running subprocess protocol
> > that lets external tools provide hunks to git's diff and blame pipelines.
> >
> > Over the past 18 years, git's diff pipeline accumulated many features that
> > operate on hunks: word diff, function context, color-moved, indent
> > heuristic, blame. External tools can replace the pipeline entirely
> > (diff.<driver>.command) or select among builtin algorithms
> > (diff.<driver>.algorithm), but there is no way for a tool to provide
> > line-change information into the pipeline. Tools that understand code
> > structure (tree-sitter parsers, format-aware analyzers, tools like
> > Difftastic and Mergiraf) must bypass git's pipeline and lose access to
> > everything downstream.
> >
> > The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
> > capability negotiation, one tool invocation per git command. The tool
> > receives file pairs and returns hunk descriptors that git feeds into the
> > standard xdiff pipeline. All output features work normally.
> >
> > Zero hunks with status=success means the tool considers the files
> > equivalent. git diff shows no output for the file, and git blame skips the
> > commit, attributing lines to earlier commits.
> >
> > On error or tool crash, git falls back silently to the builtin diff
> > algorithm. The feature is opt-in via diff.<driver>.process and
> > .gitattributes; unconfigured files are unaffected.
> >
> > The series includes git diff-process-normalize, a built-in tool that
> > compares files line by line ignoring whitespace (same logic as "git diff -w"
> > via xdiff_compare_lines):
>
> Interesting.
>
> If the goal is purely to normalize content before comparison
> (e.g. stripping comments or canonicalizing formatting), we already
> have the `textconv` mechanism.  While `textconv` is a "one-shot"
> per-file process, it is significantly simpler.
>
> I suspect, however, that the primary focus here is to allow external
> tools to provide structural alignment (e.g. for AST- aware diffs
> like Difftastic or Mergiraf) without losing the original content in
> the display.  Unlike `textconv`, which transforms the text the user
> sees, this protocol lets the display remain identical to the source
> while using a custom engine for the line-matching logic.
>
> If that is the intent, it should be stated more explicitly in the
> documentation and commit messages.  The "whitespace-normalize"
> demonstration in [PATCH 5/5] is misleading because it's exactly the
> case where `textconv` would be sufficient.
>

Thank you for looking at this.

Yes, you have correctly identified the primary focus. My intention with the
whitespace normalization example was to provide a kind of "hello world"
diff process that would showcase how such a tool could interact with
the pipeline further down (i.e., blame vs diff output). However, I do agree
that it is a confusing example because it seems to clash with something
`textconv` already provides. I will update messaging across the series to
make the true intention of these changes more clear.

> I am afraid that the use of a long-running subprocess for every
> diff/blame invocation adds significant complexity and overhead.  In
> particular, wouldn't the `blame` implementation performs a
> round-trip to the subprocess for every commit in the history?  Even
> with a persistent process, the overhead of serializing and
> deserializing the entire file content twice (old and new) for every
> commit could be prohibitive for large files or deep histories.
>
> So, I dunno.

I hear you on this point. Anecdotally, my measurement of running blame
on diff.c:

Performance:
  without process:  0.57s
  with process:       0.67s

Blame attribution:
  without process:  726 unique commits
  with process:     721 unique commits (5 whitespace-only skipped)

Skipped commits:
  0ea7d5b6f8 diff.c: fix some recent whitespace style violations
  2775d92c53 diff.c: stylefix
  4b25d091ba Fix a bunch of pointer declarations (codestyle)
  a6080a0a44 War on whitespace
  eeefa7c90e Style fixes, add a space after if/for/while.

I was imagining we could potentially optimize performance by extending the
protocol to enable passing OIDs as a capability so tools could read objects
directly without needing any serialization/deserialization.

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Michael Montalbo]

On Thu, May 21, 2026 at 10:29 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +/*
> > + * Populate the changed[] arrays from externally supplied hunks,
> > + * bypassing the diff algorithm.  Validates that hunks are in order,
> > + * non-overlapping, and within bounds.
> > + *
> > + * Returns 0 on success, -1 on validation failure.
> > + */
> > +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
> > +                                         const struct xdl_hunk *hunks,
> > +                                         size_t nr_hunks)
> > +{
> > +     size_t i;
> > +     long j, prev_old_end = 0, prev_new_end = 0;
> > +     long total_old = 0, total_new = 0;
> > +
> > +     /*
> > +      * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
> > +      * them via xdl_cleanup_records().  The allocation is nrec + 2
> > +      * elements; changed points one past the start (see xprepare.c).
> > +      */
> > +     memset(xe->xdf1.changed - 1, 0,
> > +            (xe->xdf1.nrec + 2) * sizeof(bool));
> > +     memset(xe->xdf2.changed - 1, 0,
> > +            (xe->xdf2.nrec + 2) * sizeof(bool));
>
> This, especially the starting offset of -1, looks horrible.  The
> internal layout of xdfenv_t might happen to match the way the above
> code expects, which is how xdl_prepare_ctx() may have give you, but
> it somehow feels brittle.  I guess the assumption that changed[]
> does not point at the beginning of the allocated area (e.g., it is a
> no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
> it cannot be helped.  Sigh.
>

Agreed it is ugly. I wanted to make sure the entire changed[] including
sentinels were clear as a defensive measure for downstream callers
(xdl_change_compact). I agree this results in something that is ugly
and brittle, but in the end I thought it was superior to relying on the
fact that upstream zeroes the entire changed[] array. Maybe if the
comment was more explicit about why this is happening it would be
helpful?

    /*
     * Clear changed[] arrays including sentinels.
     * xdl_prepare_env() may have dirtied them via
     * xdl_cleanup_records(), and xdl_change_compact() reads
     * the sentinel at changed[-1] during backward scans.
     */

> >  int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
> >            xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
> >       xdchange_t *xscr;
> >       xdfenv_t xe;
> >       emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
> >
> > -     if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
> > -
> > -             return -1;
> > +     if (xpp->external_hunks) {
> > +             if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
> > +                     return -1;
> > +             if (xdl_populate_hunks_from_external(&xe,
> > +                                                  xpp->external_hunks,
> > +                                                  xpp->external_hunks_nr) < 0) {
> > +                     /*
> > +                      * Invalid external hunks; fall back to the
> > +                      * builtin diff algorithm.  Re-runs
> > +                      * xdl_prepare_env() via xdl_do_diff().
> > +                      */
> > +                     xdl_free_env(&xe);
> > +                     if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> > +                             return -1;
>
> If the external tool keeps sending bogus hunks, silently falling
> back to what we would have done if there weren't any external stuff
> may be necessary to pleasantly keep using Git, but two and a half
> short comments here.
>
>  (1) "What we would have done" is exactly the same as what appears
>      in the corresponding "else" block.  Can we make sure that we do
>      not have to keep updating both copies in the future with some
>      code rearrangement?
>

How about something like this:

  if (xpp->external_hunks) {
      if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
          return -1;
      if (xdl_populate_hunks_from_external(&xe,
                       xpp->external_hunks,
                       xpp->external_hunks_nr) == 0)
          goto diff_done;
      xdl_free_env(&xe);
  }

  if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
      return -1;

  diff_done:

>  (2) The writer of the external tool may want to see some trace of
>      warning under certain flags when a failure of the tool forces
>      the receiving end to fallback.
>

In diff.c how about we emit a warning rather than a trace on
fallback:

    warning(_("diff process failed for '%s',"
              " falling back to builtin diff"),
            name_a);

>  (3) If the tool throws too many broken replies, perhaps we want to
>      disable it automatically?
>

For the RFC I wanted to keep it simple, but I definitely agree. A configurable
failure policy makes a lot of sense to me (e.g., disable after N failures).

> > +             }
> > +     } else {
> > +             if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
> > +                     return -1;
> >       }
> > +
> >       if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
> >           xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
> >           xdl_build_script(&xe, &xscr) < 0) {

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Junio C Hamano]

Michael Montalbo <mmontalbo@gmail.com> writes:

>> > +      * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
>> > +      * them via xdl_cleanup_records().  The allocation is nrec + 2
>> > +      * elements; changed points one past the start (see xprepare.c).
>> > +      */
>> > +     memset(xe->xdf1.changed - 1, 0,
>> > +            (xe->xdf1.nrec + 2) * sizeof(bool));
>> > +     memset(xe->xdf2.changed - 1, 0,
>> > +            (xe->xdf2.nrec + 2) * sizeof(bool));
>>
>> This, especially the starting offset of -1, looks horrible.  The
>> internal layout of xdfenv_t might happen to match the way the above
>> code expects, which is how xdl_prepare_ctx() may have give you, but
>> it somehow feels brittle.  I guess the assumption that changed[]
>> does not point at the beginning of the allocated area (e.g., it is a
>> no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
>> it cannot be helped.  Sigh.
>>
>
> Agreed it is ugly. I wanted to make sure the entire changed[] including
> sentinels were clear as a defensive measure for downstream callers
> (xdl_change_compact). I agree this results in something that is ugly
> and brittle, but in the end I thought it was superior to relying on the
> fact that upstream zeroes the entire changed[] array. Maybe if the
> comment was more explicit about why this is happening it would be
> helpful?

Perhaps make these memset() into calls to a helper function that is
defined in xdiff/xprepare.c with a descriptive name and placed near
where xdl_prepare_ctx() is.  That way, the patch in question does
not even have to expose the strangeness of changed[] (i.e., it has 2
more elements than it would normally contain to make the memory
region for changed[-1] and changed[N] valid, and freeing it requires
free(changed-1)) to the code path.  It only needs to say "Hey, I am
clearing changed[] arrays because of XXX" without having to say "by
the way, the memory layout of changed[] is strange this way", the
latter of which is not exactly of interest for readers of this code.

>     /*
>      * Clear changed[] arrays including sentinels.
>      * xdl_prepare_env() may have dirtied them via
>      * xdl_cleanup_records(), and xdl_change_compact() reads
>      * the sentinel at changed[-1] during backward scans.
>      */

And this belongs in xdiff/xprepare.c near that new helper function.

Re: [PATCH 1/5] xdiff: support external hunks via xpparam_t

[Michael Montalbo]

On Sun, May 24, 2026 at 1:50 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Michael Montalbo <mmontalbo@gmail.com> writes:
>
> >> > +      * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
> >> > +      * them via xdl_cleanup_records().  The allocation is nrec + 2
> >> > +      * elements; changed points one past the start (see xprepare.c).
> >> > +      */
> >> > +     memset(xe->xdf1.changed - 1, 0,
> >> > +            (xe->xdf1.nrec + 2) * sizeof(bool));
> >> > +     memset(xe->xdf2.changed - 1, 0,
> >> > +            (xe->xdf2.nrec + 2) * sizeof(bool));
> >>
> >> This, especially the starting offset of -1, looks horrible.  The
> >> internal layout of xdfenv_t might happen to match the way the above
> >> code expects, which is how xdl_prepare_ctx() may have give you, but
> >> it somehow feels brittle.  I guess the assumption that changed[]
> >> does not point at the beginning of the allocated area (e.g., it is a
> >> no-no to free(xe->xdf1.changed) or realloc() it) is so pervasive that
> >> it cannot be helped.  Sigh.
> >>
> >
> > Agreed it is ugly. I wanted to make sure the entire changed[] including
> > sentinels were clear as a defensive measure for downstream callers
> > (xdl_change_compact). I agree this results in something that is ugly
> > and brittle, but in the end I thought it was superior to relying on the
> > fact that upstream zeroes the entire changed[] array. Maybe if the
> > comment was more explicit about why this is happening it would be
> > helpful?
>
> Perhaps make these memset() into calls to a helper function that is
> defined in xdiff/xprepare.c with a descriptive name and placed near
> where xdl_prepare_ctx() is.  That way, the patch in question does
> not even have to expose the strangeness of changed[] (i.e., it has 2
> more elements than it would normally contain to make the memory
> region for changed[-1] and changed[N] valid, and freeing it requires
> free(changed-1)) to the code path.  It only needs to say "Hey, I am
> clearing changed[] arrays because of XXX" without having to say "by
> the way, the memory layout of changed[] is strange this way", the
> latter of which is not exactly of interest for readers of this code.
>
> >     /*
> >      * Clear changed[] arrays including sentinels.
> >      * xdl_prepare_env() may have dirtied them via
> >      * xdl_cleanup_records(), and xdl_change_compact() reads
> >      * the sentinel at changed[-1] during backward scans.
> >      */
>
> And this belongs in xdiff/xprepare.c near that new helper function.

That sounds a lot nicer. Will update.

[PATCH v2 0/4] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets external tools provide hunks into git's diff and blame pipelines.

diff.<driver>.process opens the door to integrating content-aware logic that
don't exist inside git: structural diff tools, format-aware analyzers, and
other tools that understand the semantics of the content being tracked.
Where diff.<driver>.command replaces the diff pipeline entirely,
diff.<driver>.process feeds hunks into it, so all downstream features (word
diff, function context, color-moved, stat, blame) work normally.

The protocol follows filter.<driver>.process: pkt-line over stdin/stdout,
capability negotiation, one tool invocation per git command. The tool
receives file pairs and returns hunk descriptors that git feeds into the
standard xdiff pipeline.

Zero hunks with status=success means the tool considers the files
equivalent. git diff shows no output for the file, and git blame skips the
commit, attributing lines to earlier commits.

On error or tool crash, git falls back silently to the builtin diff
algorithm. The feature is opt-in via diff.<driver>.process and
.gitattributes; unconfigured files are unaffected.

The blame integration calls the diff process for each commit in the file's
history. The subprocess is long-running (one startup amortized across the
traversal), but per-commit round-trips add latency. A natural follow-up
would be a capability that sends blob OIDs instead of content, allowing
tools that can read the object store directly to avoid the cost of
serializing and deserializing blob content over the pipe for each file pair.

Changes since v1:

 * Dropped the built-in diff-process-normalize tool since it obscured the
   main use case for the RFC and update series messaging accordingly.
 * Encapsulated changed[] memory layout behind xdl_clear_changed() helper in
   xprepare.c.
 * Restructured the external hunks path in xdl_diff() to fall through to the
   regular diff algorithm on validation failure instead of duplicating diff
   logic on fallback then returning.
 * Changed subprocess error reporting from trace to warning.
 * Fixed whitespace issues in the test's embedded Python.
 * Changed instances of grep to test_grep in tests.

Michael Montalbo (4):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  diff: add long-running diff process via diff.<driver>.process
  blame: consult diff process for zero-hunk detection

 Documentation/config/diff.adoc   |   8 +
 Documentation/gitattributes.adoc |  43 ++++
 Makefile                         |   1 +
 blame.c                          |  43 +++-
 diff-process.c                   | 206 +++++++++++++++++
 diff-process.h                   |  28 +++
 diff.c                           |  23 ++
 t/.gitattributes                 |   1 +
 t/t4080-diff-process.sh          | 370 +++++++++++++++++++++++++++++++
 userdiff.c                       |   7 +
 userdiff.h                       |   2 +
 xdiff-interface.c                |   7 +-
 xdiff/xdiff.h                    |  13 ++
 xdiff/xdiffi.c                   |  84 ++++++-
 xdiff/xprepare.c                 |  10 +
 xdiff/xprepare.h                 |   1 +
 16 files changed, 840 insertions(+), 7 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh


base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v1:

 1:  8c0ea0bc07 ! 1:  f887a7e2ba xdiff: support external hunks via xpparam_t
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +	long j, prev_old_end = 0, prev_new_end = 0;
      +	long total_old = 0, total_new = 0;
      +
     -+	/*
     -+	 * Clear changed[] arrays.  xdl_prepare_env() may have dirtied
     -+	 * them via xdl_cleanup_records().  The allocation is nrec + 2
     -+	 * elements; changed points one past the start (see xprepare.c).
     -+	 */
     -+	memset(xe->xdf1.changed - 1, 0,
     -+	       (xe->xdf1.nrec + 2) * sizeof(bool));
     -+	memset(xe->xdf2.changed - 1, 0,
     -+	       (xe->xdf2.nrec + 2) * sizeof(bool));
     ++	xdl_clear_changed(&xe->xdf1);
     ++	xdl_clear_changed(&xe->xdf2);
      +
      +	for (i = 0; i < nr_hunks; i++) {
      +		const struct xdl_hunk *h = &hunks[i];
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +		/* Bounds check (1-based line numbers) */
      +		if (h->old_count > 0 &&
      +		    (h->old_start < 1 ||
     -+		     h->old_start + h->old_count - 1 > xe->xdf1.nrec))
     ++		     h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
      +			return -1;
      +		if (h->new_count > 0 &&
      +		    (h->new_start < 1 ||
     -+		     h->new_start + h->new_count - 1 > xe->xdf2.nrec))
     ++		     h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
      +			return -1;
      +
      +		/* Zero-count hunks: start must still be in [1, nrec+1] */
      +		if (h->old_count == 0 &&
     -+		    (h->old_start < 1 || h->old_start > xe->xdf1.nrec + 1))
     ++		    (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
      +			return -1;
      +		if (h->new_count == 0 &&
     -+		    (h->new_start < 1 || h->new_start > xe->xdf2.nrec + 1))
     ++		    (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
      +			return -1;
      +
      +		/* Ordering: no overlap with previous hunk */
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
       	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
       
      -	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
     --
     --		return -1;
      +	if (xpp->external_hunks) {
      +		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
      +			return -1;
      +		if (xdl_populate_hunks_from_external(&xe,
      +						     xpp->external_hunks,
     -+						     xpp->external_hunks_nr) < 0) {
     -+			/*
     -+			 * Invalid external hunks; fall back to the
     -+			 * builtin diff algorithm.  Re-runs
     -+			 * xdl_prepare_env() via xdl_do_diff().
     -+			 */
     -+			xdl_free_env(&xe);
     -+			if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
     -+				return -1;
     -+		}
     -+	} else {
     -+		if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
     -+			return -1;
     - 	}
     ++						     xpp->external_hunks_nr) == 0)
     ++			goto diff_done;
     ++		xdl_free_env(&xe);
     ++	}
     + 
     ++	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
     + 		return -1;
     +-	}
     ++
     ++diff_done:
      +
       	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
       	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
       	    xdl_build_script(&xe, &xscr) < 0) {
     +
     + ## xdiff/xprepare.c ##
     +@@ xdiff/xprepare.c: int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
     + 
     + 	return 0;
     + }
     ++
     ++/*
     ++ * Reset the changed[] array so that no lines are marked as changed.
     ++ * Also clears the sentinel slots at changed[-1] and changed[nrec]
     ++ * that xdl_change_compact() relies on during backward scans.
     ++ */
     ++void xdl_clear_changed(xdfile_t *xdf)
     ++{
     ++	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
     ++}
     +
     + ## xdiff/xprepare.h ##
     +@@
     + int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
     + 		    xdfenv_t *xe);
     + void xdl_free_env(xdfenv_t *xe);
     ++void xdl_clear_changed(xdfile_t *xdf);
     + 
     + 
     + 
 2:  3bc127c800 = 2:  de6d85f9d7 userdiff: add diff.<driver>.process config
 3:  f9976fc6aa ! 3:  c25647c6e5 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
              [diff "cdiff"]
                  process = /path/to/diff-tool
      
     -    The tool receives file pairs and returns hunks describing which
     -    lines changed.  Git feeds these hunks into the standard xdiff
     -    pipeline, so all output features (word diff, function context,
     -    color) work normally.
     +    The tool provides custom line-matching: it receives file pairs
     +    and returns hunks that reference original line numbers.  Unlike
     +    textconv, which transforms the displayed content, the diff
     +    output shows the actual file while the tool controls which
     +    lines are marked as changed.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
          requests send command=hunks, pathname, and both file contents as
          packetized data.  The tool responds with hunk lines and a status
     -    packet.  On error, git falls back to the builtin diff algorithm.
     +    packet.  On error, git falls back to the builtin diff algorithm
     +    with a warning.
      
     -    Zero hunks with status=success means the tool considers the files
     -    equivalent.  Git skips diff output for that file entirely.
     +    Zero hunks with status=success means the tool considers the
     +    files equivalent.  Git skips diff output for that file.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +An external tool can provide content-aware line matching by
      +setting `diff.<name>.process` to the command that runs
      +the tool.  The tool is a long-running process that communicates via
     -+the pkt-line protocol (see
     -+linkgit:gitprotocol-long-running-process[5]).
     ++the pkt-line protocol (described in
     ++Documentation/technical/long-running-process-protocol.adoc).
      +
      +------------------------
      +*.c diff=cdiff
     @@ diff-process.c (new)
      @@
      +/*
      + * Diff process backend: communicates with a long-running external
     -+ * tool via the pkt-line protocol to obtain content-aware hunks.
     ++ * tool via the pkt-line protocol to obtain custom line-matching
     ++ * results.  Unlike textconv, which transforms the displayed content,
     ++ * hunks from a diff process reference original line numbers and
     ++ * the display shows the actual file content.
      + *
      + * Protocol: pkt-line over stdin/stdout, following the pattern of
      + * the long-running filter process protocol (see convert.c).
     @@ diff.c
       #include "userdiff.h"
      +#include "diff-process.h"
       #include "submodule.h"
     -+#include "trace2.h"
       #include "hashmap.h"
       #include "mem-pool.h"
     - #include "merge-ll.h"
      @@ diff.c: static void builtin_diff(const char *name_a,
       		xpparam_t xpp;
       		xdemitconf_t xecfg;
     @@ diff.c: static void builtin_diff(const char *name_a,
      +				xpp.external_hunks = ext_hunks;
      +				xpp.external_hunks_nr = ext_hunks_nr;
      +			} else {
     -+				trace2_data_string("diff",
     -+						   o->repo,
     -+						   "diff-process-fallback",
     -+						   name_a);
     ++				warning(_("diff process failed for '%s',"
     ++					  " falling back to builtin diff"),
     ++					name_a);
      +			}
      +		}
      +
     @@ diff.c: static void builtin_diff(const char *name_a,
       			free_diff_words_data(&ecbdata);
       		if (textconv_one)
      
     + ## t/.gitattributes ##
     +@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
     + /t8005/*.txt eol=lf
     + /t9*/*.dump eol=lf
     + /t0040*.sh whitespace=-indent-with-non-tab
     ++/t4080-diff-process.sh whitespace=-indent-with-non-tab
     +
       ## t/t4080-diff-process.sh (new) ##
      @@
      +#!/bin/sh
     @@ t/t4080-diff-process.sh (new)
      +	# appear as changed in the output.
      +	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
      +		diff boundary.c >actual &&
     -+	grep "^-OLD5" actual &&
     -+	grep "^-OLD6" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^+NEW6" actual &&
     -+	! grep "^-OLD9" actual &&
     -+	! grep "^-OLD10" actual &&
     -+	! grep "^+NEW9" actual &&
     -+	! grep "^+NEW10" actual
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^-OLD6" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^+NEW6" actual &&
     ++	! test_grep "^-OLD9" actual &&
     ++	! test_grep "^-OLD10" actual &&
     ++	! test_grep "^+NEW9" actual &&
     ++	! test_grep "^+NEW10" actual
      +'
      +
      +test_expect_success PYTHON 'diff process fallback on tool error status' '
     @@ t/t4080-diff-process.sh (new)
      +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
      +		diff boundary.c >actual &&
      +	# Fallback produces the full builtin diff (both change regions).
     -+	grep "^-OLD5" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^-OLD9" actual &&
     -+	grep "^+NEW9" actual &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
      +	# Tool was contacted (it replied with error, not crash).
     -+	grep "command=hunks pathname=boundary.c" backend.log
     ++	test_grep "command=hunks pathname=boundary.c" backend.log
      +'
      +
      +test_expect_success PYTHON 'diff process fallback on bad hunks' '
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
      +		diff boundary.c >actual &&
     -+	grep "^-OLD5" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^-OLD9" actual &&
     -+	grep "^+NEW9" actual
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual
      +'
      +
      +test_expect_success PYTHON 'diff process fallback on tool crash' '
      +	git -c diff.cdiff.process="$BACKEND --mode=crash" \
      +		diff boundary.c >actual &&
     -+	grep "^-OLD5" actual &&
     -+	grep "^+NEW5" actual &&
     -+	grep "^-OLD9" actual &&
     -+	grep "^+NEW9" actual
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual
      +'
      +
      +test_expect_success PYTHON 'diff process abort disables for session' '
     @@ t/t4080-diff-process.sh (new)
      +	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
      +		diff -- abort1.c abort2.c >actual &&
      +	# Both files should still produce diff output via fallback.
     -+	grep "return 10" actual &&
     -+	grep "return 20" actual &&
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
      +	# The tool aborts on the first file and git clears its
      +	# capability.  The second file never contacts the tool,
      +	# so the log should have exactly one entry, not two.
     -+	grep "command=hunks" backend.log >matches &&
     ++	test_grep "command=hunks" backend.log >matches &&
      +	test_line_count = 1 matches
      +'
      +
     @@ t/t4080-diff-process.sh (new)
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- multi1.c multi2.c >actual &&
     -+	grep "return 10" actual &&
     -+	grep "return 20" actual &&
     -+	grep "pathname=multi1.c" backend.log &&
     -+	grep "pathname=multi2.c" backend.log
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
     ++	test_grep "pathname=multi1.c" backend.log &&
     ++	test_grep "pathname=multi2.c" backend.log
      +'
      +
      +test_expect_success PYTHON 'diff process with --word-diff' '
     @@ t/t4080-diff-process.sh (new)
      +
      +	git -c diff.cdiff.process="$BACKEND" \
      +		diff --word-diff worddiff.c >actual &&
     -+	grep "\[-1;-\]" actual &&
     -+	grep "{+999;+}" actual
     ++	test_grep "\[-1;-\]" actual &&
     ++	test_grep "{+999;+}" actual
      +'
      +
      +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --diff-algorithm=patience worddiff.c >actual &&
     -+	grep "return 999" actual &&
     ++	test_grep "return 999" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     @@ t/t4080-diff-process.sh (new)
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		log -1 -p -- logtest.c >actual &&
     -+	grep "return 2" actual &&
     -+	grep "command=hunks pathname=logtest.c" backend.log
     ++	test_grep "return 2" actual &&
     ++	test_grep "command=hunks pathname=logtest.c" backend.log
      +'
      +
      +test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
 4:  4e6ea6d518 ! 4:  39ff53acef blame: consult diff process for zero-hunk detection
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
      +
      +	# Without zero-hunk mode, blame attributes the change.
      +	git blame blame.c >without &&
     -+	grep "$BLAME_COMMIT" without &&
     ++	test_grep "$BLAME_COMMIT" without &&
      +
      +	# With zero-hunk mode, the process considers the files equivalent
      +	# and blame skips the reformat commit.
      +	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
      +		blame blame.c >with &&
     -+	! grep "$BLAME_COMMIT" with
     ++	! test_grep "$BLAME_COMMIT" with
      +'
      +
      +
 5:  8c7359b8a1 < -:  ---------- diff-process-normalize: add built-in whitespace normalizer

-- 
gitgitgadget

[PATCH v2 1/4] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 13 ++++++++
 xdiff/xdiffi.c    | 84 +++++++++++++++++++++++++++++++++++++++++++++--
 xdiff/xprepare.c  | 10 ++++++
 xdiff/xprepare.h  |  1 +
 5 files changed, 112 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index f043330f2a..9542c0bcc2 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..2ee6f1aae3 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based, matching unified diff convention.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +97,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm. */
+	const struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index 5455b4690d..e7d6190d37 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,96 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    const struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long total_old = 0, total_new = 0;
+
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0)
+			return -1;
+
+		/* Bounds check (1-based line numbers) */
+		if (h->old_count > 0 &&
+		    (h->old_start < 1 ||
+		     h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
+			return -1;
+		if (h->new_count > 0 &&
+		    (h->new_start < 1 ||
+		     h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
+			return -1;
+
+		/* Zero-count hunks: start must still be in [1, nrec+1] */
+		if (h->old_count == 0 &&
+		    (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
+			return -1;
+		if (h->new_count == 0 &&
+		    (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
+			return -1;
+
+		/* Ordering: no overlap with previous hunk */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+		total_old += h->old_count;
+		total_new += h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 */
+	if ((long)xe->xdf1.nrec - total_old !=
+	    (long)xe->xdf2.nrec - total_new)
+		return -1;
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
+
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index cd4fc405eb..4645a9a746 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -432,3 +432,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v2 2/4] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add a new per-driver configuration key that specifies the command
for a long-running diff process.

The name follows filter.<driver>.process: a long-running subprocess
that stays alive across files within a single git invocation.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index fe710a68bf..81c0bebcce 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference original line numbers.  Unlike
textconv, which transforms the displayed content, the diff
output shows the actual file while the tool controls which
lines are marked as changed.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet.  On error, git falls back to the builtin diff algorithm
with a warning.

Zero hunks with status=success means the tool considers the
files equivalent.  Git skips diff output for that file.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   8 +
 Documentation/gitattributes.adoc |  40 ++++
 Makefile                         |   1 +
 diff-process.c                   | 206 +++++++++++++++++++
 diff-process.h                   |  28 +++
 diff.c                           |  23 +++
 t/.gitattributes                 |   1 +
 t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
 8 files changed, 645 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..4ab5f60df6 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,14 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process.
+	The tool communicates via the pkt-line protocol and returns
+	hunks that are fed into Git's diff and blame pipelines.
+	If the tool returns zero hunks, the file is treated as
+	unchanged for both diff output and blame attribution.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f20041a323..962896a0b4 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,46 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An external tool can provide content-aware line matching by
+setting `diff.<name>.process` to the command that runs
+the tool.  The tool is a long-running process that communicates via
+the pkt-line protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+The tool receives file pairs and returns hunk descriptors indicating
+which lines changed.  Git feeds these hunks into its standard diff
+pipeline, so all output features (word diff, function context,
+color) work normally.
+
+If the tool fails or returns an error, Git silently falls back to
+the builtin diff algorithm.  If the tool returns invalid hunks
+(out of bounds, overlapping), Git also falls back silently.
+
+The handshake negotiates `version=1` and `capability=hunks`.
+Per-file requests send `command=hunks` and `pathname=<path>`,
+followed by the old and new file content as packetized data.
+The tool responds with lines of the form
+`hunk <old_start> <old_count> <new_start> <new_count>`
+(1-based line numbers), a flush packet, and `status=success`.
+
+If the tool returns zero hunks with `status=success`, Git treats
+the file as having no changes and produces no diff output.
+
+Tools should ignore unknown keys in the per-file request to
+remain forward-compatible.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index cedc234173..22900368dd 100644
--- a/Makefile
+++ b/Makefile
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..801ac9e22e
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,206 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  Unlike textconv, which transforms the displayed content,
+ * hunks from a diff process reference original line numbers and
+ * the display shows the actual file content.
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * Zero hunks with status=success means the tool considers the
+ * files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int subprocess_map_initialized;
+static struct hashmap subprocess_map;
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		(struct diff_subprocess *)subprocess;
+
+	/* Uses dying pkt-line variant, same as convert.c filters. */
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *find_or_start_process(const char *cmd)
+{
+	struct diff_subprocess *entry;
+
+	if (!subprocess_map_initialized) {
+		subprocess_map_initialized = 1;
+		hashmap_init(&subprocess_map, cmd2process_cmp, NULL, 0);
+	}
+
+	entry = (struct diff_subprocess *)
+		subprocess_find_entry(&subprocess_map, cmd);
+	if (entry)
+		return entry;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start(&subprocess_map, &entry->subprocess,
+			     cmd, start_diff_process_fn)) {
+		free(entry);
+		return NULL;
+	}
+
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret;
+
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	else
+		ret = 0;
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	hunk->old_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->old_count = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_start = strtol(line, &end, 10);
+	if (end == line || *end != ' ')
+		return -1;
+	line = end;
+
+	hunk->new_count = strtol(line, &end, 10);
+	if (end == line || *end != '\0')
+		return -1;
+
+	return 0;
+}
+
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	if (!drv || !drv->process)
+		return -1;
+
+	backend = find_or_start_process(drv->process);
+	if (!backend)
+		return -1;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return -1;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto error;
+
+	if (strcmp(status.buf, "success")) {
+		if (!strcmp(status.buf, "abort"))
+			backend->supported_capabilities &= ~CAP_HUNKS;
+		goto error;
+	}
+
+	*hunks_out = hunks;
+	*nr_hunks_out = nr_hunks;
+	strbuf_release(&status);
+	return 0;
+
+error:
+	free(hunks);
+	strbuf_release(&status);
+	return -1;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..4c84951e02
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,28 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+struct userdiff_driver;
+struct xdl_hunk;
+
+/*
+ * Query a diff process for hunks describing the changes
+ * between old_buf and new_buf.
+ *
+ * The backend is a long-running subprocess configured via
+ * diff.<driver>.process.  It receives file content via
+ * pkt-line and returns hunks with 1-based line numbers.
+ *
+ * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
+ * array (caller must free) and returns 0.
+ *
+ * On failure, returns -1.  The caller should fall back to the
+ * builtin diff algorithm.
+ */
+int diff_process_get_hunks(struct userdiff_driver *drv,
+			   const char *path,
+			   const char *old_buf, long old_size,
+			   const char *new_buf, long new_size,
+			   struct xdl_hunk **hunks_out,
+			   size_t *nr_hunks_out);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 397e38b41c..1aeb0f319e 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -3991,6 +3992,7 @@ static void builtin_diff(const char *name_a,
 		xpparam_t xpp;
 		xdemitconf_t xecfg;
 		struct emit_callback ecbdata;
+		struct xdl_hunk *ext_hunks = NULL;
 		unsigned ws_rule;
 		const struct userdiff_funcname *pe;
 
@@ -4031,6 +4033,26 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (!o->ignore_driver_algorithm &&
+		    one->driver && one->driver->process) {
+			size_t ext_hunks_nr = 0;
+			if (!diff_process_get_hunks(
+				    one->driver, name_a,
+				    mf1.ptr, mf1.size,
+				    mf2.ptr, mf2.size,
+				    &ext_hunks, &ext_hunks_nr)) {
+				if (!ext_hunks_nr)
+					goto free_ab_and_return;
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = ext_hunks_nr;
+			} else {
+				warning(_("diff process failed for '%s',"
+					  " falling back to builtin diff"),
+					name_a);
+			}
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4111,6 +4133,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(ext_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/t/.gitattributes b/t/.gitattributes
index 7664c6e027..de97920cab 100644
--- a/t/.gitattributes
+++ b/t/.gitattributes
@@ -23,3 +23,4 @@ t[0-9][0-9][0-9][0-9]/* -whitespace
 /t8005/*.txt eol=lf
 /t9*/*.dump eol=lf
 /t0040*.sh whitespace=-indent-with-non-tab
+/t4080-diff-process.sh whitespace=-indent-with-non-tab
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..083e48e872
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,338 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+if test_have_prereq PYTHON
+then
+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+fi
+
+#
+# A single parametric diff process.
+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
+#
+# Modes:
+#   whole-file  - report all lines as changed (default)
+#   fixed-hunk  - always report hunk 5 2 5 2
+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
+#   zero-hunk   - return zero hunks (files considered equivalent)
+#   error       - return status=error for every request
+#   abort       - return status=abort for every request
+#   crash       - read one request then exit without responding
+#
+setup_backend () {
+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
+	import sys, os
+
+	def read_pkt():
+	    hdr = sys.stdin.buffer.read(4)
+	    if len(hdr) < 4: return None
+	    length = int(hdr, 16)
+	    if length == 0: return ""
+	    data = sys.stdin.buffer.read(length - 4)
+	    return data.decode().rstrip("\n")
+
+	def write_pkt(line):
+	    data = (line + "\n").encode()
+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    sys.stdout.buffer.flush()
+
+	def write_flush():
+	    sys.stdout.buffer.write(b"0000")
+	    sys.stdout.buffer.flush()
+
+	def read_content():
+	    chunks = []
+	    while True:
+	        hdr = sys.stdin.buffer.read(4)
+	        if len(hdr) < 4: break
+	        length = int(hdr, 16)
+	        if length == 0: break
+	        chunks.append(sys.stdin.buffer.read(length - 4))
+	    return b"".join(chunks)
+
+	mode = "whole-file"
+	logfile = None
+	for arg in sys.argv[1:]:
+	    if arg.startswith("--mode="):
+	        mode = arg[7:]
+	    elif arg.startswith("--log="):
+	        logfile = open(arg[6:], "a")
+
+	def log(msg):
+	    if logfile:
+	        logfile.write(msg + "\n")
+	        logfile.flush()
+
+	# Handshake
+	assert read_pkt() == "git-diff-client"
+	assert read_pkt() == "version=1"
+	read_pkt()
+	write_pkt("git-diff-server")
+	write_pkt("version=1")
+	write_flush()
+	while True:
+	    p = read_pkt()
+	    if p == "": break
+	write_pkt("capability=hunks")
+	write_flush()
+
+	log("ready")
+
+	while True:
+	    cmd = None
+	    pathname = None
+	    while True:
+	        p = read_pkt()
+	        if p is None: sys.exit(0)
+	        if p == "": break
+	        if p.startswith("command="): cmd = p.split("=",1)[1]
+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
+	    if cmd is None: sys.exit(0)
+	    old = read_content()
+	    new = read_content()
+	    log(f"command={cmd} pathname={pathname}")
+
+	    if mode == "error":
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	        continue
+
+	    if mode == "abort":
+	        write_flush()
+	        write_pkt("status=abort")
+	        write_flush()
+	        continue
+
+	    if mode == "crash":
+	        sys.exit(1)
+
+	    if cmd == "hunks":
+	        if mode == "fixed-hunk":
+	            write_pkt("hunk 5 2 5 2")
+	        elif mode == "bad-hunk":
+	            write_pkt("hunk 999 1 999 1")
+	        elif mode == "zero-hunk":
+	            pass
+	        else:
+	            ol = len(old.split(b"\n"))
+	            nl = len(new.split(b"\n"))
+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	        write_flush()
+	        write_pkt("status=success")
+	        write_flush()
+	    else:
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	PYEOF
+	write_script diff-process-backend <<-SHEOF
+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
+	SHEOF
+}
+
+BACKEND="./diff-process-backend"
+
+test_expect_success PYTHON 'setup' '
+	setup_backend &&
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+	git commit -m "initial"
+'
+
+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+	git commit -m "add boundary.c" &&
+
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	! test_grep "^-OLD9" actual &&
+	! test_grep "^-OLD10" actual &&
+	! test_grep "^+NEW9" actual &&
+	! test_grep "^+NEW10" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool error status' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual
+'
+
+test_expect_success PYTHON 'diff process abort disables for session' '
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add abort1.c abort2.c &&
+	git commit -m "add abort files" &&
+
+	cat >abort1.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+	cat >abort2.c <<-\EOF &&
+	int second(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- abort1.c abort2.c >actual &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool,
+	# so the log should have exactly one entry, not two.
+	test_grep "command=hunks" backend.log >matches &&
+	test_line_count = 1 matches
+'
+
+test_expect_success PYTHON 'diff process handles multiple files' '
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 1; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 2; }
+	EOF
+	git add multi1.c multi2.c &&
+	git commit -m "add multi files" &&
+
+	cat >multi1.c <<-\EOF &&
+	int one(void) { return 10; }
+	EOF
+	cat >multi2.c <<-\EOF &&
+	int two(void) { return 20; }
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- multi1.c multi2.c >actual &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	test_grep "pathname=multi1.c" backend.log &&
+	test_grep "pathname=multi2.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process with --word-diff' '
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+	git commit -m "add worddiff.c" &&
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND" \
+		diff --word-diff worddiff.c >actual &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual
+'
+
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process works with git log -p' '
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+	git commit -m "add logtest.c" &&
+
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		log -1 -p -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add zerohunk.c &&
+	git commit -m "add zerohunk.c" &&
+
+	cat >zerohunk.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		diff zerohunk.c >actual &&
+	test_must_be_empty actual
+'
+
+test_done
-- 
gitgitgadget

[PATCH v2 4/4] blame: consult diff process for zero-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns zero hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc |  3 +++
 blame.c                          | 43 +++++++++++++++++++++++++++++---
 t/t4080-diff-process.sh          | 32 ++++++++++++++++++++++++
 3 files changed, 74 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 962896a0b4..c087b4b265 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -857,6 +857,9 @@ The tool responds with lines of the form
 
 If the tool returns zero hunks with `status=success`, Git treats
 the file as having no changes and produces no diff output.
+`git blame` also consults the diff process and skips commits
+where it reports zero hunks, attributing lines to earlier commits
+instead.
 
 Tools should ignore unknown keys in the per-file request to
 remain forward-compatible.
diff --git a/blame.c b/blame.c
index a3c49d132e..8a5f14db7a 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "userdiff.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -315,16 +317,47 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data,
+		      int xdl_opts, struct index_state *istate,
+		      const char *path)
 {
 	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
+	struct xdl_hunk *ext_hunks = NULL;
+	int ret;
 
 	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+
+	if (path && istate) {
+		struct userdiff_driver *drv;
+		drv = userdiff_find_by_path(istate, path);
+		if (drv && drv->process) {
+			size_t nr = 0;
+			if (!diff_process_get_hunks(drv, path,
+						    file_a->ptr, file_a->size,
+						    file_b->ptr, file_b->size,
+						    &ext_hunks, &nr)) {
+				if (!nr) {
+					/*
+					 * Zero hunks: the diff process
+					 * considers these files equivalent.
+					 * Skip so blame looks past this
+					 * commit.
+					 */
+					return 0;
+				}
+				xpp.external_hunks = ext_hunks;
+				xpp.external_hunks_nr = nr;
+			}
+		}
+	}
+
+	ret = xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	free(ext_hunks);
+	return ret;
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1961,7 +1994,8 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts,
+		       sb->revs->diffopt.repo->index, target->path))
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
@@ -2114,7 +2148,8 @@ static void find_copy_in_blob(struct blame_scoreboard *sb,
 	 * file_p partially may match that image.
 	 */
 	memset(split, 0, sizeof(struct blame_entry [3]));
-	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts))
+	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts,
+		       NULL, NULL))
 		die("unable to generate diff (%s)",
 		    oid_to_hex(&parent->commit->object.oid));
 	/* remainder, if any, all match the preimage */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 083e48e872..50f49a9b02 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -335,4 +335,36 @@ test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
 	test_must_be_empty actual
 '
 
+test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without zero-hunk mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With zero-hunk mode, the process considers the files equivalent
+	# and blame skips the reformat commit.
+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
+		blame blame.c >with &&
+	! test_grep "$BLAME_COMMIT" with
+'
+
+
 test_done
-- 
gitgitgadget

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +struct diff_subprocess {
> +	struct subprocess_entry subprocess;
> +	unsigned int supported_capabilities;
> +};
> +
> +static int subprocess_map_initialized;
> +static struct hashmap subprocess_map;

Can we avoid introducing new global variables like these?  Would
"struct userdiff_driver" or "struct diff_options" be a good place to
hang this hashmap, perhaps?

> +static int send_file_content(int fd, const char *buf, long size)
> +{
> +	int ret;
> +
> +	if (size > 0)
> +		ret = write_packetized_from_buf_no_flush(buf, size, fd);
> +	else
> +		ret = 0;

Shouldn't "size == -24" be flagged as an invalid input?

> +	if (ret)
> +		return ret;
> +	return packet_flush_gently(fd);
> +}

> +static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
> +{
> +...
> +}

This gives a silent error diagnosis, which is good for a lower level
helper.

> +int diff_process_get_hunks(struct userdiff_driver *drv,
> +			   const char *path,
> +			   const char *old_buf, long old_size,
> +			   const char *new_buf, long new_size,
> +			   struct xdl_hunk **hunks_out,
> +			   size_t *nr_hunks_out)
> +{
> +	struct diff_subprocess *backend;
> +	struct child_process *process;
> +	int fd_in, fd_out;
> +	struct strbuf status = STRBUF_INIT;
> +	struct xdl_hunk *hunks = NULL;
> +	struct xdl_hunk hunk;
> +	size_t nr_hunks = 0, alloc_hunks = 0;
> +	int len;
> +	char *line;
> +
> +	if (!drv || !drv->process)
> +		return -1;

A driver that does not define process is not an error; it is
perfectly normal in the current world order where nobody has such an
external process and even fi this patch lands, external processes
are optional.  So here "return -1" does not mean an error, and
silent return is perfectly fine.

> +	backend = find_or_start_process(drv->process);
> +	if (!backend)
> +		return -1;

This is probably an error; the user specified drv->process, we
either tried to find or start the process and failed.  Isn't it an
event that deserves to be reported in an error message?

> +	if (!(backend->supported_capabilities & CAP_HUNKS))
> +		return -1;

Backend started, but the "hunks" feature is not supported.  Perhaps
in a year or two, this external process protocol may have become so
popular that it gained more capabilities, possibly making get_hunks
obsolete.  We may be looking at such an external process that uses
other capabilities but not this one.  This is not an error, so
silent return is perfectly fine.

> +	process = subprocess_get_child_process(&backend->subprocess);
> +	fd_in = process->in;
> +	fd_out = process->out;
> +
> +	/* Send request */
> +	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
> +	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
> +	    packet_flush_gently(fd_in))
> +		goto error;
> +
> +	/* Send old file content */
> +	if (send_file_content(fd_in, old_buf, old_size))
> +		goto error;
> +
> +	/* Send new file content */
> +	if (send_file_content(fd_in, new_buf, new_size))
> +		goto error;
> +
> +	/* Read hunks until flush packet */
> +	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
> +	       line) {
> +		if (parse_hunk_line(line, &hunk) < 0)
> +			goto error;
> +		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
> +		hunks[nr_hunks++] = hunk;
> +	}
> +	if (len < 0)
> +		goto error;
> +
> +	/* Read status */
> +	if (subprocess_read_status(fd_out, &status))
> +		goto error;
> +
> +	if (strcmp(status.buf, "success")) {
> +		if (!strcmp(status.buf, "abort"))
> +			backend->supported_capabilities &= ~CAP_HUNKS;
> +		goto error;
> +	}
> +
> +	*hunks_out = hunks;
> +	*nr_hunks_out = nr_hunks;
> +	strbuf_release(&status);
> +	return 0;
> +
> +error:

All exceptions that lead here look like events that should be
reported to the end-user.

> +	free(hunks);
> +	strbuf_release(&status);
> +	return -1;
> +}

> +/*
> + * Query a diff process for hunks describing the changes
> + * between old_buf and new_buf.
> + *
> + * The backend is a long-running subprocess configured via
> + * diff.<driver>.process.  It receives file content via
> + * pkt-line and returns hunks with 1-based line numbers.
> + *
> + * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
> + * array (caller must free) and returns 0.
> + *
> + * On failure, returns -1.  The caller should fall back to the
> + * builtin diff algorithm.
> + */

I do not agree with this.  If it is a failure, the user should fix
the external process (or disable).  It shouldn't be hidden behind a
fallback.  As I left comments, in this round of implementation,
there are conditions that returns -1 for soemthing that is not an
error (i.e., not configured, or process not supporting the
particular capability) *and* in those cases the caller should fall
back as if nothing happened.  But some error cases, the caller
should't hide them.

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Zero hunks with status=success means the tool considers the
> files equivalent.  Git skips diff output for that file.

Is "zero hunk" a common word or some random string you invented?  If
the latter, which is I am assuming it to be, you should define what
it means at/before the first use.  Here in the proposed log message,
and ...

>
> Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
> ---
>  Documentation/config/diff.adoc   |   8 +
>  Documentation/gitattributes.adoc |  40 ++++
>  Makefile                         |   1 +
>  diff-process.c                   | 206 +++++++++++++++++++
>  diff-process.h                   |  28 +++
>  diff.c                           |  23 +++
>  t/.gitattributes                 |   1 +
>  t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
>  8 files changed, 645 insertions(+)
>  create mode 100644 diff-process.c
>  create mode 100644 diff-process.h
>  create mode 100755 t/t4080-diff-process.sh
>
> diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
> index 1135a62a0a..4ab5f60df6 100644
> --- a/Documentation/config/diff.adoc
> +++ b/Documentation/config/diff.adoc
> @@ -218,6 +218,14 @@ endif::git-diff[]
>  	Set this option to `true` to make the diff driver cache the text
>  	conversion outputs.  See linkgit:gitattributes[5] for details.
>  
> +`diff.<driver>.process`::
> +	The command to run as a long-running diff process.
> +	The tool communicates via the pkt-line protocol and returns
> +	hunks that are fed into Git's diff and blame pipelines.
> +	If the tool returns zero hunks, the file is treated as
> +	unchanged for both diff output and blame attribution.
> +	See linkgit:gitattributes[5] for details.

... also here.

I do not know if you mean "the tool returns no hunks" (there is no
"hunk <old_start> <old_count> <new_start> <new_count>" line passed
from the tool over the protocol) or "the tool returns zero-hunk"
(there is a special "zero-hunk" message to signal this particular
condition sent over the protocol), and this description does not
quite help disambiguating between the two.

If the former, then avoid "zero hunks" as it sounds like a noun with
special meaning.  Yes, we can say "tool returns one hunk", "tool
returns 31 hunks", etc., so "tool returns zero hunks" may logically
be correct, but "when the tool returns no hunks with status=success"
is much less confusing, I think.

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Mon, May 25, 2026 at 6:56 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +struct diff_subprocess {
> > +     struct subprocess_entry subprocess;
> > +     unsigned int supported_capabilities;
> > +};
> > +
> > +static int subprocess_map_initialized;
> > +static struct hashmap subprocess_map;
>
> Can we avoid introducing new global variables like these?  Would
> "struct userdiff_driver" or "struct diff_options" be a good place to
> hang this hashmap, perhaps?
>

Will clean this up.

> > +static int send_file_content(int fd, const char *buf, long size)
> > +{
> > +     int ret;
> > +
> > +     if (size > 0)
> > +             ret = write_packetized_from_buf_no_flush(buf, size, fd);
> > +     else
> > +             ret = 0;
>
> Shouldn't "size == -24" be flagged as an invalid input?
>

Will fix and do a broader audit of input validation and bounds checking.

> > +     if (ret)
> > +             return ret;
> > +     return packet_flush_gently(fd);
> > +}
>
> > +static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
> > +{
> > +...
> > +}
>
> This gives a silent error diagnosis, which is good for a lower level
> helper.
>
> > +int diff_process_get_hunks(struct userdiff_driver *drv,
> > +                        const char *path,
> > +                        const char *old_buf, long old_size,
> > +                        const char *new_buf, long new_size,
> > +                        struct xdl_hunk **hunks_out,
> > +                        size_t *nr_hunks_out)
> > +{
> > +     struct diff_subprocess *backend;
> > +     struct child_process *process;
> > +     int fd_in, fd_out;
> > +     struct strbuf status = STRBUF_INIT;
> > +     struct xdl_hunk *hunks = NULL;
> > +     struct xdl_hunk hunk;
> > +     size_t nr_hunks = 0, alloc_hunks = 0;
> > +     int len;
> > +     char *line;
> > +
> > +     if (!drv || !drv->process)
> > +             return -1;
>
> A driver that does not define process is not an error; it is
> perfectly normal in the current world order where nobody has such an
> external process and even fi this patch lands, external processes
> are optional.  So here "return -1" does not mean an error, and
> silent return is perfectly fine.
>
> > +     backend = find_or_start_process(drv->process);
> > +     if (!backend)
> > +             return -1;
>
> This is probably an error; the user specified drv->process, we
> either tried to find or start the process and failed.  Isn't it an
> event that deserves to be reported in an error message?
>
> > +     if (!(backend->supported_capabilities & CAP_HUNKS))
> > +             return -1;
>
> Backend started, but the "hunks" feature is not supported.  Perhaps
> in a year or two, this external process protocol may have become so
> popular that it gained more capabilities, possibly making get_hunks
> obsolete.  We may be looking at such an external process that uses
> other capabilities but not this one.  This is not an error, so
> silent return is perfectly fine.
>
> > +     process = subprocess_get_child_process(&backend->subprocess);
> > +     fd_in = process->in;
> > +     fd_out = process->out;
> > +
> > +     /* Send request */
> > +     if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
> > +         packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
> > +         packet_flush_gently(fd_in))
> > +             goto error;
> > +
> > +     /* Send old file content */
> > +     if (send_file_content(fd_in, old_buf, old_size))
> > +             goto error;
> > +
> > +     /* Send new file content */
> > +     if (send_file_content(fd_in, new_buf, new_size))
> > +             goto error;
> > +
> > +     /* Read hunks until flush packet */
> > +     while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
> > +            line) {
> > +             if (parse_hunk_line(line, &hunk) < 0)
> > +                     goto error;
> > +             ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
> > +             hunks[nr_hunks++] = hunk;
> > +     }
> > +     if (len < 0)
> > +             goto error;
> > +
> > +     /* Read status */
> > +     if (subprocess_read_status(fd_out, &status))
> > +             goto error;
> > +
> > +     if (strcmp(status.buf, "success")) {
> > +             if (!strcmp(status.buf, "abort"))
> > +                     backend->supported_capabilities &= ~CAP_HUNKS;
> > +             goto error;
> > +     }
> > +
> > +     *hunks_out = hunks;
> > +     *nr_hunks_out = nr_hunks;
> > +     strbuf_release(&status);
> > +     return 0;
> > +
> > +error:
>
> All exceptions that lead here look like events that should be
> reported to the end-user.
>

Agreed on all points. I will restructure things so errors are flagged when
appropriate (i.e., user specified a process but one was not found / couldn't
start and exceptions) and non-errors are treated as they should be.

> > +     free(hunks);
> > +     strbuf_release(&status);
> > +     return -1;
> > +}
>
> > +/*
> > + * Query a diff process for hunks describing the changes
> > + * between old_buf and new_buf.
> > + *
> > + * The backend is a long-running subprocess configured via
> > + * diff.<driver>.process.  It receives file content via
> > + * pkt-line and returns hunks with 1-based line numbers.
> > + *
> > + * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
> > + * array (caller must free) and returns 0.
> > + *
> > + * On failure, returns -1.  The caller should fall back to the
> > + * builtin diff algorithm.
> > + */
>
> I do not agree with this.  If it is a failure, the user should fix
> the external process (or disable).  It shouldn't be hidden behind a
> fallback.  As I left comments, in this round of implementation,
> there are conditions that returns -1 for soemthing that is not an
> error (i.e., not configured, or process not supporting the
> particular capability) *and* in those cases the caller should fall
> back as if nothing happened.  But some error cases, the caller
> should't hide them.

Will address in a follow-up.

Thank you for the feedback!

Re: [PATCH v2 3/4] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Mon, May 25, 2026 at 7:26 PM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > Zero hunks with status=success means the tool considers the
> > files equivalent.  Git skips diff output for that file.
>
> Is "zero hunk" a common word or some random string you invented?  If
> the latter, which is I am assuming it to be, you should define what
> it means at/before the first use.  Here in the proposed log message,
> and ...
>
> >
> > Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
> > ---
> >  Documentation/config/diff.adoc   |   8 +
> >  Documentation/gitattributes.adoc |  40 ++++
> >  Makefile                         |   1 +
> >  diff-process.c                   | 206 +++++++++++++++++++
> >  diff-process.h                   |  28 +++
> >  diff.c                           |  23 +++
> >  t/.gitattributes                 |   1 +
> >  t/t4080-diff-process.sh          | 338 +++++++++++++++++++++++++++++++
> >  8 files changed, 645 insertions(+)
> >  create mode 100644 diff-process.c
> >  create mode 100644 diff-process.h
> >  create mode 100755 t/t4080-diff-process.sh
> >
> > diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
> > index 1135a62a0a..4ab5f60df6 100644
> > --- a/Documentation/config/diff.adoc
> > +++ b/Documentation/config/diff.adoc
> > @@ -218,6 +218,14 @@ endif::git-diff[]
> >       Set this option to `true` to make the diff driver cache the text
> >       conversion outputs.  See linkgit:gitattributes[5] for details.
> >
> > +`diff.<driver>.process`::
> > +     The command to run as a long-running diff process.
> > +     The tool communicates via the pkt-line protocol and returns
> > +     hunks that are fed into Git's diff and blame pipelines.
> > +     If the tool returns zero hunks, the file is treated as
> > +     unchanged for both diff output and blame attribution.
> > +     See linkgit:gitattributes[5] for details.
>
> ... also here.
>
> I do not know if you mean "the tool returns no hunks" (there is no
> "hunk <old_start> <old_count> <new_start> <new_count>" line passed
> from the tool over the protocol) or "the tool returns zero-hunk"
> (there is a special "zero-hunk" message to signal this particular
> condition sent over the protocol), and this description does not
> quite help disambiguating between the two.
>
> If the former, then avoid "zero hunks" as it sounds like a noun with
> special meaning.  Yes, we can say "tool returns one hunk", "tool
> returns 31 hunks", etc., so "tool returns zero hunks" may logically
> be correct, but "when the tool returns no hunks with status=success"
> is much less confusing, I think.

Yes, "zero hunks" was my own invention and I see why it's confusing. Will
update the messaging to use "no hunks" instead and do a broader sweep of
the documentation to clarify the protocol and expected tool behavior.

[PATCH v3 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's output entirely, losing downstream
features like word diff, function context, color, and blame.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes.

The first two patches add xdiff plumbing for externally supplied hunks and
the diff.<driver>.process config key. Patch 3 refactors the subprocess API
to separate process lifecycle from hashmap management, since the diff
process stores its subprocess on the userdiff driver rather than in a
hashmap. The main feature lands in patch 4. Patch 5 wires up bypass knobs
(--no-ext-diff, format-patch). Patch 6 integrates with blame so the tool can
declare commits as having no changes.

Changes since v2:

 * Extracted subprocess_start_command() from subprocess_start() so the diff
   process can reuse the startup machinery without a hashmap (new patch 3).
 * Subprocess stored on userdiff_driver, no global variables.
 * Differentiated error handling: tool failure warns (with tool name), "not
   configured" is silent, abort is voluntary.
 * Single public entry point: diff_process_fill_hunks() handles driver
   lookup, flag checks, subprocess management, and error reporting for both
   diff.c and blame.c.
 * Rewrote gitattributes protocol section to match filter process convention
   with packet diagrams and fully specified hunk format.
 * Split bypass knobs (--no-ext-diff, format-patch) into a separate commit.
 * SIGPIPE protection, stricter input validation, const correctness.
 * Changed "zero hunks" to "no hunks" throughout.

Michael Montalbo (6):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 139 +++++
 Makefile                                 |   1 +
 blame.c                                  |  40 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 288 ++++++++++
 diff-process.h                           |  39 ++
 diff.c                                   |  29 +-
 diff.h                                   |   5 +
 meson.build                              |   1 +
 sub-process.c                            |  29 +-
 sub-process.h                            |   9 +-
 t/.gitattributes                         |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 660 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  13 +
 xdiff/xdiffi.c                           |  85 ++-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 24 files changed, 1368 insertions(+), 21 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh


base-commit: 94f057755b7941b321fd11fec1b2e3ca5313a4e0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v2:

 1:  f887a7e2ba ! 1:  13eb201d63 xdiff: support external hunks via xpparam_t
     @@ Commit message
          numbers, overlapping or out-of-order hunks, negative counts, and
          violations of the synchronization invariant (unchanged line counts
          must match between files).  On validation failure, fall back to
     -    the builtin diff algorithm.
     +    the builtin diff algorithm; this re-runs xdl_prepare_env() since
     +    the first call may have dirtied the changed[] arrays.
      
          Skip trim_common_tail() in xdi_diff() when external hunks are
          present, since external hunks reference line numbers in the
     @@ xdiff/xdiff.h: typedef struct s_xpparam {
       	char **anchors;
       	size_t anchors_nr;
      +
     -+	/* Externally computed hunks: bypass the diff algorithm. */
     -+	const struct xdl_hunk *external_hunks;
     ++	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
     ++	struct xdl_hunk *external_hunks;
      +	size_t external_hunks_nr;
       } xpparam_t;
       
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      + * Returns 0 on success, -1 on validation failure.
      + */
      +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
     -+					    const struct xdl_hunk *hunks,
     ++					    struct xdl_hunk *hunks,
      +					    size_t nr_hunks)
      +{
      +	size_t i;
      +	long j, prev_old_end = 0, prev_new_end = 0;
      +	long total_old = 0, total_new = 0;
      +
     ++	/*
     ++	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
     ++	 * Clear them so only the external hunks are marked.
     ++	 */
      +	xdl_clear_changed(&xe->xdf1);
      +	xdl_clear_changed(&xe->xdf2);
      +
      +	for (i = 0; i < nr_hunks; i++) {
     -+		const struct xdl_hunk *h = &hunks[i];
     ++		struct xdl_hunk *h = &hunks[i];
      +
      +		if (h->old_count < 0 || h->new_count < 0)
      +			return -1;
     -+
     -+		/* Bounds check (1-based line numbers) */
     -+		if (h->old_count > 0 &&
     -+		    (h->old_start < 1 ||
     -+		     h->old_start + h->old_count - 1 > (long)xe->xdf1.nrec))
     -+			return -1;
     -+		if (h->new_count > 0 &&
     -+		    (h->new_start < 1 ||
     -+		     h->new_start + h->new_count - 1 > (long)xe->xdf2.nrec))
     ++		if (h->old_start < 1 || h->new_start < 1)
      +			return -1;
      +
     -+		/* Zero-count hunks: start must still be in [1, nrec+1] */
     -+		if (h->old_count == 0 &&
     -+		    (h->old_start < 1 || h->old_start > (long)xe->xdf1.nrec + 1))
     ++		/*
     ++		 * Range must fit: start + count - 1 <= nrec,
     ++		 * rewritten to avoid overflow.  Same for both sides.
     ++		 *
     ++		 * When count is 0 (pure insert/delete) the check
     ++		 * reduces to 0 > nrec - start + 1, which rejects
     ++		 * start > nrec + 1 and allows start == nrec + 1
     ++		 * (the position after the last line).
     ++		 */
     ++		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
      +			return -1;
     -+		if (h->new_count == 0 &&
     -+		    (h->new_start < 1 || h->new_start > (long)xe->xdf2.nrec + 1))
     ++		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
      +			return -1;
      +
     -+		/* Ordering: no overlap with previous hunk */
     ++		/* Ordering: no overlap with previous hunk (adjacent is OK) */
      +		if (h->old_start < prev_old_end ||
      +		    h->new_start < prev_new_end)
      +			return -1;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      -	}
      +
      +diff_done:
     -+
       	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
       	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
       	    xdl_build_script(&xe, &xscr) < 0) {
 2:  de6d85f9d7 ! 2:  58f4763c63 userdiff: add diff.<driver>.process config
     @@ Metadata
       ## Commit message ##
          userdiff: add diff.<driver>.process config
      
     -    Add a new per-driver configuration key that specifies the command
     -    for a long-running diff process.
     -
     -    The name follows filter.<driver>.process: a long-running subprocess
     -    that stays alive across files within a single git invocation.
     +    Add the process field to struct userdiff_driver and teach the
     +    config parser to populate it from diff.<driver>.process.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
 -:  ---------- > 3:  d6c833dd42 sub-process: separate process lifecycle from hashmap management
 3:  c25647c6e5 ! 4:  d044fa0ee5 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
                  process = /path/to/diff-tool
      
          The tool provides custom line-matching: it receives file pairs
     -    and returns hunks that reference original line numbers.  Unlike
     -    textconv, which transforms the displayed content, the diff
     -    output shows the actual file while the tool controls which
     -    lines are marked as changed.
     +    and returns hunks that reference line numbers in the content.
     +    When textconv is also configured, the tool receives the
     +    textconv-transformed content.  The tool controls which lines
     +    are marked as changed while the display shows the file content.
     +    Patch output features (word diff, function context, color) work
     +    normally; summary formats like --stat use their own diff path
     +    and are not affected.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
          requests send command=hunks, pathname, and both file contents as
          packetized data.  The tool responds with hunk lines and a status
     -    packet.  On error, git falls back to the builtin diff algorithm
     -    with a warning.
     +    packet (success, error, or abort).  On error, Git warns and falls
     +    back to the builtin diff algorithm for that file.  On abort, Git
     +    silently falls back for the current file and stops sending further
     +    requests to the tool for the remainder of the session.
      
     -    Zero hunks with status=success means the tool considers the
     -    files equivalent.  Git skips diff output for that file.
     +    When the tool returns no hunks followed by status=success, Git
     +    treats the file as having no changes and produces no diff output.
     +    This also means --exit-code reports no changes for that file.
     +
     +    The subprocess is stored on the userdiff_driver struct and
     +    launched on first use.  If the process fails to start, the
     +    handshake fails, or a communication error occurs mid-stream,
     +    the failure is cached on the driver to avoid retrying and
     +    re-warning on every subsequent file.
     +
     +    diff_process_fill_hunks() is the sole public entry point.  It
     +    handles driver lookup, flag checks, subprocess management, and
     +    error reporting, returning an enum that lets callers distinguish
     +    "hunks populated" from "files equivalent" from "not applicable"
     +    from "tool failure."
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ Documentation/config/diff.adoc: endif::git-diff[]
       	conversion outputs.  See linkgit:gitattributes[5] for details.
       
      +`diff.<driver>.process`::
     -+	The command to run as a long-running diff process.
     -+	The tool communicates via the pkt-line protocol and returns
     -+	hunks that are fed into Git's diff and blame pipelines.
     -+	If the tool returns zero hunks, the file is treated as
     -+	unchanged for both diff output and blame attribution.
     ++	The command to run as a long-running diff process that
     ++	provides hunks to Git's diff pipeline.
      +	See linkgit:gitattributes[5] for details.
      +
       `diff.indentHeuristic`::
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
       algorithm is not passed to the external diff driver.
       
      +Using an external diff process
     -+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     -+
     -+An external tool can provide content-aware line matching by
     -+setting `diff.<name>.process` to the command that runs
     -+the tool.  The tool is a long-running process that communicates via
     -+the pkt-line protocol (described in
     ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ++
     ++If `diff.<name>.process` is defined, Git sends the old and new file
     ++content to an external tool and receives back a list of changed
     ++regions (pairs of line ranges in the old and new file).  Git uses
     ++these instead of its builtin diff algorithm, but still controls
     ++all output formatting, so features like word diff, function context,
     ++color, and blame work normally.  This is achieved by using the
     ++long-running process protocol (described in
      +Documentation/technical/long-running-process-protocol.adoc).
     ++Unlike `diff.<name>.command`, which replaces Git's output entirely,
     ++the diff process feeds results back into the standard pipeline.
     ++
     ++First, in `.gitattributes`, assign the `diff` attribute for paths.
      +
      +------------------------
      +*.c diff=cdiff
      +------------------------
      +
     ++Then, define a "diff.<name>.process" configuration to specify
     ++the diff process command.
     ++
      +----------------------------------------------------------------
      +[diff "cdiff"]
      +  process = /path/to/diff-process-tool
      +----------------------------------------------------------------
      +
     -+The tool receives file pairs and returns hunk descriptors indicating
     -+which lines changed.  Git feeds these hunks into its standard diff
     -+pipeline, so all output features (word diff, function context,
     -+color) work normally.
     -+
     -+If the tool fails or returns an error, Git silently falls back to
     -+the builtin diff algorithm.  If the tool returns invalid hunks
     -+(out of bounds, overlapping), Git also falls back silently.
     -+
     -+The handshake negotiates `version=1` and `capability=hunks`.
     -+Per-file requests send `command=hunks` and `pathname=<path>`,
     -+followed by the old and new file content as packetized data.
     -+The tool responds with lines of the form
     -+`hunk <old_start> <old_count> <new_start> <new_count>`
     -+(1-based line numbers), a flush packet, and `status=success`.
     -+
     -+If the tool returns zero hunks with `status=success`, Git treats
     -+the file as having no changes and produces no diff output.
     -+
     -+Tools should ignore unknown keys in the per-file request to
     -+remain forward-compatible.
     ++When Git encounters the first file that needs to be diffed, it starts
     ++the process and performs the handshake.  In the handshake, the welcome
     ++message sent by Git is "git-diff-client", only version 1 is supported,
     ++and the supported capability is "hunks" (the changed regions
     ++described below).
     ++
     ++For each file, Git sends a list of "key=value" pairs terminated with
     ++a flush packet, followed by the old and new file content as packetized
     ++data, each terminated with a flush packet.  The pathname is relative
     ++to the repository root.  When `diff.<name>.textconv` is also set,
     ++the tool receives the textconv-transformed content rather than the
     ++raw blob.  Git does not send binary files to the diff process.
     ++
     ++-----------------------
     ++packet:          git> command=hunks
     ++packet:          git> pathname=path/file.c
     ++packet:          git> 0000
     ++packet:          git> OLD_CONTENT
     ++packet:          git> 0000
     ++packet:          git> NEW_CONTENT
     ++packet:          git> 0000
     ++-----------------------
     ++
     ++The tool is expected to respond with zero or more hunk lines,
     ++a flush packet, and a status packet terminated with a flush packet.
     ++Each hunk line has the form:
     ++
     ++  `hunk <old_start> <old_count> <new_start> <new_count>`
     ++
     ++where `<old_start>` and `<old_count>` identify a range of lines in
     ++the old file, and `<new_start>` and `<new_count>` identify the
     ++replacement range in the new file.  Start values are 1-based and
     ++counts are non-negative.  Ranges must not extend beyond the end of
     ++the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
     ++at line 3 in the old file were replaced by 4 lines starting at
     ++line 3 in the new file.  An `<old_count>` of 0 means no lines were
     ++removed (pure insertion); a `<new_count>` of 0 means no lines were
     ++added (pure deletion).
     ++
     ++Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
     ++file `"foo\nbar"` both have 2 lines.
     ++
     ++Hunks must be listed in order and must not overlap.  Any line
     ++not covered by a hunk is treated as unchanged, so the total
     ++number of unchanged lines must be the same on both sides.
     ++For example, if the old file has 10 lines and the hunks cover
     ++4 of them (`old_count` values summing to 4), then 6 old lines
     ++are unchanged.  The new file must also have exactly 6 lines
     ++not covered by hunks, so the `new_count` values must sum to
     ++`new_file_lines - 6`.
     ++
     ++-----------------------
     ++packet:          git< hunk 1 3 1 5
     ++packet:          git< hunk 10 2 12 2
     ++packet:          git< 0000
     ++packet:          git< status=success
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++If the tool responds with hunks and "success", Git marks those lines
     ++as changed and feeds them into the standard diff pipeline.  Patch
     ++output features (word diff, function context, color) work normally.
     ++Note that `--stat` and other summary formats use their own diff path
     ++and are not affected by the diff process.
     ++
     ++If no hunk lines precede the flush, followed by "success", Git
     ++treats the files as having no changes: `git diff` produces no output
     ++and `git blame` skips the commit, attributing lines to earlier commits.
     ++
     ++-----------------------
     ++packet:          git< 0000
     ++packet:          git< status=success
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++If the tool returns invalid hunks (out of bounds, overlapping), Git
     ++silently falls back to the builtin diff algorithm.
     ++
     ++In case the tool cannot or does not want to process the content,
     ++it is expected to respond with an "error" status.  Git warns and
     ++falls back to the builtin diff algorithm for this file.  The tool
     ++remains available for subsequent files.
     ++
     ++-----------------------
     ++packet:          git< 0000
     ++packet:          git< status=error
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++In case the tool cannot or does not want to process the content as
     ++well as any future content for the lifetime of the Git process, it
     ++is expected to respond with an "abort" status.  Git silently falls
     ++back to the builtin diff algorithm for this file and does not send
     ++further requests to the tool.
     ++
     ++-----------------------
     ++packet:          git< 0000
     ++packet:          git< status=abort
     ++packet:          git< 0000
     ++-----------------------
     ++
     ++If the tool dies during the communication or does not adhere to the
     ++protocol then Git will stop the process and fall back to the builtin
     ++diff algorithm.  Git warns once and does not restart the process for
     ++subsequent files.
     ++
     ++Tools should ignore unknown keys in the per-file request to remain
     ++forward-compatible.  Future versions of Git may send additional
     ++`command=` values; tools that receive an unrecognized command should
     ++respond with `status=error` rather than terminating.
      +
       Defining a custom hunk-header
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     @@ diff-process.c (new)
      +/*
      + * Diff process backend: communicates with a long-running external
      + * tool via the pkt-line protocol to obtain custom line-matching
     -+ * results.  Unlike textconv, which transforms the displayed content,
     -+ * hunks from a diff process reference original line numbers and
     -+ * the display shows the actual file content.
     ++ * results.  The tool controls which lines are marked as changed
     ++ * while the display shows the file content (after any textconv
     ++ * transformation, if configured).
      + *
      + * Protocol: pkt-line over stdin/stdout, following the pattern of
      + * the long-running filter process protocol (see convert.c).
     @@ diff-process.c (new)
      + *   tool< ... / flush
      + *   tool< status=success / flush
      + *
     -+ * Zero hunks with status=success means the tool considers the
     -+ * files equivalent.  Git will skip the diff for that file.
     ++ * When the tool returns no hunks with status=success, it considers
     ++ * the files equivalent.  Git will skip the diff for that file.
      + */
      +
      +#include "git-compat-util.h"
      +#include "diff-process.h"
     ++#include "diff.h"
     ++#include "gettext.h"
     ++#include "repository.h"
     ++#include "sigchain.h"
      +#include "userdiff.h"
      +#include "sub-process.h"
      +#include "pkt-line.h"
     @@ diff-process.c (new)
      +	unsigned int supported_capabilities;
      +};
      +
     -+static int subprocess_map_initialized;
     -+static struct hashmap subprocess_map;
     -+
      +static int start_diff_process_fn(struct subprocess_entry *subprocess)
      +{
      +	static int versions[] = { 1, 0 };
     @@ diff-process.c (new)
      +		{ NULL, 0 }
      +	};
      +	struct diff_subprocess *entry =
     -+		(struct diff_subprocess *)subprocess;
     ++		container_of(subprocess, struct diff_subprocess, subprocess);
      +
     -+	/* Uses dying pkt-line variant, same as convert.c filters. */
      +	return subprocess_handshake(subprocess, "git-diff",
      +				    versions, NULL,
      +				    capabilities,
      +				    &entry->supported_capabilities);
      +}
      +
     -+static struct diff_subprocess *find_or_start_process(const char *cmd)
     ++static struct diff_subprocess *get_or_launch_process(
     ++		struct userdiff_driver *drv)
      +{
      +	struct diff_subprocess *entry;
      +
     -+	if (!subprocess_map_initialized) {
     -+		subprocess_map_initialized = 1;
     -+		hashmap_init(&subprocess_map, cmd2process_cmp, NULL, 0);
     -+	}
     -+
     -+	entry = (struct diff_subprocess *)
     -+		subprocess_find_entry(&subprocess_map, cmd);
     -+	if (entry)
     -+		return entry;
     ++	if (drv->diff_subprocess)
     ++		return drv->diff_subprocess;
      +
      +	entry = xcalloc(1, sizeof(*entry));
     -+	if (subprocess_start(&subprocess_map, &entry->subprocess,
     -+			     cmd, start_diff_process_fn)) {
     ++	if (subprocess_start_command(&entry->subprocess, drv->process,
     ++				     start_diff_process_fn)) {
      +		free(entry);
     ++		drv->diff_process_failed = 1;
      +		return NULL;
      +	}
      +
     ++	drv->diff_subprocess = entry;
      +	return entry;
      +}
      +
      +static int send_file_content(int fd, const char *buf, long size)
      +{
     -+	int ret;
     ++	int ret = 0;
      +
     ++	if (size < 0)
     ++		return -1;
      +	if (size > 0)
      +		ret = write_packetized_from_buf_no_flush(buf, size, fd);
     -+	else
     -+		ret = 0;
      +	if (ret)
      +		return ret;
      +	return packet_flush_gently(fd);
     @@ diff-process.c (new)
      +{
      +	char *end;
      +
     -+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
     ++	/*
     ++	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
     ++	 * All numbers must be non-negative decimal with no leading
     ++	 * whitespace or sign characters.
     ++	 */
      +	if (!skip_prefix(line, "hunk ", &line))
      +		return -1;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->old_start = strtol(line, &end, 10);
     -+	if (end == line || *end != ' ')
     ++	if (errno || end == line || *end++ != ' ')
      +		return -1;
      +	line = end;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->old_count = strtol(line, &end, 10);
     -+	if (end == line || *end != ' ')
     ++	if (errno || end == line || *end++ != ' ')
      +		return -1;
      +	line = end;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->new_start = strtol(line, &end, 10);
     -+	if (end == line || *end != ' ')
     ++	if (errno || end == line || *end++ != ' ')
      +		return -1;
      +	line = end;
      +
     ++	if (!isdigit(*line))
     ++		return -1;
     ++	errno = 0;
      +	hunk->new_count = strtol(line, &end, 10);
     -+	if (end == line || *end != '\0')
     ++	if (errno || end == line || *end != '\0')
      +		return -1;
      +
      +	return 0;
      +}
      +
     -+int diff_process_get_hunks(struct userdiff_driver *drv,
     -+			   const char *path,
     -+			   const char *old_buf, long old_size,
     -+			   const char *new_buf, long new_size,
     -+			   struct xdl_hunk **hunks_out,
     -+			   size_t *nr_hunks_out)
     ++static enum diff_process_result get_hunks(
     ++		struct userdiff_driver *drv,
     ++		const char *path,
     ++		const char *old_buf, long old_size,
     ++		const char *new_buf, long new_size,
     ++		struct xdl_hunk **hunks_out,
     ++		size_t *nr_hunks_out)
      +{
      +	struct diff_subprocess *backend;
      +	struct child_process *process;
     @@ diff-process.c (new)
      +	int len;
      +	char *line;
      +
     -+	if (!drv || !drv->process)
     -+		return -1;
     -+
     -+	backend = find_or_start_process(drv->process);
     ++	backend = get_or_launch_process(drv);
      +	if (!backend)
     -+		return -1;
     ++		return DIFF_PROCESS_ERROR;
      +
      +	if (!(backend->supported_capabilities & CAP_HUNKS))
     -+		return -1;
     ++		return DIFF_PROCESS_SKIP;
      +
      +	process = subprocess_get_child_process(&backend->subprocess);
      +	fd_in = process->in;
      +	fd_out = process->out;
      +
     ++	sigchain_push(SIGPIPE, SIG_IGN);
     ++
      +	/* Send request */
      +	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
      +	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
      +	    packet_flush_gently(fd_in))
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Send old file content */
      +	if (send_file_content(fd_in, old_buf, old_size))
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Send new file content */
      +	if (send_file_content(fd_in, new_buf, new_size))
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Read hunks until flush packet */
      +	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
      +	       line) {
      +		if (parse_hunk_line(line, &hunk) < 0)
     -+			goto error;
     ++			goto comm_error;
      +		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
      +		hunks[nr_hunks++] = hunk;
      +	}
      +	if (len < 0)
     -+		goto error;
     ++		goto comm_error;
      +
      +	/* Read status */
      +	if (subprocess_read_status(fd_out, &status))
     -+		goto error;
     ++		goto comm_error;
     ++
     ++	if (!strcmp(status.buf, "success")) {
     ++		*hunks_out = hunks;
     ++		*nr_hunks_out = nr_hunks;
     ++		strbuf_release(&status);
     ++		sigchain_pop(SIGPIPE);
     ++		return DIFF_PROCESS_OK;
     ++	}
      +
     -+	if (strcmp(status.buf, "success")) {
     -+		if (!strcmp(status.buf, "abort"))
     -+			backend->supported_capabilities &= ~CAP_HUNKS;
     -+		goto error;
     ++	if (!strcmp(status.buf, "abort")) {
     ++		/*
     ++		 * The tool voluntarily withdrew: stop sending requests
     ++		 * but do not warn (this is not a failure).
     ++		 */
     ++		backend->supported_capabilities &= ~CAP_HUNKS;
     ++		free(hunks);
     ++		strbuf_release(&status);
     ++		sigchain_pop(SIGPIPE);
     ++		return DIFF_PROCESS_SKIP;
      +	}
      +
     -+	*hunks_out = hunks;
     -+	*nr_hunks_out = nr_hunks;
     ++	/* status=error or unknown status */
     ++	free(hunks);
      +	strbuf_release(&status);
     -+	return 0;
     -+
     -+error:
     ++	sigchain_pop(SIGPIPE);
     ++	return DIFF_PROCESS_ERROR;
     ++
     ++comm_error:
     ++	/*
     ++	 * Communication failure (broken pipe, malformed response).
     ++	 * Tear down the process and mark as failed so we do not
     ++	 * retry on every subsequent file.
     ++	 */
     ++	drv->diff_process_failed = 1;
     ++	drv->diff_subprocess = NULL;
     ++	subprocess_stop_command(&backend->subprocess);
     ++	free(backend);
      +	free(hunks);
      +	strbuf_release(&status);
     -+	return -1;
     ++	sigchain_pop(SIGPIPE);
     ++	return DIFF_PROCESS_ERROR;
     ++}
     ++
     ++enum diff_process_result diff_process_fill_hunks(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		const mmfile_t *file_a,
     ++		const mmfile_t *file_b,
     ++		xpparam_t *xpp)
     ++{
     ++	struct userdiff_driver *drv;
     ++	struct xdl_hunk *ext_hunks = NULL;
     ++	size_t nr = 0;
     ++	enum diff_process_result res;
     ++
     ++	if (!diffopt || !path)
     ++		return DIFF_PROCESS_SKIP;
     ++	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
     ++		return DIFF_PROCESS_SKIP;
     ++
     ++	drv = userdiff_find_by_path(diffopt->repo->index, path);
     ++	if (!drv || !drv->process)
     ++		return DIFF_PROCESS_SKIP;
     ++	if (drv->diff_process_failed)
     ++		return DIFF_PROCESS_SKIP;
     ++
     ++	res = get_hunks(drv, path,
     ++			file_a->ptr, file_a->size,
     ++			file_b->ptr, file_b->size,
     ++			&ext_hunks, &nr);
     ++	if (res == DIFF_PROCESS_OK) {
     ++		if (!nr) {
     ++			free(ext_hunks);
     ++			return DIFF_PROCESS_EQUIVALENT;
     ++		}
     ++		xpp->external_hunks = ext_hunks;
     ++		xpp->external_hunks_nr = nr;
     ++		return DIFF_PROCESS_OK;
     ++	}
     ++	if (res == DIFF_PROCESS_ERROR) {
     ++		warning(_("diff process '%s' failed for '%s',"
     ++			  " falling back to builtin diff"),
     ++			drv->process, path);
     ++		return DIFF_PROCESS_ERROR;
     ++	}
     ++	return DIFF_PROCESS_SKIP;
      +}
      
       ## diff-process.h (new) ##
     @@ diff-process.h (new)
      +#ifndef DIFF_PROCESS_H
      +#define DIFF_PROCESS_H
      +
     -+struct userdiff_driver;
     -+struct xdl_hunk;
     ++#include "xdiff/xdiff.h"
     ++
     ++struct diff_options;
     ++
     ++enum diff_process_result {
     ++	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
     ++	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
     ++	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
     ++	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
     ++};
      +
      +/*
     -+ * Query a diff process for hunks describing the changes
     -+ * between old_buf and new_buf.
     ++ * Consult the diff process configured for 'path' and populate
     ++ * xpp->external_hunks with the returned hunks.
      + *
     -+ * The backend is a long-running subprocess configured via
     -+ * diff.<driver>.process.  It receives file content via
     -+ * pkt-line and returns hunks with 1-based line numbers.
     ++ * Handles driver lookup, flag checks (--no-ext-diff,
     ++ * --diff-algorithm), subprocess management, and error reporting.
      + *
     -+ * On success, sets *hunks_out and *nr_hunks_out to a newly allocated
     -+ * array (caller must free) and returns 0.
     ++ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
     ++ * The caller owns xpp->external_hunks and must free() it.
      + *
     -+ * On failure, returns -1.  The caller should fall back to the
     -+ * builtin diff algorithm.
     ++ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
     ++ * (files are considered identical); caller should skip diff/blame.
     ++ * Returns DIFF_PROCESS_SKIP when no process applies; caller
     ++ * should use the builtin diff algorithm.
     ++ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
     ++ * caller should fall back to the builtin diff algorithm.
      + */
     -+int diff_process_get_hunks(struct userdiff_driver *drv,
     -+			   const char *path,
     -+			   const char *old_buf, long old_size,
     -+			   const char *new_buf, long new_size,
     -+			   struct xdl_hunk **hunks_out,
     -+			   size_t *nr_hunks_out);
     ++enum diff_process_result diff_process_fill_hunks(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		const mmfile_t *file_a,
     ++		const mmfile_t *file_b,
     ++		xpparam_t *xpp);
      +
      +#endif /* DIFF_PROCESS_H */
      
     @@ diff.c
       #include "submodule.h"
       #include "hashmap.h"
       #include "mem-pool.h"
     -@@ diff.c: static void builtin_diff(const char *name_a,
     - 		xpparam_t xpp;
     - 		xdemitconf_t xecfg;
     - 		struct emit_callback ecbdata;
     -+		struct xdl_hunk *ext_hunks = NULL;
     - 		unsigned ws_rule;
     - 		const struct userdiff_funcname *pe;
     - 
      @@ diff.c: static void builtin_diff(const char *name_a,
       		xpp.ignore_regex_nr = o->ignore_regex_nr;
       		xpp.anchors = o->anchors;
       		xpp.anchors_nr = o->anchors_nr;
      +
     -+		if (!o->ignore_driver_algorithm &&
     -+		    one->driver && one->driver->process) {
     -+			size_t ext_hunks_nr = 0;
     -+			if (!diff_process_get_hunks(
     -+				    one->driver, name_a,
     -+				    mf1.ptr, mf1.size,
     -+				    mf2.ptr, mf2.size,
     -+				    &ext_hunks, &ext_hunks_nr)) {
     -+				if (!ext_hunks_nr)
     -+					goto free_ab_and_return;
     -+				xpp.external_hunks = ext_hunks;
     -+				xpp.external_hunks_nr = ext_hunks_nr;
     -+			} else {
     -+				warning(_("diff process failed for '%s',"
     -+					  " falling back to builtin diff"),
     -+					name_a);
     -+			}
     ++		if (diff_process_fill_hunks(o, name_a,
     ++					    &mf1, &mf2, &xpp)
     ++		    == DIFF_PROCESS_EQUIVALENT) {
     ++			if (textconv_one)
     ++				free(mf1.ptr);
     ++			if (textconv_two)
     ++				free(mf2.ptr);
     ++			goto free_ab_and_return;
      +		}
      +
       		xecfg.ctxlen = o->context;
     @@ diff.c: static void builtin_diff(const char *name_a,
       		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
       					 &ecbdata, &xpp, &xecfg))
       			die("unable to generate diff for %s", one->path);
     -+		free(ext_hunks);
     ++		free(xpp.external_hunks);
       		if (o->word_diff)
       			free_diff_words_data(&ecbdata);
       		if (textconv_one)
      
     + ## diff.h ##
     +@@ diff.h: struct diff_flags {
     + 	 */
     + 	unsigned allow_external;
     + 
     ++	/** Disables diff.<driver>.process. */
     ++	unsigned no_diff_process;
     ++
     + 	/**
     + 	 * For communication between the calling program and the options parser;
     + 	 * tell the calling program to signal the presence of difference using
     +
     + ## meson.build ##
     +@@ meson.build: libgit_sources = [
     +   'diff-merges.c',
     +   'diff-lib.c',
     +   'diff-no-index.c',
     ++  'diff-process.c',
     +   'diff.c',
     +   'diffcore-break.c',
     +   'diffcore-delta.c',
     +
       ## t/.gitattributes ##
      @@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
       /t8005/*.txt eol=lf
     @@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
       /t0040*.sh whitespace=-indent-with-non-tab
      +/t4080-diff-process.sh whitespace=-indent-with-non-tab
      
     + ## t/meson.build ##
     +@@ t/meson.build: integration_tests = [
     +   't4072-diff-max-depth.sh',
     +   't4073-diff-stat-name-width.sh',
     +   't4074-diff-shifted-matched-group.sh',
     ++  't4080-diff-process.sh',
     +   't4100-apply-stat.sh',
     +   't4101-apply-nonl.sh',
     +   't4102-apply-rename.sh',
     +
       ## t/t4080-diff-process.sh (new) ##
      @@
      +#!/bin/sh
     @@ t/t4080-diff-process.sh (new)
      +#   whole-file  - report all lines as changed (default)
      +#   fixed-hunk  - always report hunk 5 2 5 2
      +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
     -+#   zero-hunk   - return zero hunks (files considered equivalent)
     ++#   bad-sync    - report hunk with mismatched unchanged totals
     ++#   overlap     - report two overlapping hunks
     ++#   no-hunks   - return no hunks (files considered equivalent)
      +#   error       - return status=error for every request
      +#   abort       - return status=abort for every request
      +#   crash       - read one request then exit without responding
     @@ t/t4080-diff-process.sh (new)
      +	    if cmd is None: sys.exit(0)
      +	    old = read_content()
      +	    new = read_content()
     -+	    log(f"command={cmd} pathname={pathname}")
     ++	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
     ++	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
     ++	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
      +
      +	    if mode == "error":
      +	        write_flush()
     @@ t/t4080-diff-process.sh (new)
      +	            write_pkt("hunk 5 2 5 2")
      +	        elif mode == "bad-hunk":
      +	            write_pkt("hunk 999 1 999 1")
     -+	        elif mode == "zero-hunk":
     ++	        elif mode == "bad-sync":
     ++	            write_pkt("hunk 1 2 1 1")
     ++	        elif mode == "overlap":
     ++	            write_pkt("hunk 1 5 1 5")
     ++	            write_pkt("hunk 3 2 3 2")
     ++	        elif mode == "no-hunks":
      +	            pass
      +	        else:
     -+	            ol = len(old.split(b"\n"))
     -+	            nl = len(new.split(b"\n"))
     ++	            ol = old.count(b"\n")
     ++	            nl = new.count(b"\n")
      +	            write_pkt(f"hunk 1 {ol} 1 {nl}")
      +	        write_flush()
      +	        write_pkt("status=success")
     @@ t/t4080-diff-process.sh (new)
      +	setup_backend &&
      +	echo "*.c diff=cdiff" >.gitattributes &&
      +	git add .gitattributes &&
     -+	git commit -m "initial"
     -+'
      +
     -+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
     ++	# boundary.c: 10 lines, changes at 5-6 and 9-10.
     ++	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
      +	cat >boundary.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh (new)
      +	OLD10
      +	EOF
      +	git add boundary.c &&
     -+	git commit -m "add boundary.c" &&
      +
     ++	# worddiff.c: single-line function, value changes 1 -> 999.
     ++	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
     ++	cat >worddiff.c <<-\EOF &&
     ++	int value(void) { return 1; }
     ++	EOF
     ++	git add worddiff.c &&
     ++
     ++	# newfile.c: single-line function, value changes 42 -> 99.
     ++	# Used by: new file, --exit-code, multiple drivers.
     ++	cat >newfile.c <<-\EOF &&
     ++	int new_func(void) { return 42; }
     ++	EOF
     ++	git add newfile.c &&
     ++
     ++	# logtest.c: single-line function for log/format-patch tests.
     ++	# Needs two commits so log -1 has a diff.
     ++	cat >logtest.c <<-\EOF &&
     ++	int logfunc(void) { return 1; }
     ++	EOF
     ++	git add logtest.c &&
     ++
     ++	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
     ++	cat >one.c <<-\EOF &&
     ++	int first(void) { return 1; }
     ++	EOF
     ++	cat >two.c <<-\EOF &&
     ++	int second(void) { return 2; }
     ++	EOF
     ++	git add one.c two.c &&
     ++
     ++	git commit -m "initial" &&
     ++
     ++	# Second commit for logtest.c (so log -1 has something to show).
     ++	cat >logtest.c <<-\EOF &&
     ++	int logfunc(void) { return 2; }
     ++	EOF
     ++	git add logtest.c &&
     ++	git commit -m "change logtest.c" &&
     ++
     ++	# Working tree modifications (not committed).
      +	cat >boundary.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh (new)
      +	NEW10
      +	EOF
      +
     ++	cat >worddiff.c <<-\EOF &&
     ++	int value(void) { return 999; }
     ++	EOF
     ++
     ++	cat >newfile.c <<-\EOF &&
     ++	int new_func(void) { return 99; }
     ++	EOF
     ++
     ++	cat >one.c <<-\EOF &&
     ++	int first(void) { return 10; }
     ++	EOF
     ++
     ++	cat >two.c <<-\EOF
     ++	int second(void) { return 20; }
     ++	EOF
     ++'
     ++
     ++#
     ++# Core behavior: the tool controls which lines are marked as changed.
     ++#
     ++
     ++test_expect_success PYTHON 'diff process hunk boundaries affect output' '
      +	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
      +	# only reports lines 5-6 as changed.  Lines 9-10 should not
      +	# appear as changed in the output.
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "^-OLD6" actual &&
      +	test_grep "^+NEW5" actual &&
      +	test_grep "^+NEW6" actual &&
     -+	! test_grep "^-OLD9" actual &&
     -+	! test_grep "^-OLD10" actual &&
     -+	! test_grep "^+NEW9" actual &&
     -+	! test_grep "^+NEW10" actual
     ++	test_grep ! "^-OLD9" actual &&
     ++	test_grep ! "^-OLD10" actual &&
     ++	test_grep ! "^+NEW9" actual &&
     ++	test_grep ! "^+NEW10" actual
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on tool error status' '
     ++test_expect_success PYTHON 'diff process works with new file' '
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
     -+		diff boundary.c >actual &&
     -+	# Fallback produces the full builtin diff (both change regions).
     -+	test_grep "^-OLD5" actual &&
     -+	test_grep "^+NEW5" actual &&
     -+	test_grep "^-OLD9" actual &&
     -+	test_grep "^+NEW9" actual &&
     -+	# Tool was contacted (it replied with error, not crash).
     -+	test_grep "command=hunks pathname=boundary.c" backend.log
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- newfile.c >actual 2>stderr &&
     ++	test_grep "return 99" actual &&
     ++	test_grep "pathname=newfile.c" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on bad hunks' '
     -+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
     -+		diff boundary.c >actual &&
     -+	test_grep "^-OLD5" actual &&
     -+	test_grep "^+NEW5" actual &&
     -+	test_grep "^-OLD9" actual &&
     -+	test_grep "^+NEW9" actual
     ++test_expect_success PYTHON 'diff process works with added file (empty old side)' '
     ++	cat >added.c <<-\EOF &&
     ++	int added(void) { return 1; }
     ++	EOF
     ++	git add added.c &&
     ++
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --cached -- added.c >actual 2>stderr &&
     ++	test_grep "added" actual &&
     ++	test_grep "pathname=added.c" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on tool crash' '
     -+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
     -+		diff boundary.c >actual &&
     -+	test_grep "^-OLD5" actual &&
     -+	test_grep "^+NEW5" actual &&
     -+	test_grep "^-OLD9" actual &&
     -+	test_grep "^+NEW9" actual
     ++test_expect_success PYTHON 'diff process skipped for binary files' '
     ++	printf "\\0binary" >binary.c &&
     ++	git add binary.c &&
     ++	git commit -m "add binary" &&
     ++	printf "\\0changed" >binary.c &&
     ++
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- binary.c >actual &&
     ++	test_grep "Binary files" actual &&
     ++	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process abort disables for session' '
     -+	cat >abort1.c <<-\EOF &&
     -+	int first(void) { return 1; }
     -+	EOF
     -+	cat >abort2.c <<-\EOF &&
     -+	int second(void) { return 2; }
     -+	EOF
     -+	git add abort1.c abort2.c &&
     -+	git commit -m "add abort files" &&
     ++test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
     ++	echo "not tracked by cdiff" >unmatched.txt &&
     ++	git add unmatched.txt &&
     ++	git commit -m "add unmatched.txt" &&
      +
     -+	cat >abort1.c <<-\EOF &&
     -+	int first(void) { return 10; }
     -+	EOF
     -+	cat >abort2.c <<-\EOF &&
     -+	int second(void) { return 20; }
     -+	EOF
     ++	echo "modified" >unmatched.txt &&
      +
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
     -+		diff -- abort1.c abort2.c >actual &&
     -+	# Both files should still produce diff output via fallback.
     -+	test_grep "return 10" actual &&
     -+	test_grep "return 20" actual &&
     -+	# The tool aborts on the first file and git clears its
     -+	# capability.  The second file never contacts the tool,
     -+	# so the log should have exactly one entry, not two.
     -+	test_grep "command=hunks" backend.log >matches &&
     -+	test_line_count = 1 matches
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- unmatched.txt >actual &&
     ++	test_grep "modified" actual &&
     ++	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process handles multiple files' '
     -+	cat >multi1.c <<-\EOF &&
     -+	int one(void) { return 1; }
     ++test_expect_success PYTHON 'multiple drivers use separate processes' '
     ++	echo "*.h diff=hdiff" >>.gitattributes &&
     ++	git add .gitattributes &&
     ++
     ++	cat >multi.h <<-\EOF &&
     ++	int header(void) { return 1; }
      +	EOF
     -+	cat >multi2.c <<-\EOF &&
     -+	int two(void) { return 2; }
     ++	git add multi.h &&
     ++	git commit -m "add multi.h" &&
     ++
     ++	cat >multi.h <<-\EOF &&
     ++	int header(void) { return 2; }
      +	EOF
     -+	git add multi1.c multi2.c &&
     -+	git commit -m "add multi files" &&
      +
     -+	cat >multi1.c <<-\EOF &&
     -+	int one(void) { return 10; }
     ++	rm -f backend-c.log backend-h.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
     ++	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
     ++		diff -- newfile.c multi.h >actual 2>stderr &&
     ++	test_grep "pathname=newfile.c" backend-c.log &&
     ++	test_grep "pathname=multi.h" backend-h.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process works alongside textconv' '
     ++	write_script uppercase-filter <<-\EOF &&
     ++	tr "a-z" "A-Z" <"$1"
      +	EOF
     -+	cat >multi2.c <<-\EOF &&
     -+	int two(void) { return 20; }
     ++
     ++	cat >textconv.c <<-\EOF &&
     ++	hello world
     ++	EOF
     ++	git add textconv.c &&
     ++	git commit -m "add textconv.c" &&
     ++
     ++	cat >textconv.c <<-\EOF &&
     ++	goodbye world
      +	EOF
      +
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     -+		diff -- multi1.c multi2.c >actual &&
     -+	test_grep "return 10" actual &&
     -+	test_grep "return 20" actual &&
     -+	test_grep "pathname=multi1.c" backend.log &&
     -+	test_grep "pathname=multi2.c" backend.log
     ++	git -c diff.cdiff.textconv="./uppercase-filter" \
     ++	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff -- textconv.c >actual 2>stderr &&
     ++	# The diff process receives textconv-transformed (uppercase) content.
     ++	test_grep "pathname=textconv.c" backend.log &&
     ++	test_grep "old=HELLO WORLD" backend.log &&
     ++	test_grep "new=GOODBYE WORLD" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     ++#
     ++# Downstream features: word diff, log, equivalent files, exit code.
     ++#
     ++
      +test_expect_success PYTHON 'diff process with --word-diff' '
     -+	cat >worddiff.c <<-\EOF &&
     -+	int value(void) { return 1; }
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --word-diff worddiff.c >actual 2>stderr &&
     ++	test_grep "\[-1;-\]" actual &&
     ++	test_grep "{+999;+}" actual &&
     ++	test_grep "pathname=worddiff.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process works with git log -p' '
     ++	# With no-hunks mode, the tool says the files are equivalent,
     ++	# so log -p should show the commit but no diff content.
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		log -1 -p -- logtest.c >actual 2>stderr &&
     ++	test_grep "change logtest.c" actual &&
     ++	test_grep ! "return 2" actual &&
     ++	test_grep "command=hunks pathname=logtest.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
     ++	cat >nohunks.c <<-\EOF &&
     ++	int zero(void) { return 0; }
      +	EOF
     -+	git add worddiff.c &&
     -+	git commit -m "add worddiff.c" &&
     ++	git add nohunks.c &&
     ++	git commit -m "add nohunks.c" &&
      +
     -+	cat >worddiff.c <<-\EOF &&
     -+	int value(void) { return 999; }
     ++	cat >nohunks.c <<-\EOF &&
     ++	int zero(void) { return 999; }
      +	EOF
      +
     -+	git -c diff.cdiff.process="$BACKEND" \
     -+		diff --word-diff worddiff.c >actual &&
     -+	test_grep "\[-1;-\]" actual &&
     -+	test_grep "{+999;+}" actual
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff nohunks.c >actual &&
     ++	test_must_be_empty actual
     ++'
     ++
     ++test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff --exit-code nohunks.c
     ++'
     ++
     ++test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
     ++	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
     ++		diff --exit-code newfile.c
      +'
      +
     ++#
     ++# Bypass mechanisms: flags and commands that skip the diff process.
     ++#
     ++
      +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
      +	rm -f backend.log &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     @@ t/t4080-diff-process.sh (new)
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process works with git log -p' '
     -+	cat >logtest.c <<-\EOF &&
     -+	int logfunc(void) { return 1; }
     -+	EOF
     -+	git add logtest.c &&
     -+	git commit -m "add logtest.c" &&
     ++test_expect_success PYTHON 'diff process not used by --stat' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --stat worddiff.c >actual &&
     ++	test_grep "worddiff.c" actual &&
     ++	test_path_is_missing backend.log
     ++'
      +
     -+	cat >logtest.c <<-\EOF &&
     -+	int logfunc(void) { return 2; }
     -+	EOF
     -+	git add logtest.c &&
     -+	git commit -m "change logtest.c" &&
     ++#
     ++# Error handling and fallback.
     ++#
      +
     ++test_expect_success PYTHON 'diff process fallback on tool error status' '
      +	rm -f backend.log &&
     -+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     -+		log -1 -p -- logtest.c >actual &&
     -+	test_grep "return 2" actual &&
     -+	test_grep "command=hunks pathname=logtest.c" backend.log
     ++	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	# Fallback produces the full builtin diff (both change regions).
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	# Tool was contacted (it replied with error, not crash).
     ++	test_grep "command=hunks pathname=boundary.c" backend.log &&
     ++	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
     -+	cat >zerohunk.c <<-\EOF &&
     -+	int zero(void) { return 0; }
     ++test_expect_success PYTHON 'diff process error keeps tool available for next file' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
     ++		diff -- one.c two.c >actual 2>stderr &&
     ++	# Unlike abort, error keeps the tool available: both files
     ++	# are sent to the tool (and both fall back).
     ++	test_grep "pathname=one.c" backend.log &&
     ++	test_grep "pathname=two.c" backend.log &&
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual
     ++'
     ++
     ++test_expect_success PYTHON 'diff process abort disables for session' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
     ++		diff -- one.c two.c >actual &&
     ++	# Both files should still produce diff output via fallback.
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
     ++	# The tool aborts on the first file and git clears its
     ++	# capability.  The second file never contacts the tool.
     ++	test_grep "pathname=one.c" backend.log &&
     ++	test_grep ! "pathname=two.c" backend.log
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on tool crash' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=crash" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	# Crash is a communication failure, so a warning is emitted.
     ++	test_grep "diff process.*failed" stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process startup failure only warns once' '
     ++	git -c diff.cdiff.process="/nonexistent/tool" \
     ++		diff -- one.c two.c >actual 2>stderr &&
     ++	# Both files produce diff output via fallback.
     ++	test_grep "return 10" actual &&
     ++	test_grep "return 20" actual &&
     ++	# Sentinel prevents repeated warnings: only one, not one per file.
     ++	test_grep "diff process.*failed" stderr >warnings &&
     ++	test_line_count = 1 warnings
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on bad hunks' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	# Invalid hunks are caught by xdiff validation, not the
     ++	# protocol layer, so no warning is emitted.
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
     ++	cat >synctest.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
      +	EOF
     -+	git add zerohunk.c &&
     -+	git commit -m "add zerohunk.c" &&
     ++	git add synctest.c &&
     ++	git commit -m "add synctest.c" &&
      +
     -+	cat >zerohunk.c <<-\EOF &&
     -+	int zero(void) { return 999; }
     ++	cat >synctest.c <<-\EOF &&
     ++	line1
     ++	changed
     ++	line3
      +	EOF
      +
     -+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
     -+		diff zerohunk.c >actual &&
     -+	test_must_be_empty actual
     ++	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
     ++	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
     ++	# The synchronization invariant fails and git falls back.
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
     ++		diff synctest.c >actual 2>stderr &&
     ++	test_grep "changed" actual
     ++'
     ++
     ++test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     ++	# boundary.c has 10 lines, so both hunks are in bounds
     ++	# but they overlap at lines 3-5, triggering the ordering check.
     ++	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "NEW5" actual
      +'
      +
      +test_done
     +
     + ## userdiff.h ##
     +@@
     + 
     + #include "notes-cache.h"
     + 
     ++struct diff_subprocess;
     + struct index_state;
     + struct repository;
     + 
     +@@ userdiff.h: struct userdiff_driver {
     + 	int textconv_want_cache;
     + 	const char *process;
     + 	char *process_owned;
     ++	struct diff_subprocess *diff_subprocess;
     ++	unsigned diff_process_failed : 1;
     + };
     + enum userdiff_driver_type {
     + 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
 -:  ---------- > 5:  f4fd9aa682 diff: bypass diff process with --no-ext-diff and in format-patch
 4:  39ff53acef ! 6:  370e766978 blame: consult diff process for zero-hunk detection
     @@ Metadata
      Author: Michael Montalbo <mmontalbo@gmail.com>
      
       ## Commit message ##
     -    blame: consult diff process for zero-hunk detection
     +    blame: consult diff process for no-hunk detection
      
          When a diff process is configured via diff.<driver>.process,
          consult it during blame's per-commit diffing.  If the process
     -    returns zero hunks for a commit's changes to a file, treat the
     +    returns no hunks for a commit's changes to a file, treat the
          commit as having no changes, causing blame to attribute lines
          to earlier commits.
      
     +    The consultation happens at the pass_blame_to_parent() callsite
     +    using diff_process_fill_hunks(), matching how builtin_diff() in
     +    diff.c uses the same function.  A new diff_hunks_xpp() variant
     +    accepts a pre-populated xpparam_t for this callsite, while the
     +    existing diff_hunks() retains its original signature and behavior.
     +    The copy-detection callsite is unaffected since it does not use
     +    the diff process.
     +
          The subprocess is long-running (one startup cost amortized
          across the blame traversal), but each commit in the file's
          history incurs a round-trip to the tool.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     - ## Documentation/gitattributes.adoc ##
     -@@ Documentation/gitattributes.adoc: The tool responds with lines of the form
     - 
     - If the tool returns zero hunks with `status=success`, Git treats
     - the file as having no changes and produces no diff output.
     -+`git blame` also consults the diff process and skips commits
     -+where it reports zero hunks, attributing lines to earlier commits
     -+instead.
     - 
     - Tools should ignore unknown keys in the per-file request to
     - remain forward-compatible.
     -
       ## blame.c ##
      @@
       #include "tag.h"
       #include "trace2.h"
       #include "blame.h"
      +#include "diff-process.h"
     -+#include "userdiff.h"
     ++#include "xdiff-interface.h"
       #include "alloc.h"
       #include "commit-slab.h"
       #include "bloom.h"
      @@ blame.c: static struct commit *fake_working_tree_commit(struct repository *r,
       
       
     - static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     + 
     +-static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
      -		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     -+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data,
     -+		      int xdl_opts, struct index_state *istate,
     -+		      const char *path)
     ++static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
     ++			  xdl_emit_hunk_consume_func_t hunk_func,
     ++			  void *cb_data, xpparam_t *xpp)
       {
     - 	xpparam_t xpp = {0};
     +-	xpparam_t xpp = {0};
       	xdemitconf_t xecfg = {0};
       	xdemitcb_t ecb = {NULL};
     -+	struct xdl_hunk *ext_hunks = NULL;
     -+	int ret;
       
     - 	xpp.flags = xdl_opts;
     +-	xpp.flags = xdl_opts;
       	xecfg.hunk_func = hunk_func;
       	ecb.priv = cb_data;
      -	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
     ++	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
     ++}
      +
     -+	if (path && istate) {
     -+		struct userdiff_driver *drv;
     -+		drv = userdiff_find_by_path(istate, path);
     -+		if (drv && drv->process) {
     -+			size_t nr = 0;
     -+			if (!diff_process_get_hunks(drv, path,
     -+						    file_a->ptr, file_a->size,
     -+						    file_b->ptr, file_b->size,
     -+						    &ext_hunks, &nr)) {
     -+				if (!nr) {
     -+					/*
     -+					 * Zero hunks: the diff process
     -+					 * considers these files equivalent.
     -+					 * Skip so blame looks past this
     -+					 * commit.
     -+					 */
     -+					return 0;
     -+				}
     -+				xpp.external_hunks = ext_hunks;
     -+				xpp.external_hunks_nr = nr;
     -+			}
     -+		}
     -+	}
     ++static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     ++		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     ++{
     ++	xpparam_t xpp = {0};
      +
     -+	ret = xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
     -+	free(ext_hunks);
     -+	return ret;
     ++	xpp.flags = xdl_opts;
     ++	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
       }
       
       static const char *get_next_line(const char *start, const char *end)
     +@@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
     + 				 struct blame_origin *parent, int ignore_diffs)
     + {
     + 	mmfile_t file_p, file_o;
     ++	xpparam_t xpp = {0};
     + 	struct blame_chunk_cb_data d;
     + 	struct blame_entry *newdest = NULL;
     + 
      @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       			 &sb->num_read_blob, ignore_diffs);
       	sb->num_get_patch++;
       
      -	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
     -+	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts,
     -+		       sb->revs->diffopt.repo->index, target->path))
     - 		die("unable to generate diff (%s -> %s)",
     - 		    oid_to_hex(&parent->commit->object.oid),
     - 		    oid_to_hex(&target->commit->object.oid));
     -@@ blame.c: static void find_copy_in_blob(struct blame_scoreboard *sb,
     - 	 * file_p partially may match that image.
     - 	 */
     - 	memset(split, 0, sizeof(struct blame_entry [3]));
     --	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts))
     -+	if (diff_hunks(file_p, &file_o, handle_split_cb, &d, sb->xdl_opts,
     -+		       NULL, NULL))
     - 		die("unable to generate diff (%s)",
     - 		    oid_to_hex(&parent->commit->object.oid));
     - 	/* remainder, if any, all match the preimage */
     +-		die("unable to generate diff (%s -> %s)",
     +-		    oid_to_hex(&parent->commit->object.oid),
     +-		    oid_to_hex(&target->commit->object.oid));
     ++	xpp.flags = sb->xdl_opts;
     ++	/*
     ++	 * If the diff process considers the files equivalent,
     ++	 * skip the diff so blame looks past this commit.
     ++	 */
     ++	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
     ++				    &file_p, &file_o, &xpp)
     ++	    != DIFF_PROCESS_EQUIVALENT) {
     ++		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
     ++				   &d, &xpp))
     ++			die("unable to generate diff (%s -> %s)",
     ++			    oid_to_hex(&parent->commit->object.oid),
     ++			    oid_to_hex(&target->commit->object.oid));
     ++	}
     ++	free(xpp.external_hunks);
     + 	/* The rest are the same as the parent */
     + 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
     + 		    parent, target, 0);
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks suppresses diff output' '
     - 	test_must_be_empty actual
     +@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     + 	test_grep "NEW5" actual
       '
       
     -+test_expect_success PYTHON 'blame skips commits with zero hunks from diff process' '
     ++#
     ++# Blame integration.
     ++#
     ++
     ++test_expect_success PYTHON 'blame uses tool-provided hunks' '
     ++	cat >blame-hunk.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
     ++	line4
     ++	original5
     ++	original6
     ++	line7
     ++	line8
     ++	line9
     ++	line10
     ++	EOF
     ++	git add blame-hunk.c &&
     ++	git commit -m "add blame-hunk.c" &&
     ++	ORIG=$(git rev-parse --short HEAD) &&
     ++
     ++	cat >blame-hunk.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
     ++	line4
     ++	changed5
     ++	changed6
     ++	line7
     ++	line8
     ++	changed9
     ++	changed10
     ++	EOF
     ++	git add blame-hunk.c &&
     ++	git commit -m "change blame-hunk.c" &&
     ++	CHANGE=$(git rev-parse --short HEAD) &&
     ++
     ++	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
     ++	# so blame should attribute lines 9-10 to the original commit
     ++	# even though the builtin diff would show them as changed.
     ++	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
     ++		blame blame-hunk.c >actual &&
     ++	sed -n "9p" actual >line9 &&
     ++	sed -n "10p" actual >line10 &&
     ++	test_grep "$ORIG" line9 &&
     ++	test_grep "$ORIG" line10 &&
     ++	sed -n "5p" actual >line5 &&
     ++	sed -n "6p" actual >line6 &&
     ++	test_grep "$CHANGE" line5 &&
     ++	test_grep "$CHANGE" line6
     ++'
     ++
     ++test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
      +	cat >blame.c <<-\EOF &&
      +	int main(void)
      +	{
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
      +	EOF
      +	git add blame.c &&
      +	git commit -m "add blame.c" &&
     ++	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
      +
      +	cat >blame.c <<-\EOF &&
      +	int main(void)
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process zero hunks sup
      +	git commit -m "reformat blame.c" &&
      +	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
      +
     -+	# Without zero-hunk mode, blame attributes the change.
     ++	# Without no-hunks mode, blame attributes the change.
      +	git blame blame.c >without &&
      +	test_grep "$BLAME_COMMIT" without &&
      +
     -+	# With zero-hunk mode, the process considers the files equivalent
     -+	# and blame skips the reformat commit.
     -+	git -c diff.cdiff.process="$BACKEND --mode=zero-hunk" \
     ++	# With no-hunks mode, the process considers the files equivalent
     ++	# and blame skips the reformat commit, attributing to the original.
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
      +		blame blame.c >with &&
     -+	! test_grep "$BLAME_COMMIT" with
     ++	test_grep ! "$BLAME_COMMIT" with &&
     ++	test_grep "$ORIG_COMMIT" with
      +'
      +
     ++test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		blame --no-ext-diff blame.c >actual &&
     ++	# Without the process, blame attributes the reformat commit normally.
     ++	test_grep "$BLAME_COMMIT" actual &&
     ++	test_path_is_missing backend.log
     ++'
     ++
     ++test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
     ++	# fixed-hunk mode would narrow blame to lines 5-6, but
     ++	# --no-ext-diff should bypass it and use the builtin diff.
     ++	rm -f backend.log &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
     ++		blame --no-ext-diff blame-hunk.c >actual &&
     ++	# Builtin diff attributes lines 9-10 to the change commit.
     ++	sed -n "9p" actual >line9 &&
     ++	test_grep "$CHANGE" line9 &&
     ++	test_path_is_missing backend.log
     ++'
      +
       test_done

-- 
gitgitgadget

[PATCH v3 1/6] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm; this re-runs xdl_prepare_env() since
the first call may have dirtied the changed[] arrays.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 13 ++++++++
 xdiff/xdiffi.c    | 85 +++++++++++++++++++++++++++++++++++++++++++++--
 xdiff/xprepare.c  | 10 ++++++
 xdiff/xprepare.h  |  1 +
 5 files changed, 113 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index f043330f2a..9542c0bcc2 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..a7e8502e4c 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,15 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based, matching unified diff convention.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +97,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
+	struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index 5455b4690d..b0a01ca856 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,97 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long total_old = 0, total_new = 0;
+
+	/*
+	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
+	 * Clear them so only the external hunks are marked.
+	 */
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0)
+			return -1;
+		if (h->old_start < 1 || h->new_start < 1)
+			return -1;
+
+		/*
+		 * Range must fit: start + count - 1 <= nrec,
+		 * rewritten to avoid overflow.  Same for both sides.
+		 *
+		 * When count is 0 (pure insert/delete) the check
+		 * reduces to 0 > nrec - start + 1, which rejects
+		 * start > nrec + 1 and allows start == nrec + 1
+		 * (the position after the last line).
+		 */
+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
+			return -1;
+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
+			return -1;
+
+		/* Ordering: no overlap with previous hunk (adjacent is OK) */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+		total_old += h->old_count;
+		total_new += h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 */
+	if ((long)xe->xdf1.nrec - total_old !=
+	    (long)xe->xdf2.nrec - total_new)
+		return -1;
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index cd4fc405eb..4645a9a746 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -432,3 +432,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v3 2/6] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index fe710a68bf..81c0bebcce 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -499,6 +499,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v3 3/6] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and
managing a hashmap that indexes running processes by command
string.  The hashmap suits callers like convert.c where many
files may share one filter process looked up by name, but
callers that manage process lifetime through their own data
structures do not need it.

Extract subprocess_start_command() and subprocess_stop_command()
so callers can reuse the child process setup and handshake
machinery without maintaining a hashmap.  subprocess_start()
and subprocess_stop() become thin wrappers that add hashmap
operations on top.

No functional change for existing callers.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 29 ++++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 32 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 83bf0a0e82..33b0bbc831 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
+
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
 
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,27 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err) {
+		return err;
+	}
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
gitgitgadget

[PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference line numbers in the content.
When textconv is also configured, the tool receives the
textconv-transformed content.  The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
normally; summary formats like --stat use their own diff path
and are not affected.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet (success, error, or abort).  On error, Git warns and falls
back to the builtin diff algorithm for that file.  On abort, Git
silently falls back for the current file and stops sending further
requests to the tool for the remainder of the session.

When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
This also means --exit-code reports no changes for that file.

The subprocess is stored on the userdiff_driver struct and
launched on first use.  If the process fails to start, the
handshake fails, or a communication error occurs mid-stream,
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.

diff_process_fill_hunks() is the sole public entry point.  It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc   |   5 +
 Documentation/gitattributes.adoc | 139 ++++++++
 Makefile                         |   1 +
 diff-process.c                   | 288 +++++++++++++++++
 diff-process.h                   |  39 +++
 diff.c                           |  13 +
 diff.h                           |   3 +
 meson.build                      |   1 +
 t/.gitattributes                 |   1 +
 t/meson.build                    |   1 +
 t/t4080-diff-process.sh          | 538 +++++++++++++++++++++++++++++++
 userdiff.h                       |   3 +
 12 files changed, 1032 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..ac0635bb3b 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f20041a323..1700bd8e97 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,145 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If `diff.<name>.process` is defined, Git sends the old and new file
+content to an external tool and receives back a list of changed
+regions (pairs of line ranges in the old and new file).  Git uses
+these instead of its builtin diff algorithm, but still controls
+all output formatting, so features like word diff, function context,
+color, and blame work normally.  This is achieved by using the
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
+the diff process feeds results back into the standard pipeline.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+Then, define a "diff.<name>.process" configuration to specify
+the diff process command.
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+When Git encounters the first file that needs to be diffed, it starts
+the process and performs the handshake.  In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
+described below).
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
+data, each terminated with a flush packet.  The pathname is relative
+to the repository root.  When `diff.<name>.textconv` is also set,
+the tool receives the textconv-transformed content rather than the
+raw blob.  Git does not send binary files to the diff process.
+
+-----------------------
+packet:          git> command=hunks
+packet:          git> pathname=path/file.c
+packet:          git> 0000
+packet:          git> OLD_CONTENT
+packet:          git> 0000
+packet:          git> NEW_CONTENT
+packet:          git> 0000
+-----------------------
+
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
+
+  `hunk <old_start> <old_count> <new_start> <new_count>`
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
+replacement range in the new file.  Start values are 1-based and
+counts are non-negative.  Ranges must not extend beyond the end of
+the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
+at line 3 in the old file were replaced by 4 lines starting at
+line 3 in the new file.  An `<old_count>` of 0 means no lines were
+removed (pure insertion); a `<new_count>` of 0 means no lines were
+added (pure deletion).
+
+Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
+Hunks must be listed in order and must not overlap.  Any line
+not covered by a hunk is treated as unchanged, so the total
+number of unchanged lines must be the same on both sides.
+For example, if the old file has 10 lines and the hunks cover
+4 of them (`old_count` values summing to 4), then 6 old lines
+are unchanged.  The new file must also have exactly 6 lines
+not covered by hunks, so the `new_count` values must sum to
+`new_file_lines - 6`.
+
+-----------------------
+packet:          git< hunk 1 3 1 5
+packet:          git< hunk 10 2 12 2
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
+as changed and feeds them into the standard diff pipeline.  Patch
+output features (word diff, function context, color) work normally.
+Note that `--stat` and other summary formats use their own diff path
+and are not affected by the diff process.
+
+If no hunk lines precede the flush, followed by "success", Git
+treats the files as having no changes: `git diff` produces no output
+and `git blame` skips the commit, attributing lines to earlier commits.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool returns invalid hunks (out of bounds, overlapping), Git
+silently falls back to the builtin diff algorithm.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status.  Git warns and
+falls back to the builtin diff algorithm for this file.  The tool
+remains available for subsequent files.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=error
+packet:          git< 0000
+-----------------------
+
+In case the tool cannot or does not want to process the content as
+well as any future content for the lifetime of the Git process, it
+is expected to respond with an "abort" status.  Git silently falls
+back to the builtin diff algorithm for this file and does not send
+further requests to the tool.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=abort
+packet:          git< 0000
+-----------------------
+
+If the tool dies during the communication or does not adhere to the
+protocol then Git will stop the process and fall back to the builtin
+diff algorithm.  Git warns once and does not restart the process for
+subsequent files.
+
+Tools should ignore unknown keys in the per-file request to remain
+forward-compatible.  Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index cedc234173..22900368dd 100644
--- a/Makefile
+++ b/Makefile
@@ -1142,6 +1142,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..d2ef9463d7
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,288 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  The tool controls which lines are marked as changed
+ * while the display shows the file content (after any textconv
+ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * When the tool returns no hunks with status=success, it considers
+ * the files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *get_or_launch_process(
+		struct userdiff_driver *drv)
+{
+	struct diff_subprocess *entry;
+
+	if (drv->diff_subprocess)
+		return drv->diff_subprocess;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start_command(&entry->subprocess, drv->process,
+				     start_diff_process_fn)) {
+		free(entry);
+		drv->diff_process_failed = 1;
+		return NULL;
+	}
+
+	drv->diff_subprocess = entry;
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret = 0;
+
+	if (size < 0)
+		return -1;
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/*
+	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
+	 * All numbers must be non-negative decimal with no leading
+	 * whitespace or sign characters.
+	 */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_count = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_count = strtol(line, &end, 10);
+	if (errno || end == line || *end != '\0')
+		return -1;
+
+	return 0;
+}
+
+static enum diff_process_result get_hunks(
+		struct userdiff_driver *drv,
+		const char *path,
+		const char *old_buf, long old_size,
+		const char *new_buf, long new_size,
+		struct xdl_hunk **hunks_out,
+		size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	backend = get_or_launch_process(drv);
+	if (!backend)
+		return DIFF_PROCESS_ERROR;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto comm_error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto comm_error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto comm_error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto comm_error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto comm_error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		*hunks_out = hunks;
+		*nr_hunks_out = nr_hunks;
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_OK;
+	}
+
+	if (!strcmp(status.buf, "abort")) {
+		/*
+		 * The tool voluntarily withdrew: stop sending requests
+		 * but do not warn (this is not a failure).
+		 */
+		backend->supported_capabilities &= ~CAP_HUNKS;
+		free(hunks);
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_SKIP;
+	}
+
+	/* status=error or unknown status */
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+
+comm_error:
+	/*
+	 * Communication failure (broken pipe, malformed response).
+	 * Tear down the process and mark as failed so we do not
+	 * retry on every subsequent file.
+	 */
+	drv->diff_process_failed = 1;
+	drv->diff_subprocess = NULL;
+	subprocess_stop_command(&backend->subprocess);
+	free(backend);
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+	struct xdl_hunk *ext_hunks = NULL;
+	size_t nr = 0;
+	enum diff_process_result res;
+
+	if (!diffopt || !path)
+		return DIFF_PROCESS_SKIP;
+	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+		return DIFF_PROCESS_SKIP;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return DIFF_PROCESS_SKIP;
+	if (drv->diff_process_failed)
+		return DIFF_PROCESS_SKIP;
+
+	res = get_hunks(drv, path,
+			file_a->ptr, file_a->size,
+			file_b->ptr, file_b->size,
+			&ext_hunks, &nr);
+	if (res == DIFF_PROCESS_OK) {
+		if (!nr) {
+			free(ext_hunks);
+			return DIFF_PROCESS_EQUIVALENT;
+		}
+		xpp->external_hunks = ext_hunks;
+		xpp->external_hunks_nr = nr;
+		return DIFF_PROCESS_OK;
+	}
+	if (res == DIFF_PROCESS_ERROR) {
+		warning(_("diff process '%s' failed for '%s',"
+			  " falling back to builtin diff"),
+			drv->process, path);
+		return DIFF_PROCESS_ERROR;
+	}
+	return DIFF_PROCESS_SKIP;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..d34b42f811
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,39 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+#include "xdiff/xdiff.h"
+
+struct diff_options;
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
+	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
+	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
+/*
+ * Consult the diff process configured for 'path' and populate
+ * xpp->external_hunks with the returned hunks.
+ *
+ * Handles driver lookup, flag checks (--no-ext-diff,
+ * --diff-algorithm), subprocess management, and error reporting.
+ *
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
+ * (files are considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
+ */
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 397e38b41c..2d5ed6ea8c 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -4031,6 +4032,17 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (diff_process_fill_hunks(o, name_a,
+					    &mf1, &mf2, &xpp)
+		    == DIFF_PROCESS_EQUIVALENT) {
+			if (textconv_one)
+				free(mf1.ptr);
+			if (textconv_two)
+				free(mf2.ptr);
+			goto free_ab_and_return;
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4111,6 +4123,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(xpp.external_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/diff.h b/diff.h
index 7eb84aadf4..d1e5a13e9e 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/** Disables diff.<driver>.process. */
+	unsigned no_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 11488623bf..8a7370b38f 100644
--- a/meson.build
+++ b/meson.build
@@ -328,6 +328,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/t/.gitattributes b/t/.gitattributes
index 7664c6e027..de97920cab 100644
--- a/t/.gitattributes
+++ b/t/.gitattributes
@@ -23,3 +23,4 @@ t[0-9][0-9][0-9][0-9]/* -whitespace
 /t8005/*.txt eol=lf
 /t9*/*.dump eol=lf
 /t0040*.sh whitespace=-indent-with-non-tab
+/t4080-diff-process.sh whitespace=-indent-with-non-tab
diff --git a/t/meson.build b/t/meson.build
index 7528e5cda5..f67208d7ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -510,6 +510,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..f159cd86d8
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,538 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+if test_have_prereq PYTHON
+then
+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+fi
+
+#
+# A single parametric diff process.
+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
+#
+# Modes:
+#   whole-file  - report all lines as changed (default)
+#   fixed-hunk  - always report hunk 5 2 5 2
+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
+#   bad-sync    - report hunk with mismatched unchanged totals
+#   overlap     - report two overlapping hunks
+#   no-hunks   - return no hunks (files considered equivalent)
+#   error       - return status=error for every request
+#   abort       - return status=abort for every request
+#   crash       - read one request then exit without responding
+#
+setup_backend () {
+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
+	import sys, os
+
+	def read_pkt():
+	    hdr = sys.stdin.buffer.read(4)
+	    if len(hdr) < 4: return None
+	    length = int(hdr, 16)
+	    if length == 0: return ""
+	    data = sys.stdin.buffer.read(length - 4)
+	    return data.decode().rstrip("\n")
+
+	def write_pkt(line):
+	    data = (line + "\n").encode()
+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    sys.stdout.buffer.flush()
+
+	def write_flush():
+	    sys.stdout.buffer.write(b"0000")
+	    sys.stdout.buffer.flush()
+
+	def read_content():
+	    chunks = []
+	    while True:
+	        hdr = sys.stdin.buffer.read(4)
+	        if len(hdr) < 4: break
+	        length = int(hdr, 16)
+	        if length == 0: break
+	        chunks.append(sys.stdin.buffer.read(length - 4))
+	    return b"".join(chunks)
+
+	mode = "whole-file"
+	logfile = None
+	for arg in sys.argv[1:]:
+	    if arg.startswith("--mode="):
+	        mode = arg[7:]
+	    elif arg.startswith("--log="):
+	        logfile = open(arg[6:], "a")
+
+	def log(msg):
+	    if logfile:
+	        logfile.write(msg + "\n")
+	        logfile.flush()
+
+	# Handshake
+	assert read_pkt() == "git-diff-client"
+	assert read_pkt() == "version=1"
+	read_pkt()
+	write_pkt("git-diff-server")
+	write_pkt("version=1")
+	write_flush()
+	while True:
+	    p = read_pkt()
+	    if p == "": break
+	write_pkt("capability=hunks")
+	write_flush()
+
+	log("ready")
+
+	while True:
+	    cmd = None
+	    pathname = None
+	    while True:
+	        p = read_pkt()
+	        if p is None: sys.exit(0)
+	        if p == "": break
+	        if p.startswith("command="): cmd = p.split("=",1)[1]
+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
+	    if cmd is None: sys.exit(0)
+	    old = read_content()
+	    new = read_content()
+	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
+	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
+	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
+
+	    if mode == "error":
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	        continue
+
+	    if mode == "abort":
+	        write_flush()
+	        write_pkt("status=abort")
+	        write_flush()
+	        continue
+
+	    if mode == "crash":
+	        sys.exit(1)
+
+	    if cmd == "hunks":
+	        if mode == "fixed-hunk":
+	            write_pkt("hunk 5 2 5 2")
+	        elif mode == "bad-hunk":
+	            write_pkt("hunk 999 1 999 1")
+	        elif mode == "bad-sync":
+	            write_pkt("hunk 1 2 1 1")
+	        elif mode == "overlap":
+	            write_pkt("hunk 1 5 1 5")
+	            write_pkt("hunk 3 2 3 2")
+	        elif mode == "no-hunks":
+	            pass
+	        else:
+	            ol = old.count(b"\n")
+	            nl = new.count(b"\n")
+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	        write_flush()
+	        write_pkt("status=success")
+	        write_flush()
+	    else:
+	        write_flush()
+	        write_pkt("status=error")
+	        write_flush()
+	PYEOF
+	write_script diff-process-backend <<-SHEOF
+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
+	SHEOF
+}
+
+BACKEND="./diff-process-backend"
+
+test_expect_success PYTHON 'setup' '
+	setup_backend &&
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# boundary.c: 10 lines, changes at 5-6 and 9-10.
+	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+
+	# worddiff.c: single-line function, value changes 1 -> 999.
+	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+
+	# newfile.c: single-line function, value changes 42 -> 99.
+	# Used by: new file, --exit-code, multiple drivers.
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 42; }
+	EOF
+	git add newfile.c &&
+
+	# logtest.c: single-line function for log/format-patch tests.
+	# Needs two commits so log -1 has a diff.
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+
+	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
+	cat >one.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >two.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add one.c two.c &&
+
+	git commit -m "initial" &&
+
+	# Second commit for logtest.c (so log -1 has something to show).
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	# Working tree modifications (not committed).
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 99; }
+	EOF
+
+	cat >one.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+
+	cat >two.c <<-\EOF
+	int second(void) { return 20; }
+	EOF
+'
+
+#
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	test_grep ! "^-OLD9" actual &&
+	test_grep ! "^-OLD10" actual &&
+	test_grep ! "^+NEW9" actual &&
+	test_grep ! "^+NEW10" actual
+'
+
+test_expect_success PYTHON 'diff process works with new file' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- newfile.c >actual 2>stderr &&
+	test_grep "return 99" actual &&
+	test_grep "pathname=newfile.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process works with added file (empty old side)' '
+	cat >added.c <<-\EOF &&
+	int added(void) { return 1; }
+	EOF
+	git add added.c &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "added" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process skipped for binary files' '
+	printf "\\0binary" >binary.c &&
+	git add binary.c &&
+	git commit -m "add binary" &&
+	printf "\\0changed" >binary.c &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- binary.c >actual &&
+	test_grep "Binary files" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
+	echo "not tracked by cdiff" >unmatched.txt &&
+	git add unmatched.txt &&
+	git commit -m "add unmatched.txt" &&
+
+	echo "modified" >unmatched.txt &&
+
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- unmatched.txt >actual &&
+	test_grep "modified" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'multiple drivers use separate processes' '
+	echo "*.h diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 1; }
+	EOF
+	git add multi.h &&
+	git commit -m "add multi.h" &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 2; }
+	EOF
+
+	rm -f backend-c.log backend-h.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+		diff -- newfile.c multi.h >actual 2>stderr &&
+	test_grep "pathname=newfile.c" backend-c.log &&
+	test_grep "pathname=multi.h" backend-h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process works alongside textconv' '
+	write_script uppercase-filter <<-\EOF &&
+	tr "a-z" "A-Z" <"$1"
+	EOF
+
+	cat >textconv.c <<-\EOF &&
+	hello world
+	EOF
+	git add textconv.c &&
+	git commit -m "add textconv.c" &&
+
+	cat >textconv.c <<-\EOF &&
+	goodbye world
+	EOF
+
+	rm -f backend.log &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- textconv.c >actual 2>stderr &&
+	# The diff process receives textconv-transformed (uppercase) content.
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=HELLO WORLD" backend.log &&
+	test_grep "new=GOODBYE WORLD" backend.log &&
+	test_must_be_empty stderr
+'
+
+#
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
+test_expect_success PYTHON 'diff process with --word-diff' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --word-diff worddiff.c >actual 2>stderr &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual &&
+	test_grep "pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process works with git log -p' '
+	# With no-hunks mode, the tool says the files are equivalent,
+	# so log -p should show the commit but no diff content.
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -1 -p -- logtest.c >actual 2>stderr &&
+	test_grep "change logtest.c" actual &&
+	test_grep ! "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add nohunks.c &&
+	git commit -m "add nohunks.c" &&
+
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff nohunks.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code nohunks.c
+'
+
+test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
+	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+		diff --exit-code newfile.c
+'
+
+#
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process not used by --stat' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --stat worddiff.c >actual &&
+	test_grep "worddiff.c" actual &&
+	test_path_is_missing backend.log
+'
+
+#
+# Error handling and fallback.
+#
+
+test_expect_success PYTHON 'diff process fallback on tool error status' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success PYTHON 'diff process error keeps tool available for next file' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Unlike abort, error keeps the tool available: both files
+	# are sent to the tool (and both fall back).
+	test_grep "pathname=one.c" backend.log &&
+	test_grep "pathname=two.c" backend.log &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual
+'
+
+test_expect_success PYTHON 'diff process abort disables for session' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- one.c two.c >actual &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool.
+	test_grep "pathname=one.c" backend.log &&
+	test_grep ! "pathname=two.c" backend.log
+'
+
+test_expect_success PYTHON 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Crash is a communication failure, so a warning is emitted.
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success PYTHON 'diff process startup failure only warns once' '
+	git -c diff.cdiff.process="/nonexistent/tool" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# Sentinel prevents repeated warnings: only one, not one per file.
+	test_grep "diff process.*failed" stderr >warnings &&
+	test_line_count = 1 warnings
+'
+
+test_expect_success PYTHON 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Invalid hunks are caught by xdiff validation, not the
+	# protocol layer, so no warning is emitted.
+	test_must_be_empty stderr
+'
+
+test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
+	cat >synctest.c <<-\EOF &&
+	line1
+	line2
+	line3
+	EOF
+	git add synctest.c &&
+	git commit -m "add synctest.c" &&
+
+	cat >synctest.c <<-\EOF &&
+	line1
+	changed
+	line3
+	EOF
+
+	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
+	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
+	# The synchronization invariant fails and git falls back.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+		diff synctest.c >actual 2>stderr &&
+	test_grep "changed" actual
+'
+
+test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
+	# boundary.c has 10 lines, so both hunks are in bounds
+	# but they overlap at lines 3-5, triggering the ordering check.
+	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "NEW5" actual
+'
+
+test_done
diff --git a/userdiff.h b/userdiff.h
index 51c26e0d41..a98eabe377 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -3,6 +3,7 @@
 
 #include "notes-cache.h"
 
+struct diff_subprocess;
 struct index_state;
 struct repository;
 
@@ -33,6 +34,8 @@ struct userdiff_driver {
 	int textconv_want_cache;
 	const char *process;
 	char *process_owned;
+	struct diff_subprocess *diff_subprocess;
+	unsigned diff_process_failed : 1;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v3 5/6] diff: bypass diff process with --no-ext-diff and in format-patch

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Make --no-ext-diff disable diff.<driver>.process in addition to
diff.<driver>.command.  Although the two mechanisms work differently
(command replaces Git's output, process feeds hunks back into the
pipeline), both invoke external tools and --no-ext-diff means
"no external tools."

Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that
sets both allow_external and no_diff_process, so a single option
controls both.  Passing --ext-diff explicitly clears
no_diff_process, so a later --ext-diff overrides an earlier
--no-ext-diff.

Disable the diff process unconditionally in format-patch so that
generated patches are always based on the builtin diff algorithm
and can be applied reliably by recipients who do not have the
external tool.

Document that --diff-algorithm also bypasses the diff process,
since it sets ignore_driver_algorithm which diff_process_fill_hunks
already checks.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/diff-algorithm-option.adoc |  3 +++
 Documentation/diff-options.adoc          |  4 +++-
 builtin/log.c                            |  7 +++++++
 diff.c                                   | 16 ++++++++++++++--
 diff.h                                   |  4 +++-
 t/t4080-diff-process.sh                  | 16 ++++++++++++++++
 6 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..4d7e2ec35f 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index 8a63b5e164..18b8b0ed24 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -825,7 +825,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::
diff --git a/builtin/log.c b/builtin/log.c
index 8c0939dd42..1ea520c12d 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2213,6 +2213,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * Disable diff.<driver>.process so that patches generated by
+	 * format-patch are always based on the builtin diff algorithm
+	 * and can be applied reliably.
+	 */
+	rev.diffopt.flags.no_diff_process = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff.c b/diff.c
index 2d5ed6ea8c..38932db084 100644
--- a/diff.c
+++ b/diff.c
@@ -5913,6 +5913,17 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.no_diff_process = unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6241,8 +6252,9 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index d1e5a13e9e..71ba389f03 100644
--- a/diff.h
+++ b/diff.h
@@ -173,7 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
-	/** Disables diff.<driver>.process. */
+	/**
+	 * Disables diff.<driver>.process.  Set by --no-ext-diff.
+	 */
 	unsigned no_diff_process;
 
 	/**
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index f159cd86d8..ee0c306abd 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -419,6 +419,22 @@ test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
 	test_path_is_missing backend.log
 '
 
+test_expect_success PYTHON 'diff process bypassed by --no-ext-diff' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --no-ext-diff worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'diff process not used by format-patch' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		format-patch -1 --stdout -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_path_is_missing backend.log
+'
+
 test_expect_success PYTHON 'diff process not used by --stat' '
 	rm -f backend.log &&
 	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
-- 
gitgitgadget

[PATCH v3 6/6] blame: consult diff process for no-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The consultation happens at the pass_blame_to_parent() callsite
using diff_process_fill_hunks(), matching how builtin_diff() in
diff.c uses the same function.  A new diff_hunks_xpp() variant
accepts a pre-populated xpparam_t for this callsite, while the
existing diff_hunks() retains its original signature and behavior.
The copy-detection callsite is unaffected since it does not use
the diff process.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                 |  40 +++++++++++----
 t/t4080-diff-process.sh | 106 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+), 9 deletions(-)

diff --git a/blame.c b/blame.c
index a3c49d132e..79f02735a4 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "xdiff-interface.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -314,17 +316,25 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 
-static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
+			  xdl_emit_hunk_consume_func_t hunk_func,
+			  void *cb_data, xpparam_t *xpp)
 {
-	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
 
-	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
+}
+
+static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+{
+	xpparam_t xpp = {0};
+
+	xpp.flags = xdl_opts;
+	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1943,6 +1953,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *parent, int ignore_diffs)
 {
 	mmfile_t file_p, file_o;
+	xpparam_t xpp = {0};
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
 
@@ -1961,10 +1972,21 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
-		die("unable to generate diff (%s -> %s)",
-		    oid_to_hex(&parent->commit->object.oid),
-		    oid_to_hex(&target->commit->object.oid));
+	xpp.flags = sb->xdl_opts;
+	/*
+	 * If the diff process considers the files equivalent,
+	 * skip the diff so blame looks past this commit.
+	 */
+	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
+				    &file_p, &file_o, &xpp)
+	    != DIFF_PROCESS_EQUIVALENT) {
+		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
+				   &d, &xpp))
+			die("unable to generate diff (%s -> %s)",
+			    oid_to_hex(&parent->commit->object.oid),
+			    oid_to_hex(&target->commit->object.oid));
+	}
+	free(xpp.external_hunks);
 	/* The rest are the same as the parent */
 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
 		    parent, target, 0);
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index ee0c306abd..fdf6da1c34 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -551,4 +551,110 @@ test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
 	test_grep "NEW5" actual
 '
 
+#
+# Blame integration.
+#
+
+test_expect_success PYTHON 'blame uses tool-provided hunks' '
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "add blame-hunk.c" &&
+	ORIG=$(git rev-parse --short HEAD) &&
+
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "change blame-hunk.c" &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+
+	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
+	# so blame should attribute lines 9-10 to the original commit
+	# even though the builtin diff would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		blame blame-hunk.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	sed -n "6p" actual >line6 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "$CHANGE" line6
+'
+
+test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	    return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	        return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without no-hunks mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With no-hunks mode, the process considers the files equivalent
+	# and blame skips the reformat commit, attributing to the original.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		blame blame.c >with &&
+	test_grep ! "$BLAME_COMMIT" with &&
+	test_grep "$ORIG_COMMIT" with
+'
+
+test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		blame --no-ext-diff blame.c >actual &&
+	# Without the process, blame attributes the reformat commit normally.
+	test_grep "$BLAME_COMMIT" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
+	# fixed-hunk mode would narrow blame to lines 5-6, but
+	# --no-ext-diff should bypass it and use the builtin diff.
+	rm -f backend.log &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		blame --no-ext-diff blame-hunk.c >actual &&
+	# Builtin diff attributes lines 9-10 to the change commit.
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v3 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
> can produce better line matching than Git's builtin diff algorithm, but
> diff.<driver>.command replaces Git's output entirely, losing downstream
> features like word diff, function context, color, and blame.

This seems to break CI on Windows; take a look at

  https://github.com/git/git/actions/runs/26709491830/job/78717295153

for an example.

Thanks.

Re: [PATCH v3 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo]

On Sun, May 31, 2026 at 3:44 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
> > can produce better line matching than Git's builtin diff algorithm, but
> > diff.<driver>.command replaces Git's output entirely, losing downstream
> > features like word diff, function context, color, and blame.
>
> This seems to break CI on Windows; take a look at
>
>   https://github.com/git/git/actions/runs/26709491830/job/78717295153
>
> for an example.
>
> Thanks.

Thanks for the heads up. Interestingly, I think Windows exposed a latent issue
with sub-process startup dying instead of erroring out when there is a space in
the path.

I've submitted https://lore.kernel.org/git/pull.2133.git.1780287309846.gitgitgadget@gmail.com/T/#u
which I believe addresses the issue, however it seems like there are
some potentially
unrelated CI failures going on right now which is making verification
a bit harder.

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Johannes Schindelin]

Hi Michael,

I stumbled about this patch when it broke CI in Git for Windows, where we
do _not_ use `NO_PYTHON`, even though Python is unavailable in the
build/test CI jobs. The existing tests handle this situation gracefully,
this here patch does not:

On Sun, 7 Jun 2026, Michael Montalbo via GitGitGadget wrote:

> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> new file mode 100755
> index 0000000000..f159cd86d8
> --- /dev/null
> +++ b/t/t4080-diff-process.sh
> @@ -0,0 +1,538 @@
> +#!/bin/sh
> +
> +test_description='diff process via long-running process'
> +
> +. ./test-lib.sh
> +
> +if test_have_prereq PYTHON
> +then
> +	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)

When neither `python3` nor `python` are available (which is the case in
the minimal Git for Windows SDK used in Git's CI runs), this fails under
`set -e`. Before even running the first test case. Resulting in an
unexpected TAP format error.

Now, we could "fix" this by imitating what `lib-p4` does (see
https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
which demonstrates that it is indeed a work-around on Windows):

-- snip --
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index fdf6da1c341e67..bd22c247ff3856 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -4,9 +4,10 @@ test_description='diff process via long-running process'
 
 . ./test-lib.sh
 
-if test_have_prereq PYTHON
+if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH"
 then
-	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
+	skip_all='python interpreter not available'
+	test_done
 fi
 
 #
-- snap --

Of course, this uncovers _another_ problem with the Python script: It uses
Python3-only `f"..."` format strings, which cannot be handled by the
Python2 to which the `PYTHON_PATH` variable in `linux-TEST-vars` points.
So this requires _another follow-up (see also
https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):

-- snip --
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index bd22c247ff3856..ba14682a9086e4 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -39,7 +39,8 @@ setup_backend () {
 
 	def write_pkt(line):
 	    data = (line + "\n").encode()
-	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
+	    hdr = "{:04x}".format(len(data) + 4).encode()
+	    sys.stdout.buffer.write(hdr + data)
 	    sys.stdout.buffer.flush()
 
 	def write_flush():
@@ -98,7 +99,8 @@ setup_backend () {
 	    new = read_content()
 	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
 	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
-	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
+	    log("command={} pathname={} old={} new={}".format(
+	        cmd, pathname, old_first, new_first))
 
 	    if mode == "error":
 	        write_flush()
@@ -130,7 +132,7 @@ setup_backend () {
 	        else:
 	            ol = old.count(b"\n")
 	            nl = new.count(b"\n")
-	            write_pkt(f"hunk 1 {ol} 1 {nl}")
+	            write_pkt("hunk 1 {} 1 {}".format(ol, nl))
 	        write_flush()
 	        write_pkt("status=success")
 	        write_flush()
-- snap --

And this is still not enough to make it work with Python2, see
https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:

-- snip --
[...]
+ git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diff boundary.c
  Traceback (most recent call last):
    File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
      assert read_pkt() == "git-diff-client"
    File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 4, in read_pkt
      hdr = sys.stdin.buffer.read(4)
  AttributeError: 'file' object has no attribute 'buffer'
-- snap --

I have experienced similar patterns in my career, where a single decision
required multiple follow-up fixes _just_ to avoid having to revert that
decision. This kind of doubling down has never ended well.

Therefore I would like to take a step back, and ask: Is it _really_ a good
idea to use Python here? Are we certain that we want to _require_ Python
to run this test and skip it if Python isn't available (as is the case in
the Windows-related parts of Git's very own CI) even if Python has nothing
at all to do with the feature that is being tested?

I don't want to be doomed to repeat history, and we can very well learn
e.g. from prior art in this very project, where the tests for the
clean/smudge filters (which _also_ want to speak pkt-line over stdio)
needlessly incurred Perl as a requirement to run the tests. It was
Matheus's heroic work in 52917a998ef3a (t0021: implementation the
rot13-filter.pl script in C, 2022-08-14) and 4d1d843be7a15 (tests: use the
new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided
that unnecessary prerequisite.

Likewise, there is `test-tool pkt-line` intended for driving the pkt-line
protocol via simple shell scripts.

So the conscious project direction has been: fold pkt-line test backends
into `test-tool` and drop the scripting-language prereq. Reintroducing the
same shape in Python would walk this back.

Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
2025-04-29) has been another clear sign that the Git project is actively
_removing_ scripting-language dependencies from the build and test
infrastructure, not adding new ones.

The clear prior art in Git's own tests for what t4080 wants to do, as of
today, is `t/helper/test-rot13-filter.c`, which could be imitated here
instead of (re-)introducing a dependency on a scripting language other
than Unix shell in Git's test suite.

The `PYTHON` prereq exists in exactly five files today, all `git p4`
related (where Python is an intrinsic prerequisite given that `git-p4.py`
_is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
`t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
and `t/t9836-git-p4-metadata-encoding-python3.sh`.

After 7cdbff14d482 (remove merge-recursive-old, 2006-11-20), this here
patch would be the first one, after almost 20 years, to re-introduce
Python as a dependency outside `git p4`.

And it would also be the first ever to embed a Python script as a heredoc:

> +fi
> +
> +#
> +# A single parametric diff process.
> +# Usage: diff-process-backend --mode=<mode> [--log=<path>]
> +#
> +# Modes:
> +#   whole-file  - report all lines as changed (default)
> +#   fixed-hunk  - always report hunk 5 2 5 2
> +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
> +#   bad-sync    - report hunk with mismatched unchanged totals
> +#   overlap     - report two overlapping hunks
> +#   no-hunks   - return no hunks (files considered equivalent)
> +#   error       - return status=error for every request
> +#   abort       - return status=abort for every request
> +#   crash       - read one request then exit without responding
> +#
> +setup_backend () {
> +	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
> +	import sys, os
> +
> +	def read_pkt():
> +	    hdr = sys.stdin.buffer.read(4)
> +	    if len(hdr) < 4: return None
> +	    length = int(hdr, 16)
> +	    if length == 0: return ""
> +	    data = sys.stdin.buffer.read(length - 4)
> +	    return data.decode().rstrip("\n")
> +
> +	def write_pkt(line):
> +	    data = (line + "\n").encode()
> +	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> +	    sys.stdout.buffer.flush()
> +
> +	def write_flush():
> +	    sys.stdout.buffer.write(b"0000")
> +	    sys.stdout.buffer.flush()
> +
> +	def read_content():
> +	    chunks = []
> +	    while True:
> +	        hdr = sys.stdin.buffer.read(4)
> +	        if len(hdr) < 4: break
> +	        length = int(hdr, 16)
> +	        if length == 0: break
> +	        chunks.append(sys.stdin.buffer.read(length - 4))
> +	    return b"".join(chunks)
> +
> +	mode = "whole-file"
> +	logfile = None
> +	for arg in sys.argv[1:]:
> +	    if arg.startswith("--mode="):
> +	        mode = arg[7:]
> +	    elif arg.startswith("--log="):
> +	        logfile = open(arg[6:], "a")
> +
> +	def log(msg):
> +	    if logfile:
> +	        logfile.write(msg + "\n")
> +	        logfile.flush()
> +
> +	# Handshake
> +	assert read_pkt() == "git-diff-client"
> +	assert read_pkt() == "version=1"
> +	read_pkt()
> +	write_pkt("git-diff-server")
> +	write_pkt("version=1")
> +	write_flush()
> +	while True:
> +	    p = read_pkt()
> +	    if p == "": break
> +	write_pkt("capability=hunks")
> +	write_flush()
> +
> +	log("ready")
> +
> +	while True:
> +	    cmd = None
> +	    pathname = None
> +	    while True:
> +	        p = read_pkt()
> +	        if p is None: sys.exit(0)
> +	        if p == "": break
> +	        if p.startswith("command="): cmd = p.split("=",1)[1]
> +	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
> +	    if cmd is None: sys.exit(0)
> +	    old = read_content()
> +	    new = read_content()
> +	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
> +	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> +	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> +
> +	    if mode == "error":
> +	        write_flush()
> +	        write_pkt("status=error")
> +	        write_flush()
> +	        continue
> +
> +	    if mode == "abort":
> +	        write_flush()
> +	        write_pkt("status=abort")
> +	        write_flush()
> +	        continue
> +
> +	    if mode == "crash":
> +	        sys.exit(1)
> +
> +	    if cmd == "hunks":
> +	        if mode == "fixed-hunk":
> +	            write_pkt("hunk 5 2 5 2")
> +	        elif mode == "bad-hunk":
> +	            write_pkt("hunk 999 1 999 1")
> +	        elif mode == "bad-sync":
> +	            write_pkt("hunk 1 2 1 1")
> +	        elif mode == "overlap":
> +	            write_pkt("hunk 1 5 1 5")
> +	            write_pkt("hunk 3 2 3 2")
> +	        elif mode == "no-hunks":
> +	            pass
> +	        else:
> +	            ol = old.count(b"\n")
> +	            nl = new.count(b"\n")
> +	            write_pkt(f"hunk 1 {ol} 1 {nl}")
> +	        write_flush()
> +	        write_pkt("status=success")
> +	        write_flush()
> +	    else:
> +	        write_flush()
> +	        write_pkt("status=error")
> +	        write_flush()
> +	PYEOF

The existing pattern is to provide larger scripts as fixtures in
associated `t/tNNNN/` directories, not as heredoc, see e.g.
`t/t1509/prepare-chroot.sh`. Writing scripts, especially lengthy ones, in
heredoc strings makes it virtually impossible to use static code analysis
or syntax highlighting to fend off banal errors.

Given the complexity of what t4080 tries to test (error, abort, crash,
bad-sync, no-hunks, multiple files in one session, capability
negotiation), it would unfortunately be infeasible to use `test-tool
pkt-line` from a shell script implementing that `diff.*.process` protocol.

So I've spiked a demo how the `test-tool diff-process-backend` could look
like (letting Opus do the menial typing, so that I can enjoy at least part
of a sunny Sunday outside), which also passes the CI build and test:
https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6

That commit is of course not intended to be used as-is; Feel free to pick
code parts of it and integrate them into your topic branch. Or write your
own test-tool helper from scratch if that's more your jam.

Ciao,
Johannes

> +	write_script diff-process-backend <<-SHEOF
> +	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
> +	SHEOF
> +}
> +
> +BACKEND="./diff-process-backend"
> +
> +test_expect_success PYTHON 'setup' '
> +	setup_backend &&
> +	echo "*.c diff=cdiff" >.gitattributes &&
> +	git add .gitattributes &&
> +
> +	# boundary.c: 10 lines, changes at 5-6 and 9-10.
> +	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
> +	cat >boundary.c <<-\EOF &&
> +	line1
> +	line2
> +	line3
> +	line4
> +	OLD5
> +	OLD6
> +	line7
> +	line8
> +	OLD9
> +	OLD10
> +	EOF
> +	git add boundary.c &&
> +
> +	# worddiff.c: single-line function, value changes 1 -> 999.
> +	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
> +	cat >worddiff.c <<-\EOF &&
> +	int value(void) { return 1; }
> +	EOF
> +	git add worddiff.c &&
> +
> +	# newfile.c: single-line function, value changes 42 -> 99.
> +	# Used by: new file, --exit-code, multiple drivers.
> +	cat >newfile.c <<-\EOF &&
> +	int new_func(void) { return 42; }
> +	EOF
> +	git add newfile.c &&
> +
> +	# logtest.c: single-line function for log/format-patch tests.
> +	# Needs two commits so log -1 has a diff.
> +	cat >logtest.c <<-\EOF &&
> +	int logfunc(void) { return 1; }
> +	EOF
> +	git add logtest.c &&
> +
> +	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
> +	cat >one.c <<-\EOF &&
> +	int first(void) { return 1; }
> +	EOF
> +	cat >two.c <<-\EOF &&
> +	int second(void) { return 2; }
> +	EOF
> +	git add one.c two.c &&
> +
> +	git commit -m "initial" &&
> +
> +	# Second commit for logtest.c (so log -1 has something to show).
> +	cat >logtest.c <<-\EOF &&
> +	int logfunc(void) { return 2; }
> +	EOF
> +	git add logtest.c &&
> +	git commit -m "change logtest.c" &&
> +
> +	# Working tree modifications (not committed).
> +	cat >boundary.c <<-\EOF &&
> +	line1
> +	line2
> +	line3
> +	line4
> +	NEW5
> +	NEW6
> +	line7
> +	line8
> +	NEW9
> +	NEW10
> +	EOF
> +
> +	cat >worddiff.c <<-\EOF &&
> +	int value(void) { return 999; }
> +	EOF
> +
> +	cat >newfile.c <<-\EOF &&
> +	int new_func(void) { return 99; }
> +	EOF
> +
> +	cat >one.c <<-\EOF &&
> +	int first(void) { return 10; }
> +	EOF
> +
> +	cat >two.c <<-\EOF
> +	int second(void) { return 20; }
> +	EOF
> +'
> +
> +#
> +# Core behavior: the tool controls which lines are marked as changed.
> +#
> +
> +test_expect_success PYTHON 'diff process hunk boundaries affect output' '
> +	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
> +	# only reports lines 5-6 as changed.  Lines 9-10 should not
> +	# appear as changed in the output.
> +	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
> +		diff boundary.c >actual &&
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^-OLD6" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^+NEW6" actual &&
> +	test_grep ! "^-OLD9" actual &&
> +	test_grep ! "^-OLD10" actual &&
> +	test_grep ! "^+NEW9" actual &&
> +	test_grep ! "^+NEW10" actual
> +'
> +
> +test_expect_success PYTHON 'diff process works with new file' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- newfile.c >actual 2>stderr &&
> +	test_grep "return 99" actual &&
> +	test_grep "pathname=newfile.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process works with added file (empty old side)' '
> +	cat >added.c <<-\EOF &&
> +	int added(void) { return 1; }
> +	EOF
> +	git add added.c &&
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --cached -- added.c >actual 2>stderr &&
> +	test_grep "added" actual &&
> +	test_grep "pathname=added.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process skipped for binary files' '
> +	printf "\\0binary" >binary.c &&
> +	git add binary.c &&
> +	git commit -m "add binary" &&
> +	printf "\\0changed" >binary.c &&
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- binary.c >actual &&
> +	test_grep "Binary files" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
> +	echo "not tracked by cdiff" >unmatched.txt &&
> +	git add unmatched.txt &&
> +	git commit -m "add unmatched.txt" &&
> +
> +	echo "modified" >unmatched.txt &&
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- unmatched.txt >actual &&
> +	test_grep "modified" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +test_expect_success PYTHON 'multiple drivers use separate processes' '
> +	echo "*.h diff=hdiff" >>.gitattributes &&
> +	git add .gitattributes &&
> +
> +	cat >multi.h <<-\EOF &&
> +	int header(void) { return 1; }
> +	EOF
> +	git add multi.h &&
> +	git commit -m "add multi.h" &&
> +
> +	cat >multi.h <<-\EOF &&
> +	int header(void) { return 2; }
> +	EOF
> +
> +	rm -f backend-c.log backend-h.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
> +	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
> +		diff -- newfile.c multi.h >actual 2>stderr &&
> +	test_grep "pathname=newfile.c" backend-c.log &&
> +	test_grep "pathname=multi.h" backend-h.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process works alongside textconv' '
> +	write_script uppercase-filter <<-\EOF &&
> +	tr "a-z" "A-Z" <"$1"
> +	EOF
> +
> +	cat >textconv.c <<-\EOF &&
> +	hello world
> +	EOF
> +	git add textconv.c &&
> +	git commit -m "add textconv.c" &&
> +
> +	cat >textconv.c <<-\EOF &&
> +	goodbye world
> +	EOF
> +
> +	rm -f backend.log &&
> +	git -c diff.cdiff.textconv="./uppercase-filter" \
> +	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff -- textconv.c >actual 2>stderr &&
> +	# The diff process receives textconv-transformed (uppercase) content.
> +	test_grep "pathname=textconv.c" backend.log &&
> +	test_grep "old=HELLO WORLD" backend.log &&
> +	test_grep "new=GOODBYE WORLD" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +#
> +# Downstream features: word diff, log, equivalent files, exit code.
> +#
> +
> +test_expect_success PYTHON 'diff process with --word-diff' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --word-diff worddiff.c >actual 2>stderr &&
> +	test_grep "\[-1;-\]" actual &&
> +	test_grep "{+999;+}" actual &&
> +	test_grep "pathname=worddiff.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process works with git log -p' '
> +	# With no-hunks mode, the tool says the files are equivalent,
> +	# so log -p should show the commit but no diff content.
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
> +		log -1 -p -- logtest.c >actual 2>stderr &&
> +	test_grep "change logtest.c" actual &&
> +	test_grep ! "return 2" actual &&
> +	test_grep "command=hunks pathname=logtest.c" backend.log &&
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
> +	cat >nohunks.c <<-\EOF &&
> +	int zero(void) { return 0; }
> +	EOF
> +	git add nohunks.c &&
> +	git commit -m "add nohunks.c" &&
> +
> +	cat >nohunks.c <<-\EOF &&
> +	int zero(void) { return 999; }
> +	EOF
> +
> +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> +		diff nohunks.c >actual &&
> +	test_must_be_empty actual
> +'
> +
> +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
> +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> +		diff --exit-code nohunks.c
> +'
> +
> +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
> +	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
> +		diff --exit-code newfile.c
> +'
> +
> +#
> +# Bypass mechanisms: flags and commands that skip the diff process.
> +#
> +
> +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --diff-algorithm=patience worddiff.c >actual &&
> +	test_grep "return 999" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +test_expect_success PYTHON 'diff process not used by --stat' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> +		diff --stat worddiff.c >actual &&
> +	test_grep "worddiff.c" actual &&
> +	test_path_is_missing backend.log
> +'
> +
> +#
> +# Error handling and fallback.
> +#
> +
> +test_expect_success PYTHON 'diff process fallback on tool error status' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> +		diff boundary.c >actual 2>stderr &&
> +	# Fallback produces the full builtin diff (both change regions).
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^-OLD9" actual &&
> +	test_grep "^+NEW9" actual &&
> +	# Tool was contacted (it replied with error, not crash).
> +	test_grep "command=hunks pathname=boundary.c" backend.log &&
> +	test_grep "diff process.*failed" stderr
> +'
> +
> +test_expect_success PYTHON 'diff process error keeps tool available for next file' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> +		diff -- one.c two.c >actual 2>stderr &&
> +	# Unlike abort, error keeps the tool available: both files
> +	# are sent to the tool (and both fall back).
> +	test_grep "pathname=one.c" backend.log &&
> +	test_grep "pathname=two.c" backend.log &&
> +	test_grep "return 10" actual &&
> +	test_grep "return 20" actual
> +'
> +
> +test_expect_success PYTHON 'diff process abort disables for session' '
> +	rm -f backend.log &&
> +	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
> +		diff -- one.c two.c >actual &&
> +	# Both files should still produce diff output via fallback.
> +	test_grep "return 10" actual &&
> +	test_grep "return 20" actual &&
> +	# The tool aborts on the first file and git clears its
> +	# capability.  The second file never contacts the tool.
> +	test_grep "pathname=one.c" backend.log &&
> +	test_grep ! "pathname=two.c" backend.log
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on tool crash' '
> +	git -c diff.cdiff.process="$BACKEND --mode=crash" \
> +		diff boundary.c >actual 2>stderr &&
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^-OLD9" actual &&
> +	test_grep "^+NEW9" actual &&
> +	# Crash is a communication failure, so a warning is emitted.
> +	test_grep "diff process.*failed" stderr
> +'
> +
> +test_expect_success PYTHON 'diff process startup failure only warns once' '
> +	git -c diff.cdiff.process="/nonexistent/tool" \
> +		diff -- one.c two.c >actual 2>stderr &&
> +	# Both files produce diff output via fallback.
> +	test_grep "return 10" actual &&
> +	test_grep "return 20" actual &&
> +	# Sentinel prevents repeated warnings: only one, not one per file.
> +	test_grep "diff process.*failed" stderr >warnings &&
> +	test_line_count = 1 warnings
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on bad hunks' '
> +	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
> +		diff boundary.c >actual 2>stderr &&
> +	test_grep "^-OLD5" actual &&
> +	test_grep "^+NEW5" actual &&
> +	test_grep "^-OLD9" actual &&
> +	test_grep "^+NEW9" actual &&
> +	# Invalid hunks are caught by xdiff validation, not the
> +	# protocol layer, so no warning is emitted.
> +	test_must_be_empty stderr
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
> +	cat >synctest.c <<-\EOF &&
> +	line1
> +	line2
> +	line3
> +	EOF
> +	git add synctest.c &&
> +	git commit -m "add synctest.c" &&
> +
> +	cat >synctest.c <<-\EOF &&
> +	line1
> +	changed
> +	line3
> +	EOF
> +
> +	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
> +	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
> +	# The synchronization invariant fails and git falls back.
> +	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
> +		diff synctest.c >actual 2>stderr &&
> +	test_grep "changed" actual
> +'
> +
> +test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
> +	# boundary.c has 10 lines, so both hunks are in bounds
> +	# but they overlap at lines 3-5, triggering the ordering check.
> +	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
> +		diff boundary.c >actual 2>stderr &&
> +	test_grep "NEW5" actual
> +'
> +
> +test_done
> diff --git a/userdiff.h b/userdiff.h
> index 51c26e0d41..a98eabe377 100644
> --- a/userdiff.h
> +++ b/userdiff.h
> @@ -3,6 +3,7 @@
>  
>  #include "notes-cache.h"
>  
> +struct diff_subprocess;
>  struct index_state;
>  struct repository;
>  
> @@ -33,6 +34,8 @@ struct userdiff_driver {
>  	int textconv_want_cache;
>  	const char *process;
>  	char *process_owned;
> +	struct diff_subprocess *diff_subprocess;
> +	unsigned diff_process_failed : 1;
>  };
>  enum userdiff_driver_type {
>  	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
> -- 
> gitgitgadget
> 
> 
> 

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Sun, Jun 7, 2026 at 7:36 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Hi Michael,
>
> I stumbled about this patch when it broke CI in Git for Windows, where we
> do _not_ use `NO_PYTHON`, even though Python is unavailable in the
> build/test CI jobs. The existing tests handle this situation gracefully,
> this here patch does not:
>
> On Sun, 7 Jun 2026, Michael Montalbo via GitGitGadget wrote:
>
> > diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> > new file mode 100755
> > index 0000000000..f159cd86d8
> > --- /dev/null
> > +++ b/t/t4080-diff-process.sh
> > @@ -0,0 +1,538 @@
> > +#!/bin/sh
> > +
> > +test_description='diff process via long-running process'
> > +
> > +. ./test-lib.sh
> > +
> > +if test_have_prereq PYTHON
> > +then
> > +     PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
>
> When neither `python3` nor `python` are available (which is the case in
> the minimal Git for Windows SDK used in Git's CI runs), this fails under
> `set -e`. Before even running the first test case. Resulting in an
> unexpected TAP format error.
>
> Now, we could "fix" this by imitating what `lib-p4` does (see
> https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
> which demonstrates that it is indeed a work-around on Windows):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index fdf6da1c341e67..bd22c247ff3856 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -4,9 +4,10 @@ test_description='diff process via long-running process'
>
>  . ./test-lib.sh
>
> -if test_have_prereq PYTHON
> +if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH"
>  then
> -       PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
> +       skip_all='python interpreter not available'
> +       test_done
>  fi
>
>  #
> -- snap --
>
> Of course, this uncovers _another_ problem with the Python script: It uses
> Python3-only `f"..."` format strings, which cannot be handled by the
> Python2 to which the `PYTHON_PATH` variable in `linux-TEST-vars` points.
> So this requires _another follow-up (see also
> https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index bd22c247ff3856..ba14682a9086e4 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -39,7 +39,8 @@ setup_backend () {
>
>         def write_pkt(line):
>             data = (line + "\n").encode()
> -           sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> +           hdr = "{:04x}".format(len(data) + 4).encode()
> +           sys.stdout.buffer.write(hdr + data)
>             sys.stdout.buffer.flush()
>
>         def write_flush():
> @@ -98,7 +99,8 @@ setup_backend () {
>             new = read_content()
>             old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
>             new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> -           log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> +           log("command={} pathname={} old={} new={}".format(
> +               cmd, pathname, old_first, new_first))
>
>             if mode == "error":
>                 write_flush()
> @@ -130,7 +132,7 @@ setup_backend () {
>                 else:
>                     ol = old.count(b"\n")
>                     nl = new.count(b"\n")
> -                   write_pkt(f"hunk 1 {ol} 1 {nl}")
> +                   write_pkt("hunk 1 {} 1 {}".format(ol, nl))
>                 write_flush()
>                 write_pkt("status=success")
>                 write_flush()
> -- snap --
>
> And this is still not enough to make it work with Python2, see
> https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:
>
> -- snip --
> [...]
> + git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diff boundary.c
>   Traceback (most recent call last):
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
>       assert read_pkt() == "git-diff-client"
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 4, in read_pkt
>       hdr = sys.stdin.buffer.read(4)
>   AttributeError: 'file' object has no attribute 'buffer'
> -- snap --
>
> I have experienced similar patterns in my career, where a single decision
> required multiple follow-up fixes _just_ to avoid having to revert that
> decision. This kind of doubling down has never ended well.
>
> Therefore I would like to take a step back, and ask: Is it _really_ a good
> idea to use Python here? Are we certain that we want to _require_ Python
> to run this test and skip it if Python isn't available (as is the case in
> the Windows-related parts of Git's very own CI) even if Python has nothing
> at all to do with the feature that is being tested?
>
> I don't want to be doomed to repeat history, and we can very well learn
> e.g. from prior art in this very project, where the tests for the
> clean/smudge filters (which _also_ want to speak pkt-line over stdio)
> needlessly incurred Perl as a requirement to run the tests. It was
> Matheus's heroic work in 52917a998ef3a (t0021: implementation the
> rot13-filter.pl script in C, 2022-08-14) and 4d1d843be7a15 (tests: use the
> new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided
> that unnecessary prerequisite.
>
> Likewise, there is `test-tool pkt-line` intended for driving the pkt-line
> protocol via simple shell scripts.
>
> So the conscious project direction has been: fold pkt-line test backends
> into `test-tool` and drop the scripting-language prereq. Reintroducing the
> same shape in Python would walk this back.
>
> Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
> 2025-04-29) has been another clear sign that the Git project is actively
> _removing_ scripting-language dependencies from the build and test
> infrastructure, not adding new ones.
>
> The clear prior art in Git's own tests for what t4080 wants to do, as of
> today, is `t/helper/test-rot13-filter.c`, which could be imitated here
> instead of (re-)introducing a dependency on a scripting language other
> than Unix shell in Git's test suite.
>
> The `PYTHON` prereq exists in exactly five files today, all `git p4`
> related (where Python is an intrinsic prerequisite given that `git-p4.py`
> _is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
> `t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
> and `t/t9836-git-p4-metadata-encoding-python3.sh`.
>
> After 7cdbff14d482 (remove merge-recursive-old, 2006-11-20), this here
> patch would be the first one, after almost 20 years, to re-introduce
> Python as a dependency outside `git p4`.
>
> And it would also be the first ever to embed a Python script as a heredoc:
>
> > +fi
> > +
> > +#
> > +# A single parametric diff process.
> > +# Usage: diff-process-backend --mode=<mode> [--log=<path>]
> > +#
> > +# Modes:
> > +#   whole-file  - report all lines as changed (default)
> > +#   fixed-hunk  - always report hunk 5 2 5 2
> > +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
> > +#   bad-sync    - report hunk with mismatched unchanged totals
> > +#   overlap     - report two overlapping hunks
> > +#   no-hunks   - return no hunks (files considered equivalent)
> > +#   error       - return status=error for every request
> > +#   abort       - return status=abort for every request
> > +#   crash       - read one request then exit without responding
> > +#
> > +setup_backend () {
> > +     cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
> > +     import sys, os
> > +
> > +     def read_pkt():
> > +         hdr = sys.stdin.buffer.read(4)
> > +         if len(hdr) < 4: return None
> > +         length = int(hdr, 16)
> > +         if length == 0: return ""
> > +         data = sys.stdin.buffer.read(length - 4)
> > +         return data.decode().rstrip("\n")
> > +
> > +     def write_pkt(line):
> > +         data = (line + "\n").encode()
> > +         sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> > +         sys.stdout.buffer.flush()
> > +
> > +     def write_flush():
> > +         sys.stdout.buffer.write(b"0000")
> > +         sys.stdout.buffer.flush()
> > +
> > +     def read_content():
> > +         chunks = []
> > +         while True:
> > +             hdr = sys.stdin.buffer.read(4)
> > +             if len(hdr) < 4: break
> > +             length = int(hdr, 16)
> > +             if length == 0: break
> > +             chunks.append(sys.stdin.buffer.read(length - 4))
> > +         return b"".join(chunks)
> > +
> > +     mode = "whole-file"
> > +     logfile = None
> > +     for arg in sys.argv[1:]:
> > +         if arg.startswith("--mode="):
> > +             mode = arg[7:]
> > +         elif arg.startswith("--log="):
> > +             logfile = open(arg[6:], "a")
> > +
> > +     def log(msg):
> > +         if logfile:
> > +             logfile.write(msg + "\n")
> > +             logfile.flush()
> > +
> > +     # Handshake
> > +     assert read_pkt() == "git-diff-client"
> > +     assert read_pkt() == "version=1"
> > +     read_pkt()
> > +     write_pkt("git-diff-server")
> > +     write_pkt("version=1")
> > +     write_flush()
> > +     while True:
> > +         p = read_pkt()
> > +         if p == "": break
> > +     write_pkt("capability=hunks")
> > +     write_flush()
> > +
> > +     log("ready")
> > +
> > +     while True:
> > +         cmd = None
> > +         pathname = None
> > +         while True:
> > +             p = read_pkt()
> > +             if p is None: sys.exit(0)
> > +             if p == "": break
> > +             if p.startswith("command="): cmd = p.split("=",1)[1]
> > +             if p.startswith("pathname="): pathname = p.split("=",1)[1]
> > +         if cmd is None: sys.exit(0)
> > +         old = read_content()
> > +         new = read_content()
> > +         old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
> > +         new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> > +         log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> > +
> > +         if mode == "error":
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "abort":
> > +             write_flush()
> > +             write_pkt("status=abort")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "crash":
> > +             sys.exit(1)
> > +
> > +         if cmd == "hunks":
> > +             if mode == "fixed-hunk":
> > +                 write_pkt("hunk 5 2 5 2")
> > +             elif mode == "bad-hunk":
> > +                 write_pkt("hunk 999 1 999 1")
> > +             elif mode == "bad-sync":
> > +                 write_pkt("hunk 1 2 1 1")
> > +             elif mode == "overlap":
> > +                 write_pkt("hunk 1 5 1 5")
> > +                 write_pkt("hunk 3 2 3 2")
> > +             elif mode == "no-hunks":
> > +                 pass
> > +             else:
> > +                 ol = old.count(b"\n")
> > +                 nl = new.count(b"\n")
> > +                 write_pkt(f"hunk 1 {ol} 1 {nl}")
> > +             write_flush()
> > +             write_pkt("status=success")
> > +             write_flush()
> > +         else:
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +     PYEOF
>
> The existing pattern is to provide larger scripts as fixtures in
> associated `t/tNNNN/` directories, not as heredoc, see e.g.
> `t/t1509/prepare-chroot.sh`. Writing scripts, especially lengthy ones, in
> heredoc strings makes it virtually impossible to use static code analysis
> or syntax highlighting to fend off banal errors.
>
> Given the complexity of what t4080 tries to test (error, abort, crash,
> bad-sync, no-hunks, multiple files in one session, capability
> negotiation), it would unfortunately be infeasible to use `test-tool
> pkt-line` from a shell script implementing that `diff.*.process` protocol.
>
> So I've spiked a demo how the `test-tool diff-process-backend` could look
> like (letting Opus do the menial typing, so that I can enjoy at least part
> of a sunny Sunday outside), which also passes the CI build and test:
> https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6
>
> That commit is of course not intended to be used as-is; Feel free to pick
> code parts of it and integrate them into your topic branch. Or write your
> own test-tool helper from scratch if that's more your jam.
>

Johannes, thank you for the great feedback. The historical context is
really helpful and
the concerns you raise make a lot of sense. I will take a look at your
spike and also work
on removing Python from the test.

> Ciao,
> Johannes
>
> > +     write_script diff-process-backend <<-SHEOF
> > +     exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
> > +     SHEOF
> > +}
> > +
> > +BACKEND="./diff-process-backend"
> > +
> > +test_expect_success PYTHON 'setup' '
> > +     setup_backend &&
> > +     echo "*.c diff=cdiff" >.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     # boundary.c: 10 lines, changes at 5-6 and 9-10.
> > +     # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     OLD5
> > +     OLD6
> > +     line7
> > +     line8
> > +     OLD9
> > +     OLD10
> > +     EOF
> > +     git add boundary.c &&
> > +
> > +     # worddiff.c: single-line function, value changes 1 -> 999.
> > +     # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 1; }
> > +     EOF
> > +     git add worddiff.c &&
> > +
> > +     # newfile.c: single-line function, value changes 42 -> 99.
> > +     # Used by: new file, --exit-code, multiple drivers.
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 42; }
> > +     EOF
> > +     git add newfile.c &&
> > +
> > +     # logtest.c: single-line function for log/format-patch tests.
> > +     # Needs two commits so log -1 has a diff.
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 1; }
> > +     EOF
> > +     git add logtest.c &&
> > +
> > +     # two.c/one.c: two-file pair for error/abort/startup-failure tests.
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 1; }
> > +     EOF
> > +     cat >two.c <<-\EOF &&
> > +     int second(void) { return 2; }
> > +     EOF
> > +     git add one.c two.c &&
> > +
> > +     git commit -m "initial" &&
> > +
> > +     # Second commit for logtest.c (so log -1 has something to show).
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 2; }
> > +     EOF
> > +     git add logtest.c &&
> > +     git commit -m "change logtest.c" &&
> > +
> > +     # Working tree modifications (not committed).
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     NEW5
> > +     NEW6
> > +     line7
> > +     line8
> > +     NEW9
> > +     NEW10
> > +     EOF
> > +
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 999; }
> > +     EOF
> > +
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 99; }
> > +     EOF
> > +
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 10; }
> > +     EOF
> > +
> > +     cat >two.c <<-\EOF
> > +     int second(void) { return 20; }
> > +     EOF
> > +'
> > +
> > +#
> > +# Core behavior: the tool controls which lines are marked as changed.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process hunk boundaries affect output' '
> > +     # The file has changes at lines 5-6 and 9-10, but fixed-hunk
> > +     # only reports lines 5-6 as changed.  Lines 9-10 should not
> > +     # appear as changed in the output.
> > +     git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
> > +             diff boundary.c >actual &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^-OLD6" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^+NEW6" actual &&
> > +     test_grep ! "^-OLD9" actual &&
> > +     test_grep ! "^-OLD10" actual &&
> > +     test_grep ! "^+NEW9" actual &&
> > +     test_grep ! "^+NEW10" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with new file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- newfile.c >actual 2>stderr &&
> > +     test_grep "return 99" actual &&
> > +     test_grep "pathname=newfile.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with added file (empty old side)' '
> > +     cat >added.c <<-\EOF &&
> > +     int added(void) { return 1; }
> > +     EOF
> > +     git add added.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --cached -- added.c >actual 2>stderr &&
> > +     test_grep "added" actual &&
> > +     test_grep "pathname=added.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process skipped for binary files' '
> > +     printf "\\0binary" >binary.c &&
> > +     git add binary.c &&
> > +     git commit -m "add binary" &&
> > +     printf "\\0changed" >binary.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- binary.c >actual &&
> > +     test_grep "Binary files" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
> > +     echo "not tracked by cdiff" >unmatched.txt &&
> > +     git add unmatched.txt &&
> > +     git commit -m "add unmatched.txt" &&
> > +
> > +     echo "modified" >unmatched.txt &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- unmatched.txt >actual &&
> > +     test_grep "modified" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'multiple drivers use separate processes' '
> > +     echo "*.h diff=hdiff" >>.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 1; }
> > +     EOF
> > +     git add multi.h &&
> > +     git commit -m "add multi.h" &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 2; }
> > +     EOF
> > +
> > +     rm -f backend-c.log backend-h.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
> > +         -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
> > +             diff -- newfile.c multi.h >actual 2>stderr &&
> > +     test_grep "pathname=newfile.c" backend-c.log &&
> > +     test_grep "pathname=multi.h" backend-h.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works alongside textconv' '
> > +     write_script uppercase-filter <<-\EOF &&
> > +     tr "a-z" "A-Z" <"$1"
> > +     EOF
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     hello world
> > +     EOF
> > +     git add textconv.c &&
> > +     git commit -m "add textconv.c" &&
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     goodbye world
> > +     EOF
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.textconv="./uppercase-filter" \
> > +         -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- textconv.c >actual 2>stderr &&
> > +     # The diff process receives textconv-transformed (uppercase) content.
> > +     test_grep "pathname=textconv.c" backend.log &&
> > +     test_grep "old=HELLO WORLD" backend.log &&
> > +     test_grep "new=GOODBYE WORLD" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +#
> > +# Downstream features: word diff, log, equivalent files, exit code.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process with --word-diff' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --word-diff worddiff.c >actual 2>stderr &&
> > +     test_grep "\[-1;-\]" actual &&
> > +     test_grep "{+999;+}" actual &&
> > +     test_grep "pathname=worddiff.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with git log -p' '
> > +     # With no-hunks mode, the tool says the files are equivalent,
> > +     # so log -p should show the commit but no diff content.
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
> > +             log -1 -p -- logtest.c >actual 2>stderr &&
> > +     test_grep "change logtest.c" actual &&
> > +     test_grep ! "return 2" actual &&
> > +     test_grep "command=hunks pathname=logtest.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 0; }
> > +     EOF
> > +     git add nohunks.c &&
> > +     git commit -m "add nohunks.c" &&
> > +
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 999; }
> > +     EOF
> > +
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff nohunks.c >actual &&
> > +     test_must_be_empty actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff --exit-code nohunks.c
> > +'
> > +
> > +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
> > +     test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
> > +             diff --exit-code newfile.c
> > +'
> > +
> > +#
> > +# Bypass mechanisms: flags and commands that skip the diff process.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --diff-algorithm=patience worddiff.c >actual &&
> > +     test_grep "return 999" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not used by --stat' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --stat worddiff.c >actual &&
> > +     test_grep "worddiff.c" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +#
> > +# Error handling and fallback.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool error status' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     # Fallback produces the full builtin diff (both change regions).
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Tool was contacted (it replied with error, not crash).
> > +     test_grep "command=hunks pathname=boundary.c" backend.log &&
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process error keeps tool available for next file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Unlike abort, error keeps the tool available: both files
> > +     # are sent to the tool (and both fall back).
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep "pathname=two.c" backend.log &&
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process abort disables for session' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
> > +             diff -- one.c two.c >actual &&
> > +     # Both files should still produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # The tool aborts on the first file and git clears its
> > +     # capability.  The second file never contacts the tool.
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep ! "pathname=two.c" backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool crash' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=crash" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Crash is a communication failure, so a warning is emitted.
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process startup failure only warns once' '
> > +     git -c diff.cdiff.process="/nonexistent/tool" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Both files produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # Sentinel prevents repeated warnings: only one, not one per file.
> > +     test_grep "diff process.*failed" stderr >warnings &&
> > +     test_line_count = 1 warnings
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on bad hunks' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Invalid hunks are caught by xdiff validation, not the
> > +     # protocol layer, so no warning is emitted.
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     EOF
> > +     git add synctest.c &&
> > +     git commit -m "add synctest.c" &&
> > +
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     changed
> > +     line3
> > +     EOF
> > +
> > +     # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
> > +     # line as changed, leaving 1 unchanged old vs 2 unchanged new.
> > +     # The synchronization invariant fails and git falls back.
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
> > +             diff synctest.c >actual 2>stderr &&
> > +     test_grep "changed" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
> > +     # boundary.c has 10 lines, so both hunks are in bounds
> > +     # but they overlap at lines 3-5, triggering the ordering check.
> > +     git -c diff.cdiff.process="$BACKEND --mode=overlap" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "NEW5" actual
> > +'
> > +
> > +test_done
> > diff --git a/userdiff.h b/userdiff.h
> > index 51c26e0d41..a98eabe377 100644
> > --- a/userdiff.h
> > +++ b/userdiff.h
> > @@ -3,6 +3,7 @@
> >
> >  #include "notes-cache.h"
> >
> > +struct diff_subprocess;
> >  struct index_state;
> >  struct repository;
> >
> > @@ -33,6 +34,8 @@ struct userdiff_driver {
> >       int textconv_want_cache;
> >       const char *process;
> >       char *process_owned;
> > +     struct diff_subprocess *diff_subprocess;
> > +     unsigned diff_process_failed : 1;
> >  };
> >  enum userdiff_driver_type {
> >       USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
> > --
> > gitgitgadget
> >
> >
> >

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo]

On Sun, Jun 7, 2026 at 7:36 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> Hi Michael,
>
> I stumbled about this patch when it broke CI in Git for Windows, where we
> do _not_ use `NO_PYTHON`, even though Python is unavailable in the
> build/test CI jobs. The existing tests handle this situation gracefully,
> this here patch does not:
>
> On Sun, 7 Jun 2026, Michael Montalbo via GitGitGadget wrote:
>
> > diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> > new file mode 100755
> > index 0000000000..f159cd86d8
> > --- /dev/null
> > +++ b/t/t4080-diff-process.sh
> > @@ -0,0 +1,538 @@
> > +#!/bin/sh
> > +
> > +test_description='diff process via long-running process'
> > +
> > +. ./test-lib.sh
> > +
> > +if test_have_prereq PYTHON
> > +then
> > +     PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
>
> When neither `python3` nor `python` are available (which is the case in
> the minimal Git for Windows SDK used in Git's CI runs), this fails under
> `set -e`. Before even running the first test case. Resulting in an
> unexpected TAP format error.
>
> Now, we could "fix" this by imitating what `lib-p4` does (see
> https://github.com/dscho/git/commit/bd0b5570c744f678911a67a62da63f30655f20d8
> which demonstrates that it is indeed a work-around on Windows):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index fdf6da1c341e67..bd22c247ff3856 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -4,9 +4,10 @@ test_description='diff process via long-running process'
>
>  . ./test-lib.sh
>
> -if test_have_prereq PYTHON
> +if ! test_have_prereq PYTHON || ! test -x "$PYTHON_PATH"
>  then
> -       PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
> +       skip_all='python interpreter not available'
> +       test_done
>  fi
>
>  #
> -- snap --
>
> Of course, this uncovers _another_ problem with the Python script: It uses
> Python3-only `f"..."` format strings, which cannot be handled by the
> Python2 to which the `PYTHON_PATH` variable in `linux-TEST-vars` points.
> So this requires _another follow-up (see also
> https://github.com/dscho/git/commit/c12a9f4c80e5ce8db0fe370fac46fb45be2b775f):
>
> -- snip --
> diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
> index bd22c247ff3856..ba14682a9086e4 100755
> --- a/t/t4080-diff-process.sh
> +++ b/t/t4080-diff-process.sh
> @@ -39,7 +39,8 @@ setup_backend () {
>
>         def write_pkt(line):
>             data = (line + "\n").encode()
> -           sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> +           hdr = "{:04x}".format(len(data) + 4).encode()
> +           sys.stdout.buffer.write(hdr + data)
>             sys.stdout.buffer.flush()
>
>         def write_flush():
> @@ -98,7 +99,8 @@ setup_backend () {
>             new = read_content()
>             old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
>             new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> -           log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> +           log("command={} pathname={} old={} new={}".format(
> +               cmd, pathname, old_first, new_first))
>
>             if mode == "error":
>                 write_flush()
> @@ -130,7 +132,7 @@ setup_backend () {
>                 else:
>                     ol = old.count(b"\n")
>                     nl = new.count(b"\n")
> -                   write_pkt(f"hunk 1 {ol} 1 {nl}")
> +                   write_pkt("hunk 1 {} 1 {}".format(ol, nl))
>                 write_flush()
>                 write_pkt("status=success")
>                 write_flush()
> -- snap --
>
> And this is still not enough to make it work with Python2, see
> https://github.com/dscho/git/actions/runs/27091523842/job/79955895737:
>
> -- snip --
> [...]
> + git -c diff.cdiff.process=./diff-process-backend --mode=fixed-hunk diff boundary.c
>   Traceback (most recent call last):
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 45, in <module>
>       assert read_pkt() == "git-diff-client"
>     File "/__w/git/git/t/trash directory.t4080-diff-process/diff-process-backend.py", line 4, in read_pkt
>       hdr = sys.stdin.buffer.read(4)
>   AttributeError: 'file' object has no attribute 'buffer'
> -- snap --
>
> I have experienced similar patterns in my career, where a single decision
> required multiple follow-up fixes _just_ to avoid having to revert that
> decision. This kind of doubling down has never ended well.
>
> Therefore I would like to take a step back, and ask: Is it _really_ a good
> idea to use Python here? Are we certain that we want to _require_ Python
> to run this test and skip it if Python isn't available (as is the case in
> the Windows-related parts of Git's very own CI) even if Python has nothing
> at all to do with the feature that is being tested?
>
> I don't want to be doomed to repeat history, and we can very well learn
> e.g. from prior art in this very project, where the tests for the
> clean/smudge filters (which _also_ want to speak pkt-line over stdio)
> needlessly incurred Perl as a requirement to run the tests. It was
> Matheus's heroic work in 52917a998ef3a (t0021: implementation the
> rot13-filter.pl script in C, 2022-08-14) and 4d1d843be7a15 (tests: use the
> new C rot13-filter helper to avoid PERL prereq, 2022-08-14) that avoided
> that unnecessary prerequisite.
>
> Likewise, there is `test-tool pkt-line` intended for driving the pkt-line
> protocol via simple shell scripts.
>
> So the conscious project direction has been: fold pkt-line test backends
> into `test-tool` and drop the scripting-language prereq. Reintroducing the
> same shape in Python would walk this back.
>
> Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
> 2025-04-29) has been another clear sign that the Git project is actively
> _removing_ scripting-language dependencies from the build and test
> infrastructure, not adding new ones.
>

Now I wonder if the extension / addition of more Perl test infra with my other
series:

https://lore.kernel.org/git/pull.2135.git.1780559158.gitgitgadget@gmail.com/T/

also goes against the project direction w.r.t. removing scripting languages.
Maybe I should also re-evaluate the usage of Perl there. I am leveraging
existing shell parsing logic written in Perl, but maybe the preference for
Perl-based lint rules is a mistake and should be avoided.

> The clear prior art in Git's own tests for what t4080 wants to do, as of
> today, is `t/helper/test-rot13-filter.c`, which could be imitated here
> instead of (re-)introducing a dependency on a scripting language other
> than Unix shell in Git's test suite.
>
> The `PYTHON` prereq exists in exactly five files today, all `git p4`
> related (where Python is an intrinsic prerequisite given that `git-p4.py`
> _is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
> `t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
> and `t/t9836-git-p4-metadata-encoding-python3.sh`.
>
> After 7cdbff14d482 (remove merge-recursive-old, 2006-11-20), this here
> patch would be the first one, after almost 20 years, to re-introduce
> Python as a dependency outside `git p4`.
>
> And it would also be the first ever to embed a Python script as a heredoc:
>
> > +fi
> > +
> > +#
> > +# A single parametric diff process.
> > +# Usage: diff-process-backend --mode=<mode> [--log=<path>]
> > +#
> > +# Modes:
> > +#   whole-file  - report all lines as changed (default)
> > +#   fixed-hunk  - always report hunk 5 2 5 2
> > +#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
> > +#   bad-sync    - report hunk with mismatched unchanged totals
> > +#   overlap     - report two overlapping hunks
> > +#   no-hunks   - return no hunks (files considered equivalent)
> > +#   error       - return status=error for every request
> > +#   abort       - return status=abort for every request
> > +#   crash       - read one request then exit without responding
> > +#
> > +setup_backend () {
> > +     cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
> > +     import sys, os
> > +
> > +     def read_pkt():
> > +         hdr = sys.stdin.buffer.read(4)
> > +         if len(hdr) < 4: return None
> > +         length = int(hdr, 16)
> > +         if length == 0: return ""
> > +         data = sys.stdin.buffer.read(length - 4)
> > +         return data.decode().rstrip("\n")
> > +
> > +     def write_pkt(line):
> > +         data = (line + "\n").encode()
> > +         sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
> > +         sys.stdout.buffer.flush()
> > +
> > +     def write_flush():
> > +         sys.stdout.buffer.write(b"0000")
> > +         sys.stdout.buffer.flush()
> > +
> > +     def read_content():
> > +         chunks = []
> > +         while True:
> > +             hdr = sys.stdin.buffer.read(4)
> > +             if len(hdr) < 4: break
> > +             length = int(hdr, 16)
> > +             if length == 0: break
> > +             chunks.append(sys.stdin.buffer.read(length - 4))
> > +         return b"".join(chunks)
> > +
> > +     mode = "whole-file"
> > +     logfile = None
> > +     for arg in sys.argv[1:]:
> > +         if arg.startswith("--mode="):
> > +             mode = arg[7:]
> > +         elif arg.startswith("--log="):
> > +             logfile = open(arg[6:], "a")
> > +
> > +     def log(msg):
> > +         if logfile:
> > +             logfile.write(msg + "\n")
> > +             logfile.flush()
> > +
> > +     # Handshake
> > +     assert read_pkt() == "git-diff-client"
> > +     assert read_pkt() == "version=1"
> > +     read_pkt()
> > +     write_pkt("git-diff-server")
> > +     write_pkt("version=1")
> > +     write_flush()
> > +     while True:
> > +         p = read_pkt()
> > +         if p == "": break
> > +     write_pkt("capability=hunks")
> > +     write_flush()
> > +
> > +     log("ready")
> > +
> > +     while True:
> > +         cmd = None
> > +         pathname = None
> > +         while True:
> > +             p = read_pkt()
> > +             if p is None: sys.exit(0)
> > +             if p == "": break
> > +             if p.startswith("command="): cmd = p.split("=",1)[1]
> > +             if p.startswith("pathname="): pathname = p.split("=",1)[1]
> > +         if cmd is None: sys.exit(0)
> > +         old = read_content()
> > +         new = read_content()
> > +         old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
> > +         new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
> > +         log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
> > +
> > +         if mode == "error":
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "abort":
> > +             write_flush()
> > +             write_pkt("status=abort")
> > +             write_flush()
> > +             continue
> > +
> > +         if mode == "crash":
> > +             sys.exit(1)
> > +
> > +         if cmd == "hunks":
> > +             if mode == "fixed-hunk":
> > +                 write_pkt("hunk 5 2 5 2")
> > +             elif mode == "bad-hunk":
> > +                 write_pkt("hunk 999 1 999 1")
> > +             elif mode == "bad-sync":
> > +                 write_pkt("hunk 1 2 1 1")
> > +             elif mode == "overlap":
> > +                 write_pkt("hunk 1 5 1 5")
> > +                 write_pkt("hunk 3 2 3 2")
> > +             elif mode == "no-hunks":
> > +                 pass
> > +             else:
> > +                 ol = old.count(b"\n")
> > +                 nl = new.count(b"\n")
> > +                 write_pkt(f"hunk 1 {ol} 1 {nl}")
> > +             write_flush()
> > +             write_pkt("status=success")
> > +             write_flush()
> > +         else:
> > +             write_flush()
> > +             write_pkt("status=error")
> > +             write_flush()
> > +     PYEOF
>
> The existing pattern is to provide larger scripts as fixtures in
> associated `t/tNNNN/` directories, not as heredoc, see e.g.
> `t/t1509/prepare-chroot.sh`. Writing scripts, especially lengthy ones, in
> heredoc strings makes it virtually impossible to use static code analysis
> or syntax highlighting to fend off banal errors.
>
> Given the complexity of what t4080 tries to test (error, abort, crash,
> bad-sync, no-hunks, multiple files in one session, capability
> negotiation), it would unfortunately be infeasible to use `test-tool
> pkt-line` from a shell script implementing that `diff.*.process` protocol.
>
> So I've spiked a demo how the `test-tool diff-process-backend` could look
> like (letting Opus do the menial typing, so that I can enjoy at least part
> of a sunny Sunday outside), which also passes the CI build and test:
> https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6
>
> That commit is of course not intended to be used as-is; Feel free to pick
> code parts of it and integrate them into your topic branch. Or write your
> own test-tool helper from scratch if that's more your jam.
>
> Ciao,
> Johannes
>
> > +     write_script diff-process-backend <<-SHEOF
> > +     exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
> > +     SHEOF
> > +}
> > +
> > +BACKEND="./diff-process-backend"
> > +
> > +test_expect_success PYTHON 'setup' '
> > +     setup_backend &&
> > +     echo "*.c diff=cdiff" >.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     # boundary.c: 10 lines, changes at 5-6 and 9-10.
> > +     # Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     OLD5
> > +     OLD6
> > +     line7
> > +     line8
> > +     OLD9
> > +     OLD10
> > +     EOF
> > +     git add boundary.c &&
> > +
> > +     # worddiff.c: single-line function, value changes 1 -> 999.
> > +     # Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 1; }
> > +     EOF
> > +     git add worddiff.c &&
> > +
> > +     # newfile.c: single-line function, value changes 42 -> 99.
> > +     # Used by: new file, --exit-code, multiple drivers.
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 42; }
> > +     EOF
> > +     git add newfile.c &&
> > +
> > +     # logtest.c: single-line function for log/format-patch tests.
> > +     # Needs two commits so log -1 has a diff.
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 1; }
> > +     EOF
> > +     git add logtest.c &&
> > +
> > +     # two.c/one.c: two-file pair for error/abort/startup-failure tests.
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 1; }
> > +     EOF
> > +     cat >two.c <<-\EOF &&
> > +     int second(void) { return 2; }
> > +     EOF
> > +     git add one.c two.c &&
> > +
> > +     git commit -m "initial" &&
> > +
> > +     # Second commit for logtest.c (so log -1 has something to show).
> > +     cat >logtest.c <<-\EOF &&
> > +     int logfunc(void) { return 2; }
> > +     EOF
> > +     git add logtest.c &&
> > +     git commit -m "change logtest.c" &&
> > +
> > +     # Working tree modifications (not committed).
> > +     cat >boundary.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     line4
> > +     NEW5
> > +     NEW6
> > +     line7
> > +     line8
> > +     NEW9
> > +     NEW10
> > +     EOF
> > +
> > +     cat >worddiff.c <<-\EOF &&
> > +     int value(void) { return 999; }
> > +     EOF
> > +
> > +     cat >newfile.c <<-\EOF &&
> > +     int new_func(void) { return 99; }
> > +     EOF
> > +
> > +     cat >one.c <<-\EOF &&
> > +     int first(void) { return 10; }
> > +     EOF
> > +
> > +     cat >two.c <<-\EOF
> > +     int second(void) { return 20; }
> > +     EOF
> > +'
> > +
> > +#
> > +# Core behavior: the tool controls which lines are marked as changed.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process hunk boundaries affect output' '
> > +     # The file has changes at lines 5-6 and 9-10, but fixed-hunk
> > +     # only reports lines 5-6 as changed.  Lines 9-10 should not
> > +     # appear as changed in the output.
> > +     git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
> > +             diff boundary.c >actual &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^-OLD6" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^+NEW6" actual &&
> > +     test_grep ! "^-OLD9" actual &&
> > +     test_grep ! "^-OLD10" actual &&
> > +     test_grep ! "^+NEW9" actual &&
> > +     test_grep ! "^+NEW10" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with new file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- newfile.c >actual 2>stderr &&
> > +     test_grep "return 99" actual &&
> > +     test_grep "pathname=newfile.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with added file (empty old side)' '
> > +     cat >added.c <<-\EOF &&
> > +     int added(void) { return 1; }
> > +     EOF
> > +     git add added.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --cached -- added.c >actual 2>stderr &&
> > +     test_grep "added" actual &&
> > +     test_grep "pathname=added.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process skipped for binary files' '
> > +     printf "\\0binary" >binary.c &&
> > +     git add binary.c &&
> > +     git commit -m "add binary" &&
> > +     printf "\\0changed" >binary.c &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- binary.c >actual &&
> > +     test_grep "Binary files" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
> > +     echo "not tracked by cdiff" >unmatched.txt &&
> > +     git add unmatched.txt &&
> > +     git commit -m "add unmatched.txt" &&
> > +
> > +     echo "modified" >unmatched.txt &&
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- unmatched.txt >actual &&
> > +     test_grep "modified" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'multiple drivers use separate processes' '
> > +     echo "*.h diff=hdiff" >>.gitattributes &&
> > +     git add .gitattributes &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 1; }
> > +     EOF
> > +     git add multi.h &&
> > +     git commit -m "add multi.h" &&
> > +
> > +     cat >multi.h <<-\EOF &&
> > +     int header(void) { return 2; }
> > +     EOF
> > +
> > +     rm -f backend-c.log backend-h.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
> > +         -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
> > +             diff -- newfile.c multi.h >actual 2>stderr &&
> > +     test_grep "pathname=newfile.c" backend-c.log &&
> > +     test_grep "pathname=multi.h" backend-h.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works alongside textconv' '
> > +     write_script uppercase-filter <<-\EOF &&
> > +     tr "a-z" "A-Z" <"$1"
> > +     EOF
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     hello world
> > +     EOF
> > +     git add textconv.c &&
> > +     git commit -m "add textconv.c" &&
> > +
> > +     cat >textconv.c <<-\EOF &&
> > +     goodbye world
> > +     EOF
> > +
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.textconv="./uppercase-filter" \
> > +         -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff -- textconv.c >actual 2>stderr &&
> > +     # The diff process receives textconv-transformed (uppercase) content.
> > +     test_grep "pathname=textconv.c" backend.log &&
> > +     test_grep "old=HELLO WORLD" backend.log &&
> > +     test_grep "new=GOODBYE WORLD" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +#
> > +# Downstream features: word diff, log, equivalent files, exit code.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process with --word-diff' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --word-diff worddiff.c >actual 2>stderr &&
> > +     test_grep "\[-1;-\]" actual &&
> > +     test_grep "{+999;+}" actual &&
> > +     test_grep "pathname=worddiff.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process works with git log -p' '
> > +     # With no-hunks mode, the tool says the files are equivalent,
> > +     # so log -p should show the commit but no diff content.
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
> > +             log -1 -p -- logtest.c >actual 2>stderr &&
> > +     test_grep "change logtest.c" actual &&
> > +     test_grep ! "return 2" actual &&
> > +     test_grep "command=hunks pathname=logtest.c" backend.log &&
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 0; }
> > +     EOF
> > +     git add nohunks.c &&
> > +     git commit -m "add nohunks.c" &&
> > +
> > +     cat >nohunks.c <<-\EOF &&
> > +     int zero(void) { return 999; }
> > +     EOF
> > +
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff nohunks.c >actual &&
> > +     test_must_be_empty actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
> > +             diff --exit-code nohunks.c
> > +'
> > +
> > +test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
> > +     test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
> > +             diff --exit-code newfile.c
> > +'
> > +
> > +#
> > +# Bypass mechanisms: flags and commands that skip the diff process.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --diff-algorithm=patience worddiff.c >actual &&
> > +     test_grep "return 999" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process not used by --stat' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --log=backend.log" \
> > +             diff --stat worddiff.c >actual &&
> > +     test_grep "worddiff.c" actual &&
> > +     test_path_is_missing backend.log
> > +'
> > +
> > +#
> > +# Error handling and fallback.
> > +#
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool error status' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     # Fallback produces the full builtin diff (both change regions).
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Tool was contacted (it replied with error, not crash).
> > +     test_grep "command=hunks pathname=boundary.c" backend.log &&
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process error keeps tool available for next file' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Unlike abort, error keeps the tool available: both files
> > +     # are sent to the tool (and both fall back).
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep "pathname=two.c" backend.log &&
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process abort disables for session' '
> > +     rm -f backend.log &&
> > +     git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
> > +             diff -- one.c two.c >actual &&
> > +     # Both files should still produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # The tool aborts on the first file and git clears its
> > +     # capability.  The second file never contacts the tool.
> > +     test_grep "pathname=one.c" backend.log &&
> > +     test_grep ! "pathname=two.c" backend.log
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on tool crash' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=crash" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Crash is a communication failure, so a warning is emitted.
> > +     test_grep "diff process.*failed" stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process startup failure only warns once' '
> > +     git -c diff.cdiff.process="/nonexistent/tool" \
> > +             diff -- one.c two.c >actual 2>stderr &&
> > +     # Both files produce diff output via fallback.
> > +     test_grep "return 10" actual &&
> > +     test_grep "return 20" actual &&
> > +     # Sentinel prevents repeated warnings: only one, not one per file.
> > +     test_grep "diff process.*failed" stderr >warnings &&
> > +     test_line_count = 1 warnings
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on bad hunks' '
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "^-OLD5" actual &&
> > +     test_grep "^+NEW5" actual &&
> > +     test_grep "^-OLD9" actual &&
> > +     test_grep "^+NEW9" actual &&
> > +     # Invalid hunks are caught by xdiff validation, not the
> > +     # protocol layer, so no warning is emitted.
> > +     test_must_be_empty stderr
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     line2
> > +     line3
> > +     EOF
> > +     git add synctest.c &&
> > +     git commit -m "add synctest.c" &&
> > +
> > +     cat >synctest.c <<-\EOF &&
> > +     line1
> > +     changed
> > +     line3
> > +     EOF
> > +
> > +     # bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
> > +     # line as changed, leaving 1 unchanged old vs 2 unchanged new.
> > +     # The synchronization invariant fails and git falls back.
> > +     git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
> > +             diff synctest.c >actual 2>stderr &&
> > +     test_grep "changed" actual
> > +'
> > +
> > +test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
> > +     # boundary.c has 10 lines, so both hunks are in bounds
> > +     # but they overlap at lines 3-5, triggering the ordering check.
> > +     git -c diff.cdiff.process="$BACKEND --mode=overlap" \
> > +             diff boundary.c >actual 2>stderr &&
> > +     test_grep "NEW5" actual
> > +'
> > +
> > +test_done
> > diff --git a/userdiff.h b/userdiff.h
> > index 51c26e0d41..a98eabe377 100644
> > --- a/userdiff.h
> > +++ b/userdiff.h
> > @@ -3,6 +3,7 @@
> >
> >  #include "notes-cache.h"
> >
> > +struct diff_subprocess;
> >  struct index_state;
> >  struct repository;
> >
> > @@ -33,6 +34,8 @@ struct userdiff_driver {
> >       int textconv_want_cache;
> >       const char *process;
> >       char *process_owned;
> > +     struct diff_subprocess *diff_subprocess;
> > +     unsigned diff_process_failed : 1;
> >  };
> >  enum userdiff_driver_type {
> >       USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
> > --
> > gitgitgadget
> >
> >
> >

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> So the conscious project direction has been: fold pkt-line test backends
> into `test-tool` and drop the scripting-language prereq. Reintroducing the
> same shape in Python would walk this back.
> ...
> The `PYTHON` prereq exists in exactly five files today, all `git p4`
> related (where Python is an intrinsic prerequisite given that `git-p4.py`
> _is_ written in Python): `t/lib-git-p4.sh`, `t/t9802-git-p4-filetype.sh`,
> `t/t9810-git-p4-rcs.sh`, `t/t9835-git-p4-metadata-encoding-python2.sh`,
> and `t/t9836-git-p4-metadata-encoding-python3.sh`.
> ...
> That commit is of course not intended to be used as-is; Feel free to pick
> code parts of it and integrate them into your topic branch. Or write your
> own test-tool helper from scratch if that's more your jam.

Showing better direction to new folks with such a clear thinking is
very much appreciated.  Even though it is natural and perfectly OK
for tests that interacts with parts of Git that are written in these
languages (e.g., we are OK for gitweb tests to require Perl), we
should consciously keep ourselves clean and not adding unnecessary
dependencies.

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

Michael Montalbo <mmontalbo@gmail.com> writes:

> On Sun, Jun 7, 2026 at 7:36 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>>
>> Hi Michael,
>>
>> I stumbled about this patch when it broke CI in Git for Windows, where we
>> do _not_ use `NO_PYTHON`, even though Python is unavailable in the
>> build/test CI jobs. The existing tests handle this situation gracefully,
>> this here patch does not:
>> ...
>> Given the complexity of what t4080 tries to test (error, abort, crash,
>> bad-sync, no-hunks, multiple files in one session, capability
>> negotiation), it would unfortunately be infeasible to use `test-tool
>> pkt-line` from a shell script implementing that `diff.*.process` protocol.
>>
>> So I've spiked a demo how the `test-tool diff-process-backend` could look
>> like (letting Opus do the menial typing, so that I can enjoy at least part
>> of a sunny Sunday outside), which also passes the CI build and test:
>> https://github.com/dscho/git/commit/b6e3c93381b00929476c3a00155f7cf7334a22e6
>>
>> That commit is of course not intended to be used as-is; Feel free to pick
>> code parts of it and integrate them into your topic branch. Or write your
>> own test-tool helper from scratch if that's more your jam.
>>
>
> Johannes, thank you for the great feedback. The historical context is
> really helpful and
> the concerns you raise make a lot of sense. I will take a look at your
> spike and also work
> on removing Python from the test.

Another request.

Please do not force readers to scroll through a ~800 line message
just to read only 5 lines of response from you.  Keep relevant parts
of the message you are responding to in your message to help readers
understand the context in which your response was made, but trim
everything else that is not relevant from your quote.

Thanks.

Re: [PATCH v3 4/6] diff: add long-running diff process via diff.<driver>.process

[Junio C Hamano]

Michael Montalbo <mmontalbo@gmail.com> writes:

>> So the conscious project direction has been: fold pkt-line test backends
>> into `test-tool` and drop the scripting-language prereq. Reintroducing the
>> same shape in Python would walk this back.
>>
>> Patrick's careful effort in 27bd8ee311719 (Merge branch 'ps/fewer-perl',
>> 2025-04-29) has been another clear sign that the Git project is actively
>> _removing_ scripting-language dependencies from the build and test
>> infrastructure, not adding new ones.
>
> Now I wonder if the extension / addition of more Perl test infra with my other
> series:
>
> https://lore.kernel.org/git/pull.2135.git.1780559158.gitgitgadget@gmail.com/T/
>
> also goes against the project direction w.r.t. removing scripting languages.
> Maybe I should also re-evaluate the usage of Perl there. I am leveraging
> existing shell parsing logic written in Perl, but maybe the preference for
> Perl-based lint rules is a mistake and should be avoided.

That sounds prudent (even though it is outside the scope of _this_
topic, of course).

Thanks.

[PATCH v4 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's output entirely, losing downstream
features like word diff, function context, color, and blame.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes.

 * Patch 1: xdiff plumbing for externally supplied hunks.
 * Patch 2: diff.<driver>.process config key.
 * Patch 3: refactor subprocess API to separate process lifecycle from
   hashmap management, since the diff process stores its subprocess on the
   userdiff driver rather than in a hashmap.
 * Patch 4: the main feature.
 * Patch 5: bypass knobs (--no-ext-diff, format-patch).
 * Patch 6: blame integration so the tool can declare commits as having no
   changes.

Changes since v3:

 * Replaced Python test backend with C test-tool helper (thanks to Johannes
   Schindelin).
 * Added test coverage cases for deleted file, malformed hunk line, and
   missing capability.
 * Fixed potential overflow in synchronization invariant check by counting
   from changed[] arrays instead of accumulating.
 * Accept start=0 with count=0 in the hunk protocol, matching what git diff
   itself emits for empty file sides.
 * Warn on external hunk validation failure with specific reasons (range
   exceeded, overlap, sync mismatch) to help tool authors debug their
   implementations.
 * Test backend follows the same convention (start=0 when count=0 for empty
   file sides).

Michael Montalbo (6):
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 143 ++++++
 Makefile                                 |   2 +
 blame.c                                  |  40 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 297 ++++++++++++
 diff-process.h                           |  39 ++
 diff.c                                   |  29 +-
 diff.h                                   |   5 +
 meson.build                              |   1 +
 sub-process.c                            |  28 +-
 sub-process.h                            |   9 +-
 t/helper/meson.build                     |   1 +
 t/helper/test-diff-process-backend.c     | 299 ++++++++++++
 t/helper/test-tool.c                     |   1 +
 t/helper/test-tool.h                     |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 553 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  14 +
 xdiff/xdiffi.c                           | 123 ++++-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 27 files changed, 1614 insertions(+), 21 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh


base-commit: ea97ad8d017de0c9037451a78008a0fd60abea0c
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v3:

 1:  13eb201d63 ! 1:  03f261dfe2 xdiff: support external hunks via xpparam_t
     @@ xdiff/xdiff.h: typedef struct s_mmbuffer {
       
      +/*
      + * Hunk descriptor for externally computed diffs.
     -+ * Line numbers are 1-based, matching unified diff convention.
     ++ * Line numbers are 1-based; a start of 0 is accepted when
     ++ * count is 0 (empty file side, matching git diff output).
      + */
      +struct xdl_hunk {
      +	long old_start, old_count;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +{
      +	size_t i;
      +	long j, prev_old_end = 0, prev_new_end = 0;
     -+	long total_old = 0, total_new = 0;
     ++	long changed_old = 0, changed_new = 0;
      +
      +	/*
      +	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +	for (i = 0; i < nr_hunks; i++) {
      +		struct xdl_hunk *h = &hunks[i];
      +
     -+		if (h->old_count < 0 || h->new_count < 0)
     ++		if (h->old_count < 0 || h->new_count < 0) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"negative count (old=%ld, new=%ld)",
     ++				(uintmax_t)(i + 1),
     ++				h->old_count, h->new_count);
      +			return -1;
     -+		if (h->old_start < 1 || h->new_start < 1)
     ++		}
     ++		if (h->old_start < 1 || h->new_start < 1) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"start must be >= 1 (old=%ld, new=%ld)",
     ++				(uintmax_t)(i + 1),
     ++				h->old_start, h->new_start);
      +			return -1;
     ++		}
      +
      +		/*
      +		 * Range must fit: start + count - 1 <= nrec,
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +		 * start > nrec + 1 and allows start == nrec + 1
      +		 * (the position after the last line).
      +		 */
     -+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1)
     ++		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"old range %ld+%ld exceeds %lu lines",
     ++				(uintmax_t)(i + 1),
     ++				h->old_start, h->old_count,
     ++				(unsigned long)xe->xdf1.nrec);
      +			return -1;
     -+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
     ++		}
     ++		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"new range %ld+%ld exceeds %lu lines",
     ++				(uintmax_t)(i + 1),
     ++				h->new_start, h->new_count,
     ++				(unsigned long)xe->xdf2.nrec);
      +			return -1;
     ++		}
      +
      +		/* Ordering: no overlap with previous hunk (adjacent is OK) */
      +		if (h->old_start < prev_old_end ||
     -+		    h->new_start < prev_new_end)
     ++		    h->new_start < prev_new_end) {
     ++			warning("diff process hunk %"PRIuMAX": "
     ++				"overlaps with previous hunk",
     ++				(uintmax_t)(i + 1));
      +			return -1;
     ++		}
      +
      +		for (j = 0; j < h->old_count; j++)
      +			xe->xdf1.changed[h->old_start - 1 + j] = true;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +
      +		prev_old_end = h->old_start + h->old_count;
      +		prev_new_end = h->new_start + h->new_count;
     -+		total_old += h->old_count;
     -+		total_new += h->new_count;
      +	}
      +
      +	/*
      +	 * Synchronization invariant: unchanged line counts must match.
      +	 * Otherwise xdl_build_script() would walk off one array.
     ++	 *
     ++	 * Count changed lines from the arrays rather than accumulating
     ++	 * during the loop to avoid any overflow in the summation.
      +	 */
     -+	if ((long)xe->xdf1.nrec - total_old !=
     -+	    (long)xe->xdf2.nrec - total_new)
     ++	for (j = 0; j < (long)xe->xdf1.nrec; j++)
     ++		if (xe->xdf1.changed[j])
     ++			changed_old++;
     ++	for (j = 0; j < (long)xe->xdf2.nrec; j++)
     ++		if (xe->xdf2.changed[j])
     ++			changed_new++;
     ++	if ((long)xe->xdf1.nrec - changed_old !=
     ++	    (long)xe->xdf2.nrec - changed_new) {
     ++		warning("diff process: unchanged line count mismatch "
     ++			"(old: %ld unchanged, new: %ld unchanged)",
     ++			(long)xe->xdf1.nrec - changed_old,
     ++			(long)xe->xdf2.nrec - changed_new);
      +		return -1;
     ++	}
      +
      +	return 0;
      +}
 2:  58f4763c63 = 2:  30617ee17b userdiff: add diff.<driver>.process config
 3:  d6c833dd42 ! 3:  459e485e6d sub-process: separate process lifecycle from hashmap management
     @@ Commit message
          and subprocess_stop() become thin wrappers that add hashmap
          operations on top.
      
     -    No functional change for existing callers.
     -
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
       ## sub-process.c ##
     @@ sub-process.c: void subprocess_stop(struct hashmap *hashmap, struct subprocess_e
       	kill(entry->process.pid, SIGTERM);
       	finish_command(&entry->process);
      +}
     -+
     + 
      +void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
      +{
      +	if (!entry)
      +		return;
     - 
     ++
      +	subprocess_stop_command(entry);
       	hashmap_remove(hashmap, &entry->ent, NULL);
       }
     @@ sub-process.c: int subprocess_start(struct hashmap *hashmap, struct subprocess_e
      +	int err;
      +
      +	err = subprocess_start_command(entry, cmd, startfn);
     -+	if (err) {
     ++	if (err)
      +		return err;
     -+	}
      +
      +	hashmap_entry_init(&entry->ent, strhash(cmd));
       	hashmap_add(hashmap, &entry->ent);
 4:  d044fa0ee5 ! 4:  10b3980f59 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
          textconv-transformed content.  The tool controls which lines
          are marked as changed while the display shows the file content.
          Patch output features (word diff, function context, color) work
     -    normally; summary formats like --stat use their own diff path
     -    and are not affected.
     +    normally; --stat uses its own diff codepath and never consults
     +    the diff process.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
          requests send command=hunks, pathname, and both file contents as
     @@ Commit message
          "hunks populated" from "files equivalent" from "not applicable"
          from "tool failure."
      
     +    Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
       ## Documentation/config/diff.adoc ##
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +at line 3 in the old file were replaced by 4 lines starting at
      +line 3 in the new file.  An `<old_count>` of 0 means no lines were
      +removed (pure insertion); a `<new_count>` of 0 means no lines were
     -+added (pure deletion).
     ++added (pure deletion).  A start value of 0 is accepted when
     ++the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
     ++added file), matching what `git diff` itself emits for empty
     ++file sides.
      +
      +Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
      +file `"foo\nbar"` both have 2 lines.
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +packet:          git< 0000
      +-----------------------
      +
     -+If the tool returns invalid hunks (out of bounds, overlapping), Git
     -+silently falls back to the builtin diff algorithm.
     ++If the tool returns invalid hunks (out of bounds, overlapping, or
     ++mismatched unchanged line counts), Git warns and falls back to the
     ++builtin diff algorithm.
      +
      +In case the tool cannot or does not want to process the content,
      +it is expected to respond with an "error" status.  Git warns and
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
       
      
       ## Makefile ##
     +@@ Makefile: TEST_BUILTINS_OBJS += test-csprng.o
     + TEST_BUILTINS_OBJS += test-date.o
     + TEST_BUILTINS_OBJS += test-delete-gpgsig.o
     + TEST_BUILTINS_OBJS += test-delta.o
     ++TEST_BUILTINS_OBJS += test-diff-process-backend.o
     + TEST_BUILTINS_OBJS += test-dir-iterator.o
     + TEST_BUILTINS_OBJS += test-drop-caches.o
     + TEST_BUILTINS_OBJS += test-dump-cache-tree.o
      @@ Makefile: LIB_OBJS += diff-delta.o
       LIB_OBJS += diff-merges.o
       LIB_OBJS += diff-lib.o
     @@ diff-process.c (new)
      +	if (errno || end == line || *end != '\0')
      +		return -1;
      +
     ++	/*
     ++	 * git diff emits start=0 when count=0 (empty file side).
     ++	 * Normalize to 1-based so downstream validation can assume start >= 1.
     ++	 */
     ++	if (!hunk->old_count && !hunk->old_start)
     ++		hunk->old_start = 1;
     ++	if (!hunk->new_count && !hunk->new_start)
     ++		hunk->new_start = 1;
     ++
      +	return 0;
      +}
      +
     @@ meson.build: libgit_sources = [
         'diffcore-break.c',
         'diffcore-delta.c',
      
     - ## t/.gitattributes ##
     -@@ t/.gitattributes: t[0-9][0-9][0-9][0-9]/* -whitespace
     - /t8005/*.txt eol=lf
     - /t9*/*.dump eol=lf
     - /t0040*.sh whitespace=-indent-with-non-tab
     -+/t4080-diff-process.sh whitespace=-indent-with-non-tab
     + ## t/helper/meson.build ##
     +@@ t/helper/meson.build: test_tool_sources = [
     +   'test-date.c',
     +   'test-delete-gpgsig.c',
     +   'test-delta.c',
     ++  'test-diff-process-backend.c',
     +   'test-dir-iterator.c',
     +   'test-drop-caches.c',
     +   'test-dump-cache-tree.c',
     +
     + ## t/helper/test-diff-process-backend.c (new) ##
     +@@
     ++/*
     ++ * Test backend for the long-running diff process protocol
     ++ * (see diff-process.c and Documentation/gitattributes.adoc).
     ++ *
     ++ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
     ++ *
     ++ * Implements the server side of the pkt-line handshake and a per-file
     ++ * response loop.  The --mode= switch selects the response shape
     ++ * (success, error, abort, crash, malformed hunks).
     ++ *
     ++ * Per-file request from Git:
     ++ *
     ++ *   packet:          git> command=hunks
     ++ *   packet:          git> pathname=<path>
     ++ *   packet:          git> 0000
     ++ *   packet:          git> OLD_CONTENT
     ++ *   packet:          git> 0000
     ++ *   packet:          git> NEW_CONTENT
     ++ *   packet:          git> 0000
     ++ *
     ++ * Response varies by --mode (default: whole-file):
     ++ *
     ++ *   whole-file   packet: git< hunk 1 <old_lines> 1 <new_lines>
     ++ *   fixed-hunk   packet: git< hunk 5 2 5 2
     ++ *   no-hunks     (no hunk packets)
     ++ *   bad-hunk     packet: git< hunk 999 1 999 1
     ++ *   bad-parse    packet: git< garbage not a hunk
     ++ *   bad-sync     packet: git< hunk 1 2 1 1
     ++ *   overlap      packet: git< hunk 1 5 1 5
     ++ *                packet: git< hunk 3 2 3 2
     ++ *   no-cap       (omits capability=hunks during handshake)
     ++ *   error        (status=error instead of status=success)
     ++ *   abort        (status=abort instead of status=success)
     ++ *   crash        exit(1) before sending any response
     ++ *
     ++ * All non-error/abort modes end with:
     ++ *
     ++ *   packet:          git< 0000
     ++ *   packet:          git< status=success
     ++ *   packet:          git< 0000
     ++ *
     ++ * Each request is logged to --log as:
     ++ *
     ++ *   command=<cmd> pathname=<path> old=<first line> new=<first line>
     ++ */
     ++
     ++#include "test-tool.h"
     ++#include "pkt-line.h"
     ++#include "parse-options.h"
     ++#include "strbuf.h"
     ++
     ++static FILE *logfile;
     ++
     ++enum mode {
     ++	MODE_WHOLE_FILE,
     ++	MODE_FIXED_HUNK,
     ++	MODE_NO_HUNKS,
     ++	MODE_BAD_HUNK,
     ++	MODE_BAD_PARSE,
     ++	MODE_BAD_SYNC,
     ++	MODE_OVERLAP,
     ++	MODE_NO_CAP,
     ++	MODE_ERROR,
     ++	MODE_ABORT,
     ++	MODE_CRASH,
     ++};
     ++
     ++static enum mode parse_mode(const char *s)
     ++{
     ++	if (!strcmp(s, "whole-file"))
     ++		return MODE_WHOLE_FILE;
     ++	if (!strcmp(s, "fixed-hunk"))
     ++		return MODE_FIXED_HUNK;
     ++	if (!strcmp(s, "no-hunks"))
     ++		return MODE_NO_HUNKS;
     ++	if (!strcmp(s, "bad-hunk"))
     ++		return MODE_BAD_HUNK;
     ++	if (!strcmp(s, "bad-parse"))
     ++		return MODE_BAD_PARSE;
     ++	if (!strcmp(s, "bad-sync"))
     ++		return MODE_BAD_SYNC;
     ++	if (!strcmp(s, "overlap"))
     ++		return MODE_OVERLAP;
     ++	if (!strcmp(s, "no-cap"))
     ++		return MODE_NO_CAP;
     ++	if (!strcmp(s, "error"))
     ++		return MODE_ERROR;
     ++	if (!strcmp(s, "abort"))
     ++		return MODE_ABORT;
     ++	if (!strcmp(s, "crash"))
     ++		return MODE_CRASH;
     ++	die("unknown --mode=%s", s);
     ++}
     ++
     ++/*
     ++ * Read "key=value" packets up to a flush, capturing "command" and
     ++ * "pathname".  Returns 1 if a request was read, 0 on EOF.
     ++ *
     ++ * The first packet uses the gentle variant so that a clean shutdown
     ++ * by Git (EOF) does not produce a spurious "the remote end hung up
     ++ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
     ++ * variant: once inside a request, truncation is a protocol violation
     ++ * and dying loudly is the correct response.
     ++ */
     ++static int read_request_header(char **command, char **pathname)
     ++{
     ++	int first = 1;
     ++	char *line;
     ++
     ++	*command = *pathname = NULL;
     ++	for (;;) {
     ++		const char *value;
     ++
     ++		if (first) {
     ++			if (packet_read_line_gently(0, NULL, &line) < 0)
     ++				return 0;
     ++			first = 0;
     ++		} else {
     ++			line = packet_read_line(0, NULL);
     ++		}
     ++		if (!line)
     ++			break;
     ++		if (skip_prefix(line, "command=", &value))
     ++			*command = xstrdup(value);
     ++		else if (skip_prefix(line, "pathname=", &value))
     ++			*pathname = xstrdup(value);
     ++	}
     ++	return 1;
     ++}
     ++
     ++static size_t count_lines(const struct strbuf *buf)
     ++{
     ++	size_t lines = 0;
     ++
     ++	for (size_t i = 0; i < buf->len; i++)
     ++		if (buf->buf[i] == '\n')
     ++			lines++;
     ++
     ++	return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
     ++}
     ++
     ++static void send_status(const char *status)
     ++{
     ++	packet_flush(1);
     ++	packet_write_fmt(1, "%s\n", status);
     ++	packet_flush(1);
     ++}
     ++
     ++static void respond(enum mode mode,
     ++		    const struct strbuf *old_buf,
     ++		    const struct strbuf *new_buf)
     ++{
     ++	switch (mode) {
     ++	case MODE_ERROR:
     ++		send_status("status=error");
     ++		return;
     ++	case MODE_ABORT:
     ++		send_status("status=abort");
     ++		return;
     ++	case MODE_CRASH:
     ++		exit(1);
     ++	case MODE_FIXED_HUNK:
     ++		packet_write_fmt(1, "hunk 5 2 5 2\n");
     ++		break;
     ++	case MODE_BAD_HUNK:
     ++		packet_write_fmt(1, "hunk 999 1 999 1\n");
     ++		break;
     ++	case MODE_BAD_PARSE:
     ++		packet_write_fmt(1, "garbage not a hunk\n");
     ++		break;
     ++	case MODE_BAD_SYNC:
     ++		packet_write_fmt(1, "hunk 1 2 1 1\n");
     ++		break;
     ++	case MODE_OVERLAP:
     ++		packet_write_fmt(1, "hunk 1 5 1 5\n");
     ++		packet_write_fmt(1, "hunk 3 2 3 2\n");
     ++		break;
     ++	case MODE_NO_HUNKS:
     ++		break;
     ++	case MODE_NO_CAP:
     ++	case MODE_WHOLE_FILE: {
     ++		size_t old_lines = count_lines(old_buf);
     ++		size_t new_lines = count_lines(new_buf);
     ++		/*
     ++		 * Match git diff output: start=0 when count=0
     ++		 * (empty file side), 1 otherwise.
     ++		 */
     ++		packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
     ++				 " %"PRIuMAX" %"PRIuMAX"\n",
     ++				 (uintmax_t)(old_lines ? 1 : 0),
     ++				 (uintmax_t)old_lines,
     ++				 (uintmax_t)(new_lines ? 1 : 0),
     ++				 (uintmax_t)new_lines);
     ++		break;
     ++	}
     ++	}
     ++	send_status("status=success");
     ++}
     ++
     ++static void command_loop(enum mode mode)
     ++{
     ++	for (;;) {
     ++		char *command = NULL, *pathname = NULL;
     ++		struct strbuf obuf = STRBUF_INIT;
     ++		struct strbuf nbuf = STRBUF_INIT;
     ++
     ++		if (!read_request_header(&command, &pathname))
     ++			break; /* EOF: Git closed its end */
     ++
     ++		read_packetized_to_strbuf(0, &obuf, 0);
     ++		read_packetized_to_strbuf(0, &nbuf, 0);
     ++
     ++		if (logfile) {
     ++			fprintf(logfile,
     ++				"command=%s pathname=%s old=%.*s new=%.*s\n",
     ++				command ? command : "(none)",
     ++				pathname ? pathname : "(none)",
     ++				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
     ++				obuf.buf,
     ++				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
     ++				nbuf.buf);
     ++			fflush(logfile);
     ++		}
     ++
     ++		respond(mode, &obuf, &nbuf);
     ++
     ++		free(command);
     ++		free(pathname);
     ++		strbuf_release(&obuf);
     ++		strbuf_release(&nbuf);
     ++	}
     ++}
     ++
     ++static void handshake(enum mode mode)
     ++{
     ++	char *line;
     ++
     ++	line = packet_read_line(0, NULL);
     ++	if (!line || strcmp(line, "git-diff-client"))
     ++		die("bad welcome: '%s'", line ? line : "(eof)");
     ++	line = packet_read_line(0, NULL);
     ++	if (!line || strcmp(line, "version=1"))
     ++		die("bad version: '%s'", line ? line : "(eof)");
     ++	if (packet_read_line(0, NULL))
     ++		die("expected flush after version");
     ++
     ++	packet_write_fmt(1, "git-diff-server\n");
     ++	packet_write_fmt(1, "version=1\n");
     ++	packet_flush(1);
     ++
     ++	/* Drain capabilities advertised by Git */
     ++	while ((line = packet_read_line(0, NULL)))
     ++		; /* drain */
     ++
     ++	/* Respond with our capabilities (or none for no-cap mode) */
     ++	if (mode != MODE_NO_CAP)
     ++		packet_write_fmt(1, "capability=hunks\n");
     ++	packet_flush(1);
     ++}
     ++
     ++static const char *const usage_str[] = {
     ++	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
     ++	NULL
     ++};
     ++
     ++int cmd__diff_process_backend(int argc, const char **argv)
     ++{
     ++	const char *mode_str = NULL, *log_path = NULL;
     ++	enum mode mode = MODE_WHOLE_FILE;
     ++	struct option options[] = {
     ++		OPT_STRING(0, "mode", &mode_str, "mode",
     ++			   "response shape: whole-file (default), fixed-hunk,"
     ++			   " no-hunks, bad-hunk, bad-sync, overlap, error,"
     ++			   " abort, crash"),
     ++		OPT_STRING(0, "log", &log_path, "path",
     ++			   "append per-request summary to this file"),
     ++		OPT_END()
     ++	};
     ++
     ++	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
     ++	if (argc)
     ++		usage_with_options(usage_str, options);
     ++
     ++	if (mode_str)
     ++		mode = parse_mode(mode_str);
     ++
     ++	if (log_path) {
     ++		logfile = fopen(log_path, "a");
     ++		if (!logfile)
     ++			die_errno("failed to open log '%s'", log_path);
     ++	}
     ++
     ++	handshake(mode);
     ++	command_loop(mode);
     ++
     ++	if (logfile && fclose(logfile))
     ++		die_errno("error closing log");
     ++	return 0;
     ++}
     +
     + ## t/helper/test-tool.c ##
     +@@ t/helper/test-tool.c: static struct test_cmd cmds[] = {
     + 	{ "date", cmd__date },
     + 	{ "delete-gpgsig", cmd__delete_gpgsig },
     + 	{ "delta", cmd__delta },
     ++	{ "diff-process-backend", cmd__diff_process_backend },
     + 	{ "dir-iterator", cmd__dir_iterator },
     + 	{ "drop-caches", cmd__drop_caches },
     + 	{ "dump-cache-tree", cmd__dump_cache_tree },
     +
     + ## t/helper/test-tool.h ##
     +@@ t/helper/test-tool.h: int cmd__csprng(int argc, const char **argv);
     + int cmd__date(int argc, const char **argv);
     + int cmd__delta(int argc, const char **argv);
     + int cmd__delete_gpgsig(int argc, const char **argv);
     ++int cmd__diff_process_backend(int argc, const char **argv);
     + int cmd__dir_iterator(int argc, const char **argv);
     + int cmd__drop_caches(int argc, const char **argv);
     + int cmd__dump_cache_tree(int argc, const char **argv);
      
       ## t/meson.build ##
      @@ t/meson.build: integration_tests = [
     @@ t/t4080-diff-process.sh (new)
      +
      +. ./test-lib.sh
      +
     -+if test_have_prereq PYTHON
     -+then
     -+	PYTHON_PATH=$(command -v python3) || PYTHON_PATH=$(command -v python)
     -+fi
     ++# See t/helper/test-diff-process-backend.c for the backend implementation
     ++# and available --mode= options.
      +
     -+#
     -+# A single parametric diff process.
     -+# Usage: diff-process-backend --mode=<mode> [--log=<path>]
     -+#
     -+# Modes:
     -+#   whole-file  - report all lines as changed (default)
     -+#   fixed-hunk  - always report hunk 5 2 5 2
     -+#   bad-hunk    - report out-of-bounds hunk 999 1 999 1
     -+#   bad-sync    - report hunk with mismatched unchanged totals
     -+#   overlap     - report two overlapping hunks
     -+#   no-hunks   - return no hunks (files considered equivalent)
     -+#   error       - return status=error for every request
     -+#   abort       - return status=abort for every request
     -+#   crash       - read one request then exit without responding
     -+#
     -+setup_backend () {
     -+	cat >"$TRASH_DIRECTORY/diff-process-backend.py" <<-\PYEOF
     -+	import sys, os
     -+
     -+	def read_pkt():
     -+	    hdr = sys.stdin.buffer.read(4)
     -+	    if len(hdr) < 4: return None
     -+	    length = int(hdr, 16)
     -+	    if length == 0: return ""
     -+	    data = sys.stdin.buffer.read(length - 4)
     -+	    return data.decode().rstrip("\n")
     -+
     -+	def write_pkt(line):
     -+	    data = (line + "\n").encode()
     -+	    sys.stdout.buffer.write(f"{len(data)+4:04x}".encode() + data)
     -+	    sys.stdout.buffer.flush()
     -+
     -+	def write_flush():
     -+	    sys.stdout.buffer.write(b"0000")
     -+	    sys.stdout.buffer.flush()
     -+
     -+	def read_content():
     -+	    chunks = []
     -+	    while True:
     -+	        hdr = sys.stdin.buffer.read(4)
     -+	        if len(hdr) < 4: break
     -+	        length = int(hdr, 16)
     -+	        if length == 0: break
     -+	        chunks.append(sys.stdin.buffer.read(length - 4))
     -+	    return b"".join(chunks)
     -+
     -+	mode = "whole-file"
     -+	logfile = None
     -+	for arg in sys.argv[1:]:
     -+	    if arg.startswith("--mode="):
     -+	        mode = arg[7:]
     -+	    elif arg.startswith("--log="):
     -+	        logfile = open(arg[6:], "a")
     -+
     -+	def log(msg):
     -+	    if logfile:
     -+	        logfile.write(msg + "\n")
     -+	        logfile.flush()
     -+
     -+	# Handshake
     -+	assert read_pkt() == "git-diff-client"
     -+	assert read_pkt() == "version=1"
     -+	read_pkt()
     -+	write_pkt("git-diff-server")
     -+	write_pkt("version=1")
     -+	write_flush()
     -+	while True:
     -+	    p = read_pkt()
     -+	    if p == "": break
     -+	write_pkt("capability=hunks")
     -+	write_flush()
     -+
     -+	log("ready")
     -+
     -+	while True:
     -+	    cmd = None
     -+	    pathname = None
     -+	    while True:
     -+	        p = read_pkt()
     -+	        if p is None: sys.exit(0)
     -+	        if p == "": break
     -+	        if p.startswith("command="): cmd = p.split("=",1)[1]
     -+	        if p.startswith("pathname="): pathname = p.split("=",1)[1]
     -+	    if cmd is None: sys.exit(0)
     -+	    old = read_content()
     -+	    new = read_content()
     -+	    old_first = old.split(b"\n")[0].decode(errors="replace") if old else ""
     -+	    new_first = new.split(b"\n")[0].decode(errors="replace") if new else ""
     -+	    log(f"command={cmd} pathname={pathname} old={old_first} new={new_first}")
     -+
     -+	    if mode == "error":
     -+	        write_flush()
     -+	        write_pkt("status=error")
     -+	        write_flush()
     -+	        continue
     -+
     -+	    if mode == "abort":
     -+	        write_flush()
     -+	        write_pkt("status=abort")
     -+	        write_flush()
     -+	        continue
     -+
     -+	    if mode == "crash":
     -+	        sys.exit(1)
     -+
     -+	    if cmd == "hunks":
     -+	        if mode == "fixed-hunk":
     -+	            write_pkt("hunk 5 2 5 2")
     -+	        elif mode == "bad-hunk":
     -+	            write_pkt("hunk 999 1 999 1")
     -+	        elif mode == "bad-sync":
     -+	            write_pkt("hunk 1 2 1 1")
     -+	        elif mode == "overlap":
     -+	            write_pkt("hunk 1 5 1 5")
     -+	            write_pkt("hunk 3 2 3 2")
     -+	        elif mode == "no-hunks":
     -+	            pass
     -+	        else:
     -+	            ol = old.count(b"\n")
     -+	            nl = new.count(b"\n")
     -+	            write_pkt(f"hunk 1 {ol} 1 {nl}")
     -+	        write_flush()
     -+	        write_pkt("status=success")
     -+	        write_flush()
     -+	    else:
     -+	        write_flush()
     -+	        write_pkt("status=error")
     -+	        write_flush()
     -+	PYEOF
     -+	write_script diff-process-backend <<-SHEOF
     -+	exec "$PYTHON_PATH" "$TRASH_DIRECTORY/diff-process-backend.py" "\$@"
     -+	SHEOF
     -+}
     -+
     -+BACKEND="./diff-process-backend"
     ++BACKEND="test-tool diff-process-backend"
      +
     -+test_expect_success PYTHON 'setup' '
     -+	setup_backend &&
     ++test_expect_success 'setup' '
      +	echo "*.c diff=cdiff" >.gitattributes &&
      +	git add .gitattributes &&
      +
     @@ t/t4080-diff-process.sh (new)
      +	git add worddiff.c &&
      +
      +	# newfile.c: single-line function, value changes 42 -> 99.
     -+	# Used by: new file, --exit-code, multiple drivers.
     ++	# Used by: modified file, --exit-code, multiple drivers.
      +	cat >newfile.c <<-\EOF &&
      +	int new_func(void) { return 42; }
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +# Core behavior: the tool controls which lines are marked as changed.
      +#
      +
     -+test_expect_success PYTHON 'diff process hunk boundaries affect output' '
     ++test_expect_success 'diff process hunk boundaries affect output' '
      +	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
      +	# only reports lines 5-6 as changed.  Lines 9-10 should not
      +	# appear as changed in the output.
     @@ t/t4080-diff-process.sh (new)
      +	test_grep ! "^+NEW10" actual
      +'
      +
     -+test_expect_success PYTHON 'diff process works with new file' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process works with modified file' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- newfile.c >actual 2>stderr &&
      +	test_grep "return 99" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process works with added file (empty old side)' '
     ++test_expect_success 'diff process works with added file (empty old side)' '
      +	cat >added.c <<-\EOF &&
      +	int added(void) { return 1; }
      +	EOF
      +	git add added.c &&
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --cached -- added.c >actual 2>stderr &&
      +	test_grep "added" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process skipped for binary files' '
     ++test_expect_success 'diff process works with deleted file (empty new side)' '
     ++	git add added.c &&
     ++	git commit -m "commit added.c" &&
     ++	git rm added.c &&
     ++
     ++	test_when_finished "rm -f backend.log" &&
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff --cached -- added.c >actual 2>stderr &&
     ++	test_grep "deleted file" actual &&
     ++	test_grep "pathname=added.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process skipped for binary files' '
      +	printf "\\0binary" >binary.c &&
      +	git add binary.c &&
      +	git commit -m "add binary" &&
      +	printf "\\0changed" >binary.c &&
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- binary.c >actual &&
      +	test_grep "Binary files" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process not consulted for unmatched driver' '
     ++test_expect_success 'diff process not consulted for unmatched driver' '
      +	echo "not tracked by cdiff" >unmatched.txt &&
      +	git add unmatched.txt &&
      +	git commit -m "add unmatched.txt" &&
      +
      +	echo "modified" >unmatched.txt &&
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- unmatched.txt >actual &&
      +	test_grep "modified" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'multiple drivers use separate processes' '
     ++test_expect_success 'multiple drivers use separate processes' '
      +	echo "*.h diff=hdiff" >>.gitattributes &&
      +	git add .gitattributes &&
      +
     @@ t/t4080-diff-process.sh (new)
      +	int header(void) { return 2; }
      +	EOF
      +
     -+	rm -f backend-c.log backend-h.log &&
     ++	test_when_finished "rm -f backend-c.log backend-h.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
      +	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
      +		diff -- newfile.c multi.h >actual 2>stderr &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process works alongside textconv' '
     ++test_expect_success 'diff process works alongside textconv' '
      +	write_script uppercase-filter <<-\EOF &&
      +	tr "a-z" "A-Z" <"$1"
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +	goodbye world
      +	EOF
      +
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.textconv="./uppercase-filter" \
      +	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff -- textconv.c >actual 2>stderr &&
     @@ t/t4080-diff-process.sh (new)
      +# Downstream features: word diff, log, equivalent files, exit code.
      +#
      +
     -+test_expect_success PYTHON 'diff process with --word-diff' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process with --word-diff' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --word-diff worddiff.c >actual 2>stderr &&
      +	test_grep "\[-1;-\]" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process works with git log -p' '
     ++test_expect_success 'diff process works with git log -p' '
      +	# With no-hunks mode, the tool says the files are equivalent,
      +	# so log -p should show the commit but no diff content.
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
      +		log -1 -p -- logtest.c >actual 2>stderr &&
      +	test_grep "change logtest.c" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process no hunks suppresses diff output' '
     ++test_expect_success 'diff process no hunks suppresses diff output' '
      +	cat >nohunks.c <<-\EOF &&
      +	int zero(void) { return 0; }
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +	test_must_be_empty actual
      +'
      +
     -+test_expect_success PYTHON 'diff process no hunks with --exit-code returns success' '
     ++test_expect_success 'diff process no hunks with --exit-code returns success' '
      +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
      +		diff --exit-code nohunks.c
      +'
      +
     -+test_expect_success PYTHON 'diff process with --exit-code and hunks returns failure' '
     ++test_expect_success 'diff process with --exit-code and hunks returns failure' '
      +	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
      +		diff --exit-code newfile.c
      +'
     @@ t/t4080-diff-process.sh (new)
      +# Bypass mechanisms: flags and commands that skip the diff process.
      +#
      +
     -+test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process bypassed by --diff-algorithm' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --diff-algorithm=patience worddiff.c >actual &&
      +	test_grep "return 999" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process not used by --stat' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process not used by --stat' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --stat worddiff.c >actual &&
      +	test_grep "worddiff.c" actual &&
     @@ t/t4080-diff-process.sh (new)
      +# Error handling and fallback.
      +#
      +
     -+test_expect_success PYTHON 'diff process fallback on tool error status' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process fallback on tool error status' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
      +		diff boundary.c >actual 2>stderr &&
      +	# Fallback produces the full builtin diff (both change regions).
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process error keeps tool available for next file' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process error keeps tool available for next file' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
      +		diff -- one.c two.c >actual 2>stderr &&
      +	# Unlike abort, error keeps the tool available: both files
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "pathname=one.c" backend.log &&
      +	test_grep "pathname=two.c" backend.log &&
      +	test_grep "return 10" actual &&
     -+	test_grep "return 20" actual
     ++	test_grep "return 20" actual &&
     ++	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process abort disables for session' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process abort disables for session' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
     -+		diff -- one.c two.c >actual &&
     ++		diff -- one.c two.c >actual 2>stderr &&
      +	# Both files should still produce diff output via fallback.
      +	test_grep "return 10" actual &&
      +	test_grep "return 20" actual &&
      +	# The tool aborts on the first file and git clears its
      +	# capability.  The second file never contacts the tool.
      +	test_grep "pathname=one.c" backend.log &&
     -+	test_grep ! "pathname=two.c" backend.log
     ++	test_grep ! "pathname=two.c" backend.log &&
     ++	test_must_be_empty stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on tool crash' '
     ++test_expect_success 'diff process fallback on tool crash' '
      +	git -c diff.cdiff.process="$BACKEND --mode=crash" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "^-OLD5" actual &&
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "diff process.*failed" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process startup failure only warns once' '
     ++test_expect_success 'diff process startup failure only warns once' '
      +	git -c diff.cdiff.process="/nonexistent/tool" \
      +		diff -- one.c two.c >actual 2>stderr &&
      +	# Both files produce diff output via fallback.
     @@ t/t4080-diff-process.sh (new)
      +	test_line_count = 1 warnings
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on bad hunks' '
     ++
     ++test_expect_success 'diff process fallback on bad hunks' '
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "^-OLD5" actual &&
      +	test_grep "^+NEW5" actual &&
      +	test_grep "^-OLD9" actual &&
      +	test_grep "^+NEW9" actual &&
     -+	# Invalid hunks are caught by xdiff validation, not the
     -+	# protocol layer, so no warning is emitted.
     -+	test_must_be_empty stderr
     ++	test_grep "exceeds.*lines" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on mismatched unchanged totals' '
     ++test_expect_success 'diff process fallback on mismatched unchanged totals' '
      +	cat >synctest.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh (new)
      +	# The synchronization invariant fails and git falls back.
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
      +		diff synctest.c >actual 2>stderr &&
     -+	test_grep "changed" actual
     ++	test_grep "changed" actual &&
     ++	test_grep "unchanged line count mismatch" stderr
      +'
      +
     -+test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     ++test_expect_success 'diff process fallback on overlapping hunks' '
      +	# boundary.c has 10 lines, so both hunks are in bounds
      +	# but they overlap at lines 3-5, triggering the ordering check.
      +	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
      +		diff boundary.c >actual 2>stderr &&
     -+	test_grep "NEW5" actual
     ++	test_grep "NEW5" actual &&
     ++	test_grep "overlaps with previous" stderr
     ++'
     ++
     ++test_expect_success 'diff process fallback on malformed hunk line' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual
     ++'
     ++
     ++test_expect_success 'diff process skipped when tool omits capability' '
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_must_be_empty stderr
      +'
      +
      +test_done
 5:  f4fd9aa682 ! 5:  6ec6716ea4 diff: bypass diff process with --no-ext-diff and in format-patch
     @@ Commit message
          external tool.
      
          Document that --diff-algorithm also bypasses the diff process,
     -    since it sets ignore_driver_algorithm which diff_process_fill_hunks
     -    already checks.
     +    since it forces the builtin algorithm.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ diff.h: struct diff_flags {
       	/**
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process bypassed by --diff-algorithm' '
     +@@ t/t4080-diff-process.sh: test_expect_success 'diff process bypassed by --diff-algorithm' '
       	test_path_is_missing backend.log
       '
       
     -+test_expect_success PYTHON 'diff process bypassed by --no-ext-diff' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process bypassed by --no-ext-diff' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		diff --no-ext-diff worddiff.c >actual &&
      +	test_grep "return 999" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'diff process not used by format-patch' '
     -+	rm -f backend.log &&
     ++test_expect_success 'diff process not used by format-patch' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
      +		format-patch -1 --stdout -- logtest.c >actual &&
      +	test_grep "return 2" actual &&
      +	test_path_is_missing backend.log
      +'
      +
     - test_expect_success PYTHON 'diff process not used by --stat' '
     - 	rm -f backend.log &&
     + test_expect_success 'diff process not used by --stat' '
     + 	test_when_finished "rm -f backend.log" &&
       	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
 6:  370e766978 ! 6:  3dadafa1bc blame: consult diff process for no-hunk detection
     @@ Commit message
          The consultation happens at the pass_blame_to_parent() callsite
          using diff_process_fill_hunks(), matching how builtin_diff() in
          diff.c uses the same function.  A new diff_hunks_xpp() variant
     -    accepts a pre-populated xpparam_t for this callsite, while the
     -    existing diff_hunks() retains its original signature and behavior.
     -    The copy-detection callsite is unaffected since it does not use
     -    the diff process.
     +    accepts a pre-populated xpparam_t so callers can pass external
     +    hunks, while the existing diff_hunks() retains its original
     +    signature and behavior.  The copy-detection callsite is
     +    unaffected since it does not use the diff process.
      
          The subprocess is long-running (one startup cost amortized
          across the blame traversal), but each commit in the file's
     @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       		    parent, target, 0);
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on overlapping hunks' '
     - 	test_grep "NEW5" actual
     +@@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omits capability' '
     + 	test_must_be_empty stderr
       '
       
      +#
      +# Blame integration.
      +#
      +
     -+test_expect_success PYTHON 'blame uses tool-provided hunks' '
     ++test_expect_success 'blame uses tool-provided hunks' '
      +	cat >blame-hunk.c <<-\EOF &&
      +	line1
      +	line2
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	test_grep "$CHANGE" line6
      +'
      +
     -+test_expect_success PYTHON 'blame skips commits with no hunks from diff process' '
     ++test_expect_success 'blame skips commits with no hunks from diff process' '
      +	cat >blame.c <<-\EOF &&
     -+	int main(void)
     -+	{
     -+	    return 0;
     ++	int main(void) {
     ++	return 0;
      +	}
      +	EOF
      +	git add blame.c &&
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	cat >blame.c <<-\EOF &&
      +	int main(void)
      +	{
     -+	        return 0;
     ++	return 0;
      +	}
      +	EOF
      +	git add blame.c &&
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	test_grep "$ORIG_COMMIT" with
      +'
      +
     -+test_expect_success PYTHON 'blame --no-ext-diff bypasses diff process' '
     -+	rm -f backend.log &&
     ++test_expect_success 'blame --no-ext-diff bypasses diff process' '
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
      +		blame --no-ext-diff blame.c >actual &&
      +	# Without the process, blame attributes the reformat commit normally.
     @@ t/t4080-diff-process.sh: test_expect_success PYTHON 'diff process fallback on ov
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success PYTHON 'blame --no-ext-diff uses builtin hunks' '
     ++test_expect_success 'blame --no-ext-diff uses builtin hunks' '
      +	# fixed-hunk mode would narrow blame to lines 5-6, but
      +	# --no-ext-diff should bypass it and use the builtin diff.
     -+	rm -f backend.log &&
     ++	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
      +		blame --no-ext-diff blame-hunk.c >actual &&
      +	# Builtin diff attributes lines 9-10 to the change commit.

-- 
gitgitgadget

[PATCH v4 1/6] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use: reject out-of-bounds line
numbers, overlapping or out-of-order hunks, negative counts, and
violations of the synchronization invariant (unchanged line counts
must match between files).  On validation failure, fall back to
the builtin diff algorithm; this re-runs xdl_prepare_env() since
the first call may have dirtied the changed[] arrays.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |   7 ++-
 xdiff/xdiff.h     |  14 ++++++
 xdiff/xdiffi.c    | 123 +++++++++++++++++++++++++++++++++++++++++++++-
 xdiff/xprepare.c  |  10 ++++
 xdiff/xprepare.h  |   1 +
 5 files changed, 152 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index 5ee2b96d0a..76a24fc589 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..dd4915fe16 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,16 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs.
+ * Line numbers are 1-based; a start of 0 is accepted when
+ * count is 0 (empty file side, matching git diff output).
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +98,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
+	struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index c5a892f91e..bf820b52e3 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,135 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  Validates that hunks are in order,
+ * non-overlapping, and within bounds.
+ *
+ * Returns 0 on success, -1 on validation failure.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j, prev_old_end = 0, prev_new_end = 0;
+	long changed_old = 0, changed_new = 0;
+
+	/*
+	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
+	 * Clear them so only the external hunks are marked.
+	 */
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count < 0 || h->new_count < 0) {
+			warning("diff process hunk %"PRIuMAX": "
+				"negative count (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_count, h->new_count);
+			return -1;
+		}
+		if (h->old_start < 1 || h->new_start < 1) {
+			warning("diff process hunk %"PRIuMAX": "
+				"start must be >= 1 (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_start, h->new_start);
+			return -1;
+		}
+
+		/*
+		 * Range must fit: start + count - 1 <= nrec,
+		 * rewritten to avoid overflow.  Same for both sides.
+		 *
+		 * When count is 0 (pure insert/delete) the check
+		 * reduces to 0 > nrec - start + 1, which rejects
+		 * start > nrec + 1 and allows start == nrec + 1
+		 * (the position after the last line).
+		 */
+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
+			warning("diff process hunk %"PRIuMAX": "
+				"old range %ld+%ld exceeds %lu lines",
+				(uintmax_t)(i + 1),
+				h->old_start, h->old_count,
+				(unsigned long)xe->xdf1.nrec);
+			return -1;
+		}
+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
+			warning("diff process hunk %"PRIuMAX": "
+				"new range %ld+%ld exceeds %lu lines",
+				(uintmax_t)(i + 1),
+				h->new_start, h->new_count,
+				(unsigned long)xe->xdf2.nrec);
+			return -1;
+		}
+
+		/* Ordering: no overlap with previous hunk (adjacent is OK) */
+		if (h->old_start < prev_old_end ||
+		    h->new_start < prev_new_end) {
+			warning("diff process hunk %"PRIuMAX": "
+				"overlaps with previous hunk",
+				(uintmax_t)(i + 1));
+			return -1;
+		}
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+	}
+
+	/*
+	 * Synchronization invariant: unchanged line counts must match.
+	 * Otherwise xdl_build_script() would walk off one array.
+	 *
+	 * Count changed lines from the arrays rather than accumulating
+	 * during the loop to avoid any overflow in the summation.
+	 */
+	for (j = 0; j < (long)xe->xdf1.nrec; j++)
+		if (xe->xdf1.changed[j])
+			changed_old++;
+	for (j = 0; j < (long)xe->xdf2.nrec; j++)
+		if (xe->xdf2.changed[j])
+			changed_new++;
+	if ((long)xe->xdf1.nrec - changed_old !=
+	    (long)xe->xdf2.nrec - changed_new) {
+		warning("diff process: unchanged line count mismatch "
+			"(old: %ld unchanged, new: %ld unchanged)",
+			(long)xe->xdf1.nrec - changed_old,
+			(long)xe->xdf2.nrec - changed_new);
+		return -1;
+	}
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index 11bada2608..f4ab935332 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v4 2/6] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index b5412e6bc3..7547874aa2 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v4 3/6] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and
managing a hashmap that indexes running processes by command
string.  The hashmap suits callers like convert.c where many
files may share one filter process looked up by name, but
callers that manage process lifetime through their own data
structures do not need it.

Extract subprocess_start_command() and subprocess_stop_command()
so callers can reuse the child process setup and handshake
machinery without maintaining a hashmap.  subprocess_start()
and subprocess_stop() become thin wrappers that add hashmap
operations on top.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 28 +++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 83bf0a0e82..5468939338 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
 
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
+
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err)
+		return err;
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
gitgitgadget

[PATCH v4 4/6] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference line numbers in the content.
When textconv is also configured, the tool receives the
textconv-transformed content.  The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
normally; --stat uses its own diff codepath and never consults
the diff process.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, and both file contents as
packetized data.  The tool responds with hunk lines and a status
packet (success, error, or abort).  On error, Git warns and falls
back to the builtin diff algorithm for that file.  On abort, Git
silently falls back for the current file and stops sending further
requests to the tool for the remainder of the session.

When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
This also means --exit-code reports no changes for that file.

The subprocess is stored on the userdiff_driver struct and
launched on first use.  If the process fails to start, the
handshake fails, or a communication error occurs mid-stream,
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.

diff_process_fill_hunks() is the sole public entry point.  It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc       |   5 +
 Documentation/gitattributes.adoc     | 143 +++++++++
 Makefile                             |   2 +
 diff-process.c                       | 297 ++++++++++++++++++
 diff-process.h                       |  39 +++
 diff.c                               |  13 +
 diff.h                               |   3 +
 meson.build                          |   1 +
 t/helper/meson.build                 |   1 +
 t/helper/test-diff-process-backend.c | 299 ++++++++++++++++++
 t/helper/test-tool.c                 |   1 +
 t/helper/test-tool.h                 |   1 +
 t/meson.build                        |   1 +
 t/t4080-diff-process.sh              | 432 +++++++++++++++++++++++++++
 userdiff.h                           |   3 +
 15 files changed, 1241 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..ac0635bb3b 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index bd76167a45..49ed11d069 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -821,6 +821,149 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If `diff.<name>.process` is defined, Git sends the old and new file
+content to an external tool and receives back a list of changed
+regions (pairs of line ranges in the old and new file).  Git uses
+these instead of its builtin diff algorithm, but still controls
+all output formatting, so features like word diff, function context,
+color, and blame work normally.  This is achieved by using the
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
+the diff process feeds results back into the standard pipeline.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+Then, define a "diff.<name>.process" configuration to specify
+the diff process command.
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+When Git encounters the first file that needs to be diffed, it starts
+the process and performs the handshake.  In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
+described below).
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
+data, each terminated with a flush packet.  The pathname is relative
+to the repository root.  When `diff.<name>.textconv` is also set,
+the tool receives the textconv-transformed content rather than the
+raw blob.  Git does not send binary files to the diff process.
+
+-----------------------
+packet:          git> command=hunks
+packet:          git> pathname=path/file.c
+packet:          git> 0000
+packet:          git> OLD_CONTENT
+packet:          git> 0000
+packet:          git> NEW_CONTENT
+packet:          git> 0000
+-----------------------
+
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
+
+  `hunk <old_start> <old_count> <new_start> <new_count>`
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
+replacement range in the new file.  Start values are 1-based and
+counts are non-negative.  Ranges must not extend beyond the end of
+the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
+at line 3 in the old file were replaced by 4 lines starting at
+line 3 in the new file.  An `<old_count>` of 0 means no lines were
+removed (pure insertion); a `<new_count>` of 0 means no lines were
+added (pure deletion).  A start value of 0 is accepted when
+the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
+added file), matching what `git diff` itself emits for empty
+file sides.
+
+Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
+Hunks must be listed in order and must not overlap.  Any line
+not covered by a hunk is treated as unchanged, so the total
+number of unchanged lines must be the same on both sides.
+For example, if the old file has 10 lines and the hunks cover
+4 of them (`old_count` values summing to 4), then 6 old lines
+are unchanged.  The new file must also have exactly 6 lines
+not covered by hunks, so the `new_count` values must sum to
+`new_file_lines - 6`.
+
+-----------------------
+packet:          git< hunk 1 3 1 5
+packet:          git< hunk 10 2 12 2
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
+as changed and feeds them into the standard diff pipeline.  Patch
+output features (word diff, function context, color) work normally.
+Note that `--stat` and other summary formats use their own diff path
+and are not affected by the diff process.
+
+If no hunk lines precede the flush, followed by "success", Git
+treats the files as having no changes: `git diff` produces no output
+and `git blame` skips the commit, attributing lines to earlier commits.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool returns invalid hunks (out of bounds, overlapping, or
+mismatched unchanged line counts), Git warns and falls back to the
+builtin diff algorithm.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status.  Git warns and
+falls back to the builtin diff algorithm for this file.  The tool
+remains available for subsequent files.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=error
+packet:          git< 0000
+-----------------------
+
+In case the tool cannot or does not want to process the content as
+well as any future content for the lifetime of the Git process, it
+is expected to respond with an "abort" status.  Git silently falls
+back to the builtin diff algorithm for this file and does not send
+further requests to the tool.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=abort
+packet:          git< 0000
+-----------------------
+
+If the tool dies during the communication or does not adhere to the
+protocol then Git will stop the process and fall back to the builtin
+diff algorithm.  Git warns once and does not restart the process for
+subsequent files.
+
+Tools should ignore unknown keys in the per-file request to remain
+forward-compatible.  Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index 0976a69b4c..5de4718b24 100644
--- a/Makefile
+++ b/Makefile
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o
 TEST_BUILTINS_OBJS += test-date.o
 TEST_BUILTINS_OBJS += test-delete-gpgsig.o
 TEST_BUILTINS_OBJS += test-delta.o
+TEST_BUILTINS_OBJS += test-diff-process-backend.o
 TEST_BUILTINS_OBJS += test-dir-iterator.o
 TEST_BUILTINS_OBJS += test-drop-caches.o
 TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..fec63cf233
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,297 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  The tool controls which lines are marked as changed
+ * while the display shows the file content (after any textconv
+ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * When the tool returns no hunks with status=success, it considers
+ * the files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *get_or_launch_process(
+		struct userdiff_driver *drv)
+{
+	struct diff_subprocess *entry;
+
+	if (drv->diff_subprocess)
+		return drv->diff_subprocess;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start_command(&entry->subprocess, drv->process,
+				     start_diff_process_fn)) {
+		free(entry);
+		drv->diff_process_failed = 1;
+		return NULL;
+	}
+
+	drv->diff_subprocess = entry;
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret = 0;
+
+	if (size < 0)
+		return -1;
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
+{
+	char *end;
+
+	/*
+	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
+	 * All numbers must be non-negative decimal with no leading
+	 * whitespace or sign characters.
+	 */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->old_count = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_start = strtol(line, &end, 10);
+	if (errno || end == line || *end++ != ' ')
+		return -1;
+	line = end;
+
+	if (!isdigit(*line))
+		return -1;
+	errno = 0;
+	hunk->new_count = strtol(line, &end, 10);
+	if (errno || end == line || *end != '\0')
+		return -1;
+
+	/*
+	 * git diff emits start=0 when count=0 (empty file side).
+	 * Normalize to 1-based so downstream validation can assume start >= 1.
+	 */
+	if (!hunk->old_count && !hunk->old_start)
+		hunk->old_start = 1;
+	if (!hunk->new_count && !hunk->new_start)
+		hunk->new_start = 1;
+
+	return 0;
+}
+
+static enum diff_process_result get_hunks(
+		struct userdiff_driver *drv,
+		const char *path,
+		const char *old_buf, long old_size,
+		const char *new_buf, long new_size,
+		struct xdl_hunk **hunks_out,
+		size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	int len;
+	char *line;
+
+	backend = get_or_launch_process(drv);
+	if (!backend)
+		return DIFF_PROCESS_ERROR;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_flush_gently(fd_in))
+		goto comm_error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto comm_error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto comm_error;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &hunk) < 0)
+			goto comm_error;
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto comm_error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		*hunks_out = hunks;
+		*nr_hunks_out = nr_hunks;
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_OK;
+	}
+
+	if (!strcmp(status.buf, "abort")) {
+		/*
+		 * The tool voluntarily withdrew: stop sending requests
+		 * but do not warn (this is not a failure).
+		 */
+		backend->supported_capabilities &= ~CAP_HUNKS;
+		free(hunks);
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_SKIP;
+	}
+
+	/* status=error or unknown status */
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+
+comm_error:
+	/*
+	 * Communication failure (broken pipe, malformed response).
+	 * Tear down the process and mark as failed so we do not
+	 * retry on every subsequent file.
+	 */
+	drv->diff_process_failed = 1;
+	drv->diff_subprocess = NULL;
+	subprocess_stop_command(&backend->subprocess);
+	free(backend);
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+	struct xdl_hunk *ext_hunks = NULL;
+	size_t nr = 0;
+	enum diff_process_result res;
+
+	if (!diffopt || !path)
+		return DIFF_PROCESS_SKIP;
+	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+		return DIFF_PROCESS_SKIP;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return DIFF_PROCESS_SKIP;
+	if (drv->diff_process_failed)
+		return DIFF_PROCESS_SKIP;
+
+	res = get_hunks(drv, path,
+			file_a->ptr, file_a->size,
+			file_b->ptr, file_b->size,
+			&ext_hunks, &nr);
+	if (res == DIFF_PROCESS_OK) {
+		if (!nr) {
+			free(ext_hunks);
+			return DIFF_PROCESS_EQUIVALENT;
+		}
+		xpp->external_hunks = ext_hunks;
+		xpp->external_hunks_nr = nr;
+		return DIFF_PROCESS_OK;
+	}
+	if (res == DIFF_PROCESS_ERROR) {
+		warning(_("diff process '%s' failed for '%s',"
+			  " falling back to builtin diff"),
+			drv->process, path);
+		return DIFF_PROCESS_ERROR;
+	}
+	return DIFF_PROCESS_SKIP;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..d34b42f811
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,39 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+#include "xdiff/xdiff.h"
+
+struct diff_options;
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
+	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
+	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
+/*
+ * Consult the diff process configured for 'path' and populate
+ * xpp->external_hunks with the returned hunks.
+ *
+ * Handles driver lookup, flag checks (--no-ext-diff,
+ * --diff-algorithm), subprocess management, and error reporting.
+ *
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
+ * (files are considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
+ */
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 5a584fa1d5..3d97a188b9 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -4054,6 +4055,17 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		if (diff_process_fill_hunks(o, name_a,
+					    &mf1, &mf2, &xpp)
+		    == DIFF_PROCESS_EQUIVALENT) {
+			if (textconv_one)
+				free(mf1.ptr);
+			if (textconv_two)
+				free(mf2.ptr);
+			goto free_ab_and_return;
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4134,6 +4146,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(xpp.external_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/diff.h b/diff.h
index bb5cddaf34..7dc157968d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/** Disables diff.<driver>.process. */
+	unsigned no_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 3247697f74..aa532f5200 100644
--- a/meson.build
+++ b/meson.build
@@ -328,6 +328,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 3235f10ab8..6abcda4afb 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -12,6 +12,7 @@ test_tool_sources = [
   'test-date.c',
   'test-delete-gpgsig.c',
   'test-delta.c',
+  'test-diff-process-backend.c',
   'test-dir-iterator.c',
   'test-drop-caches.c',
   'test-dump-cache-tree.c',
diff --git a/t/helper/test-diff-process-backend.c b/t/helper/test-diff-process-backend.c
new file mode 100644
index 0000000000..ad392694e6
--- /dev/null
+++ b/t/helper/test-diff-process-backend.c
@@ -0,0 +1,299 @@
+/*
+ * Test backend for the long-running diff process protocol
+ * (see diff-process.c and Documentation/gitattributes.adoc).
+ *
+ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
+ *
+ * Implements the server side of the pkt-line handshake and a per-file
+ * response loop.  The --mode= switch selects the response shape
+ * (success, error, abort, crash, malformed hunks).
+ *
+ * Per-file request from Git:
+ *
+ *   packet:          git> command=hunks
+ *   packet:          git> pathname=<path>
+ *   packet:          git> 0000
+ *   packet:          git> OLD_CONTENT
+ *   packet:          git> 0000
+ *   packet:          git> NEW_CONTENT
+ *   packet:          git> 0000
+ *
+ * Response varies by --mode (default: whole-file):
+ *
+ *   whole-file   packet: git< hunk 1 <old_lines> 1 <new_lines>
+ *   fixed-hunk   packet: git< hunk 5 2 5 2
+ *   no-hunks     (no hunk packets)
+ *   bad-hunk     packet: git< hunk 999 1 999 1
+ *   bad-parse    packet: git< garbage not a hunk
+ *   bad-sync     packet: git< hunk 1 2 1 1
+ *   overlap      packet: git< hunk 1 5 1 5
+ *                packet: git< hunk 3 2 3 2
+ *   no-cap       (omits capability=hunks during handshake)
+ *   error        (status=error instead of status=success)
+ *   abort        (status=abort instead of status=success)
+ *   crash        exit(1) before sending any response
+ *
+ * All non-error/abort modes end with:
+ *
+ *   packet:          git< 0000
+ *   packet:          git< status=success
+ *   packet:          git< 0000
+ *
+ * Each request is logged to --log as:
+ *
+ *   command=<cmd> pathname=<path> old=<first line> new=<first line>
+ */
+
+#include "test-tool.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "strbuf.h"
+
+static FILE *logfile;
+
+enum mode {
+	MODE_WHOLE_FILE,
+	MODE_FIXED_HUNK,
+	MODE_NO_HUNKS,
+	MODE_BAD_HUNK,
+	MODE_BAD_PARSE,
+	MODE_BAD_SYNC,
+	MODE_OVERLAP,
+	MODE_NO_CAP,
+	MODE_ERROR,
+	MODE_ABORT,
+	MODE_CRASH,
+};
+
+static enum mode parse_mode(const char *s)
+{
+	if (!strcmp(s, "whole-file"))
+		return MODE_WHOLE_FILE;
+	if (!strcmp(s, "fixed-hunk"))
+		return MODE_FIXED_HUNK;
+	if (!strcmp(s, "no-hunks"))
+		return MODE_NO_HUNKS;
+	if (!strcmp(s, "bad-hunk"))
+		return MODE_BAD_HUNK;
+	if (!strcmp(s, "bad-parse"))
+		return MODE_BAD_PARSE;
+	if (!strcmp(s, "bad-sync"))
+		return MODE_BAD_SYNC;
+	if (!strcmp(s, "overlap"))
+		return MODE_OVERLAP;
+	if (!strcmp(s, "no-cap"))
+		return MODE_NO_CAP;
+	if (!strcmp(s, "error"))
+		return MODE_ERROR;
+	if (!strcmp(s, "abort"))
+		return MODE_ABORT;
+	if (!strcmp(s, "crash"))
+		return MODE_CRASH;
+	die("unknown --mode=%s", s);
+}
+
+/*
+ * Read "key=value" packets up to a flush, capturing "command" and
+ * "pathname".  Returns 1 if a request was read, 0 on EOF.
+ *
+ * The first packet uses the gentle variant so that a clean shutdown
+ * by Git (EOF) does not produce a spurious "the remote end hung up
+ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
+ * variant: once inside a request, truncation is a protocol violation
+ * and dying loudly is the correct response.
+ */
+static int read_request_header(char **command, char **pathname)
+{
+	int first = 1;
+	char *line;
+
+	*command = *pathname = NULL;
+	for (;;) {
+		const char *value;
+
+		if (first) {
+			if (packet_read_line_gently(0, NULL, &line) < 0)
+				return 0;
+			first = 0;
+		} else {
+			line = packet_read_line(0, NULL);
+		}
+		if (!line)
+			break;
+		if (skip_prefix(line, "command=", &value))
+			*command = xstrdup(value);
+		else if (skip_prefix(line, "pathname=", &value))
+			*pathname = xstrdup(value);
+	}
+	return 1;
+}
+
+static size_t count_lines(const struct strbuf *buf)
+{
+	size_t lines = 0;
+
+	for (size_t i = 0; i < buf->len; i++)
+		if (buf->buf[i] == '\n')
+			lines++;
+
+	return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
+}
+
+static void send_status(const char *status)
+{
+	packet_flush(1);
+	packet_write_fmt(1, "%s\n", status);
+	packet_flush(1);
+}
+
+static void respond(enum mode mode,
+		    const struct strbuf *old_buf,
+		    const struct strbuf *new_buf)
+{
+	switch (mode) {
+	case MODE_ERROR:
+		send_status("status=error");
+		return;
+	case MODE_ABORT:
+		send_status("status=abort");
+		return;
+	case MODE_CRASH:
+		exit(1);
+	case MODE_FIXED_HUNK:
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		break;
+	case MODE_BAD_HUNK:
+		packet_write_fmt(1, "hunk 999 1 999 1\n");
+		break;
+	case MODE_BAD_PARSE:
+		packet_write_fmt(1, "garbage not a hunk\n");
+		break;
+	case MODE_BAD_SYNC:
+		packet_write_fmt(1, "hunk 1 2 1 1\n");
+		break;
+	case MODE_OVERLAP:
+		packet_write_fmt(1, "hunk 1 5 1 5\n");
+		packet_write_fmt(1, "hunk 3 2 3 2\n");
+		break;
+	case MODE_NO_HUNKS:
+		break;
+	case MODE_NO_CAP:
+	case MODE_WHOLE_FILE: {
+		size_t old_lines = count_lines(old_buf);
+		size_t new_lines = count_lines(new_buf);
+		/*
+		 * Match git diff output: start=0 when count=0
+		 * (empty file side), 1 otherwise.
+		 */
+		packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
+				 " %"PRIuMAX" %"PRIuMAX"\n",
+				 (uintmax_t)(old_lines ? 1 : 0),
+				 (uintmax_t)old_lines,
+				 (uintmax_t)(new_lines ? 1 : 0),
+				 (uintmax_t)new_lines);
+		break;
+	}
+	}
+	send_status("status=success");
+}
+
+static void command_loop(enum mode mode)
+{
+	for (;;) {
+		char *command = NULL, *pathname = NULL;
+		struct strbuf obuf = STRBUF_INIT;
+		struct strbuf nbuf = STRBUF_INIT;
+
+		if (!read_request_header(&command, &pathname))
+			break; /* EOF: Git closed its end */
+
+		read_packetized_to_strbuf(0, &obuf, 0);
+		read_packetized_to_strbuf(0, &nbuf, 0);
+
+		if (logfile) {
+			fprintf(logfile,
+				"command=%s pathname=%s old=%.*s new=%.*s\n",
+				command ? command : "(none)",
+				pathname ? pathname : "(none)",
+				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
+				obuf.buf,
+				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
+				nbuf.buf);
+			fflush(logfile);
+		}
+
+		respond(mode, &obuf, &nbuf);
+
+		free(command);
+		free(pathname);
+		strbuf_release(&obuf);
+		strbuf_release(&nbuf);
+	}
+}
+
+static void handshake(enum mode mode)
+{
+	char *line;
+
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "git-diff-client"))
+		die("bad welcome: '%s'", line ? line : "(eof)");
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "version=1"))
+		die("bad version: '%s'", line ? line : "(eof)");
+	if (packet_read_line(0, NULL))
+		die("expected flush after version");
+
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Drain capabilities advertised by Git */
+	while ((line = packet_read_line(0, NULL)))
+		; /* drain */
+
+	/* Respond with our capabilities (or none for no-cap mode) */
+	if (mode != MODE_NO_CAP)
+		packet_write_fmt(1, "capability=hunks\n");
+	packet_flush(1);
+}
+
+static const char *const usage_str[] = {
+	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
+	NULL
+};
+
+int cmd__diff_process_backend(int argc, const char **argv)
+{
+	const char *mode_str = NULL, *log_path = NULL;
+	enum mode mode = MODE_WHOLE_FILE;
+	struct option options[] = {
+		OPT_STRING(0, "mode", &mode_str, "mode",
+			   "response shape: whole-file (default), fixed-hunk,"
+			   " no-hunks, bad-hunk, bad-sync, overlap, error,"
+			   " abort, crash"),
+		OPT_STRING(0, "log", &log_path, "path",
+			   "append per-request summary to this file"),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
+	if (argc)
+		usage_with_options(usage_str, options);
+
+	if (mode_str)
+		mode = parse_mode(mode_str);
+
+	if (log_path) {
+		logfile = fopen(log_path, "a");
+		if (!logfile)
+			die_errno("failed to open log '%s'", log_path);
+	}
+
+	handshake(mode);
+	command_loop(mode);
+
+	if (logfile && fclose(logfile))
+		die_errno("error closing log");
+	return 0;
+}
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index b71a22b43b..3c3f95269c 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = {
 	{ "date", cmd__date },
 	{ "delete-gpgsig", cmd__delete_gpgsig },
 	{ "delta", cmd__delta },
+	{ "diff-process-backend", cmd__diff_process_backend },
 	{ "dir-iterator", cmd__dir_iterator },
 	{ "drop-caches", cmd__drop_caches },
 	{ "dump-cache-tree", cmd__dump_cache_tree },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f2885b33d5..a5bb755516 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);
 int cmd__date(int argc, const char **argv);
 int cmd__delta(int argc, const char **argv);
 int cmd__delete_gpgsig(int argc, const char **argv);
+int cmd__diff_process_backend(int argc, const char **argv);
 int cmd__dir_iterator(int argc, const char **argv);
 int cmd__drop_caches(int argc, const char **argv);
 int cmd__dump_cache_tree(int argc, const char **argv);
diff --git a/t/meson.build b/t/meson.build
index c5832fee05..027855ced7 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -512,6 +512,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..9bb579b564
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,432 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+. ./test-lib.sh
+
+# See t/helper/test-diff-process-backend.c for the backend implementation
+# and available --mode= options.
+
+BACKEND="test-tool diff-process-backend"
+
+test_expect_success 'setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# boundary.c: 10 lines, changes at 5-6 and 9-10.
+	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+
+	# worddiff.c: single-line function, value changes 1 -> 999.
+	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+
+	# newfile.c: single-line function, value changes 42 -> 99.
+	# Used by: modified file, --exit-code, multiple drivers.
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 42; }
+	EOF
+	git add newfile.c &&
+
+	# logtest.c: single-line function for log/format-patch tests.
+	# Needs two commits so log -1 has a diff.
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+
+	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
+	cat >one.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >two.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add one.c two.c &&
+
+	git commit -m "initial" &&
+
+	# Second commit for logtest.c (so log -1 has something to show).
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	# Working tree modifications (not committed).
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 99; }
+	EOF
+
+	cat >one.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+
+	cat >two.c <<-\EOF
+	int second(void) { return 20; }
+	EOF
+'
+
+#
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
+test_expect_success 'diff process hunk boundaries affect output' '
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	test_grep ! "^-OLD9" actual &&
+	test_grep ! "^-OLD10" actual &&
+	test_grep ! "^+NEW9" actual &&
+	test_grep ! "^+NEW10" actual
+'
+
+test_expect_success 'diff process works with modified file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- newfile.c >actual 2>stderr &&
+	test_grep "return 99" actual &&
+	test_grep "pathname=newfile.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with added file (empty old side)' '
+	cat >added.c <<-\EOF &&
+	int added(void) { return 1; }
+	EOF
+	git add added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "added" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with deleted file (empty new side)' '
+	git add added.c &&
+	git commit -m "commit added.c" &&
+	git rm added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "deleted file" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process skipped for binary files' '
+	printf "\\0binary" >binary.c &&
+	git add binary.c &&
+	git commit -m "add binary" &&
+	printf "\\0changed" >binary.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- binary.c >actual &&
+	test_grep "Binary files" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not consulted for unmatched driver' '
+	echo "not tracked by cdiff" >unmatched.txt &&
+	git add unmatched.txt &&
+	git commit -m "add unmatched.txt" &&
+
+	echo "modified" >unmatched.txt &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- unmatched.txt >actual &&
+	test_grep "modified" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'multiple drivers use separate processes' '
+	echo "*.h diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 1; }
+	EOF
+	git add multi.h &&
+	git commit -m "add multi.h" &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 2; }
+	EOF
+
+	test_when_finished "rm -f backend-c.log backend-h.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+		diff -- newfile.c multi.h >actual 2>stderr &&
+	test_grep "pathname=newfile.c" backend-c.log &&
+	test_grep "pathname=multi.h" backend-h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works alongside textconv' '
+	write_script uppercase-filter <<-\EOF &&
+	tr "a-z" "A-Z" <"$1"
+	EOF
+
+	cat >textconv.c <<-\EOF &&
+	hello world
+	EOF
+	git add textconv.c &&
+	git commit -m "add textconv.c" &&
+
+	cat >textconv.c <<-\EOF &&
+	goodbye world
+	EOF
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- textconv.c >actual 2>stderr &&
+	# The diff process receives textconv-transformed (uppercase) content.
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=HELLO WORLD" backend.log &&
+	test_grep "new=GOODBYE WORLD" backend.log &&
+	test_must_be_empty stderr
+'
+
+#
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
+test_expect_success 'diff process with --word-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --word-diff worddiff.c >actual 2>stderr &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual &&
+	test_grep "pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with git log -p' '
+	# With no-hunks mode, the tool says the files are equivalent,
+	# so log -p should show the commit but no diff content.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -1 -p -- logtest.c >actual 2>stderr &&
+	test_grep "change logtest.c" actual &&
+	test_grep ! "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process no hunks suppresses diff output' '
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add nohunks.c &&
+	git commit -m "add nohunks.c" &&
+
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff nohunks.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success 'diff process no hunks with --exit-code returns success' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code nohunks.c
+'
+
+test_expect_success 'diff process with --exit-code and hunks returns failure' '
+	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+		diff --exit-code newfile.c
+'
+
+#
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
+test_expect_success 'diff process bypassed by --diff-algorithm' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not used by --stat' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --stat worddiff.c >actual &&
+	test_grep "worddiff.c" actual &&
+	test_path_is_missing backend.log
+'
+
+#
+# Error handling and fallback.
+#
+
+test_expect_success 'diff process fallback on tool error status' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process error keeps tool available for next file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Unlike abort, error keeps the tool available: both files
+	# are sent to the tool (and both fall back).
+	test_grep "pathname=one.c" backend.log &&
+	test_grep "pathname=two.c" backend.log &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process abort disables for session' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool.
+	test_grep "pathname=one.c" backend.log &&
+	test_grep ! "pathname=two.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Crash is a communication failure, so a warning is emitted.
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process startup failure only warns once' '
+	git -c diff.cdiff.process="/nonexistent/tool" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# Sentinel prevents repeated warnings: only one, not one per file.
+	test_grep "diff process.*failed" stderr >warnings &&
+	test_line_count = 1 warnings
+'
+
+
+test_expect_success 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "exceeds.*lines" stderr
+'
+
+test_expect_success 'diff process fallback on mismatched unchanged totals' '
+	cat >synctest.c <<-\EOF &&
+	line1
+	line2
+	line3
+	EOF
+	git add synctest.c &&
+	git commit -m "add synctest.c" &&
+
+	cat >synctest.c <<-\EOF &&
+	line1
+	changed
+	line3
+	EOF
+
+	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
+	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
+	# The synchronization invariant fails and git falls back.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+		diff synctest.c >actual 2>stderr &&
+	test_grep "changed" actual &&
+	test_grep "unchanged line count mismatch" stderr
+'
+
+test_expect_success 'diff process fallback on overlapping hunks' '
+	# boundary.c has 10 lines, so both hunks are in bounds
+	# but they overlap at lines 3-5, triggering the ordering check.
+	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "NEW5" actual &&
+	test_grep "overlaps with previous" stderr
+'
+
+test_expect_success 'diff process fallback on malformed hunk line' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual
+'
+
+test_expect_success 'diff process skipped when tool omits capability' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_must_be_empty stderr
+'
+
+test_done
diff --git a/userdiff.h b/userdiff.h
index 51c26e0d41..a98eabe377 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -3,6 +3,7 @@
 
 #include "notes-cache.h"
 
+struct diff_subprocess;
 struct index_state;
 struct repository;
 
@@ -33,6 +34,8 @@ struct userdiff_driver {
 	int textconv_want_cache;
 	const char *process;
 	char *process_owned;
+	struct diff_subprocess *diff_subprocess;
+	unsigned diff_process_failed : 1;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v4 5/6] diff: bypass diff process with --no-ext-diff and in format-patch

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Make --no-ext-diff disable diff.<driver>.process in addition to
diff.<driver>.command.  Although the two mechanisms work differently
(command replaces Git's output, process feeds hunks back into the
pipeline), both invoke external tools and --no-ext-diff means
"no external tools."

Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that
sets both allow_external and no_diff_process, so a single option
controls both.  Passing --ext-diff explicitly clears
no_diff_process, so a later --ext-diff overrides an earlier
--no-ext-diff.

Disable the diff process unconditionally in format-patch so that
generated patches are always based on the builtin diff algorithm
and can be applied reliably by recipients who do not have the
external tool.

Document that --diff-algorithm also bypasses the diff process,
since it forces the builtin algorithm.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/diff-algorithm-option.adoc |  3 +++
 Documentation/diff-options.adoc          |  4 +++-
 builtin/log.c                            |  7 +++++++
 diff.c                                   | 16 ++++++++++++++--
 diff.h                                   |  4 +++-
 t/t4080-diff-process.sh                  | 16 ++++++++++++++++
 6 files changed, 46 insertions(+), 4 deletions(-)

diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..4d7e2ec35f 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index c8242e2462..a884445211 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -833,7 +833,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::
diff --git a/builtin/log.c b/builtin/log.c
index e464b30af4..363052f468 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * Disable diff.<driver>.process so that patches generated by
+	 * format-patch are always based on the builtin diff algorithm
+	 * and can be applied reliably.
+	 */
+	rev.diffopt.flags.no_diff_process = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff.c b/diff.c
index 3d97a188b9..4d9cb9b26b 100644
--- a/diff.c
+++ b/diff.c
@@ -5936,6 +5936,17 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.no_diff_process = unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6266,8 +6277,9 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index 7dc157968d..bc7da6986a 100644
--- a/diff.h
+++ b/diff.h
@@ -173,7 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
-	/** Disables diff.<driver>.process. */
+	/**
+	 * Disables diff.<driver>.process.  Set by --no-ext-diff.
+	 */
 	unsigned no_diff_process;
 
 	/**
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 9bb579b564..df4d08e31f 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -295,6 +295,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' '
 	test_path_is_missing backend.log
 '
 
+test_expect_success 'diff process bypassed by --no-ext-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --no-ext-diff worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not used by format-patch' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		format-patch -1 --stdout -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_path_is_missing backend.log
+'
+
 test_expect_success 'diff process not used by --stat' '
 	test_when_finished "rm -f backend.log" &&
 	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
-- 
gitgitgadget

[PATCH v4 6/6] blame: consult diff process for no-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

The consultation happens at the pass_blame_to_parent() callsite
using diff_process_fill_hunks(), matching how builtin_diff() in
diff.c uses the same function.  A new diff_hunks_xpp() variant
accepts a pre-populated xpparam_t so callers can pass external
hunks, while the existing diff_hunks() retains its original
signature and behavior.  The copy-detection callsite is
unaffected since it does not use the diff process.

The subprocess is long-running (one startup cost amortized
across the blame traversal), but each commit in the file's
history incurs a round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                 |  40 +++++++++++----
 t/t4080-diff-process.sh | 105 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 136 insertions(+), 9 deletions(-)

diff --git a/blame.c b/blame.c
index 977cbb7097..354e6c15f4 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "xdiff-interface.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -314,17 +316,25 @@ static struct commit *fake_working_tree_commit(struct repository *r,
 
 
 
-static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
-		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
+			  xdl_emit_hunk_consume_func_t hunk_func,
+			  void *cb_data, xpparam_t *xpp)
 {
-	xpparam_t xpp = {0};
 	xdemitconf_t xecfg = {0};
 	xdemitcb_t ecb = {NULL};
 
-	xpp.flags = xdl_opts;
 	xecfg.hunk_func = hunk_func;
 	ecb.priv = cb_data;
-	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
+	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
+}
+
+static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
+{
+	xpparam_t xpp = {0};
+
+	xpp.flags = xdl_opts;
+	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
 }
 
 static const char *get_next_line(const char *start, const char *end)
@@ -1943,6 +1953,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *parent, int ignore_diffs)
 {
 	mmfile_t file_p, file_o;
+	xpparam_t xpp = {0};
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
 
@@ -1961,10 +1972,21 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
-		die("unable to generate diff (%s -> %s)",
-		    oid_to_hex(&parent->commit->object.oid),
-		    oid_to_hex(&target->commit->object.oid));
+	xpp.flags = sb->xdl_opts;
+	/*
+	 * If the diff process considers the files equivalent,
+	 * skip the diff so blame looks past this commit.
+	 */
+	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
+				    &file_p, &file_o, &xpp)
+	    != DIFF_PROCESS_EQUIVALENT) {
+		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
+				   &d, &xpp))
+			die("unable to generate diff (%s -> %s)",
+			    oid_to_hex(&parent->commit->object.oid),
+			    oid_to_hex(&target->commit->object.oid));
+	}
+	free(xpp.external_hunks);
 	/* The rest are the same as the parent */
 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
 		    parent, target, 0);
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index df4d08e31f..9fc3c01eec 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -445,4 +445,109 @@ test_expect_success 'diff process skipped when tool omits capability' '
 	test_must_be_empty stderr
 '
 
+#
+# Blame integration.
+#
+
+test_expect_success 'blame uses tool-provided hunks' '
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "add blame-hunk.c" &&
+	ORIG=$(git rev-parse --short HEAD) &&
+
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "change blame-hunk.c" &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+
+	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
+	# so blame should attribute lines 9-10 to the original commit
+	# even though the builtin diff would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		blame blame-hunk.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	sed -n "6p" actual >line6 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "$CHANGE" line6
+'
+
+test_expect_success 'blame skips commits with no hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void) {
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without no-hunks mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With no-hunks mode, the process considers the files equivalent
+	# and blame skips the reformat commit, attributing to the original.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		blame blame.c >with &&
+	test_grep ! "$BLAME_COMMIT" with &&
+	test_grep "$ORIG_COMMIT" with
+'
+
+test_expect_success 'blame --no-ext-diff bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		blame --no-ext-diff blame.c >actual &&
+	# Without the process, blame attributes the reformat commit normally.
+	test_grep "$BLAME_COMMIT" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame --no-ext-diff uses builtin hunks' '
+	# fixed-hunk mode would narrow blame to lines 5-6, but
+	# --no-ext-diff should bypass it and use the builtin diff.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		blame --no-ext-diff blame-hunk.c >actual &&
+	# Builtin diff attributes lines 9-10 to the change commit.
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
 test_done
-- 
gitgitgadget

Re: [PREVIEW v4 0/6] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo]

On Sun, Jun 14, 2026 at 11:55 AM Michael Montalbo via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> This series adds diff.<driver>.process, a long-running subprocess protocol
> that lets an external tool control which lines Git considers changed while
> Git handles all output formatting...

Now I'm realizing there are some diff flags that are not properly supported
yet with an external diff provider. Flags like -L and the stat-related
flags should probably behave like blame and consult the external diff
process. In general, there should be a more explicit and comprehensive
explanation for which flags interact with external diff processes and
which do not, included principled reasons for why that is the case.

[PATCH v5 0/9] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's diff output with the program's own
output, so display features like word diff, function context, and color
cannot operate on it; and because the program is consulted only for that
patch output, blame, --stat, and git log -L fall back to Git's builtin line
matching and cannot benefit from the tool at all.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes, which propagates through
patch output, the --stat summary, blame, and git log -L. The request also
carries the two blobs' object names (old-oid/new-oid) so a tool can cache
its analysis keyed on the pair.

 * Patch 1: document how an external diff driver (diff.<driver>.command)
   relates to the rest of Git's diff features, so the contrast with the new
   process driver is clear.
 * Patch 2: xdiff plumbing for externally supplied hunks.
 * Patch 3: diff.<driver>.process config key.
 * Patch 4: refactor subprocess API to separate process lifecycle from
   hashmap management, since the diff process stores its subprocess on the
   userdiff driver rather than in a hashmap.
 * Patch 5: the main feature, including the old-oid/new-oid request metadata
   for blob-pair caching.
 * Patch 6: bypass knobs (--no-ext-diff, format-patch).
 * Patch 7: blame integration so the tool can declare commits as having no
   changes; introduces the shared xdi_diff_process() consult-then-diff
   helper that blame and git log -L both use.
 * Patch 8: --stat/--numstat/--shortstat consult the tool, so the summary
   agrees with the patch output.
 * Patch 9: git log -L range tracking consults the tool, so a reformat-only
   commit is dropped from the log rather than shown with an empty diff.

A "Which features consult the diff process" section in gitattributes(5) lays
out, per feature, why each does or does not consult the process (patch
output, blame, summary formats, and the -L line-range view do; pickaxe -G,
patch-id, merge, range-diff, --check, and --raw do not, with reasons).
Combined diffs (--cc) remain on the builtin algorithm and are noted as
future work.

Changes since v4:

 * New preparatory doc patch (patch 1): document how an external diff driver
   (diff.<driver>.command) relates to the rest of Git's diff features, so
   the contrast with the process driver added later in the series is
   explicit.

 * New: --stat/--numstat/--shortstat now consult the process (patch 8). A
   file the tool reports as equivalent contributes no stat line, matching
   the empty patch git diff produces for it. Summary formats do not apply
   textconv, so the tool is fed raw content there, as the builtin --stat
   already counts raw lines.

 * New: git log -L range tracking now consults the process (patch 9).
   Previously the tracking pass used the builtin diff while the displayed
   diff used the tool, so a reformat-only commit could be selected by
   tracking and then rendered with an empty diff. Tracking now agrees with
   the display and the commit is dropped.

 * New: the request carries the old and new blob object names
   (old-oid/new-oid), so a tool can cache its line matching keyed on the
   blob pair. A side's oid is sent only when the tool receives that raw
   blob; it is omitted under textconv (which rewrites the bytes) and for a
   working-tree side with no stored object. This is where the process
   differs from diff.<name>.command, which never composes with textconv and
   so always feeds the raw blob its oid names.

 * Refactor: blame and git log -L now consult the process through a single
   xdi_diff_process() helper instead of open-coding the consult-then-diff
   dance; builtin_diff() keeps its own path so it can short-circuit
   equivalent files before its funcname/word-diff setup. The whitespace
   bypass keys off the effective diff parameters (xpp), which removes
   blame's separate -w guard, and blame's diff_hunks() sheds an intermediate
   variant it no longer needs.

 * Correctness: external-hunk validation now checks per-gap alignment, not
   just the total unchanged-line count. xdl_build_script() walks the two
   files in lockstep over unchanged lines, so a hunk set whose totals
   balance but whose per-gap line counts do not (e.g. hunk 1 1 3 1)
   previously passed validation and produced a corrupt diff, --stat, and
   blame attribution. Such responses are now rejected and Git falls back to
   the builtin diff.

 * Robustness: hunk accumulation is bounded by the two files' combined line
   count, so a misbehaving tool that floods hunk lines cannot grow memory
   without bound before validation.

 * Forward-compat: the hunk-line parser now ignores unknown trailing fields
   after the four counts, so a future protocol version can append a field
   without older clients rejecting it.

 * Feature interactions: the whitespace-ignoring options (-w,
   --ignore-space-change, --ignore-blank-lines, ...), -I, and --anchored
   bypass the process (the tool is never told about them and could not honor
   them), and git blame -w does the same. A change that only adds or removes
   the trailing newline cannot be expressed as line hunks, so it also falls
   back to the builtin diff (preserving the "\ No newline at end of file"
   marker).

 * Documentation: added the per-feature "Which features consult the diff
   process" section; documented that the process trusts the tool's notion of
   "unchanged" (it is not byte-validated, so like git diff -w such a patch
   may not apply against the old blob), that --exit-code/--quiet report
   success for tool- equivalent files, and that diff.<name>.command takes
   precedence when both it and .process are configured.

 * Tests: t4080 grew coverage for the per-gap check, the hunk- flood cap,
   the whitespace/-w bypasses, the trailing-newline and added/deleted-file
   fallbacks, multi-hunk output through patch and --stat, --stat
   --exit-code, a mode-only change, and a mixed equivalent/changed
   multi-file diffstat.

Michael Montalbo (9):
  gitattributes: document how external diff drivers relate to diff
    features
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection
  diff: consult diff process for --stat counts
  line-log: consult diff process for range tracking

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 274 +++++++
 Makefile                                 |   2 +
 blame.c                                  |  24 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 529 +++++++++++++
 diff-process.h                           |  75 ++
 diff.c                                   |  57 +-
 diff.h                                   |   6 +
 line-log.c                               |  33 +-
 meson.build                              |   1 +
 sub-process.c                            |  28 +-
 sub-process.h                            |   9 +-
 t/helper/meson.build                     |   1 +
 t/helper/test-diff-process-backend.c     | 381 +++++++++
 t/helper/test-tool.c                     |   1 +
 t/helper/test-tool.h                     |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 950 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  16 +
 xdiff/xdiffi.c                           |  84 +-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 28 files changed, 2504 insertions(+), 18 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh


base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v5
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v4:

  -:  ---------- >  1:  0fd994a3d3 gitattributes: document how external diff drivers relate to diff features
  1:  03f261dfe2 !  2:  2004502549 xdiff: support external hunks via xpparam_t
     @@ Commit message
          the diff algorithm, then continues through compaction and emission
          as usual.
      
     -    Validate supplied hunks before use: reject out-of-bounds line
     -    numbers, overlapping or out-of-order hunks, negative counts, and
     -    violations of the synchronization invariant (unchanged line counts
     -    must match between files).  On validation failure, fall back to
     -    the builtin diff algorithm; this re-runs xdl_prepare_env() since
     -    the first call may have dirtied the changed[] arrays.
     +    Validate supplied hunks before use.  Out-of-bounds line numbers,
     +    overlapping or out-of-order hunks, and misaligned unchanged runs are
     +    treated as a malformed tool response: xdl_populate_hunks_from_external()
     +    warns, returns -1, and xdl_diff() falls back to the builtin diff
     +    algorithm for that file.  The run of unchanged lines between two hunks
     +    (and before the first and after the last) must be the same length on
     +    both sides; xdl_build_script() walks the two files in lockstep over
     +    unchanged lines, so a balanced total is not enough.  Non-negative
     +    counts and 1-based starts are instead caller preconditions, checked
     +    with BUG(), since the caller normalizes hunks before this point.
     +
     +    On rejection xdl_diff() frees the environment it prepared and falls
     +    through to xdl_do_diff(), which prepares a fresh one for the builtin
     +    pass.
      
          Skip trim_common_tail() in xdi_diff() when external hunks are
          present, since external hunks reference line numbers in the
     @@ xdiff/xdiff.h: typedef struct s_mmbuffer {
       } mmbuffer_t;
       
      +/*
     -+ * Hunk descriptor for externally computed diffs.
     -+ * Line numbers are 1-based; a start of 0 is accepted when
     -+ * count is 0 (empty file side, matching git diff output).
     ++ * Hunk descriptor for externally computed diffs, in xdiff's own
     ++ * coordinates: line numbers are 1-based and a hunk's start is the
     ++ * first line it covers.  A caller translates any external "empty side"
     ++ * idiom (such as git diff's start-0/count-0) to a 1-based start before
     ++ * handing hunks over.
      + */
      +struct xdl_hunk {
      +	long old_start, old_count;
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
       
      +/*
      + * Populate the changed[] arrays from externally supplied hunks,
     -+ * bypassing the diff algorithm.  Validates that hunks are in order,
     -+ * non-overlapping, and within bounds.
     ++ * bypassing the diff algorithm.  The caller normalizes and validates
     ++ * the hunks first (order, overlap, and lockstep alignment), so this
     ++ * only marks lines changed after asserting the memory-safety
     ++ * preconditions it depends on: non-negative counts and 1-based starts
     ++ * (checked with BUG()), and an in-bounds range (a silent -1 so the
     ++ * caller can fall back to the builtin diff rather than index changed[]
     ++ * out of range).  Keeping this diagnostic-free leaves user-facing
     ++ * messages to the git layer.
      + *
     -+ * Returns 0 on success, -1 on validation failure.
     ++ * Returns 0 on success, -1 if a hunk is out of range.
      + */
      +static int xdl_populate_hunks_from_external(xdfenv_t *xe,
      +					    struct xdl_hunk *hunks,
      +					    size_t nr_hunks)
      +{
      +	size_t i;
     -+	long j, prev_old_end = 0, prev_new_end = 0;
     -+	long changed_old = 0, changed_new = 0;
     ++	long j;
      +
      +	/*
      +	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
     @@ xdiff/xdiffi.c: static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdf
      +	for (i = 0; i < nr_hunks; i++) {
      +		struct xdl_hunk *h = &hunks[i];
      +
     -+		if (h->old_count < 0 || h->new_count < 0) {
     -+			warning("diff process hunk %"PRIuMAX": "
     ++		/*
     ++		 * Non-negative counts and 1-based starts are caller
     ++		 * preconditions (it normalizes hunks into xdiff coordinates
     ++		 * before this point), so a violation is a bug, not a bad
     ++		 * tool response.
     ++		 */
     ++		if (h->old_count < 0 || h->new_count < 0)
     ++			BUG("external hunk %"PRIuMAX": "
      +				"negative count (old=%ld, new=%ld)",
      +				(uintmax_t)(i + 1),
      +				h->old_count, h->new_count);
     -+			return -1;
     -+		}
     -+		if (h->old_start < 1 || h->new_start < 1) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"start must be >= 1 (old=%ld, new=%ld)",
     ++		if (h->old_start < 1 || h->new_start < 1)
     ++			BUG("external hunk %"PRIuMAX": "
     ++				"start not 1-based (old=%ld, new=%ld)",
      +				(uintmax_t)(i + 1),
      +				h->old_start, h->new_start);
     -+			return -1;
     -+		}
      +
      +		/*
     -+		 * Range must fit: start + count - 1 <= nrec,
     -+		 * rewritten to avoid overflow.  Same for both sides.
     -+		 *
     -+		 * When count is 0 (pure insert/delete) the check
     -+		 * reduces to 0 > nrec - start + 1, which rejects
     -+		 * start > nrec + 1 and allows start == nrec + 1
     -+		 * (the position after the last line).
     ++		 * The caller validates ordering, overlap and lockstep
     ++		 * alignment (and diagnoses a bad response).  This is only a
     ++		 * silent in-bounds guard so the marking loop cannot index
     ++		 * changed[] out of range: start + count - 1 <= nrec,
     ++		 * rewritten to avoid overflow.  A count of 0 (pure
     ++		 * insert/delete) allows start == nrec + 1, the position
     ++		 * after the last line.  On a miss, return -1 and let the
     ++		 * caller fall back to the builtin diff.
      +		 */
     -+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"old range %ld+%ld exceeds %lu lines",
     -+				(uintmax_t)(i + 1),
     -+				h->old_start, h->old_count,
     -+				(unsigned long)xe->xdf1.nrec);
     -+			return -1;
     -+		}
     -+		if (h->new_count > (long)xe->xdf2.nrec - h->new_start + 1) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"new range %ld+%ld exceeds %lu lines",
     -+				(uintmax_t)(i + 1),
     -+				h->new_start, h->new_count,
     -+				(unsigned long)xe->xdf2.nrec);
     ++		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1 ||
     ++		    h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
      +			return -1;
     -+		}
     -+
     -+		/* Ordering: no overlap with previous hunk (adjacent is OK) */
     -+		if (h->old_start < prev_old_end ||
     -+		    h->new_start < prev_new_end) {
     -+			warning("diff process hunk %"PRIuMAX": "
     -+				"overlaps with previous hunk",
     -+				(uintmax_t)(i + 1));
     -+			return -1;
     -+		}
      +
      +		for (j = 0; j < h->old_count; j++)
      +			xe->xdf1.changed[h->old_start - 1 + j] = true;
      +		for (j = 0; j < h->new_count; j++)
      +			xe->xdf2.changed[h->new_start - 1 + j] = true;
     -+
     -+		prev_old_end = h->old_start + h->old_count;
     -+		prev_new_end = h->new_start + h->new_count;
     -+	}
     -+
     -+	/*
     -+	 * Synchronization invariant: unchanged line counts must match.
     -+	 * Otherwise xdl_build_script() would walk off one array.
     -+	 *
     -+	 * Count changed lines from the arrays rather than accumulating
     -+	 * during the loop to avoid any overflow in the summation.
     -+	 */
     -+	for (j = 0; j < (long)xe->xdf1.nrec; j++)
     -+		if (xe->xdf1.changed[j])
     -+			changed_old++;
     -+	for (j = 0; j < (long)xe->xdf2.nrec; j++)
     -+		if (xe->xdf2.changed[j])
     -+			changed_new++;
     -+	if ((long)xe->xdf1.nrec - changed_old !=
     -+	    (long)xe->xdf2.nrec - changed_new) {
     -+		warning("diff process: unchanged line count mismatch "
     -+			"(old: %ld unchanged, new: %ld unchanged)",
     -+			(long)xe->xdf1.nrec - changed_old,
     -+			(long)xe->xdf2.nrec - changed_new);
     -+		return -1;
      +	}
      +
      +	return 0;
  2:  30617ee17b =  3:  926cf01af6 userdiff: add diff.<driver>.process config
  3:  459e485e6d =  4:  363d459ff6 sub-process: separate process lifecycle from hashmap management
  4:  10b3980f59 !  5:  d003bc1f15 diff: add long-running diff process via diff.<driver>.process
     @@ Commit message
          textconv-transformed content.  The tool controls which lines
          are marked as changed while the display shows the file content.
          Patch output features (word diff, function context, color) work
     -    normally; --stat uses its own diff codepath and never consults
     -    the diff process.
     +    normally.  A new "Which features consult the diff process"
     +    documentation section lays out which features use the tool's hunks,
     +    which compute independently, and why; the summary formats such as
     +    --stat still use the builtin diff for now.
      
          The handshake negotiates version=1 and capability=hunks.  Per-file
     -    requests send command=hunks, pathname, and both file contents as
     -    packetized data.  The tool responds with hunk lines and a status
     -    packet (success, error, or abort).  On error, Git warns and falls
     -    back to the builtin diff algorithm for that file.  On abort, Git
     -    silently falls back for the current file and stops sending further
     -    requests to the tool for the remainder of the session.
     +    requests send command=hunks, pathname, the old and new blob object
     +    names as old-oid/new-oid, and both file contents as packetized data.
     +    The tool responds with hunk lines and a status packet (success,
     +    error, or abort).  On error, Git warns and falls back to the builtin
     +    diff algorithm for that file.  On abort, Git silently falls back for
     +    the current file and stops sending further requests to the tool for
     +    the remainder of the session.
     +
     +    old-oid/new-oid name the two blobs so a tool can cache its analysis
     +    keyed on the pair.  A side's oid is sent only when the content the
     +    tool receives is that raw blob: it is omitted under textconv, which
     +    rewrites the bytes, and for a working-tree side with no stored
     +    object, so an oid that is sent always names the bytes the tool
     +    receives.  This is where the process protocol diverges from
     +    diff.<driver>.command, which never composes with textconv (the
     +    command replaces the whole diff and always gets the raw blob).  Tools
     +    ignore unknown request keys, so old tools skip them.
      
          When the tool returns no hunks followed by status=success, Git
          treats the file as having no changes and produces no diff output.
     @@ Commit message
          the failure is cached on the driver to avoid retrying and
          re-warning on every subsequent file.
      
     +    Git falls back to the builtin diff (rather than consulting the
     +    tool) when an option the tool cannot honor is in effect: the
     +    whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and
     +    --anchored.  The bypass keys off the effective diff parameters (xpp)
     +    rather than diffopt, so a later caller whose flags live elsewhere is
     +    covered uniformly.  A change that only adds or removes the trailing
     +    newline is likewise not expressible as hunks, so it too uses the
     +    builtin diff.  The hunk parser ignores unknown trailing fields on a
     +    hunk line for response forward-compatibility.
     +
     +    Hunk accumulation is bounded by the combined byte count of the two
     +    files, so a misbehaving tool that floods hunk lines cannot grow
     +    memory without bound before validation runs.
     +
          diff_process_fill_hunks() is the sole public entry point.  It
          handles driver lookup, flag checks, subprocess management, and
          error reporting, returning an enum that lets callers distinguish
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +long-running process protocol (described in
      +Documentation/technical/long-running-process-protocol.adoc).
      +Unlike `diff.<name>.command`, which replaces Git's output entirely,
     -+the diff process feeds results back into the standard pipeline.
     ++the diff process feeds results back into the standard pipeline.  If
     ++both are configured for a path, `diff.<name>.command` takes precedence
     ++for the patch output it replaces; the summary formats, `git blame`,
     ++and `git log -L` never run the command and still consult the process.
      +
      +First, in `.gitattributes`, assign the `diff` attribute for paths.
      +
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +the process and performs the handshake.  In the handshake, the welcome
      +message sent by Git is "git-diff-client", only version 1 is supported,
      +and the supported capability is "hunks" (the changed regions
     -+described below).
     ++described below).  The tool replies with "git-diff-server", the
     ++version it supports, and the capabilities it supports.
      +
      +For each file, Git sends a list of "key=value" pairs terminated with
      +a flush packet, followed by the old and new file content as packetized
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +-----------------------
      +packet:          git> command=hunks
      +packet:          git> pathname=path/file.c
     ++packet:          git> old-oid=<hex>
     ++packet:          git> new-oid=<hex>
      +packet:          git> 0000
      +packet:          git> OLD_CONTENT
      +packet:          git> 0000
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +packet:          git> 0000
      +-----------------------
      +
     ++The optional `old-oid` and `new-oid` keys give the object names of the
     ++old and new blobs, so a tool can cache its analysis keyed on the pair.
     ++A side's key is sent only when the content for that side is the raw
     ++blob it names: it is omitted when the content is textconv-transformed,
     ++and for a working-tree side that has no stored object.  A tool that
     ++does not recognize these keys ignores them.
     ++
      +The tool is expected to respond with zero or more hunk lines,
      +a flush packet, and a status packet terminated with a flush packet.
      +Each hunk line has the form:
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +
      +where `<old_start>` and `<old_count>` identify a range of lines in
      +the old file, and `<new_start>` and `<new_count>` identify the
     -+replacement range in the new file.  Start values are 1-based and
     -+counts are non-negative.  Ranges must not extend beyond the end of
     -+the file.  For example, `hunk 3 2 3 4` means that 2 lines starting
     -+at line 3 in the old file were replaced by 4 lines starting at
     -+line 3 in the new file.  An `<old_count>` of 0 means no lines were
     -+removed (pure insertion); a `<new_count>` of 0 means no lines were
     -+added (pure deletion).  A start value of 0 is accepted when
     -+the corresponding count is 0 (e.g., `hunk 0 0 1 5` for a newly
     -+added file), matching what `git diff` itself emits for empty
     -+file sides.
     ++replacement range in the new file.  The four fields are separated by
     ++single spaces.  Start values are 1-based and counts are non-negative.
     ++For example, `hunk 3 2 3 4` means that 2 lines starting at line 3 in
     ++the old file were replaced by 4 lines starting at line 3 in the new
     ++file.  An `<old_count>` of 0 means no lines were removed (pure
     ++insertion); a `<new_count>` of 0 means no lines were added (pure
     ++deletion).  For a side with a count of 0 (a pure insertion or
     ++deletion) the start is the 1-based line the change sits before,
     ++ranging from 1 to one past the last line (the line count plus 1, to
     ++place the change at the end of the file); like every start it must
     ++keep the unchanged runs aligned on both sides (see below), so for a
     ++given change it takes one specific value, not an arbitrary one.  A
     ++start of 0 is also accepted and treated as 1, matching the
     ++empty-file-side form `git diff` emits (e.g. `hunk 0 0 1 5` for a newly
     ++added file).  A nonzero range must not extend beyond the end of the
     ++file.  Git ignores any extra
     ++whitespace-separated tokens after `<new_count>`, so a future protocol
     ++version can append fields to a hunk line (for example a "moved"
     ++marker) without older tools rejecting it.
      +
      +Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
      +file `"foo\nbar"` both have 2 lines.
      +
     -+Hunks must be listed in order and must not overlap.  Any line
     -+not covered by a hunk is treated as unchanged, so the total
     -+number of unchanged lines must be the same on both sides.
     -+For example, if the old file has 10 lines and the hunks cover
     -+4 of them (`old_count` values summing to 4), then 6 old lines
     -+are unchanged.  The new file must also have exactly 6 lines
     -+not covered by hunks, so the `new_count` values must sum to
     -+`new_file_lines - 6`.
     ++Hunks must be listed in order and must not overlap.  Any line not
     ++covered by a hunk is treated as unchanged and is paired, in order,
     ++with the unchanged lines on the other side.  Each run of unchanged
     ++lines between two hunks (and the run before the first hunk and
     ++after the last) must therefore be the same length on both sides,
     ++not merely equal in total.  For the hunks `1 3 1 5` and `10 2 12 2`
     ++below, lines 4-9 of the old file and lines 6-11 of the new file are
     ++both the six unchanged lines between the two hunks.  A response that
     ++balances only the total unchanged count but misaligns one of these
     ++runs is rejected, and Git falls back to the builtin diff.
     ++
     ++Git does not check that the lines a hunk leaves unchanged are
     ++byte-for-byte identical between the two sides; it pairs them by
     ++position and shows the new side as context.  A tool may therefore
     ++report lines that differ textually (a pure reformatting, say) as
     ++unchanged, and the diff reflects that judgment.  This is
     ++the point of a semantic backend, but it means a misbehaving tool can
     ++produce a diff whose context does not match the old blob; as with
     ++`git diff -w`, such a patch may not apply against the old content.
      +
      +-----------------------
      +packet:          git< hunk 1 3 1 5
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +-----------------------
      +
      +If the tool responds with hunks and "success", Git marks those lines
     -+as changed and feeds them into the standard diff pipeline.  Patch
     -+output features (word diff, function context, color) work normally.
     -+Note that `--stat` and other summary formats use their own diff path
     -+and are not affected by the diff process.
     ++as changed and feeds them into the standard diff pipeline.  Git may
     ++still slide or regroup those changes against matching context for
     ++display, exactly as it compacts its own diffs, so the tool controls
     ++which lines are reported as changed, not the precise hunk boundaries.
     ++Patch output features (word diff, function context, color) work
     ++normally.  Summary formats such as `--stat` still compute their counts
     ++with the builtin diff for now; see "Which features consult the diff
     ++process" below for the full picture and the reasoning behind it.
      +
      +If no hunk lines precede the flush, followed by "success", Git
     -+treats the files as having no changes: `git diff` produces no output
     -+and `git blame` skips the commit, attributing lines to earlier commits.
     ++treats the files as having no changes: `git diff` produces no output,
     ++`git diff --exit-code` and `--quiet` report success even though the
     ++stored blobs differ, and `git blame` skips the commit, attributing
     ++lines to earlier commits.
     ++The one exception is a change that only adds or removes the file's
     ++trailing newline: it cannot be expressed as line hunks, so when the
     ++line content otherwise matches Git keeps the builtin diff for that
     ++file (preserving the `\ No newline at end of file` marker) instead of
     ++treating the two sides as equal.
      +
      +-----------------------
      +packet:          git< 0000
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +packet:          git< 0000
      +-----------------------
      +
     -+If the tool returns invalid hunks (out of bounds, overlapping, or
     -+mismatched unchanged line counts), Git warns and falls back to the
     -+builtin diff algorithm.
     ++If the tool returns well-formed but invalid hunks (out of bounds,
     ++overlapping, or with misaligned unchanged runs), Git warns and falls
     ++back to the builtin diff for that file; the tool stays available for
     ++subsequent files.  A malformed hunk line, by contrast (bad syntax, a
     ++nonzero count paired with a start of 0, or more hunks than the file
     ++has lines), is a protocol violation: Git stops the process and does
     ++not send it further requests, as described below.
      +
      +In case the tool cannot or does not want to process the content,
      +it is expected to respond with an "error" status.  Git warns and
     -+falls back to the builtin diff algorithm for this file.  The tool
     ++falls back to the builtin diff algorithm for this file, treating any
     ++status other than "success" or "abort" the same way.  The tool
      +remains available for subsequent files.
      +
      +-----------------------
     @@ Documentation/gitattributes.adoc: NOTE: If `diff.<name>.command` is defined for
      +forward-compatible.  Future versions of Git may send additional
      +`command=` values; tools that receive an unrecognized command should
      +respond with `status=error` rather than terminating.
     ++
     ++Which features consult the diff process
     ++^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     ++
     ++The diff process answers a single question: given two blobs, which
     ++line ranges differ?  Whether a particular feature consults it follows
     ++from whether that is the question the feature is really asking.
     ++
     ++Features that ask "which lines changed" use the tool's hunks in place
     ++of the builtin algorithm:
     ++
     ++- `git diff` patch output, together with everything layered on it:
     ++  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
     ++  headers, and the `-L` line-range display.  These operate on the
     ++  lines the patch step already emitted, so they reflect the tool's
     ++  hunks without any further negotiation.
     ++- `git blame`: a commit whose change the tool reports as equivalent is
     ++  skipped, and its lines are attributed to an earlier commit.
     ++
     ++Features that ask a different question do not consult the process, by
     ++design:
     ++
     ++- The pickaxe `-G<regex>` searches the textual diff for a pattern; it
     ++  asks "does this string appear in the diff," not "did these lines
     ++  change."  (`-S` runs at an earlier stage and is likewise unaffected.)
     ++- `git patch-id` must produce a stable hash for `git rebase` and
     ++  cherry-pick detection; deriving it from a configured tool would make
     ++  equal patches hash differently from machine to machine.
     ++- The merge machinery (`git merge-tree`, `rerere`) computes merge
     ++  content and conflict signatures rather than display output, so the
     ++  tool's hunks must not alter its results.
     ++- `git range-diff` diffs patch text, not source blobs, so source-file
     ++  hunks do not apply to it.
     ++- `--check` reports whitespace errors in added lines using the builtin
     ++  diff's notion of which lines are added, not the tool's.  It can
     ++  therefore flag (and exit non-zero on) a line the tool treats as
     ++  unchanged and that `git diff` shows as context.  Whitespace breakage
     ++  is a property of the literal bytes, so `--check` keeps the builtin
     ++  partition deliberately; a future change could wire it to the tool if
     ++  matching `git diff` exactly became desirable.
     ++- `--raw`, `--name-only`, and `--name-status` compare object ids at
     ++  the tree level and never run a line-level diff at all.
     ++
     ++Some features ask "which lines changed" but still use the builtin
     ++algorithm for now, and may consult the process in a later change: the
     ++summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
     ++commit selection and parent range propagation (as distinct from its
     ++display, which is covered above); and combined diffs (`--cc` and merge
     ++diffs), whose protocol would have to be extended from a single old/new
     ++pair to one comparison per merge parent.
     ++
     ++`--diff-algorithm` bypasses the process entirely, for every feature
     ++listed above.  The whitespace-ignoring options (`-w`,
     ++`--ignore-space-change`, `--ignore-blank-lines`, and the like),
     ++`-I<regex>`, and `--anchored` also bypass it for the affected files:
     ++the tool is never told about these options, so it could not honor
     ++them, and Git falls back to the builtin diff, which does.
      +
       Defining a custom hunk-header
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     @@ diff-process.c (new)
      + *   tool< capability=hunks / flush
      + *
      + * Per-file:
     -+ *   git> command=hunks / pathname=<path> / flush
     ++ *   git> command=hunks / pathname=<path> / [old-oid=<hex>] / [new-oid=<hex>] / flush
      + *   git> <old content packetized> / flush
      + *   git> <new content packetized> / flush
      + *   tool< hunk <old_start> <old_count> <new_start> <new_count>
     @@ diff-process.c (new)
      +#include "diff-process.h"
      +#include "diff.h"
      +#include "gettext.h"
     ++#include "hex.h"
      +#include "repository.h"
      +#include "sigchain.h"
      +#include "userdiff.h"
     @@ diff-process.c (new)
      +	return packet_flush_gently(fd);
      +}
      +
     -+static int parse_hunk_line(const char *line, struct xdl_hunk *hunk)
     ++/*
     ++ * A hunk in the diff process's presentation coordinates: the line
     ++ * numbering it reports over the protocol.  Kept distinct from struct
     ++ * xdl_hunk (xdiff's coordinates) so that only translated hunks ever
     ++ * reach the diff algorithm; diff_process_hunk_to_xdl() is the single
     ++ * crossing point.
     ++ */
     ++struct diff_process_hunk {
     ++	long old_start, old_count;
     ++	long new_start, new_count;
     ++};
     ++
     ++/*
     ++ * Parse one non-negative decimal field of a hunk line into *out and
     ++ * advance *line past it.  Fields must be plain decimal with no leading
     ++ * whitespace or sign (isdigit() takes an unsigned char to stay defined
     ++ * for high-bit bytes).  The first three fields are followed by a single
     ++ * space; the last (is_last) is followed by end-of-string or a space.
     ++ * Trailing space-separated tokens after the last field are allowed and
     ++ * ignored, so a future protocol version can append fields (e.g. a
     ++ * "moved" marker) without older tools rejecting the line -- mirroring
     ++ * the request-side rule that tools ignore unknown keys.
     ++ */
     ++static int parse_hunk_field(const char **line, long *out, int is_last)
      +{
     ++	const char *p = *line;
      +	char *end;
      +
     -+	/*
     -+	 * Format: "hunk <old_start> <old_count> <new_start> <new_count>"
     -+	 * All numbers must be non-negative decimal with no leading
     -+	 * whitespace or sign characters.
     -+	 */
     -+	if (!skip_prefix(line, "hunk ", &line))
     -+		return -1;
     -+
     -+	if (!isdigit(*line))
     ++	if (!isdigit((unsigned char)*p))
      +		return -1;
      +	errno = 0;
     -+	hunk->old_start = strtol(line, &end, 10);
     -+	if (errno || end == line || *end++ != ' ')
     ++	*out = strtol(p, &end, 10);
     ++	if (errno || end == p)
      +		return -1;
     -+	line = end;
     ++	if (is_last) {
     ++		if (*end != '\0' && *end != ' ')
     ++			return -1;
     ++	} else {
     ++		if (*end != ' ')
     ++			return -1;
     ++		end++;
     ++	}
     ++	*line = end;
     ++	return 0;
     ++}
      +
     -+	if (!isdigit(*line))
     ++static int parse_hunk_line(const char *line,
     ++			   struct diff_process_hunk *presented)
     ++{
     ++	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
     ++	if (!skip_prefix(line, "hunk ", &line))
      +		return -1;
     -+	errno = 0;
     -+	hunk->old_count = strtol(line, &end, 10);
     -+	if (errno || end == line || *end++ != ' ')
     ++	if (parse_hunk_field(&line, &presented->old_start, 0) ||
     ++	    parse_hunk_field(&line, &presented->old_count, 0) ||
     ++	    parse_hunk_field(&line, &presented->new_start, 0) ||
     ++	    parse_hunk_field(&line, &presented->new_count, 1))
      +		return -1;
     -+	line = end;
     ++	return 0;
     ++}
      +
     -+	if (!isdigit(*line))
     -+		return -1;
     -+	errno = 0;
     -+	hunk->new_start = strtol(line, &end, 10);
     -+	if (errno || end == line || *end++ != ' ')
     -+		return -1;
     -+	line = end;
     ++/*
     ++ * Translate a hunk from the diff process's presentation coordinates
     ++ * into xdiff's.
     ++ *
     ++ * Protocol starts are already 1-based positions (the line a change
     ++ * sits before), the same numbering xdiff uses, so the only adjustment
     ++ * is for an empty file side: "git diff" addresses it with a start of 0
     ++ * and a count of 0 (e.g. "0 0 1 5" adds five lines to an empty old
     ++ * side), and since xdiff uses start-1 as an array index that 0 becomes
     ++ * 1 here.  This is NOT the full inverse of xdl_emit_hunk_hdr()
     ++ * (xdiff/xutils.c): that emitter shifts a count-0 range to start-1 for
     ++ * the displayed "@@" header, but the protocol keeps the unshifted
     ++ * 1-based position for a mid-file insert or delete.  This is the single
     ++ * point where presentation coordinates become xdiff coordinates, so
     ++ * xdl_populate_hunks_from_external() may assume 1-based starts.
     ++ *
     ++ * Returns -1 for a start of 0 paired with a nonzero count, which names
     ++ * no line in either coordinate system.  (parse_hunk_line() already
     ++ * guarantees non-negative starts and counts.)
     ++ */
     ++static int diff_process_hunk_to_xdl(const struct diff_process_hunk *presented,
     ++				    struct xdl_hunk *xdl)
     ++{
     ++	long old_start = presented->old_start;
     ++	long new_start = presented->new_start;
      +
     -+	if (!isdigit(*line))
     ++	if ((!old_start && presented->old_count) ||
     ++	    (!new_start && presented->new_count))
      +		return -1;
     -+	errno = 0;
     -+	hunk->new_count = strtol(line, &end, 10);
     -+	if (errno || end == line || *end != '\0')
     -+		return -1;
     -+
     -+	/*
     -+	 * git diff emits start=0 when count=0 (empty file side).
     -+	 * Normalize to 1-based so downstream validation can assume start >= 1.
     -+	 */
     -+	if (!hunk->old_count && !hunk->old_start)
     -+		hunk->old_start = 1;
     -+	if (!hunk->new_count && !hunk->new_start)
     -+		hunk->new_start = 1;
     -+
     ++	if (!old_start)
     ++		old_start = 1;
     ++	if (!new_start)
     ++		new_start = 1;
     ++
     ++	xdl->old_start = old_start;
     ++	xdl->old_count = presented->old_count;
     ++	xdl->new_start = new_start;
     ++	xdl->new_count = presented->new_count;
      +	return 0;
      +}
      +
     @@ diff-process.c (new)
      +		const char *path,
      +		const char *old_buf, long old_size,
      +		const char *new_buf, long new_size,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
      +		struct xdl_hunk **hunks_out,
      +		size_t *nr_hunks_out)
      +{
     @@ diff-process.c (new)
      +	int fd_in, fd_out;
      +	struct strbuf status = STRBUF_INIT;
      +	struct xdl_hunk *hunks = NULL;
     ++	struct diff_process_hunk presented;
      +	struct xdl_hunk hunk;
      +	size_t nr_hunks = 0, alloc_hunks = 0;
     ++	size_t max_hunks;
      +	int len;
      +	char *line;
      +
     @@ diff-process.c (new)
      +
      +	/* Send request */
      +	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
     -+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
     -+	    packet_flush_gently(fd_in))
     ++	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path))
     ++		goto comm_error;
     ++	/*
     ++	 * old-oid/new-oid let the tool key a cache on the blob pair.  A
     ++	 * side is sent only when its content is the raw blob (the caller
     ++	 * passes NULL otherwise, e.g. for textconv'd content), so an oid
     ++	 * that is present always names the bytes the tool receives.
     ++	 */
     ++	if (oid_a &&
     ++	    packet_write_fmt_gently(fd_in, "old-oid=%s\n", oid_to_hex(oid_a)))
     ++		goto comm_error;
     ++	if (oid_b &&
     ++	    packet_write_fmt_gently(fd_in, "new-oid=%s\n", oid_to_hex(oid_b)))
     ++		goto comm_error;
     ++	if (packet_flush_gently(fd_in))
      +		goto comm_error;
      +
      +	/* Send old file content */
     @@ diff-process.c (new)
      +	if (send_file_content(fd_in, new_buf, new_size))
      +		goto comm_error;
      +
     ++	/*
     ++	 * Hunks are non-overlapping and each useful hunk covers at least
     ++	 * one line, so a valid response cannot contain more hunks than the
     ++	 * two files have lines, which is bounded by their byte sizes.  Cap
     ++	 * the accumulation accordingly so a misbehaving tool that floods
     ++	 * hunk lines cannot drive unbounded memory growth before validation.
     ++	 */
     ++	max_hunks = (size_t)old_size + (size_t)new_size + 1;
     ++
      +	/* Read hunks until flush packet */
      +	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
      +	       line) {
     -+		if (parse_hunk_line(line, &hunk) < 0)
     ++		if (parse_hunk_line(line, &presented) < 0)
     ++			goto comm_error;
     ++		if (diff_process_hunk_to_xdl(&presented, &hunk) < 0)
     ++			goto comm_error;
     ++		if (nr_hunks >= max_hunks) {
     ++			warning(_("diff process '%s' sent too many hunks"
     ++				  " for '%s'"), drv->process, path);
      +			goto comm_error;
     ++		}
      +		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
      +		hunks[nr_hunks++] = hunk;
      +	}
     @@ diff-process.c (new)
      +	return DIFF_PROCESS_ERROR;
      +}
      +
     ++/*
     ++ * Whether exactly one of the two blobs ends in a newline.  A change
     ++ * that only adds or removes the trailing newline is not expressible as
     ++ * line hunks, so a tool comparing lines reports the files as equal.
     ++ */
     ++static int eof_newline_differs(const mmfile_t *a, const mmfile_t *b)
     ++{
     ++	int a_nl = a->size > 0 && a->ptr[a->size - 1] == '\n';
     ++	int b_nl = b->size > 0 && b->ptr[b->size - 1] == '\n';
     ++	return a_nl != b_nl;
     ++}
     ++
     ++/*
     ++ * Number of lines in a blob, matching xdiff's record count: one per
     ++ * newline, plus one more if the last line has no trailing newline.
     ++ */
     ++static long count_lines(const char *buf, long size)
     ++{
     ++	long lines = 0, i;
     ++
     ++	for (i = 0; i < size; i++)
     ++		if (buf[i] == '\n')
     ++			lines++;
     ++	if (size > 0 && buf[size - 1] != '\n')
     ++		lines++;
     ++	return lines;
     ++}
     ++
     ++/*
     ++ * Validate the tool's hunks (already in xdiff coordinates) against the
     ++ * two blobs before they bypass the diff algorithm.  Each hunk must fit
     ++ * within its file, the hunks must be ordered and non-overlapping, and
     ++ * the unchanged run before each hunk (and after the last) must be the
     ++ * same length on both sides -- xdl_build_script() walks the two files
     ++ * in lockstep over unchanged lines, so a mismatched gap desynchronizes
     ++ * it and yields a corrupt diff even when the totals balance.  This is
     ++ * the git layer's job so xdiff stays diagnostic-free; on a bad response
     ++ * we warn and the caller falls back to the builtin diff.  Returns 0 if
     ++ * valid, -1 (after warning) otherwise.
     ++ */
     ++static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,
     ++				   long old_lines, long new_lines,
     ++				   const char *process, const char *path)
     ++{
     ++	size_t i;
     ++	long prev_old_end = 0, prev_new_end = 0;
     ++
     ++	for (i = 0; i < nr; i++) {
     ++		const struct xdl_hunk *h = &hunks[i];
     ++
     ++		if (h->old_count > old_lines - h->old_start + 1 ||
     ++		    h->new_count > new_lines - h->new_start + 1) {
     ++			warning(_("diff process '%s' returned a hunk past the "
     ++				  "end of '%s'; using the builtin diff"),
     ++				process, path);
     ++			return -1;
     ++		}
     ++		if (h->old_start < prev_old_end || h->new_start < prev_new_end) {
     ++			warning(_("diff process '%s' returned overlapping hunks "
     ++				  "for '%s'; using the builtin diff"),
     ++				process, path);
     ++			return -1;
     ++		}
     ++		if (h->old_start - prev_old_end != h->new_start - prev_new_end) {
     ++			warning(_("diff process '%s' returned hunks that leave "
     ++				  "'%s' misaligned; using the builtin diff"),
     ++				process, path);
     ++			return -1;
     ++		}
     ++		prev_old_end = h->old_start + h->old_count;
     ++		prev_new_end = h->new_start + h->new_count;
     ++	}
     ++	if (old_lines - prev_old_end != new_lines - prev_new_end) {
     ++		warning(_("diff process '%s' returned hunks that leave '%s' "
     ++			  "misaligned; using the builtin diff"),
     ++			process, path);
     ++		return -1;
     ++	}
     ++	return 0;
     ++}
     ++
      +enum diff_process_result diff_process_fill_hunks(
      +		struct diff_options *diffopt,
      +		const char *path,
      +		const mmfile_t *file_a,
      +		const mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
      +		xpparam_t *xpp)
      +{
      +	struct userdiff_driver *drv;
     @@ diff-process.c (new)
      +		return DIFF_PROCESS_SKIP;
      +	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
      +		return DIFF_PROCESS_SKIP;
     ++	/*
     ++	 * Whitespace-ignoring, regex-ignore (-I) and anchored options
     ++	 * change which lines count as different, but the tool is never
     ++	 * told about them, so its hunks could not honor them.  Rather
     ++	 * than silently override the user's request, fall back to the
     ++	 * builtin diff, which does honor these flags.  Key this off xpp
     ++	 * (the parameters this diff actually runs with) rather than
     ++	 * diffopt, so a caller like blame that keeps its flags outside
     ++	 * diffopt is covered without a separate guard of its own.
     ++	 */
     ++	if ((xpp->flags & (XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES)) ||
     ++	    xpp->ignore_regex_nr || xpp->anchors_nr)
     ++		return DIFF_PROCESS_SKIP;
      +
      +	drv = userdiff_find_by_path(diffopt->repo->index, path);
      +	if (!drv || !drv->process)
     @@ diff-process.c (new)
      +	res = get_hunks(drv, path,
      +			file_a->ptr, file_a->size,
      +			file_b->ptr, file_b->size,
     ++			oid_a, oid_b,
      +			&ext_hunks, &nr);
      +	if (res == DIFF_PROCESS_OK) {
      +		if (!nr) {
      +			free(ext_hunks);
     ++			/*
     ++			 * Zero hunks means the tool considers the line
     ++			 * content identical, but it cannot express a
     ++			 * trailing-newline-only change.  When that is the
     ++			 * actual difference, fall back to the builtin diff
     ++			 * so the "\ No newline at end of file" marker is
     ++			 * preserved instead of reporting the files equal.
     ++			 */
     ++			if (eof_newline_differs(file_a, file_b))
     ++				return DIFF_PROCESS_SKIP;
      +			return DIFF_PROCESS_EQUIVALENT;
      +		}
     ++		if (validate_external_hunks(ext_hunks, nr,
     ++					    count_lines(file_a->ptr, file_a->size),
     ++					    count_lines(file_b->ptr, file_b->size),
     ++					    drv->process, path) < 0) {
     ++			free(ext_hunks);
     ++			return DIFF_PROCESS_SKIP;
     ++		}
      +		xpp->external_hunks = ext_hunks;
      +		xpp->external_hunks_nr = nr;
      +		return DIFF_PROCESS_OK;
     @@ diff-process.h (new)
      +#include "xdiff/xdiff.h"
      +
      +struct diff_options;
     ++struct object_id;
      +
      +enum diff_process_result {
     -+	DIFF_PROCESS_ERROR = -1, /* tool failure: warned, fell back */
     ++	DIFF_PROCESS_ERROR = -1, /* failed; caller falls back to builtin */
      +	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
     -+	DIFF_PROCESS_SKIP,       /* no process configured: use builtin */
     ++	DIFF_PROCESS_SKIP,       /* process did not apply: use builtin */
      +	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
      +};
      +
     @@ diff-process.h (new)
      + * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
      + * The caller owns xpp->external_hunks and must free() it.
      + *
     -+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks
     -+ * (files are considered identical); caller should skip diff/blame.
     ++ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks and
     ++ * the blobs are not a trailing-newline-only change (files are
     ++ * considered identical); caller should skip diff/blame.
      + * Returns DIFF_PROCESS_SKIP when no process applies; caller
      + * should use the builtin diff algorithm.
      + * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
      + * caller should fall back to the builtin diff algorithm.
     ++ *
     ++ * oid_a/oid_b, when non-NULL, are sent to the tool as old-oid/new-oid
     ++ * so it can key a cache on the blob pair.  Pass NULL for a side whose
     ++ * content is not the raw blob (e.g. textconv'd) or whose object name is
     ++ * unknown, so any oid that is sent always names the bytes the tool
     ++ * receives.
      + */
      +enum diff_process_result diff_process_fill_hunks(
      +		struct diff_options *diffopt,
      +		const char *path,
      +		const mmfile_t *file_a,
      +		const mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
      +		xpparam_t *xpp);
      +
      +#endif /* DIFF_PROCESS_H */
     @@ diff.c: static void builtin_diff(const char *name_a,
       		xpp.anchors = o->anchors;
       		xpp.anchors_nr = o->anchors_nr;
      +
     -+		if (diff_process_fill_hunks(o, name_a,
     -+					    &mf1, &mf2, &xpp)
     ++		/*
     ++		 * Send the blob oids only for a side whose content is the
     ++		 * raw blob: textconv rewrites the bytes, and a working-tree
     ++		 * side has no stored oid, so pass NULL there rather than an
     ++		 * oid that would not name what the tool receives.
     ++		 */
     ++		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
     ++					    (textconv_one || !one->oid_valid) ? NULL : &one->oid,
     ++					    (textconv_two || !two->oid_valid) ? NULL : &two->oid,
     ++					    &xpp)
      +		    == DIFF_PROCESS_EQUIVALENT) {
      +			if (textconv_one)
      +				free(mf1.ptr);
     @@ t/helper/test-diff-process-backend.c (new)
      + *
      + *   packet:          git> command=hunks
      + *   packet:          git> pathname=<path>
     ++ *   packet:          git> [old-oid=<hex>]   (omitted for textconv/worktree)
     ++ *   packet:          git> [new-oid=<hex>]
      + *   packet:          git> 0000
      + *   packet:          git> OLD_CONTENT
      + *   packet:          git> 0000
     @@ t/helper/test-diff-process-backend.c (new)
      + *
      + * Response varies by --mode (default: whole-file):
      + *
     -+ *   whole-file   packet: git< hunk 1 <old_lines> 1 <new_lines>
     ++ *   whole-file   packet: git< hunk <1|0> <old_lines> <1|0> <new_lines>
     ++ *                (start is 0 for an empty side, matching git diff)
      + *   fixed-hunk   packet: git< hunk 5 2 5 2
      + *   no-hunks     (no hunk packets)
      + *   bad-hunk     packet: git< hunk 999 1 999 1
      + *   bad-parse    packet: git< garbage not a hunk
      + *   bad-sync     packet: git< hunk 1 2 1 1
     ++ *   bad-gap      packet: git< hunk 1 1 3 1
     ++ *   bad-start    packet: git< hunk 0 1 1 1
     ++ *   multi-hunk   packet: git< hunk 5 2 5 2
     ++ *                packet: git< hunk 9 2 9 2
     ++ *   insert       packet: git< hunk 3 0 3 2   (mid-file count-0 insertion)
     ++ *   flood        packet: git< hunk 1 1 1 1   (x100000)
      + *   overlap      packet: git< hunk 1 5 1 5
      + *                packet: git< hunk 3 2 3 2
      + *   no-cap       (omits capability=hunks during handshake)
     @@ t/helper/test-diff-process-backend.c (new)
      + *   abort        (status=abort instead of status=success)
      + *   crash        exit(1) before sending any response
      + *
     -+ * All non-error/abort modes end with:
     ++ * All success modes (not error/abort/crash) end with:
      + *
      + *   packet:          git< 0000
      + *   packet:          git< status=success
     @@ t/helper/test-diff-process-backend.c (new)
      + *
      + * Each request is logged to --log as:
      + *
     -+ *   command=<cmd> pathname=<path> old=<first line> new=<first line>
     ++ *   command=<cmd> pathname=<path> old-oid=<hex> new-oid=<hex> old=<first line> new=<first line>
      + */
      +
      +#include "test-tool.h"
     @@ t/helper/test-diff-process-backend.c (new)
      +	MODE_BAD_HUNK,
      +	MODE_BAD_PARSE,
      +	MODE_BAD_SYNC,
     ++	MODE_BAD_GAP,
     ++	MODE_BAD_START,
     ++	MODE_MULTI_HUNK,
     ++	MODE_INSERT,
     ++	MODE_FLOOD,
      +	MODE_OVERLAP,
      +	MODE_NO_CAP,
      +	MODE_ERROR,
     @@ t/helper/test-diff-process-backend.c (new)
      +		return MODE_BAD_PARSE;
      +	if (!strcmp(s, "bad-sync"))
      +		return MODE_BAD_SYNC;
     ++	if (!strcmp(s, "bad-gap"))
     ++		return MODE_BAD_GAP;
     ++	if (!strcmp(s, "bad-start"))
     ++		return MODE_BAD_START;
     ++	if (!strcmp(s, "multi-hunk"))
     ++		return MODE_MULTI_HUNK;
     ++	if (!strcmp(s, "insert"))
     ++		return MODE_INSERT;
     ++	if (!strcmp(s, "flood"))
     ++		return MODE_FLOOD;
      +	if (!strcmp(s, "overlap"))
      +		return MODE_OVERLAP;
      +	if (!strcmp(s, "no-cap"))
     @@ t/helper/test-diff-process-backend.c (new)
      + * variant: once inside a request, truncation is a protocol violation
      + * and dying loudly is the correct response.
      + */
     -+static int read_request_header(char **command, char **pathname)
     ++static int read_request_header(char **command, char **pathname,
     ++			       char **old_oid, char **new_oid)
      +{
      +	int first = 1;
      +	char *line;
      +
     -+	*command = *pathname = NULL;
     ++	*command = *pathname = *old_oid = *new_oid = NULL;
      +	for (;;) {
      +		const char *value;
      +
     @@ t/helper/test-diff-process-backend.c (new)
      +			*command = xstrdup(value);
      +		else if (skip_prefix(line, "pathname=", &value))
      +			*pathname = xstrdup(value);
     ++		else if (skip_prefix(line, "old-oid=", &value))
     ++			*old_oid = xstrdup(value);
     ++		else if (skip_prefix(line, "new-oid=", &value))
     ++			*new_oid = xstrdup(value);
      +	}
      +	return 1;
      +}
     @@ t/helper/test-diff-process-backend.c (new)
      +	case MODE_BAD_SYNC:
      +		packet_write_fmt(1, "hunk 1 2 1 1\n");
      +		break;
     ++	case MODE_BAD_GAP:
     ++		/*
     ++		 * Globally balanced (1 changed line on each side, so the
     ++		 * total unchanged counts match) but the gap before the
     ++		 * change differs between sides: old line 1 vs new line 3.
     ++		 * Exercises the per-gap lockstep-alignment check.
     ++		 */
     ++		packet_write_fmt(1, "hunk 1 1 3 1\n");
     ++		break;
     ++	case MODE_BAD_START:
     ++		/*
     ++		 * A start of 0 is valid only for an empty (count 0) range;
     ++		 * pairing it with a nonzero count names no line in either
     ++		 * the protocol's or xdiff's coordinates, so the translation
     ++		 * rejects it and git falls back to the builtin diff.
     ++		 */
     ++		packet_write_fmt(1, "hunk 0 1 1 1\n");
     ++		break;
     ++	case MODE_MULTI_HUNK:
     ++		/*
     ++		 * Two valid, non-overlapping, gap-aligned hunks.  Exercises
     ++		 * the accepting branch of the per-gap lockstep check with a
     ++		 * non-zero previous-hunk end (the realistic two-region case).
     ++		 */
     ++		packet_write_fmt(1, "hunk 5 2 5 2\n");
     ++		packet_write_fmt(1, "hunk 9 2 9 2\n");
     ++		break;
     ++	case MODE_INSERT:
     ++		/*
     ++		 * A mid-file pure insertion (count 0 on the old side) in the
     ++		 * protocol's 1-based-position form: 2 lines inserted before
     ++		 * old line 3.  Exercises the count-0 path, which uses the
     ++		 * unshifted position (not git diff's "-3,0" display start).
     ++		 */
     ++		packet_write_fmt(1, "hunk 3 0 3 2\n");
     ++		break;
     ++	case MODE_FLOOD: {
     ++		/*
     ++		 * Emit far more hunks than any small file has lines, so Git
     ++		 * trips its accumulation cap and falls back before reading
     ++		 * them all.
     ++		 */
     ++		int i;
     ++		for (i = 0; i < 100000; i++)
     ++			packet_write_fmt(1, "hunk 1 1 1 1\n");
     ++		break;
     ++	}
      +	case MODE_OVERLAP:
      +		packet_write_fmt(1, "hunk 1 5 1 5\n");
      +		packet_write_fmt(1, "hunk 3 2 3 2\n");
     @@ t/helper/test-diff-process-backend.c (new)
      +{
      +	for (;;) {
      +		char *command = NULL, *pathname = NULL;
     ++		char *old_oid = NULL, *new_oid = NULL;
      +		struct strbuf obuf = STRBUF_INIT;
      +		struct strbuf nbuf = STRBUF_INIT;
      +
     -+		if (!read_request_header(&command, &pathname))
     ++		if (!read_request_header(&command, &pathname,
     ++					 &old_oid, &new_oid))
      +			break; /* EOF: Git closed its end */
      +
      +		read_packetized_to_strbuf(0, &obuf, 0);
     @@ t/helper/test-diff-process-backend.c (new)
      +
      +		if (logfile) {
      +			fprintf(logfile,
     -+				"command=%s pathname=%s old=%.*s new=%.*s\n",
     ++				"command=%s pathname=%s old-oid=%s new-oid=%s"
     ++				" old=%.*s new=%.*s\n",
      +				command ? command : "(none)",
      +				pathname ? pathname : "(none)",
     ++				old_oid ? old_oid : "(none)",
     ++				new_oid ? new_oid : "(none)",
      +				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
      +				obuf.buf,
      +				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
     @@ t/helper/test-diff-process-backend.c (new)
      +
      +		free(command);
      +		free(pathname);
     ++		free(old_oid);
     ++		free(new_oid);
      +		strbuf_release(&obuf);
      +		strbuf_release(&nbuf);
      +	}
     @@ t/helper/test-diff-process-backend.c (new)
      +	enum mode mode = MODE_WHOLE_FILE;
      +	struct option options[] = {
      +		OPT_STRING(0, "mode", &mode_str, "mode",
     -+			   "response shape: whole-file (default), fixed-hunk,"
     -+			   " no-hunks, bad-hunk, bad-sync, overlap, error,"
     -+			   " abort, crash"),
     ++			   "response shape (default whole-file);"
     ++			   " see the file header for the full list of modes"),
      +		OPT_STRING(0, "log", &log_path, "path",
      +			   "append per-request summary to this file"),
      +		OPT_END()
     @@ t/t4080-diff-process.sh (new)
      +
      +test_description='diff process via long-running process'
      +
     ++TEST_PASSES_SANITIZE_LEAK=true
      +. ./test-lib.sh
      +
      +# See t/helper/test-diff-process-backend.c for the backend implementation
     @@ t/t4080-diff-process.sh (new)
      +	EOF
      +	git add logtest.c &&
      +
     -+	# two.c/one.c: two-file pair for error/abort/startup-failure tests.
     ++	# one.c/two.c: two-file pair for error/abort/startup-failure tests.
      +	cat >one.c <<-\EOF &&
      +	int first(void) { return 1; }
      +	EOF
     @@ t/t4080-diff-process.sh (new)
      +	test_grep ! "^+NEW10" actual
      +'
      +
     ++test_expect_success 'diff process accepts valid multi-hunk output' '
     ++	# multi-hunk reports both changed regions (5-6 and 9-10) as two
     ++	# gap-aligned hunks.  This exercises the accepting branch of the
     ++	# per-gap lockstep check (non-zero previous-hunk end) and must
     ++	# produce a correct two-region diff with the lines between the
     ++	# hunks kept as context.
     ++	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	test_grep "^ line7" actual &&
     ++	test_grep "^ line8" actual &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process accepts a mid-file count-0 insertion' '
     ++	# insert mode reports "hunk 3 0 3 2": a pure insertion (count 0 on
     ++	# the old side) in the protocol 1-based-position form.  Exercises
     ++	# the count-0 hunk path that the other valid-hunk modes (full
     ++	# replacements, equal-count modifies) never hit.  Empty stderr is
     ++	# the discriminator: a mishandled count-0 start would be rejected
     ++	# by the lockstep check and warn.
     ++	cat >insert.c <<-\EOF &&
     ++	a
     ++	b
     ++	c
     ++	d
     ++	e
     ++	EOF
     ++	git add insert.c &&
     ++	git commit -m "add insert.c" &&
     ++	cat >insert.c <<-\EOF &&
     ++	a
     ++	b
     ++	X
     ++	Y
     ++	c
     ++	d
     ++	e
     ++	EOF
     ++	git -c diff.cdiff.process="$BACKEND --mode=insert" \
     ++		diff insert.c >actual 2>stderr &&
     ++	test_grep "^+X" actual &&
     ++	test_grep "^+Y" actual &&
     ++	test_grep "^ c" actual &&
     ++	test_must_be_empty stderr
     ++'
     ++
      +test_expect_success 'diff process works with modified file' '
      +	test_when_finished "rm -f backend.log" &&
      +	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     @@ t/t4080-diff-process.sh (new)
      +		diff --exit-code nohunks.c
      +'
      +
     ++test_expect_success 'diff process equivalent commit: --exit-code and --quiet agree' '
     ++	# A committed blob pair (not a worktree file) whose oids differ but
     ++	# the tool reports equivalent.  --exit-code and --quiet must agree
     ++	# with the shown diff (empty) and report success, not fall back to
     ++	# the byte-level "oids differ" answer.
     ++	cat >ecq.c <<-\EOF &&
     ++	alpha
     ++	EOF
     ++	git add ecq.c &&
     ++	git commit -m "ecq v1" &&
     ++	cat >ecq.c <<-\EOF &&
     ++	beta
     ++	EOF
     ++	git add ecq.c &&
     ++	git commit -m "ecq v2" &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff --exit-code HEAD^ HEAD -- ecq.c &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
     ++		diff --quiet HEAD^ HEAD -- ecq.c
     ++'
     ++
     ++test_expect_success 'diff process falls back for trailing-newline-only change' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	printf "a\nb\nc\n" >eofnl.c &&
     ++	git add eofnl.c &&
     ++	git commit -m "add eofnl.c" &&
     ++	printf "a\nb\nc" >eofnl.c &&
     ++	# Same lines, only the final newline removed.  The tool reports
     ++	# no hunks (it sees identical lines), but that change is not
     ++	# expressible as hunks, so git falls back to the builtin diff
     ++	# rather than treating the files as equivalent.
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		diff eofnl.c >actual 2>stderr &&
     ++	test_grep "No newline at end of file" actual &&
     ++	test_grep "pathname=eofnl.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process falls back for added file (empty old side)' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	printf "x\ny\nz\n" >addnl.c &&
     ++	git add addnl.c &&
     ++	# The empty old side has no trailing newline while the new side
     ++	# does, so the newline fallback shows the addition rather than
     ++	# letting no-hunks suppress the whole new file.
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
     ++		diff --cached addnl.c >actual 2>stderr &&
     ++	test_grep "^+x" actual &&
     ++	test_grep "pathname=addnl.c" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
      +test_expect_success 'diff process with --exit-code and hunks returns failure' '
      +	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
      +		diff --exit-code newfile.c
     @@ t/t4080-diff-process.sh (new)
      +	test_path_is_missing backend.log
      +'
      +
     -+test_expect_success 'diff process not used by --stat' '
     ++test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
      +	test_when_finished "rm -f backend.log" &&
     -+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     -+		diff --stat worddiff.c >actual &&
     -+	test_grep "worddiff.c" actual &&
     -+	test_path_is_missing backend.log
     ++	printf "a\nb\nc\n" >wsbypass.c &&
     ++	git add wsbypass.c &&
     ++	git commit -m "add wsbypass.c" &&
     ++	printf "a\n  b  \nc\n" >wsbypass.c &&
     ++	# The tool is never told about these options and could not honor
     ++	# them, so git bypasses the process for each (covering the whole
     ++	# XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES mask, not just -w).
     ++	for opt in -w -b --ignore-space-at-eol --ignore-blank-lines
     ++	do
     ++		rm -f backend.log &&
     ++		git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++			diff $opt wsbypass.c >actual 2>stderr &&
     ++		test_path_is_missing backend.log &&
     ++		test_must_be_empty stderr ||
     ++		return 1
     ++	done &&
     ++	# -w additionally suppresses the whitespace-only change via the
     ++	# builtin diff that now runs.
     ++	git -c diff.cdiff.process="$BACKEND" diff -w wsbypass.c >actual &&
     ++	test_must_be_empty actual
      +'
      +
      +#
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "^+NEW5" actual &&
      +	test_grep "^-OLD9" actual &&
      +	test_grep "^+NEW9" actual &&
     -+	test_grep "exceeds.*lines" stderr
     ++	test_grep "hunk past the end" stderr
      +'
      +
      +test_expect_success 'diff process fallback on mismatched unchanged totals' '
     @@ t/t4080-diff-process.sh (new)
      +	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
      +		diff synctest.c >actual 2>stderr &&
      +	test_grep "changed" actual &&
     -+	test_grep "unchanged line count mismatch" stderr
     ++	test_grep "misaligned" stderr
     ++'
     ++
     ++test_expect_success 'diff process fallback on misaligned hunk gap' '
     ++	# bad-gap reports hunk 1 1 3 1 on boundary.c: one changed line
     ++	# on each side, so the total unchanged counts match, but the
     ++	# unchanged run before the change differs (old line 1 vs new
     ++	# line 3).  A global count check would accept this and emit a
     ++	# corrupt diff; the per-gap lockstep check rejects it and git
     ++	# falls back to the builtin algorithm.
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-gap" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	# The builtin fallback shows both changed regions as additions
     ++	# (a corrupt-accepted hunk would show NEW5 only as context).
     ++	test_grep "^+NEW5" actual &&
     ++	test_grep "^+NEW9" actual &&
     ++	test_grep "misaligned" stderr
      +'
      +
      +test_expect_success 'diff process fallback on overlapping hunks' '
      +	# boundary.c has 10 lines, so both hunks are in bounds
     -+	# but they overlap at lines 3-5, triggering the ordering check.
     ++	# but they overlap at lines 3-4, triggering the ordering check.
      +	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "NEW5" actual &&
     -+	test_grep "overlaps with previous" stderr
     ++	test_grep "overlapping hunks" stderr
      +'
      +
      +test_expect_success 'diff process fallback on malformed hunk line' '
     @@ t/t4080-diff-process.sh (new)
      +	test_grep "^+NEW5" actual
      +'
      +
     -+test_expect_success 'diff process skipped when tool omits capability' '
     -+	git -c diff.cdiff.process="$BACKEND --mode=no-cap" \
     ++test_expect_success 'diff process fallback on start 0 with nonzero count' '
     ++	# bad-start reports hunk 0 1 1 1.  A start of 0 is valid only for
     ++	# an empty (count 0) range, so the presentation-to-xdiff
     ++	# translation rejects it and git falls back to the builtin diff
     ++	# instead of handing xdiff an out-of-range start.
     ++	git -c diff.cdiff.process="$BACKEND --mode=bad-start" \
      +		diff boundary.c >actual 2>stderr &&
      +	test_grep "^-OLD5" actual &&
      +	test_grep "^+NEW5" actual &&
     ++	test_grep "diff process.*failed" stderr
     ++'
     ++
     ++test_expect_success 'diff process caps a flood of hunks and falls back' '
     ++	# flood emits far more hunks than the file has lines.  Git must
     ++	# stop accumulating and fall back to the builtin diff rather than
     ++	# grow memory without bound.
     ++	git -c diff.cdiff.process="$BACKEND --mode=flood" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "too many hunks" stderr
     ++'
     ++
     ++test_expect_success 'diff process skipped when tool omits capability' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	git -c diff.cdiff.process="$BACKEND --mode=no-cap --log=backend.log" \
     ++		diff boundary.c >actual 2>stderr &&
     ++	# Builtin diff runs: all changes appear, including lines 9-10
     ++	# that a tool-provided hunk would have narrowed away.
     ++	test_grep "^-OLD5" actual &&
     ++	test_grep "^-OLD9" actual &&
     ++	# The process launched (creating the log) but was
     ++	# never sent a per-file request, so no hunks command is logged.
     ++	test_path_is_file backend.log &&
     ++	test_grep ! "command=hunks" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process receives old-oid and new-oid for a blob pair' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	cat >oidpair.c <<-\EOF &&
     ++	int f(void) { return 1; }
     ++	EOF
     ++	git add oidpair.c &&
     ++	git commit -m "oidpair v1" &&
     ++	old=$(git rev-parse HEAD:oidpair.c) &&
     ++
     ++	cat >oidpair.c <<-\EOF &&
     ++	int f(void) { return 2; }
     ++	EOF
     ++	git add oidpair.c &&
     ++	git commit -m "oidpair v2" &&
     ++	new=$(git rev-parse HEAD:oidpair.c) &&
     ++
     ++	# Both sides are stored blobs, so their object names are sent.
     ++	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff HEAD^ HEAD -- oidpair.c >actual 2>stderr &&
     ++	test_grep "old-oid=$old new-oid=$new" backend.log &&
     ++	test_must_be_empty stderr
     ++'
     ++
     ++test_expect_success 'diff process omits old-oid and new-oid for textconv content' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	write_script oidcat <<-\EOF &&
     ++	cat "$1"
     ++	EOF
     ++	cat >oidtc.c <<-\EOF &&
     ++	alpha
     ++	EOF
     ++	git add oidtc.c &&
     ++	git commit -m "oidtc v1" &&
     ++	cat >oidtc.c <<-\EOF &&
     ++	beta
     ++	EOF
     ++	git add oidtc.c &&
     ++	git commit -m "oidtc v2" &&
     ++
     ++	# textconv rewrites the bytes, so the raw-blob object name that
     ++	# would otherwise identify each side is omitted.
     ++	git -c diff.cdiff.textconv="./oidcat" \
     ++	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
     ++		diff HEAD^ HEAD -- oidtc.c >actual 2>stderr &&
     ++	test_grep "pathname=oidtc.c" backend.log &&
     ++	test_grep "old-oid=(none) new-oid=(none)" backend.log &&
      +	test_must_be_empty stderr
      +'
      +
  5:  6ec6716ea4 !  6:  b2e80f014e diff: bypass diff process with --no-ext-diff and in format-patch
     @@ Documentation/diff-options.adoc: endif::git-format-patch[]
       `--textconv`::
       `--no-textconv`::
      
     + ## Documentation/gitattributes.adoc ##
     +@@ Documentation/gitattributes.adoc: display, which is covered above); and combined diffs (`--cc` and merge
     + diffs), whose protocol would have to be extended from a single old/new
     + pair to one comparison per merge parent.
     + 
     +-`--diff-algorithm` bypasses the process entirely, for every feature
     +-listed above.  The whitespace-ignoring options (`-w`,
     +-`--ignore-space-change`, `--ignore-blank-lines`, and the like),
     ++`--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
     ++for every feature listed above.  The whitespace-ignoring options
     ++(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like),
     + `-I<regex>`, and `--anchored` also bypass it for the affected files:
     + the tool is never told about these options, so it could not honor
     + them, and Git falls back to the builtin diff, which does.
     +
       ## builtin/log.c ##
      @@ builtin/log.c: int cmd_format_patch(int argc,
       	if (argc > 1)
     @@ diff.h: struct diff_flags {
       
      -	/** Disables diff.<driver>.process. */
      +	/**
     -+	 * Disables diff.<driver>.process.  Set by --no-ext-diff.
     ++	 * Disables diff.<driver>.process.  Set by --no-ext-diff and by
     ++	 * format-patch.
      +	 */
       	unsigned no_diff_process;
       
     @@ t/t4080-diff-process.sh: test_expect_success 'diff process bypassed by --diff-al
      +	test_path_is_missing backend.log
      +'
      +
     - test_expect_success 'diff process not used by --stat' '
     + test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
       	test_when_finished "rm -f backend.log" &&
     - 	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
     + 	printf "a\nb\nc\n" >wsbypass.c &&
  6:  3dadafa1bc !  7:  cf5bb8984a blame: consult diff process for no-hunk detection
     @@ Commit message
          commit as having no changes, causing blame to attribute lines
          to earlier commits.
      
     -    The consultation happens at the pass_blame_to_parent() callsite
     -    using diff_process_fill_hunks(), matching how builtin_diff() in
     -    diff.c uses the same function.  A new diff_hunks_xpp() variant
     -    accepts a pre-populated xpparam_t so callers can pass external
     -    hunks, while the existing diff_hunks() retains its original
     -    signature and behavior.  The copy-detection callsite is
     -    unaffected since it does not use the diff process.
     +    Introduce xdi_diff_process(), a process-aware xdi_diff() that
     +    consults the process, runs xdiff on the tool's hunks or on the
     +    builtin algorithm when it does not apply, frees the hunks, and
     +    reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller
     +    can drop or skip the change.  It is the shared consult-then-diff path
     +    for consumers that work on raw hunks: blame's pass_blame_to_parent()
     +    uses it here, and git log -L reuses it later.  builtin_diff() keeps
     +    consulting the process directly, because it tests for equivalence
     +    early, before its funcname-pattern and word-diff setup, so a
     +    reformat-only file short-circuits without that work.
      
     -    The subprocess is long-running (one startup cost amortized
     -    across the blame traversal), but each commit in the file's
     -    history incurs a round-trip to the tool.
     +    Blame's -w option is not communicated to the process and it could not
     +    honor it, so blame must fall back to the builtin diff there.  Because
     +    blame keeps its whitespace flags in sb->xdl_opts rather than diffopt,
     +    the process bypass keys off xpp (the flags the diff actually runs
     +    with), which covers blame without a guard of its own.
     +
     +    The subprocess is long-running (one startup cost amortized across the
     +    blame traversal), but each commit in the file's history incurs a
     +    round-trip to the tool.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ blame.c
       #include "alloc.h"
       #include "commit-slab.h"
       #include "bloom.h"
     -@@ blame.c: static struct commit *fake_working_tree_commit(struct repository *r,
     - 
     - 
     - 
     --static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     --		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     -+static int diff_hunks_xpp(mmfile_t *file_a, mmfile_t *file_b,
     -+			  xdl_emit_hunk_consume_func_t hunk_func,
     -+			  void *cb_data, xpparam_t *xpp)
     - {
     --	xpparam_t xpp = {0};
     - 	xdemitconf_t xecfg = {0};
     - 	xdemitcb_t ecb = {NULL};
     - 
     --	xpp.flags = xdl_opts;
     - 	xecfg.hunk_func = hunk_func;
     - 	ecb.priv = cb_data;
     --	return xdi_diff(file_a, file_b, &xpp, &xecfg, &ecb);
     -+	return xdi_diff(file_a, file_b, xpp, &xecfg, &ecb);
     -+}
     -+
     -+static int diff_hunks(mmfile_t *file_a, mmfile_t *file_b,
     -+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data, int xdl_opts)
     -+{
     -+	xpparam_t xpp = {0};
     -+
     -+	xpp.flags = xdl_opts;
     -+	return diff_hunks_xpp(file_a, file_b, hunk_func, cb_data, &xpp);
     - }
     - 
     - static const char *get_next_line(const char *start, const char *end)
      @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       				 struct blame_origin *parent, int ignore_diffs)
       {
       	mmfile_t file_p, file_o;
      +	xpparam_t xpp = {0};
     ++	xdemitconf_t xecfg = {0};
     ++	xdemitcb_t ecb = {NULL};
       	struct blame_chunk_cb_data d;
       	struct blame_entry *newdest = NULL;
       
     @@ blame.c: static void pass_blame_to_parent(struct blame_scoreboard *sb,
       	sb->num_get_patch++;
       
      -	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
     --		die("unable to generate diff (%s -> %s)",
     --		    oid_to_hex(&parent->commit->object.oid),
     --		    oid_to_hex(&target->commit->object.oid));
      +	xpp.flags = sb->xdl_opts;
     ++	xecfg.hunk_func = blame_chunk_cb;
     ++	ecb.priv = &d;
      +	/*
     -+	 * If the diff process considers the files equivalent,
     -+	 * skip the diff so blame looks past this commit.
     ++	 * Consult the diff process, then attribute the resulting chunks
     ++	 * via blame_chunk_cb.  It bypasses the process for the whitespace-
     ++	 * ignoring options it cannot honor (they live in xpp.flags, which
     ++	 * the consultation checks), and when the process reports the blobs
     ++	 * equivalent it runs no diff, so blame passes this commit and looks
     ++	 * past it.  Look up the driver by the parent (old) path, as
     ++	 * builtin_diff() does with name_a, so a renamed file resolves to the
     ++	 * same driver across diff, blame, and line-log.  Pass no
     ++	 * old-oid/new-oid: blame diffs each blob pair once, so the tool gains
     ++	 * nothing from a per-invocation cache key.
      +	 */
     -+	if (diff_process_fill_hunks(&sb->revs->diffopt, target->path,
     -+				    &file_p, &file_o, &xpp)
     -+	    != DIFF_PROCESS_EQUIVALENT) {
     -+		if (diff_hunks_xpp(&file_p, &file_o, blame_chunk_cb,
     -+				   &d, &xpp))
     -+			die("unable to generate diff (%s -> %s)",
     -+			    oid_to_hex(&parent->commit->object.oid),
     -+			    oid_to_hex(&target->commit->object.oid));
     -+	}
     -+	free(xpp.external_hunks);
     - 	/* The rest are the same as the parent */
     - 	blame_chunk(&d.dstq, &d.srcq, INT_MAX, d.offset, INT_MAX, 0,
     - 		    parent, target, 0);
     ++	if (xdi_diff_process(&sb->revs->diffopt, parent->path,
     ++			     &file_p, &file_o, NULL, NULL, &xpp, &xecfg, &ecb)
     ++	    == DIFF_PROCESS_ERROR)
     + 		die("unable to generate diff (%s -> %s)",
     + 		    oid_to_hex(&parent->commit->object.oid),
     + 		    oid_to_hex(&target->commit->object.oid));
     +
     + ## diff-process.c ##
     +@@
     + #include "sub-process.h"
     + #include "pkt-line.h"
     + #include "strbuf.h"
     ++#include "xdiff-interface.h"
     + #include "xdiff/xdiff.h"
     + 
     + #define CAP_HUNKS (1u << 0)
     +@@ diff-process.c: enum diff_process_result diff_process_fill_hunks(
     + 	}
     + 	return DIFF_PROCESS_SKIP;
     + }
     ++
     ++enum diff_process_result xdi_diff_process(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		mmfile_t *file_a,
     ++		mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
     ++		xpparam_t *xpp,
     ++		xdemitconf_t *xecfg,
     ++		xdemitcb_t *ecb)
     ++{
     ++	enum diff_process_result res;
     ++
     ++	/*
     ++	 * Consult the diff process, then run xdiff either constrained to
     ++	 * the tool's hunks or, when the process does not apply, computing
     ++	 * the diff itself as a fallback.  EQUIVALENT short-circuits: the
     ++	 * caller decides what "no change" means for it (drop the commit,
     ++	 * skip the file, ...), so xdiff is not run.
     ++	 *
     ++	 * A SKIP/ERROR from the process just selects the builtin path
     ++	 * (its warning, if any, was already emitted), so the result then
     ++	 * reflects whether xdiff itself succeeded, not the process.
     ++	 */
     ++	res = diff_process_fill_hunks(diffopt, path, file_a, file_b,
     ++				      oid_a, oid_b, xpp);
     ++	if (res == DIFF_PROCESS_EQUIVALENT)
     ++		return res;
     ++
     ++	res = xdi_diff(file_a, file_b, xpp, xecfg, ecb) < 0
     ++		? DIFF_PROCESS_ERROR : DIFF_PROCESS_OK;
     ++
     ++	FREE_AND_NULL(xpp->external_hunks);
     ++	xpp->external_hunks_nr = 0;
     ++	return res;
     ++}
     +
     + ## diff-process.h ##
     +@@ diff-process.h: enum diff_process_result diff_process_fill_hunks(
     + 		const struct object_id *oid_b,
     + 		xpparam_t *xpp);
     + 
     ++/*
     ++ * Process-aware xdi_diff(): consult the diff process for 'path', then
     ++ * run xdiff either constrained to the tool's hunks or computing the
     ++ * diff itself when the process does not apply or fails.  Frees any
     ++ * hunks it obtained before returning.
     ++ *
     ++ * Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool
     ++ * reports the blobs equal, so the caller can drop or skip the change;
     ++ * DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and
     ++ * DIFF_PROCESS_ERROR if xdiff itself errored.
     ++ *
     ++ * The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as
     ++ * for a direct xdi_diff() call.  oid_a/oid_b are forwarded to
     ++ * diff_process_fill_hunks() (see there).
     ++ */
     ++enum diff_process_result xdi_diff_process(
     ++		struct diff_options *diffopt,
     ++		const char *path,
     ++		mmfile_t *file_a,
     ++		mmfile_t *file_b,
     ++		const struct object_id *oid_a,
     ++		const struct object_id *oid_b,
     ++		xpparam_t *xpp,
     ++		xdemitconf_t *xecfg,
     ++		xdemitcb_t *ecb);
     ++
     + #endif /* DIFF_PROCESS_H */
      
       ## t/t4080-diff-process.sh ##
     -@@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omits capability' '
     +@@ t/t4080-diff-process.sh: test_expect_success 'diff process omits old-oid and new-oid for textconv content
       	test_must_be_empty stderr
       '
       
     @@ t/t4080-diff-process.sh: test_expect_success 'diff process skipped when tool omi
      +	test_grep "$CHANGE" line9 &&
      +	test_path_is_missing backend.log
      +'
     ++
     ++test_expect_success 'blame -w bypasses diff process' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	printf "alpha\nbeta\ngamma\n" >blamew.c &&
     ++	git add blamew.c &&
     ++	git commit -m "add blamew.c" &&
     ++	orig=$(git rev-parse --short HEAD) &&
     ++	printf "alpha\n   beta   \ngamma\n" >blamew.c &&
     ++	git commit -am "reindent beta" &&
     ++	reindent=$(git rev-parse --short HEAD) &&
     ++	# blame -w must ignore the whitespace-only change and attribute
     ++	# beta to the original commit, not the reindent commit.  The tool
     ++	# is never told about -w, so blame must bypass it (not let tool
     ++	# hunks override -w).
     ++	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=backend.log" \
     ++		blame -w blamew.c >actual &&
     ++	sed -n "2p" actual >line2 &&
     ++	test_grep "$orig" line2 &&
     ++	test_grep ! "$reindent" line2 &&
     ++	test_path_is_missing backend.log
     ++'
      +
       test_done
  -:  ---------- >  8:  c1d02d0e15 diff: consult diff process for --stat counts
  -:  ---------- >  9:  c3c17ba8fc line-log: consult diff process for range tracking

-- 
gitgitgadget

[PATCH v5 1/9] gitattributes: document how external diff drivers relate to diff features

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

The "Defining an external diff driver" section explains how to
configure diff.<driver>.command but not how the driver relates to the
rest of Git's diff machinery.  In particular, the command only
replaces the textual patch: word diff, function context, color, and
the like cannot apply to its output, while the summary formats, blame,
and git log -L do not run it at all and keep using the builtin diff.

Spell this out so the scope of an external diff driver is clear.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index bd76167a45..2c4fbfd7f1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -784,6 +784,16 @@ with the above configuration, i.e. `j-c-diff`, with 7
 parameters, just like `GIT_EXTERNAL_DIFF` program is called.
 See linkgit:git[1] for details.
 
+An external diff driver replaces the patch Git would otherwise
+produce for the path: Git runs the command and shows its output in
+place of its own.  Output features that post-process Git's diff do
+not apply to it; word diff, function context (`-W`), `--color-moved`,
+and coloring all act on Git's builtin diff, not the driver's output.
+The driver is consulted only when Git generates a textual patch.  The
+summary formats (`--stat`, `--numstat`, `--shortstat`, and
+`--dirstat`), `git blame`, and `git log -L` do not run it and
+continue to use Git's builtin diff.
+
 If the program is able to ignore certain changes (similar to
 `git diff --ignore-space-change`), then also set the option
 `trustExitCode` to true.  It is then expected to return exit code 1 if
-- 
gitgitgadget

[PATCH v5 2/9] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use.  Out-of-bounds line numbers,
overlapping or out-of-order hunks, and misaligned unchanged runs are
treated as a malformed tool response: xdl_populate_hunks_from_external()
warns, returns -1, and xdl_diff() falls back to the builtin diff
algorithm for that file.  The run of unchanged lines between two hunks
(and before the first and after the last) must be the same length on
both sides; xdl_build_script() walks the two files in lockstep over
unchanged lines, so a balanced total is not enough.  Non-negative
counts and 1-based starts are instead caller preconditions, checked
with BUG(), since the caller normalizes hunks before this point.

On rejection xdl_diff() frees the environment it prepared and falls
through to xdl_do_diff(), which prepares a fresh one for the builtin
pass.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 16 +++++++++
 xdiff/xdiffi.c    | 84 +++++++++++++++++++++++++++++++++++++++++++++--
 xdiff/xprepare.c  | 10 ++++++
 xdiff/xprepare.h  |  1 +
 5 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index db6938689f..1fa16af668 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -124,7 +124,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..4736bcdb07 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,18 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs, in xdiff's own
+ * coordinates: line numbers are 1-based and a hunk's start is the
+ * first line it covers.  A caller translates any external "empty side"
+ * idiom (such as git diff's start-0/count-0) to a 1-based start before
+ * handing hunks over.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +100,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
+	struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index c5a892f91e..73a456f5dd 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,96 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  The caller normalizes and validates
+ * the hunks first (order, overlap, and lockstep alignment), so this
+ * only marks lines changed after asserting the memory-safety
+ * preconditions it depends on: non-negative counts and 1-based starts
+ * (checked with BUG()), and an in-bounds range (a silent -1 so the
+ * caller can fall back to the builtin diff rather than index changed[]
+ * out of range).  Keeping this diagnostic-free leaves user-facing
+ * messages to the git layer.
+ *
+ * Returns 0 on success, -1 if a hunk is out of range.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j;
+
+	/*
+	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
+	 * Clear them so only the external hunks are marked.
+	 */
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		struct xdl_hunk *h = &hunks[i];
+
+		/*
+		 * Non-negative counts and 1-based starts are caller
+		 * preconditions (it normalizes hunks into xdiff coordinates
+		 * before this point), so a violation is a bug, not a bad
+		 * tool response.
+		 */
+		if (h->old_count < 0 || h->new_count < 0)
+			BUG("external hunk %"PRIuMAX": "
+				"negative count (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_count, h->new_count);
+		if (h->old_start < 1 || h->new_start < 1)
+			BUG("external hunk %"PRIuMAX": "
+				"start not 1-based (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_start, h->new_start);
+
+		/*
+		 * The caller validates ordering, overlap and lockstep
+		 * alignment (and diagnoses a bad response).  This is only a
+		 * silent in-bounds guard so the marking loop cannot index
+		 * changed[] out of range: start + count - 1 <= nrec,
+		 * rewritten to avoid overflow.  A count of 0 (pure
+		 * insert/delete) allows start == nrec + 1, the position
+		 * after the last line.  On a miss, return -1 and let the
+		 * caller fall back to the builtin diff.
+		 */
+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1 ||
+		    h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+	}
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index 11bada2608..f4ab935332 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v5 3/9] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index b5412e6bc3..7547874aa2 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v5 4/9] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and
managing a hashmap that indexes running processes by command
string.  The hashmap suits callers like convert.c where many
files may share one filter process looked up by name, but
callers that manage process lifetime through their own data
structures do not need it.

Extract subprocess_start_command() and subprocess_stop_command()
so callers can reuse the child process setup and handshake
machinery without maintaining a hashmap.  subprocess_start()
and subprocess_stop() become thin wrappers that add hashmap
operations on top.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 28 +++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 2d5c965169..3cef42b088 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
 
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
+
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err)
+		return err;
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
gitgitgadget

[PATCH v5 5/9] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference line numbers in the content.
When textconv is also configured, the tool receives the
textconv-transformed content.  The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
normally.  A new "Which features consult the diff process"
documentation section lays out which features use the tool's hunks,
which compute independently, and why; the summary formats such as
--stat still use the builtin diff for now.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, the old and new blob object
names as old-oid/new-oid, and both file contents as packetized data.
The tool responds with hunk lines and a status packet (success,
error, or abort).  On error, Git warns and falls back to the builtin
diff algorithm for that file.  On abort, Git silently falls back for
the current file and stops sending further requests to the tool for
the remainder of the session.

old-oid/new-oid name the two blobs so a tool can cache its analysis
keyed on the pair.  A side's oid is sent only when the content the
tool receives is that raw blob: it is omitted under textconv, which
rewrites the bytes, and for a working-tree side with no stored
object, so an oid that is sent always names the bytes the tool
receives.  This is where the process protocol diverges from
diff.<driver>.command, which never composes with textconv (the
command replaces the whole diff and always gets the raw blob).  Tools
ignore unknown request keys, so old tools skip them.

When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
This also means --exit-code reports no changes for that file.

The subprocess is stored on the userdiff_driver struct and
launched on first use.  If the process fails to start, the
handshake fails, or a communication error occurs mid-stream,
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.

Git falls back to the builtin diff (rather than consulting the
tool) when an option the tool cannot honor is in effect: the
whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and
--anchored.  The bypass keys off the effective diff parameters (xpp)
rather than diffopt, so a later caller whose flags live elsewhere is
covered uniformly.  A change that only adds or removes the trailing
newline is likewise not expressible as hunks, so it too uses the
builtin diff.  The hunk parser ignores unknown trailing fields on a
hunk line for response forward-compatibility.

Hunk accumulation is bounded by the combined byte count of the two
files, so a misbehaving tool that floods hunk lines cannot grow
memory without bound before validation runs.

diff_process_fill_hunks() is the sole public entry point.  It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc       |   5 +
 Documentation/gitattributes.adoc     | 249 +++++++++++
 Makefile                             |   2 +
 diff-process.c                       | 491 ++++++++++++++++++++
 diff-process.h                       |  49 ++
 diff.c                               |  21 +
 diff.h                               |   3 +
 meson.build                          |   1 +
 t/helper/meson.build                 |   1 +
 t/helper/test-diff-process-backend.c | 381 ++++++++++++++++
 t/helper/test-tool.c                 |   1 +
 t/helper/test-tool.h                 |   1 +
 t/meson.build                        |   1 +
 t/t4080-diff-process.sh              | 645 +++++++++++++++++++++++++++
 userdiff.h                           |   3 +
 15 files changed, 1854 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..ac0635bb3b 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 2c4fbfd7f1..f4ca4a8c7e 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -831,6 +831,255 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If `diff.<name>.process` is defined, Git sends the old and new file
+content to an external tool and receives back a list of changed
+regions (pairs of line ranges in the old and new file).  Git uses
+these instead of its builtin diff algorithm, but still controls
+all output formatting, so features like word diff, function context,
+color, and blame work normally.  This is achieved by using the
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
+the diff process feeds results back into the standard pipeline.  If
+both are configured for a path, `diff.<name>.command` takes precedence
+for the patch output it replaces; the summary formats, `git blame`,
+and `git log -L` never run the command and still consult the process.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+Then, define a "diff.<name>.process" configuration to specify
+the diff process command.
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+When Git encounters the first file that needs to be diffed, it starts
+the process and performs the handshake.  In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
+described below).  The tool replies with "git-diff-server", the
+version it supports, and the capabilities it supports.
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
+data, each terminated with a flush packet.  The pathname is relative
+to the repository root.  When `diff.<name>.textconv` is also set,
+the tool receives the textconv-transformed content rather than the
+raw blob.  Git does not send binary files to the diff process.
+
+-----------------------
+packet:          git> command=hunks
+packet:          git> pathname=path/file.c
+packet:          git> old-oid=<hex>
+packet:          git> new-oid=<hex>
+packet:          git> 0000
+packet:          git> OLD_CONTENT
+packet:          git> 0000
+packet:          git> NEW_CONTENT
+packet:          git> 0000
+-----------------------
+
+The optional `old-oid` and `new-oid` keys give the object names of the
+old and new blobs, so a tool can cache its analysis keyed on the pair.
+A side's key is sent only when the content for that side is the raw
+blob it names: it is omitted when the content is textconv-transformed,
+and for a working-tree side that has no stored object.  A tool that
+does not recognize these keys ignores them.
+
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
+
+  `hunk <old_start> <old_count> <new_start> <new_count>`
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
+replacement range in the new file.  The four fields are separated by
+single spaces.  Start values are 1-based and counts are non-negative.
+For example, `hunk 3 2 3 4` means that 2 lines starting at line 3 in
+the old file were replaced by 4 lines starting at line 3 in the new
+file.  An `<old_count>` of 0 means no lines were removed (pure
+insertion); a `<new_count>` of 0 means no lines were added (pure
+deletion).  For a side with a count of 0 (a pure insertion or
+deletion) the start is the 1-based line the change sits before,
+ranging from 1 to one past the last line (the line count plus 1, to
+place the change at the end of the file); like every start it must
+keep the unchanged runs aligned on both sides (see below), so for a
+given change it takes one specific value, not an arbitrary one.  A
+start of 0 is also accepted and treated as 1, matching the
+empty-file-side form `git diff` emits (e.g. `hunk 0 0 1 5` for a newly
+added file).  A nonzero range must not extend beyond the end of the
+file.  Git ignores any extra
+whitespace-separated tokens after `<new_count>`, so a future protocol
+version can append fields to a hunk line (for example a "moved"
+marker) without older tools rejecting it.
+
+Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
+Hunks must be listed in order and must not overlap.  Any line not
+covered by a hunk is treated as unchanged and is paired, in order,
+with the unchanged lines on the other side.  Each run of unchanged
+lines between two hunks (and the run before the first hunk and
+after the last) must therefore be the same length on both sides,
+not merely equal in total.  For the hunks `1 3 1 5` and `10 2 12 2`
+below, lines 4-9 of the old file and lines 6-11 of the new file are
+both the six unchanged lines between the two hunks.  A response that
+balances only the total unchanged count but misaligns one of these
+runs is rejected, and Git falls back to the builtin diff.
+
+Git does not check that the lines a hunk leaves unchanged are
+byte-for-byte identical between the two sides; it pairs them by
+position and shows the new side as context.  A tool may therefore
+report lines that differ textually (a pure reformatting, say) as
+unchanged, and the diff reflects that judgment.  This is
+the point of a semantic backend, but it means a misbehaving tool can
+produce a diff whose context does not match the old blob; as with
+`git diff -w`, such a patch may not apply against the old content.
+
+-----------------------
+packet:          git< hunk 1 3 1 5
+packet:          git< hunk 10 2 12 2
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
+as changed and feeds them into the standard diff pipeline.  Git may
+still slide or regroup those changes against matching context for
+display, exactly as it compacts its own diffs, so the tool controls
+which lines are reported as changed, not the precise hunk boundaries.
+Patch output features (word diff, function context, color) work
+normally.  Summary formats such as `--stat` still compute their counts
+with the builtin diff for now; see "Which features consult the diff
+process" below for the full picture and the reasoning behind it.
+
+If no hunk lines precede the flush, followed by "success", Git
+treats the files as having no changes: `git diff` produces no output,
+`git diff --exit-code` and `--quiet` report success even though the
+stored blobs differ, and `git blame` skips the commit, attributing
+lines to earlier commits.
+The one exception is a change that only adds or removes the file's
+trailing newline: it cannot be expressed as line hunks, so when the
+line content otherwise matches Git keeps the builtin diff for that
+file (preserving the `\ No newline at end of file` marker) instead of
+treating the two sides as equal.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool returns well-formed but invalid hunks (out of bounds,
+overlapping, or with misaligned unchanged runs), Git warns and falls
+back to the builtin diff for that file; the tool stays available for
+subsequent files.  A malformed hunk line, by contrast (bad syntax, a
+nonzero count paired with a start of 0, or more hunks than the file
+has lines), is a protocol violation: Git stops the process and does
+not send it further requests, as described below.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status.  Git warns and
+falls back to the builtin diff algorithm for this file, treating any
+status other than "success" or "abort" the same way.  The tool
+remains available for subsequent files.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=error
+packet:          git< 0000
+-----------------------
+
+In case the tool cannot or does not want to process the content as
+well as any future content for the lifetime of the Git process, it
+is expected to respond with an "abort" status.  Git silently falls
+back to the builtin diff algorithm for this file and does not send
+further requests to the tool.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=abort
+packet:          git< 0000
+-----------------------
+
+If the tool dies during the communication or does not adhere to the
+protocol then Git will stop the process and fall back to the builtin
+diff algorithm.  Git warns once and does not restart the process for
+subsequent files.
+
+Tools should ignore unknown keys in the per-file request to remain
+forward-compatible.  Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
+
+Which features consult the diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The diff process answers a single question: given two blobs, which
+line ranges differ?  Whether a particular feature consults it follows
+from whether that is the question the feature is really asking.
+
+Features that ask "which lines changed" use the tool's hunks in place
+of the builtin algorithm:
+
+- `git diff` patch output, together with everything layered on it:
+  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
+  headers, and the `-L` line-range display.  These operate on the
+  lines the patch step already emitted, so they reflect the tool's
+  hunks without any further negotiation.
+- `git blame`: a commit whose change the tool reports as equivalent is
+  skipped, and its lines are attributed to an earlier commit.
+
+Features that ask a different question do not consult the process, by
+design:
+
+- The pickaxe `-G<regex>` searches the textual diff for a pattern; it
+  asks "does this string appear in the diff," not "did these lines
+  change."  (`-S` runs at an earlier stage and is likewise unaffected.)
+- `git patch-id` must produce a stable hash for `git rebase` and
+  cherry-pick detection; deriving it from a configured tool would make
+  equal patches hash differently from machine to machine.
+- The merge machinery (`git merge-tree`, `rerere`) computes merge
+  content and conflict signatures rather than display output, so the
+  tool's hunks must not alter its results.
+- `git range-diff` diffs patch text, not source blobs, so source-file
+  hunks do not apply to it.
+- `--check` reports whitespace errors in added lines using the builtin
+  diff's notion of which lines are added, not the tool's.  It can
+  therefore flag (and exit non-zero on) a line the tool treats as
+  unchanged and that `git diff` shows as context.  Whitespace breakage
+  is a property of the literal bytes, so `--check` keeps the builtin
+  partition deliberately; a future change could wire it to the tool if
+  matching `git diff` exactly became desirable.
+- `--raw`, `--name-only`, and `--name-status` compare object ids at
+  the tree level and never run a line-level diff at all.
+
+Some features ask "which lines changed" but still use the builtin
+algorithm for now, and may consult the process in a later change: the
+summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
+commit selection and parent range propagation (as distinct from its
+display, which is covered above); and combined diffs (`--cc` and merge
+diffs), whose protocol would have to be extended from a single old/new
+pair to one comparison per merge parent.
+
+`--diff-algorithm` bypasses the process entirely, for every feature
+listed above.  The whitespace-ignoring options (`-w`,
+`--ignore-space-change`, `--ignore-blank-lines`, and the like),
+`-I<regex>`, and `--anchored` also bypass it for the affected files:
+the tool is never told about these options, so it could not honor
+them, and Git falls back to the builtin diff, which does.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index 1cec251f43..1314c10463 100644
--- a/Makefile
+++ b/Makefile
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o
 TEST_BUILTINS_OBJS += test-date.o
 TEST_BUILTINS_OBJS += test-delete-gpgsig.o
 TEST_BUILTINS_OBJS += test-delta.o
+TEST_BUILTINS_OBJS += test-diff-process-backend.o
 TEST_BUILTINS_OBJS += test-dir-iterator.o
 TEST_BUILTINS_OBJS += test-drop-caches.o
 TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..4c748fdd2a
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,491 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  The tool controls which lines are marked as changed
+ * while the display shows the file content (after any textconv
+ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / [old-oid=<hex>] / [new-oid=<hex>] / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * When the tool returns no hunks with status=success, it considers
+ * the files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
+#include "hex.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *get_or_launch_process(
+		struct userdiff_driver *drv)
+{
+	struct diff_subprocess *entry;
+
+	if (drv->diff_subprocess)
+		return drv->diff_subprocess;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start_command(&entry->subprocess, drv->process,
+				     start_diff_process_fn)) {
+		free(entry);
+		drv->diff_process_failed = 1;
+		return NULL;
+	}
+
+	drv->diff_subprocess = entry;
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret = 0;
+
+	if (size < 0)
+		return -1;
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+/*
+ * A hunk in the diff process's presentation coordinates: the line
+ * numbering it reports over the protocol.  Kept distinct from struct
+ * xdl_hunk (xdiff's coordinates) so that only translated hunks ever
+ * reach the diff algorithm; diff_process_hunk_to_xdl() is the single
+ * crossing point.
+ */
+struct diff_process_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
+/*
+ * Parse one non-negative decimal field of a hunk line into *out and
+ * advance *line past it.  Fields must be plain decimal with no leading
+ * whitespace or sign (isdigit() takes an unsigned char to stay defined
+ * for high-bit bytes).  The first three fields are followed by a single
+ * space; the last (is_last) is followed by end-of-string or a space.
+ * Trailing space-separated tokens after the last field are allowed and
+ * ignored, so a future protocol version can append fields (e.g. a
+ * "moved" marker) without older tools rejecting the line -- mirroring
+ * the request-side rule that tools ignore unknown keys.
+ */
+static int parse_hunk_field(const char **line, long *out, int is_last)
+{
+	const char *p = *line;
+	char *end;
+
+	if (!isdigit((unsigned char)*p))
+		return -1;
+	errno = 0;
+	*out = strtol(p, &end, 10);
+	if (errno || end == p)
+		return -1;
+	if (is_last) {
+		if (*end != '\0' && *end != ' ')
+			return -1;
+	} else {
+		if (*end != ' ')
+			return -1;
+		end++;
+	}
+	*line = end;
+	return 0;
+}
+
+static int parse_hunk_line(const char *line,
+			   struct diff_process_hunk *presented)
+{
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+	if (parse_hunk_field(&line, &presented->old_start, 0) ||
+	    parse_hunk_field(&line, &presented->old_count, 0) ||
+	    parse_hunk_field(&line, &presented->new_start, 0) ||
+	    parse_hunk_field(&line, &presented->new_count, 1))
+		return -1;
+	return 0;
+}
+
+/*
+ * Translate a hunk from the diff process's presentation coordinates
+ * into xdiff's.
+ *
+ * Protocol starts are already 1-based positions (the line a change
+ * sits before), the same numbering xdiff uses, so the only adjustment
+ * is for an empty file side: "git diff" addresses it with a start of 0
+ * and a count of 0 (e.g. "0 0 1 5" adds five lines to an empty old
+ * side), and since xdiff uses start-1 as an array index that 0 becomes
+ * 1 here.  This is NOT the full inverse of xdl_emit_hunk_hdr()
+ * (xdiff/xutils.c): that emitter shifts a count-0 range to start-1 for
+ * the displayed "@@" header, but the protocol keeps the unshifted
+ * 1-based position for a mid-file insert or delete.  This is the single
+ * point where presentation coordinates become xdiff coordinates, so
+ * xdl_populate_hunks_from_external() may assume 1-based starts.
+ *
+ * Returns -1 for a start of 0 paired with a nonzero count, which names
+ * no line in either coordinate system.  (parse_hunk_line() already
+ * guarantees non-negative starts and counts.)
+ */
+static int diff_process_hunk_to_xdl(const struct diff_process_hunk *presented,
+				    struct xdl_hunk *xdl)
+{
+	long old_start = presented->old_start;
+	long new_start = presented->new_start;
+
+	if ((!old_start && presented->old_count) ||
+	    (!new_start && presented->new_count))
+		return -1;
+	if (!old_start)
+		old_start = 1;
+	if (!new_start)
+		new_start = 1;
+
+	xdl->old_start = old_start;
+	xdl->old_count = presented->old_count;
+	xdl->new_start = new_start;
+	xdl->new_count = presented->new_count;
+	return 0;
+}
+
+static enum diff_process_result get_hunks(
+		struct userdiff_driver *drv,
+		const char *path,
+		const char *old_buf, long old_size,
+		const char *new_buf, long new_size,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		struct xdl_hunk **hunks_out,
+		size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct diff_process_hunk presented;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	size_t max_hunks;
+	int len;
+	char *line;
+
+	backend = get_or_launch_process(drv);
+	if (!backend)
+		return DIFF_PROCESS_ERROR;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path))
+		goto comm_error;
+	/*
+	 * old-oid/new-oid let the tool key a cache on the blob pair.  A
+	 * side is sent only when its content is the raw blob (the caller
+	 * passes NULL otherwise, e.g. for textconv'd content), so an oid
+	 * that is present always names the bytes the tool receives.
+	 */
+	if (oid_a &&
+	    packet_write_fmt_gently(fd_in, "old-oid=%s\n", oid_to_hex(oid_a)))
+		goto comm_error;
+	if (oid_b &&
+	    packet_write_fmt_gently(fd_in, "new-oid=%s\n", oid_to_hex(oid_b)))
+		goto comm_error;
+	if (packet_flush_gently(fd_in))
+		goto comm_error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto comm_error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto comm_error;
+
+	/*
+	 * Hunks are non-overlapping and each useful hunk covers at least
+	 * one line, so a valid response cannot contain more hunks than the
+	 * two files have lines, which is bounded by their byte sizes.  Cap
+	 * the accumulation accordingly so a misbehaving tool that floods
+	 * hunk lines cannot drive unbounded memory growth before validation.
+	 */
+	max_hunks = (size_t)old_size + (size_t)new_size + 1;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &presented) < 0)
+			goto comm_error;
+		if (diff_process_hunk_to_xdl(&presented, &hunk) < 0)
+			goto comm_error;
+		if (nr_hunks >= max_hunks) {
+			warning(_("diff process '%s' sent too many hunks"
+				  " for '%s'"), drv->process, path);
+			goto comm_error;
+		}
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto comm_error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		*hunks_out = hunks;
+		*nr_hunks_out = nr_hunks;
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_OK;
+	}
+
+	if (!strcmp(status.buf, "abort")) {
+		/*
+		 * The tool voluntarily withdrew: stop sending requests
+		 * but do not warn (this is not a failure).
+		 */
+		backend->supported_capabilities &= ~CAP_HUNKS;
+		free(hunks);
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_SKIP;
+	}
+
+	/* status=error or unknown status */
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+
+comm_error:
+	/*
+	 * Communication failure (broken pipe, malformed response).
+	 * Tear down the process and mark as failed so we do not
+	 * retry on every subsequent file.
+	 */
+	drv->diff_process_failed = 1;
+	drv->diff_subprocess = NULL;
+	subprocess_stop_command(&backend->subprocess);
+	free(backend);
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+/*
+ * Whether exactly one of the two blobs ends in a newline.  A change
+ * that only adds or removes the trailing newline is not expressible as
+ * line hunks, so a tool comparing lines reports the files as equal.
+ */
+static int eof_newline_differs(const mmfile_t *a, const mmfile_t *b)
+{
+	int a_nl = a->size > 0 && a->ptr[a->size - 1] == '\n';
+	int b_nl = b->size > 0 && b->ptr[b->size - 1] == '\n';
+	return a_nl != b_nl;
+}
+
+/*
+ * Number of lines in a blob, matching xdiff's record count: one per
+ * newline, plus one more if the last line has no trailing newline.
+ */
+static long count_lines(const char *buf, long size)
+{
+	long lines = 0, i;
+
+	for (i = 0; i < size; i++)
+		if (buf[i] == '\n')
+			lines++;
+	if (size > 0 && buf[size - 1] != '\n')
+		lines++;
+	return lines;
+}
+
+/*
+ * Validate the tool's hunks (already in xdiff coordinates) against the
+ * two blobs before they bypass the diff algorithm.  Each hunk must fit
+ * within its file, the hunks must be ordered and non-overlapping, and
+ * the unchanged run before each hunk (and after the last) must be the
+ * same length on both sides -- xdl_build_script() walks the two files
+ * in lockstep over unchanged lines, so a mismatched gap desynchronizes
+ * it and yields a corrupt diff even when the totals balance.  This is
+ * the git layer's job so xdiff stays diagnostic-free; on a bad response
+ * we warn and the caller falls back to the builtin diff.  Returns 0 if
+ * valid, -1 (after warning) otherwise.
+ */
+static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,
+				   long old_lines, long new_lines,
+				   const char *process, const char *path)
+{
+	size_t i;
+	long prev_old_end = 0, prev_new_end = 0;
+
+	for (i = 0; i < nr; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count > old_lines - h->old_start + 1 ||
+		    h->new_count > new_lines - h->new_start + 1) {
+			warning(_("diff process '%s' returned a hunk past the "
+				  "end of '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		if (h->old_start < prev_old_end || h->new_start < prev_new_end) {
+			warning(_("diff process '%s' returned overlapping hunks "
+				  "for '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		if (h->old_start - prev_old_end != h->new_start - prev_new_end) {
+			warning(_("diff process '%s' returned hunks that leave "
+				  "'%s' misaligned; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+	}
+	if (old_lines - prev_old_end != new_lines - prev_new_end) {
+		warning(_("diff process '%s' returned hunks that leave '%s' "
+			  "misaligned; using the builtin diff"),
+			process, path);
+		return -1;
+	}
+	return 0;
+}
+
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+	struct xdl_hunk *ext_hunks = NULL;
+	size_t nr = 0;
+	enum diff_process_result res;
+
+	if (!diffopt || !path)
+		return DIFF_PROCESS_SKIP;
+	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+		return DIFF_PROCESS_SKIP;
+	/*
+	 * Whitespace-ignoring, regex-ignore (-I) and anchored options
+	 * change which lines count as different, but the tool is never
+	 * told about them, so its hunks could not honor them.  Rather
+	 * than silently override the user's request, fall back to the
+	 * builtin diff, which does honor these flags.  Key this off xpp
+	 * (the parameters this diff actually runs with) rather than
+	 * diffopt, so a caller like blame that keeps its flags outside
+	 * diffopt is covered without a separate guard of its own.
+	 */
+	if ((xpp->flags & (XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES)) ||
+	    xpp->ignore_regex_nr || xpp->anchors_nr)
+		return DIFF_PROCESS_SKIP;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return DIFF_PROCESS_SKIP;
+	if (drv->diff_process_failed)
+		return DIFF_PROCESS_SKIP;
+
+	res = get_hunks(drv, path,
+			file_a->ptr, file_a->size,
+			file_b->ptr, file_b->size,
+			oid_a, oid_b,
+			&ext_hunks, &nr);
+	if (res == DIFF_PROCESS_OK) {
+		if (!nr) {
+			free(ext_hunks);
+			/*
+			 * Zero hunks means the tool considers the line
+			 * content identical, but it cannot express a
+			 * trailing-newline-only change.  When that is the
+			 * actual difference, fall back to the builtin diff
+			 * so the "\ No newline at end of file" marker is
+			 * preserved instead of reporting the files equal.
+			 */
+			if (eof_newline_differs(file_a, file_b))
+				return DIFF_PROCESS_SKIP;
+			return DIFF_PROCESS_EQUIVALENT;
+		}
+		if (validate_external_hunks(ext_hunks, nr,
+					    count_lines(file_a->ptr, file_a->size),
+					    count_lines(file_b->ptr, file_b->size),
+					    drv->process, path) < 0) {
+			free(ext_hunks);
+			return DIFF_PROCESS_SKIP;
+		}
+		xpp->external_hunks = ext_hunks;
+		xpp->external_hunks_nr = nr;
+		return DIFF_PROCESS_OK;
+	}
+	if (res == DIFF_PROCESS_ERROR) {
+		warning(_("diff process '%s' failed for '%s',"
+			  " falling back to builtin diff"),
+			drv->process, path);
+		return DIFF_PROCESS_ERROR;
+	}
+	return DIFF_PROCESS_SKIP;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..8d00dafe1d
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,49 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+#include "xdiff/xdiff.h"
+
+struct diff_options;
+struct object_id;
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* failed; caller falls back to builtin */
+	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
+	DIFF_PROCESS_SKIP,       /* process did not apply: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
+/*
+ * Consult the diff process configured for 'path' and populate
+ * xpp->external_hunks with the returned hunks.
+ *
+ * Handles driver lookup, flag checks (--no-ext-diff,
+ * --diff-algorithm), subprocess management, and error reporting.
+ *
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks and
+ * the blobs are not a trailing-newline-only change (files are
+ * considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
+ *
+ * oid_a/oid_b, when non-NULL, are sent to the tool as old-oid/new-oid
+ * so it can key a cache on the blob pair.  Pass NULL for a side whose
+ * content is not the raw blob (e.g. textconv'd) or whose object name is
+ * unknown, so any oid that is sent always names the bytes the tool
+ * receives.
+ */
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index 2a9d0d8687..af31072858 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -4055,6 +4056,25 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		/*
+		 * Send the blob oids only for a side whose content is the
+		 * raw blob: textconv rewrites the bytes, and a working-tree
+		 * side has no stored oid, so pass NULL there rather than an
+		 * oid that would not name what the tool receives.
+		 */
+		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
+					    (textconv_one || !one->oid_valid) ? NULL : &one->oid,
+					    (textconv_two || !two->oid_valid) ? NULL : &two->oid,
+					    &xpp)
+		    == DIFF_PROCESS_EQUIVALENT) {
+			if (textconv_one)
+				free(mf1.ptr);
+			if (textconv_two)
+				free(mf2.ptr);
+			goto free_ab_and_return;
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4135,6 +4155,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(xpp.external_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/diff.h b/diff.h
index bb5cddaf34..7dc157968d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/** Disables diff.<driver>.process. */
+	unsigned no_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 3247697f74..aa532f5200 100644
--- a/meson.build
+++ b/meson.build
@@ -328,6 +328,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 3235f10ab8..6abcda4afb 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -12,6 +12,7 @@ test_tool_sources = [
   'test-date.c',
   'test-delete-gpgsig.c',
   'test-delta.c',
+  'test-diff-process-backend.c',
   'test-dir-iterator.c',
   'test-drop-caches.c',
   'test-dump-cache-tree.c',
diff --git a/t/helper/test-diff-process-backend.c b/t/helper/test-diff-process-backend.c
new file mode 100644
index 0000000000..c2ec532c4a
--- /dev/null
+++ b/t/helper/test-diff-process-backend.c
@@ -0,0 +1,381 @@
+/*
+ * Test backend for the long-running diff process protocol
+ * (see diff-process.c and Documentation/gitattributes.adoc).
+ *
+ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
+ *
+ * Implements the server side of the pkt-line handshake and a per-file
+ * response loop.  The --mode= switch selects the response shape
+ * (success, error, abort, crash, malformed hunks).
+ *
+ * Per-file request from Git:
+ *
+ *   packet:          git> command=hunks
+ *   packet:          git> pathname=<path>
+ *   packet:          git> [old-oid=<hex>]   (omitted for textconv/worktree)
+ *   packet:          git> [new-oid=<hex>]
+ *   packet:          git> 0000
+ *   packet:          git> OLD_CONTENT
+ *   packet:          git> 0000
+ *   packet:          git> NEW_CONTENT
+ *   packet:          git> 0000
+ *
+ * Response varies by --mode (default: whole-file):
+ *
+ *   whole-file   packet: git< hunk <1|0> <old_lines> <1|0> <new_lines>
+ *                (start is 0 for an empty side, matching git diff)
+ *   fixed-hunk   packet: git< hunk 5 2 5 2
+ *   no-hunks     (no hunk packets)
+ *   bad-hunk     packet: git< hunk 999 1 999 1
+ *   bad-parse    packet: git< garbage not a hunk
+ *   bad-sync     packet: git< hunk 1 2 1 1
+ *   bad-gap      packet: git< hunk 1 1 3 1
+ *   bad-start    packet: git< hunk 0 1 1 1
+ *   multi-hunk   packet: git< hunk 5 2 5 2
+ *                packet: git< hunk 9 2 9 2
+ *   insert       packet: git< hunk 3 0 3 2   (mid-file count-0 insertion)
+ *   flood        packet: git< hunk 1 1 1 1   (x100000)
+ *   overlap      packet: git< hunk 1 5 1 5
+ *                packet: git< hunk 3 2 3 2
+ *   no-cap       (omits capability=hunks during handshake)
+ *   error        (status=error instead of status=success)
+ *   abort        (status=abort instead of status=success)
+ *   crash        exit(1) before sending any response
+ *
+ * All success modes (not error/abort/crash) end with:
+ *
+ *   packet:          git< 0000
+ *   packet:          git< status=success
+ *   packet:          git< 0000
+ *
+ * Each request is logged to --log as:
+ *
+ *   command=<cmd> pathname=<path> old-oid=<hex> new-oid=<hex> old=<first line> new=<first line>
+ */
+
+#include "test-tool.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "strbuf.h"
+
+static FILE *logfile;
+
+enum mode {
+	MODE_WHOLE_FILE,
+	MODE_FIXED_HUNK,
+	MODE_NO_HUNKS,
+	MODE_BAD_HUNK,
+	MODE_BAD_PARSE,
+	MODE_BAD_SYNC,
+	MODE_BAD_GAP,
+	MODE_BAD_START,
+	MODE_MULTI_HUNK,
+	MODE_INSERT,
+	MODE_FLOOD,
+	MODE_OVERLAP,
+	MODE_NO_CAP,
+	MODE_ERROR,
+	MODE_ABORT,
+	MODE_CRASH,
+};
+
+static enum mode parse_mode(const char *s)
+{
+	if (!strcmp(s, "whole-file"))
+		return MODE_WHOLE_FILE;
+	if (!strcmp(s, "fixed-hunk"))
+		return MODE_FIXED_HUNK;
+	if (!strcmp(s, "no-hunks"))
+		return MODE_NO_HUNKS;
+	if (!strcmp(s, "bad-hunk"))
+		return MODE_BAD_HUNK;
+	if (!strcmp(s, "bad-parse"))
+		return MODE_BAD_PARSE;
+	if (!strcmp(s, "bad-sync"))
+		return MODE_BAD_SYNC;
+	if (!strcmp(s, "bad-gap"))
+		return MODE_BAD_GAP;
+	if (!strcmp(s, "bad-start"))
+		return MODE_BAD_START;
+	if (!strcmp(s, "multi-hunk"))
+		return MODE_MULTI_HUNK;
+	if (!strcmp(s, "insert"))
+		return MODE_INSERT;
+	if (!strcmp(s, "flood"))
+		return MODE_FLOOD;
+	if (!strcmp(s, "overlap"))
+		return MODE_OVERLAP;
+	if (!strcmp(s, "no-cap"))
+		return MODE_NO_CAP;
+	if (!strcmp(s, "error"))
+		return MODE_ERROR;
+	if (!strcmp(s, "abort"))
+		return MODE_ABORT;
+	if (!strcmp(s, "crash"))
+		return MODE_CRASH;
+	die("unknown --mode=%s", s);
+}
+
+/*
+ * Read "key=value" packets up to a flush, capturing "command" and
+ * "pathname".  Returns 1 if a request was read, 0 on EOF.
+ *
+ * The first packet uses the gentle variant so that a clean shutdown
+ * by Git (EOF) does not produce a spurious "the remote end hung up
+ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
+ * variant: once inside a request, truncation is a protocol violation
+ * and dying loudly is the correct response.
+ */
+static int read_request_header(char **command, char **pathname,
+			       char **old_oid, char **new_oid)
+{
+	int first = 1;
+	char *line;
+
+	*command = *pathname = *old_oid = *new_oid = NULL;
+	for (;;) {
+		const char *value;
+
+		if (first) {
+			if (packet_read_line_gently(0, NULL, &line) < 0)
+				return 0;
+			first = 0;
+		} else {
+			line = packet_read_line(0, NULL);
+		}
+		if (!line)
+			break;
+		if (skip_prefix(line, "command=", &value))
+			*command = xstrdup(value);
+		else if (skip_prefix(line, "pathname=", &value))
+			*pathname = xstrdup(value);
+		else if (skip_prefix(line, "old-oid=", &value))
+			*old_oid = xstrdup(value);
+		else if (skip_prefix(line, "new-oid=", &value))
+			*new_oid = xstrdup(value);
+	}
+	return 1;
+}
+
+static size_t count_lines(const struct strbuf *buf)
+{
+	size_t lines = 0;
+
+	for (size_t i = 0; i < buf->len; i++)
+		if (buf->buf[i] == '\n')
+			lines++;
+
+	return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
+}
+
+static void send_status(const char *status)
+{
+	packet_flush(1);
+	packet_write_fmt(1, "%s\n", status);
+	packet_flush(1);
+}
+
+static void respond(enum mode mode,
+		    const struct strbuf *old_buf,
+		    const struct strbuf *new_buf)
+{
+	switch (mode) {
+	case MODE_ERROR:
+		send_status("status=error");
+		return;
+	case MODE_ABORT:
+		send_status("status=abort");
+		return;
+	case MODE_CRASH:
+		exit(1);
+	case MODE_FIXED_HUNK:
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		break;
+	case MODE_BAD_HUNK:
+		packet_write_fmt(1, "hunk 999 1 999 1\n");
+		break;
+	case MODE_BAD_PARSE:
+		packet_write_fmt(1, "garbage not a hunk\n");
+		break;
+	case MODE_BAD_SYNC:
+		packet_write_fmt(1, "hunk 1 2 1 1\n");
+		break;
+	case MODE_BAD_GAP:
+		/*
+		 * Globally balanced (1 changed line on each side, so the
+		 * total unchanged counts match) but the gap before the
+		 * change differs between sides: old line 1 vs new line 3.
+		 * Exercises the per-gap lockstep-alignment check.
+		 */
+		packet_write_fmt(1, "hunk 1 1 3 1\n");
+		break;
+	case MODE_BAD_START:
+		/*
+		 * A start of 0 is valid only for an empty (count 0) range;
+		 * pairing it with a nonzero count names no line in either
+		 * the protocol's or xdiff's coordinates, so the translation
+		 * rejects it and git falls back to the builtin diff.
+		 */
+		packet_write_fmt(1, "hunk 0 1 1 1\n");
+		break;
+	case MODE_MULTI_HUNK:
+		/*
+		 * Two valid, non-overlapping, gap-aligned hunks.  Exercises
+		 * the accepting branch of the per-gap lockstep check with a
+		 * non-zero previous-hunk end (the realistic two-region case).
+		 */
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		packet_write_fmt(1, "hunk 9 2 9 2\n");
+		break;
+	case MODE_INSERT:
+		/*
+		 * A mid-file pure insertion (count 0 on the old side) in the
+		 * protocol's 1-based-position form: 2 lines inserted before
+		 * old line 3.  Exercises the count-0 path, which uses the
+		 * unshifted position (not git diff's "-3,0" display start).
+		 */
+		packet_write_fmt(1, "hunk 3 0 3 2\n");
+		break;
+	case MODE_FLOOD: {
+		/*
+		 * Emit far more hunks than any small file has lines, so Git
+		 * trips its accumulation cap and falls back before reading
+		 * them all.
+		 */
+		int i;
+		for (i = 0; i < 100000; i++)
+			packet_write_fmt(1, "hunk 1 1 1 1\n");
+		break;
+	}
+	case MODE_OVERLAP:
+		packet_write_fmt(1, "hunk 1 5 1 5\n");
+		packet_write_fmt(1, "hunk 3 2 3 2\n");
+		break;
+	case MODE_NO_HUNKS:
+		break;
+	case MODE_NO_CAP:
+	case MODE_WHOLE_FILE: {
+		size_t old_lines = count_lines(old_buf);
+		size_t new_lines = count_lines(new_buf);
+		/*
+		 * Match git diff output: start=0 when count=0
+		 * (empty file side), 1 otherwise.
+		 */
+		packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
+				 " %"PRIuMAX" %"PRIuMAX"\n",
+				 (uintmax_t)(old_lines ? 1 : 0),
+				 (uintmax_t)old_lines,
+				 (uintmax_t)(new_lines ? 1 : 0),
+				 (uintmax_t)new_lines);
+		break;
+	}
+	}
+	send_status("status=success");
+}
+
+static void command_loop(enum mode mode)
+{
+	for (;;) {
+		char *command = NULL, *pathname = NULL;
+		char *old_oid = NULL, *new_oid = NULL;
+		struct strbuf obuf = STRBUF_INIT;
+		struct strbuf nbuf = STRBUF_INIT;
+
+		if (!read_request_header(&command, &pathname,
+					 &old_oid, &new_oid))
+			break; /* EOF: Git closed its end */
+
+		read_packetized_to_strbuf(0, &obuf, 0);
+		read_packetized_to_strbuf(0, &nbuf, 0);
+
+		if (logfile) {
+			fprintf(logfile,
+				"command=%s pathname=%s old-oid=%s new-oid=%s"
+				" old=%.*s new=%.*s\n",
+				command ? command : "(none)",
+				pathname ? pathname : "(none)",
+				old_oid ? old_oid : "(none)",
+				new_oid ? new_oid : "(none)",
+				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
+				obuf.buf,
+				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
+				nbuf.buf);
+			fflush(logfile);
+		}
+
+		respond(mode, &obuf, &nbuf);
+
+		free(command);
+		free(pathname);
+		free(old_oid);
+		free(new_oid);
+		strbuf_release(&obuf);
+		strbuf_release(&nbuf);
+	}
+}
+
+static void handshake(enum mode mode)
+{
+	char *line;
+
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "git-diff-client"))
+		die("bad welcome: '%s'", line ? line : "(eof)");
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "version=1"))
+		die("bad version: '%s'", line ? line : "(eof)");
+	if (packet_read_line(0, NULL))
+		die("expected flush after version");
+
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Drain capabilities advertised by Git */
+	while ((line = packet_read_line(0, NULL)))
+		; /* drain */
+
+	/* Respond with our capabilities (or none for no-cap mode) */
+	if (mode != MODE_NO_CAP)
+		packet_write_fmt(1, "capability=hunks\n");
+	packet_flush(1);
+}
+
+static const char *const usage_str[] = {
+	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
+	NULL
+};
+
+int cmd__diff_process_backend(int argc, const char **argv)
+{
+	const char *mode_str = NULL, *log_path = NULL;
+	enum mode mode = MODE_WHOLE_FILE;
+	struct option options[] = {
+		OPT_STRING(0, "mode", &mode_str, "mode",
+			   "response shape (default whole-file);"
+			   " see the file header for the full list of modes"),
+		OPT_STRING(0, "log", &log_path, "path",
+			   "append per-request summary to this file"),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
+	if (argc)
+		usage_with_options(usage_str, options);
+
+	if (mode_str)
+		mode = parse_mode(mode_str);
+
+	if (log_path) {
+		logfile = fopen(log_path, "a");
+		if (!logfile)
+			die_errno("failed to open log '%s'", log_path);
+	}
+
+	handshake(mode);
+	command_loop(mode);
+
+	if (logfile && fclose(logfile))
+		die_errno("error closing log");
+	return 0;
+}
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index b71a22b43b..3c3f95269c 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = {
 	{ "date", cmd__date },
 	{ "delete-gpgsig", cmd__delete_gpgsig },
 	{ "delta", cmd__delta },
+	{ "diff-process-backend", cmd__diff_process_backend },
 	{ "dir-iterator", cmd__dir_iterator },
 	{ "drop-caches", cmd__drop_caches },
 	{ "dump-cache-tree", cmd__dump_cache_tree },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f2885b33d5..a5bb755516 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);
 int cmd__date(int argc, const char **argv);
 int cmd__delta(int argc, const char **argv);
 int cmd__delete_gpgsig(int argc, const char **argv);
+int cmd__diff_process_backend(int argc, const char **argv);
 int cmd__dir_iterator(int argc, const char **argv);
 int cmd__drop_caches(int argc, const char **argv);
 int cmd__dump_cache_tree(int argc, const char **argv);
diff --git a/t/meson.build b/t/meson.build
index 3219264fe7..6afbfa6a87 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -512,6 +512,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..3b75df082e
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,645 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+# See t/helper/test-diff-process-backend.c for the backend implementation
+# and available --mode= options.
+
+BACKEND="test-tool diff-process-backend"
+
+test_expect_success 'setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# boundary.c: 10 lines, changes at 5-6 and 9-10.
+	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+
+	# worddiff.c: single-line function, value changes 1 -> 999.
+	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+
+	# newfile.c: single-line function, value changes 42 -> 99.
+	# Used by: modified file, --exit-code, multiple drivers.
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 42; }
+	EOF
+	git add newfile.c &&
+
+	# logtest.c: single-line function for log/format-patch tests.
+	# Needs two commits so log -1 has a diff.
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+
+	# one.c/two.c: two-file pair for error/abort/startup-failure tests.
+	cat >one.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >two.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add one.c two.c &&
+
+	git commit -m "initial" &&
+
+	# Second commit for logtest.c (so log -1 has something to show).
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	# Working tree modifications (not committed).
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 99; }
+	EOF
+
+	cat >one.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+
+	cat >two.c <<-\EOF
+	int second(void) { return 20; }
+	EOF
+'
+
+#
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
+test_expect_success 'diff process hunk boundaries affect output' '
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	test_grep ! "^-OLD9" actual &&
+	test_grep ! "^-OLD10" actual &&
+	test_grep ! "^+NEW9" actual &&
+	test_grep ! "^+NEW10" actual
+'
+
+test_expect_success 'diff process accepts valid multi-hunk output' '
+	# multi-hunk reports both changed regions (5-6 and 9-10) as two
+	# gap-aligned hunks.  This exercises the accepting branch of the
+	# per-gap lockstep check (non-zero previous-hunk end) and must
+	# produce a correct two-region diff with the lines between the
+	# hunks kept as context.
+	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "^ line7" actual &&
+	test_grep "^ line8" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process accepts a mid-file count-0 insertion' '
+	# insert mode reports "hunk 3 0 3 2": a pure insertion (count 0 on
+	# the old side) in the protocol 1-based-position form.  Exercises
+	# the count-0 hunk path that the other valid-hunk modes (full
+	# replacements, equal-count modifies) never hit.  Empty stderr is
+	# the discriminator: a mishandled count-0 start would be rejected
+	# by the lockstep check and warn.
+	cat >insert.c <<-\EOF &&
+	a
+	b
+	c
+	d
+	e
+	EOF
+	git add insert.c &&
+	git commit -m "add insert.c" &&
+	cat >insert.c <<-\EOF &&
+	a
+	b
+	X
+	Y
+	c
+	d
+	e
+	EOF
+	git -c diff.cdiff.process="$BACKEND --mode=insert" \
+		diff insert.c >actual 2>stderr &&
+	test_grep "^+X" actual &&
+	test_grep "^+Y" actual &&
+	test_grep "^ c" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with modified file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- newfile.c >actual 2>stderr &&
+	test_grep "return 99" actual &&
+	test_grep "pathname=newfile.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with added file (empty old side)' '
+	cat >added.c <<-\EOF &&
+	int added(void) { return 1; }
+	EOF
+	git add added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "added" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with deleted file (empty new side)' '
+	git add added.c &&
+	git commit -m "commit added.c" &&
+	git rm added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "deleted file" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process skipped for binary files' '
+	printf "\\0binary" >binary.c &&
+	git add binary.c &&
+	git commit -m "add binary" &&
+	printf "\\0changed" >binary.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- binary.c >actual &&
+	test_grep "Binary files" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not consulted for unmatched driver' '
+	echo "not tracked by cdiff" >unmatched.txt &&
+	git add unmatched.txt &&
+	git commit -m "add unmatched.txt" &&
+
+	echo "modified" >unmatched.txt &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- unmatched.txt >actual &&
+	test_grep "modified" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'multiple drivers use separate processes' '
+	echo "*.h diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 1; }
+	EOF
+	git add multi.h &&
+	git commit -m "add multi.h" &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 2; }
+	EOF
+
+	test_when_finished "rm -f backend-c.log backend-h.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+		diff -- newfile.c multi.h >actual 2>stderr &&
+	test_grep "pathname=newfile.c" backend-c.log &&
+	test_grep "pathname=multi.h" backend-h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works alongside textconv' '
+	write_script uppercase-filter <<-\EOF &&
+	tr "a-z" "A-Z" <"$1"
+	EOF
+
+	cat >textconv.c <<-\EOF &&
+	hello world
+	EOF
+	git add textconv.c &&
+	git commit -m "add textconv.c" &&
+
+	cat >textconv.c <<-\EOF &&
+	goodbye world
+	EOF
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- textconv.c >actual 2>stderr &&
+	# The diff process receives textconv-transformed (uppercase) content.
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=HELLO WORLD" backend.log &&
+	test_grep "new=GOODBYE WORLD" backend.log &&
+	test_must_be_empty stderr
+'
+
+#
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
+test_expect_success 'diff process with --word-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --word-diff worddiff.c >actual 2>stderr &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual &&
+	test_grep "pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with git log -p' '
+	# With no-hunks mode, the tool says the files are equivalent,
+	# so log -p should show the commit but no diff content.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -1 -p -- logtest.c >actual 2>stderr &&
+	test_grep "change logtest.c" actual &&
+	test_grep ! "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process no hunks suppresses diff output' '
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add nohunks.c &&
+	git commit -m "add nohunks.c" &&
+
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff nohunks.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success 'diff process no hunks with --exit-code returns success' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code nohunks.c
+'
+
+test_expect_success 'diff process equivalent commit: --exit-code and --quiet agree' '
+	# A committed blob pair (not a worktree file) whose oids differ but
+	# the tool reports equivalent.  --exit-code and --quiet must agree
+	# with the shown diff (empty) and report success, not fall back to
+	# the byte-level "oids differ" answer.
+	cat >ecq.c <<-\EOF &&
+	alpha
+	EOF
+	git add ecq.c &&
+	git commit -m "ecq v1" &&
+	cat >ecq.c <<-\EOF &&
+	beta
+	EOF
+	git add ecq.c &&
+	git commit -m "ecq v2" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code HEAD^ HEAD -- ecq.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --quiet HEAD^ HEAD -- ecq.c
+'
+
+test_expect_success 'diff process falls back for trailing-newline-only change' '
+	test_when_finished "rm -f backend.log" &&
+	printf "a\nb\nc\n" >eofnl.c &&
+	git add eofnl.c &&
+	git commit -m "add eofnl.c" &&
+	printf "a\nb\nc" >eofnl.c &&
+	# Same lines, only the final newline removed.  The tool reports
+	# no hunks (it sees identical lines), but that change is not
+	# expressible as hunks, so git falls back to the builtin diff
+	# rather than treating the files as equivalent.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff eofnl.c >actual 2>stderr &&
+	test_grep "No newline at end of file" actual &&
+	test_grep "pathname=eofnl.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process falls back for added file (empty old side)' '
+	test_when_finished "rm -f backend.log" &&
+	printf "x\ny\nz\n" >addnl.c &&
+	git add addnl.c &&
+	# The empty old side has no trailing newline while the new side
+	# does, so the newline fallback shows the addition rather than
+	# letting no-hunks suppress the whole new file.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --cached addnl.c >actual 2>stderr &&
+	test_grep "^+x" actual &&
+	test_grep "pathname=addnl.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process with --exit-code and hunks returns failure' '
+	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+		diff --exit-code newfile.c
+'
+
+#
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
+test_expect_success 'diff process bypassed by --diff-algorithm' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
+	test_when_finished "rm -f backend.log" &&
+	printf "a\nb\nc\n" >wsbypass.c &&
+	git add wsbypass.c &&
+	git commit -m "add wsbypass.c" &&
+	printf "a\n  b  \nc\n" >wsbypass.c &&
+	# The tool is never told about these options and could not honor
+	# them, so git bypasses the process for each (covering the whole
+	# XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES mask, not just -w).
+	for opt in -w -b --ignore-space-at-eol --ignore-blank-lines
+	do
+		rm -f backend.log &&
+		git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+			diff $opt wsbypass.c >actual 2>stderr &&
+		test_path_is_missing backend.log &&
+		test_must_be_empty stderr ||
+		return 1
+	done &&
+	# -w additionally suppresses the whitespace-only change via the
+	# builtin diff that now runs.
+	git -c diff.cdiff.process="$BACKEND" diff -w wsbypass.c >actual &&
+	test_must_be_empty actual
+'
+
+#
+# Error handling and fallback.
+#
+
+test_expect_success 'diff process fallback on tool error status' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process error keeps tool available for next file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Unlike abort, error keeps the tool available: both files
+	# are sent to the tool (and both fall back).
+	test_grep "pathname=one.c" backend.log &&
+	test_grep "pathname=two.c" backend.log &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process abort disables for session' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool.
+	test_grep "pathname=one.c" backend.log &&
+	test_grep ! "pathname=two.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Crash is a communication failure, so a warning is emitted.
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process startup failure only warns once' '
+	git -c diff.cdiff.process="/nonexistent/tool" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# Sentinel prevents repeated warnings: only one, not one per file.
+	test_grep "diff process.*failed" stderr >warnings &&
+	test_line_count = 1 warnings
+'
+
+
+test_expect_success 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "hunk past the end" stderr
+'
+
+test_expect_success 'diff process fallback on mismatched unchanged totals' '
+	cat >synctest.c <<-\EOF &&
+	line1
+	line2
+	line3
+	EOF
+	git add synctest.c &&
+	git commit -m "add synctest.c" &&
+
+	cat >synctest.c <<-\EOF &&
+	line1
+	changed
+	line3
+	EOF
+
+	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
+	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
+	# The synchronization invariant fails and git falls back.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+		diff synctest.c >actual 2>stderr &&
+	test_grep "changed" actual &&
+	test_grep "misaligned" stderr
+'
+
+test_expect_success 'diff process fallback on misaligned hunk gap' '
+	# bad-gap reports hunk 1 1 3 1 on boundary.c: one changed line
+	# on each side, so the total unchanged counts match, but the
+	# unchanged run before the change differs (old line 1 vs new
+	# line 3).  A global count check would accept this and emit a
+	# corrupt diff; the per-gap lockstep check rejects it and git
+	# falls back to the builtin algorithm.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-gap" \
+		diff boundary.c >actual 2>stderr &&
+	# The builtin fallback shows both changed regions as additions
+	# (a corrupt-accepted hunk would show NEW5 only as context).
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "misaligned" stderr
+'
+
+test_expect_success 'diff process fallback on overlapping hunks' '
+	# boundary.c has 10 lines, so both hunks are in bounds
+	# but they overlap at lines 3-4, triggering the ordering check.
+	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "NEW5" actual &&
+	test_grep "overlapping hunks" stderr
+'
+
+test_expect_success 'diff process fallback on malformed hunk line' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual
+'
+
+test_expect_success 'diff process fallback on start 0 with nonzero count' '
+	# bad-start reports hunk 0 1 1 1.  A start of 0 is valid only for
+	# an empty (count 0) range, so the presentation-to-xdiff
+	# translation rejects it and git falls back to the builtin diff
+	# instead of handing xdiff an out-of-range start.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-start" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process caps a flood of hunks and falls back' '
+	# flood emits far more hunks than the file has lines.  Git must
+	# stop accumulating and fall back to the builtin diff rather than
+	# grow memory without bound.
+	git -c diff.cdiff.process="$BACKEND --mode=flood" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "too many hunks" stderr
+'
+
+test_expect_success 'diff process skipped when tool omits capability' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-cap --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Builtin diff runs: all changes appear, including lines 9-10
+	# that a tool-provided hunk would have narrowed away.
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD9" actual &&
+	# The process launched (creating the log) but was
+	# never sent a per-file request, so no hunks command is logged.
+	test_path_is_file backend.log &&
+	test_grep ! "command=hunks" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process receives old-oid and new-oid for a blob pair' '
+	test_when_finished "rm -f backend.log" &&
+	cat >oidpair.c <<-\EOF &&
+	int f(void) { return 1; }
+	EOF
+	git add oidpair.c &&
+	git commit -m "oidpair v1" &&
+	old=$(git rev-parse HEAD:oidpair.c) &&
+
+	cat >oidpair.c <<-\EOF &&
+	int f(void) { return 2; }
+	EOF
+	git add oidpair.c &&
+	git commit -m "oidpair v2" &&
+	new=$(git rev-parse HEAD:oidpair.c) &&
+
+	# Both sides are stored blobs, so their object names are sent.
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff HEAD^ HEAD -- oidpair.c >actual 2>stderr &&
+	test_grep "old-oid=$old new-oid=$new" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process omits old-oid and new-oid for textconv content' '
+	test_when_finished "rm -f backend.log" &&
+	write_script oidcat <<-\EOF &&
+	cat "$1"
+	EOF
+	cat >oidtc.c <<-\EOF &&
+	alpha
+	EOF
+	git add oidtc.c &&
+	git commit -m "oidtc v1" &&
+	cat >oidtc.c <<-\EOF &&
+	beta
+	EOF
+	git add oidtc.c &&
+	git commit -m "oidtc v2" &&
+
+	# textconv rewrites the bytes, so the raw-blob object name that
+	# would otherwise identify each side is omitted.
+	git -c diff.cdiff.textconv="./oidcat" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff HEAD^ HEAD -- oidtc.c >actual 2>stderr &&
+	test_grep "pathname=oidtc.c" backend.log &&
+	test_grep "old-oid=(none) new-oid=(none)" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_done
diff --git a/userdiff.h b/userdiff.h
index 51c26e0d41..a98eabe377 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -3,6 +3,7 @@
 
 #include "notes-cache.h"
 
+struct diff_subprocess;
 struct index_state;
 struct repository;
 
@@ -33,6 +34,8 @@ struct userdiff_driver {
 	int textconv_want_cache;
 	const char *process;
 	char *process_owned;
+	struct diff_subprocess *diff_subprocess;
+	unsigned diff_process_failed : 1;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v5 6/9] diff: bypass diff process with --no-ext-diff and in format-patch

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Make --no-ext-diff disable diff.<driver>.process in addition to
diff.<driver>.command.  Although the two mechanisms work differently
(command replaces Git's output, process feeds hunks back into the
pipeline), both invoke external tools and --no-ext-diff means
"no external tools."

Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that
sets both allow_external and no_diff_process, so a single option
controls both.  Passing --ext-diff explicitly clears
no_diff_process, so a later --ext-diff overrides an earlier
--no-ext-diff.

Disable the diff process unconditionally in format-patch so that
generated patches are always based on the builtin diff algorithm
and can be applied reliably by recipients who do not have the
external tool.

Document that --diff-algorithm also bypasses the diff process,
since it forces the builtin algorithm.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/diff-algorithm-option.adoc |  3 +++
 Documentation/diff-options.adoc          |  4 +++-
 Documentation/gitattributes.adoc         |  6 +++---
 builtin/log.c                            |  7 +++++++
 diff.c                                   | 16 ++++++++++++++--
 diff.h                                   |  5 ++++-
 t/t4080-diff-process.sh                  | 16 ++++++++++++++++
 7 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..4d7e2ec35f 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index c8242e2462..a884445211 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -833,7 +833,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f4ca4a8c7e..a03fb9deb1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -1073,9 +1073,9 @@ display, which is covered above); and combined diffs (`--cc` and merge
 diffs), whose protocol would have to be extended from a single old/new
 pair to one comparison per merge parent.
 
-`--diff-algorithm` bypasses the process entirely, for every feature
-listed above.  The whitespace-ignoring options (`-w`,
-`--ignore-space-change`, `--ignore-blank-lines`, and the like),
+`--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
+for every feature listed above.  The whitespace-ignoring options
+(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like),
 `-I<regex>`, and `--anchored` also bypass it for the affected files:
 the tool is never told about these options, so it could not honor
 them, and Git falls back to the builtin diff, which does.
diff --git a/builtin/log.c b/builtin/log.c
index d027ce1e0b..7821a61143 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * Disable diff.<driver>.process so that patches generated by
+	 * format-patch are always based on the builtin diff algorithm
+	 * and can be applied reliably.
+	 */
+	rev.diffopt.flags.no_diff_process = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff.c b/diff.c
index af31072858..a9a732629e 100644
--- a/diff.c
+++ b/diff.c
@@ -5948,6 +5948,17 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.no_diff_process = unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6278,8 +6289,9 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index 7dc157968d..ee034d240d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,7 +173,10 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
-	/** Disables diff.<driver>.process. */
+	/**
+	 * Disables diff.<driver>.process.  Set by --no-ext-diff and by
+	 * format-patch.
+	 */
 	unsigned no_diff_process;
 
 	/**
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 3b75df082e..7e71b70ab9 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -398,6 +398,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' '
 	test_path_is_missing backend.log
 '
 
+test_expect_success 'diff process bypassed by --no-ext-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --no-ext-diff worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not used by format-patch' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		format-patch -1 --stdout -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_path_is_missing backend.log
+'
+
 test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
 	test_when_finished "rm -f backend.log" &&
 	printf "a\nb\nc\n" >wsbypass.c &&
-- 
gitgitgadget

[PATCH v5 7/9] blame: consult diff process for no-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

Introduce xdi_diff_process(), a process-aware xdi_diff() that
consults the process, runs xdiff on the tool's hunks or on the
builtin algorithm when it does not apply, frees the hunks, and
reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller
can drop or skip the change.  It is the shared consult-then-diff path
for consumers that work on raw hunks: blame's pass_blame_to_parent()
uses it here, and git log -L reuses it later.  builtin_diff() keeps
consulting the process directly, because it tests for equivalence
early, before its funcname-pattern and word-diff setup, so a
reformat-only file short-circuits without that work.

Blame's -w option is not communicated to the process and it could not
honor it, so blame must fall back to the builtin diff there.  Because
blame keeps its whitespace flags in sb->xdl_opts rather than diffopt,
the process bypass keys off xpp (the flags the diff actually runs
with), which covers blame without a guard of its own.

The subprocess is long-running (one startup cost amortized across the
blame traversal), but each commit in the file's history incurs a
round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                 |  24 +++++++-
 diff-process.c          |  38 ++++++++++++
 diff-process.h          |  26 +++++++++
 t/t4080-diff-process.sh | 126 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 213 insertions(+), 1 deletion(-)

diff --git a/blame.c b/blame.c
index 126e232416..932a04c4e8 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "xdiff-interface.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -1946,6 +1948,9 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *parent, int ignore_diffs)
 {
 	mmfile_t file_p, file_o;
+	xpparam_t xpp = {0};
+	xdemitconf_t xecfg = {0};
+	xdemitcb_t ecb = {NULL};
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
 
@@ -1964,7 +1969,24 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	xpp.flags = sb->xdl_opts;
+	xecfg.hunk_func = blame_chunk_cb;
+	ecb.priv = &d;
+	/*
+	 * Consult the diff process, then attribute the resulting chunks
+	 * via blame_chunk_cb.  It bypasses the process for the whitespace-
+	 * ignoring options it cannot honor (they live in xpp.flags, which
+	 * the consultation checks), and when the process reports the blobs
+	 * equivalent it runs no diff, so blame passes this commit and looks
+	 * past it.  Look up the driver by the parent (old) path, as
+	 * builtin_diff() does with name_a, so a renamed file resolves to the
+	 * same driver across diff, blame, and line-log.  Pass no
+	 * old-oid/new-oid: blame diffs each blob pair once, so the tool gains
+	 * nothing from a per-invocation cache key.
+	 */
+	if (xdi_diff_process(&sb->revs->diffopt, parent->path,
+			     &file_p, &file_o, NULL, NULL, &xpp, &xecfg, &ecb)
+	    == DIFF_PROCESS_ERROR)
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
diff --git a/diff-process.c b/diff-process.c
index 4c748fdd2a..191b2b67b2 100644
--- a/diff-process.c
+++ b/diff-process.c
@@ -37,6 +37,7 @@
 #include "sub-process.h"
 #include "pkt-line.h"
 #include "strbuf.h"
+#include "xdiff-interface.h"
 #include "xdiff/xdiff.h"
 
 #define CAP_HUNKS (1u << 0)
@@ -489,3 +490,40 @@ enum diff_process_result diff_process_fill_hunks(
 	}
 	return DIFF_PROCESS_SKIP;
 }
+
+enum diff_process_result xdi_diff_process(
+		struct diff_options *diffopt,
+		const char *path,
+		mmfile_t *file_a,
+		mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp,
+		xdemitconf_t *xecfg,
+		xdemitcb_t *ecb)
+{
+	enum diff_process_result res;
+
+	/*
+	 * Consult the diff process, then run xdiff either constrained to
+	 * the tool's hunks or, when the process does not apply, computing
+	 * the diff itself as a fallback.  EQUIVALENT short-circuits: the
+	 * caller decides what "no change" means for it (drop the commit,
+	 * skip the file, ...), so xdiff is not run.
+	 *
+	 * A SKIP/ERROR from the process just selects the builtin path
+	 * (its warning, if any, was already emitted), so the result then
+	 * reflects whether xdiff itself succeeded, not the process.
+	 */
+	res = diff_process_fill_hunks(diffopt, path, file_a, file_b,
+				      oid_a, oid_b, xpp);
+	if (res == DIFF_PROCESS_EQUIVALENT)
+		return res;
+
+	res = xdi_diff(file_a, file_b, xpp, xecfg, ecb) < 0
+		? DIFF_PROCESS_ERROR : DIFF_PROCESS_OK;
+
+	FREE_AND_NULL(xpp->external_hunks);
+	xpp->external_hunks_nr = 0;
+	return res;
+}
diff --git a/diff-process.h b/diff-process.h
index 8d00dafe1d..5e5b514b77 100644
--- a/diff-process.h
+++ b/diff-process.h
@@ -46,4 +46,30 @@ enum diff_process_result diff_process_fill_hunks(
 		const struct object_id *oid_b,
 		xpparam_t *xpp);
 
+/*
+ * Process-aware xdi_diff(): consult the diff process for 'path', then
+ * run xdiff either constrained to the tool's hunks or computing the
+ * diff itself when the process does not apply or fails.  Frees any
+ * hunks it obtained before returning.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool
+ * reports the blobs equal, so the caller can drop or skip the change;
+ * DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and
+ * DIFF_PROCESS_ERROR if xdiff itself errored.
+ *
+ * The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as
+ * for a direct xdi_diff() call.  oid_a/oid_b are forwarded to
+ * diff_process_fill_hunks() (see there).
+ */
+enum diff_process_result xdi_diff_process(
+		struct diff_options *diffopt,
+		const char *path,
+		mmfile_t *file_a,
+		mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp,
+		xdemitconf_t *xecfg,
+		xdemitcb_t *ecb);
+
 #endif /* DIFF_PROCESS_H */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 7e71b70ab9..694c94edb2 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -658,4 +658,130 @@ test_expect_success 'diff process omits old-oid and new-oid for textconv content
 	test_must_be_empty stderr
 '
 
+#
+# Blame integration.
+#
+
+test_expect_success 'blame uses tool-provided hunks' '
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "add blame-hunk.c" &&
+	ORIG=$(git rev-parse --short HEAD) &&
+
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "change blame-hunk.c" &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+
+	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
+	# so blame should attribute lines 9-10 to the original commit
+	# even though the builtin diff would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		blame blame-hunk.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	sed -n "6p" actual >line6 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "$CHANGE" line6
+'
+
+test_expect_success 'blame skips commits with no hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void) {
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without no-hunks mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With no-hunks mode, the process considers the files equivalent
+	# and blame skips the reformat commit, attributing to the original.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		blame blame.c >with &&
+	test_grep ! "$BLAME_COMMIT" with &&
+	test_grep "$ORIG_COMMIT" with
+'
+
+test_expect_success 'blame --no-ext-diff bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		blame --no-ext-diff blame.c >actual &&
+	# Without the process, blame attributes the reformat commit normally.
+	test_grep "$BLAME_COMMIT" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame --no-ext-diff uses builtin hunks' '
+	# fixed-hunk mode would narrow blame to lines 5-6, but
+	# --no-ext-diff should bypass it and use the builtin diff.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		blame --no-ext-diff blame-hunk.c >actual &&
+	# Builtin diff attributes lines 9-10 to the change commit.
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame -w bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	printf "alpha\nbeta\ngamma\n" >blamew.c &&
+	git add blamew.c &&
+	git commit -m "add blamew.c" &&
+	orig=$(git rev-parse --short HEAD) &&
+	printf "alpha\n   beta   \ngamma\n" >blamew.c &&
+	git commit -am "reindent beta" &&
+	reindent=$(git rev-parse --short HEAD) &&
+	# blame -w must ignore the whitespace-only change and attribute
+	# beta to the original commit, not the reindent commit.  The tool
+	# is never told about -w, so blame must bypass it (not let tool
+	# hunks override -w).
+	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=backend.log" \
+		blame -w blamew.c >actual &&
+	sed -n "2p" actual >line2 &&
+	test_grep "$orig" line2 &&
+	test_grep ! "$reindent" line2 &&
+	test_path_is_missing backend.log
+'
+
 test_done
-- 
gitgitgadget

[PATCH v5 8/9] diff: consult diff process for --stat counts

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

builtin_diff() already consults a configured diff.<driver>.process: a
file the tool reports as equivalent emits no patch, and otherwise the
tool's hunks drive the output.  builtin_diffstat() ran its own xdiff
and ignored the process, so "git diff --stat" still counted a
byte-level change for a file that "git diff" showed as unchanged.

Consult diff_process_fill_hunks() before the stat xdiff, as
builtin_diff() does.  On DIFF_PROCESS_EQUIVALENT, skip the xdiff so
the file keeps its zero inserted and deleted counts and the existing
"nothing changed" pruning drops it, matching the empty patch.
Otherwise the tool's hunks, or the builtin fallback, feed the counts
through the shared xpparam_t.

Like the builtin summary path, builtin_diffstat() does not apply
textconv, so the process is consulted on the raw blob content here,
unlike builtin_diff() which sends textconv'd content.  This keeps
"git diff --stat" counting raw lines as it does today; the asymmetry
between patch output and summary counts under textconv predates this
change.  Because the content is the raw blob, the stat path sends the
blob object names to the tool (old-oid/new-oid) for any stored blob,
where the patch path omits the oid under textconv.

Move the summary formats out of the "not yet wired" group of the
"Which features consult the diff process" documentation and into the
list of features that use the tool's hunks, noting the raw,
non-textconv content they receive.  Document that the line-counting
--dirstat=lines follows these counts while the default --dirstat does
not, and that summary formats and blame (only under --textconv) differ
from patch output in whether they textconv the content the tool sees.

Add tests covering counts from the tool's hunks (--numstat,
--shortstat), an equivalent file producing no stat line, --stat
--exit-code, the raw non-textconv content the tool receives, a
multi-file mix of equivalent and changed files, and a mode-only
change.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc |  33 +++++---
 diff.c                           |  20 ++++-
 t/t4080-diff-process.sh          | 130 +++++++++++++++++++++++++++++++
 3 files changed, 172 insertions(+), 11 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index a03fb9deb1..7cdede6b21 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -874,7 +874,10 @@ a flush packet, followed by the old and new file content as packetized
 data, each terminated with a flush packet.  The pathname is relative
 to the repository root.  When `diff.<name>.textconv` is also set,
 the tool receives the textconv-transformed content rather than the
-raw blob.  Git does not send binary files to the diff process.
+raw blob, matching what the consuming feature itself diffs: patch
+output is textconv'd, the summary formats (noted below) are not, and
+`git blame` applies textconv only under `--textconv`.  Git does not
+send binary files to the diff process.
 
 -----------------------
 packet:          git> command=hunks
@@ -960,8 +963,8 @@ still slide or regroup those changes against matching context for
 display, exactly as it compacts its own diffs, so the tool controls
 which lines are reported as changed, not the precise hunk boundaries.
 Patch output features (word diff, function context, color) work
-normally.  Summary formats such as `--stat` still compute their counts
-with the builtin diff for now; see "Which features consult the diff
+normally, as do summary formats like `--stat`.  Not every feature
+consults the process, though; see "Which features consult the diff
 process" below for the full picture and the reasoning behind it.
 
 If no hunk lines precede the flush, followed by "success", Git
@@ -1040,6 +1043,17 @@ of the builtin algorithm:
   hunks without any further negotiation.
 - `git blame`: a commit whose change the tool reports as equivalent is
   skipped, and its lines are attributed to an earlier commit.
+- `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted
+  counts come from the tool's hunks, so a file the tool calls
+  equivalent contributes no stat line, matching the empty patch that
+  `git diff` produces for it.  These summary formats do not apply
+  textconv (just as the builtin summary path does not), so the tool
+  is consulted on the raw blob content even when a `textconv` is also
+  configured for patch output; this mirrors how builtin `--stat`
+  already counts raw lines rather than the textconv'd view.  The
+  line-counting `--dirstat=lines` uses these same counts; the default
+  `--dirstat`, which weighs byte changes, is computed on its own and
+  does not consult the tool.
 
 Features that ask a different question do not consult the process, by
 design:
@@ -1065,13 +1079,12 @@ design:
 - `--raw`, `--name-only`, and `--name-status` compare object ids at
   the tree level and never run a line-level diff at all.
 
-Some features ask "which lines changed" but still use the builtin
-algorithm for now, and may consult the process in a later change: the
-summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
-commit selection and parent range propagation (as distinct from its
-display, which is covered above); and combined diffs (`--cc` and merge
-diffs), whose protocol would have to be extended from a single old/new
-pair to one comparison per merge parent.
+Two cases ask "which lines changed" but still use the builtin
+algorithm, and may consult the process in a later change: `git log
+-L`'s commit selection and parent range propagation (as distinct from
+its display, which is covered above), and combined diffs (`--cc` and
+merge diffs), whose protocol would have to be extended from a single
+old/new pair to one comparison per merge parent.
 
 `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
 for every feature listed above.  The whitespace-ignoring options
diff --git a/diff.c b/diff.c
index a9a732629e..f3c8267a39 100644
--- a/diff.c
+++ b/diff.c
@@ -4269,9 +4269,27 @@ static void builtin_diffstat(const char *name_a, const char *name_b,
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_NO_HUNK_HDR;
-		if (xdi_diff_outf(&mf1, &mf2, NULL,
+		/*
+		 * Consult the diff process so --stat reflects the
+		 * tool's view of which lines changed rather than the
+		 * builtin line diff.  --stat never applies textconv, so
+		 * the tool is fed the same raw mmfiles the stat itself
+		 * diffs (unlike builtin_diff, which consults the process
+		 * on textconv'd content).
+		 * When the tool reports the files as equivalent we skip
+		 * xdiff entirely, leaving added and deleted at zero so
+		 * the file is pruned below, just as builtin_diff() emits
+		 * no patch for an equivalent file.
+		 */
+		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
+					    one->oid_valid ? &one->oid : NULL,
+					    two->oid_valid ? &two->oid : NULL,
+					    &xpp)
+		    != DIFF_PROCESS_EQUIVALENT &&
+		    xdi_diff_outf(&mf1, &mf2, NULL,
 				  diffstat_consume, diffstat, &xpp, &xecfg))
 			die("unable to generate diffstat for %s", one->path);
+		free(xpp.external_hunks);
 
 		if (DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two)) {
 			struct diffstat_file *file =
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 694c94edb2..e1c7256747 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -282,6 +282,21 @@ test_expect_success 'diff process works alongside textconv' '
 	test_must_be_empty stderr
 '
 
+test_expect_success 'diff process --stat is fed raw, not textconv, content' '
+	# Reuses textconv.c from the previous test (committed "hello
+	# world", modified to "goodbye world").  Unlike patch output,
+	# --stat does not apply textconv, so the tool sees raw lowercase
+	# content here even with a textconv configured.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --stat -- textconv.c >actual 2>stderr &&
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=hello world" backend.log &&
+	test_grep "new=goodbye world" backend.log &&
+	test_must_be_empty stderr
+'
+
 #
 # Downstream features: word diff, log, equivalent files, exit code.
 #
@@ -386,6 +401,121 @@ test_expect_success 'diff process with --exit-code and hunks returns failure' '
 		diff --exit-code newfile.c
 '
 
+test_expect_success 'diff process feeds --numstat counts' '
+	# fixed-hunk reports only lines 5-6 as changed, so the stat
+	# counts come from the tool (2/2), not the builtin diff (4/4).
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		diff --numstat boundary.c >actual 2>stderr &&
+	printf "2\t2\tboundary.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process --numstat sums multi-hunk counts' '
+	# multi-hunk reports both 2-line regions (5-6 and 9-10), so the
+	# counts add up across both hunks: 4 inserted, 4 deleted.  This
+	# exercises the two-region hunk path through builtin_diffstat.
+	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
+		diff --numstat boundary.c >actual &&
+	printf "4\t4\tboundary.c\n" >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success 'diff process equivalent files produce no --stat line' '
+	# A file the tool calls equivalent contributes no stat line,
+	# matching the empty patch that git diff produces for it.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --stat worddiff.c >actual 2>stderr &&
+	test_must_be_empty actual &&
+	test_grep "command=hunks pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process feeds --shortstat counts' '
+	# fixed-hunk reports lines 5-6 only, so the summary counts come
+	# from the tool (2 insertions, 2 deletions), not builtin (4/4).
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff --shortstat boundary.c >actual &&
+	test_grep "2 insertions" actual &&
+	test_grep "2 deletions" actual
+'
+
+test_expect_success 'diff process equivalent file makes --stat --exit-code succeed' '
+	# The tool reports worddiff.c equivalent, so --exit-code reports
+	# no change (0); the builtin diff would report a change (1).
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --stat --exit-code worddiff.c &&
+	test_expect_code 1 git diff --no-ext-diff --stat --exit-code worddiff.c
+'
+
+test_expect_success 'diff process --numstat with mixed equivalent and changed files' '
+	test_when_finished "rm -f c.log h.log" &&
+	# Self-contained fixtures: *.c uses whole-file (changed); *.mh
+	# uses no-hunks (equivalent).
+	echo "*.mh diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+	printf "int a(void) { return 1; }\n" >mixed.c &&
+	printf "int b(void) { return 1; }\n" >mixed.mh &&
+	git add mixed.c mixed.mh &&
+	git commit -m "add mixed fixtures" &&
+	printf "int a(void) { return 2; }\n" >mixed.c &&
+	printf "int b(void) { return 2; }\n" >mixed.mh &&
+	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=c.log" \
+	    -c diff.hdiff.process="$BACKEND --mode=no-hunks --log=h.log" \
+		diff --numstat mixed.c mixed.mh >actual 2>stderr &&
+	test_grep "mixed.c" actual &&
+	test_grep ! "mixed.mh" actual &&
+	test_grep "pathname=mixed.c" c.log &&
+	test_grep "pathname=mixed.mh" h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success POSIXPERM 'diff process keeps mode-only change in --stat' '
+	test_when_finished "rm -f backend.log" &&
+	cat >modeonly.c <<-\EOF &&
+	int m(void) { return 1; }
+	EOF
+	git add modeonly.c &&
+	git commit -m "add modeonly.c" &&
+	cat >modeonly.c <<-\EOF &&
+	int m(void) { return 2; }
+	EOF
+	git add modeonly.c &&
+	test_chmod +x modeonly.c &&
+	git commit -m "edit and chmod modeonly.c" &&
+	# Content and mode both changed, but no-hunks reports the content
+	# equivalent.  The tool is consulted (counts are zero, not the
+	# builtin 1/1), yet the mode change keeps the file from being
+	# pruned.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --stat HEAD^ HEAD >actual 2>stderr &&
+	test_grep "modeonly.c" actual &&
+	test_grep "command=hunks pathname=modeonly.c" backend.log &&
+	test_grep ! "1 insertion" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process not consulted for default --dirstat' '
+	# The default (change-based) --dirstat algorithm counts via its
+	# own path and never contacts the tool (here --dirstat=0 just
+	# sets a 0% threshold), so the change is still reported even
+	# though no-hunks would call it equivalent.  --dirstat=lines
+	# instead uses the process-aware stat path.
+	test_when_finished "rm -f backend.log" &&
+	mkdir -p dsub &&
+	printf "a\nb\nc\n" >dsub/d.c &&
+	git add dsub/d.c &&
+	git commit -m "add dsub/d.c" &&
+	printf "a\nB\nc\n" >dsub/d.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --dirstat=0 dsub/d.c >actual &&
+	test_grep "dsub" actual &&
+	test_path_is_missing backend.log
+'
+
 #
 # Bypass mechanisms: flags and commands that skip the diff process.
 #
-- 
gitgitgadget

[PATCH v5 9/9] line-log: consult diff process for range tracking

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

git log -L tracks line ranges by diffing each commit against its
parent in collect_diff().  This pass used the builtin diff while the
displayed diff (builtin_diff()) consults a configured
diff.<driver>.process, so the two could disagree: a reformat-only
commit selected by builtin tracking was then rendered with an empty
diff because the tool reported the files equivalent.

Consult the process in collect_diff() too, mirroring the blame
integration.  When the tool reports the files equivalent, collect no
ranges; the tracked range then maps across unchanged and the commit
drops out of the log, matching what is displayed.  Like the summary
formats, the tracking pass diffs raw content, so the tool is consulted
on the raw blobs here.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc | 20 ++++++++++---------
 line-log.c                       | 33 ++++++++++++++++++++++++++++----
 t/t4080-diff-process.sh          | 33 ++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 13 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 7cdede6b21..8021dc8e39 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -1037,12 +1037,16 @@ Features that ask "which lines changed" use the tool's hunks in place
 of the builtin algorithm:
 
 - `git diff` patch output, together with everything layered on it:
-  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
-  headers, and the `-L` line-range display.  These operate on the
-  lines the patch step already emitted, so they reflect the tool's
-  hunks without any further negotiation.
+  word diff, function context (`-W`), `--color-moved`, and the `@@`
+  hunk headers.  These operate on the lines the patch step already
+  emitted, so they reflect the tool's hunks without any further
+  negotiation.
 - `git blame`: a commit whose change the tool reports as equivalent is
   skipped, and its lines are attributed to an earlier commit.
+- `git log -L`: both the line-range display and the underlying range
+  tracking consult the tool, so a commit it reports as equivalent is
+  dropped from the log (its tracked range maps across unchanged)
+  rather than selected and then shown with an empty diff.
 - `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted
   counts come from the tool's hunks, so a file the tool calls
   equivalent contributes no stat line, matching the empty patch that
@@ -1079,11 +1083,9 @@ design:
 - `--raw`, `--name-only`, and `--name-status` compare object ids at
   the tree level and never run a line-level diff at all.
 
-Two cases ask "which lines changed" but still use the builtin
-algorithm, and may consult the process in a later change: `git log
--L`'s commit selection and parent range propagation (as distinct from
-its display, which is covered above), and combined diffs (`--cc` and
-merge diffs), whose protocol would have to be extended from a single
+Combined diffs (`--cc` and merge diffs) ask "which lines changed" but
+still use the builtin algorithm, and may consult the process in a
+later change; their protocol would have to be extended from a single
 old/new pair to one comparison per merge parent.
 
 `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
diff --git a/line-log.c b/line-log.c
index 5fc75ae275..97b3e0a31d 100644
--- a/line-log.c
+++ b/line-log.c
@@ -7,11 +7,11 @@
 #include "tag.h"
 #include "tree.h"
 #include "diff.h"
+#include "diff-process.h"
 #include "commit.h"
 #include "decorate.h"
 #include "repository.h"
 #include "revision.h"
-#include "xdiff-interface.h"
 #include "strbuf.h"
 #include "line-log.h"
 #include "setup.h"
@@ -330,12 +330,15 @@ static int collect_diff_cb(long start_a, long count_a,
 	return 0;
 }
 
-static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges *out)
+static int collect_diff(struct diff_options *diffopt, const char *path,
+			mmfile_t *parent, mmfile_t *target,
+			struct diff_ranges *out)
 {
 	struct collect_diff_cbdata cbdata = {NULL};
 	xpparam_t xpp;
 	xdemitconf_t xecfg;
 	xdemitcb_t ecb;
+	int ret = 0;
 
 	memset(&xpp, 0, sizeof(xpp));
 	memset(&xecfg, 0, sizeof(xecfg));
@@ -345,7 +348,23 @@ static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges *
 	xecfg.hunk_func = collect_diff_cb;
 	memset(&ecb, 0, sizeof(ecb));
 	ecb.priv = &cbdata;
-	return xdi_diff(parent, target, &xpp, &xecfg, &ecb);
+
+	/*
+	 * Consult the diff process so range tracking agrees with the
+	 * diff that will be shown.  When the tool reports the files as
+	 * equivalent we collect no ranges, so the tracked range maps
+	 * across unchanged and the commit drops out of the log, rather
+	 * than being selected here but rendered with an empty diff by
+	 * the process-aware builtin_diff().  Blob oids are not threaded to
+	 * this path yet, so pass NULL and send no old-oid/new-oid (a later
+	 * change can supply the pair, where they would let the tool cache
+	 * across the range-tracking and display passes over the same
+	 * commit).
+	 */
+	if (xdi_diff_process(diffopt, path, parent, target,
+			     NULL, NULL, &xpp, &xecfg, &ecb) == DIFF_PROCESS_ERROR)
+		ret = -1;
+	return ret;
 }
 
 /*
@@ -927,7 +946,13 @@ static int process_diff_filepair(struct rev_info *rev,
 	}
 
 	diff_ranges_init(&diff);
-	if (collect_diff(&file_parent, &file_target, &diff))
+	/*
+	 * Select the driver by the old (parent) path, as builtin_diff() does
+	 * with name_a, so a renamed file resolves to the same driver for
+	 * range tracking as for the diff that is shown.
+	 */
+	if (collect_diff(&rev->diffopt, pair->one->path,
+			 &file_parent, &file_target, &diff))
 		die("unable to generate diff for %s", pair->one->path);
 
 	/* NEEDSWORK should apply some heuristics to prevent mismatches */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index e1c7256747..c20ab15ecd 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -914,4 +914,37 @@ test_expect_success 'blame -w bypasses diff process' '
 	test_path_is_missing backend.log
 '
 
+#
+# Line-log (git log -L) range tracking.
+#
+
+test_expect_success 'diff process drops equivalent commit from log -L' '
+	test_when_finished "rm -f backend.log" &&
+	cat >linelog.c <<-\EOF &&
+	int tracked(void) { return 1; }
+	EOF
+	git add linelog.c &&
+	git commit -m "add linelog.c" &&
+
+	cat >linelog.c <<-\EOF &&
+	int tracked(void) { return 2; }
+	EOF
+	git commit -am "change tracked line" &&
+
+	# Builtin line tracking selects the change commit.
+	git log --no-ext-diff -L1,1:linelog.c --format="%s" >builtin &&
+	test_grep "change tracked line" builtin &&
+
+	# With the tool reporting the change as equivalent, tracking
+	# drops the commit (the range maps across unchanged) instead of
+	# selecting it and rendering an empty diff.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -L1,1:linelog.c --format="%s" >actual &&
+	test_grep ! "change tracked line" actual &&
+	# The creating commit still appears, so the change commit was
+	# selectively dropped rather than the whole log going empty.
+	test_grep "add linelog.c" actual &&
+	test_grep "command=hunks pathname=linelog.c" backend.log
+'
+
 test_done
-- 
gitgitgadget

Re: [PATCH v5 0/9] [RFC] diff: add diff.<driver>.process for external hunk providers

[Junio C Hamano]

"Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:

> A "Which features consult the diff process" section in gitattributes(5) lays
> out, per feature, why each does or does not consult the process (patch
> output, blame, summary formats, and the -L line-range view do; pickaxe -G,
> patch-id, merge, range-diff, --check, and --raw do not, with reasons).
> Combined diffs (--cc) remain on the builtin algorithm and are noted as
> future work.
>
> Changes since v4:

This round does not play well with the mm/line-log-limited-ops
topic, unfortunately, it seems.

Re: [PATCH v5 0/9] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo]

On Thu, Jul 16, 2026 at 9:40 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> "Michael Montalbo via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > A "Which features consult the diff process" section in gitattributes(5) lays
> > out, per feature, why each does or does not consult the process (patch
> > output, blame, summary formats, and the -L line-range view do; pickaxe -G,
> > patch-id, merge, range-diff, --check, and --raw do not, with reasons).
> > Combined diffs (--cc) remain on the builtin algorithm and are noted as
> > future work.
> >
> > Changes since v4:
>
> This round does not play well with the mm/line-log-limited-ops
> topic, unfortunately, it seems.

Ah, thanks for the heads up and apologies for the hiccup. I will
rebase this series on top of mm/line-log-limited-ops and make
sure the two topics function together correctly.

[PATCH v6 0/9] [RFC] diff: add diff.<driver>.process for external hunk providers

[Michael Montalbo via GitGitGadget]

Language-aware diff tools (e.g., Difftastic) and format-specific analyzers
can produce better line matching than Git's builtin diff algorithm, but
diff.<driver>.command replaces Git's diff output with the program's own
output, so display features like word diff, function context, and color
cannot operate on it; and because the program is consulted only for that
patch output, blame, --stat, and git log -L fall back to Git's builtin line
matching and cannot benefit from the tool at all.

This series adds diff.<driver>.process, a long-running subprocess protocol
that lets an external tool control which lines Git considers changed while
Git handles all output formatting. The protocol follows
filter.<driver>.process: pkt-line over stdin/stdout, capability negotiation,
one process per Git invocation.

The tool receives both file versions and returns changed regions (line
ranges in the old and new file). Git validates and feeds them into the xdiff
pipeline in place of the builtin diff algorithm. When the tool returns no
hunks, Git treats the files as having no changes, which propagates through
patch output, the --stat summary, blame, and git log -L. The request also
carries the two blobs' object names (old-oid/new-oid) so a tool can cache
its analysis keyed on the pair.

 * Patch 1: document how an external diff driver (diff.<driver>.command)
   relates to the rest of Git's diff features, so the contrast with the new
   process driver is clear.
 * Patch 2: xdiff plumbing for externally supplied hunks.
 * Patch 3: diff.<driver>.process config key.
 * Patch 4: refactor subprocess API to separate process lifecycle from
   hashmap management, since the diff process stores its subprocess on the
   userdiff driver rather than in a hashmap.
 * Patch 5: the main feature, including the old-oid/new-oid request metadata
   for blob-pair caching.
 * Patch 6: bypass knobs (--no-ext-diff, format-patch).
 * Patch 7: blame integration so the tool can declare commits as having no
   changes; introduces the shared xdi_diff_process() consult-then-diff
   helper that blame and git log -L both use.
 * Patch 8: --stat/--numstat/--shortstat consult the tool, so the summary
   agrees with the patch output.
 * Patch 9: git log -L range tracking consults the tool, so a reformat-only
   commit is dropped from the log rather than shown with an empty diff.

A "Which features consult the diff process" section in gitattributes(5) lays
out, per feature, why each does or does not consult the process (patch
output, blame, summary formats, and the -L line-range view do; pickaxe -G,
patch-id, merge, range-diff, --check, and --raw do not, with reasons).
Combined diffs (--cc) remain on the builtin algorithm and are noted as
future work.

Changes since v5:

 * Changed series to be based on top of the in-flight
   mm/line-log-limited-ops:
   
   https://lore.kernel.org/git/pull.2152.v2.git.1782581342.gitgitgadget@gmail.com/

 * builtin_diffstat() now routes the process's hunks through that topic's -L
   line-range filter, so "git log -L --stat" scopes the tool's changed-line
   counts to the tracked range (patch 8, with a new t4080 test).

Michael Montalbo (9):
  gitattributes: document how external diff drivers relate to diff
    features
  xdiff: support external hunks via xpparam_t
  userdiff: add diff.<driver>.process config
  sub-process: separate process lifecycle from hashmap management
  diff: add long-running diff process via diff.<driver>.process
  diff: bypass diff process with --no-ext-diff and in format-patch
  blame: consult diff process for no-hunk detection
  diff: consult diff process for --stat counts
  line-log: consult diff process for range tracking

 Documentation/config/diff.adoc           |   5 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |   4 +-
 Documentation/gitattributes.adoc         | 274 +++++++
 Makefile                                 |   2 +
 blame.c                                  |  24 +-
 builtin/log.c                            |   7 +
 diff-process.c                           | 529 ++++++++++++
 diff-process.h                           |  75 ++
 diff.c                                   |  84 +-
 diff.h                                   |   6 +
 line-log.c                               |  33 +-
 meson.build                              |   1 +
 sub-process.c                            |  28 +-
 sub-process.h                            |   9 +-
 t/helper/meson.build                     |   1 +
 t/helper/test-diff-process-backend.c     | 381 +++++++++
 t/helper/test-tool.c                     |   1 +
 t/helper/test-tool.h                     |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 996 +++++++++++++++++++++++
 userdiff.c                               |   7 +
 userdiff.h                               |   5 +
 xdiff-interface.c                        |   7 +-
 xdiff/xdiff.h                            |  16 +
 xdiff/xdiffi.c                           |  84 +-
 xdiff/xprepare.c                         |  10 +
 xdiff/xprepare.h                         |   1 +
 28 files changed, 2566 insertions(+), 29 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh


base-commit: f67c51df064d2b64b257bd8c17d757cc0ce1b7fc
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2120%2Fmmontalbo%2Fmm%2Fstructural-diff-backend-clean-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2120/mmontalbo/mm/structural-diff-backend-clean-v6
Pull-Request: https://github.com/gitgitgadget/git/pull/2120

Range-diff vs v5:

  1:  0fd994a3d3 =  1:  b4a1ff4dea gitattributes: document how external diff drivers relate to diff features
  2:  2004502549 =  2:  ed2db0ac59 xdiff: support external hunks via xpparam_t
  3:  926cf01af6 =  3:  115e31d806 userdiff: add diff.<driver>.process config
  4:  363d459ff6 =  4:  850c7cbcf5 sub-process: separate process lifecycle from hashmap management
  5:  d003bc1f15 =  5:  4526809b7f diff: add long-running diff process via diff.<driver>.process
  6:  b2e80f014e =  6:  4795743ab9 diff: bypass diff process with --no-ext-diff and in format-patch
  7:  cf5bb8984a =  7:  00573a88a9 blame: consult diff process for no-hunk detection
  8:  c1d02d0e15 !  8:  7e3ba56967 diff: consult diff process for --stat counts
     @@ Commit message
          Otherwise the tool's hunks, or the builtin fallback, feed the counts
          through the shared xpparam_t.
      
     +    Under -L, route the surviving hunks through the same line-range filter
     +    builtin_diffstat() already uses for a tracked range, so a
     +    process-provided diff is scoped to that range: "git log -L<range>
     +    --stat" counts the tool's changed lines within the range rather than
     +    the builtin line diff's.
     +
          Like the builtin summary path, builtin_diffstat() does not apply
          textconv, so the process is consulted on the raw blob content here,
          unlike builtin_diff() which sends textconv'd content.  This keeps
     @@ Commit message
          Add tests covering counts from the tool's hunks (--numstat,
          --shortstat), an equivalent file producing no stat line, --stat
          --exit-code, the raw non-textconv content the tool receives, a
     -    multi-file mix of equivalent and changed files, and a mode-only
     -    change.
     +    multi-file mix of equivalent and changed files, a mode-only change,
     +    and a range-scoped --stat under "git log -L" that reflects the tool's
     +    hunks.
      
          Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
      
     @@ diff.c: static void builtin_diffstat(const char *name_a, const char *name_b,
       		xecfg.ctxlen = o->context;
       		xecfg.interhunkctxlen = o->interhunkcontext;
       		xecfg.flags = XDL_EMIT_NO_HUNK_HDR;
     --		if (xdi_diff_outf(&mf1, &mf2, NULL,
     +-
     +-		if (p->line_ranges) {
     +-			struct line_range_filter lr_filter;
     +-
     +-			line_range_filter_init(&lr_filter, p->line_ranges,
     +-					       diffstat_consume, diffstat);
     +-
     +-			if (line_range_filter_diff(&lr_filter, &mf1, &mf2,
     +-						   &xpp, &xecfg))
      +		/*
      +		 * Consult the diff process so --stat reflects the
      +		 * tool's view of which lines changed rather than the
     @@ diff.c: static void builtin_diffstat(const char *name_a, const char *name_b,
      +		 * xdiff entirely, leaving added and deleted at zero so
      +		 * the file is pruned below, just as builtin_diff() emits
      +		 * no patch for an equivalent file.
     ++		 *
     ++		 * Under -L, feed the tool's hunks through the same
     ++		 * line-range filter the builtin stat uses, so a
     ++		 * process-provided diff is scoped to the tracked range.
      +		 */
      +		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
      +					    one->oid_valid ? &one->oid : NULL,
      +					    two->oid_valid ? &two->oid : NULL,
      +					    &xpp)
     -+		    != DIFF_PROCESS_EQUIVALENT &&
     -+		    xdi_diff_outf(&mf1, &mf2, NULL,
     - 				  diffstat_consume, diffstat, &xpp, &xecfg))
     - 			die("unable to generate diffstat for %s", one->path);
     ++		    != DIFF_PROCESS_EQUIVALENT) {
     ++			if (p->line_ranges) {
     ++				struct line_range_filter lr_filter;
     ++
     ++				line_range_filter_init(&lr_filter, p->line_ranges,
     ++						       diffstat_consume, diffstat);
     ++
     ++				if (line_range_filter_diff(&lr_filter, &mf1, &mf2,
     ++							   &xpp, &xecfg))
     ++					die("unable to generate diffstat for %s",
     ++					    one->path);
     ++			} else if (xdi_diff_outf(&mf1, &mf2, NULL, diffstat_consume,
     ++						 diffstat, &xpp, &xecfg))
     + 				die("unable to generate diffstat for %s",
     + 				    one->path);
     +-		} else if (xdi_diff_outf(&mf1, &mf2, NULL,
     +-				  diffstat_consume, diffstat, &xpp, &xecfg))
     +-			die("unable to generate diffstat for %s", one->path);
     ++		}
      +		free(xpp.external_hunks);
       
       		if (DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two)) {
     @@ t/t4080-diff-process.sh: test_expect_success 'diff process with --exit-code and
      +	test_grep "2 deletions" actual
      +'
      +
     ++test_expect_success 'diff process scopes --stat to the tracked range under log -L' '
     ++	test_when_finished "rm -f backend.log" &&
     ++	cat >rangestat.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
     ++	line4
     ++	OLD5
     ++	OLD6
     ++	line7
     ++	line8
     ++	OLD9
     ++	OLD10
     ++	EOF
     ++	git add rangestat.c &&
     ++	git commit -m "add rangestat.c" &&
     ++
     ++	cat >rangestat.c <<-\EOF &&
     ++	line1
     ++	line2
     ++	line3
     ++	line4
     ++	NEW5
     ++	NEW6
     ++	line7
     ++	line8
     ++	NEW9
     ++	NEW10
     ++	EOF
     ++	git add rangestat.c &&
     ++	git commit -m "change rangestat.c" &&
     ++
     ++	# The file changes at lines 5-6 and 9-10, but fixed-hunk reports
     ++	# only 5-6.  The builtin line diff counts both regions (4/4); the
     ++	# tool hunks flow through the same line-range filter the stat uses,
     ++	# so the range-scoped stat reflects the tool view instead (2/2).
     ++	git log --no-ext-diff -L1,10:rangestat.c --oneline --stat >builtin &&
     ++	test_grep "4 insertions(+), 4 deletions(-)" builtin &&
     ++
     ++	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
     ++		log -L1,10:rangestat.c --oneline --stat >actual &&
     ++	test_grep "2 insertions(+), 2 deletions(-)" actual &&
     ++	test_grep ! "4 insertions" actual &&
     ++	test_grep "command=hunks pathname=rangestat.c" backend.log
     ++'
     ++
      +test_expect_success 'diff process equivalent file makes --stat --exit-code succeed' '
      +	# The tool reports worddiff.c equivalent, so --exit-code reports
      +	# no change (0); the builtin diff would report a change (1).
  9:  c3c17ba8fc =  9:  ffa6954d67 line-log: consult diff process for range tracking

-- 
gitgitgadget

[PATCH v6 1/9] gitattributes: document how external diff drivers relate to diff features

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

The "Defining an external diff driver" section explains how to
configure diff.<driver>.command but not how the driver relates to the
rest of Git's diff machinery.  In particular, the command only
replaces the textual patch: word diff, function context, color, and
the like cannot apply to its output, while the summary formats, blame,
and git log -L do not run it at all and keep using the builtin diff.

Spell this out so the scope of an external diff driver is clear.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index bd76167a45..2c4fbfd7f1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -784,6 +784,16 @@ with the above configuration, i.e. `j-c-diff`, with 7
 parameters, just like `GIT_EXTERNAL_DIFF` program is called.
 See linkgit:git[1] for details.
 
+An external diff driver replaces the patch Git would otherwise
+produce for the path: Git runs the command and shows its output in
+place of its own.  Output features that post-process Git's diff do
+not apply to it; word diff, function context (`-W`), `--color-moved`,
+and coloring all act on Git's builtin diff, not the driver's output.
+The driver is consulted only when Git generates a textual patch.  The
+summary formats (`--stat`, `--numstat`, `--shortstat`, and
+`--dirstat`), `git blame`, and `git log -L` do not run it and
+continue to use Git's builtin diff.
+
 If the program is able to ignore certain changes (similar to
 `git diff --ignore-space-change`), then also set the option
 `trustExitCode` to true.  It is then expected to return exit code 1 if
-- 
gitgitgadget

[PATCH v6 2/9] xdiff: support external hunks via xpparam_t

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add two new xpparam_t fields (external_hunks, external_hunks_nr)
that let callers supply pre-computed hunks.  When set, xdl_diff()
populates the changed[] arrays from these hunks instead of running
the diff algorithm, then continues through compaction and emission
as usual.

Validate supplied hunks before use.  Out-of-bounds line numbers,
overlapping or out-of-order hunks, and misaligned unchanged runs are
treated as a malformed tool response: xdl_populate_hunks_from_external()
warns, returns -1, and xdl_diff() falls back to the builtin diff
algorithm for that file.  The run of unchanged lines between two hunks
(and before the first and after the last) must be the same length on
both sides; xdl_build_script() walks the two files in lockstep over
unchanged lines, so a balanced total is not enough.  Non-negative
counts and 1-based starts are instead caller preconditions, checked
with BUG(), since the caller normalizes hunks before this point.

On rejection xdl_diff() frees the environment it prepared and falls
through to xdl_do_diff(), which prepares a fresh one for the builtin
pass.

Skip trim_common_tail() in xdi_diff() when external hunks are
present, since external hunks reference line numbers in the
original content.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 xdiff-interface.c |  7 +++-
 xdiff/xdiff.h     | 16 +++++++++
 xdiff/xdiffi.c    | 84 +++++++++++++++++++++++++++++++++++++++++++++--
 xdiff/xprepare.c  | 10 ++++++
 xdiff/xprepare.h  |  1 +
 5 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/xdiff-interface.c b/xdiff-interface.c
index 32e04630ee..afe4c12e9e 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -143,7 +143,12 @@ int xdi_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp, xdemitconf_t co
 	if (mf1->size > MAX_XDIFF_SIZE || mf2->size > MAX_XDIFF_SIZE)
 		return -1;
 
-	if (!xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
+	/*
+	 * External hunks reference line numbers in the original content;
+	 * trimming the tail would change line counts and invalidate them.
+	 */
+	if (!xpp->external_hunks &&
+	    !xecfg->ctxlen && !(xecfg->flags & XDL_EMIT_FUNCCONTEXT))
 		trim_common_tail(&a, &b);
 
 	return xdl_diff(&a, &b, xpp, xecfg, xecb);
diff --git a/xdiff/xdiff.h b/xdiff/xdiff.h
index dc370712e9..4736bcdb07 100644
--- a/xdiff/xdiff.h
+++ b/xdiff/xdiff.h
@@ -78,6 +78,18 @@ typedef struct s_mmbuffer {
 	long size;
 } mmbuffer_t;
 
+/*
+ * Hunk descriptor for externally computed diffs, in xdiff's own
+ * coordinates: line numbers are 1-based and a hunk's start is the
+ * first line it covers.  A caller translates any external "empty side"
+ * idiom (such as git diff's start-0/count-0) to a 1-based start before
+ * handing hunks over.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 typedef struct s_xpparam {
 	unsigned long flags;
 
@@ -88,6 +100,10 @@ typedef struct s_xpparam {
 	/* See Documentation/diff-options.adoc. */
 	char **anchors;
 	size_t anchors_nr;
+
+	/* Externally computed hunks: bypass the diff algorithm.  Owned by caller. */
+	struct xdl_hunk *external_hunks;
+	size_t external_hunks_nr;
 } xpparam_t;
 
 typedef struct s_xdemitcb {
diff --git a/xdiff/xdiffi.c b/xdiff/xdiffi.c
index c5a892f91e..73a456f5dd 100644
--- a/xdiff/xdiffi.c
+++ b/xdiff/xdiffi.c
@@ -1085,16 +1085,96 @@ static void xdl_mark_ignorable_regex(xdchange_t *xscr, const xdfenv_t *xe,
 	}
 }
 
+/*
+ * Populate the changed[] arrays from externally supplied hunks,
+ * bypassing the diff algorithm.  The caller normalizes and validates
+ * the hunks first (order, overlap, and lockstep alignment), so this
+ * only marks lines changed after asserting the memory-safety
+ * preconditions it depends on: non-negative counts and 1-based starts
+ * (checked with BUG()), and an in-bounds range (a silent -1 so the
+ * caller can fall back to the builtin diff rather than index changed[]
+ * out of range).  Keeping this diagnostic-free leaves user-facing
+ * messages to the git layer.
+ *
+ * Returns 0 on success, -1 if a hunk is out of range.
+ */
+static int xdl_populate_hunks_from_external(xdfenv_t *xe,
+					    struct xdl_hunk *hunks,
+					    size_t nr_hunks)
+{
+	size_t i;
+	long j;
+
+	/*
+	 * xdl_prepare_env() may dirty changed[] via xdl_cleanup_records().
+	 * Clear them so only the external hunks are marked.
+	 */
+	xdl_clear_changed(&xe->xdf1);
+	xdl_clear_changed(&xe->xdf2);
+
+	for (i = 0; i < nr_hunks; i++) {
+		struct xdl_hunk *h = &hunks[i];
+
+		/*
+		 * Non-negative counts and 1-based starts are caller
+		 * preconditions (it normalizes hunks into xdiff coordinates
+		 * before this point), so a violation is a bug, not a bad
+		 * tool response.
+		 */
+		if (h->old_count < 0 || h->new_count < 0)
+			BUG("external hunk %"PRIuMAX": "
+				"negative count (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_count, h->new_count);
+		if (h->old_start < 1 || h->new_start < 1)
+			BUG("external hunk %"PRIuMAX": "
+				"start not 1-based (old=%ld, new=%ld)",
+				(uintmax_t)(i + 1),
+				h->old_start, h->new_start);
+
+		/*
+		 * The caller validates ordering, overlap and lockstep
+		 * alignment (and diagnoses a bad response).  This is only a
+		 * silent in-bounds guard so the marking loop cannot index
+		 * changed[] out of range: start + count - 1 <= nrec,
+		 * rewritten to avoid overflow.  A count of 0 (pure
+		 * insert/delete) allows start == nrec + 1, the position
+		 * after the last line.  On a miss, return -1 and let the
+		 * caller fall back to the builtin diff.
+		 */
+		if (h->old_count > (long)xe->xdf1.nrec - h->old_start + 1 ||
+		    h->new_count > (long)xe->xdf2.nrec - h->new_start + 1)
+			return -1;
+
+		for (j = 0; j < h->old_count; j++)
+			xe->xdf1.changed[h->old_start - 1 + j] = true;
+		for (j = 0; j < h->new_count; j++)
+			xe->xdf2.changed[h->new_start - 1 + j] = true;
+	}
+
+	return 0;
+}
+
 int xdl_diff(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 	     xdemitconf_t const *xecfg, xdemitcb_t *ecb) {
 	xdchange_t *xscr;
 	xdfenv_t xe;
 	emit_func_t ef = xecfg->hunk_func ? xdl_call_hunk_func : xdl_emit_diff;
 
-	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0) {
+	if (xpp->external_hunks) {
+		if (xdl_prepare_env(mf1, mf2, xpp, &xe) < 0)
+			return -1;
+		if (xdl_populate_hunks_from_external(&xe,
+						     xpp->external_hunks,
+						     xpp->external_hunks_nr) == 0)
+			goto diff_done;
+		xdl_free_env(&xe);
+	}
 
+	if (xdl_do_diff(mf1, mf2, xpp, &xe) < 0)
 		return -1;
-	}
+
+diff_done:
 	if (xdl_change_compact(&xe.xdf1, &xe.xdf2, xpp->flags) < 0 ||
 	    xdl_change_compact(&xe.xdf2, &xe.xdf1, xpp->flags) < 0 ||
 	    xdl_build_script(&xe, &xscr) < 0) {
diff --git a/xdiff/xprepare.c b/xdiff/xprepare.c
index 11bada2608..f4ab935332 100644
--- a/xdiff/xprepare.c
+++ b/xdiff/xprepare.c
@@ -471,3 +471,13 @@ int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 
 	return 0;
 }
+
+/*
+ * Reset the changed[] array so that no lines are marked as changed.
+ * Also clears the sentinel slots at changed[-1] and changed[nrec]
+ * that xdl_change_compact() relies on during backward scans.
+ */
+void xdl_clear_changed(xdfile_t *xdf)
+{
+	memset(xdf->changed - 1, 0, (xdf->nrec + 2) * sizeof(bool));
+}
diff --git a/xdiff/xprepare.h b/xdiff/xprepare.h
index 947d9fc1bb..0413baf07b 100644
--- a/xdiff/xprepare.h
+++ b/xdiff/xprepare.h
@@ -28,6 +28,7 @@
 int xdl_prepare_env(mmfile_t *mf1, mmfile_t *mf2, xpparam_t const *xpp,
 		    xdfenv_t *xe);
 void xdl_free_env(xdfenv_t *xe);
+void xdl_clear_changed(xdfile_t *xdf);
 
 
 
-- 
gitgitgadget

[PATCH v6 3/9] userdiff: add diff.<driver>.process config

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index b5412e6bc3..7547874aa2 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v6 4/9] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

subprocess_start() and subprocess_stop() couple two concerns:
managing a child process (setup, handshake, teardown) and
managing a hashmap that indexes running processes by command
string.  The hashmap suits callers like convert.c where many
files may share one filter process looked up by name, but
callers that manage process lifetime through their own data
structures do not need it.

Extract subprocess_start_command() and subprocess_stop_command()
so callers can reuse the child process setup and handshake
machinery without maintaining a hashmap.  subprocess_start()
and subprocess_stop() become thin wrappers that add hashmap
operations on top.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 28 +++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 83bf0a0e82..5468939338 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
 
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
+
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err)
+		return err;
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
gitgitgadget

[PATCH v6 5/9] diff: add long-running diff process via diff.<driver>.process

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Add support for external diff processes that communicate via the
long-running process protocol (pkt-line over stdin/stdout).

A diff process is configured per userdiff driver:

    [diff "cdiff"]
        process = /path/to/diff-tool

The tool provides custom line-matching: it receives file pairs
and returns hunks that reference line numbers in the content.
When textconv is also configured, the tool receives the
textconv-transformed content.  The tool controls which lines
are marked as changed while the display shows the file content.
Patch output features (word diff, function context, color) work
normally.  A new "Which features consult the diff process"
documentation section lays out which features use the tool's hunks,
which compute independently, and why; the summary formats such as
--stat still use the builtin diff for now.

The handshake negotiates version=1 and capability=hunks.  Per-file
requests send command=hunks, pathname, the old and new blob object
names as old-oid/new-oid, and both file contents as packetized data.
The tool responds with hunk lines and a status packet (success,
error, or abort).  On error, Git warns and falls back to the builtin
diff algorithm for that file.  On abort, Git silently falls back for
the current file and stops sending further requests to the tool for
the remainder of the session.

old-oid/new-oid name the two blobs so a tool can cache its analysis
keyed on the pair.  A side's oid is sent only when the content the
tool receives is that raw blob: it is omitted under textconv, which
rewrites the bytes, and for a working-tree side with no stored
object, so an oid that is sent always names the bytes the tool
receives.  This is where the process protocol diverges from
diff.<driver>.command, which never composes with textconv (the
command replaces the whole diff and always gets the raw blob).  Tools
ignore unknown request keys, so old tools skip them.

When the tool returns no hunks followed by status=success, Git
treats the file as having no changes and produces no diff output.
This also means --exit-code reports no changes for that file.

The subprocess is stored on the userdiff_driver struct and
launched on first use.  If the process fails to start, the
handshake fails, or a communication error occurs mid-stream,
the failure is cached on the driver to avoid retrying and
re-warning on every subsequent file.

Git falls back to the builtin diff (rather than consulting the
tool) when an option the tool cannot honor is in effect: the
whitespace-ignoring flags, --ignore-blank-lines, -I<regex>, and
--anchored.  The bypass keys off the effective diff parameters (xpp)
rather than diffopt, so a later caller whose flags live elsewhere is
covered uniformly.  A change that only adds or removes the trailing
newline is likewise not expressible as hunks, so it too uses the
builtin diff.  The hunk parser ignores unknown trailing fields on a
hunk line for response forward-compatibility.

Hunk accumulation is bounded by the combined byte count of the two
files, so a misbehaving tool that floods hunk lines cannot grow
memory without bound before validation runs.

diff_process_fill_hunks() is the sole public entry point.  It
handles driver lookup, flag checks, subprocess management, and
error reporting, returning an enum that lets callers distinguish
"hunks populated" from "files equivalent" from "not applicable"
from "tool failure."

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc       |   5 +
 Documentation/gitattributes.adoc     | 249 +++++++++++
 Makefile                             |   2 +
 diff-process.c                       | 491 ++++++++++++++++++++
 diff-process.h                       |  49 ++
 diff.c                               |  21 +
 diff.h                               |   3 +
 meson.build                          |   1 +
 t/helper/meson.build                 |   1 +
 t/helper/test-diff-process-backend.c | 381 ++++++++++++++++
 t/helper/test-tool.c                 |   1 +
 t/helper/test-tool.h                 |   1 +
 t/meson.build                        |   1 +
 t/t4080-diff-process.sh              | 645 +++++++++++++++++++++++++++
 userdiff.h                           |   3 +
 15 files changed, 1854 insertions(+)
 create mode 100644 diff-process.c
 create mode 100644 diff-process.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..ac0635bb3b 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,11 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running diff process that
+	provides hunks to Git's diff pipeline.
+	See linkgit:gitattributes[5] for details.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 2c4fbfd7f1..f4ca4a8c7e 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -831,6 +831,255 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Using an external diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If `diff.<name>.process` is defined, Git sends the old and new file
+content to an external tool and receives back a list of changed
+regions (pairs of line ranges in the old and new file).  Git uses
+these instead of its builtin diff algorithm, but still controls
+all output formatting, so features like word diff, function context,
+color, and blame work normally.  This is achieved by using the
+long-running process protocol (described in
+Documentation/technical/long-running-process-protocol.adoc).
+Unlike `diff.<name>.command`, which replaces Git's output entirely,
+the diff process feeds results back into the standard pipeline.  If
+both are configured for a path, `diff.<name>.command` takes precedence
+for the patch output it replaces; the summary formats, `git blame`,
+and `git log -L` never run the command and still consult the process.
+
+First, in `.gitattributes`, assign the `diff` attribute for paths.
+
+------------------------
+*.c diff=cdiff
+------------------------
+
+Then, define a "diff.<name>.process" configuration to specify
+the diff process command.
+
+----------------------------------------------------------------
+[diff "cdiff"]
+  process = /path/to/diff-process-tool
+----------------------------------------------------------------
+
+When Git encounters the first file that needs to be diffed, it starts
+the process and performs the handshake.  In the handshake, the welcome
+message sent by Git is "git-diff-client", only version 1 is supported,
+and the supported capability is "hunks" (the changed regions
+described below).  The tool replies with "git-diff-server", the
+version it supports, and the capabilities it supports.
+
+For each file, Git sends a list of "key=value" pairs terminated with
+a flush packet, followed by the old and new file content as packetized
+data, each terminated with a flush packet.  The pathname is relative
+to the repository root.  When `diff.<name>.textconv` is also set,
+the tool receives the textconv-transformed content rather than the
+raw blob.  Git does not send binary files to the diff process.
+
+-----------------------
+packet:          git> command=hunks
+packet:          git> pathname=path/file.c
+packet:          git> old-oid=<hex>
+packet:          git> new-oid=<hex>
+packet:          git> 0000
+packet:          git> OLD_CONTENT
+packet:          git> 0000
+packet:          git> NEW_CONTENT
+packet:          git> 0000
+-----------------------
+
+The optional `old-oid` and `new-oid` keys give the object names of the
+old and new blobs, so a tool can cache its analysis keyed on the pair.
+A side's key is sent only when the content for that side is the raw
+blob it names: it is omitted when the content is textconv-transformed,
+and for a working-tree side that has no stored object.  A tool that
+does not recognize these keys ignores them.
+
+The tool is expected to respond with zero or more hunk lines,
+a flush packet, and a status packet terminated with a flush packet.
+Each hunk line has the form:
+
+  `hunk <old_start> <old_count> <new_start> <new_count>`
+
+where `<old_start>` and `<old_count>` identify a range of lines in
+the old file, and `<new_start>` and `<new_count>` identify the
+replacement range in the new file.  The four fields are separated by
+single spaces.  Start values are 1-based and counts are non-negative.
+For example, `hunk 3 2 3 4` means that 2 lines starting at line 3 in
+the old file were replaced by 4 lines starting at line 3 in the new
+file.  An `<old_count>` of 0 means no lines were removed (pure
+insertion); a `<new_count>` of 0 means no lines were added (pure
+deletion).  For a side with a count of 0 (a pure insertion or
+deletion) the start is the 1-based line the change sits before,
+ranging from 1 to one past the last line (the line count plus 1, to
+place the change at the end of the file); like every start it must
+keep the unchanged runs aligned on both sides (see below), so for a
+given change it takes one specific value, not an arbitrary one.  A
+start of 0 is also accepted and treated as 1, matching the
+empty-file-side form `git diff` emits (e.g. `hunk 0 0 1 5` for a newly
+added file).  A nonzero range must not extend beyond the end of the
+file.  Git ignores any extra
+whitespace-separated tokens after `<new_count>`, so a future protocol
+version can append fields to a hunk line (for example a "moved"
+marker) without older tools rejecting it.
+
+Lines are delimited by newlines.  A file `"foo\nbar\n"` and a
+file `"foo\nbar"` both have 2 lines.
+
+Hunks must be listed in order and must not overlap.  Any line not
+covered by a hunk is treated as unchanged and is paired, in order,
+with the unchanged lines on the other side.  Each run of unchanged
+lines between two hunks (and the run before the first hunk and
+after the last) must therefore be the same length on both sides,
+not merely equal in total.  For the hunks `1 3 1 5` and `10 2 12 2`
+below, lines 4-9 of the old file and lines 6-11 of the new file are
+both the six unchanged lines between the two hunks.  A response that
+balances only the total unchanged count but misaligns one of these
+runs is rejected, and Git falls back to the builtin diff.
+
+Git does not check that the lines a hunk leaves unchanged are
+byte-for-byte identical between the two sides; it pairs them by
+position and shows the new side as context.  A tool may therefore
+report lines that differ textually (a pure reformatting, say) as
+unchanged, and the diff reflects that judgment.  This is
+the point of a semantic backend, but it means a misbehaving tool can
+produce a diff whose context does not match the old blob; as with
+`git diff -w`, such a patch may not apply against the old content.
+
+-----------------------
+packet:          git< hunk 1 3 1 5
+packet:          git< hunk 10 2 12 2
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool responds with hunks and "success", Git marks those lines
+as changed and feeds them into the standard diff pipeline.  Git may
+still slide or regroup those changes against matching context for
+display, exactly as it compacts its own diffs, so the tool controls
+which lines are reported as changed, not the precise hunk boundaries.
+Patch output features (word diff, function context, color) work
+normally.  Summary formats such as `--stat` still compute their counts
+with the builtin diff for now; see "Which features consult the diff
+process" below for the full picture and the reasoning behind it.
+
+If no hunk lines precede the flush, followed by "success", Git
+treats the files as having no changes: `git diff` produces no output,
+`git diff --exit-code` and `--quiet` report success even though the
+stored blobs differ, and `git blame` skips the commit, attributing
+lines to earlier commits.
+The one exception is a change that only adds or removes the file's
+trailing newline: it cannot be expressed as line hunks, so when the
+line content otherwise matches Git keeps the builtin diff for that
+file (preserving the `\ No newline at end of file` marker) instead of
+treating the two sides as equal.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+If the tool returns well-formed but invalid hunks (out of bounds,
+overlapping, or with misaligned unchanged runs), Git warns and falls
+back to the builtin diff for that file; the tool stays available for
+subsequent files.  A malformed hunk line, by contrast (bad syntax, a
+nonzero count paired with a start of 0, or more hunks than the file
+has lines), is a protocol violation: Git stops the process and does
+not send it further requests, as described below.
+
+In case the tool cannot or does not want to process the content,
+it is expected to respond with an "error" status.  Git warns and
+falls back to the builtin diff algorithm for this file, treating any
+status other than "success" or "abort" the same way.  The tool
+remains available for subsequent files.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=error
+packet:          git< 0000
+-----------------------
+
+In case the tool cannot or does not want to process the content as
+well as any future content for the lifetime of the Git process, it
+is expected to respond with an "abort" status.  Git silently falls
+back to the builtin diff algorithm for this file and does not send
+further requests to the tool.
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=abort
+packet:          git< 0000
+-----------------------
+
+If the tool dies during the communication or does not adhere to the
+protocol then Git will stop the process and fall back to the builtin
+diff algorithm.  Git warns once and does not restart the process for
+subsequent files.
+
+Tools should ignore unknown keys in the per-file request to remain
+forward-compatible.  Future versions of Git may send additional
+`command=` values; tools that receive an unrecognized command should
+respond with `status=error` rather than terminating.
+
+Which features consult the diff process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The diff process answers a single question: given two blobs, which
+line ranges differ?  Whether a particular feature consults it follows
+from whether that is the question the feature is really asking.
+
+Features that ask "which lines changed" use the tool's hunks in place
+of the builtin algorithm:
+
+- `git diff` patch output, together with everything layered on it:
+  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
+  headers, and the `-L` line-range display.  These operate on the
+  lines the patch step already emitted, so they reflect the tool's
+  hunks without any further negotiation.
+- `git blame`: a commit whose change the tool reports as equivalent is
+  skipped, and its lines are attributed to an earlier commit.
+
+Features that ask a different question do not consult the process, by
+design:
+
+- The pickaxe `-G<regex>` searches the textual diff for a pattern; it
+  asks "does this string appear in the diff," not "did these lines
+  change."  (`-S` runs at an earlier stage and is likewise unaffected.)
+- `git patch-id` must produce a stable hash for `git rebase` and
+  cherry-pick detection; deriving it from a configured tool would make
+  equal patches hash differently from machine to machine.
+- The merge machinery (`git merge-tree`, `rerere`) computes merge
+  content and conflict signatures rather than display output, so the
+  tool's hunks must not alter its results.
+- `git range-diff` diffs patch text, not source blobs, so source-file
+  hunks do not apply to it.
+- `--check` reports whitespace errors in added lines using the builtin
+  diff's notion of which lines are added, not the tool's.  It can
+  therefore flag (and exit non-zero on) a line the tool treats as
+  unchanged and that `git diff` shows as context.  Whitespace breakage
+  is a property of the literal bytes, so `--check` keeps the builtin
+  partition deliberately; a future change could wire it to the tool if
+  matching `git diff` exactly became desirable.
+- `--raw`, `--name-only`, and `--name-status` compare object ids at
+  the tree level and never run a line-level diff at all.
+
+Some features ask "which lines changed" but still use the builtin
+algorithm for now, and may consult the process in a later change: the
+summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
+commit selection and parent range propagation (as distinct from its
+display, which is covered above); and combined diffs (`--cc` and merge
+diffs), whose protocol would have to be extended from a single old/new
+pair to one comparison per merge parent.
+
+`--diff-algorithm` bypasses the process entirely, for every feature
+listed above.  The whitespace-ignoring options (`-w`,
+`--ignore-space-change`, `--ignore-blank-lines`, and the like),
+`-I<regex>`, and `--anchored` also bypass it for the affected files:
+the tool is never told about these options, so it could not honor
+them, and Git falls back to the builtin diff, which does.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index b31ecb0756..3a167c73de 100644
--- a/Makefile
+++ b/Makefile
@@ -811,6 +811,7 @@ TEST_BUILTINS_OBJS += test-csprng.o
 TEST_BUILTINS_OBJS += test-date.o
 TEST_BUILTINS_OBJS += test-delete-gpgsig.o
 TEST_BUILTINS_OBJS += test-delta.o
+TEST_BUILTINS_OBJS += test-diff-process-backend.o
 TEST_BUILTINS_OBJS += test-dir-iterator.o
 TEST_BUILTINS_OBJS += test-drop-caches.o
 TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1140,6 +1141,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..4c748fdd2a
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,491 @@
+/*
+ * Diff process backend: communicates with a long-running external
+ * tool via the pkt-line protocol to obtain custom line-matching
+ * results.  The tool controls which lines are marked as changed
+ * while the display shows the file content (after any textconv
+ * transformation, if configured).
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   tool< git-diff-server / version=1 / flush
+ *   git> capability=hunks / flush
+ *   tool< capability=hunks / flush
+ *
+ * Per-file:
+ *   git> command=hunks / pathname=<path> / [old-oid=<hex>] / [new-oid=<hex>] / flush
+ *   git> <old content packetized> / flush
+ *   git> <new content packetized> / flush
+ *   tool< hunk <old_start> <old_count> <new_start> <new_count>
+ *   tool< ... / flush
+ *   tool< status=success / flush
+ *
+ * When the tool returns no hunks with status=success, it considers
+ * the files equivalent.  Git will skip the diff for that file.
+ */
+
+#include "git-compat-util.h"
+#include "diff-process.h"
+#include "diff.h"
+#include "gettext.h"
+#include "hex.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+#include "xdiff/xdiff.h"
+
+#define CAP_HUNKS (1u << 0)
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	unsigned int supported_capabilities;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks", CAP_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+static struct diff_subprocess *get_or_launch_process(
+		struct userdiff_driver *drv)
+{
+	struct diff_subprocess *entry;
+
+	if (drv->diff_subprocess)
+		return drv->diff_subprocess;
+
+	entry = xcalloc(1, sizeof(*entry));
+	if (subprocess_start_command(&entry->subprocess, drv->process,
+				     start_diff_process_fn)) {
+		free(entry);
+		drv->diff_process_failed = 1;
+		return NULL;
+	}
+
+	drv->diff_subprocess = entry;
+	return entry;
+}
+
+static int send_file_content(int fd, const char *buf, long size)
+{
+	int ret = 0;
+
+	if (size < 0)
+		return -1;
+	if (size > 0)
+		ret = write_packetized_from_buf_no_flush(buf, size, fd);
+	if (ret)
+		return ret;
+	return packet_flush_gently(fd);
+}
+
+/*
+ * A hunk in the diff process's presentation coordinates: the line
+ * numbering it reports over the protocol.  Kept distinct from struct
+ * xdl_hunk (xdiff's coordinates) so that only translated hunks ever
+ * reach the diff algorithm; diff_process_hunk_to_xdl() is the single
+ * crossing point.
+ */
+struct diff_process_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
+/*
+ * Parse one non-negative decimal field of a hunk line into *out and
+ * advance *line past it.  Fields must be plain decimal with no leading
+ * whitespace or sign (isdigit() takes an unsigned char to stay defined
+ * for high-bit bytes).  The first three fields are followed by a single
+ * space; the last (is_last) is followed by end-of-string or a space.
+ * Trailing space-separated tokens after the last field are allowed and
+ * ignored, so a future protocol version can append fields (e.g. a
+ * "moved" marker) without older tools rejecting the line -- mirroring
+ * the request-side rule that tools ignore unknown keys.
+ */
+static int parse_hunk_field(const char **line, long *out, int is_last)
+{
+	const char *p = *line;
+	char *end;
+
+	if (!isdigit((unsigned char)*p))
+		return -1;
+	errno = 0;
+	*out = strtol(p, &end, 10);
+	if (errno || end == p)
+		return -1;
+	if (is_last) {
+		if (*end != '\0' && *end != ' ')
+			return -1;
+	} else {
+		if (*end != ' ')
+			return -1;
+		end++;
+	}
+	*line = end;
+	return 0;
+}
+
+static int parse_hunk_line(const char *line,
+			   struct diff_process_hunk *presented)
+{
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+	if (parse_hunk_field(&line, &presented->old_start, 0) ||
+	    parse_hunk_field(&line, &presented->old_count, 0) ||
+	    parse_hunk_field(&line, &presented->new_start, 0) ||
+	    parse_hunk_field(&line, &presented->new_count, 1))
+		return -1;
+	return 0;
+}
+
+/*
+ * Translate a hunk from the diff process's presentation coordinates
+ * into xdiff's.
+ *
+ * Protocol starts are already 1-based positions (the line a change
+ * sits before), the same numbering xdiff uses, so the only adjustment
+ * is for an empty file side: "git diff" addresses it with a start of 0
+ * and a count of 0 (e.g. "0 0 1 5" adds five lines to an empty old
+ * side), and since xdiff uses start-1 as an array index that 0 becomes
+ * 1 here.  This is NOT the full inverse of xdl_emit_hunk_hdr()
+ * (xdiff/xutils.c): that emitter shifts a count-0 range to start-1 for
+ * the displayed "@@" header, but the protocol keeps the unshifted
+ * 1-based position for a mid-file insert or delete.  This is the single
+ * point where presentation coordinates become xdiff coordinates, so
+ * xdl_populate_hunks_from_external() may assume 1-based starts.
+ *
+ * Returns -1 for a start of 0 paired with a nonzero count, which names
+ * no line in either coordinate system.  (parse_hunk_line() already
+ * guarantees non-negative starts and counts.)
+ */
+static int diff_process_hunk_to_xdl(const struct diff_process_hunk *presented,
+				    struct xdl_hunk *xdl)
+{
+	long old_start = presented->old_start;
+	long new_start = presented->new_start;
+
+	if ((!old_start && presented->old_count) ||
+	    (!new_start && presented->new_count))
+		return -1;
+	if (!old_start)
+		old_start = 1;
+	if (!new_start)
+		new_start = 1;
+
+	xdl->old_start = old_start;
+	xdl->old_count = presented->old_count;
+	xdl->new_start = new_start;
+	xdl->new_count = presented->new_count;
+	return 0;
+}
+
+static enum diff_process_result get_hunks(
+		struct userdiff_driver *drv,
+		const char *path,
+		const char *old_buf, long old_size,
+		const char *new_buf, long new_size,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		struct xdl_hunk **hunks_out,
+		size_t *nr_hunks_out)
+{
+	struct diff_subprocess *backend;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct diff_process_hunk presented;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0;
+	size_t max_hunks;
+	int len;
+	char *line;
+
+	backend = get_or_launch_process(drv);
+	if (!backend)
+		return DIFF_PROCESS_ERROR;
+
+	if (!(backend->supported_capabilities & CAP_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&backend->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	/* Send request */
+	if (packet_write_fmt_gently(fd_in, "command=hunks\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path))
+		goto comm_error;
+	/*
+	 * old-oid/new-oid let the tool key a cache on the blob pair.  A
+	 * side is sent only when its content is the raw blob (the caller
+	 * passes NULL otherwise, e.g. for textconv'd content), so an oid
+	 * that is present always names the bytes the tool receives.
+	 */
+	if (oid_a &&
+	    packet_write_fmt_gently(fd_in, "old-oid=%s\n", oid_to_hex(oid_a)))
+		goto comm_error;
+	if (oid_b &&
+	    packet_write_fmt_gently(fd_in, "new-oid=%s\n", oid_to_hex(oid_b)))
+		goto comm_error;
+	if (packet_flush_gently(fd_in))
+		goto comm_error;
+
+	/* Send old file content */
+	if (send_file_content(fd_in, old_buf, old_size))
+		goto comm_error;
+
+	/* Send new file content */
+	if (send_file_content(fd_in, new_buf, new_size))
+		goto comm_error;
+
+	/*
+	 * Hunks are non-overlapping and each useful hunk covers at least
+	 * one line, so a valid response cannot contain more hunks than the
+	 * two files have lines, which is bounded by their byte sizes.  Cap
+	 * the accumulation accordingly so a misbehaving tool that floods
+	 * hunk lines cannot drive unbounded memory growth before validation.
+	 */
+	max_hunks = (size_t)old_size + (size_t)new_size + 1;
+
+	/* Read hunks until flush packet */
+	while ((len = packet_read_line_gently(fd_out, NULL, &line)) >= 0 &&
+	       line) {
+		if (parse_hunk_line(line, &presented) < 0)
+			goto comm_error;
+		if (diff_process_hunk_to_xdl(&presented, &hunk) < 0)
+			goto comm_error;
+		if (nr_hunks >= max_hunks) {
+			warning(_("diff process '%s' sent too many hunks"
+				  " for '%s'"), drv->process, path);
+			goto comm_error;
+		}
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+	if (len < 0)
+		goto comm_error;
+
+	/* Read status */
+	if (subprocess_read_status(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		*hunks_out = hunks;
+		*nr_hunks_out = nr_hunks;
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_OK;
+	}
+
+	if (!strcmp(status.buf, "abort")) {
+		/*
+		 * The tool voluntarily withdrew: stop sending requests
+		 * but do not warn (this is not a failure).
+		 */
+		backend->supported_capabilities &= ~CAP_HUNKS;
+		free(hunks);
+		strbuf_release(&status);
+		sigchain_pop(SIGPIPE);
+		return DIFF_PROCESS_SKIP;
+	}
+
+	/* status=error or unknown status */
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+
+comm_error:
+	/*
+	 * Communication failure (broken pipe, malformed response).
+	 * Tear down the process and mark as failed so we do not
+	 * retry on every subsequent file.
+	 */
+	drv->diff_process_failed = 1;
+	drv->diff_subprocess = NULL;
+	subprocess_stop_command(&backend->subprocess);
+	free(backend);
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+/*
+ * Whether exactly one of the two blobs ends in a newline.  A change
+ * that only adds or removes the trailing newline is not expressible as
+ * line hunks, so a tool comparing lines reports the files as equal.
+ */
+static int eof_newline_differs(const mmfile_t *a, const mmfile_t *b)
+{
+	int a_nl = a->size > 0 && a->ptr[a->size - 1] == '\n';
+	int b_nl = b->size > 0 && b->ptr[b->size - 1] == '\n';
+	return a_nl != b_nl;
+}
+
+/*
+ * Number of lines in a blob, matching xdiff's record count: one per
+ * newline, plus one more if the last line has no trailing newline.
+ */
+static long count_lines(const char *buf, long size)
+{
+	long lines = 0, i;
+
+	for (i = 0; i < size; i++)
+		if (buf[i] == '\n')
+			lines++;
+	if (size > 0 && buf[size - 1] != '\n')
+		lines++;
+	return lines;
+}
+
+/*
+ * Validate the tool's hunks (already in xdiff coordinates) against the
+ * two blobs before they bypass the diff algorithm.  Each hunk must fit
+ * within its file, the hunks must be ordered and non-overlapping, and
+ * the unchanged run before each hunk (and after the last) must be the
+ * same length on both sides -- xdl_build_script() walks the two files
+ * in lockstep over unchanged lines, so a mismatched gap desynchronizes
+ * it and yields a corrupt diff even when the totals balance.  This is
+ * the git layer's job so xdiff stays diagnostic-free; on a bad response
+ * we warn and the caller falls back to the builtin diff.  Returns 0 if
+ * valid, -1 (after warning) otherwise.
+ */
+static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,
+				   long old_lines, long new_lines,
+				   const char *process, const char *path)
+{
+	size_t i;
+	long prev_old_end = 0, prev_new_end = 0;
+
+	for (i = 0; i < nr; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (h->old_count > old_lines - h->old_start + 1 ||
+		    h->new_count > new_lines - h->new_start + 1) {
+			warning(_("diff process '%s' returned a hunk past the "
+				  "end of '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		if (h->old_start < prev_old_end || h->new_start < prev_new_end) {
+			warning(_("diff process '%s' returned overlapping hunks "
+				  "for '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		if (h->old_start - prev_old_end != h->new_start - prev_new_end) {
+			warning(_("diff process '%s' returned hunks that leave "
+				  "'%s' misaligned; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		prev_old_end = h->old_start + h->old_count;
+		prev_new_end = h->new_start + h->new_count;
+	}
+	if (old_lines - prev_old_end != new_lines - prev_new_end) {
+		warning(_("diff process '%s' returned hunks that leave '%s' "
+			  "misaligned; using the builtin diff"),
+			process, path);
+		return -1;
+	}
+	return 0;
+}
+
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+	struct xdl_hunk *ext_hunks = NULL;
+	size_t nr = 0;
+	enum diff_process_result res;
+
+	if (!diffopt || !path)
+		return DIFF_PROCESS_SKIP;
+	if (diffopt->flags.no_diff_process || diffopt->ignore_driver_algorithm)
+		return DIFF_PROCESS_SKIP;
+	/*
+	 * Whitespace-ignoring, regex-ignore (-I) and anchored options
+	 * change which lines count as different, but the tool is never
+	 * told about them, so its hunks could not honor them.  Rather
+	 * than silently override the user's request, fall back to the
+	 * builtin diff, which does honor these flags.  Key this off xpp
+	 * (the parameters this diff actually runs with) rather than
+	 * diffopt, so a caller like blame that keeps its flags outside
+	 * diffopt is covered without a separate guard of its own.
+	 */
+	if ((xpp->flags & (XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES)) ||
+	    xpp->ignore_regex_nr || xpp->anchors_nr)
+		return DIFF_PROCESS_SKIP;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return DIFF_PROCESS_SKIP;
+	if (drv->diff_process_failed)
+		return DIFF_PROCESS_SKIP;
+
+	res = get_hunks(drv, path,
+			file_a->ptr, file_a->size,
+			file_b->ptr, file_b->size,
+			oid_a, oid_b,
+			&ext_hunks, &nr);
+	if (res == DIFF_PROCESS_OK) {
+		if (!nr) {
+			free(ext_hunks);
+			/*
+			 * Zero hunks means the tool considers the line
+			 * content identical, but it cannot express a
+			 * trailing-newline-only change.  When that is the
+			 * actual difference, fall back to the builtin diff
+			 * so the "\ No newline at end of file" marker is
+			 * preserved instead of reporting the files equal.
+			 */
+			if (eof_newline_differs(file_a, file_b))
+				return DIFF_PROCESS_SKIP;
+			return DIFF_PROCESS_EQUIVALENT;
+		}
+		if (validate_external_hunks(ext_hunks, nr,
+					    count_lines(file_a->ptr, file_a->size),
+					    count_lines(file_b->ptr, file_b->size),
+					    drv->process, path) < 0) {
+			free(ext_hunks);
+			return DIFF_PROCESS_SKIP;
+		}
+		xpp->external_hunks = ext_hunks;
+		xpp->external_hunks_nr = nr;
+		return DIFF_PROCESS_OK;
+	}
+	if (res == DIFF_PROCESS_ERROR) {
+		warning(_("diff process '%s' failed for '%s',"
+			  " falling back to builtin diff"),
+			drv->process, path);
+		return DIFF_PROCESS_ERROR;
+	}
+	return DIFF_PROCESS_SKIP;
+}
diff --git a/diff-process.h b/diff-process.h
new file mode 100644
index 0000000000..8d00dafe1d
--- /dev/null
+++ b/diff-process.h
@@ -0,0 +1,49 @@
+#ifndef DIFF_PROCESS_H
+#define DIFF_PROCESS_H
+
+#include "xdiff/xdiff.h"
+
+struct diff_options;
+struct object_id;
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* failed; caller falls back to builtin */
+	DIFF_PROCESS_OK = 0,     /* hunks populated in xpp */
+	DIFF_PROCESS_SKIP,       /* process did not apply: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* tool says files are equivalent */
+};
+
+/*
+ * Consult the diff process configured for 'path' and populate
+ * xpp->external_hunks with the returned hunks.
+ *
+ * Handles driver lookup, flag checks (--no-ext-diff,
+ * --diff-algorithm), subprocess management, and error reporting.
+ *
+ * Returns DIFF_PROCESS_OK when hunks are populated in xpp.
+ * The caller owns xpp->external_hunks and must free() it.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT when the tool returns no hunks and
+ * the blobs are not a trailing-newline-only change (files are
+ * considered identical); caller should skip diff/blame.
+ * Returns DIFF_PROCESS_SKIP when no process applies; caller
+ * should use the builtin diff algorithm.
+ * Returns DIFF_PROCESS_ERROR on tool failure (already warned);
+ * caller should fall back to the builtin diff algorithm.
+ *
+ * oid_a/oid_b, when non-NULL, are sent to the tool as old-oid/new-oid
+ * so it can key a cache on the blob pair.  Pass NULL for a side whose
+ * content is not the raw blob (e.g. textconv'd) or whose object name is
+ * unknown, so any oid that is sent always names the bytes the tool
+ * receives.
+ */
+enum diff_process_result diff_process_fill_hunks(
+		struct diff_options *diffopt,
+		const char *path,
+		const mmfile_t *file_a,
+		const mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp);
+
+#endif /* DIFF_PROCESS_H */
diff --git a/diff.c b/diff.c
index a8f346621b..aab012f922 100644
--- a/diff.c
+++ b/diff.c
@@ -25,6 +25,7 @@
 #include "utf8.h"
 #include "odb.h"
 #include "userdiff.h"
+#include "diff-process.h"
 #include "submodule.h"
 #include "hashmap.h"
 #include "mem-pool.h"
@@ -4164,6 +4165,25 @@ static void builtin_diff(const char *name_a,
 		xpp.ignore_regex_nr = o->ignore_regex_nr;
 		xpp.anchors = o->anchors;
 		xpp.anchors_nr = o->anchors_nr;
+
+		/*
+		 * Send the blob oids only for a side whose content is the
+		 * raw blob: textconv rewrites the bytes, and a working-tree
+		 * side has no stored oid, so pass NULL there rather than an
+		 * oid that would not name what the tool receives.
+		 */
+		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
+					    (textconv_one || !one->oid_valid) ? NULL : &one->oid,
+					    (textconv_two || !two->oid_valid) ? NULL : &two->oid,
+					    &xpp)
+		    == DIFF_PROCESS_EQUIVALENT) {
+			if (textconv_one)
+				free(mf1.ptr);
+			if (textconv_two)
+				free(mf2.ptr);
+			goto free_ab_and_return;
+		}
+
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_FUNCNAMES;
@@ -4208,6 +4228,7 @@ static void builtin_diff(const char *name_a,
 		} else if (xdi_diff_outf(&mf1, &mf2, NULL, fn_out_consume,
 					 &ecbdata, &xpp, &xecfg))
 			die("unable to generate diff for %s", one->path);
+		free(xpp.external_hunks);
 		if (o->word_diff)
 			free_diff_words_data(&ecbdata);
 		if (textconv_one)
diff --git a/diff.h b/diff.h
index bb5cddaf34..7dc157968d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,9 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/** Disables diff.<driver>.process. */
+	unsigned no_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 064fe2e2f1..613244b35d 100644
--- a/meson.build
+++ b/meson.build
@@ -328,6 +328,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 3235f10ab8..6abcda4afb 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -12,6 +12,7 @@ test_tool_sources = [
   'test-date.c',
   'test-delete-gpgsig.c',
   'test-delta.c',
+  'test-diff-process-backend.c',
   'test-dir-iterator.c',
   'test-drop-caches.c',
   'test-dump-cache-tree.c',
diff --git a/t/helper/test-diff-process-backend.c b/t/helper/test-diff-process-backend.c
new file mode 100644
index 0000000000..c2ec532c4a
--- /dev/null
+++ b/t/helper/test-diff-process-backend.c
@@ -0,0 +1,381 @@
+/*
+ * Test backend for the long-running diff process protocol
+ * (see diff-process.c and Documentation/gitattributes.adoc).
+ *
+ * Usage: test-tool diff-process-backend --mode=<mode> [--log=<path>]
+ *
+ * Implements the server side of the pkt-line handshake and a per-file
+ * response loop.  The --mode= switch selects the response shape
+ * (success, error, abort, crash, malformed hunks).
+ *
+ * Per-file request from Git:
+ *
+ *   packet:          git> command=hunks
+ *   packet:          git> pathname=<path>
+ *   packet:          git> [old-oid=<hex>]   (omitted for textconv/worktree)
+ *   packet:          git> [new-oid=<hex>]
+ *   packet:          git> 0000
+ *   packet:          git> OLD_CONTENT
+ *   packet:          git> 0000
+ *   packet:          git> NEW_CONTENT
+ *   packet:          git> 0000
+ *
+ * Response varies by --mode (default: whole-file):
+ *
+ *   whole-file   packet: git< hunk <1|0> <old_lines> <1|0> <new_lines>
+ *                (start is 0 for an empty side, matching git diff)
+ *   fixed-hunk   packet: git< hunk 5 2 5 2
+ *   no-hunks     (no hunk packets)
+ *   bad-hunk     packet: git< hunk 999 1 999 1
+ *   bad-parse    packet: git< garbage not a hunk
+ *   bad-sync     packet: git< hunk 1 2 1 1
+ *   bad-gap      packet: git< hunk 1 1 3 1
+ *   bad-start    packet: git< hunk 0 1 1 1
+ *   multi-hunk   packet: git< hunk 5 2 5 2
+ *                packet: git< hunk 9 2 9 2
+ *   insert       packet: git< hunk 3 0 3 2   (mid-file count-0 insertion)
+ *   flood        packet: git< hunk 1 1 1 1   (x100000)
+ *   overlap      packet: git< hunk 1 5 1 5
+ *                packet: git< hunk 3 2 3 2
+ *   no-cap       (omits capability=hunks during handshake)
+ *   error        (status=error instead of status=success)
+ *   abort        (status=abort instead of status=success)
+ *   crash        exit(1) before sending any response
+ *
+ * All success modes (not error/abort/crash) end with:
+ *
+ *   packet:          git< 0000
+ *   packet:          git< status=success
+ *   packet:          git< 0000
+ *
+ * Each request is logged to --log as:
+ *
+ *   command=<cmd> pathname=<path> old-oid=<hex> new-oid=<hex> old=<first line> new=<first line>
+ */
+
+#include "test-tool.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "strbuf.h"
+
+static FILE *logfile;
+
+enum mode {
+	MODE_WHOLE_FILE,
+	MODE_FIXED_HUNK,
+	MODE_NO_HUNKS,
+	MODE_BAD_HUNK,
+	MODE_BAD_PARSE,
+	MODE_BAD_SYNC,
+	MODE_BAD_GAP,
+	MODE_BAD_START,
+	MODE_MULTI_HUNK,
+	MODE_INSERT,
+	MODE_FLOOD,
+	MODE_OVERLAP,
+	MODE_NO_CAP,
+	MODE_ERROR,
+	MODE_ABORT,
+	MODE_CRASH,
+};
+
+static enum mode parse_mode(const char *s)
+{
+	if (!strcmp(s, "whole-file"))
+		return MODE_WHOLE_FILE;
+	if (!strcmp(s, "fixed-hunk"))
+		return MODE_FIXED_HUNK;
+	if (!strcmp(s, "no-hunks"))
+		return MODE_NO_HUNKS;
+	if (!strcmp(s, "bad-hunk"))
+		return MODE_BAD_HUNK;
+	if (!strcmp(s, "bad-parse"))
+		return MODE_BAD_PARSE;
+	if (!strcmp(s, "bad-sync"))
+		return MODE_BAD_SYNC;
+	if (!strcmp(s, "bad-gap"))
+		return MODE_BAD_GAP;
+	if (!strcmp(s, "bad-start"))
+		return MODE_BAD_START;
+	if (!strcmp(s, "multi-hunk"))
+		return MODE_MULTI_HUNK;
+	if (!strcmp(s, "insert"))
+		return MODE_INSERT;
+	if (!strcmp(s, "flood"))
+		return MODE_FLOOD;
+	if (!strcmp(s, "overlap"))
+		return MODE_OVERLAP;
+	if (!strcmp(s, "no-cap"))
+		return MODE_NO_CAP;
+	if (!strcmp(s, "error"))
+		return MODE_ERROR;
+	if (!strcmp(s, "abort"))
+		return MODE_ABORT;
+	if (!strcmp(s, "crash"))
+		return MODE_CRASH;
+	die("unknown --mode=%s", s);
+}
+
+/*
+ * Read "key=value" packets up to a flush, capturing "command" and
+ * "pathname".  Returns 1 if a request was read, 0 on EOF.
+ *
+ * The first packet uses the gentle variant so that a clean shutdown
+ * by Git (EOF) does not produce a spurious "the remote end hung up
+ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
+ * variant: once inside a request, truncation is a protocol violation
+ * and dying loudly is the correct response.
+ */
+static int read_request_header(char **command, char **pathname,
+			       char **old_oid, char **new_oid)
+{
+	int first = 1;
+	char *line;
+
+	*command = *pathname = *old_oid = *new_oid = NULL;
+	for (;;) {
+		const char *value;
+
+		if (first) {
+			if (packet_read_line_gently(0, NULL, &line) < 0)
+				return 0;
+			first = 0;
+		} else {
+			line = packet_read_line(0, NULL);
+		}
+		if (!line)
+			break;
+		if (skip_prefix(line, "command=", &value))
+			*command = xstrdup(value);
+		else if (skip_prefix(line, "pathname=", &value))
+			*pathname = xstrdup(value);
+		else if (skip_prefix(line, "old-oid=", &value))
+			*old_oid = xstrdup(value);
+		else if (skip_prefix(line, "new-oid=", &value))
+			*new_oid = xstrdup(value);
+	}
+	return 1;
+}
+
+static size_t count_lines(const struct strbuf *buf)
+{
+	size_t lines = 0;
+
+	for (size_t i = 0; i < buf->len; i++)
+		if (buf->buf[i] == '\n')
+			lines++;
+
+	return lines + (buf->len > 0 && buf->buf[buf->len - 1] != '\n');
+}
+
+static void send_status(const char *status)
+{
+	packet_flush(1);
+	packet_write_fmt(1, "%s\n", status);
+	packet_flush(1);
+}
+
+static void respond(enum mode mode,
+		    const struct strbuf *old_buf,
+		    const struct strbuf *new_buf)
+{
+	switch (mode) {
+	case MODE_ERROR:
+		send_status("status=error");
+		return;
+	case MODE_ABORT:
+		send_status("status=abort");
+		return;
+	case MODE_CRASH:
+		exit(1);
+	case MODE_FIXED_HUNK:
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		break;
+	case MODE_BAD_HUNK:
+		packet_write_fmt(1, "hunk 999 1 999 1\n");
+		break;
+	case MODE_BAD_PARSE:
+		packet_write_fmt(1, "garbage not a hunk\n");
+		break;
+	case MODE_BAD_SYNC:
+		packet_write_fmt(1, "hunk 1 2 1 1\n");
+		break;
+	case MODE_BAD_GAP:
+		/*
+		 * Globally balanced (1 changed line on each side, so the
+		 * total unchanged counts match) but the gap before the
+		 * change differs between sides: old line 1 vs new line 3.
+		 * Exercises the per-gap lockstep-alignment check.
+		 */
+		packet_write_fmt(1, "hunk 1 1 3 1\n");
+		break;
+	case MODE_BAD_START:
+		/*
+		 * A start of 0 is valid only for an empty (count 0) range;
+		 * pairing it with a nonzero count names no line in either
+		 * the protocol's or xdiff's coordinates, so the translation
+		 * rejects it and git falls back to the builtin diff.
+		 */
+		packet_write_fmt(1, "hunk 0 1 1 1\n");
+		break;
+	case MODE_MULTI_HUNK:
+		/*
+		 * Two valid, non-overlapping, gap-aligned hunks.  Exercises
+		 * the accepting branch of the per-gap lockstep check with a
+		 * non-zero previous-hunk end (the realistic two-region case).
+		 */
+		packet_write_fmt(1, "hunk 5 2 5 2\n");
+		packet_write_fmt(1, "hunk 9 2 9 2\n");
+		break;
+	case MODE_INSERT:
+		/*
+		 * A mid-file pure insertion (count 0 on the old side) in the
+		 * protocol's 1-based-position form: 2 lines inserted before
+		 * old line 3.  Exercises the count-0 path, which uses the
+		 * unshifted position (not git diff's "-3,0" display start).
+		 */
+		packet_write_fmt(1, "hunk 3 0 3 2\n");
+		break;
+	case MODE_FLOOD: {
+		/*
+		 * Emit far more hunks than any small file has lines, so Git
+		 * trips its accumulation cap and falls back before reading
+		 * them all.
+		 */
+		int i;
+		for (i = 0; i < 100000; i++)
+			packet_write_fmt(1, "hunk 1 1 1 1\n");
+		break;
+	}
+	case MODE_OVERLAP:
+		packet_write_fmt(1, "hunk 1 5 1 5\n");
+		packet_write_fmt(1, "hunk 3 2 3 2\n");
+		break;
+	case MODE_NO_HUNKS:
+		break;
+	case MODE_NO_CAP:
+	case MODE_WHOLE_FILE: {
+		size_t old_lines = count_lines(old_buf);
+		size_t new_lines = count_lines(new_buf);
+		/*
+		 * Match git diff output: start=0 when count=0
+		 * (empty file side), 1 otherwise.
+		 */
+		packet_write_fmt(1, "hunk %"PRIuMAX" %"PRIuMAX
+				 " %"PRIuMAX" %"PRIuMAX"\n",
+				 (uintmax_t)(old_lines ? 1 : 0),
+				 (uintmax_t)old_lines,
+				 (uintmax_t)(new_lines ? 1 : 0),
+				 (uintmax_t)new_lines);
+		break;
+	}
+	}
+	send_status("status=success");
+}
+
+static void command_loop(enum mode mode)
+{
+	for (;;) {
+		char *command = NULL, *pathname = NULL;
+		char *old_oid = NULL, *new_oid = NULL;
+		struct strbuf obuf = STRBUF_INIT;
+		struct strbuf nbuf = STRBUF_INIT;
+
+		if (!read_request_header(&command, &pathname,
+					 &old_oid, &new_oid))
+			break; /* EOF: Git closed its end */
+
+		read_packetized_to_strbuf(0, &obuf, 0);
+		read_packetized_to_strbuf(0, &nbuf, 0);
+
+		if (logfile) {
+			fprintf(logfile,
+				"command=%s pathname=%s old-oid=%s new-oid=%s"
+				" old=%.*s new=%.*s\n",
+				command ? command : "(none)",
+				pathname ? pathname : "(none)",
+				old_oid ? old_oid : "(none)",
+				new_oid ? new_oid : "(none)",
+				(int)(strchrnul(obuf.buf, '\n') - obuf.buf),
+				obuf.buf,
+				(int)(strchrnul(nbuf.buf, '\n') - nbuf.buf),
+				nbuf.buf);
+			fflush(logfile);
+		}
+
+		respond(mode, &obuf, &nbuf);
+
+		free(command);
+		free(pathname);
+		free(old_oid);
+		free(new_oid);
+		strbuf_release(&obuf);
+		strbuf_release(&nbuf);
+	}
+}
+
+static void handshake(enum mode mode)
+{
+	char *line;
+
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "git-diff-client"))
+		die("bad welcome: '%s'", line ? line : "(eof)");
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "version=1"))
+		die("bad version: '%s'", line ? line : "(eof)");
+	if (packet_read_line(0, NULL))
+		die("expected flush after version");
+
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Drain capabilities advertised by Git */
+	while ((line = packet_read_line(0, NULL)))
+		; /* drain */
+
+	/* Respond with our capabilities (or none for no-cap mode) */
+	if (mode != MODE_NO_CAP)
+		packet_write_fmt(1, "capability=hunks\n");
+	packet_flush(1);
+}
+
+static const char *const usage_str[] = {
+	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
+	NULL
+};
+
+int cmd__diff_process_backend(int argc, const char **argv)
+{
+	const char *mode_str = NULL, *log_path = NULL;
+	enum mode mode = MODE_WHOLE_FILE;
+	struct option options[] = {
+		OPT_STRING(0, "mode", &mode_str, "mode",
+			   "response shape (default whole-file);"
+			   " see the file header for the full list of modes"),
+		OPT_STRING(0, "log", &log_path, "path",
+			   "append per-request summary to this file"),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
+	if (argc)
+		usage_with_options(usage_str, options);
+
+	if (mode_str)
+		mode = parse_mode(mode_str);
+
+	if (log_path) {
+		logfile = fopen(log_path, "a");
+		if (!logfile)
+			die_errno("failed to open log '%s'", log_path);
+	}
+
+	handshake(mode);
+	command_loop(mode);
+
+	if (logfile && fclose(logfile))
+		die_errno("error closing log");
+	return 0;
+}
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index b71a22b43b..3c3f95269c 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = {
 	{ "date", cmd__date },
 	{ "delete-gpgsig", cmd__delete_gpgsig },
 	{ "delta", cmd__delta },
+	{ "diff-process-backend", cmd__diff_process_backend },
 	{ "dir-iterator", cmd__dir_iterator },
 	{ "drop-caches", cmd__drop_caches },
 	{ "dump-cache-tree", cmd__dump_cache_tree },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f2885b33d5..a5bb755516 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);
 int cmd__date(int argc, const char **argv);
 int cmd__delta(int argc, const char **argv);
 int cmd__delete_gpgsig(int argc, const char **argv);
+int cmd__diff_process_backend(int argc, const char **argv);
 int cmd__dir_iterator(int argc, const char **argv);
 int cmd__drop_caches(int argc, const char **argv);
 int cmd__dump_cache_tree(int argc, const char **argv);
diff --git a/t/meson.build b/t/meson.build
index c5832fee05..027855ced7 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -512,6 +512,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..3b75df082e
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,645 @@
+#!/bin/sh
+
+test_description='diff process via long-running process'
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+# See t/helper/test-diff-process-backend.c for the backend implementation
+# and available --mode= options.
+
+BACKEND="test-tool diff-process-backend"
+
+test_expect_success 'setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# boundary.c: 10 lines, changes at 5-6 and 9-10.
+	# Used by: hunk boundaries, error fallback, crash, bad hunks, overlap.
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add boundary.c &&
+
+	# worddiff.c: single-line function, value changes 1 -> 999.
+	# Used by: word-diff, --diff-algorithm, --no-ext-diff, --stat.
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 1; }
+	EOF
+	git add worddiff.c &&
+
+	# newfile.c: single-line function, value changes 42 -> 99.
+	# Used by: modified file, --exit-code, multiple drivers.
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 42; }
+	EOF
+	git add newfile.c &&
+
+	# logtest.c: single-line function for log/format-patch tests.
+	# Needs two commits so log -1 has a diff.
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 1; }
+	EOF
+	git add logtest.c &&
+
+	# one.c/two.c: two-file pair for error/abort/startup-failure tests.
+	cat >one.c <<-\EOF &&
+	int first(void) { return 1; }
+	EOF
+	cat >two.c <<-\EOF &&
+	int second(void) { return 2; }
+	EOF
+	git add one.c two.c &&
+
+	git commit -m "initial" &&
+
+	# Second commit for logtest.c (so log -1 has something to show).
+	cat >logtest.c <<-\EOF &&
+	int logfunc(void) { return 2; }
+	EOF
+	git add logtest.c &&
+	git commit -m "change logtest.c" &&
+
+	# Working tree modifications (not committed).
+	cat >boundary.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+
+	cat >worddiff.c <<-\EOF &&
+	int value(void) { return 999; }
+	EOF
+
+	cat >newfile.c <<-\EOF &&
+	int new_func(void) { return 99; }
+	EOF
+
+	cat >one.c <<-\EOF &&
+	int first(void) { return 10; }
+	EOF
+
+	cat >two.c <<-\EOF
+	int second(void) { return 20; }
+	EOF
+'
+
+#
+# Core behavior: the tool controls which lines are marked as changed.
+#
+
+test_expect_success 'diff process hunk boundaries affect output' '
+	# The file has changes at lines 5-6 and 9-10, but fixed-hunk
+	# only reports lines 5-6 as changed.  Lines 9-10 should not
+	# appear as changed in the output.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff boundary.c >actual &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD6" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW6" actual &&
+	test_grep ! "^-OLD9" actual &&
+	test_grep ! "^-OLD10" actual &&
+	test_grep ! "^+NEW9" actual &&
+	test_grep ! "^+NEW10" actual
+'
+
+test_expect_success 'diff process accepts valid multi-hunk output' '
+	# multi-hunk reports both changed regions (5-6 and 9-10) as two
+	# gap-aligned hunks.  This exercises the accepting branch of the
+	# per-gap lockstep check (non-zero previous-hunk end) and must
+	# produce a correct two-region diff with the lines between the
+	# hunks kept as context.
+	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "^ line7" actual &&
+	test_grep "^ line8" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process accepts a mid-file count-0 insertion' '
+	# insert mode reports "hunk 3 0 3 2": a pure insertion (count 0 on
+	# the old side) in the protocol 1-based-position form.  Exercises
+	# the count-0 hunk path that the other valid-hunk modes (full
+	# replacements, equal-count modifies) never hit.  Empty stderr is
+	# the discriminator: a mishandled count-0 start would be rejected
+	# by the lockstep check and warn.
+	cat >insert.c <<-\EOF &&
+	a
+	b
+	c
+	d
+	e
+	EOF
+	git add insert.c &&
+	git commit -m "add insert.c" &&
+	cat >insert.c <<-\EOF &&
+	a
+	b
+	X
+	Y
+	c
+	d
+	e
+	EOF
+	git -c diff.cdiff.process="$BACKEND --mode=insert" \
+		diff insert.c >actual 2>stderr &&
+	test_grep "^+X" actual &&
+	test_grep "^+Y" actual &&
+	test_grep "^ c" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with modified file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- newfile.c >actual 2>stderr &&
+	test_grep "return 99" actual &&
+	test_grep "pathname=newfile.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with added file (empty old side)' '
+	cat >added.c <<-\EOF &&
+	int added(void) { return 1; }
+	EOF
+	git add added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "added" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with deleted file (empty new side)' '
+	git add added.c &&
+	git commit -m "commit added.c" &&
+	git rm added.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --cached -- added.c >actual 2>stderr &&
+	test_grep "deleted file" actual &&
+	test_grep "pathname=added.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process skipped for binary files' '
+	printf "\\0binary" >binary.c &&
+	git add binary.c &&
+	git commit -m "add binary" &&
+	printf "\\0changed" >binary.c &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- binary.c >actual &&
+	test_grep "Binary files" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not consulted for unmatched driver' '
+	echo "not tracked by cdiff" >unmatched.txt &&
+	git add unmatched.txt &&
+	git commit -m "add unmatched.txt" &&
+
+	echo "modified" >unmatched.txt &&
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- unmatched.txt >actual &&
+	test_grep "modified" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'multiple drivers use separate processes' '
+	echo "*.h diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 1; }
+	EOF
+	git add multi.h &&
+	git commit -m "add multi.h" &&
+
+	cat >multi.h <<-\EOF &&
+	int header(void) { return 2; }
+	EOF
+
+	test_when_finished "rm -f backend-c.log backend-h.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend-c.log" \
+	    -c diff.hdiff.process="$BACKEND --log=backend-h.log" \
+		diff -- newfile.c multi.h >actual 2>stderr &&
+	test_grep "pathname=newfile.c" backend-c.log &&
+	test_grep "pathname=multi.h" backend-h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works alongside textconv' '
+	write_script uppercase-filter <<-\EOF &&
+	tr "a-z" "A-Z" <"$1"
+	EOF
+
+	cat >textconv.c <<-\EOF &&
+	hello world
+	EOF
+	git add textconv.c &&
+	git commit -m "add textconv.c" &&
+
+	cat >textconv.c <<-\EOF &&
+	goodbye world
+	EOF
+
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff -- textconv.c >actual 2>stderr &&
+	# The diff process receives textconv-transformed (uppercase) content.
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=HELLO WORLD" backend.log &&
+	test_grep "new=GOODBYE WORLD" backend.log &&
+	test_must_be_empty stderr
+'
+
+#
+# Downstream features: word diff, log, equivalent files, exit code.
+#
+
+test_expect_success 'diff process with --word-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --word-diff worddiff.c >actual 2>stderr &&
+	test_grep "\[-1;-\]" actual &&
+	test_grep "{+999;+}" actual &&
+	test_grep "pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process works with git log -p' '
+	# With no-hunks mode, the tool says the files are equivalent,
+	# so log -p should show the commit but no diff content.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -1 -p -- logtest.c >actual 2>stderr &&
+	test_grep "change logtest.c" actual &&
+	test_grep ! "return 2" actual &&
+	test_grep "command=hunks pathname=logtest.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process no hunks suppresses diff output' '
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 0; }
+	EOF
+	git add nohunks.c &&
+	git commit -m "add nohunks.c" &&
+
+	cat >nohunks.c <<-\EOF &&
+	int zero(void) { return 999; }
+	EOF
+
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff nohunks.c >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success 'diff process no hunks with --exit-code returns success' '
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code nohunks.c
+'
+
+test_expect_success 'diff process equivalent commit: --exit-code and --quiet agree' '
+	# A committed blob pair (not a worktree file) whose oids differ but
+	# the tool reports equivalent.  --exit-code and --quiet must agree
+	# with the shown diff (empty) and report success, not fall back to
+	# the byte-level "oids differ" answer.
+	cat >ecq.c <<-\EOF &&
+	alpha
+	EOF
+	git add ecq.c &&
+	git commit -m "ecq v1" &&
+	cat >ecq.c <<-\EOF &&
+	beta
+	EOF
+	git add ecq.c &&
+	git commit -m "ecq v2" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --exit-code HEAD^ HEAD -- ecq.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --quiet HEAD^ HEAD -- ecq.c
+'
+
+test_expect_success 'diff process falls back for trailing-newline-only change' '
+	test_when_finished "rm -f backend.log" &&
+	printf "a\nb\nc\n" >eofnl.c &&
+	git add eofnl.c &&
+	git commit -m "add eofnl.c" &&
+	printf "a\nb\nc" >eofnl.c &&
+	# Same lines, only the final newline removed.  The tool reports
+	# no hunks (it sees identical lines), but that change is not
+	# expressible as hunks, so git falls back to the builtin diff
+	# rather than treating the files as equivalent.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff eofnl.c >actual 2>stderr &&
+	test_grep "No newline at end of file" actual &&
+	test_grep "pathname=eofnl.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process falls back for added file (empty old side)' '
+	test_when_finished "rm -f backend.log" &&
+	printf "x\ny\nz\n" >addnl.c &&
+	git add addnl.c &&
+	# The empty old side has no trailing newline while the new side
+	# does, so the newline fallback shows the addition rather than
+	# letting no-hunks suppress the whole new file.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --cached addnl.c >actual 2>stderr &&
+	test_grep "^+x" actual &&
+	test_grep "pathname=addnl.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process with --exit-code and hunks returns failure' '
+	test_expect_code 1 git -c diff.cdiff.process="$BACKEND" \
+		diff --exit-code newfile.c
+'
+
+#
+# Bypass mechanisms: flags and commands that skip the diff process.
+#
+
+test_expect_success 'diff process bypassed by --diff-algorithm' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --diff-algorithm=patience worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
+	test_when_finished "rm -f backend.log" &&
+	printf "a\nb\nc\n" >wsbypass.c &&
+	git add wsbypass.c &&
+	git commit -m "add wsbypass.c" &&
+	printf "a\n  b  \nc\n" >wsbypass.c &&
+	# The tool is never told about these options and could not honor
+	# them, so git bypasses the process for each (covering the whole
+	# XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES mask, not just -w).
+	for opt in -w -b --ignore-space-at-eol --ignore-blank-lines
+	do
+		rm -f backend.log &&
+		git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+			diff $opt wsbypass.c >actual 2>stderr &&
+		test_path_is_missing backend.log &&
+		test_must_be_empty stderr ||
+		return 1
+	done &&
+	# -w additionally suppresses the whitespace-only change via the
+	# builtin diff that now runs.
+	git -c diff.cdiff.process="$BACKEND" diff -w wsbypass.c >actual &&
+	test_must_be_empty actual
+'
+
+#
+# Error handling and fallback.
+#
+
+test_expect_success 'diff process fallback on tool error status' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Fallback produces the full builtin diff (both change regions).
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Tool was contacted (it replied with error, not crash).
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process error keeps tool available for next file' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=error --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Unlike abort, error keeps the tool available: both files
+	# are sent to the tool (and both fall back).
+	test_grep "pathname=one.c" backend.log &&
+	test_grep "pathname=two.c" backend.log &&
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process abort disables for session' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=abort --log=backend.log" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files should still produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# The tool aborts on the first file and git clears its
+	# capability.  The second file never contacts the tool.
+	test_grep "pathname=one.c" backend.log &&
+	test_grep ! "pathname=two.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process fallback on tool crash' '
+	git -c diff.cdiff.process="$BACKEND --mode=crash" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	# Crash is a communication failure, so a warning is emitted.
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process startup failure only warns once' '
+	git -c diff.cdiff.process="/nonexistent/tool" \
+		diff -- one.c two.c >actual 2>stderr &&
+	# Both files produce diff output via fallback.
+	test_grep "return 10" actual &&
+	test_grep "return 20" actual &&
+	# Sentinel prevents repeated warnings: only one, not one per file.
+	test_grep "diff process.*failed" stderr >warnings &&
+	test_line_count = 1 warnings
+'
+
+
+test_expect_success 'diff process fallback on bad hunks' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-hunk" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "^-OLD9" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "hunk past the end" stderr
+'
+
+test_expect_success 'diff process fallback on mismatched unchanged totals' '
+	cat >synctest.c <<-\EOF &&
+	line1
+	line2
+	line3
+	EOF
+	git add synctest.c &&
+	git commit -m "add synctest.c" &&
+
+	cat >synctest.c <<-\EOF &&
+	line1
+	changed
+	line3
+	EOF
+
+	# bad-sync reports hunk 1 2 1 1: marks 2 old lines and 1 new
+	# line as changed, leaving 1 unchanged old vs 2 unchanged new.
+	# The synchronization invariant fails and git falls back.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-sync" \
+		diff synctest.c >actual 2>stderr &&
+	test_grep "changed" actual &&
+	test_grep "misaligned" stderr
+'
+
+test_expect_success 'diff process fallback on misaligned hunk gap' '
+	# bad-gap reports hunk 1 1 3 1 on boundary.c: one changed line
+	# on each side, so the total unchanged counts match, but the
+	# unchanged run before the change differs (old line 1 vs new
+	# line 3).  A global count check would accept this and emit a
+	# corrupt diff; the per-gap lockstep check rejects it and git
+	# falls back to the builtin algorithm.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-gap" \
+		diff boundary.c >actual 2>stderr &&
+	# The builtin fallback shows both changed regions as additions
+	# (a corrupt-accepted hunk would show NEW5 only as context).
+	test_grep "^+NEW5" actual &&
+	test_grep "^+NEW9" actual &&
+	test_grep "misaligned" stderr
+'
+
+test_expect_success 'diff process fallback on overlapping hunks' '
+	# boundary.c has 10 lines, so both hunks are in bounds
+	# but they overlap at lines 3-4, triggering the ordering check.
+	git -c diff.cdiff.process="$BACKEND --mode=overlap" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "NEW5" actual &&
+	test_grep "overlapping hunks" stderr
+'
+
+test_expect_success 'diff process fallback on malformed hunk line' '
+	git -c diff.cdiff.process="$BACKEND --mode=bad-parse" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual
+'
+
+test_expect_success 'diff process fallback on start 0 with nonzero count' '
+	# bad-start reports hunk 0 1 1 1.  A start of 0 is valid only for
+	# an empty (count 0) range, so the presentation-to-xdiff
+	# translation rejects it and git falls back to the builtin diff
+	# instead of handing xdiff an out-of-range start.
+	git -c diff.cdiff.process="$BACKEND --mode=bad-start" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "^+NEW5" actual &&
+	test_grep "diff process.*failed" stderr
+'
+
+test_expect_success 'diff process caps a flood of hunks and falls back' '
+	# flood emits far more hunks than the file has lines.  Git must
+	# stop accumulating and fall back to the builtin diff rather than
+	# grow memory without bound.
+	git -c diff.cdiff.process="$BACKEND --mode=flood" \
+		diff boundary.c >actual 2>stderr &&
+	test_grep "^-OLD5" actual &&
+	test_grep "too many hunks" stderr
+'
+
+test_expect_success 'diff process skipped when tool omits capability' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-cap --log=backend.log" \
+		diff boundary.c >actual 2>stderr &&
+	# Builtin diff runs: all changes appear, including lines 9-10
+	# that a tool-provided hunk would have narrowed away.
+	test_grep "^-OLD5" actual &&
+	test_grep "^-OLD9" actual &&
+	# The process launched (creating the log) but was
+	# never sent a per-file request, so no hunks command is logged.
+	test_path_is_file backend.log &&
+	test_grep ! "command=hunks" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process receives old-oid and new-oid for a blob pair' '
+	test_when_finished "rm -f backend.log" &&
+	cat >oidpair.c <<-\EOF &&
+	int f(void) { return 1; }
+	EOF
+	git add oidpair.c &&
+	git commit -m "oidpair v1" &&
+	old=$(git rev-parse HEAD:oidpair.c) &&
+
+	cat >oidpair.c <<-\EOF &&
+	int f(void) { return 2; }
+	EOF
+	git add oidpair.c &&
+	git commit -m "oidpair v2" &&
+	new=$(git rev-parse HEAD:oidpair.c) &&
+
+	# Both sides are stored blobs, so their object names are sent.
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff HEAD^ HEAD -- oidpair.c >actual 2>stderr &&
+	test_grep "old-oid=$old new-oid=$new" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process omits old-oid and new-oid for textconv content' '
+	test_when_finished "rm -f backend.log" &&
+	write_script oidcat <<-\EOF &&
+	cat "$1"
+	EOF
+	cat >oidtc.c <<-\EOF &&
+	alpha
+	EOF
+	git add oidtc.c &&
+	git commit -m "oidtc v1" &&
+	cat >oidtc.c <<-\EOF &&
+	beta
+	EOF
+	git add oidtc.c &&
+	git commit -m "oidtc v2" &&
+
+	# textconv rewrites the bytes, so the raw-blob object name that
+	# would otherwise identify each side is omitted.
+	git -c diff.cdiff.textconv="./oidcat" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff HEAD^ HEAD -- oidtc.c >actual 2>stderr &&
+	test_grep "pathname=oidtc.c" backend.log &&
+	test_grep "old-oid=(none) new-oid=(none)" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_done
diff --git a/userdiff.h b/userdiff.h
index 51c26e0d41..a98eabe377 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -3,6 +3,7 @@
 
 #include "notes-cache.h"
 
+struct diff_subprocess;
 struct index_state;
 struct repository;
 
@@ -33,6 +34,8 @@ struct userdiff_driver {
 	int textconv_want_cache;
 	const char *process;
 	char *process_owned;
+	struct diff_subprocess *diff_subprocess;
+	unsigned diff_process_failed : 1;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
gitgitgadget

[PATCH v6 6/9] diff: bypass diff process with --no-ext-diff and in format-patch

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

Make --no-ext-diff disable diff.<driver>.process in addition to
diff.<driver>.command.  Although the two mechanisms work differently
(command replaces Git's output, process feeds hunks back into the
pipeline), both invoke external tools and --no-ext-diff means
"no external tools."

Replace the OPT_BOOL for --ext-diff with an OPT_CALLBACK that
sets both allow_external and no_diff_process, so a single option
controls both.  Passing --ext-diff explicitly clears
no_diff_process, so a later --ext-diff overrides an earlier
--no-ext-diff.

Disable the diff process unconditionally in format-patch so that
generated patches are always based on the builtin diff algorithm
and can be applied reliably by recipients who do not have the
external tool.

Document that --diff-algorithm also bypasses the diff process,
since it forces the builtin algorithm.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/diff-algorithm-option.adoc |  3 +++
 Documentation/diff-options.adoc          |  4 +++-
 Documentation/gitattributes.adoc         |  6 +++---
 builtin/log.c                            |  7 +++++++
 diff.c                                   | 16 ++++++++++++++--
 diff.h                                   |  5 ++++-
 t/t4080-diff-process.sh                  | 16 ++++++++++++++++
 7 files changed, 50 insertions(+), 7 deletions(-)

diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..4d7e2ec35f 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+If you explicitly choose a diff algorithm, it also bypasses
+`diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index 8a63b5e164..18b8b0ed24 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -825,7 +825,9 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff helpers, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
 
 `--textconv`::
 `--no-textconv`::
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index f4ca4a8c7e..a03fb9deb1 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -1073,9 +1073,9 @@ display, which is covered above); and combined diffs (`--cc` and merge
 diffs), whose protocol would have to be extended from a single old/new
 pair to one comparison per merge parent.
 
-`--diff-algorithm` bypasses the process entirely, for every feature
-listed above.  The whitespace-ignoring options (`-w`,
-`--ignore-space-change`, `--ignore-blank-lines`, and the like),
+`--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
+for every feature listed above.  The whitespace-ignoring options
+(`-w`, `--ignore-space-change`, `--ignore-blank-lines`, and the like),
 `-I<regex>`, and `--anchored` also bypass it for the affected files:
 the tool is never told about these options, so it could not honor
 them, and Git falls back to the builtin diff, which does.
diff --git a/builtin/log.c b/builtin/log.c
index e464b30af4..363052f468 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2217,6 +2217,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * Disable diff.<driver>.process so that patches generated by
+	 * format-patch are always based on the builtin diff algorithm
+	 * and can be applied reliably.
+	 */
+	rev.diffopt.flags.no_diff_process = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff.c b/diff.c
index aab012f922..1a487bb353 100644
--- a/diff.c
+++ b/diff.c
@@ -6071,6 +6071,17 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.no_diff_process = unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6401,8 +6412,9 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index 7dc157968d..ee034d240d 100644
--- a/diff.h
+++ b/diff.h
@@ -173,7 +173,10 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
-	/** Disables diff.<driver>.process. */
+	/**
+	 * Disables diff.<driver>.process.  Set by --no-ext-diff and by
+	 * format-patch.
+	 */
 	unsigned no_diff_process;
 
 	/**
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 3b75df082e..7e71b70ab9 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -398,6 +398,22 @@ test_expect_success 'diff process bypassed by --diff-algorithm' '
 	test_path_is_missing backend.log
 '
 
+test_expect_success 'diff process bypassed by --no-ext-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --no-ext-diff worddiff.c >actual &&
+	test_grep "return 999" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not used by format-patch' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		format-patch -1 --stdout -- logtest.c >actual &&
+	test_grep "return 2" actual &&
+	test_path_is_missing backend.log
+'
+
 test_expect_success 'diff process bypassed under whitespace-ignoring flags' '
 	test_when_finished "rm -f backend.log" &&
 	printf "a\nb\nc\n" >wsbypass.c &&
-- 
gitgitgadget

[PATCH v6 7/9] blame: consult diff process for no-hunk detection

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

When a diff process is configured via diff.<driver>.process,
consult it during blame's per-commit diffing.  If the process
returns no hunks for a commit's changes to a file, treat the
commit as having no changes, causing blame to attribute lines
to earlier commits.

Introduce xdi_diff_process(), a process-aware xdi_diff() that
consults the process, runs xdiff on the tool's hunks or on the
builtin algorithm when it does not apply, frees the hunks, and
reports DIFF_PROCESS_EQUIVALENT (without running xdiff) so the caller
can drop or skip the change.  It is the shared consult-then-diff path
for consumers that work on raw hunks: blame's pass_blame_to_parent()
uses it here, and git log -L reuses it later.  builtin_diff() keeps
consulting the process directly, because it tests for equivalence
early, before its funcname-pattern and word-diff setup, so a
reformat-only file short-circuits without that work.

Blame's -w option is not communicated to the process and it could not
honor it, so blame must fall back to the builtin diff there.  Because
blame keeps its whitespace flags in sb->xdl_opts rather than diffopt,
the process bypass keys off xpp (the flags the diff actually runs
with), which covers blame without a guard of its own.

The subprocess is long-running (one startup cost amortized across the
blame traversal), but each commit in the file's history incurs a
round-trip to the tool.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                 |  24 +++++++-
 diff-process.c          |  38 ++++++++++++
 diff-process.h          |  26 +++++++++
 t/t4080-diff-process.sh | 126 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 213 insertions(+), 1 deletion(-)

diff --git a/blame.c b/blame.c
index 977cbb7097..d159de0367 100644
--- a/blame.c
+++ b/blame.c
@@ -19,6 +19,8 @@
 #include "tag.h"
 #include "trace2.h"
 #include "blame.h"
+#include "diff-process.h"
+#include "xdiff-interface.h"
 #include "alloc.h"
 #include "commit-slab.h"
 #include "bloom.h"
@@ -1943,6 +1945,9 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *parent, int ignore_diffs)
 {
 	mmfile_t file_p, file_o;
+	xpparam_t xpp = {0};
+	xdemitconf_t xecfg = {0};
+	xdemitcb_t ecb = {NULL};
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
 
@@ -1961,7 +1966,24 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 			 &sb->num_read_blob, ignore_diffs);
 	sb->num_get_patch++;
 
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	xpp.flags = sb->xdl_opts;
+	xecfg.hunk_func = blame_chunk_cb;
+	ecb.priv = &d;
+	/*
+	 * Consult the diff process, then attribute the resulting chunks
+	 * via blame_chunk_cb.  It bypasses the process for the whitespace-
+	 * ignoring options it cannot honor (they live in xpp.flags, which
+	 * the consultation checks), and when the process reports the blobs
+	 * equivalent it runs no diff, so blame passes this commit and looks
+	 * past it.  Look up the driver by the parent (old) path, as
+	 * builtin_diff() does with name_a, so a renamed file resolves to the
+	 * same driver across diff, blame, and line-log.  Pass no
+	 * old-oid/new-oid: blame diffs each blob pair once, so the tool gains
+	 * nothing from a per-invocation cache key.
+	 */
+	if (xdi_diff_process(&sb->revs->diffopt, parent->path,
+			     &file_p, &file_o, NULL, NULL, &xpp, &xecfg, &ecb)
+	    == DIFF_PROCESS_ERROR)
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
diff --git a/diff-process.c b/diff-process.c
index 4c748fdd2a..191b2b67b2 100644
--- a/diff-process.c
+++ b/diff-process.c
@@ -37,6 +37,7 @@
 #include "sub-process.h"
 #include "pkt-line.h"
 #include "strbuf.h"
+#include "xdiff-interface.h"
 #include "xdiff/xdiff.h"
 
 #define CAP_HUNKS (1u << 0)
@@ -489,3 +490,40 @@ enum diff_process_result diff_process_fill_hunks(
 	}
 	return DIFF_PROCESS_SKIP;
 }
+
+enum diff_process_result xdi_diff_process(
+		struct diff_options *diffopt,
+		const char *path,
+		mmfile_t *file_a,
+		mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp,
+		xdemitconf_t *xecfg,
+		xdemitcb_t *ecb)
+{
+	enum diff_process_result res;
+
+	/*
+	 * Consult the diff process, then run xdiff either constrained to
+	 * the tool's hunks or, when the process does not apply, computing
+	 * the diff itself as a fallback.  EQUIVALENT short-circuits: the
+	 * caller decides what "no change" means for it (drop the commit,
+	 * skip the file, ...), so xdiff is not run.
+	 *
+	 * A SKIP/ERROR from the process just selects the builtin path
+	 * (its warning, if any, was already emitted), so the result then
+	 * reflects whether xdiff itself succeeded, not the process.
+	 */
+	res = diff_process_fill_hunks(diffopt, path, file_a, file_b,
+				      oid_a, oid_b, xpp);
+	if (res == DIFF_PROCESS_EQUIVALENT)
+		return res;
+
+	res = xdi_diff(file_a, file_b, xpp, xecfg, ecb) < 0
+		? DIFF_PROCESS_ERROR : DIFF_PROCESS_OK;
+
+	FREE_AND_NULL(xpp->external_hunks);
+	xpp->external_hunks_nr = 0;
+	return res;
+}
diff --git a/diff-process.h b/diff-process.h
index 8d00dafe1d..5e5b514b77 100644
--- a/diff-process.h
+++ b/diff-process.h
@@ -46,4 +46,30 @@ enum diff_process_result diff_process_fill_hunks(
 		const struct object_id *oid_b,
 		xpparam_t *xpp);
 
+/*
+ * Process-aware xdi_diff(): consult the diff process for 'path', then
+ * run xdiff either constrained to the tool's hunks or computing the
+ * diff itself when the process does not apply or fails.  Frees any
+ * hunks it obtained before returning.
+ *
+ * Returns DIFF_PROCESS_EQUIVALENT (without running xdiff) when the tool
+ * reports the blobs equal, so the caller can drop or skip the change;
+ * DIFF_PROCESS_OK when xdiff ran (on tool hunks or builtin); and
+ * DIFF_PROCESS_ERROR if xdiff itself errored.
+ *
+ * The caller fills xpp (flags, ignore_regex, anchors) and xecfg/ecb as
+ * for a direct xdi_diff() call.  oid_a/oid_b are forwarded to
+ * diff_process_fill_hunks() (see there).
+ */
+enum diff_process_result xdi_diff_process(
+		struct diff_options *diffopt,
+		const char *path,
+		mmfile_t *file_a,
+		mmfile_t *file_b,
+		const struct object_id *oid_a,
+		const struct object_id *oid_b,
+		xpparam_t *xpp,
+		xdemitconf_t *xecfg,
+		xdemitcb_t *ecb);
+
 #endif /* DIFF_PROCESS_H */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 7e71b70ab9..694c94edb2 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -658,4 +658,130 @@ test_expect_success 'diff process omits old-oid and new-oid for textconv content
 	test_must_be_empty stderr
 '
 
+#
+# Blame integration.
+#
+
+test_expect_success 'blame uses tool-provided hunks' '
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "add blame-hunk.c" &&
+	ORIG=$(git rev-parse --short HEAD) &&
+
+	cat >blame-hunk.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add blame-hunk.c &&
+	git commit -m "change blame-hunk.c" &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+
+	# With fixed-hunk mode the tool reports only lines 5-6 as changed,
+	# so blame should attribute lines 9-10 to the original commit
+	# even though the builtin diff would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		blame blame-hunk.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	sed -n "6p" actual >line6 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "$CHANGE" line6
+'
+
+test_expect_success 'blame skips commits with no hunks from diff process' '
+	cat >blame.c <<-\EOF &&
+	int main(void) {
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "add blame.c" &&
+	ORIG_COMMIT=$(git rev-parse --short HEAD) &&
+
+	cat >blame.c <<-\EOF &&
+	int main(void)
+	{
+	return 0;
+	}
+	EOF
+	git add blame.c &&
+	git commit -m "reformat blame.c" &&
+	BLAME_COMMIT=$(git rev-parse --short HEAD) &&
+
+	# Without no-hunks mode, blame attributes the change.
+	git blame blame.c >without &&
+	test_grep "$BLAME_COMMIT" without &&
+
+	# With no-hunks mode, the process considers the files equivalent
+	# and blame skips the reformat commit, attributing to the original.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		blame blame.c >with &&
+	test_grep ! "$BLAME_COMMIT" with &&
+	test_grep "$ORIG_COMMIT" with
+'
+
+test_expect_success 'blame --no-ext-diff bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		blame --no-ext-diff blame.c >actual &&
+	# Without the process, blame attributes the reformat commit normally.
+	test_grep "$BLAME_COMMIT" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame --no-ext-diff uses builtin hunks' '
+	# fixed-hunk mode would narrow blame to lines 5-6, but
+	# --no-ext-diff should bypass it and use the builtin diff.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		blame --no-ext-diff blame-hunk.c >actual &&
+	# Builtin diff attributes lines 9-10 to the change commit.
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'blame -w bypasses diff process' '
+	test_when_finished "rm -f backend.log" &&
+	printf "alpha\nbeta\ngamma\n" >blamew.c &&
+	git add blamew.c &&
+	git commit -m "add blamew.c" &&
+	orig=$(git rev-parse --short HEAD) &&
+	printf "alpha\n   beta   \ngamma\n" >blamew.c &&
+	git commit -am "reindent beta" &&
+	reindent=$(git rev-parse --short HEAD) &&
+	# blame -w must ignore the whitespace-only change and attribute
+	# beta to the original commit, not the reindent commit.  The tool
+	# is never told about -w, so blame must bypass it (not let tool
+	# hunks override -w).
+	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=backend.log" \
+		blame -w blamew.c >actual &&
+	sed -n "2p" actual >line2 &&
+	test_grep "$orig" line2 &&
+	test_grep ! "$reindent" line2 &&
+	test_path_is_missing backend.log
+'
+
 test_done
-- 
gitgitgadget

[PATCH v6 8/9] diff: consult diff process for --stat counts

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

builtin_diff() already consults a configured diff.<driver>.process: a
file the tool reports as equivalent emits no patch, and otherwise the
tool's hunks drive the output.  builtin_diffstat() ran its own xdiff
and ignored the process, so "git diff --stat" still counted a
byte-level change for a file that "git diff" showed as unchanged.

Consult diff_process_fill_hunks() before the stat xdiff, as
builtin_diff() does.  On DIFF_PROCESS_EQUIVALENT, skip the xdiff so
the file keeps its zero inserted and deleted counts and the existing
"nothing changed" pruning drops it, matching the empty patch.
Otherwise the tool's hunks, or the builtin fallback, feed the counts
through the shared xpparam_t.

Under -L, route the surviving hunks through the same line-range filter
builtin_diffstat() already uses for a tracked range, so a
process-provided diff is scoped to that range: "git log -L<range>
--stat" counts the tool's changed lines within the range rather than
the builtin line diff's.

Like the builtin summary path, builtin_diffstat() does not apply
textconv, so the process is consulted on the raw blob content here,
unlike builtin_diff() which sends textconv'd content.  This keeps
"git diff --stat" counting raw lines as it does today; the asymmetry
between patch output and summary counts under textconv predates this
change.  Because the content is the raw blob, the stat path sends the
blob object names to the tool (old-oid/new-oid) for any stored blob,
where the patch path omits the oid under textconv.

Move the summary formats out of the "not yet wired" group of the
"Which features consult the diff process" documentation and into the
list of features that use the tool's hunks, noting the raw,
non-textconv content they receive.  Document that the line-counting
--dirstat=lines follows these counts while the default --dirstat does
not, and that summary formats and blame (only under --textconv) differ
from patch output in whether they textconv the content the tool sees.

Add tests covering counts from the tool's hunks (--numstat,
--shortstat), an equivalent file producing no stat line, --stat
--exit-code, the raw non-textconv content the tool receives, a
multi-file mix of equivalent and changed files, a mode-only change,
and a range-scoped --stat under "git log -L" that reflects the tool's
hunks.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc |  33 ++++--
 diff.c                           |  47 ++++++---
 t/t4080-diff-process.sh          | 176 +++++++++++++++++++++++++++++++
 3 files changed, 234 insertions(+), 22 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index a03fb9deb1..7cdede6b21 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -874,7 +874,10 @@ a flush packet, followed by the old and new file content as packetized
 data, each terminated with a flush packet.  The pathname is relative
 to the repository root.  When `diff.<name>.textconv` is also set,
 the tool receives the textconv-transformed content rather than the
-raw blob.  Git does not send binary files to the diff process.
+raw blob, matching what the consuming feature itself diffs: patch
+output is textconv'd, the summary formats (noted below) are not, and
+`git blame` applies textconv only under `--textconv`.  Git does not
+send binary files to the diff process.
 
 -----------------------
 packet:          git> command=hunks
@@ -960,8 +963,8 @@ still slide or regroup those changes against matching context for
 display, exactly as it compacts its own diffs, so the tool controls
 which lines are reported as changed, not the precise hunk boundaries.
 Patch output features (word diff, function context, color) work
-normally.  Summary formats such as `--stat` still compute their counts
-with the builtin diff for now; see "Which features consult the diff
+normally, as do summary formats like `--stat`.  Not every feature
+consults the process, though; see "Which features consult the diff
 process" below for the full picture and the reasoning behind it.
 
 If no hunk lines precede the flush, followed by "success", Git
@@ -1040,6 +1043,17 @@ of the builtin algorithm:
   hunks without any further negotiation.
 - `git blame`: a commit whose change the tool reports as equivalent is
   skipped, and its lines are attributed to an earlier commit.
+- `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted
+  counts come from the tool's hunks, so a file the tool calls
+  equivalent contributes no stat line, matching the empty patch that
+  `git diff` produces for it.  These summary formats do not apply
+  textconv (just as the builtin summary path does not), so the tool
+  is consulted on the raw blob content even when a `textconv` is also
+  configured for patch output; this mirrors how builtin `--stat`
+  already counts raw lines rather than the textconv'd view.  The
+  line-counting `--dirstat=lines` uses these same counts; the default
+  `--dirstat`, which weighs byte changes, is computed on its own and
+  does not consult the tool.
 
 Features that ask a different question do not consult the process, by
 design:
@@ -1065,13 +1079,12 @@ design:
 - `--raw`, `--name-only`, and `--name-status` compare object ids at
   the tree level and never run a line-level diff at all.
 
-Some features ask "which lines changed" but still use the builtin
-algorithm for now, and may consult the process in a later change: the
-summary formats (`--stat`, `--numstat`, `--shortstat`); `git log -L`'s
-commit selection and parent range propagation (as distinct from its
-display, which is covered above); and combined diffs (`--cc` and merge
-diffs), whose protocol would have to be extended from a single old/new
-pair to one comparison per merge parent.
+Two cases ask "which lines changed" but still use the builtin
+algorithm, and may consult the process in a later change: `git log
+-L`'s commit selection and parent range propagation (as distinct from
+its display, which is covered above), and combined diffs (`--cc` and
+merge diffs), whose protocol would have to be extended from a single
+old/new pair to one comparison per merge parent.
 
 `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
 for every feature listed above.  The whitespace-ignoring options
diff --git a/diff.c b/diff.c
index 1a487bb353..241c07fc63 100644
--- a/diff.c
+++ b/diff.c
@@ -4342,20 +4342,43 @@ static void builtin_diffstat(const char *name_a, const char *name_b,
 		xecfg.ctxlen = o->context;
 		xecfg.interhunkctxlen = o->interhunkcontext;
 		xecfg.flags = XDL_EMIT_NO_HUNK_HDR;
-
-		if (p->line_ranges) {
-			struct line_range_filter lr_filter;
-
-			line_range_filter_init(&lr_filter, p->line_ranges,
-					       diffstat_consume, diffstat);
-
-			if (line_range_filter_diff(&lr_filter, &mf1, &mf2,
-						   &xpp, &xecfg))
+		/*
+		 * Consult the diff process so --stat reflects the
+		 * tool's view of which lines changed rather than the
+		 * builtin line diff.  --stat never applies textconv, so
+		 * the tool is fed the same raw mmfiles the stat itself
+		 * diffs (unlike builtin_diff, which consults the process
+		 * on textconv'd content).
+		 * When the tool reports the files as equivalent we skip
+		 * xdiff entirely, leaving added and deleted at zero so
+		 * the file is pruned below, just as builtin_diff() emits
+		 * no patch for an equivalent file.
+		 *
+		 * Under -L, feed the tool's hunks through the same
+		 * line-range filter the builtin stat uses, so a
+		 * process-provided diff is scoped to the tracked range.
+		 */
+		if (diff_process_fill_hunks(o, name_a, &mf1, &mf2,
+					    one->oid_valid ? &one->oid : NULL,
+					    two->oid_valid ? &two->oid : NULL,
+					    &xpp)
+		    != DIFF_PROCESS_EQUIVALENT) {
+			if (p->line_ranges) {
+				struct line_range_filter lr_filter;
+
+				line_range_filter_init(&lr_filter, p->line_ranges,
+						       diffstat_consume, diffstat);
+
+				if (line_range_filter_diff(&lr_filter, &mf1, &mf2,
+							   &xpp, &xecfg))
+					die("unable to generate diffstat for %s",
+					    one->path);
+			} else if (xdi_diff_outf(&mf1, &mf2, NULL, diffstat_consume,
+						 diffstat, &xpp, &xecfg))
 				die("unable to generate diffstat for %s",
 				    one->path);
-		} else if (xdi_diff_outf(&mf1, &mf2, NULL,
-				  diffstat_consume, diffstat, &xpp, &xecfg))
-			die("unable to generate diffstat for %s", one->path);
+		}
+		free(xpp.external_hunks);
 
 		if (DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two)) {
 			struct diffstat_file *file =
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 694c94edb2..118d0f9464 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -282,6 +282,21 @@ test_expect_success 'diff process works alongside textconv' '
 	test_must_be_empty stderr
 '
 
+test_expect_success 'diff process --stat is fed raw, not textconv, content' '
+	# Reuses textconv.c from the previous test (committed "hello
+	# world", modified to "goodbye world").  Unlike patch output,
+	# --stat does not apply textconv, so the tool sees raw lowercase
+	# content here even with a textconv configured.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.textconv="./uppercase-filter" \
+	    -c diff.cdiff.process="$BACKEND --log=backend.log" \
+		diff --stat -- textconv.c >actual 2>stderr &&
+	test_grep "pathname=textconv.c" backend.log &&
+	test_grep "old=hello world" backend.log &&
+	test_grep "new=goodbye world" backend.log &&
+	test_must_be_empty stderr
+'
+
 #
 # Downstream features: word diff, log, equivalent files, exit code.
 #
@@ -386,6 +401,167 @@ test_expect_success 'diff process with --exit-code and hunks returns failure' '
 		diff --exit-code newfile.c
 '
 
+test_expect_success 'diff process feeds --numstat counts' '
+	# fixed-hunk reports only lines 5-6 as changed, so the stat
+	# counts come from the tool (2/2), not the builtin diff (4/4).
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		diff --numstat boundary.c >actual 2>stderr &&
+	printf "2\t2\tboundary.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks pathname=boundary.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process --numstat sums multi-hunk counts' '
+	# multi-hunk reports both 2-line regions (5-6 and 9-10), so the
+	# counts add up across both hunks: 4 inserted, 4 deleted.  This
+	# exercises the two-region hunk path through builtin_diffstat.
+	git -c diff.cdiff.process="$BACKEND --mode=multi-hunk" \
+		diff --numstat boundary.c >actual &&
+	printf "4\t4\tboundary.c\n" >expect &&
+	test_cmp expect actual
+'
+
+test_expect_success 'diff process equivalent files produce no --stat line' '
+	# A file the tool calls equivalent contributes no stat line,
+	# matching the empty patch that git diff produces for it.
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --stat worddiff.c >actual 2>stderr &&
+	test_must_be_empty actual &&
+	test_grep "command=hunks pathname=worddiff.c" backend.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process feeds --shortstat counts' '
+	# fixed-hunk reports lines 5-6 only, so the summary counts come
+	# from the tool (2 insertions, 2 deletions), not builtin (4/4).
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk" \
+		diff --shortstat boundary.c >actual &&
+	test_grep "2 insertions" actual &&
+	test_grep "2 deletions" actual
+'
+
+test_expect_success 'diff process scopes --stat to the tracked range under log -L' '
+	test_when_finished "rm -f backend.log" &&
+	cat >rangestat.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	OLD5
+	OLD6
+	line7
+	line8
+	OLD9
+	OLD10
+	EOF
+	git add rangestat.c &&
+	git commit -m "add rangestat.c" &&
+
+	cat >rangestat.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	NEW5
+	NEW6
+	line7
+	line8
+	NEW9
+	NEW10
+	EOF
+	git add rangestat.c &&
+	git commit -m "change rangestat.c" &&
+
+	# The file changes at lines 5-6 and 9-10, but fixed-hunk reports
+	# only 5-6.  The builtin line diff counts both regions (4/4); the
+	# tool hunks flow through the same line-range filter the stat uses,
+	# so the range-scoped stat reflects the tool view instead (2/2).
+	git log --no-ext-diff -L1,10:rangestat.c --oneline --stat >builtin &&
+	test_grep "4 insertions(+), 4 deletions(-)" builtin &&
+
+	git -c diff.cdiff.process="$BACKEND --mode=fixed-hunk --log=backend.log" \
+		log -L1,10:rangestat.c --oneline --stat >actual &&
+	test_grep "2 insertions(+), 2 deletions(-)" actual &&
+	test_grep ! "4 insertions" actual &&
+	test_grep "command=hunks pathname=rangestat.c" backend.log
+'
+
+test_expect_success 'diff process equivalent file makes --stat --exit-code succeed' '
+	# The tool reports worddiff.c equivalent, so --exit-code reports
+	# no change (0); the builtin diff would report a change (1).
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks" \
+		diff --stat --exit-code worddiff.c &&
+	test_expect_code 1 git diff --no-ext-diff --stat --exit-code worddiff.c
+'
+
+test_expect_success 'diff process --numstat with mixed equivalent and changed files' '
+	test_when_finished "rm -f c.log h.log" &&
+	# Self-contained fixtures: *.c uses whole-file (changed); *.mh
+	# uses no-hunks (equivalent).
+	echo "*.mh diff=hdiff" >>.gitattributes &&
+	git add .gitattributes &&
+	printf "int a(void) { return 1; }\n" >mixed.c &&
+	printf "int b(void) { return 1; }\n" >mixed.mh &&
+	git add mixed.c mixed.mh &&
+	git commit -m "add mixed fixtures" &&
+	printf "int a(void) { return 2; }\n" >mixed.c &&
+	printf "int b(void) { return 2; }\n" >mixed.mh &&
+	git -c diff.cdiff.process="$BACKEND --mode=whole-file --log=c.log" \
+	    -c diff.hdiff.process="$BACKEND --mode=no-hunks --log=h.log" \
+		diff --numstat mixed.c mixed.mh >actual 2>stderr &&
+	test_grep "mixed.c" actual &&
+	test_grep ! "mixed.mh" actual &&
+	test_grep "pathname=mixed.c" c.log &&
+	test_grep "pathname=mixed.mh" h.log &&
+	test_must_be_empty stderr
+'
+
+test_expect_success POSIXPERM 'diff process keeps mode-only change in --stat' '
+	test_when_finished "rm -f backend.log" &&
+	cat >modeonly.c <<-\EOF &&
+	int m(void) { return 1; }
+	EOF
+	git add modeonly.c &&
+	git commit -m "add modeonly.c" &&
+	cat >modeonly.c <<-\EOF &&
+	int m(void) { return 2; }
+	EOF
+	git add modeonly.c &&
+	test_chmod +x modeonly.c &&
+	git commit -m "edit and chmod modeonly.c" &&
+	# Content and mode both changed, but no-hunks reports the content
+	# equivalent.  The tool is consulted (counts are zero, not the
+	# builtin 1/1), yet the mode change keeps the file from being
+	# pruned.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --stat HEAD^ HEAD >actual 2>stderr &&
+	test_grep "modeonly.c" actual &&
+	test_grep "command=hunks pathname=modeonly.c" backend.log &&
+	test_grep ! "1 insertion" actual &&
+	test_must_be_empty stderr
+'
+
+test_expect_success 'diff process not consulted for default --dirstat' '
+	# The default (change-based) --dirstat algorithm counts via its
+	# own path and never contacts the tool (here --dirstat=0 just
+	# sets a 0% threshold), so the change is still reported even
+	# though no-hunks would call it equivalent.  --dirstat=lines
+	# instead uses the process-aware stat path.
+	test_when_finished "rm -f backend.log" &&
+	mkdir -p dsub &&
+	printf "a\nb\nc\n" >dsub/d.c &&
+	git add dsub/d.c &&
+	git commit -m "add dsub/d.c" &&
+	printf "a\nB\nc\n" >dsub/d.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		diff --dirstat=0 dsub/d.c >actual &&
+	test_grep "dsub" actual &&
+	test_path_is_missing backend.log
+'
+
 #
 # Bypass mechanisms: flags and commands that skip the diff process.
 #
-- 
gitgitgadget

[PATCH v6 9/9] line-log: consult diff process for range tracking

[Michael Montalbo via GitGitGadget]

From: Michael Montalbo <mmontalbo@gmail.com>

git log -L tracks line ranges by diffing each commit against its
parent in collect_diff().  This pass used the builtin diff while the
displayed diff (builtin_diff()) consults a configured
diff.<driver>.process, so the two could disagree: a reformat-only
commit selected by builtin tracking was then rendered with an empty
diff because the tool reported the files equivalent.

Consult the process in collect_diff() too, mirroring the blame
integration.  When the tool reports the files equivalent, collect no
ranges; the tracked range then maps across unchanged and the commit
drops out of the log, matching what is displayed.  Like the summary
formats, the tracking pass diffs raw content, so the tool is consulted
on the raw blobs here.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc | 20 ++++++++++---------
 line-log.c                       | 33 ++++++++++++++++++++++++++++----
 t/t4080-diff-process.sh          | 33 ++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 13 deletions(-)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index 7cdede6b21..8021dc8e39 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -1037,12 +1037,16 @@ Features that ask "which lines changed" use the tool's hunks in place
 of the builtin algorithm:
 
 - `git diff` patch output, together with everything layered on it:
-  word diff, function context (`-W`), `--color-moved`, the `@@` hunk
-  headers, and the `-L` line-range display.  These operate on the
-  lines the patch step already emitted, so they reflect the tool's
-  hunks without any further negotiation.
+  word diff, function context (`-W`), `--color-moved`, and the `@@`
+  hunk headers.  These operate on the lines the patch step already
+  emitted, so they reflect the tool's hunks without any further
+  negotiation.
 - `git blame`: a commit whose change the tool reports as equivalent is
   skipped, and its lines are attributed to an earlier commit.
+- `git log -L`: both the line-range display and the underlying range
+  tracking consult the tool, so a commit it reports as equivalent is
+  dropped from the log (its tracked range maps across unchanged)
+  rather than selected and then shown with an empty diff.
 - `--stat`, `--numstat`, and `--shortstat`: the inserted and deleted
   counts come from the tool's hunks, so a file the tool calls
   equivalent contributes no stat line, matching the empty patch that
@@ -1079,11 +1083,9 @@ design:
 - `--raw`, `--name-only`, and `--name-status` compare object ids at
   the tree level and never run a line-level diff at all.
 
-Two cases ask "which lines changed" but still use the builtin
-algorithm, and may consult the process in a later change: `git log
--L`'s commit selection and parent range propagation (as distinct from
-its display, which is covered above), and combined diffs (`--cc` and
-merge diffs), whose protocol would have to be extended from a single
+Combined diffs (`--cc` and merge diffs) ask "which lines changed" but
+still use the builtin algorithm, and may consult the process in a
+later change; their protocol would have to be extended from a single
 old/new pair to one comparison per merge parent.
 
 `--no-ext-diff` and `--diff-algorithm` bypass the process entirely,
diff --git a/line-log.c b/line-log.c
index 5fc75ae275..97b3e0a31d 100644
--- a/line-log.c
+++ b/line-log.c
@@ -7,11 +7,11 @@
 #include "tag.h"
 #include "tree.h"
 #include "diff.h"
+#include "diff-process.h"
 #include "commit.h"
 #include "decorate.h"
 #include "repository.h"
 #include "revision.h"
-#include "xdiff-interface.h"
 #include "strbuf.h"
 #include "line-log.h"
 #include "setup.h"
@@ -330,12 +330,15 @@ static int collect_diff_cb(long start_a, long count_a,
 	return 0;
 }
 
-static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges *out)
+static int collect_diff(struct diff_options *diffopt, const char *path,
+			mmfile_t *parent, mmfile_t *target,
+			struct diff_ranges *out)
 {
 	struct collect_diff_cbdata cbdata = {NULL};
 	xpparam_t xpp;
 	xdemitconf_t xecfg;
 	xdemitcb_t ecb;
+	int ret = 0;
 
 	memset(&xpp, 0, sizeof(xpp));
 	memset(&xecfg, 0, sizeof(xecfg));
@@ -345,7 +348,23 @@ static int collect_diff(mmfile_t *parent, mmfile_t *target, struct diff_ranges *
 	xecfg.hunk_func = collect_diff_cb;
 	memset(&ecb, 0, sizeof(ecb));
 	ecb.priv = &cbdata;
-	return xdi_diff(parent, target, &xpp, &xecfg, &ecb);
+
+	/*
+	 * Consult the diff process so range tracking agrees with the
+	 * diff that will be shown.  When the tool reports the files as
+	 * equivalent we collect no ranges, so the tracked range maps
+	 * across unchanged and the commit drops out of the log, rather
+	 * than being selected here but rendered with an empty diff by
+	 * the process-aware builtin_diff().  Blob oids are not threaded to
+	 * this path yet, so pass NULL and send no old-oid/new-oid (a later
+	 * change can supply the pair, where they would let the tool cache
+	 * across the range-tracking and display passes over the same
+	 * commit).
+	 */
+	if (xdi_diff_process(diffopt, path, parent, target,
+			     NULL, NULL, &xpp, &xecfg, &ecb) == DIFF_PROCESS_ERROR)
+		ret = -1;
+	return ret;
 }
 
 /*
@@ -927,7 +946,13 @@ static int process_diff_filepair(struct rev_info *rev,
 	}
 
 	diff_ranges_init(&diff);
-	if (collect_diff(&file_parent, &file_target, &diff))
+	/*
+	 * Select the driver by the old (parent) path, as builtin_diff() does
+	 * with name_a, so a renamed file resolves to the same driver for
+	 * range tracking as for the diff that is shown.
+	 */
+	if (collect_diff(&rev->diffopt, pair->one->path,
+			 &file_parent, &file_target, &diff))
 		die("unable to generate diff for %s", pair->one->path);
 
 	/* NEEDSWORK should apply some heuristics to prevent mismatches */
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
index 118d0f9464..9584a458b1 100755
--- a/t/t4080-diff-process.sh
+++ b/t/t4080-diff-process.sh
@@ -960,4 +960,37 @@ test_expect_success 'blame -w bypasses diff process' '
 	test_path_is_missing backend.log
 '
 
+#
+# Line-log (git log -L) range tracking.
+#
+
+test_expect_success 'diff process drops equivalent commit from log -L' '
+	test_when_finished "rm -f backend.log" &&
+	cat >linelog.c <<-\EOF &&
+	int tracked(void) { return 1; }
+	EOF
+	git add linelog.c &&
+	git commit -m "add linelog.c" &&
+
+	cat >linelog.c <<-\EOF &&
+	int tracked(void) { return 2; }
+	EOF
+	git commit -am "change tracked line" &&
+
+	# Builtin line tracking selects the change commit.
+	git log --no-ext-diff -L1,1:linelog.c --format="%s" >builtin &&
+	test_grep "change tracked line" builtin &&
+
+	# With the tool reporting the change as equivalent, tracking
+	# drops the commit (the range maps across unchanged) instead of
+	# selecting it and rendering an empty diff.
+	git -c diff.cdiff.process="$BACKEND --mode=no-hunks --log=backend.log" \
+		log -L1,1:linelog.c --format="%s" >actual &&
+	test_grep ! "change tracked line" actual &&
+	# The creating commit still appears, so the change commit was
+	# selectively dropped rather than the whole log going empty.
+	test_grep "add linelog.c" actual &&
+	test_grep "command=hunks pathname=linelog.c" backend.log
+'
+
 test_done
-- 
gitgitgadget

[RFC PATCH v7 0/10] diff: add provider interface and initial providers

[Michael Montalbo]

Every in-process diff in Git reduces, at one point, to a single
question: given two blobs and the settings the diff runs under, which
line ranges changed?  The answer is the diff's hunks: for each change,
the position and length of the range on the old side and on the new.
Each consumer asks in its own shape:

- blame diffs each suspect's blob against its parent's, taking only the
  coordinates through xdiff's hunk callback;
- the stat formats keep only the added and deleted counts;
- patch output emits from the hunks, with xdiff interleaving context and
  content around them;
- log -L maps the tracked range across each commit from the coordinates.

In every case the answer is computed the same way: load both blobs and
run xdiff.  That is the only source, so nothing that already holds the
answer, or that would answer differently on purpose, can supply it
instead.  Sometimes that is what we want, which is why patch-id and
format-patch stay on the builtin computation throughout: patch-id needs
identical hashes on every machine, and a format-patch must apply for
recipients who share none of the sender's configuration.  Other times
another source would be useful.

This RFC sketches a direction.  The unified series shows one interface
carrying two example providers and their interaction; it is not shaped
to merge as one topic.  If the direction holds, the work returns as
separate reviewable series (see Roadmap).  The two examples are
demonstrations, each an RFC on its own: diff.<driver>.process, the RFC
cooking as mm/diff-process-hunks, lets a configured external process
answer with its own notion of which lines changed, and the diff-hunks
store, new in this thread, remembers what xdiff computed and serves it
back.  One is authoritative and external, one a cache and in-process.

Three pieces:

- A hunk provider interface (diff-provider.h) is the point of the
  series.  A provider is an alternate source for the answer: asked with
  the pair's object ids and the diff settings, before any blob is
  loaded, it may supply the hunks in place of the builtin computation.
  A miss falls through to that computation, and every answer passes one
  shared validity check first.  The providers form a chain the
  repository owns, built on first consultation and released in
  repo_clear(), so provider state such as a running process never
  outlives its repository.  Chain order is the authority, and the
  terminal provider is the builtin computation itself, so the interface
  never exists without an implementor: patch 02 ships it answering every
  request the way the consumers did before.  A consumer states its
  request in one struct and reads one set of outcomes (answered,
  unanswered, or failed); it never names a provider, and a provider
  added later maps onto those outcomes inside the interface, so consumer
  code is written once.  Because every diff now walks the chain even
  with no store or process configured, the default path was measured
  against the pre-series base and runs within noise (a 5000-commit
  log --stat and a long-history blame, ratio 1.00 either way).

- The diff-hunks store shows the non-authoritative side: an in-process
  cache at $GIT_DIR/objects/info/diff-hunks that may only reproduce the
  builtin diff, so serving from it never changes a command's output.  It
  is read by default and written only when a repository owner opts in,
  warming it as a side effect of diff work the command already does:

      GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null

  A warmed store then serves the stat formats and blame from stored
  coordinates instead of a fresh diff: on git.git a 5000-commit log
  --stat runs about 1.9x faster, and blame reads the same entries
  opportunistically (full numbers in [1]).  Its format and keying, what
  it may not serve, and how it handles corruption and staleness are in
  git-diff-hunks(1), gitformat-diff-hunks(5), and [2].  The interface
  point is small: a cache drops in as the provider that stands aside
  wherever an authoritative one answers.

- diff.<driver>.process shows the authoritative side: an external
  process, configured per driver, whose answers may deliberately differ
  from the builtin diff and outrank the store.  Git asks it for a pair by
  object names alone, so it answers before any blob is read, which suits
  a cache or a process that fetches the blobs itself.  Consulting is
  opt-in per command, following the allow_textconv precedent, and a pair
  the process cannot answer falls back to the builtin diff.  The
  protocol, the per-command gate, how failures are handled, and the
  versioning that lets it grow are in gitattributes(5) and footnotes [3]
  and [4].  The interface point, again, is small: an external,
  authoritative provider joins the same chain ahead of the cache, and
  neither consumer learns it is there.  A later content-carrying request
  would extend it to the pairs and consumers this identity-only form
  leaves on the builtin diff.

The series stops at the coordinates.  A consumer that needs the changed
text, such as patch output, would have only its hunk selection replaced,
with xdiff still emitting content from the blobs; that machinery is the
content enrichment sketched in the Roadmap.  Establishing the framework
on coordinates first keeps this series one design: the question, the
interface, and two providers answering by identity.

Shape of the series:

  01     documentation: how external diff drivers relate to the
         features layered on the diff
  02     the provider interface: the request and outcome types, the
         emit entry point, the shared validity check, and the
         repository-owned chain with its terminal builtin provider
  03     the store: on-disk format, library, and the diff-hunks command
  04     recording: the stat walk computes, sums, and records
         trim-stable pairs (writes gated off by default)
  05     reading: the consult entry point and the store's registration
         as a provider; the request gains the object ids and diff
         options
  06     blame reading through the interface's emit path
  07-09  process preparation: sub-process lifecycle split, a gentle
         status read for an optional process, and the
         diff.<driver>.process config
  10     the process provider, oid-only, at the head of the chain, with
         the per-command gate; the request gains the path

Roadmap:

This RFC asks whether the direction is right, not for these ten patches
to merge as one topic.  If it holds, the work returns in reviewable
pieces:

- the interface and the store (patches 01 through 06): a cache with
  measured numbers and no external-process machinery

- the process provider (patches 07 through 10) on the same interface

- the content enrichment (the content-carrying request, patch output and
  log -L consulting, and the xdiff machinery that feeds a provider's
  hunks into emission) once the identity-keyed framework settles.

Several design questions are left for those series.

mm/diff-process-hunks in seen would be dropped in favor of this thread
and its split.

The series applies on the line-log topic (mm/line-log-limited-ops)
rebased onto current master.  The topic rewrites the same
builtin_diffstat() region this series touches, and current master
includes 061a68e443 (sub-process: use gentle handshake to avoid die()
on startup failure), which this topic leans on: a process that dies
during the handshake degrades to the builtin diff like every other
failure.  A trial merge against seen shows no interaction with other
topics beyond the mm/diff-process-hunks replacement above.

The base (line-log topic on current master) and the full series are
available at:

  git fetch https://github.com/mmontalbo/git mm/line-log-stat-formats-followup
  git fetch https://github.com/mmontalbo/git mm/hunk-providers-oid-first

Changes since v6:

This is a restructuring, not an incremental reroll, so a range-diff
against v6 is unreadable; the map of what changed:

- The series now leads with the hunk provider interface and brings the
  diff-hunks store in as its in-process implementation (patches 02
  through 06, new to this thread).  It keeps only the identity-keyed
  half of the external diff process protocol from mm/diff-process-hunks.
- v6's gitattributes documentation, sub-process split, and userdiff
  config return close to their v6 form as patches 01, 07, and 09.  Patch
  08 is new: a gentle status read so a protocol error in an optional
  process degrades to the builtin diff instead of dying.
- v6's protocol patch returns as patch 10, reduced to the oid-only
  request, consulting through the interface, and carrying a per-command
  gate (v6's bypass patch folds into it).
- v6's blame and stat consults return as identity-keyed consults
  (patches 05, 06, and 10); their content legs, along with v6's xdiff
  external-hunks machinery, content-carrying request, and line-log
  consult, are withheld for the content enrichment.

Footnotes:

[1] Store numbers, measured with hyperfine against the same build with
    core.diffHunks=false.  The warm is a full cold build of the store;
    the blame speedup is file-dependent (see the coverage limitation):

    git.git (82,912 commits, --all)
      warm log --all --stat      20.9 s        store 28 MB, verify 39 ms
      log --stat -5000           1.91x    (1.38 s -> 0.72 s)
      blame diff.c               1.26x     (509 ms -> 403 ms)
      blame hit rate             54% (896 of 1653 pairs)

    linux (1,445,548 commits, --all)
      warm log --all --stat      714 s       store 298 MB, verify 415 ms
      log --stat -5000           1.43x    (2.29 s -> 1.60 s)
      blame kernel/sched/core.c  1.43x     (1.59 s -> 1.11 s)
      blame hit rate             74% (2414 of 3263 pairs)

[2] The store is its own file because nothing existing is addressed by a
    blob pair: notes attach to single objects, commit-graph chunks to
    commits.  One entry per pair, keyed by (old blob, new blob,
    xdl_opts) and recorded only when the pair's trimmed and untrimmed
    diffs agree, serves blame at zero context and the stat formats at
    any -U (divergent pairs are 0.4-0.5% of a warm and always compute).
    The writer fsyncs and commits atomically, and a reader bounds-checks
    every record and treats an unparsable file as absent; the trailing
    checksum is checked by git diff-hunks verify, not on every read, the
    same read-time trust the commit-graph and multi-pack-index take.
    There is deliberately no fsck integration, expiry, or background
    maintenance: the store is derivable at any time, so the recovery
    path is git diff-hunks clear and a re-warm.  New commits make it
    incomplete, not wrong; a later warm seeds from the file and pays
    only for what is new.

[3] Consulting the process is allowed per command, like textconv: git
    diff, git log and git show, and git blame consult it; the plumbing
    diff commands do not unless --ext-diff or --diff-process is given,
    and the interactive-patch machinery, format-patch, and range-diff
    stay builtin.  Options the process is never told about select no
    process, and an object id is sent only when it names the exact bytes
    diffed (a pair under an active object replacement is not sent).  The
    command comes from local configuration, as with filter.<name>
    .process: attributes select only a driver name, so cloning cannot
    cause a process to run.  gitattributes(5) has the full gate.

[4] The protocol is versioned and capability-negotiated, and extends
    without breaking deployed processes: a process ignores request keys
    it does not know, Git ignores trailing tokens on a hunk line so
    fields can be appended, and new request forms arrive as capabilities
    a process may decline.  Announcing a capability Git did not request
    aborts the command, the filter protocol's handshake rule.  The
    content-carrying request is the natural first extension; markers for
    formatting-only changes and function or token boundaries are
    candidates beyond it.

Michael Montalbo (10):
  gitattributes: document how external diff drivers relate to diff
    features
  diff: introduce a hunk provider interface
  diff-hunks: add the store format, library, and command
  diff: record precomputed hunks during stat output
  diff: read precomputed hunks for stat output
  blame: read precomputed hunks
  sub-process: separate process lifecycle from hashmap management
  sub-process: add a gentle status read
  userdiff: add diff.<driver>.process config
  diff: consult oid-only hunk providers via diff.<driver>.process

 .gitignore                               |    1 +
 Documentation/Makefile                   |    1 +
 Documentation/config.adoc                |    2 +
 Documentation/config/core.adoc           |   10 +-
 Documentation/config/diff-hunks.adoc     |    8 +
 Documentation/config/diff.adoc           |    6 +
 Documentation/diff-algorithm-option.adoc |    3 +
 Documentation/diff-options.adoc          |   15 +-
 Documentation/git-diff-hunks.adoc        |  146 +++
 Documentation/gitattributes.adoc         |  171 ++++
 Documentation/gitformat-diff-hunks.adoc  |  129 +++
 Documentation/meson.build                |    2 +
 Makefile                                 |    5 +
 blame.c                                  |   81 +-
 builtin.h                                |    1 +
 builtin/blame.c                          |    9 +-
 builtin/diff-hunks.c                     |   53 ++
 builtin/diff-tree.c                      |    3 +
 builtin/diff.c                           |   11 +
 builtin/log.c                            |   19 +
 chunk-format.c                           |   62 +-
 chunk-format.h                           |   14 +
 command-list.txt                         |    2 +
 diff-hunks.c                             | 1034 ++++++++++++++++++++++
 diff-hunks.h                             |  141 +++
 diff-process.c                           |  669 ++++++++++++++
 diff-provider-internal.h                 |  130 +++
 diff-provider.c                          |  190 ++++
 diff-provider.h                          |  159 ++++
 diff.c                                   |  294 +++++-
 diff.h                                   |   47 +
 environment.c                            |    1 +
 git.c                                    |    1 +
 meson.build                              |    4 +
 odb.c                                    |    2 +
 odb.h                                    |    4 +
 range-diff.c                             |    6 +
 repo-settings.c                          |    1 +
 repo-settings.h                          |    1 +
 repository.c                             |    3 +
 repository.h                             |    8 +
 sub-process.c                            |   52 +-
 sub-process.h                            |   19 +-
 t/helper/meson.build                     |    1 +
 t/helper/test-diff-process-backend.c     |  349 ++++++++
 t/helper/test-tool.c                     |    1 +
 t/helper/test-tool.h                     |    1 +
 t/meson.build                            |    3 +
 t/perf/p4218-diff-hunks.sh               |   48 +
 t/t4080-diff-process.sh                  |  593 +++++++++++++
 t/t4220-diff-hunks.sh                    |  819 +++++++++++++++++
 t/t4220/README                           |   55 ++
 t/t4220/trim-divergent-new               |  319 +++++++
 t/t4220/trim-divergent-old               |  316 +++++++
 userdiff.c                               |    7 +
 userdiff.h                               |    2 +
 write-or-die.h                           |    7 +-
 xdiff-interface.h                        |   12 +
 58 files changed, 5989 insertions(+), 64 deletions(-)
 create mode 100644 Documentation/config/diff-hunks.adoc
 create mode 100644 Documentation/git-diff-hunks.adoc
 create mode 100644 Documentation/gitformat-diff-hunks.adoc
 create mode 100644 builtin/diff-hunks.c
 create mode 100644 diff-hunks.c
 create mode 100644 diff-hunks.h
 create mode 100644 diff-process.c
 create mode 100644 diff-provider-internal.h
 create mode 100644 diff-provider.c
 create mode 100644 diff-provider.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/perf/p4218-diff-hunks.sh
 create mode 100755 t/t4080-diff-process.sh
 create mode 100755 t/t4220-diff-hunks.sh
 create mode 100644 t/t4220/README
 create mode 100644 t/t4220/trim-divergent-new
 create mode 100644 t/t4220/trim-divergent-old


base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
prerequisite-patch-id: 6270dea79c9f06530737cefa3e1a0a39a1be7877
prerequisite-patch-id: 46fcc16a7a2ed760a1134d2a92c87699f3ec7bdb
prerequisite-patch-id: c1e3da243003d060e429bc2196ae02b3453f01f9
prerequisite-patch-id: 4ad4e273494d4e8503706c21bfdc90a5d7ce116a
prerequisite-patch-id: f7fa1367756daafa83f4f030a5c7b6dc3dbb70d7
prerequisite-patch-id: 829e76c9fec655a07f9383086a35bff3290b1c74
prerequisite-patch-id: 5c5a0d61ae9b6d628d05f1eb5df046758f3111a8
-- 
2.54.0

[RFC PATCH v7 01/10] gitattributes: document how external diff drivers relate to diff features

[Michael Montalbo]

The "Defining an external diff driver" section explains how to
configure diff.<driver>.command but not how the driver relates to the
rest of Git's diff machinery.  In particular, the command only
replaces the textual patch: word diff, function context, color, and
the like cannot apply to its output, while the summary formats, blame,
and git log -L do not run it at all and keep using the builtin diff.

Spell this out so the scope of an external diff driver is clear.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/gitattributes.adoc | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index bd76167a45..da773e2924 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -784,6 +784,17 @@ with the above configuration, i.e. `j-c-diff`, with 7
 parameters, just like `GIT_EXTERNAL_DIFF` program is called.
 See linkgit:git[1] for details.
 
+An external diff driver replaces the patch Git would otherwise
+produce for the path: Git runs the command and shows its output in
+place of its own.  Output features that post-process Git's diff do
+not apply to the driver's output; word diff, function context (`-W`),
+`--color-moved`, and coloring all act on Git's builtin diff, not the
+driver's output.
+The driver is consulted only when Git generates a textual patch.  The
+summary formats (`--stat`, `--numstat`, `--shortstat`, and
+`--dirstat`), `git blame`, and `git log -L` do not run it and
+continue to use Git's builtin diff.
+
 If the program is able to ignore certain changes (similar to
 `git diff --ignore-space-change`), then also set the option
 `trustExitCode` to true.  It is then expected to return exit code 1 if
-- 
2.54.0

[RFC PATCH v7 02/10] diff: introduce a hunk provider interface

[Michael Montalbo]

To learn which line ranges changed between two blobs, every consumer in
the diff machinery loads both blobs and runs xdiff.  There is no other
way to supply that answer, even when it is known elsewhere: a cache may
hold the ranges from the last time the pair was diffed, and a
format-aware process may have its own idea of which lines changed.
Either could answer from the blob object ids alone, but the loading and
computing are hard-wired into each consumer, so such an answer has no
place to enter.

Introduce the hunk provider interface, diff-provider.h, between asking
the question and computing the answer.  A provider answers a request
made of the pair's identity, its blob object ids and the parameters
that determine the diff.  A provider is either authoritative, so its
answer may deliberately differ from the builtin diff, or not, so its
answer must reproduce the builtin result exactly.  Every answer served
from identity passes diff_provider_check_hunk() before a consumer sees
it: coordinates fit int32, hunks are ordered and non-overlapping, and
the unchanged runs between them match on both sides.  A failing answer
is discarded and the pair falls through as unanswered.

Providers are repository-lifecycle objects.  Each repository owns a
chain of them, built on first consultation and released from
repo_clear(), so a submodule gets its own providers and no provider
state outlives the repository it serves.  The chain has a fixed
composition, and each provider gates itself per request, passing when
it does not apply.  Chain order is the authority: the first answer
wins.  A provider may instead refuse a pair whose request is shaped by
parameters its recording key cannot express.  After a refusal, no later
provider answers the pair from identity, and the consumer must not
record what it computes for it.  The last provider is the builtin
computation, the only one that computes rather than answering from
identity, so a walk given a fill callback always ends in an answer,
refusal or not.

The walk in diff-provider.c maps a provider's four dispositions
(answer, pass, fail, refuse) onto the consumer-facing outcomes, and
checks with BUG() that only the computing provider fails and that it
passes on a walk with no fill callback.  The implementor contract, the
provider struct, its dispositions, and the shared check, lives in
diff-provider-internal.h, as refs/refs-internal.h is to refs.h;
consumers see only diff-provider.h.

The consumer surface is two types.  struct diff_provider_request names
what is diffed and under which parameters; each later commit that
consults on more state adds the field it keys on (the object ids and
diff options, then the path).  enum diff_provider_outcome flattens two
dependent axes into four points: the response state (answered,
unanswered, failed) and, only when unanswered, whether the caller may
record what it computes.  The record rule rides in the outcome, not a
separate flag, so -Wswitch forces every consumer to place the no-record
arm.  A provider added later maps onto these values inside the walk, so
consumer code is written once.

diff_provider_emit_hunks() is the consumer entry: the caller states the
request, a hunk callback, and a content-loading callback that reaches
the terminal provider only when the ranges are computed.  Blame's
pass_blame_to_parent() is the first consumer, since it knows both blob
ids before reading either blob; its loads move into the fill callback.
With only the terminal provider registered, every request still
computes, so behavior is unchanged.  (Blame's -C/-M split detection
diffs partial buffers with no blob identity and stays on xdi_diff().)

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Makefile                 |   1 +
 blame.c                  |  36 +++++++--
 diff-provider-internal.h | 122 +++++++++++++++++++++++++++++++
 diff-provider.c          | 154 +++++++++++++++++++++++++++++++++++++++
 diff-provider.h          | 130 +++++++++++++++++++++++++++++++++
 meson.build              |   1 +
 repository.c             |   3 +
 repository.h             |   8 ++
 8 files changed, 447 insertions(+), 8 deletions(-)
 create mode 100644 diff-provider-internal.h
 create mode 100644 diff-provider.c
 create mode 100644 diff-provider.h

diff --git a/Makefile b/Makefile
index 98e995e4be..50c96807d6 100644
--- a/Makefile
+++ b/Makefile
@@ -1151,6 +1151,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-provider.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
 LIB_OBJS += diffcore-delta.o
diff --git a/blame.c b/blame.c
index 126e232416..c3ef9c17f7 100644
--- a/blame.c
+++ b/blame.c
@@ -23,6 +23,7 @@
 #include "commit-slab.h"
 #include "bloom.h"
 #include "commit-graph.h"
+#include "diff-provider.h"
 
 define_commit_slab(blame_suspects, struct blame_origin *);
 static struct blame_suspects blame_suspects;
@@ -1936,6 +1937,28 @@ static int blame_chunk_cb(long start_a, long count_a,
 	return 0;
 }
 
+struct blame_diff_fill_data {
+	struct blame_scoreboard *sb;
+	struct blame_origin *parent, *target;
+	int ignore_diffs;
+};
+
+/*
+ * Content load for diff_provider_emit_hunks(): runs when the diff is
+ * computed.
+ */
+static int blame_diff_fill(void *data, mmfile_t *old_file, mmfile_t *new_file)
+{
+	struct blame_diff_fill_data *f = data;
+
+	fill_origin_blob(&f->sb->revs->diffopt, f->parent, old_file,
+			 &f->sb->num_read_blob, f->ignore_diffs);
+	fill_origin_blob(&f->sb->revs->diffopt, f->target, new_file,
+			 &f->sb->num_read_blob, f->ignore_diffs);
+	f->sb->num_get_patch++;
+	return 0;
+}
+
 /*
  * We are looking at the origin 'target' and aiming to pass blame
  * for the lines it is suspected to its parent.  Run diff to find
@@ -1945,9 +1968,11 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 				 struct blame_origin *target,
 				 struct blame_origin *parent, int ignore_diffs)
 {
-	mmfile_t file_p, file_o;
 	struct blame_chunk_cb_data d;
 	struct blame_entry *newdest = NULL;
+	struct blame_diff_fill_data fill_data = { sb, parent, target, ignore_diffs };
+	xpparam_t xpp = { .flags = sb->xdl_opts };
+	struct diff_provider_request req = { .repo = sb->repo, .xpp = &xpp };
 
 	if (!target->suspects)
 		return; /* nothing remains for this target */
@@ -1958,13 +1983,8 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 	d.ignore_diffs = ignore_diffs;
 	d.dstq = &newdest; d.srcq = &target->suspects;
 
-	fill_origin_blob(&sb->revs->diffopt, parent, &file_p,
-			 &sb->num_read_blob, ignore_diffs);
-	fill_origin_blob(&sb->revs->diffopt, target, &file_o,
-			 &sb->num_read_blob, ignore_diffs);
-	sb->num_get_patch++;
-
-	if (diff_hunks(&file_p, &file_o, blame_chunk_cb, &d, sb->xdl_opts))
+	if (diff_provider_emit_hunks(&req, blame_diff_fill, &fill_data,
+				     blame_chunk_cb, &d) == DIFF_PROVIDER_ERROR)
 		die("unable to generate diff (%s -> %s)",
 		    oid_to_hex(&parent->commit->object.oid),
 		    oid_to_hex(&target->commit->object.oid));
diff --git a/diff-provider-internal.h b/diff-provider-internal.h
new file mode 100644
index 0000000000..8dd8e4b0dc
--- /dev/null
+++ b/diff-provider-internal.h
@@ -0,0 +1,122 @@
+#ifndef DIFF_PROVIDER_INTERNAL_H
+#define DIFF_PROVIDER_INTERNAL_H
+
+#include "diff-provider.h"
+
+/*
+ * The implementor-facing half of the hunk provider interface: the
+ * provider chain a repository owns, and the rules a provider applies
+ * to its own answer before any consumer sees it.  Provider
+ * implementations include this header; consumers of the interface
+ * use only diff-provider.h.
+ */
+
+/*
+ * A provider's verdict on one request.  Only the chain walk
+ * (diff-provider.c) sees these; it maps the dispositions of a whole
+ * walk onto the public outcome set.
+ */
+enum diff_provider_disposition {
+	/*
+	 * The provider failed to produce the answer it owns.  Only
+	 * the computing provider returns this: its compute leg is
+	 * the one part of a consultation that can fail, and the walk
+	 * ends with the public error outcome.
+	 */
+	DIFF_PROVIDER_DISP_ERROR = -1,
+
+	/*
+	 * Answered: every hunk of the pair has been emitted through
+	 * the consumer's callback.
+	 */
+	DIFF_PROVIDER_DISP_ANSWERED = 0,
+
+	/* Not this provider's request: the walk consults the next one. */
+	DIFF_PROVIDER_DISP_PASS,
+
+	/*
+	 * The pair must not be answered from identity nor recorded:
+	 * the request is shaped by parameters the provider's
+	 * recording key cannot express, so a recorded answer would
+	 * not match this request, and this request's result must not
+	 * be recorded under that key.  The walk goes on, but consults
+	 * only the computing provider, and its fall-through outcome
+	 * tells the consumer not to record.
+	 */
+	DIFF_PROVIDER_DISP_STOP_NO_RECORD,
+};
+
+/*
+ * One provider in a repository's chain (repository.h).  The chain is
+ * assembled in diff-provider.c with a fixed composition; whether a
+ * provider applies to a request is decided by nobody but the
+ * provider, whose consult gates itself and passes.  Chain position
+ * carries the authority resolution: an earlier provider's answer or
+ * refusal outranks every provider after it.
+ */
+struct diff_provider {
+	/*
+	 * Consult this provider for one request.  fill is NULL on a
+	 * consult-only walk; only the computing provider reads it,
+	 * and it must pass when fill is NULL.
+	 */
+	enum diff_provider_disposition
+		(*consult)(struct diff_provider *provider,
+			   const struct diff_provider_request *req,
+			   diff_provider_fill_fn fill, void *fill_data,
+			   xdl_emit_hunk_consume_func_t hunk_cb,
+			   void *cb_data);
+
+	/*
+	 * Tear down the provider's state, or NULL when it owns none.
+	 * Runs when the owning repository is cleared; the chain frees
+	 * the provider itself afterwards.
+	 */
+	void (*release)(struct diff_provider *provider);
+
+	void *state;
+
+	/*
+	 * Set on the provider that loads content and computes rather
+	 * than answering from the request's identity.  It alone is
+	 * still consulted after a stop-no-record: an identity answer
+	 * may no longer be served, but the computation must still
+	 * run.
+	 */
+	unsigned computes:1;
+
+	struct diff_provider *next;
+};
+
+/*
+ * Incremental well-formedness check for a provider-supplied hunk
+ * sequence, shared by every provider.  Each coordinate, and each
+ * hunk's end (its start plus count), must fit int32 (a consumer may
+ * truncate to int, and a provider may serialize as such); hunks must
+ * be in order and must not overlap; and the unchanged run between
+ * hunks must be the same length on both sides, or a consumer that
+ * walks the two files in lockstep desynchronizes.  Every rule
+ * constrains differences between coordinates, so the check applies
+ * to 0-based and 1-based sequences alike.
+ *
+ * Feed the hunks in order to a zero-initialized struct; the first
+ * nonzero return names the violated rule, and the whole sequence must
+ * then be discarded unemitted.
+ */
+struct diff_provider_hunks_check {
+	int64_t prev_old_end, prev_new_end;
+};
+
+enum diff_provider_hunks_error {
+	DIFF_PROVIDER_HUNKS_OK = 0,
+	DIFF_PROVIDER_HUNKS_RANGE,      /* negative or beyond int32 */
+	DIFF_PROVIDER_HUNKS_OVERLAP,    /* out of order or overlapping */
+	DIFF_PROVIDER_HUNKS_MISALIGNED, /* unchanged runs differ in length */
+};
+
+enum diff_provider_hunks_error
+diff_provider_check_hunk(struct diff_provider_hunks_check *c,
+			  long old_start, long old_count,
+			  long new_start, long new_count);
+
+#endif /* DIFF_PROVIDER_INTERNAL_H */
diff --git a/diff-provider.c b/diff-provider.c
new file mode 100644
index 0000000000..b69854fb63
--- /dev/null
+++ b/diff-provider.c
@@ -0,0 +1,154 @@
+#include "git-compat-util.h"
+#include "diff-provider-internal.h"
+#include "repository.h"
+
+/*
+ * The terminal provider: the builtin computation.  A request that
+ * carries a fill callback is answered by loading the pair's content
+ * and running xdiff, so a walk that reaches it never falls through
+ * to the consumer.  On a consult-only walk it passes, and the walk's
+ * fall-through outcome tells the consumer to compute.
+ */
+static enum diff_provider_disposition
+builtin_consult(struct diff_provider *provider UNUSED,
+		const struct diff_provider_request *req,
+		diff_provider_fill_fn fill, void *fill_data,
+		xdl_emit_hunk_consume_func_t hunk_cb, void *cb_data)
+{
+	xdemitconf_t xecfg = { .hunk_func = hunk_cb };
+	xdemitcb_t ecb = { .priv = cb_data };
+	mmfile_t old_file, new_file;
+
+	if (!fill)
+		return DIFF_PROVIDER_DISP_PASS;
+	if (fill(fill_data, &old_file, &new_file) < 0)
+		return DIFF_PROVIDER_DISP_ERROR;
+	if (xdi_diff(&old_file, &new_file, req->xpp, &xecfg, &ecb) < 0)
+		return DIFF_PROVIDER_DISP_ERROR;
+	return DIFF_PROVIDER_DISP_ANSWERED;
+}
+
+static struct diff_provider *builtin_provider_new(void)
+{
+	struct diff_provider *p = xcalloc(1, sizeof(*p));
+
+	p->consult = builtin_consult;
+	p->computes = 1;
+	return p;
+}
+
+/*
+ * The repository's chain, assembled on first walk.  The composition
+ * is fixed; the builtin computation is the terminal provider, so the
+ * chain always ends in an implementor that can answer.  Nothing is
+ * decided per repository here; each provider gates itself per
+ * request.
+ */
+static struct diff_provider *provider_chain(struct repository *r)
+{
+	struct diff_provider **tail = &r->diff_providers;
+
+	if (*tail)
+		return *tail;
+	*tail = builtin_provider_new();
+	return r->diff_providers;
+}
+
+void diff_providers_clear(struct repository *r)
+{
+	struct diff_provider *p = r->diff_providers;
+
+	while (p) {
+		struct diff_provider *next = p->next;
+
+		if (p->release)
+			p->release(p);
+		free(p);
+		p = next;
+	}
+	r->diff_providers = NULL;
+}
+
+/*
+ * The walk behind diff_provider_emit_hunks(): consult the chain in
+ * order and map its dispositions onto the outcome set.  The first
+ * answer ends the walk.  A stop-no-record disposition
+ * (diff-provider-internal.h) is a refusal, not a pass: the provider
+ * does not answer, but rules the pair out of identity service and
+ * out of recording, so from then on the walk consults only the
+ * computing provider, and a walk that ends unanswered carries the
+ * no-record verdict.  With a fill callback the terminal provider
+ * computes instead of passing, so an emit walk returns only
+ * answered or error.
+ */
+static enum diff_provider_outcome
+walk_providers(const struct diff_provider_request *req,
+	       diff_provider_fill_fn fill, void *fill_data,
+	       xdl_emit_hunk_consume_func_t hunk_cb, void *cb_data)
+{
+	struct diff_provider *p;
+	int no_record = 0;
+
+	for (p = provider_chain(req->repo); p; p = p->next) {
+		enum diff_provider_disposition disp;
+
+		if (no_record && !p->computes)
+			continue;
+		disp = p->consult(p, req, fill, fill_data,
+				  hunk_cb, cb_data);
+		if (disp == DIFF_PROVIDER_DISP_ERROR && !p->computes)
+			BUG("only the computing provider may return the "
+			    "error disposition");
+		if (p->computes && !fill && disp != DIFF_PROVIDER_DISP_PASS)
+			BUG("the computing provider must pass on a "
+			    "fill-less walk");
+		switch (disp) {
+		case DIFF_PROVIDER_DISP_ANSWERED:
+			return DIFF_PROVIDER_ANSWERED;
+		case DIFF_PROVIDER_DISP_PASS:
+			continue;
+		case DIFF_PROVIDER_DISP_STOP_NO_RECORD:
+			no_record = 1;
+			continue;
+		case DIFF_PROVIDER_DISP_ERROR:
+			return DIFF_PROVIDER_ERROR;
+		}
+	}
+	return no_record ? DIFF_PROVIDER_UNANSWERED_NO_RECORD :
+		DIFF_PROVIDER_UNANSWERED;
+}
+
+enum diff_provider_hunks_error
+diff_provider_check_hunk(struct diff_provider_hunks_check *c,
+			  long old_start, long old_count,
+			  long new_start, long new_count)
+{
+	if (old_start < 0 || old_count < 0 ||
+	    new_start < 0 || new_count < 0 ||
+	    old_start > INT32_MAX || old_count > INT32_MAX ||
+	    new_start > INT32_MAX || new_count > INT32_MAX ||
+	    (int64_t)old_start + old_count > INT32_MAX ||
+	    (int64_t)new_start + new_count > INT32_MAX)
+		return DIFF_PROVIDER_HUNKS_RANGE;
+	if (old_start < c->prev_old_end || new_start < c->prev_new_end)
+		return DIFF_PROVIDER_HUNKS_OVERLAP;
+	if (old_start - c->prev_old_end != new_start - c->prev_new_end)
+		return DIFF_PROVIDER_HUNKS_MISALIGNED;
+	/*
+	 * With each field bounded to int32 above, the int64 sums cannot
+	 * overflow even where long is 32-bit, and the range rule has
+	 * already capped them at INT32_MAX.
+	 */
+	c->prev_old_end = (int64_t)old_start + old_count;
+	c->prev_new_end = (int64_t)new_start + new_count;
+	return DIFF_PROVIDER_HUNKS_OK;
+}
+
+enum diff_provider_outcome
+diff_provider_emit_hunks(const struct diff_provider_request *req,
+			 diff_provider_fill_fn fill, void *fill_data,
+			 xdl_emit_hunk_consume_func_t hunk_cb,
+			 void *cb_data)
+{
+	return walk_providers(req, fill, fill_data, hunk_cb, cb_data);
+}
diff --git a/diff-provider.h b/diff-provider.h
new file mode 100644
index 0000000000..1a7e4e299c
--- /dev/null
+++ b/diff-provider.h
@@ -0,0 +1,130 @@
+#ifndef DIFF_PROVIDER_H
+#define DIFF_PROVIDER_H
+
+#include "xdiff-interface.h"
+
+/*
+ * The hunk provider interface sits between naming a pair of file
+ * versions to diff and computing their changed line ranges.
+ * Consumers that operate on hunk coordinates route their diff
+ * through here, so that a provider can answer for the pair before
+ * its content is loaded.
+ *
+ * A hunk provider answers a consumer's request from the pair's
+ * identity (its blob object ids) and the parameters that determine
+ * the diff; a request no provider answers falls through to the
+ * consumer's own computation.  A provider is either authoritative for
+ * its requests, meaning its answer may deliberately differ from the
+ * builtin diff, or not, meaning its answer must reproduce the builtin
+ * result exactly.  The interface resolves that authority through a
+ * provider chain owned by the repository, built on first consultation
+ * and released by repo_clear(): chain order is the resolution, and
+ * the builtin computation itself is the chain's terminal provider.  A
+ * consumer never names a provider; it reads the outcome below.
+ * Every answer a provider serves from identity passes the shared
+ * coordinate check (diff-provider-internal.h) before any consumer
+ * sees it.
+ */
+
+struct repository;
+
+/*
+ * The result of a consultation: two dependent axes flattened into
+ * their four valid points.  The first axis is the state of the
+ * response: the pair was answered, no provider answered, or (from
+ * diff_provider_emit_hunks() alone) the attempt failed.  The second
+ * axis exists only in the unanswered state: whether what the caller
+ * computes for this request may be recorded, the one rule the
+ * interface imposes on an otherwise free caller.  The rule travels
+ * in the outcome because the knowledge is a provider's while the
+ * recording is the caller's, and it shares the enum with the state,
+ * rather than riding a separate flag, so that no meaningless
+ * combination is representable and -Wswitch forces every consumer
+ * that switches to place the no-record arm.
+ *
+ * These values describe consultations, not providers: the set does
+ * not grow when a provider is added; a new provider maps onto these
+ * values inside the interface, so consumer code is written once.
+ * Each entry point returns a subrange of the set (stated at its
+ * declaration); a switch over this enum should list every value and
+ * omit "default:" so -Wswitch keeps it exhaustive, and a caller for
+ * whom only one value is actionable may compare against that value
+ * alone.
+ */
+enum diff_provider_outcome {
+	/*
+	 * Loading or diffing the pair failed.  Returned only by
+	 * diff_provider_emit_hunks(), whose compute leg is the only
+	 * part of a consultation that can fail.
+	 */
+	DIFF_PROVIDER_ERROR = -1,
+
+	/*
+	 * The request is answered: every hunk of the pair has been
+	 * emitted through the callback.  An authoritative provider
+	 * that finds the pair equivalent answers with no hunks at
+	 * all, so a callback that never fired is an answer, not an
+	 * accident.
+	 */
+	DIFF_PROVIDER_ANSWERED = 0,
+
+	/*
+	 * No provider answered.  What happens next is the caller's
+	 * business, typically computing the diff itself; a result it
+	 * computes for this request may be recorded.
+	 */
+	DIFF_PROVIDER_UNANSWERED,
+
+	/*
+	 * No provider answered, and what the caller computes for
+	 * this request must not be recorded: either an authoritative
+	 * provider owns the pair and declined this request, or the
+	 * request is shaped by parameters outside the recording key,
+	 * the key a recorded result is later served by.
+	 */
+	DIFF_PROVIDER_UNANSWERED_NO_RECORD,
+};
+
+/*
+ * A consultation request.  The interface consults providers from
+ * these fields alone; no content is loaded before an answer.
+ *
+ * repo owns the provider chain the request walks.  xpp carries the
+ * parameters the diff runs with.  Each provider gates itself on the
+ * fields that concern it.
+ */
+struct diff_provider_request {
+	struct repository *repo;
+	const xpparam_t *xpp;
+};
+
+/*
+ * Load the pair's content.  Called at most once per request, only
+ * when the ranges are computed rather than provided.  The buffers
+ * borrow storage owned by the callback's owner.
+ */
+typedef int (*diff_provider_fill_fn)(void *data, mmfile_t *old_file,
+				     mmfile_t *new_file);
+
+/*
+ * Consult the providers and, when no identity answer serves the
+ * request, load the pair's content through fill and compute its
+ * exact changed ranges (context 0).  Emits to hunk_cb either way and
+ * returns DIFF_PROVIDER_ANSWERED, or DIFF_PROVIDER_ERROR when fill
+ * or the diff fails.  The unanswered outcomes are never returned: a
+ * pair no provider answers is computed here instead of in the caller.
+ */
+enum diff_provider_outcome
+diff_provider_emit_hunks(const struct diff_provider_request *req,
+			 diff_provider_fill_fn fill, void *fill_data,
+			 xdl_emit_hunk_consume_func_t hunk_cb,
+			 void *cb_data);
+
+/*
+ * Release the repository's provider chain: stop any provider-owned
+ * processes and free the providers.  Called by repo_clear(); the
+ * chain builds again on the next consultation.
+ */
+void diff_providers_clear(struct repository *r);
+
+#endif /* DIFF_PROVIDER_H */
diff --git a/meson.build b/meson.build
index f7c40ea079..539a50f90e 100644
--- a/meson.build
+++ b/meson.build
@@ -356,6 +356,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-provider.c',
   'diff.c',
   'diffcore-break.c',
   'diffcore-delta.c',
diff --git a/repository.c b/repository.c
index 2ef0778846..c9418f0ae1 100644
--- a/repository.c
+++ b/repository.c
@@ -5,6 +5,7 @@
 #include "odb.h"
 #include "odb/source.h"
 #include "config.h"
+#include "diff-provider.h"
 #include "gettext.h"
 #include "object.h"
 #include "lockfile.h"
@@ -383,6 +384,8 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->submodule_prefix);
 	FREE_AND_NULL(repo->ref_storage_payload);
 
+	diff_providers_clear(repo);
+
 	odb_free(repo->objects);
 	repo->objects = NULL;
 
diff --git a/repository.h b/repository.h
index b767307911..8b4747fb2c 100644
--- a/repository.h
+++ b/repository.h
@@ -7,6 +7,7 @@
 #include "environment.h"
 
 struct config_set;
+struct diff_provider;
 struct git_hash_algo;
 struct index_state;
 struct lock_file;
@@ -161,6 +162,13 @@ struct repository {
 	/* Repository's remotes and associated structures. */
 	struct remote_state *remote_state;
 
+	/*
+	 * The repository's diff hunk provider chain, NULL until the
+	 * first consultation builds it (diff-provider.c); repo_clear()
+	 * releases it.
+	 */
+	struct diff_provider *diff_providers;
+
 	/* Repository's current hash algorithm, as serialized on disk. */
 	const struct git_hash_algo *hash_algo;
 
-- 
2.54.0

[RFC PATCH v7 03/10] diff-hunks: add the store format, library, and command

[Michael Montalbo]

Blame and "git log --stat" recover hunk coordinates by diffing blob
pairs, and recompute them on every run.  Add a cache of those
coordinates at $GIT_DIR/objects/info/diff-hunks, beside the
commit-graph, so a later run can look them up instead of decompressing
the blobs and running xdiff again.

The store is a single chunk-format file (see gitformat-chunk(5)): an
8-byte header, a DHIX index of fixed-size entries sorted by key, a DHDT
segment of hunk records, and a trailing hash checksum.  An entry is
keyed by the two blob object ids and the xdl_opts the pair was diffed
under, so a stored result is served only where that exact key recurs,
independent of path.  A zero-context diff trims unchanged lines from
hunk edges and can pick a different but equally valid set of hunks than
an untrimmed diff, so a recording caller stores a pair only when its
trimmed and untrimmed diffs are identical; such an entry answers any
consumer at any context, and the rare divergent pair is always
computed.  Identical hunk blocks are interned once and shared across
keys.

The library provides a reader (repo_diff_hunks_store and _replay, gated
by core.diffHunks), loaded once and cached on the object database as the
commit-graph is, and a writer that accumulates entries and flushes them
in one atomic pass.  An absent, corrupt, or disabled store reads as all
misses.  A record with no hunks is invalid too: replaying it would claim
the pair equivalent, which the store never asserts, so it reads as a
miss.

Ordinary reads are diagnostic-free.  Loading parses the chunk table
through read_table_of_contents_quiet(), new in chunk-format, which
prints nothing on a malformed table and takes the repository's hash
algorithm rather than the_hash_algo, so the file is bounds-checked under
the algorithm it is keyed by.

The flush closes the repository's mmapped store and forgets that loading
was attempted before committing the lockfile.  A warming run that also
reads may hold the file it is replacing mapped, and the rename must not
land on a live mapping, which Windows refuses; a read after the flush
then observes the committed file.  commit-graph closes its graph before
committing for the same reason.

Writing is off by default, enabled per run by GIT_DIFF_HUNKS_WRITE or
persistently by diffHunks.write, the environment winning.  A writer
seeds from the existing store, so a flush merges rather than replaces.
The seed's checksum is verified first: a corrupt store is discarded, not
rewritten with a fresh checksum verify could no longer catch.  An entry
that fails the shared diff_provider_check_hunk() or names no blob is
dropped with a warning, since it would only ever read as a miss.  A seed
that discarded or dropped anything forces the flush even when the
warming run computed nothing new.  The writer fsyncs through a new
diff-hunks core.fsync component.

"git diff-hunks" inspects and manages the file: "verify" checks the
checksum, chunk table, sort order, entry bounds, and every entry's hunk
sequence against that shared check, so a store whose entries could only
read as misses fails verify; "clear" removes the file.  Later patches
wire the readers and the writer into the diff and blame paths.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 .gitignore                              |   1 +
 Documentation/Makefile                  |   1 +
 Documentation/config.adoc               |   2 +
 Documentation/config/core.adoc          |  10 +-
 Documentation/config/diff-hunks.adoc    |   8 +
 Documentation/git-diff-hunks.adoc       | 146 ++++
 Documentation/gitformat-diff-hunks.adoc | 129 ++++
 Documentation/meson.build               |   2 +
 Makefile                                |   2 +
 builtin.h                               |   1 +
 builtin/diff-hunks.c                    |  53 ++
 chunk-format.c                          |  62 +-
 chunk-format.h                          |  14 +
 command-list.txt                        |   2 +
 diff-hunks.c                            | 916 ++++++++++++++++++++++++
 diff-hunks.h                            | 117 +++
 environment.c                           |   1 +
 git.c                                   |   1 +
 meson.build                             |   2 +
 odb.c                                   |   2 +
 odb.h                                   |   4 +
 repo-settings.c                         |   1 +
 repo-settings.h                         |   1 +
 write-or-die.h                          |   7 +-
 24 files changed, 1467 insertions(+), 18 deletions(-)
 create mode 100644 Documentation/config/diff-hunks.adoc
 create mode 100644 Documentation/git-diff-hunks.adoc
 create mode 100644 Documentation/gitformat-diff-hunks.adoc
 create mode 100644 builtin/diff-hunks.c
 create mode 100644 diff-hunks.c
 create mode 100644 diff-hunks.h

diff --git a/.gitignore b/.gitignore
index 4da58c6754..4173111c01 100644
--- a/.gitignore
+++ b/.gitignore
@@ -56,6 +56,7 @@
 /git-diagnose
 /git-diff
 /git-diff-files
+/git-diff-hunks
 /git-diff-index
 /git-diff-pairs
 /git-diff-tree
diff --git a/Documentation/Makefile b/Documentation/Makefile
index 2699f0b24a..170fcee66e 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -33,6 +33,7 @@ MAN5_TXT += gitattributes.adoc
 MAN5_TXT += gitformat-bundle.adoc
 MAN5_TXT += gitformat-chunk.adoc
 MAN5_TXT += gitformat-commit-graph.adoc
+MAN5_TXT += gitformat-diff-hunks.adoc
 MAN5_TXT += gitformat-index.adoc
 MAN5_TXT += gitformat-loose.adoc
 MAN5_TXT += gitformat-pack.adoc
diff --git a/Documentation/config.adoc b/Documentation/config.adoc
index 1ef72de62f..8a172d52f3 100644
--- a/Documentation/config.adoc
+++ b/Documentation/config.adoc
@@ -472,6 +472,8 @@ include::config/credential.adoc[]
 
 include::config/diff.adoc[]
 
+include::config/diff-hunks.adoc[]
+
 include::config/difftool.adoc[]
 
 include::config/extensions.adoc[]
diff --git a/Documentation/config/core.adoc b/Documentation/config/core.adoc
index a0ebf03e2e..9595619c61 100644
--- a/Documentation/config/core.adoc
+++ b/Documentation/config/core.adoc
@@ -670,12 +670,13 @@ but risks losing recent work in the event of an unclean system shutdown.
 * `pack` hardens objects added to the repo in packfile form.
 * `pack-metadata` hardens packfile bitmaps and indexes.
 * `commit-graph` hardens the commit-graph file.
+* `diff-hunks` hardens the diff-hunks store.
 * `index` hardens the index when it is modified.
 * `objects` is an aggregate option that is equivalent to
   `loose-object,pack`.
 * `reference` hardens references modified in the repo.
 * `derived-metadata` is an aggregate option that is equivalent to
-  `pack-metadata,commit-graph`.
+  `pack-metadata,commit-graph,diff-hunks`.
 * `committed` is an aggregate option that is currently equivalent to
   `objects`. This mode sacrifices some performance to ensure that work
   that is committed to the repository with `git commit` or similar commands
@@ -750,6 +751,13 @@ core.commitGraph::
 	to parse the graph structure of commits. Defaults to true. See
 	linkgit:git-commit-graph[1] for more information.
 
+core.diffHunks::
+	If true, then Git will consult the diff-hunks store (if it
+	exists) to skip recomputing diff hunk coordinates in commands
+	such as `git log --stat` and linkgit:git-blame[1]. This controls
+	only reading; writing the store is controlled by `diffHunks.write`.
+	See linkgit:git-diff-hunks[1] for more information. Defaults to true.
+
 core.useReplaceRefs::
 	If set to `false`, behave as if the `--no-replace-objects`
 	option was given on the command line. See linkgit:git[1] and
diff --git a/Documentation/config/diff-hunks.adoc b/Documentation/config/diff-hunks.adoc
new file mode 100644
index 0000000000..ad76d1c6a9
--- /dev/null
+++ b/Documentation/config/diff-hunks.adoc
@@ -0,0 +1,8 @@
+diffHunks.write::
+	If true, diff-producing commands (`git diff`, `git log`,
+	`git show`, or `git diff-tree` with a `--stat`, `--numstat`, or
+	`--shortstat` format) write the hunks
+	they compute to the diff-hunks store, filling it as a side effect.
+	The `GIT_DIFF_HUNKS_WRITE` environment variable overrides this for
+	a single invocation. Reading the store is controlled separately by
+	`core.diffHunks`. See linkgit:git-diff-hunks[1]. Defaults to false.
diff --git a/Documentation/git-diff-hunks.adoc b/Documentation/git-diff-hunks.adoc
new file mode 100644
index 0000000000..25cab2ea7d
--- /dev/null
+++ b/Documentation/git-diff-hunks.adoc
@@ -0,0 +1,146 @@
+git-diff-hunks(1)
+=================
+
+NAME
+----
+git-diff-hunks - Inspect and manage the diff-hunks store
+
+SYNOPSIS
+--------
+[synopsis]
+git diff-hunks verify
+git diff-hunks clear
+
+DESCRIPTION
+-----------
+
+The diff-hunks store is a cache of diff hunk coordinates, the line
+ranges that changed between two blobs, so that commands
+which need them, such as linkgit:git-blame[1] and `git log` and `git diff`
+with the `--stat`, `--numstat`, and `--shortstat` formats, can skip
+running the diff algorithm, and blame can skip loading the blob
+content. (The summary formats still test each pair for binariness,
+which can load the blobs.)
+
+The store is a single file, `$GIT_DIR/objects/info/diff-hunks`. Reading is
+enabled by default; writing is off by default. A `git diff`, `git log`,
+`git show`, or `git diff-tree` that produces one of the stat formats
+fills the store as a side effect, but only when writing is enabled for
+that run (see "WARMING THE STORE" below), so ordinary reads never
+modify the repository. When the store does not have the pair, holds a
+different object hash, the file is unreadable, or an object replacement
+redirects one of the blobs, the consumer falls back to computing the
+diff. A store only speeds up these commands; it never changes their
+output.
+
+`git diff-hunks` itself only inspects and manages the file. See
+linkgit:gitformat-diff-hunks[5] for the file format.
+
+WARMING THE STORE
+-----------------
+
+The store is filled by running ordinary commands with writing enabled.
+Turn writing on for a single invocation with the `GIT_DIFF_HUNKS_WRITE`
+environment variable, or persistently with the `diffHunks.write`
+configuration; the environment variable takes precedence. A repository
+owner warms the store by running the diff-producing commands they care
+about with writing on, for example:
+
+	GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
+
+A `--stat` walk records one entry per blob pair;
+linkgit:git-blame[1] replays the coordinates and the summary formats
+sum the counts, so a single warming walk serves both.
+A warming run seeds from the existing store and rewrites the file
+with the newly computed pairs merged in, so a later run adds to what
+earlier runs recorded rather than discarding it.
+
+A walk records only the pairs it diffs. `git log --all --stat` diffs
+each commit against its first parent, so a blame that follows a
+merge's second parent computes those pairs itself: blame coverage is
+partial on history with merges. Warming with a walk that also diffs
+the other parents, for example `git log --all -m --stat`, raises
+blame coverage at the cost of a larger store and a longer warming
+run.
+
+COMMANDS
+--------
+
+`verify`::
+	Check the integrity of the store: the trailing hash checksum, the
+	chunk table of contents, the sort order of the index, and the
+	bounds of every entry. Exits with non-zero status if the store is
+	corrupt. An absent store is valid.
+
+`clear`::
+	Remove the store file.
+
+CORRECTNESS
+-----------
+
+A stored result is interchangeable with a freshly computed one because an
+entry is keyed by the inputs that determine the diff:
+
+* the object IDs of the old and new blob, so a result is used only for
+  the exact contents it was computed from; and
+* the diff algorithm and ignore flags (`xdl_opts`) the hunks were
+  computed under. A lookup whose `xdl_opts` differ from a stored entry
+  misses. This is why, for example, `blame -w` and
+  `--diff-algorithm=<algorithm>` (including a per-path
+  `diff.<driver>.algorithm`) do not reuse entries recorded under the
+  default settings: they change `xdl_opts`.
+
+The context length is not part of the key because only trim-stable
+pairs are recorded: pairs whose zero-context trimmed diff and untrimmed
+diff are identical, so one entry answers blame (zero context) and the
+summary formats (any context) alike. The rare pair where
+the zero-context trimming optimization picks a different but
+equally valid set of hunks is
+never recorded and is always computed.
+
+Some options shape the hunks in ways the key does not express, so a
+diff that uses them is excluded from the store in both directions:
+break detection (`-B`), `--ignore-matching-lines` (`-I`), and
+`--anchored`. `--ignore-blank-lines` is different: it is an ignore
+flag and therefore part of the key, but the summary formats exclude
+it anyway, because it coalesces hunks differently between the code
+path that emits text and the one that replays coordinates, so a
+served answer would not match a store-less run.
+linkgit:git-blame[1] additionally does not
+consult the store for reverse blame, ignored revisions, or paths with a
+textconv driver.
+
+The store carries a trailing hash checksum, but readers do not
+re-checksum it on every load. As with the commit-graph and
+multi-pack-index, the writer fsyncs the file (honoring `core.fsync`) and
+commits it atomically, so a committed store is intact; every offset and
+count is still bounds-checked as it is read. The checksum is verified by
+`git diff-hunks verify`, not on the read path, so structural corruption
+that fails a bounds check is read as an absent entry, while a record
+that stays within bounds but whose bytes were altered is served until
+`verify` detects the mismatch.
+
+CONFIGURATION
+-------------
+
+`core.diffHunks`::
+	Whether commands read the store. Defaults to true. See
+	linkgit:git-config[1].
+
+`diffHunks.write`::
+	Whether diff-producing commands write to the store. Defaults to
+	false. The `GIT_DIFF_HUNKS_WRITE` environment variable overrides it
+	for a single invocation. See linkgit:git-config[1].
+
+Writing the store honors the `core.fsync` configuration through the
+`diff-hunks` component; see linkgit:git-config[1].
+
+SEE ALSO
+--------
+linkgit:git-blame[1],
+linkgit:git-log[1],
+linkgit:gitformat-diff-hunks[5]
+
+GIT
+---
+Part of the linkgit:git[1] suite
diff --git a/Documentation/gitformat-diff-hunks.adoc b/Documentation/gitformat-diff-hunks.adoc
new file mode 100644
index 0000000000..f75ab73dd9
--- /dev/null
+++ b/Documentation/gitformat-diff-hunks.adoc
@@ -0,0 +1,129 @@
+gitformat-diff-hunks(5)
+=======================
+
+NAME
+----
+gitformat-diff-hunks - Precomputed diff hunk store format
+
+SYNOPSIS
+--------
+[verse]
+$GIT_DIR/objects/info/diff-hunks
+
+DESCRIPTION
+-----------
+
+The diff-hunks store memoizes diff hunk coordinates so that commands
+that need them, such as `git log --stat` and linkgit:git-blame[1], can
+skip running the diff algorithm (and, for blame, loading the blob
+content; the summary formats still test each pair for binariness,
+which can load the blobs). See
+linkgit:git-diff-hunks[1] for how the store is filled and managed and the
+configuration that controls it.
+
+The store is a single file, `$GIT_DIR/objects/info/diff-hunks`, written
+in one pass and replaced atomically, so a reader sees either the old
+file or the complete new one.
+
+Entries are keyed by the object IDs of the blob pair that was diffed
+and by the diff algorithm and ignore flags (`xdl_opts`) the pair was
+diffed under. A blob pair fully determines the diff input, so an entry
+is valid regardless of which commits, branches, or index states the
+pair was encountered in, and identical diffs performed in different
+contexts share one entry. A reader whose `xdl_opts` differ from an
+entry does not match it and falls back to computing the diff.
+
+FILE FORMAT
+-----------
+
+All multi-byte integers are stored in network byte order. The file is an
+8-byte header, the chunk table of contents and chunk data described in
+linkgit:gitformat-chunk[5], and a trailing checksum.
+
+HEADER
+~~~~~~
+
+- 4-byte signature: `DHPF` (diff-hunks precomputed format)
+- 1-byte version number: currently 1
+- 1-byte hash version: 1 for SHA-1, 2 for SHA-256. A store whose hash
+  function differs from the repository's is ignored.
+- 1-byte number of chunks
+- 1-byte reserved
+
+CHUNK LOOKUP
+~~~~~~~~~~~~
+
+A table of contents in the format of linkgit:gitformat-chunk[5], listing
+the offset of each chunk. Both chunks below are required; a file missing
+either is treated as corrupt.
+
+CHUNK DATA
+~~~~~~~~~~
+
+DHIX (index)::
+	A sorted sequence of fixed-size entries. Each entry is the old
+	blob object ID, the new blob object ID, a 4-byte `xdl_opts`
+	value, and a 4-byte offset into the DHDT chunk. Entries are
+	sorted by old object ID, then new object ID, then `xdl_opts`,
+	so lookups can use binary search on the full key.
+
+DHDT (hunk data)::
+	For each index entry, at its offset: a 4-byte hunk count followed
+	by that many 16-byte hunk records. A hunk record is four 4-byte
+	values: old start, old count, new start, new count.
+	Starts are 0-based line numbers in the old and new blob; counts
+	are numbers of lines. The hunk count is at least 1: a record with
+	no hunks would claim the blob pair equivalent, which the store
+	never records, so readers treat such a record as invalid.
+	Identical hunk blocks are stored once:
+	distinct index entries whose recorded hunks are byte-for-byte
+	equal point at the same offset.
+
+TRAILER
+~~~~~~~
+
+A checksum of all preceding bytes, computed with the repository hash
+function.
+
+CORRECTNESS
+-----------
+
+Serving hunks from a valid store produces the same output as recomputing
+the diff. The diff of a blob pair is not unique: a zero context length
+triggers xdiff's common-tail trimming, which can pick a different but
+equally valid set of hunks than an untrimmed diff does. A pair is
+therefore recorded only when its trimmed and untrimmed diffs are
+identical, which is the common case. Such an entry answers any consumer
+at any context: git-blame replays its coordinates directly (it diffs at
+zero context), and diffstat sums its per-hunk line counts, which the
+context length does not change. The rare pair whose two diffs differ is
+never recorded, so every consumer computes it.
+
+A store that cannot be used is ignored, and the consumer falls back to
+computing the diff. Every offset and count read from the file is
+bounds-checked, so a store that is missing, truncated, of an unknown
+version, or of a different object hash does not change the diff output
+and does not produce a diagnostic; `git diff-hunks verify` is what
+reports corruption.
+
+The store is not re-checksummed on the read path. The writer fsyncs the
+file (honoring `core.fsync`) and commits it atomically, so a
+committed store is intact, the same trust model the commit-graph and
+multi-pack-index use. The trailing checksum is recomputed by
+`git diff-hunks verify` to detect corruption.
+
+The checksum detects corruption but does not prove who wrote the file. A
+reader trusts the coordinates in a store that passes its checks, so
+anything able to write a checksum-valid file at the store path can
+influence output, the same as it could by writing objects directly.
+
+LIMITATIONS
+-----------
+
+- Hunk counts, offsets, and line coordinates are 32-bit, capping the
+  hunk data at 4 GiB and a single entry at roughly 268 million hunks.
+  A result whose coordinates cannot be represented is not recorded.
+
+GIT
+---
+Part of the linkgit:git[1] suite
diff --git a/Documentation/meson.build b/Documentation/meson.build
index f4854f802d..85f37da47e 100644
--- a/Documentation/meson.build
+++ b/Documentation/meson.build
@@ -41,6 +41,7 @@ manpages = {
   'git-describe.adoc' : 1,
   'git-diagnose.adoc' : 1,
   'git-diff-files.adoc' : 1,
+  'git-diff-hunks.adoc' : 1,
   'git-diff-index.adoc' : 1,
   'git-diff-pairs.adoc' : 1,
   'git-difftool.adoc' : 1,
@@ -175,6 +176,7 @@ manpages = {
   'gitformat-bundle.adoc' : 5,
   'gitformat-chunk.adoc' : 5,
   'gitformat-commit-graph.adoc' : 5,
+  'gitformat-diff-hunks.adoc' : 5,
   'gitformat-index.adoc' : 5,
   'gitformat-loose.adoc' : 5,
   'gitformat-pack.adoc' : 5,
diff --git a/Makefile b/Makefile
index 50c96807d6..11a06934b3 100644
--- a/Makefile
+++ b/Makefile
@@ -1159,6 +1159,7 @@ LIB_OBJS += diffcore-order.o
 LIB_OBJS += diffcore-pickaxe.o
 LIB_OBJS += diffcore-rename.o
 LIB_OBJS += diffcore-rotate.o
+LIB_OBJS += diff-hunks.o
 LIB_OBJS += dir-iterator.o
 LIB_OBJS += dir.o
 LIB_OBJS += editor.o
@@ -1421,6 +1422,7 @@ BUILTIN_OBJS += builtin/credential.o
 BUILTIN_OBJS += builtin/describe.o
 BUILTIN_OBJS += builtin/diagnose.o
 BUILTIN_OBJS += builtin/diff-files.o
+BUILTIN_OBJS += builtin/diff-hunks.o
 BUILTIN_OBJS += builtin/diff-index.o
 BUILTIN_OBJS += builtin/diff-pairs.o
 BUILTIN_OBJS += builtin/diff-tree.o
diff --git a/builtin.h b/builtin.h
index 4e47a4ebd3..7e64da9f43 100644
--- a/builtin.h
+++ b/builtin.h
@@ -175,6 +175,7 @@ int cmd_credential_store(int argc, const char **argv, const char *prefix, struct
 int cmd_describe(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diagnose(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff_files(int argc, const char **argv, const char *prefix, struct repository *repo);
+int cmd_diff_hunks(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff_index(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff(int argc, const char **argv, const char *prefix, struct repository *repo);
 int cmd_diff_pairs(int argc, const char **argv, const char *prefix, struct repository *repo);
diff --git a/builtin/diff-hunks.c b/builtin/diff-hunks.c
new file mode 100644
index 0000000000..3aea2dad56
--- /dev/null
+++ b/builtin/diff-hunks.c
@@ -0,0 +1,53 @@
+#include "builtin.h"
+#include "config.h"
+#include "diff-hunks.h"
+#include "gettext.h"
+#include "parse-options.h"
+#include "repository.h"
+
+static const char * const diff_hunks_usage[] = {
+	N_("git diff-hunks verify"),
+	N_("git diff-hunks clear"),
+	NULL
+};
+
+static int cmd_diff_hunks_verify(int argc, const char **argv,
+				 const char *prefix UNUSED,
+				 struct repository *r)
+{
+	struct option options[] = { OPT_END() };
+
+	argc = parse_options(argc, argv, NULL, options, diff_hunks_usage, 0);
+	if (argc)
+		usage_with_options(diff_hunks_usage, options);
+	return diff_hunks_verify(r) ? 1 : 0;
+}
+
+static int cmd_diff_hunks_clear(int argc, const char **argv,
+				const char *prefix UNUSED,
+				struct repository *r)
+{
+	struct option options[] = { OPT_END() };
+
+	argc = parse_options(argc, argv, NULL, options, diff_hunks_usage, 0);
+	if (argc)
+		usage_with_options(diff_hunks_usage, options);
+	return diff_hunks_clear(r) ? 1 : 0;
+}
+
+int cmd_diff_hunks(int argc, const char **argv, const char *prefix,
+		   struct repository *repo)
+{
+	parse_opt_subcommand_fn *fn = NULL;
+	struct option options[] = {
+		OPT_SUBCOMMAND("verify", &fn, cmd_diff_hunks_verify),
+		OPT_SUBCOMMAND("clear", &fn, cmd_diff_hunks_clear),
+		OPT_END()
+	};
+
+	repo_config(repo, git_default_config, NULL);
+
+	argc = parse_options(argc, argv, prefix, options, diff_hunks_usage, 0);
+
+	return fn(argc, argv, prefix, repo);
+}
diff --git a/chunk-format.c b/chunk-format.c
index 51b5a2c959..34ab2750f7 100644
--- a/chunk-format.c
+++ b/chunk-format.c
@@ -101,12 +101,14 @@ int write_chunkfile(struct chunkfile *cf, void *data)
 	return result;
 }
 
-int read_table_of_contents(struct chunkfile *cf,
-			   const unsigned char *mfile,
-			   size_t mfile_size,
-			   uint64_t toc_offset,
-			   int toc_length,
-			   unsigned expected_alignment)
+static int read_table_of_contents_1(struct chunkfile *cf,
+				    const unsigned char *mfile,
+				    size_t mfile_size,
+				    uint64_t toc_offset,
+				    int toc_length,
+				    unsigned expected_alignment,
+				    const struct git_hash_algo *algo,
+				    int quiet)
 {
 	int i;
 	uint32_t chunk_id;
@@ -121,12 +123,14 @@ int read_table_of_contents(struct chunkfile *cf,
 		chunk_offset = get_be64(table_of_contents + 4);
 
 		if (!chunk_id) {
-			error(_("terminating chunk id appears earlier than expected"));
+			if (!quiet)
+				error(_("terminating chunk id appears earlier than expected"));
 			return 1;
 		}
 		if (chunk_offset % expected_alignment != 0) {
-			error(_("chunk id %"PRIx32" not %d-byte aligned"),
-			      chunk_id, expected_alignment);
+			if (!quiet)
+				error(_("chunk id %"PRIx32" not %d-byte aligned"),
+				      chunk_id, expected_alignment);
 			return 1;
 		}
 
@@ -134,16 +138,18 @@ int read_table_of_contents(struct chunkfile *cf,
 		next_chunk_offset = get_be64(table_of_contents + 4);
 
 		if (next_chunk_offset < chunk_offset ||
-		    next_chunk_offset > mfile_size - the_hash_algo->rawsz) {
-			error(_("improper chunk offset(s) %"PRIx64" and %"PRIx64""),
-			      chunk_offset, next_chunk_offset);
+		    next_chunk_offset > mfile_size - algo->rawsz) {
+			if (!quiet)
+				error(_("improper chunk offset(s) %"PRIx64" and %"PRIx64""),
+				      chunk_offset, next_chunk_offset);
 			return -1;
 		}
 
 		for (i = 0; i < cf->chunks_nr; i++) {
 			if (cf->chunks[i].id == chunk_id) {
-				error(_("duplicate chunk ID %"PRIx32" found"),
-					chunk_id);
+				if (!quiet)
+					error(_("duplicate chunk ID %"PRIx32" found"),
+					      chunk_id);
 				return -1;
 			}
 		}
@@ -156,13 +162,39 @@ int read_table_of_contents(struct chunkfile *cf,
 
 	chunk_id = get_be32(table_of_contents);
 	if (chunk_id) {
-		error(_("final chunk has non-zero id %"PRIx32""), chunk_id);
+		if (!quiet)
+			error(_("final chunk has non-zero id %"PRIx32""), chunk_id);
 		return -1;
 	}
 
 	return 0;
 }
 
+int read_table_of_contents(struct chunkfile *cf,
+			   const unsigned char *mfile,
+			   size_t mfile_size,
+			   uint64_t toc_offset,
+			   int toc_length,
+			   unsigned expected_alignment)
+{
+	return read_table_of_contents_1(cf, mfile, mfile_size, toc_offset,
+					toc_length, expected_alignment,
+					the_hash_algo, 0);
+}
+
+int read_table_of_contents_quiet(struct chunkfile *cf,
+				 const unsigned char *mfile,
+				 size_t mfile_size,
+				 uint64_t toc_offset,
+				 int toc_length,
+				 unsigned expected_alignment,
+				 const struct git_hash_algo *algo)
+{
+	return read_table_of_contents_1(cf, mfile, mfile_size, toc_offset,
+					toc_length, expected_alignment,
+					algo, 1);
+}
+
 struct pair_chunk_data {
 	const unsigned char **p;
 	size_t *size;
diff --git a/chunk-format.h b/chunk-format.h
index 212a0a6af1..bc31302ed0 100644
--- a/chunk-format.h
+++ b/chunk-format.h
@@ -39,6 +39,20 @@ int read_table_of_contents(struct chunkfile *cf,
 			   int toc_length,
 			   unsigned expected_alignment);
 
+/*
+ * Like read_table_of_contents(), for a reader that treats a malformed
+ * table as an absent file rather than reporting it: nothing is printed
+ * on failure, and the trailing-checksum bound is computed with the
+ * given hash algorithm instead of the_hash_algo.
+ */
+int read_table_of_contents_quiet(struct chunkfile *cf,
+				 const unsigned char *mfile,
+				 size_t mfile_size,
+				 uint64_t toc_offset,
+				 int toc_length,
+				 unsigned expected_alignment,
+				 const struct git_hash_algo *algo);
+
 #define CHUNK_NOT_FOUND (-2)
 
 /*
diff --git a/command-list.txt b/command-list.txt
index 21b802c420..e7b241e6ad 100644
--- a/command-list.txt
+++ b/command-list.txt
@@ -95,6 +95,7 @@ git-describe                            mainporcelain
 git-diagnose                            ancillaryinterrogators
 git-diff                                mainporcelain           info
 git-diff-files                          plumbinginterrogators
+git-diff-hunks                          plumbingmanipulators
 git-diff-index                          plumbinginterrogators
 git-diff-pairs                          plumbinginterrogators
 git-diff-tree                           plumbinginterrogators
@@ -223,6 +224,7 @@ gitfaq                                  guide
 gitformat-bundle                        developerinterfaces
 gitformat-chunk                         developerinterfaces
 gitformat-commit-graph                  developerinterfaces
+gitformat-diff-hunks                    developerinterfaces
 gitformat-index                         developerinterfaces
 gitformat-pack                          developerinterfaces
 gitformat-signature                     developerinterfaces
diff --git a/diff-hunks.c b/diff-hunks.c
new file mode 100644
index 0000000000..eeaaa2466a
--- /dev/null
+++ b/diff-hunks.c
@@ -0,0 +1,916 @@
+/*
+ * Precomputed diff hunks, keyed by diff input.
+ *
+ * A single store at .git/objects/info/diff-hunks maps an (old blob,
+ * new blob, xdl_opts) key to the hunk coordinates of diffing the pair.
+ * The key determines the diff result (only trim-stable pairs are
+ * recorded; see diff-hunks.h), so an entry is valid in any context it
+ * recurs in, independent of path. Reading is on by default
+ * (core.diffHunks); writing is off by default and enabled per run or
+ * by configuration (see diff_hunks_write_enabled), so an ordinary
+ * command populates the store only during a warming run the
+ * repository owner opts into.
+ *
+ * File layout:
+ *   Header:  "DHPF"(4) + version(1) + hash_version(1)
+ *            + num_chunks(1) + reserved(1)
+ *   Table of contents (chunk-format)
+ *   DHIX chunk: sorted entries, each
+ *       old_blob_oid, new_blob_oid, xdl_opts(4), hdat_offset(4)
+ *   DHDT chunk: per entry, num_hunks(4) followed by that many 16-byte hunks
+ *   Trailing hash checksum
+ */
+#include "git-compat-util.h"
+#include "chunk-format.h"
+#include "config.h"
+#include "csum-file.h"
+#include "diff-hunks.h"
+#include "diff-provider-internal.h"
+#include "gettext.h"
+#include "hash.h"
+#include "hashmap.h"
+#include "lockfile.h"
+#include "odb.h"
+#include "path.h"
+#include "repo-settings.h"
+#include "repository.h"
+#include "strbuf.h"
+#include "wrapper.h"
+
+#define DIFF_HUNKS_SIGNATURE 0x44485046 /* "DHPF" */
+/*
+ * Bump when the on-disk format changes, or when xdiff's emitted hunk
+ * coordinates change for a fixed (blobs, xdl_opts) key: an old store
+ * would otherwise serve stale hunks and change command output.
+ */
+#define DIFF_HUNKS_VERSION 1
+#define DIFF_HUNKS_HEADER_SIZE 8
+
+#define DIFF_HUNKS_CHUNKID_INDEX 0x44484958 /* "DHIX" */
+#define DIFF_HUNKS_CHUNKID_DATA 0x44484454 /* "DHDT" */
+
+/*
+ * Each hunk is 16 bytes on disk:
+ * old_start(4) old_count(4) new_start(4) new_count(4)
+ */
+#define DIFF_HUNKS_HUNK_SIZE (4 * sizeof(uint32_t))
+
+/*
+ * Result of a store lookup: num_hunks records encoded in the store's mmap,
+ * valid until the store is freed. Read them with nth_precomputed_hunk().
+ */
+struct precomputed_entry {
+	uint32_t num_hunks;
+	const unsigned char *hunk_data;
+};
+
+/* Decode a single hunk from the raw on-disk format. */
+static inline void decode_precomputed_hunk(const unsigned char *data,
+					   struct precomputed_hunk *h)
+{
+	h->old_start = get_be32(data);
+	h->old_count = get_be32(data + 4);
+	h->new_start = get_be32(data + 8);
+	h->new_count = get_be32(data + 12);
+}
+
+/* Decode the nth hunk of a lookup result into *h. */
+static inline void nth_precomputed_hunk(const struct precomputed_entry *e,
+					uint32_t n, struct precomputed_hunk *h)
+{
+	decode_precomputed_hunk(e->hunk_data + (size_t)n * DIFF_HUNKS_HUNK_SIZE, h);
+}
+
+/* Byte length of the (old_oid, new_oid, xdl_opts) lookup key. */
+static size_t store_index_key_size(const struct git_hash_algo *algo)
+{
+	return 2 * algo->rawsz + sizeof(uint32_t);
+}
+
+/* Index entry: the lookup key followed by the 4-byte offset into DHDT. */
+static size_t store_index_entry_size(const struct git_hash_algo *algo)
+{
+	return store_index_key_size(algo) + sizeof(uint32_t);
+}
+
+/*
+ * The smallest a valid store file can be: the header, a table of contents
+ * with one entry per chunk plus a terminating entry, and the trailing
+ * checksum.
+ */
+static size_t store_min_size(const struct git_hash_algo *algo,
+			     uint8_t num_chunks)
+{
+	size_t toc_size = (num_chunks + 1) * CHUNK_TOC_ENTRY_SIZE;
+
+	return DIFF_HUNKS_HEADER_SIZE + toc_size + algo->rawsz;
+}
+
+/*
+ * Decode an index entry's key into pointers to the two oids and the
+ * xdl_opts value (on-disk: old_oid, new_oid, then xdl_opts as a
+ * big-endian uint32).
+ */
+static void decode_store_index_key(const unsigned char *entry, unsigned int rawsz,
+			     const unsigned char **old_hash,
+			     const unsigned char **new_hash,
+			     uint32_t *xdl_opts)
+{
+	*old_hash = entry;
+	*new_hash = entry + rawsz;
+	*xdl_opts = get_be32(entry + 2 * rawsz);
+}
+
+/* The DHDT offset stored in an index entry, in the field after its key. */
+static uint32_t index_entry_hdat_offset(const unsigned char *entry, size_t keysz)
+{
+	return get_be32(entry + keysz);
+}
+
+static char *diff_hunks_store_path(struct repository *r)
+{
+	return xstrfmt("%s/info/diff-hunks", repo_get_object_directory(r));
+}
+
+struct diff_hunks_store {
+	const unsigned char *data;
+	size_t data_len;
+	const struct git_hash_algo *hash_algo;
+	const unsigned char *index;
+	uint32_t num_entries;
+	const unsigned char *hdat;
+	size_t hdat_size;
+};
+
+static void free_store(struct diff_hunks_store *s)
+{
+	if (!s)
+		return;
+	if (s->data)
+		munmap((void *)s->data, s->data_len);
+	free(s);
+}
+
+/*
+ * Open, mmap, and parse the store at fname. Returns the parsed store
+ * or NULL on any error. The diff output is unaffected either way;
+ * corruption is reported by verify, not treated as fatal here.
+ */
+static struct diff_hunks_store *load_store_at(
+		const struct git_hash_algo *repo_algo, const char *fname)
+{
+	struct diff_hunks_store *s;
+	struct chunkfile *cf;
+	int fd;
+	struct stat st;
+	void *data;
+	const unsigned char *p;
+	uint8_t num_chunks;
+	size_t index_size, entry_size, data_len;
+
+	fd = git_open(fname);
+	if (fd < 0)
+		return NULL;
+	if (fstat(fd, &st) || st.st_size < DIFF_HUNKS_HEADER_SIZE) {
+		close(fd);
+		return NULL;
+	}
+	data_len = xsize_t(st.st_size);
+	data = xmmap(NULL, data_len, PROT_READ, MAP_PRIVATE, fd, 0);
+	close(fd);
+	p = data;
+
+	num_chunks = p[6];
+
+	/*
+	 * Reject a file that is not a readable store: wrong signature,
+	 * version, or object hash, or too small to hold the table of
+	 * contents that read_table_of_contents() walks (it dereferences
+	 * each entry before range-checking its offset).
+	 */
+	if (get_be32(p) != DIFF_HUNKS_SIGNATURE ||
+	    p[4] != DIFF_HUNKS_VERSION ||
+	    p[5] != oid_version(repo_algo) ||
+	    data_len < store_min_size(repo_algo, num_chunks)) {
+		munmap(data, data_len);
+		return NULL;
+	}
+
+	/*
+	 * The trailing checksum is not verified here: the writer fsyncs
+	 * and commits atomically, so a committed file is intact, and
+	 * every record is bounds-checked at read (see precomputed_entry_at).
+	 * The checksum is checked separately, by diff_hunks_verify().
+	 */
+
+	CALLOC_ARRAY(s, 1);
+	s->data = data;
+	s->data_len = data_len;
+	s->hash_algo = repo_algo;
+
+	cf = init_chunkfile(NULL);
+	if (read_table_of_contents_quiet(cf, p, data_len,
+					 DIFF_HUNKS_HEADER_SIZE, num_chunks, 1,
+					 repo_algo) ||
+	    pair_chunk(cf, DIFF_HUNKS_CHUNKID_INDEX, &s->index, &index_size) ||
+	    pair_chunk(cf, DIFF_HUNKS_CHUNKID_DATA, &s->hdat, &s->hdat_size)) {
+		free_chunkfile(cf);
+		goto corrupt;
+	}
+	free_chunkfile(cf);
+
+	entry_size = store_index_entry_size(s->hash_algo);
+	if (index_size % entry_size)
+		goto corrupt;
+	s->num_entries = index_size / entry_size;
+	return s;
+
+corrupt:
+	free_store(s);
+	return NULL;
+}
+
+static struct diff_hunks_store *diff_hunks_store_load(struct repository *r)
+{
+	struct diff_hunks_store *s;
+	char *fname;
+
+	prepare_repo_settings(r);
+	if (!r->settings.core_diff_hunks)
+		return NULL;
+
+	fname = diff_hunks_store_path(r);
+	s = load_store_at(r->hash_algo, fname);
+	free(fname);
+	return s;
+}
+
+struct diff_hunks_store *repo_diff_hunks_store(struct repository *r)
+{
+	if (!r->objects)
+		return NULL;
+	if (r->objects->diff_hunks_store_attempted)
+		return r->objects->diff_hunks_store;
+	r->objects->diff_hunks_store_attempted = 1;
+	r->objects->diff_hunks_store = diff_hunks_store_load(r);
+	return r->objects->diff_hunks_store;
+}
+
+void close_diff_hunks_store(struct object_database *o)
+{
+	if (!o->diff_hunks_store)
+		return;
+	free_store(o->diff_hunks_store);
+	o->diff_hunks_store = NULL;
+}
+
+/*
+ * Fill *out with the hunk record at offset in the data chunk, and return
+ * 1 if the record is in bounds, 0 otherwise. The read path does not
+ * re-verify the checksum, and a valid checksum would not bound the count
+ * anyway, so a read must call this and use *out only when it returns
+ * non-zero.
+ *
+ * A record is a be32 hunk count followed by that many DIFF_HUNKS_HUNK_SIZE
+ * hunks. "remaining" tracks the bytes from offset to the end of the data
+ * chunk: it must hold the count, and after the count is consumed it must
+ * hold every hunk. The bounds are written as subtraction and division
+ * (never addition or multiplication) so a crafted offset or count cannot
+ * overflow them.
+ */
+static int precomputed_entry_at(const struct diff_hunks_store *s,
+				uint32_t offset, struct precomputed_entry *out)
+{
+	size_t remaining;
+	uint32_t num_hunks;
+
+	if (offset >= s->hdat_size)
+		return 0;
+	remaining = s->hdat_size - offset;
+	if (remaining < sizeof(uint32_t))
+		return 0;
+
+	num_hunks = get_be32(s->hdat + offset);
+	remaining -= sizeof(uint32_t);
+	if (num_hunks > remaining / DIFF_HUNKS_HUNK_SIZE)
+		return 0;
+
+	out->num_hunks = num_hunks;
+	out->hunk_data = s->hdat + offset + sizeof(uint32_t);
+	return 1;
+}
+
+struct lookup_key {
+	const struct object_id *old_oid;
+	const struct object_id *new_oid;
+	int xdl_opts;
+	unsigned int rawsz;
+};
+
+/*
+ * The store's total order over (old_oid, new_oid, xdl_opts), defined
+ * once so the write-side sort (writer_entry_cmp) and the read-side
+ * search (store_bsearch_cmp) order the keys identically.
+ */
+static int cmp_store_index_key(const unsigned char *old_a, const unsigned char *new_a,
+			 uint32_t opts_a,
+			 const unsigned char *old_b, const unsigned char *new_b,
+			 uint32_t opts_b, unsigned int rawsz)
+{
+	int cmp = memcmp(old_a, old_b, rawsz);
+	if (!cmp)
+		cmp = memcmp(new_a, new_b, rawsz);
+	if (!cmp)
+		cmp = (opts_a > opts_b) - (opts_a < opts_b);
+	return cmp;
+}
+
+static int store_bsearch_cmp(const void *key, const void *entry_ptr)
+{
+	const struct lookup_key *k = key;
+	const unsigned char *old_hash, *new_hash;
+	uint32_t xdl_opts;
+
+	decode_store_index_key(entry_ptr, k->rawsz, &old_hash, &new_hash,
+			 &xdl_opts);
+	return cmp_store_index_key(k->old_oid->hash, k->new_oid->hash,
+			     (uint32_t)k->xdl_opts,
+			     old_hash, new_hash, xdl_opts, k->rawsz);
+}
+
+static int store_get_one(struct diff_hunks_store *s, const struct lookup_key *key,
+			 struct precomputed_entry *out)
+{
+	size_t entry_size = store_index_entry_size(s->hash_algo);
+	const unsigned char *found;
+
+	found = bsearch(key, s->index, s->num_entries, entry_size,
+			store_bsearch_cmp);
+	if (!found)
+		return 0;
+	return precomputed_entry_at(s,
+				    index_entry_hdat_offset(found, store_index_key_size(s->hash_algo)),
+				    out);
+}
+
+static int diff_hunks_store_get(struct diff_hunks_store *s,
+			 const struct object_id *old_oid,
+			 const struct object_id *new_oid,
+			 int xdl_opts,
+			 struct precomputed_entry *out)
+{
+	struct lookup_key key;
+
+	if (!s)
+		return 0;
+	/* The null OID names no blob and cannot key an entry. */
+	if (is_null_oid(old_oid) || is_null_oid(new_oid))
+		return 0;
+
+	key.old_oid = old_oid;
+	key.new_oid = new_oid;
+	key.xdl_opts = xdl_opts;
+	key.rawsz = s->hash_algo->rawsz;
+
+	return store_get_one(s, &key, out);
+}
+
+/*
+ * A recorded hunk sequence must satisfy the provider interface's
+ * shared check (diff_provider_check_hunk()) before it may be replayed:
+ * coordinates decode from be32 into long, which is 32-bit on some
+ * platforms, so a crafted value can decode negative or out of order.
+ * An entry that fails reads as a miss, so the caller recomputes.
+ */
+static int replayable_hunks(const struct precomputed_entry *e)
+{
+	struct diff_provider_hunks_check c = { 0 };
+	uint32_t i;
+
+	/*
+	 * Replaying a record with no hunks would assert the blob pair
+	 * equivalent, a claim the store must never make (the writer
+	 * refuses to record one), so such a record is invalid.
+	 */
+	if (!e->num_hunks)
+		return 0;
+	for (i = 0; i < e->num_hunks; i++) {
+		struct precomputed_hunk h;
+		nth_precomputed_hunk(e, i, &h);
+		if (diff_provider_check_hunk(&c, h.old_start, h.old_count,
+					      h.new_start, h.new_count))
+			return 0;
+	}
+	return 1;
+}
+
+int diff_hunks_replay(struct diff_hunks_store *s,
+		      const struct object_id *old_oid,
+		      const struct object_id *new_oid,
+		      int xdl_opts,
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data)
+{
+	struct precomputed_entry e;
+	uint32_t i;
+
+	if (!diff_hunks_store_get(s, old_oid, new_oid, xdl_opts, &e) ||
+	    !replayable_hunks(&e))
+		return 0;
+	for (i = 0; i < e.num_hunks; i++) {
+		struct precomputed_hunk h;
+		nth_precomputed_hunk(&e, i, &h);
+		hunk_func(h.old_start, h.old_count,
+			  h.new_start, h.new_count, cb_data);
+	}
+	return 1;
+}
+
+/* Validate one store file. Returns 0 if valid or absent, -1 on any error. */
+static int verify_store_at(struct repository *r, const char *fname)
+{
+	struct diff_hunks_store *s;
+	size_t entry_size;
+	uint32_t i;
+	int fd;
+	int ret = 0;
+
+	/*
+	 * A file that cannot be opened is not evidence of corruption:
+	 * report the open error, and reserve the corruption diagnostics
+	 * below for a file that was read and failed to parse.
+	 */
+	fd = git_open(fname);
+	if (fd < 0) {
+		if (errno == ENOENT)
+			return 0; /* absent is valid */
+		return error_errno(_("unable to open diff-hunks store %s"),
+				   fname);
+	}
+	close(fd);
+	s = load_store_at(r->hash_algo, fname);
+	if (!s)
+		return error(_("diff-hunks store failed to load (corrupt "
+			       "header or hash mismatch): %s"), fname);
+	if (!hashfile_checksum_valid(r->hash_algo, s->data, s->data_len)) {
+		error(_("diff-hunks store has incorrect checksum and is "
+			"likely corrupt: %s"), fname);
+		free_store(s);
+		return -1;
+	}
+
+	entry_size = store_index_entry_size(s->hash_algo);
+	for (i = 0; i < s->num_entries; i++) {
+		const unsigned char *ep = s->index + st_mult(entry_size, i);
+		size_t keysz = store_index_key_size(s->hash_algo);
+		uint32_t offset = index_entry_hdat_offset(ep, keysz);
+		struct precomputed_entry pe;
+
+		/*
+		 * Keyed by (old_oid, new_oid, xdl_opts), increasing.  memcmp
+		 * matches cmp_store_index_key's integer comparison of
+		 * xdl_opts because it is non-negative, so its big-endian
+		 * bytes order the same as its value.
+		 */
+		if (i > 0 && memcmp(ep - entry_size, ep, keysz) >= 0) {
+			error(_("diff-hunks entry %u not in sorted order"), i);
+			ret = -1;
+		}
+		if (!precomputed_entry_at(s, offset, &pe)) {
+			error(_("diff-hunks entry %u has out-of-bounds hunk "
+				"data"), i);
+			ret = -1;
+		} else if (!replayable_hunks(&pe)) {
+			error(_("diff-hunks entry %u holds an invalid hunk "
+				"sequence"), i);
+			ret = -1;
+		}
+	}
+
+	free_store(s);
+	return ret;
+}
+
+int diff_hunks_verify(struct repository *r)
+{
+	char *fname = diff_hunks_store_path(r);
+	int ret = 0;
+
+	if (verify_store_at(r, fname))
+		ret = -1;
+	free(fname);
+	return ret;
+}
+
+int diff_hunks_clear(struct repository *r)
+{
+	char *fname = diff_hunks_store_path(r);
+	int ret = 0;
+
+	if (unlink(fname) && errno != ENOENT)
+		ret = error_errno(_("unable to remove %s"), fname);
+	free(fname);
+	return ret;
+}
+
+struct writer_entry {
+	struct object_id old_oid;
+	struct object_id new_oid;
+	int xdl_opts;
+	uint32_t hdat_offset;
+};
+
+struct diff_hunks_writer {
+	struct repository *r;
+	struct writer_entry *entries;
+	size_t nr, alloc;
+	size_t seed_nr;		/* nr after seeding; finish skips a no-op flush */
+	unsigned force_flush : 1;	/* seed pruned: rewrite even a no-op warm */
+	struct strbuf hdat;
+	struct hashmap dedup;	/* hunk block content -> offset in hdat */
+};
+
+/* A record of one distinct hunk block already present in hdat. */
+struct dedup_entry {
+	struct hashmap_entry ent;
+	uint32_t offset;
+	uint32_t len;
+};
+
+static int dedup_cmp(const void *cmp_data,
+		     const struct hashmap_entry *a,
+		     const struct hashmap_entry *b,
+		     const void *keydata UNUSED)
+{
+	const struct diff_hunks_writer *writer = cmp_data;
+	const struct dedup_entry *ea = container_of(a, const struct dedup_entry, ent);
+	const struct dedup_entry *eb = container_of(b, const struct dedup_entry, ent);
+
+	if (ea->len != eb->len)
+		return 1;
+	return memcmp(writer->hdat.buf + ea->offset,
+		      writer->hdat.buf + eb->offset, ea->len);
+}
+
+static struct diff_hunks_writer *diff_hunks_writer_new(struct repository *r)
+{
+	struct diff_hunks_writer *w;
+
+	CALLOC_ARRAY(w, 1);
+	w->r = r;
+	strbuf_init(&w->hdat, 0);
+	hashmap_init(&w->dedup, dedup_cmp, w, 0);
+	return w;
+}
+
+static void strbuf_put_be32(struct strbuf *sb, uint32_t val)
+{
+	unsigned char buf[4];
+	put_be32(buf, val);
+	strbuf_add(sb, buf, 4);
+}
+
+/*
+ * The hunk block just appended at `start` is deduplicated: if an
+ * identical block is already in hdat, this copy is dropped and the
+ * earlier offset returned; otherwise it is kept and remembered.
+ * Distinct keys that diff to the same hunks then share one block.
+ */
+static uint32_t intern_block(struct diff_hunks_writer *w, size_t start)
+{
+	size_t len = w->hdat.len - start;
+	struct dedup_entry key, *found, *added;
+
+	hashmap_entry_init(&key.ent, memhash(w->hdat.buf + start, len));
+	key.offset = (uint32_t)start;
+	key.len = (uint32_t)len;
+
+	found = hashmap_get_entry(&w->dedup, &key, ent, NULL);
+	if (found) {
+		strbuf_setlen(&w->hdat, start);
+		return found->offset;
+	}
+
+	added = xmalloc(sizeof(*added));
+	hashmap_entry_init(&added->ent, key.ent.hash);
+	added->offset = key.offset;
+	added->len = key.len;
+	hashmap_add(&w->dedup, &added->ent);
+	return key.offset;
+}
+
+int diff_hunks_writer_add(struct diff_hunks_writer *w,
+			  const struct object_id *old_oid,
+			  const struct object_id *new_oid,
+			  int xdl_opts,
+			  const struct precomputed_hunk *hunks,
+			  size_t nr_hunks)
+{
+	struct writer_entry *e;
+	size_t i, block_start;
+
+	if (!w)
+		return 0;
+	/*
+	 * The block appended for this entry is sizeof(uint32_t) +
+	 * nr_hunks * DIFF_HUNKS_HUNK_SIZE bytes. Bound nr_hunks so that
+	 * length fits the uint32_t the dedup index records (and so the
+	 * count itself fits the uint32_t written to the store).
+	 */
+	if (!nr_hunks ||
+	    nr_hunks > (UINT32_MAX - sizeof(uint32_t)) / DIFF_HUNKS_HUNK_SIZE ||
+	    is_null_oid(old_oid) || is_null_oid(new_oid))
+		return 0;
+	if (w->hdat.len > UINT32_MAX)
+		return 0;
+	/*
+	 * Coordinates are stored as 32-bit values; a result that cannot
+	 * round-trip is dropped rather than silently truncated.
+	 */
+	for (i = 0; i < nr_hunks; i++)
+		if ((uintmax_t)hunks[i].old_start > (uintmax_t)INT32_MAX ||
+		    (uintmax_t)hunks[i].old_count > (uintmax_t)INT32_MAX ||
+		    (uintmax_t)hunks[i].new_start > (uintmax_t)INT32_MAX ||
+		    (uintmax_t)hunks[i].new_count > (uintmax_t)INT32_MAX)
+			return 0;
+
+	ALLOC_GROW(w->entries, w->nr + 1, w->alloc);
+	e = &w->entries[w->nr++];
+	oidcpy(&e->old_oid, old_oid);
+	oidcpy(&e->new_oid, new_oid);
+	e->xdl_opts = xdl_opts;
+
+	block_start = w->hdat.len;
+	strbuf_put_be32(&w->hdat, (uint32_t)nr_hunks);
+	for (i = 0; i < nr_hunks; i++) {
+		strbuf_put_be32(&w->hdat, hunks[i].old_start);
+		strbuf_put_be32(&w->hdat, hunks[i].old_count);
+		strbuf_put_be32(&w->hdat, hunks[i].new_start);
+		strbuf_put_be32(&w->hdat, hunks[i].new_count);
+	}
+	e->hdat_offset = intern_block(w, block_start);
+	return 1;
+}
+
+/*
+ * Seed the writer with fname's entries so a rewrite preserves them,
+ * setting *pruned when the rewrite will not carry the whole file
+ * forward: the file failed its checksum and was discarded outright, or
+ * individual entries were dropped because they failed the replayable
+ * check or the writer refused them (a key naming no blob).  A
+ * rewrite re-checksums, so corruption must not be carried forward:
+ * that would launder it into a checksum-valid file that verify can no
+ * longer catch.  This path already reads the whole file, so verify the
+ * checksum here (the reader keeps trusting committed files, without
+ * re-checksumming); an invalid
+ * entry reads as a miss anyway, so dropping it heals the store rather
+ * than losing anything a reader could use.
+ */
+static void diff_hunks_writer_seed(struct diff_hunks_writer *w,
+				   const char *fname, int *pruned)
+{
+	struct diff_hunks_store *s = load_store_at(w->r->hash_algo, fname);
+	unsigned int rawsz;
+	size_t entry_size, keysz;
+	struct precomputed_hunk *hunks = NULL;
+	size_t hunks_alloc = 0;
+	uint32_t i, dropped = 0;
+
+	if (!s)
+		return;
+	if (!hashfile_checksum_valid(w->r->hash_algo, s->data, s->data_len)) {
+		warning(_("diff-hunks store %s failed its checksum; "
+			  "discarding it"), fname);
+		free_store(s);
+		*pruned = 1;
+		return;
+	}
+	rawsz = s->hash_algo->rawsz;
+	entry_size = store_index_entry_size(s->hash_algo);
+	keysz = store_index_key_size(s->hash_algo);
+
+	for (i = 0; i < s->num_entries; i++) {
+		const unsigned char *ep = s->index + st_mult(entry_size, i);
+		const unsigned char *old_hash, *new_hash;
+		struct object_id old_oid, new_oid;
+		uint32_t xdl_opts, j;
+		struct precomputed_entry pe;
+
+		decode_store_index_key(ep, rawsz, &old_hash, &new_hash,
+				 &xdl_opts);
+		oidread(&old_oid, old_hash, s->hash_algo);
+		oidread(&new_oid, new_hash, s->hash_algo);
+		if (!precomputed_entry_at(s, index_entry_hdat_offset(ep, keysz), &pe) ||
+		    !replayable_hunks(&pe)) {
+			dropped++;
+			continue;
+		}
+		ALLOC_GROW(hunks, pe.num_hunks, hunks_alloc);
+		for (j = 0; j < pe.num_hunks; j++)
+			nth_precomputed_hunk(&pe, j, &hunks[j]);
+		if (!diff_hunks_writer_add(w, &old_oid, &new_oid,
+					   (int)xdl_opts, hunks, pe.num_hunks))
+			dropped++;
+	}
+	if (dropped) {
+		warning(Q_("diff-hunks store %s: dropping %u invalid entry",
+			   "diff-hunks store %s: dropping %u invalid entries",
+			   dropped), fname, dropped);
+		*pruned = 1;
+	}
+	free(hunks);
+	free_store(s);
+}
+
+/*
+ * Writing is off by default. It is enabled per invocation by the
+ * GIT_DIFF_HUNKS_WRITE environment variable, or persistently by the
+ * diffHunks.write config, with the environment variable winning when
+ * set. Only a warming run (a diff or log the repository owner chooses
+ * to run with writing on) enables it, so ordinary reads never mutate
+ * the store.
+ */
+static int diff_hunks_write_enabled(struct repository *r)
+{
+	const char *env = getenv("GIT_DIFF_HUNKS_WRITE");
+	int val;
+
+	if (env) {
+		/*
+		 * This is a warming opt-in, so an unparseable value must not
+		 * abort an ordinary read command: treat it as disabled.
+		 */
+		val = git_parse_maybe_bool(env);
+		return val < 0 ? 0 : val;
+	}
+	if (!repo_config_get_bool(r, "diffhunks.write", &val))
+		return val;
+	return 0;
+}
+
+struct diff_hunks_writer *diff_hunks_writer_maybe_new(struct repository *r)
+{
+	struct diff_hunks_writer *w;
+	char *fname;
+	int pruned;
+
+	if (!diff_hunks_write_enabled(r))
+		return NULL;
+	/*
+	 * Seed from the existing store so a flush merges with it rather
+	 * than replacing it: a later warm adds newly computed pairs
+	 * without discarding what earlier warms recorded.
+	 */
+	w = diff_hunks_writer_new(r);
+	fname = diff_hunks_store_path(r);
+	pruned = 0;
+	diff_hunks_writer_seed(w, fname, &pruned);
+	free(fname);
+	w->seed_nr = w->nr;
+	/*
+	 * A pruning seed means the file on disk holds material the
+	 * rewrite must not preserve; flush even if this warm computes
+	 * nothing new, so the store on disk is repaired rather than
+	 * left serving what the seed refused.
+	 */
+	w->force_flush = !!pruned;
+	return w;
+}
+
+static int writer_entry_cmp(const void *va, const void *vb, void *ctx)
+{
+	const struct writer_entry *a = va, *b = vb;
+	unsigned int rawsz = *(const unsigned int *)ctx;
+	return cmp_store_index_key(a->old_oid.hash, a->new_oid.hash,
+			     (uint32_t)a->xdl_opts,
+			     b->old_oid.hash, b->new_oid.hash,
+			     (uint32_t)b->xdl_opts,
+			     rawsz);
+}
+
+struct write_ctx {
+	struct diff_hunks_writer *w;
+	unsigned int rawsz;
+};
+
+static int write_index_chunk(struct hashfile *f, void *data)
+{
+	struct write_ctx *ctx = data;
+	size_t i;
+
+	for (i = 0; i < ctx->w->nr; i++) {
+		hashwrite(f, ctx->w->entries[i].old_oid.hash, ctx->rawsz);
+		hashwrite(f, ctx->w->entries[i].new_oid.hash, ctx->rawsz);
+		hashwrite_be32(f, ctx->w->entries[i].xdl_opts);
+		hashwrite_be32(f, ctx->w->entries[i].hdat_offset);
+	}
+	return 0;
+}
+
+static int write_data_chunk(struct hashfile *f, void *data)
+{
+	struct write_ctx *ctx = data;
+	hashwrite(f, ctx->w->hdat.buf, ctx->w->hdat.len);
+	return 0;
+}
+
+/* Sort, dedup, and write the accumulated entries to the file at fname. */
+static int diff_hunks_writer_flush(struct diff_hunks_writer *w, char *fname)
+{
+	struct lock_file lk = LOCK_INIT;
+	struct hashfile *f;
+	struct chunkfile *cf;
+	unsigned int rawsz = w->r->hash_algo->rawsz;
+	struct write_ctx ctx = { w, rawsz };
+	size_t entry_size;
+
+	QSORT_S(w->entries, w->nr, writer_entry_cmp, &rawsz);
+
+	/*
+	 * The same blob pair recurs across history (reverts, cherry-
+	 * picks); identical keys carry identical hunks, so keep one of
+	 * each. The index must stay duplicate-free for binary search.
+	 */
+	if (w->nr > 1) {
+		size_t kept = 1, i;
+		for (i = 1; i < w->nr; i++)
+			if (writer_entry_cmp(&w->entries[kept - 1],
+					      &w->entries[i], &rawsz))
+				w->entries[kept++] = w->entries[i];
+		w->nr = kept;
+	}
+
+	if (safe_create_leading_directories(w->r, fname)) {
+		error(_("unable to create directory for %s"), fname);
+		return -1;
+	}
+	if (hold_lock_file_for_update(&lk, fname, 0) < 0) {
+		error_errno(_("unable to lock %s"), fname);
+		return -1;
+	}
+	adjust_shared_perm(w->r, get_lock_file_path(&lk));
+	f = hashfd(w->r->hash_algo, get_lock_file_fd(&lk),
+		   get_lock_file_path(&lk));
+
+	entry_size = store_index_entry_size(w->r->hash_algo);
+	cf = init_chunkfile(f);
+	add_chunk(cf, DIFF_HUNKS_CHUNKID_INDEX, w->nr * entry_size,
+		  write_index_chunk);
+	add_chunk(cf, DIFF_HUNKS_CHUNKID_DATA, w->hdat.len, write_data_chunk);
+
+	hashwrite_be32(f, DIFF_HUNKS_SIGNATURE);
+	hashwrite_u8(f, DIFF_HUNKS_VERSION);
+	hashwrite_u8(f, oid_version(w->r->hash_algo));
+	hashwrite_u8(f, get_num_chunks(cf));
+	hashwrite_u8(f, 0); /* reserved */
+
+	write_chunkfile(cf, &ctx);
+	free_chunkfile(cf);
+
+	/*
+	 * fsync per the user's configuration (like commit-graph and the
+	 * multi-pack-index), then commit atomically. Readers trust the
+	 * committed file rather than re-checksumming it; diff_hunks_verify()
+	 * checks the checksum separately.
+	 */
+	finalize_hashfile(f, NULL, FSYNC_COMPONENT_DIFF_HUNKS,
+			  CSUM_HASH_IN_STREAM | CSUM_FSYNC);
+	/*
+	 * This same process may hold the current store mmapped (a warm
+	 * that also reads); the commit below renames over it, which must
+	 * never land on a live mapping (Windows refuses it).  Close the
+	 * store and clear the load-attempted flag first, so the next
+	 * read loads the committed file.
+	 */
+	if (w->r->objects) {
+		close_diff_hunks_store(w->r->objects);
+		w->r->objects->diff_hunks_store_attempted = 0;
+	}
+	if (commit_lock_file(&lk)) {
+		error_errno(_("unable to write %s"), fname);
+		return -1;
+	}
+	return 0;
+}
+
+static void diff_hunks_writer_free(struct diff_hunks_writer *w)
+{
+	if (!w)
+		return;
+	hashmap_clear_and_free(&w->dedup, struct dedup_entry, ent);
+	free(w->entries);
+	strbuf_release(&w->hdat);
+	free(w);
+}
+
+void diff_hunks_writer_finish(struct diff_hunks_writer *w)
+{
+	if (!w)
+		return;
+	/* Skip the flush when the warm recorded nothing beyond its seed. */
+	if (w->nr != w->seed_nr || w->force_flush) {
+		char *fname = diff_hunks_store_path(w->r);
+		diff_hunks_writer_flush(w, fname);
+		free(fname);
+	}
+	diff_hunks_writer_free(w);
+}
diff --git a/diff-hunks.h b/diff-hunks.h
new file mode 100644
index 0000000000..ef9ee3f417
--- /dev/null
+++ b/diff-hunks.h
@@ -0,0 +1,117 @@
+#ifndef DIFF_HUNKS_H
+#define DIFF_HUNKS_H
+
+#include "hash.h"
+#include "xdiff-interface.h"	/* xdl_emit_hunk_consume_func_t */
+
+struct object_id;
+struct repository;
+struct object_database;
+
+/*
+ * A persistent store of precomputed diff hunk coordinates, at
+ * .git/objects/info/diff-hunks. Entries are keyed by the two blobs diffed
+ * and the xdl_opts they were diffed under, so a cached result is valid
+ * in any context that key recurs in, independent of path. The xdl_opts
+ * key component mirrors the (always non-negative) diff_options field it
+ * projects from, and is serialized and compared as a 4-byte big-endian
+ * integer.
+ *
+ * The hunks a pair produces are not unique. They vary with the xdiff
+ * algorithm and ignore flags (xdl_opts, part of the key), and with
+ * whether the diff was trimmed: a zero-context diff runs
+ * trim_common_tail, which can pick a different but equally valid set of
+ * hunks than an untrimmed diff. The store holds one entry per key, so a
+ * pair is recorded only when its trimmed and untrimmed diffs are
+ * identical (the recording caller checks); such an entry serves a
+ * consumer at any context. The rare pair where the two diffs differ is
+ * never recorded and is always computed.
+ *
+ * The store is a cache: ordinary commands read it and fall back to
+ * computing the diff when it is absent, stale, or corrupt. It is filled
+ * as a side effect of diff and log runs, but only when writing is
+ * enabled (such a write-enabled run is a warming run); writing is off
+ * by default, so an ordinary command reads the store without recording
+ * into it.
+ */
+
+/*
+ * A hunk's coordinates. The type is long to match the xdiff emit
+ * callback; the values are a diff's line numbers and counts, always
+ * within the int32 range the on-disk format stores (see
+ * diff_hunks_writer_add()).
+ */
+struct precomputed_hunk {
+	long old_start;
+	long old_count;
+	long new_start;
+	long new_count;
+};
+
+/*
+ * The repository's store, loaded once on first use and cached on the
+ * object database. Returns NULL when reading is disabled
+ * (core.diffHunks=false), the store is absent, or it fails to parse
+ * (wrong signature, version, or object hash, or a corrupt structure).
+ * The lookup functions below accept a NULL store and treat it as
+ * empty (every lookup misses), so callers need not check for NULL.
+ * The object database owns the store; callers must not free it.
+ */
+struct diff_hunks_store *repo_diff_hunks_store(struct repository *r);
+
+/* Free the repository's cached store, at object-database teardown. */
+void close_diff_hunks_store(struct object_database *o);
+
+/*
+ * Replay the recorded hunks of an (old blob, new blob) pair diffed
+ * under xdl_opts through hunk_func. The sequence is validated before
+ * any callback runs: on a hit (return 1) every hunk is emitted, on a
+ * miss (return 0: absent pair, xdl_opts mismatch, or an entry that
+ * fails validation) nothing is emitted, so a caller may accumulate
+ * directly into its result.
+ */
+int diff_hunks_replay(struct diff_hunks_store *s,
+		      const struct object_id *old_oid,
+		      const struct object_id *new_oid,
+		      int xdl_opts,
+		      xdl_emit_hunk_consume_func_t hunk_func, void *cb_data);
+
+/*
+ * A warming run's writer: it accumulates the hunks it computes in memory
+ * and flushes them to the store in one pass at finish.
+ */
+struct diff_hunks_writer;
+
+/*
+ * Return a writer for a warming run, or NULL when writing is disabled
+ * (the default). diff_hunks_writer_add() tolerates a NULL writer, so a
+ * caller may attach the result unconditionally. Pair with
+ * diff_hunks_writer_finish().
+ */
+struct diff_hunks_writer *diff_hunks_writer_maybe_new(struct repository *r);
+
+/*
+ * Record a blob pair's hunks as computed under xdl_opts; a later lookup
+ * with a matching key is served these hunks. The caller must have
+ * checked that the pair's trimmed and untrimmed diffs are identical
+ * (see the top of this file), so the entry answers at any context.
+ * NULL-safe. Returns 1 when the entry was recorded, 0 when the writer
+ * refused it (no hunks, a null object id, or values the on-disk
+ * 32-bit fields cannot hold).
+ */
+int diff_hunks_writer_add(struct diff_hunks_writer *w,
+			  const struct object_id *old_oid,
+			  const struct object_id *new_oid,
+			  int xdl_opts,
+			  const struct precomputed_hunk *hunks,
+			  size_t nr_hunks);
+
+/* Flush the accumulated entries to the store and free the writer. NULL-safe. */
+void diff_hunks_writer_finish(struct diff_hunks_writer *w);
+
+/* Remove the store file. Returns 0 (incl. absent) or -1. */
+int diff_hunks_clear(struct repository *r);
+/* Validate the store. Returns 0 if valid/absent, -1 if corrupt. */
+int diff_hunks_verify(struct repository *r);
+
+#endif /* DIFF_HUNKS_H */
diff --git a/environment.c b/environment.c
index c663113e8a..a0e6d0b9b3 100644
--- a/environment.c
+++ b/environment.c
@@ -239,6 +239,7 @@ static const struct fsync_component_name {
 	{ "pack", FSYNC_COMPONENT_PACK },
 	{ "pack-metadata", FSYNC_COMPONENT_PACK_METADATA },
 	{ "commit-graph", FSYNC_COMPONENT_COMMIT_GRAPH },
+	{ "diff-hunks", FSYNC_COMPONENT_DIFF_HUNKS },
 	{ "index", FSYNC_COMPONENT_INDEX },
 	{ "objects", FSYNC_COMPONENTS_OBJECTS },
 	{ "reference", FSYNC_COMPONENT_REFERENCE },
diff --git a/git.c b/git.c
index e5f1811b6b..cb149e4b4e 100644
--- a/git.c
+++ b/git.c
@@ -566,6 +566,7 @@ static struct cmd_struct commands[] = {
 	{ "diagnose", cmd_diagnose, RUN_SETUP_GENTLY },
 	{ "diff", cmd_diff, NO_PARSEOPT },
 	{ "diff-files", cmd_diff_files, RUN_SETUP | NEED_WORK_TREE | NO_PARSEOPT },
+	{ "diff-hunks", cmd_diff_hunks, RUN_SETUP },
 	{ "diff-index", cmd_diff_index, RUN_SETUP | NO_PARSEOPT },
 	{ "diff-pairs", cmd_diff_pairs, RUN_SETUP | NO_PARSEOPT },
 	{ "diff-tree", cmd_diff_tree, RUN_SETUP | NO_PARSEOPT },
diff --git a/meson.build b/meson.build
index 539a50f90e..391d9da93c 100644
--- a/meson.build
+++ b/meson.build
@@ -364,6 +364,7 @@ libgit_sources = [
   'diffcore-pickaxe.c',
   'diffcore-rename.c',
   'diffcore-rotate.c',
+  'diff-hunks.c',
   'dir-iterator.c',
   'dir.c',
   'editor.c',
@@ -633,6 +634,7 @@ builtin_sources = [
   'builtin/describe.c',
   'builtin/diagnose.c',
   'builtin/diff-files.c',
+  'builtin/diff-hunks.c',
   'builtin/diff-index.c',
   'builtin/diff-pairs.c',
   'builtin/diff-tree.c',
diff --git a/odb.c b/odb.c
index cf6e7938c0..b300cd522d 100644
--- a/odb.c
+++ b/odb.c
@@ -2,6 +2,7 @@
 #include "abspath.h"
 #include "commit-graph.h"
 #include "config.h"
+#include "diff-hunks.h"
 #include "dir.h"
 #include "environment.h"
 #include "gettext.h"
@@ -1033,6 +1034,7 @@ void odb_close(struct object_database *o)
 	for (source = o->sources; source; source = source->next)
 		odb_source_close(source);
 	close_commit_graph(o);
+	close_diff_hunks_store(o);
 }
 
 static void odb_free_sources(struct object_database *o)
diff --git a/odb.h b/odb.h
index 7995bed97b..949d55668f 100644
--- a/odb.h
+++ b/odb.h
@@ -8,6 +8,7 @@
 #include "thread-utils.h"
 
 struct cached_object_entry;
+struct diff_hunks_store;
 struct list_objects_filter_options;
 struct odb_source_inmemory;
 struct packed_git;
@@ -76,6 +77,9 @@ struct object_database {
 	struct commit_graph *commit_graph;
 	unsigned commit_graph_attempted : 1; /* if loading has been attempted */
 
+	struct diff_hunks_store *diff_hunks_store;
+	unsigned diff_hunks_store_attempted : 1; /* if loading has been attempted */
+
 	/*
 	 * This is meant to hold a *small* number of objects that you would
 	 * want odb_read_object() to be able to return, but yet you do not want
diff --git a/repo-settings.c b/repo-settings.c
index f3be3b8c5a..c3015356ba 100644
--- a/repo-settings.c
+++ b/repo-settings.c
@@ -77,6 +77,7 @@ void prepare_repo_settings(struct repository *r)
 	repo_cfg_bool(r, "pack.usesparse", &r->settings.pack_use_sparse, 1);
 	repo_cfg_bool(r, "pack.usepathwalk", &r->settings.pack_use_path_walk, 0);
 	repo_cfg_bool(r, "core.multipackindex", &r->settings.core_multi_pack_index, 1);
+	repo_cfg_bool(r, "core.diffhunks", &r->settings.core_diff_hunks, 1);
 	repo_cfg_bool(r, "index.sparse", &r->settings.sparse_index, 0);
 	repo_cfg_bool(r, "index.skiphash", &r->settings.index_skip_hash, r->settings.index_skip_hash);
 	repo_cfg_bool(r, "pack.readreverseindex", &r->settings.pack_read_reverse_index, 1);
diff --git a/repo-settings.h b/repo-settings.h
index e5253ead02..615a55cac4 100644
--- a/repo-settings.h
+++ b/repo-settings.h
@@ -22,6 +22,7 @@ struct repo_settings {
 	int core_commit_graph;
 	int commit_graph_generation_version;
 	int commit_graph_changed_paths_version;
+	int core_diff_hunks;
 	int gc_write_commit_graph;
 	int fetch_write_commit_graph;
 	int command_requires_full_index;
diff --git a/write-or-die.h b/write-or-die.h
index ff0408bd84..35ed324307 100644
--- a/write-or-die.h
+++ b/write-or-die.h
@@ -22,13 +22,15 @@ enum fsync_component {
 	FSYNC_COMPONENT_INDEX			= 1 << 4,
 	FSYNC_COMPONENT_REFERENCE		= 1 << 5,
 	FSYNC_COMPONENT_OBJECT_MAP		= 1 << 6,
+	FSYNC_COMPONENT_DIFF_HUNKS		= 1 << 7,
 };
 
 #define FSYNC_COMPONENTS_OBJECTS (FSYNC_COMPONENT_LOOSE_OBJECT | \
 				  FSYNC_COMPONENT_PACK)
 
 #define FSYNC_COMPONENTS_DERIVED_METADATA (FSYNC_COMPONENT_PACK_METADATA | \
-					   FSYNC_COMPONENT_COMMIT_GRAPH)
+					   FSYNC_COMPONENT_COMMIT_GRAPH | \
+					   FSYNC_COMPONENT_DIFF_HUNKS)
 
 #define FSYNC_COMPONENTS_DEFAULT ((FSYNC_COMPONENTS_OBJECTS | \
 				   FSYNC_COMPONENTS_DERIVED_METADATA) & \
@@ -46,7 +48,8 @@ enum fsync_component {
 			      FSYNC_COMPONENT_COMMIT_GRAPH | \
 			      FSYNC_COMPONENT_INDEX | \
 			      FSYNC_COMPONENT_REFERENCE | \
-			      FSYNC_COMPONENT_OBJECT_MAP)
+			      FSYNC_COMPONENT_OBJECT_MAP | \
+			      FSYNC_COMPONENT_DIFF_HUNKS)
 
 #ifndef FSYNC_COMPONENTS_PLATFORM_DEFAULT
 #define FSYNC_COMPONENTS_PLATFORM_DEFAULT FSYNC_COMPONENTS_DEFAULT
-- 
2.54.0

[RFC PATCH v7 04/10] diff: record precomputed hunks during stat output

[Michael Montalbo]

The diff-hunks store has a writer, but nothing fills it.  Teach
builtin_diffstat() to do so: on a warming run (a writer is attached), a
modified pair's stat is produced by collecting the pair's hunk
coordinates instead of emitting text, the counts are summed from those
hunks, and the pair is recorded.  A run without a writer is unchanged,
and nothing reads the store yet; the read side arrives next.

The store records one context-free entry per pair, and only for a
trim-stable pair: one whose zero-context trimmed diff (what blame will
read) and untrimmed diff (whose counts a nonzero-context stat matches)
are identical.  The warming path computes both and hands them to
diff_hunks_writer_record_stable(), new here, which records only when
they agree; a divergent pair is never recorded and every consumer
computes it.  The warming run displays the counts it shows a store-less
run: the trimmed ones, since xdi_diff trims at zero context, while the
untrimmed counts serve only the stability comparison.

Not everything the stat path computes may be recorded.
--ignore-blank-lines is part of the key, but it coalesces hunks
differently between the text-emitting and coordinate-callback paths, so
a recorded entry would not match a store-less run's --stat.  -I
patterns, --anchored, and break detection (-B) shape the diff outside
the key entirely; the guard for those three sits in this consumer for
now and moves into the store's own provider when it registers, next.  A
"log -L" range-scoped stat is not the whole-pair diff the key describes,
so it does not record.  Recording also requires both sides to be valid
regular files whose blobs the key can name: a working-tree side,
textconv output, or a gitlink has no usable id.

"git diff", "git log", "git show", and "git diff-tree" with the --stat,
--numstat, and --shortstat formats attach a writer when writing is
enabled and flush it when the traversal finishes, so a warming run such
as

        GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null

fills the cache as a side effect of the diff work the command already
does.  Writing is controlled by diffHunks.write and GIT_DIFF_HUNKS_WRITE.

Add the write half of t4220:

- ordinary commands never create the store, and creation is gated off
  by default, the environment overriding the config;
- a warming run builds a store that verifies, and a second refreshes it
  in place;
- a warming run displays parity at zero context on a trim-divergent
  pair, committed as a fixture (small synthetic pairs cannot diverge:
  minimal diffs add and delete equal counts, and trimming preserves
  that);
- binary and mode-only pairs do not break the writer;
- a corrupt store is discarded at seed;
- verify and clear run against the files a warming run builds.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 builtin/diff-tree.c        |   3 +
 builtin/diff.c             |  10 ++
 builtin/log.c              |   7 +
 diff-hunks.c               |  32 ++++
 diff-hunks.h               |  18 ++-
 diff.c                     | 218 +++++++++++++++++++++----
 diff.h                     |  17 ++
 t/meson.build              |   1 +
 t/t4220-diff-hunks.sh      | 184 +++++++++++++++++++++
 t/t4220/README             |  55 +++++++
 t/t4220/trim-divergent-new | 319 +++++++++++++++++++++++++++++++++++++
 t/t4220/trim-divergent-old | 316 ++++++++++++++++++++++++++++++++++++
 12 files changed, 1150 insertions(+), 30 deletions(-)
 create mode 100755 t/t4220-diff-hunks.sh
 create mode 100644 t/t4220/README
 create mode 100644 t/t4220/trim-divergent-new
 create mode 100644 t/t4220/trim-divergent-old

diff --git a/builtin/diff-tree.c b/builtin/diff-tree.c
index 8b8f8b54e4..296c6a137e 100644
--- a/builtin/diff-tree.c
+++ b/builtin/diff-tree.c
@@ -170,6 +170,8 @@ int cmd_diff_tree(int argc,
 
 	opt->diffopt.rotate_to_strict = 1;
 
+	diff_hunks_attach(&opt->diffopt);
+
 	/*
 	 * NOTE!  We expect "a..b" to expand to "^a b" but it is
 	 * perfectly valid for revision range parser to yield "b ^a",
@@ -234,5 +236,6 @@ int cmd_diff_tree(int argc,
 		diff_free(&opt->diffopt);
 	}
 
+	diff_hunks_detach(&opt->diffopt);
 	return diff_result_code(opt);
 }
diff --git a/builtin/diff.c b/builtin/diff.c
index 18b1083e98..a2ad63ac8c 100644
--- a/builtin/diff.c
+++ b/builtin/diff.c
@@ -568,6 +568,15 @@ int cmd_diff(int argc,
 		}
 	}
 
+	/*
+	 * The hunk store is keyed by blob pair, so any diff whose
+	 * file pairs carry known blob object IDs (tree-to-tree,
+	 * index-to-tree) can consult the same entries that
+	 * "git log --stat" and "git blame" use; pairs without known
+	 * blobs bypass it at lookup time.
+	 */
+	diff_hunks_attach(&rev.diffopt);
+
 	symdiff_prepare(&rev, &sdiff);
 	for (i = 0; i < rev.pending.nr; i++) {
 		struct object_array_entry *entry = &rev.pending.objects[i];
@@ -648,6 +657,7 @@ int cmd_diff(int argc,
 	result = diff_result_code(&rev);
 	if (1 < rev.diffopt.skip_stat_unmatch)
 		refresh_index_quietly();
+	diff_hunks_detach(&rev.diffopt);
 	release_revisions(&rev);
 	object_array_clear(&ent);
 	symdiff_release(&sdiff);
diff --git a/builtin/log.c b/builtin/log.c
index 350b35c556..6903bdd5ff 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -693,8 +693,11 @@ int cmd_show(int argc,
 	opt.tweak = show_setup_revisions_tweak;
 	cmd_log_init(argc, argv, prefix, &rev, &opt, &cfg);
 
+	diff_hunks_attach(&rev.diffopt);
+
 	if (!rev.no_walk) {
 		ret = cmd_log_walk(&rev);
+		diff_hunks_detach(&rev.diffopt);
 		release_revisions(&rev);
 		log_config_release(&cfg);
 		return ret;
@@ -767,6 +770,7 @@ int cmd_show(int argc,
 	}
 
 	rev.diffopt.no_free = 0;
+	diff_hunks_detach(&rev.diffopt);
 	diff_free(&rev.diffopt);
 	release_revisions(&rev);
 	log_config_release(&cfg);
@@ -846,8 +850,11 @@ int cmd_log(int argc,
 	opt.tweak = log_setup_revisions_tweak;
 	cmd_log_init(argc, argv, prefix, &rev, &opt, &cfg);
 
+	diff_hunks_attach(&rev.diffopt);
+
 	ret = cmd_log_walk(&rev);
 
+	diff_hunks_detach(&rev.diffopt);
 	release_revisions(&rev);
 	log_config_release(&cfg);
 	return ret;
diff --git a/diff-hunks.c b/diff-hunks.c
index eeaaa2466a..23cefe055b 100644
--- a/diff-hunks.c
+++ b/diff-hunks.c
@@ -651,6 +651,38 @@ int diff_hunks_writer_add(struct diff_hunks_writer *w,
 	return 1;
 }
 
+void diff_hunks_writer_record_stable(struct diff_hunks_writer *w,
+				     const struct object_id *old_oid,
+				     const struct object_id *new_oid,
+				     int xdl_opts,
+				     const struct precomputed_hunk *trimmed,
+				     size_t nr_trimmed,
+				     const struct precomputed_hunk *full,
+				     size_t nr_full)
+{
+	size_t i;
+
+	if (!w)
+		return;
+	/*
+	 * Record only a trim-stable pair, one whose trimmed and
+	 * untrimmed diffs are identical, so the single entry answers
+	 * any consumer at any context (see the top of this file).  A
+	 * pair where the two diffs differ is never recorded and every
+	 * consumer computes it.
+	 */
+	if (nr_trimmed != nr_full)
+		return;
+	for (i = 0; i < nr_trimmed; i++)
+		if (trimmed[i].old_start != full[i].old_start ||
+		    trimmed[i].old_count != full[i].old_count ||
+		    trimmed[i].new_start != full[i].new_start ||
+		    trimmed[i].new_count != full[i].new_count)
+			return;
+	diff_hunks_writer_add(w, old_oid, new_oid, xdl_opts,
+			      trimmed, nr_trimmed);
+}
+
 /*
  * Seed the writer with fname's entries so a rewrite preserves them,
  * setting *pruned when the rewrite will not carry the whole file
diff --git a/diff-hunks.h b/diff-hunks.h
index ef9ee3f417..89e56f4b1b 100644
--- a/diff-hunks.h
+++ b/diff-hunks.h
@@ -94,7 +94,8 @@ struct diff_hunks_writer *diff_hunks_writer_maybe_new(struct repository *r);
  * Record a blob pair's hunks as computed under xdl_opts; a later lookup
  * with a matching key is served these hunks. The caller must have
  * checked that the pair's trimmed and untrimmed diffs are identical
- * (see the top of this file), so the entry answers at any context.
+ * (see the top of this file), so the entry answers at any context;
+ * diff_hunks_writer_record_stable() below performs that check.
  * NULL-safe. Returns 1 when the entry was recorded, 0 when the writer
  * refused it (no hunks, a null object id, or values the on-disk
  * 32-bit fields cannot hold).
@@ -106,6 +107,21 @@ int diff_hunks_writer_add(struct diff_hunks_writer *w,
 			  const struct precomputed_hunk *hunks,
 			  size_t nr_hunks);
 
+/*
+ * Record the pair only if it is trim-stable: the recording caller
+ * hands over both the trimmed (xdi_diff) and untrimmed (xdl_diff)
+ * zero-context hunk sequences it computed, and the entry is added
+ * only when the two are identical. NULL-safe.
+ */
+void diff_hunks_writer_record_stable(struct diff_hunks_writer *w,
+				     const struct object_id *old_oid,
+				     const struct object_id *new_oid,
+				     int xdl_opts,
+				     const struct precomputed_hunk *trimmed,
+				     size_t nr_trimmed,
+				     const struct precomputed_hunk *full,
+				     size_t nr_full);
+
 /* Flush the accumulated entries to the store and free the writer. NULL-safe. */
 void diff_hunks_writer_finish(struct diff_hunks_writer *w);
 
diff --git a/diff.c b/diff.c
index 9ef4328afa..11ec88dc8c 100644
--- a/diff.c
+++ b/diff.c
@@ -16,6 +16,7 @@
 #include "revision.h"
 #include "quote.h"
 #include "diff.h"
+#include "diff-hunks.h"
 #include "diffcore.h"
 #include "delta.h"
 #include "hex.h"
@@ -2929,6 +2930,72 @@ static struct diffstat_file *diffstat_add(struct diffstat_t *diffstat,
 	return x;
 }
 
+struct diffstat_hunk_cb_data {
+	struct precomputed_hunk **h;
+	size_t *nr, *alloc;
+};
+
+/*
+ * Hunk callback that appends each hunk's coordinates to a growable
+ * array, so one xdiff pass can both sum a diffstat and record hunks for
+ * the store.
+ */
+static int diffstat_hunk_cb(long start_a, long count_a,
+			    long start_b, long count_b,
+			    void *cb_data)
+{
+	struct diffstat_hunk_cb_data *d = cb_data;
+
+	ALLOC_GROW(*d->h, *d->nr + 1, *d->alloc);
+	(*d->h)[*d->nr].old_start = start_a;
+	(*d->h)[*d->nr].old_count = count_a;
+	(*d->h)[*d->nr].new_start = start_b;
+	(*d->h)[*d->nr].new_count = count_b;
+	(*d->nr)++;
+	return 0;
+}
+
+/*
+ * Collect the hunks of the two files at zero context. diff_fn chooses
+ * whether trimming runs: xdi_diff applies trim_common_tail, yielding the
+ * zero-context hunks blame reads; xdl_diff does not, yielding the
+ * untrimmed hunks. Both run at zero context, so the untrimmed hunks are
+ * not grouped the way a nonzero context would group them; diffstat only
+ * sums their counts, which grouping does not change. Sets *ph (caller
+ * frees) and *ph_nr.
+ */
+typedef int (*xdiff_fn)(mmfile_t *, mmfile_t *, xpparam_t const *,
+			xdemitconf_t const *, xdemitcb_t *);
+static int collect_hunks(xdiff_fn diff_fn, mmfile_t *mf1, mmfile_t *mf2,
+			       xpparam_t *xpp, struct precomputed_hunk **ph,
+			       size_t *ph_nr)
+{
+	size_t ph_alloc = 0;
+	xdemitcb_t ecb = { 0 };
+	xdemitconf_t xecfg = { 0 };
+	struct diffstat_hunk_cb_data cd = { ph, ph_nr, &ph_alloc };
+
+	*ph = NULL;
+	*ph_nr = 0;
+	xecfg.hunk_func = diffstat_hunk_cb;
+	ecb.priv = &cd;
+	return diff_fn(mf1, mf2, xpp, &xecfg, &ecb);
+}
+
+void diff_hunks_attach(struct diff_options *o)
+{
+	if (!(o->output_format &
+	      (DIFF_FORMAT_DIFFSTAT | DIFF_FORMAT_SHORTSTAT | DIFF_FORMAT_NUMSTAT)))
+		return;
+	o->hunks_writer = diff_hunks_writer_maybe_new(o->repo);
+}
+
+void diff_hunks_detach(struct diff_options *o)
+{
+	diff_hunks_writer_finish(o->hunks_writer);
+	o->hunks_writer = NULL;
+}
+
 static int diffstat_consume(void *priv, char *line, unsigned long len)
 {
 	struct diffstat_t *diffstat = priv;
@@ -4253,6 +4320,87 @@ static const char *get_compact_summary(const struct diff_filepair *p, int is_ren
 	return NULL;
 }
 
+/*
+ * Fill data->added/deleted for a modified pair by collecting its hunk
+ * coordinates, and record them into the store. Runs only on a warming
+ * run; returns 1 when it produced the counts, 0 when the caller must
+ * compute the diffstat itself.
+ *
+ * --ignore-blank-lines is excluded: that flag is part of the store
+ * key, but it coalesces hunks differently between the emit and
+ * hunk-callback paths, so a recorded entry would not match a
+ * store-less run's --stat output. (--inter-hunk-context is not
+ * excluded: it only groups hunks, and diffstat sums their counts,
+ * which grouping does not change.) Recording requires both sides to
+ * be valid regular files whose blobs the key can name.
+ */
+static int diffstat_from_hunks(struct diff_options *o,
+			       struct diff_filespec *one,
+			       struct diff_filespec *two,
+			       struct diffstat_file *data)
+{
+	struct precomputed_hunk *ph_trim, *ph_full, *counts;
+	size_t n_trim, n_full, n_counts, k;
+	mmfile_t mf1, mf2;
+	xpparam_t xpp = { .flags = o->xdl_opts,
+			  .ignore_regex = o->ignore_regex,
+			  .ignore_regex_nr = o->ignore_regex_nr,
+			  .anchors = o->anchors,
+			  .anchors_nr = o->anchors_nr };
+
+	if (o->xdl_opts & XDF_IGNORE_BLANK_LINES)
+		return 0;
+
+	/* Not a warming run: the caller computes the diffstat. */
+	if (!o->hunks_writer)
+		return 0;
+	/*
+	 * -I patterns, --anchored anchors, and break detection (-B)
+	 * shape the diff outside the store key, so what they compute
+	 * must not be recorded under it.
+	 */
+	if (o->ignore_regex_nr || o->anchors_nr || o->break_opt != -1)
+		return 0;
+	/* Recording needs blobs the key can name, on both sides. */
+	if (!one->oid_valid || !two->oid_valid ||
+	    S_ISGITLINK(one->mode) || S_ISGITLINK(two->mode) ||
+	    !DIFF_FILE_VALID(one) || !DIFF_FILE_VALID(two) ||
+	    !S_ISREG(one->mode) || !S_ISREG(two->mode))
+		return 0;
+
+	if (fill_mmfile(o->repo, &mf1, one) < 0 ||
+	    fill_mmfile(o->repo, &mf2, two) < 0)
+		die("unable to read files to diff");
+
+	/*
+	 * Compute the zero-context trimmed diff (what blame reads) and the
+	 * untrimmed diff (whose counts a nonzero-context stat matches).
+	 * xdi_diff runs first: it enforces the size limit, so the xdl_diff
+	 * call is already bounded.
+	 */
+	if (collect_hunks(xdi_diff, &mf1, &mf2, &xpp, &ph_trim, &n_trim) ||
+	    collect_hunks(xdl_diff, &mf1, &mf2, &xpp, &ph_full, &n_full))
+		die("unable to generate diffstat for %s", one->path);
+
+	/*
+	 * Match a store-less run: at zero context xdi_diff trims, so sum the
+	 * trimmed diff; otherwise sum the untrimmed one.
+	 */
+	counts = o->context ? ph_full : ph_trim;
+	n_counts = o->context ? n_full : n_trim;
+	for (k = 0; k < n_counts; k++) {
+		data->added += counts[k].new_count;
+		data->deleted += counts[k].old_count;
+	}
+
+	diff_hunks_writer_record_stable(o->hunks_writer, &one->oid, &two->oid,
+					o->xdl_opts, ph_trim, n_trim,
+					ph_full, n_full);
+	free(ph_trim);
+	free(ph_full);
+	return 1;
+}
+
 static void builtin_diffstat(const char *name_a, const char *name_b,
 			     struct diff_filespec *one,
 			     struct diff_filespec *two,
@@ -4304,38 +4452,50 @@ static void builtin_diffstat(const char *name_a, const char *name_b,
 	}
 
 	else if (may_differ) {
-		/* Crazy xdl interfaces.. */
-		xpparam_t xpp;
-		xdemitconf_t xecfg;
-
-		if (fill_mmfile(o->repo, &mf1, one) < 0 ||
-		    fill_mmfile(o->repo, &mf2, two) < 0)
-			die("unable to read files to diff");
-
-		memset(&xpp, 0, sizeof(xpp));
-		memset(&xecfg, 0, sizeof(xecfg));
-		xpp.flags = o->xdl_opts;
-		xpp.ignore_regex = o->ignore_regex;
-		xpp.ignore_regex_nr = o->ignore_regex_nr;
-		xpp.anchors = o->anchors;
-		xpp.anchors_nr = o->anchors_nr;
-		xecfg.ctxlen = o->context;
-		xecfg.interhunkctxlen = o->interhunkcontext;
-		xecfg.flags = XDL_EMIT_NO_HUNK_HDR;
-
-		if (p->line_ranges) {
-			struct line_range_filter lr_filter;
-
-			line_range_filter_init(&lr_filter, p->line_ranges,
-					       diffstat_consume, diffstat);
+		/*
+		 * Record into the diff-hunks store on a warming run. A
+		 * "log -L" range-scoped stat is not the whole-pair diff
+		 * the store keys, so it does not record. Otherwise diff
+		 * normally.
+		 */
+		if (p->line_ranges || !diffstat_from_hunks(o, one, two, data)) {
+			/* Crazy xdl interfaces.. */
+			xpparam_t xpp;
+			xdemitconf_t xecfg;
+
+			if (fill_mmfile(o->repo, &mf1, one) < 0 ||
+			    fill_mmfile(o->repo, &mf2, two) < 0)
+				die("unable to read files to diff");
+
+			memset(&xpp, 0, sizeof(xpp));
+			memset(&xecfg, 0, sizeof(xecfg));
+			xpp.flags = o->xdl_opts;
+			xpp.ignore_regex = o->ignore_regex;
+			xpp.ignore_regex_nr = o->ignore_regex_nr;
+			xpp.anchors = o->anchors;
+			xpp.anchors_nr = o->anchors_nr;
+			xecfg.ctxlen = o->context;
+			xecfg.interhunkctxlen = o->interhunkcontext;
+			xecfg.flags = XDL_EMIT_NO_HUNK_HDR;
 
-			if (line_range_filter_diff(&lr_filter, &mf1, &mf2,
-						   &xpp, &xecfg))
+			if (p->line_ranges) {
+				struct line_range_filter lr_filter;
+
+				line_range_filter_init(&lr_filter,
+						       p->line_ranges,
+						       diffstat_consume,
+						       diffstat);
+
+				if (line_range_filter_diff(&lr_filter, &mf1,
+							   &mf2, &xpp, &xecfg))
+					die("unable to generate diffstat for %s",
+					    one->path);
+			} else if (xdi_diff_outf(&mf1, &mf2, NULL,
+						 diffstat_consume, diffstat,
+						 &xpp, &xecfg))
 				die("unable to generate diffstat for %s",
 				    one->path);
-		} else if (xdi_diff_outf(&mf1, &mf2, NULL,
-				  diffstat_consume, diffstat, &xpp, &xecfg))
-			die("unable to generate diffstat for %s", one->path);
+		}
 
 		if (DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two)) {
 			struct diffstat_file *file =
diff --git a/diff.h b/diff.h
index bb5cddaf34..3d44de39ff 100644
--- a/diff.h
+++ b/diff.h
@@ -420,6 +420,13 @@ struct diff_options {
 	 */
 	int max_depth;
 	int max_depth_valid;
+
+	/*
+	 * Precomputed diff hunks (see diff-hunks.h). When hunks_writer is
+	 * set (a warming run), diffstat records the hunks it computes;
+	 * the writer is attached only for the stat output formats.
+	 */
+	struct diff_hunks_writer *hunks_writer;
 };
 
 unsigned diff_filter_bit(char status);
@@ -668,6 +675,16 @@ void diffcore_fix_diff_index(void);
 int diff_queue_is_empty(struct diff_options *o);
 void diff_flush(struct diff_options*);
 void diff_free(struct diff_options*);
+
+/*
+ * Attach a diff-hunks writer to a diff producing a stat format, so a
+ * warming run records the hunks it computes; a no-op when writing is off
+ * or for other formats. Pair with diff_hunks_detach() once the diff is
+ * done.
+ */
+void diff_hunks_attach(struct diff_options *o);
+void diff_hunks_detach(struct diff_options *o);
+
 void diff_warn_rename_limit(const char *varname, int needed, int degraded_cc);
 
 /* diff-raw status letters */
diff --git a/t/meson.build b/t/meson.build
index 8ae6ab6c5f..c63c30ba63 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -583,6 +583,7 @@ integration_tests = [
   't4216-log-bloom.sh',
   't4217-log-limit.sh',
   't4219-log-follow-merge.sh',
+  't4220-diff-hunks.sh',
   't4252-am-options.sh',
   't4253-am-keep-cr-dos.sh',
   't4254-am-corrupt.sh',
diff --git a/t/t4220-diff-hunks.sh b/t/t4220-diff-hunks.sh
new file mode 100755
index 0000000000..c677831946
--- /dev/null
+++ b/t/t4220-diff-hunks.sh
@@ -0,0 +1,184 @@
+#!/bin/sh
+
+test_description='precomputed diff hunks store (git diff-hunks)
+
+The store maps an (old blob, new blob, diff settings) key to the hunks of
+diffing the pair. It is a cache: reading is on by default
+(core.diffHunks), while writing is
+off by default and enabled per run by GIT_DIFF_HUNKS_WRITE (or the
+diffHunks.write config), so a diff or log warms the store only when the
+owner opts in. These tests check that a warmed store never changes
+output, that lookups honor the diff settings, and that a corrupt store is
+read as absent while verify reports the corruption.'
+
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+
+STORE=.git/objects/info/diff-hunks
+
+# Warm the store the way a repository owner would: a stat walk with
+# writing enabled. A --stat walk records one entry per trim-stable blob
+# pair, serving blame and the summary formats alike. Extra arguments
+# (e.g. -c options) are passed to git before "log".
+warm () {
+	GIT_DIFF_HUNKS_WRITE=1 git "$@" log --all --stat >/dev/null
+}
+
+# Run a command with the store disabled, for ground truth.
+no_store () {
+	git -c core.diffhunks=false "$@"
+}
+
+test_expect_success 'setup' '
+	test_commit initial file.txt "line 1" &&
+	test_commit second file.txt "line 1
+line 2" &&
+	test_commit third file.txt "line 1
+line 2
+line 3" &&
+	test_commit fourth file.txt "changed line 1
+line 2
+line 3
+line 4"
+'
+
+test_expect_success 'ordinary commands do not create the store' '
+	git log --stat >/dev/null &&
+	git blame file.txt >/dev/null &&
+	git diff --stat second third >/dev/null &&
+	test_path_is_missing $STORE
+'
+
+test_expect_success 'writing is gated by env and config, env wins' '
+	test_when_finished "git diff-hunks clear" &&
+	# The diffHunks.write config enables writing.
+	git -c diffHunks.write=true log --all --stat >/dev/null &&
+	test_path_is_file $STORE &&
+	git diff-hunks clear &&
+	# GIT_DIFF_HUNKS_WRITE overrides the config: 0 disables it.
+	GIT_DIFF_HUNKS_WRITE=0 git -c diffHunks.write=true log --all --stat >/dev/null &&
+	test_path_is_missing $STORE &&
+	# and enables it without any config.
+	GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null &&
+	test_path_is_file $STORE
+'
+
+test_expect_success 'a warm builds a store that verifies' '
+	warm &&
+	test_path_is_file $STORE &&
+	git diff-hunks verify
+'
+
+test_expect_success 'a second warming run refreshes the store in place' '
+	warm &&
+	test_commit fifth file.txt "brand new line" &&
+	warm &&
+	git diff-hunks verify &&
+	no_store log --stat >expect &&
+	git log --stat >actual &&
+	test_cmp expect actual
+'
+
+# A warming run displays the diffstat it computes. At zero context xdi_diff
+# trims, so the displayed counts must be the trimmed ones (what a store-less
+# run shows), not the untrimmed ones the writer compares against when it
+# decides whether the pair is stable enough to record.
+test_expect_success 'warming --stat at zero context matches a store-less run' '
+	git init -q warm-u0 &&
+	(
+		cd warm-u0 &&
+		cp "$TEST_DIRECTORY/t4220/trim-divergent-old" div.sh &&
+		git add div.sh && git commit -q -m old &&
+		cp "$TEST_DIRECTORY/t4220/trim-divergent-new" div.sh &&
+		git add div.sh && git commit -q -m new &&
+		git -c core.diffhunks=false log -1 --format= -U0 --stat -- div.sh >expect &&
+		GIT_DIFF_HUNKS_WRITE=1 git log -1 --format= -U0 --stat -- div.sh >got &&
+		test_cmp expect got
+	)
+'
+
+test_expect_success 'show and diff-tree --stat use the store' '
+	test_when_finished "git diff-hunks clear" &&
+	# diff_hunks_attach() runs for show and diff-tree: a write-enabled
+	# --stat records into the store (without the attach there is no
+	# writer, so nothing is written).
+	git diff-hunks clear &&
+	GIT_DIFF_HUNKS_WRITE=1 git show --stat fourth >/dev/null &&
+	test_path_is_file "$STORE" &&
+	git diff-hunks clear &&
+	GIT_DIFF_HUNKS_WRITE=1 git diff-tree --stat fourth >/dev/null &&
+	test_path_is_file "$STORE" &&
+	# Reading never changes their output.
+	git diff-hunks clear &&
+	no_store show --stat fourth >expect_show &&
+	no_store diff-tree --stat fourth >expect_dt &&
+	warm &&
+	git show --stat fourth >got_show &&
+	git diff-tree --stat fourth >got_dt &&
+	test_cmp expect_show got_show &&
+	test_cmp expect_dt got_dt
+'
+
+# Cover the pair shapes an object walk encounters: binary and
+# mode-only changes produce no text hunks to record.
+test_expect_success 'binary and mode-only changes do not break the writer' '
+	printf "\\000\\001\\002" >bin.dat &&
+	git add bin.dat &&
+	git commit -m binary-1 &&
+	printf "\\000\\001\\003\\004" >bin.dat &&
+	git add bin.dat &&
+	git commit -m binary-2 &&
+	echo "mode content" >mode.txt &&
+	git add mode.txt &&
+	git commit -m mode-1 &&
+	test_chmod +x mode.txt &&
+	git commit -m mode-2 &&
+	no_store log --stat >expect &&
+	warm &&
+	git log --stat >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'verify succeeds on a valid store and on an absent one' '
+	warm &&
+	git diff-hunks verify &&
+	git diff-hunks clear &&
+	test_path_is_missing $STORE &&
+	git diff-hunks verify
+'
+
+test_expect_success 'verify detects a checksum mismatch' '
+	test_when_finished "git diff-hunks clear" &&
+	warm &&
+	fsize=$(test_file_size $STORE) &&
+	mid=$((fsize / 2)) &&
+	printf "\\377" | dd of=$STORE bs=1 seek=$mid count=1 conv=notrunc 2>/dev/null &&
+	test_must_fail git diff-hunks verify
+'
+
+test_expect_success 'a warm discards a corrupt store rather than seeding from it' '
+	test_when_finished "git diff-hunks clear" &&
+	warm &&
+	# Corrupt the checksum: the next warm must not carry the corrupt
+	# entries forward into a fresh checksum-valid file; it discards
+	# them (with a warning) and rewrites a store that verifies.
+	fsize=$(test_file_size $STORE) &&
+	printf "\\377" | dd of=$STORE bs=1 seek=$((fsize / 2)) count=1 conv=notrunc 2>/dev/null &&
+	warm 2>err &&
+	test_grep "failed its checksum" err &&
+	git diff-hunks verify &&
+	no_store log --stat >expect &&
+	git log --stat >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'diff-hunks clear removes the store file' '
+	warm &&
+	test_path_is_file $STORE &&
+	git diff-hunks clear &&
+	test_path_is_missing $STORE
+'
+
+test_done
diff --git a/t/t4220/README b/t/t4220/README
new file mode 100644
index 0000000000..b850fe7c61
--- /dev/null
+++ b/t/t4220/README
@@ -0,0 +1,55 @@
+t4220 diff-hunks test fixtures
+==============================
+
+trim-divergent-old, trim-divergent-new
+--------------------------------------
+
+Two revisions of a single real file, used by t4220-diff-hunks.sh to
+exercise a "trim-divergent" blob pair: one whose diff hunk counts change
+with the amount of context, so the trimmed and untrimmed results
+disagree.
+
+They are two versions of git's own t/t6002-rev-list-bisect.sh, taken from
+git.git history around:
+
+  090af9957c ("t6002: fix use of `expr` with `set -e`",
+  Patrick Steinhardt, 2026-04-21)
+
+which rewrites `$(expr ...)` arithmetic as `$((...))` and reformats a few
+test_expect_success blocks.
+
+  trim-divergent-old = 090af9957c^:t/t6002-rev-list-bisect.sh  (blob daa009c9a1)
+  trim-divergent-new = 090af9957c :t/t6002-rev-list-bisect.sh  (blob f2de40b5ed)
+
+To regenerate them from any git.git checkout:
+
+  git show 090af9957c^:t/t6002-rev-list-bisect.sh >trim-divergent-old
+  git show 090af9957c:t/t6002-rev-list-bisect.sh  >trim-divergent-new
+
+Why this pair
+-------------
+
+The diff-hunks store records only "trim-stable" pairs: those whose hunks
+are identical whether or not xdiff trims the common head and tail (which
+it does at zero context, in trim_common_tail).  This pair is deliberately
+NOT trim-stable:
+
+  diff -U0 reports  9 added / 6 deleted
+  diff -U3 reports 10 added / 7 deleted
+
+Because the counts diverge with context, the writer must refuse to record
+this pair and every command must recompute it from the blobs.  t4220 uses
+it to prove that the displayed counts stay correct at each context, and
+that a divergent pair is never served from the store.  See
+t4220-diff-hunks.sh ("a trim-divergent file is correct at each context"
+and the store-poison test).
+
+Why not a synthesized fixture
+-----------------------------
+
+The divergence needs real content that makes xdiff's common-tail trimming
+shift a hunk boundary while the added/deleted balance stays equal.  A
+minimal hand-written file that reliably triggers the -U0 vs -U3 count
+disagreement has not been found yet; until one is, this real pair is kept
+verbatim.  If you synthesize a smaller equivalent, replace these two files
+and delete this note.
diff --git a/t/t4220/trim-divergent-new b/t/t4220/trim-divergent-new
new file mode 100644
index 0000000000..f2de40b5ed
--- /dev/null
+++ b/t/t4220/trim-divergent-new
@@ -0,0 +1,319 @@
+#!/bin/sh
+#
+# Copyright (c) 2005 Jon Seymour
+#
+test_description='Tests git rev-list --bisect functionality'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-t6000.sh # t6xxx specific functions
+
+# usage: test_bisection max-diff bisect-option head ^prune...
+#
+# e.g. test_bisection 1 --bisect l1 ^l0
+#
+test_bisection_diff()
+{
+	_max_diff=$1
+	_bisect_option=$2
+	shift 2
+	_bisection=$(git rev-list $_bisect_option "$@")
+	_list_size=$(git rev-list "$@" | wc -l)
+        _head=$1
+	shift 1
+	_bisection_size=$(git rev-list $_bisection "$@" | wc -l)
+	[ -n "$_list_size" -a -n "$_bisection_size" ] ||
+	error "test_bisection_diff failed"
+
+	# Test if bisection size is close to half of list size within
+	# tolerance.
+	#
+	_bisect_err=$(($_list_size - $_bisection_size * 2))
+	if test "$_bisect_err" -lt 0
+	then
+		_bisect_err=$((0 - $_bisect_err))
+	fi
+	_bisect_err=$(($_bisect_err / 2)) ; # floor
+
+	test_expect_success "bisection diff $_bisect_option $_head $* <= $_max_diff" '
+		test $_bisect_err -le $_max_diff
+	'
+}
+
+date >path0
+git update-index --add path0
+save_tag tree git write-tree
+on_committer_date "00:00" hide_error save_tag root unique_commit root tree
+on_committer_date "00:01" save_tag l0 unique_commit l0 tree -p root
+on_committer_date "00:02" save_tag l1 unique_commit l1 tree -p l0
+on_committer_date "00:03" save_tag l2 unique_commit l2 tree -p l1
+on_committer_date "00:04" save_tag a0 unique_commit a0 tree -p l2
+on_committer_date "00:05" save_tag a1 unique_commit a1 tree -p a0
+on_committer_date "00:06" save_tag b1 unique_commit b1 tree -p a0
+on_committer_date "00:07" save_tag c1 unique_commit c1 tree -p b1
+on_committer_date "00:08" save_tag b2 unique_commit b2 tree -p b1
+on_committer_date "00:09" save_tag b3 unique_commit b2 tree -p b2
+on_committer_date "00:10" save_tag c2 unique_commit c2 tree -p c1 -p b2
+on_committer_date "00:11" save_tag c3 unique_commit c3 tree -p c2
+on_committer_date "00:12" save_tag a2 unique_commit a2 tree -p a1
+on_committer_date "00:13" save_tag a3 unique_commit a3 tree -p a2
+on_committer_date "00:14" save_tag b4 unique_commit b4 tree -p b3 -p a3
+on_committer_date "00:15" save_tag a4 unique_commit a4 tree -p a3 -p b4 -p c3
+on_committer_date "00:16" save_tag l3 unique_commit l3 tree -p a4
+on_committer_date "00:17" save_tag l4 unique_commit l4 tree -p l3
+on_committer_date "00:18" save_tag l5 unique_commit l5 tree -p l4
+git update-ref HEAD $(tag l5)
+
+
+#     E
+#    / \
+#   e1  |
+#   |   |
+#   e2  |
+#   |   |
+#   e3  |
+#   |   |
+#   e4  |
+#   |   |
+#   |   f1
+#   |   |
+#   |   f2
+#   |   |
+#   |   f3
+#   |   |
+#   |   f4
+#   |   |
+#   e5  |
+#   |   |
+#   e6  |
+#   |   |
+#   e7  |
+#   |   |
+#   e8  |
+#    \ /
+#     F
+
+
+on_committer_date "00:00" hide_error save_tag F unique_commit F tree
+on_committer_date "00:01" save_tag e8 unique_commit e8 tree -p F
+on_committer_date "00:02" save_tag e7 unique_commit e7 tree -p e8
+on_committer_date "00:03" save_tag e6 unique_commit e6 tree -p e7
+on_committer_date "00:04" save_tag e5 unique_commit e5 tree -p e6
+on_committer_date "00:05" save_tag f4 unique_commit f4 tree -p F
+on_committer_date "00:06" save_tag f3 unique_commit f3 tree -p f4
+on_committer_date "00:07" save_tag f2 unique_commit f2 tree -p f3
+on_committer_date "00:08" save_tag f1 unique_commit f1 tree -p f2
+on_committer_date "00:09" save_tag e4 unique_commit e4 tree -p e5
+on_committer_date "00:10" save_tag e3 unique_commit e3 tree -p e4
+on_committer_date "00:11" save_tag e2 unique_commit e2 tree -p e3
+on_committer_date "00:12" save_tag e1 unique_commit e1 tree -p e2
+on_committer_date "00:13" save_tag E unique_commit E tree -p e1 -p f1
+
+on_committer_date "00:00" hide_error save_tag U unique_commit U tree
+on_committer_date "00:01" save_tag u0 unique_commit u0 tree -p U
+on_committer_date "00:01" save_tag u1 unique_commit u1 tree -p u0
+on_committer_date "00:02" save_tag u2 unique_commit u2 tree -p u0
+on_committer_date "00:03" save_tag u3 unique_commit u3 tree -p u0
+on_committer_date "00:04" save_tag u4 unique_commit u4 tree -p u0
+on_committer_date "00:05" save_tag u5 unique_commit u5 tree -p u0
+on_committer_date "00:06" save_tag V unique_commit V tree -p u1 -p u2 -p u3 -p u4 -p u5
+
+test_sequence()
+{
+	_bisect_option=$1
+
+	test_bisection_diff 0 $_bisect_option l0 ^root
+	test_bisection_diff 0 $_bisect_option l1 ^root
+	test_bisection_diff 0 $_bisect_option l2 ^root
+	test_bisection_diff 0 $_bisect_option a0 ^root
+	test_bisection_diff 0 $_bisect_option a1 ^root
+	test_bisection_diff 0 $_bisect_option a2 ^root
+	test_bisection_diff 0 $_bisect_option a3 ^root
+	test_bisection_diff 0 $_bisect_option b1 ^root
+	test_bisection_diff 0 $_bisect_option b2 ^root
+	test_bisection_diff 0 $_bisect_option b3 ^root
+	test_bisection_diff 0 $_bisect_option c1 ^root
+	test_bisection_diff 0 $_bisect_option c2 ^root
+	test_bisection_diff 0 $_bisect_option c3 ^root
+	test_bisection_diff 0 $_bisect_option E ^F
+	test_bisection_diff 0 $_bisect_option e1 ^F
+	test_bisection_diff 0 $_bisect_option e2 ^F
+	test_bisection_diff 0 $_bisect_option e3 ^F
+	test_bisection_diff 0 $_bisect_option e4 ^F
+	test_bisection_diff 0 $_bisect_option e5 ^F
+	test_bisection_diff 0 $_bisect_option e6 ^F
+	test_bisection_diff 0 $_bisect_option e7 ^F
+	test_bisection_diff 0 $_bisect_option f1 ^F
+	test_bisection_diff 0 $_bisect_option f2 ^F
+	test_bisection_diff 0 $_bisect_option f3 ^F
+	test_bisection_diff 0 $_bisect_option f4 ^F
+	test_bisection_diff 0 $_bisect_option E ^F
+
+	test_bisection_diff 1 $_bisect_option V ^U
+	test_bisection_diff 0 $_bisect_option V ^U ^u1 ^u2 ^u3
+	test_bisection_diff 0 $_bisect_option u1 ^U
+	test_bisection_diff 0 $_bisect_option u2 ^U
+	test_bisection_diff 0 $_bisect_option u3 ^U
+	test_bisection_diff 0 $_bisect_option u4 ^U
+	test_bisection_diff 0 $_bisect_option u5 ^U
+
+#
+# the following illustrates Linus' binary bug blatt idea.
+#
+# assume the bug is actually at l3, but you don't know that - all you know is that l3 is broken
+# and it wasn't broken before
+#
+# keep bisecting the list, advancing the "bad" head and accumulating "good" heads until
+# the bisection point is the head - this is the bad point.
+#
+
+test_output_expect_success "$_bisect_option l5 ^root" 'git rev-list $_bisect_option l5 ^root' <<EOF
+c3
+EOF
+
+test_output_expect_success "$_bisect_option l5 ^root ^c3" 'git rev-list $_bisect_option l5 ^root ^c3' <<EOF
+b4
+EOF
+
+test_output_expect_success "$_bisect_option l5 ^root ^c3 ^b4" 'git rev-list $_bisect_option l5 ^c3 ^b4' <<EOF
+l3
+EOF
+
+test_output_expect_success "$_bisect_option l3 ^root ^c3 ^b4" 'git rev-list $_bisect_option l3 ^root ^c3 ^b4' <<EOF
+a4
+EOF
+
+test_output_expect_success "$_bisect_option l5 ^b3 ^a3 ^b4 ^a4" 'git rev-list $_bisect_option l3 ^b3 ^a3 ^a4' <<EOF
+l3
+EOF
+
+#
+# if l3 is bad, then l4 is bad too - so advance the bad pointer by making b4 the known bad head
+#
+
+test_output_expect_success "$_bisect_option l4 ^a2 ^a3 ^b ^a4" 'git rev-list $_bisect_option l4 ^a2 ^a3 ^a4' <<EOF
+l3
+EOF
+
+test_output_expect_success "$_bisect_option l3 ^a2 ^a3 ^b ^a4" 'git rev-list $_bisect_option l3 ^a2 ^a3 ^a4' <<EOF
+l3
+EOF
+
+# found!
+
+#
+# as another example, let's consider a4 to be the bad head, in which case
+#
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4' <<EOF
+c2
+EOF
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4 ^c2" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4 ^c2' <<EOF
+c3
+EOF
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4 ^c2 ^c3" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4 ^c2 ^c3' <<EOF
+a4
+EOF
+
+# found!
+
+#
+# or consider c3 to be the bad head
+#
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4' <<EOF
+c2
+EOF
+
+test_output_expect_success "$_bisect_option c3 ^a2 ^a3 ^b4 ^c2" 'git rev-list $_bisect_option c3 ^a2 ^a3 ^b4 ^c2' <<EOF
+c3
+EOF
+
+# found!
+
+}
+
+test_sequence "--bisect"
+
+#
+#
+
+test_expect_success 'set up fake --bisect refs' '
+	git update-ref refs/bisect/bad c3 &&
+	good=$(git rev-parse b1) &&
+	git update-ref refs/bisect/good-$good $good &&
+	good=$(git rev-parse c1) &&
+	git update-ref refs/bisect/good-$good $good
+'
+
+test_expect_success 'rev-list --bisect can default to good/bad refs' '
+	# the only thing between c3 and c1 is c2
+	git rev-parse c2 >expect &&
+	git rev-list --bisect >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'rev-parse --bisect can default to good/bad refs' '
+	git rev-parse c3 ^b1 ^c1 >expect &&
+	git rev-parse --bisect >actual &&
+
+	# output order depends on the refnames, which in turn depends on
+	# the exact sha1s. We just want to make sure we have the same set
+	# of lines in any order.
+	sort <expect >expect.sorted &&
+	sort <actual >actual.sorted &&
+	test_cmp expect.sorted actual.sorted
+'
+
+test_output_expect_success '--bisect --first-parent' 'git rev-list --bisect --first-parent E ^F' <<EOF
+e4
+EOF
+
+test_output_expect_success '--first-parent' 'git rev-list --first-parent E ^F' <<EOF
+E
+e1
+e2
+e3
+e4
+e5
+e6
+e7
+e8
+EOF
+
+test_output_expect_success '--bisect-vars --first-parent' 'git rev-list --bisect-vars --first-parent E ^F' <<EOF
+bisect_rev='e5'
+bisect_nr=4
+bisect_good=4
+bisect_bad=3
+bisect_all=9
+bisect_steps=2
+EOF
+
+test_expect_success '--bisect-all --first-parent' '
+	cat >expect.unsorted <<-EOF &&
+	$(git rev-parse E) (tag: E, dist=0)
+	$(git rev-parse e1) (tag: e1, dist=1)
+	$(git rev-parse e2) (tag: e2, dist=2)
+	$(git rev-parse e3) (tag: e3, dist=3)
+	$(git rev-parse e4) (tag: e4, dist=4)
+	$(git rev-parse e5) (tag: e5, dist=4)
+	$(git rev-parse e6) (tag: e6, dist=3)
+	$(git rev-parse e7) (tag: e7, dist=2)
+	$(git rev-parse e8) (tag: e8, dist=1)
+	EOF
+
+	# expect results to be ordered by distance (descending),
+	# commit hash (ascending)
+	sort -k4,4r -k1,1 expect.unsorted >expect &&
+	git rev-list --bisect-all --first-parent E ^F >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success '--bisect without any revisions' '
+	git rev-list --bisect HEAD..HEAD >out &&
+	test_must_be_empty out
+'
+
+test_done
diff --git a/t/t4220/trim-divergent-old b/t/t4220/trim-divergent-old
new file mode 100644
index 0000000000..daa009c9a1
--- /dev/null
+++ b/t/t4220/trim-divergent-old
@@ -0,0 +1,316 @@
+#!/bin/sh
+#
+# Copyright (c) 2005 Jon Seymour
+#
+test_description='Tests git rev-list --bisect functionality'
+
+. ./test-lib.sh
+. "$TEST_DIRECTORY"/lib-t6000.sh # t6xxx specific functions
+
+# usage: test_bisection max-diff bisect-option head ^prune...
+#
+# e.g. test_bisection 1 --bisect l1 ^l0
+#
+test_bisection_diff()
+{
+	_max_diff=$1
+	_bisect_option=$2
+	shift 2
+	_bisection=$(git rev-list $_bisect_option "$@")
+	_list_size=$(git rev-list "$@" | wc -l)
+        _head=$1
+	shift 1
+	_bisection_size=$(git rev-list $_bisection "$@" | wc -l)
+	[ -n "$_list_size" -a -n "$_bisection_size" ] ||
+	error "test_bisection_diff failed"
+
+	# Test if bisection size is close to half of list size within
+	# tolerance.
+	#
+	_bisect_err=$(expr $_list_size - $_bisection_size \* 2)
+	test "$_bisect_err" -lt 0 && _bisect_err=$(expr 0 - $_bisect_err)
+	_bisect_err=$(expr $_bisect_err / 2) ; # floor
+
+	test_expect_success \
+	"bisection diff $_bisect_option $_head $* <= $_max_diff" \
+	'test $_bisect_err -le $_max_diff'
+}
+
+date >path0
+git update-index --add path0
+save_tag tree git write-tree
+on_committer_date "00:00" hide_error save_tag root unique_commit root tree
+on_committer_date "00:01" save_tag l0 unique_commit l0 tree -p root
+on_committer_date "00:02" save_tag l1 unique_commit l1 tree -p l0
+on_committer_date "00:03" save_tag l2 unique_commit l2 tree -p l1
+on_committer_date "00:04" save_tag a0 unique_commit a0 tree -p l2
+on_committer_date "00:05" save_tag a1 unique_commit a1 tree -p a0
+on_committer_date "00:06" save_tag b1 unique_commit b1 tree -p a0
+on_committer_date "00:07" save_tag c1 unique_commit c1 tree -p b1
+on_committer_date "00:08" save_tag b2 unique_commit b2 tree -p b1
+on_committer_date "00:09" save_tag b3 unique_commit b2 tree -p b2
+on_committer_date "00:10" save_tag c2 unique_commit c2 tree -p c1 -p b2
+on_committer_date "00:11" save_tag c3 unique_commit c3 tree -p c2
+on_committer_date "00:12" save_tag a2 unique_commit a2 tree -p a1
+on_committer_date "00:13" save_tag a3 unique_commit a3 tree -p a2
+on_committer_date "00:14" save_tag b4 unique_commit b4 tree -p b3 -p a3
+on_committer_date "00:15" save_tag a4 unique_commit a4 tree -p a3 -p b4 -p c3
+on_committer_date "00:16" save_tag l3 unique_commit l3 tree -p a4
+on_committer_date "00:17" save_tag l4 unique_commit l4 tree -p l3
+on_committer_date "00:18" save_tag l5 unique_commit l5 tree -p l4
+git update-ref HEAD $(tag l5)
+
+
+#     E
+#    / \
+#   e1  |
+#   |   |
+#   e2  |
+#   |   |
+#   e3  |
+#   |   |
+#   e4  |
+#   |   |
+#   |   f1
+#   |   |
+#   |   f2
+#   |   |
+#   |   f3
+#   |   |
+#   |   f4
+#   |   |
+#   e5  |
+#   |   |
+#   e6  |
+#   |   |
+#   e7  |
+#   |   |
+#   e8  |
+#    \ /
+#     F
+
+
+on_committer_date "00:00" hide_error save_tag F unique_commit F tree
+on_committer_date "00:01" save_tag e8 unique_commit e8 tree -p F
+on_committer_date "00:02" save_tag e7 unique_commit e7 tree -p e8
+on_committer_date "00:03" save_tag e6 unique_commit e6 tree -p e7
+on_committer_date "00:04" save_tag e5 unique_commit e5 tree -p e6
+on_committer_date "00:05" save_tag f4 unique_commit f4 tree -p F
+on_committer_date "00:06" save_tag f3 unique_commit f3 tree -p f4
+on_committer_date "00:07" save_tag f2 unique_commit f2 tree -p f3
+on_committer_date "00:08" save_tag f1 unique_commit f1 tree -p f2
+on_committer_date "00:09" save_tag e4 unique_commit e4 tree -p e5
+on_committer_date "00:10" save_tag e3 unique_commit e3 tree -p e4
+on_committer_date "00:11" save_tag e2 unique_commit e2 tree -p e3
+on_committer_date "00:12" save_tag e1 unique_commit e1 tree -p e2
+on_committer_date "00:13" save_tag E unique_commit E tree -p e1 -p f1
+
+on_committer_date "00:00" hide_error save_tag U unique_commit U tree
+on_committer_date "00:01" save_tag u0 unique_commit u0 tree -p U
+on_committer_date "00:01" save_tag u1 unique_commit u1 tree -p u0
+on_committer_date "00:02" save_tag u2 unique_commit u2 tree -p u0
+on_committer_date "00:03" save_tag u3 unique_commit u3 tree -p u0
+on_committer_date "00:04" save_tag u4 unique_commit u4 tree -p u0
+on_committer_date "00:05" save_tag u5 unique_commit u5 tree -p u0
+on_committer_date "00:06" save_tag V unique_commit V tree -p u1 -p u2 -p u3 -p u4 -p u5
+
+test_sequence()
+{
+	_bisect_option=$1
+
+	test_bisection_diff 0 $_bisect_option l0 ^root
+	test_bisection_diff 0 $_bisect_option l1 ^root
+	test_bisection_diff 0 $_bisect_option l2 ^root
+	test_bisection_diff 0 $_bisect_option a0 ^root
+	test_bisection_diff 0 $_bisect_option a1 ^root
+	test_bisection_diff 0 $_bisect_option a2 ^root
+	test_bisection_diff 0 $_bisect_option a3 ^root
+	test_bisection_diff 0 $_bisect_option b1 ^root
+	test_bisection_diff 0 $_bisect_option b2 ^root
+	test_bisection_diff 0 $_bisect_option b3 ^root
+	test_bisection_diff 0 $_bisect_option c1 ^root
+	test_bisection_diff 0 $_bisect_option c2 ^root
+	test_bisection_diff 0 $_bisect_option c3 ^root
+	test_bisection_diff 0 $_bisect_option E ^F
+	test_bisection_diff 0 $_bisect_option e1 ^F
+	test_bisection_diff 0 $_bisect_option e2 ^F
+	test_bisection_diff 0 $_bisect_option e3 ^F
+	test_bisection_diff 0 $_bisect_option e4 ^F
+	test_bisection_diff 0 $_bisect_option e5 ^F
+	test_bisection_diff 0 $_bisect_option e6 ^F
+	test_bisection_diff 0 $_bisect_option e7 ^F
+	test_bisection_diff 0 $_bisect_option f1 ^F
+	test_bisection_diff 0 $_bisect_option f2 ^F
+	test_bisection_diff 0 $_bisect_option f3 ^F
+	test_bisection_diff 0 $_bisect_option f4 ^F
+	test_bisection_diff 0 $_bisect_option E ^F
+
+	test_bisection_diff 1 $_bisect_option V ^U
+	test_bisection_diff 0 $_bisect_option V ^U ^u1 ^u2 ^u3
+	test_bisection_diff 0 $_bisect_option u1 ^U
+	test_bisection_diff 0 $_bisect_option u2 ^U
+	test_bisection_diff 0 $_bisect_option u3 ^U
+	test_bisection_diff 0 $_bisect_option u4 ^U
+	test_bisection_diff 0 $_bisect_option u5 ^U
+
+#
+# the following illustrates Linus' binary bug blatt idea.
+#
+# assume the bug is actually at l3, but you don't know that - all you know is that l3 is broken
+# and it wasn't broken before
+#
+# keep bisecting the list, advancing the "bad" head and accumulating "good" heads until
+# the bisection point is the head - this is the bad point.
+#
+
+test_output_expect_success "$_bisect_option l5 ^root" 'git rev-list $_bisect_option l5 ^root' <<EOF
+c3
+EOF
+
+test_output_expect_success "$_bisect_option l5 ^root ^c3" 'git rev-list $_bisect_option l5 ^root ^c3' <<EOF
+b4
+EOF
+
+test_output_expect_success "$_bisect_option l5 ^root ^c3 ^b4" 'git rev-list $_bisect_option l5 ^c3 ^b4' <<EOF
+l3
+EOF
+
+test_output_expect_success "$_bisect_option l3 ^root ^c3 ^b4" 'git rev-list $_bisect_option l3 ^root ^c3 ^b4' <<EOF
+a4
+EOF
+
+test_output_expect_success "$_bisect_option l5 ^b3 ^a3 ^b4 ^a4" 'git rev-list $_bisect_option l3 ^b3 ^a3 ^a4' <<EOF
+l3
+EOF
+
+#
+# if l3 is bad, then l4 is bad too - so advance the bad pointer by making b4 the known bad head
+#
+
+test_output_expect_success "$_bisect_option l4 ^a2 ^a3 ^b ^a4" 'git rev-list $_bisect_option l4 ^a2 ^a3 ^a4' <<EOF
+l3
+EOF
+
+test_output_expect_success "$_bisect_option l3 ^a2 ^a3 ^b ^a4" 'git rev-list $_bisect_option l3 ^a2 ^a3 ^a4' <<EOF
+l3
+EOF
+
+# found!
+
+#
+# as another example, let's consider a4 to be the bad head, in which case
+#
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4' <<EOF
+c2
+EOF
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4 ^c2" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4 ^c2' <<EOF
+c3
+EOF
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4 ^c2 ^c3" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4 ^c2 ^c3' <<EOF
+a4
+EOF
+
+# found!
+
+#
+# or consider c3 to be the bad head
+#
+
+test_output_expect_success "$_bisect_option a4 ^a2 ^a3 ^b4" 'git rev-list $_bisect_option a4 ^a2 ^a3 ^b4' <<EOF
+c2
+EOF
+
+test_output_expect_success "$_bisect_option c3 ^a2 ^a3 ^b4 ^c2" 'git rev-list $_bisect_option c3 ^a2 ^a3 ^b4 ^c2' <<EOF
+c3
+EOF
+
+# found!
+
+}
+
+test_sequence "--bisect"
+
+#
+#
+
+test_expect_success 'set up fake --bisect refs' '
+	git update-ref refs/bisect/bad c3 &&
+	good=$(git rev-parse b1) &&
+	git update-ref refs/bisect/good-$good $good &&
+	good=$(git rev-parse c1) &&
+	git update-ref refs/bisect/good-$good $good
+'
+
+test_expect_success 'rev-list --bisect can default to good/bad refs' '
+	# the only thing between c3 and c1 is c2
+	git rev-parse c2 >expect &&
+	git rev-list --bisect >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'rev-parse --bisect can default to good/bad refs' '
+	git rev-parse c3 ^b1 ^c1 >expect &&
+	git rev-parse --bisect >actual &&
+
+	# output order depends on the refnames, which in turn depends on
+	# the exact sha1s. We just want to make sure we have the same set
+	# of lines in any order.
+	sort <expect >expect.sorted &&
+	sort <actual >actual.sorted &&
+	test_cmp expect.sorted actual.sorted
+'
+
+test_output_expect_success '--bisect --first-parent' 'git rev-list --bisect --first-parent E ^F' <<EOF
+e4
+EOF
+
+test_output_expect_success '--first-parent' 'git rev-list --first-parent E ^F' <<EOF
+E
+e1
+e2
+e3
+e4
+e5
+e6
+e7
+e8
+EOF
+
+test_output_expect_success '--bisect-vars --first-parent' 'git rev-list --bisect-vars --first-parent E ^F' <<EOF
+bisect_rev='e5'
+bisect_nr=4
+bisect_good=4
+bisect_bad=3
+bisect_all=9
+bisect_steps=2
+EOF
+
+test_expect_success '--bisect-all --first-parent' '
+	cat >expect.unsorted <<-EOF &&
+	$(git rev-parse E) (tag: E, dist=0)
+	$(git rev-parse e1) (tag: e1, dist=1)
+	$(git rev-parse e2) (tag: e2, dist=2)
+	$(git rev-parse e3) (tag: e3, dist=3)
+	$(git rev-parse e4) (tag: e4, dist=4)
+	$(git rev-parse e5) (tag: e5, dist=4)
+	$(git rev-parse e6) (tag: e6, dist=3)
+	$(git rev-parse e7) (tag: e7, dist=2)
+	$(git rev-parse e8) (tag: e8, dist=1)
+	EOF
+
+	# expect results to be ordered by distance (descending),
+	# commit hash (ascending)
+	sort -k4,4r -k1,1 expect.unsorted >expect &&
+	git rev-list --bisect-all --first-parent E ^F >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success '--bisect without any revisions' '
+	git rev-list --bisect HEAD..HEAD >out &&
+	test_must_be_empty out
+'
+
+test_done
-- 
2.54.0

[RFC PATCH v7 05/10] diff: read precomputed hunks for stat output

[Michael Montalbo]

Teach builtin_diffstat() to consult the hunk provider interface through
diff_provider_consult(), new here: the consult-only entry that answers
without loading content or computing, so it never returns
DIFF_PROVIDER_ERROR.  On an answer, the summing callback accumulates the
provided counts directly into the diffstat entry; the blobs were already
loaded for the binary check, so an answer saves the diff run, not the
content load (blame, taught next, skips its loads too).  On an
unanswered outcome it computes as before and, with a writer attached,
records what it computed; on unanswered-no-record it computes without
recording.

The provider behind the consult is the diff-hunks store, registered in
front of the terminal builtin computation.  Its consult serves a
recorded pair through diff_hunks_replay(), which validates the sequence
before any hunk reaches the callback, so direct accumulation is safe.
The request gains the pair's object ids and the diff options read by the
exclusions below.  A side whose bytes are not a stored blob, such as a
working-tree file or a gitlink, has a NULL id; the store passes it by and
the terminal provider computes it.  diff_provider_emit_hunks() walks the
same chain, so blame's requests follow these rules the moment blame
supplies identity.  The walk also insists, as a BUG check, that a
request's diff options belong to the repository whose chain it walks.

Each exclusion lives with the provider whose key cannot express it.  -I
patterns and --anchored shape the diff outside the store key, and break
detection (-B) rescores the pair outside it; the store's consult maps
all three to stop-no-record, so such a request is neither served nor
recorded for any consumer.  The consumer-side guard the recording commit
carried for those three comes out here.  The compile-time assert on
xpparam_t's layout sits next to that decision, forcing an explicit
keying decision whenever a diff parameter is added.  The stat consumer
keeps only the exclusion that is not about the key: --ignore-blank-lines
is part of the key but coalesces hunks differently between the
text-emitting and coordinate-callback paths, so the consumer returns
before consulting.  A "log -L" range-scoped stat neither reads nor
records; the line-range filter computes it as before.

"git diff", "git log", "git show", and "git diff-tree" with the --stat,
--numstat, and --shortstat formats consult the interface.  Reading is
controlled by core.diffHunks.

An answer is invisible in the output, so the store counts the pairs it
serves and the consultations it cannot, and diff_hunks_read_stats()
reports both; the stat path emits the hits as a trace2 "read-hits" datum
for tests and tuning.  The counters live on the store because only the
store knows whether a consultation reached it, and none of its exclusion
legs reaches the replay, so none counts as a miss.

Extend t4220 with the read half:

- output parity with and without the store, at several context lengths
  and both directions, and reversed pairs keying apart;
- the consultation made visible through the read-hits datum, and the
  trim-divergent pair correct at every context;
- the settings that must bypass the store doing so in both directions
  (-I, -B, --anchored, --ignore-blank-lines), asserted through the trace
  rather than output parity alone, which a coincidentally equal count
  could satisfy;
- a driver-forced algorithm keying apart rather than bypassing: it is
  part of the key, so a read under it misses the default entries and a
  warm records under its own.

A "log -L" range-scoped stat neither reads nor records.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 builtin/log.c            |   7 +
 diff-hunks.c             |  88 +++++++++++-
 diff-hunks.h             |   8 ++
 diff-provider-internal.h |   7 +
 diff-provider.c          |  62 +++++++--
 diff-provider.h          |  28 +++-
 diff.c                   |  86 ++++++++----
 diff.h                   |  20 ++-
 t/t4220-diff-hunks.sh    | 280 +++++++++++++++++++++++++++++++++++++++
 9 files changed, 539 insertions(+), 47 deletions(-)

diff --git a/builtin/log.c b/builtin/log.c
index 6903bdd5ff..0b72477f56 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -2225,6 +2225,13 @@ int cmd_format_patch(int argc,
 	if (argc > 1)
 		die(_("unrecognized argument: %s"), argv[1]);
 
+	/*
+	 * A patch generated by format-patch carries the builtin diffstat,
+	 * not one served from a local store, so its counts do not depend
+	 * on whether the sender warmed the store.
+	 */
+	rev.diffopt.flags.no_precomputed_hunks = 1;
+
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME_STATUS)
diff --git a/diff-hunks.c b/diff-hunks.c
index 23cefe055b..9830df07fa 100644
--- a/diff-hunks.c
+++ b/diff-hunks.c
@@ -26,6 +26,7 @@
 #include "csum-file.h"
 #include "diff-hunks.h"
 #include "diff-provider-internal.h"
+#include "diff.h"
 #include "gettext.h"
 #include "hash.h"
 #include "hashmap.h"
@@ -140,6 +141,10 @@ struct diff_hunks_store {
 	uint32_t num_entries;
 	const unsigned char *hdat;
 	size_t hdat_size;
+
+	/* Consultation counters; see diff_hunks_read_stats(). */
+	unsigned long read_hits;
+	unsigned long read_misses;
 };
 
 static void free_store(struct diff_hunks_store *s)
@@ -256,6 +261,15 @@ struct diff_hunks_store *repo_diff_hunks_store(struct repository *r)
 	return r->objects->diff_hunks_store;
 }
 
+void diff_hunks_read_stats(struct repository *r,
+			   unsigned long *hits, unsigned long *misses)
+{
+	struct diff_hunks_store *s = repo_diff_hunks_store(r);
+
+	*hits = s ? s->read_hits : 0;
+	*misses = s ? s->read_misses : 0;
+}
+
 void close_diff_hunks_store(struct object_database *o)
 {
 	if (!o->diff_hunks_store)
@@ -413,18 +427,90 @@ int diff_hunks_replay(struct diff_hunks_store *s,
 	struct precomputed_entry e;
 	uint32_t i;
 
+	if (!s)
+		return 0;
 	if (!diff_hunks_store_get(s, old_oid, new_oid, xdl_opts, &e) ||
-	    !replayable_hunks(&e))
+	    !replayable_hunks(&e)) {
+		s->read_misses++;
 		return 0;
+	}
 	for (i = 0; i < e.num_hunks; i++) {
 		struct precomputed_hunk h;
 		nth_precomputed_hunk(&e, i, &h);
 		hunk_func(h.old_start, h.old_count,
 			  h.new_start, h.new_count, cb_data);
 	}
+	s->read_hits++;
 	return 1;
 }
 
+/*
+ * The store's consult implementation.  The store is not
+ * authoritative, so it serves a recorded pair or passes; what the
+ * recording key cannot express, it excludes here with the
+ * stop-no-record disposition.  None of those legs reaches
+ * diff_hunks_replay(), so none of them counts as a miss.
+ */
+static enum diff_provider_disposition
+diff_hunks_store_consult(struct diff_provider *provider UNUSED,
+			 const struct diff_provider_request *req,
+			 diff_provider_fill_fn fill UNUSED,
+			 void *fill_data UNUSED,
+			 xdl_emit_hunk_consume_func_t hunk_cb, void *cb_data)
+{
+	/*
+	 * xpparam_t is the consult's parameter input.  Its flags are
+	 * the store key's xdl_opts; ignore_regex (-I) and anchors
+	 * (--anchored) shape the diff outside the key, so such a
+	 * request is neither served nor recorded.
+	 *
+	 * Adding an xpparam_t field fires this assert (its size no
+	 * longer matches the reference struct).  To clear it: (1) add
+	 * the field to the reference struct below; then (2) decide how
+	 * it affects the key: make it part of the key, or exclude
+	 * diffs that use it here with the disposition below.  The
+	 * assert only tracks size: a same-size reorder or a changed
+	 * field meaning slips past, so re-read the fields when it
+	 * fires.
+	 */
+	(void)BUILD_ASSERT_OR_ZERO(sizeof(xpparam_t) == sizeof(struct {
+		unsigned long flags;
+		regex_t **ignore_regex;
+		size_t ignore_regex_nr;
+		char **anchors;
+		size_t anchors_nr;
+	}));
+	if (req->xpp->ignore_regex_nr || req->xpp->anchors_nr)
+		return DIFF_PROVIDER_DISP_STOP_NO_RECORD;
+	/*
+	 * Break detection (-B) rescores the pair outside xpparam_t, so
+	 * it is outside the key for the same reason.
+	 */
+	if (req->diffopt && req->diffopt->break_opt != -1)
+		return DIFF_PROVIDER_DISP_STOP_NO_RECORD;
+
+	if (!req->old_oid || !req->new_oid)
+		return DIFF_PROVIDER_DISP_PASS;
+	if (diff_hunks_replay(repo_diff_hunks_store(req->repo),
+			      req->old_oid, req->new_oid,
+			      req->xpp->flags, hunk_cb, cb_data))
+		return DIFF_PROVIDER_DISP_ANSWERED;
+	return DIFF_PROVIDER_DISP_PASS;
+}
+
+/*
+ * The provider borrows the repository's store through
+ * repo_diff_hunks_store() per request; the object database owns the
+ * file and tears it down, so there is nothing to release here.
+ */
+struct diff_provider *diff_hunks_store_provider_new(void)
+{
+	struct diff_provider *p = xcalloc(1, sizeof(*p));
+
+	p->consult = diff_hunks_store_consult;
+	return p;
+}
+
 /* Validate one store file. Returns 0 if valid or absent, -1 on any error. */
 static int verify_store_at(struct repository *r, const char *fname)
 {
diff --git a/diff-hunks.h b/diff-hunks.h
index 89e56f4b1b..c58500d05c 100644
--- a/diff-hunks.h
+++ b/diff-hunks.h
@@ -62,6 +62,14 @@ struct diff_hunks_store *repo_diff_hunks_store(struct repository *r);
 /* Free the repository's cached store, at object-database teardown. */
 void close_diff_hunks_store(struct object_database *o);
 
+/*
+ * Consultation counters for the repository's store: pairs the store
+ * served (hits) and pairs it was consulted for but could not serve
+ * (misses). Both zero when reading is disabled or no store exists.
+ */
+void diff_hunks_read_stats(struct repository *r,
+			   unsigned long *hits, unsigned long *misses);
+
 /*
  * Replay the recorded hunks of an (old blob, new blob) pair diffed
  * under xdl_opts through hunk_func. The sequence is validated before
diff --git a/diff-provider-internal.h b/diff-provider-internal.h
index 8dd8e4b0dc..8ad3e481e7 100644
--- a/diff-provider-internal.h
+++ b/diff-provider-internal.h
@@ -88,6 +88,13 @@ struct diff_provider {
 	struct diff_provider *next;
 };
 
+/*
+ * The providers Git ships, besides the builtin computation that
+ * diff-provider.c holds itself.  Each call returns a fresh provider
+ * for one repository's chain.
+ */
+struct diff_provider *diff_hunks_store_provider_new(void);
+
 /*
  * Incremental well-formedness check for a provider-supplied hunk
  * sequence, shared by every provider.  Each coordinate, and each
diff --git a/diff-provider.c b/diff-provider.c
index b69854fb63..66a9909eaa 100644
--- a/diff-provider.c
+++ b/diff-provider.c
@@ -1,5 +1,7 @@
 #include "git-compat-util.h"
+#include "diff.h"
 #include "diff-provider-internal.h"
+#include "replace-object.h"
 #include "repository.h"
 
 /*
@@ -39,10 +41,11 @@ static struct diff_provider *builtin_provider_new(void)
 
 /*
  * The repository's chain, assembled on first walk.  The composition
- * is fixed; the builtin computation is the terminal provider, so the
- * chain always ends in an implementor that can answer.  Nothing is
- * decided per repository here; each provider gates itself per
- * request.
+ * is fixed, and the order is the authority resolution: the store is
+ * consulted before the builtin computation, the terminal provider,
+ * so the chain always ends in an implementor that can answer.
+ * Nothing is decided per repository here; each provider gates itself
+ * per request.
  */
 static struct diff_provider *provider_chain(struct repository *r)
 {
@@ -50,6 +53,8 @@ static struct diff_provider *provider_chain(struct repository *r)
 
 	if (*tail)
 		return *tail;
+	*tail = diff_hunks_store_provider_new();
+	tail = &(*tail)->next;
 	*tail = builtin_provider_new();
 	return r->diff_providers;
 }
@@ -70,16 +75,16 @@ void diff_providers_clear(struct repository *r)
 }
 
 /*
- * The walk behind diff_provider_emit_hunks(): consult the chain in
- * order and map its dispositions onto the outcome set.  The first
- * answer ends the walk.  A stop-no-record disposition
- * (diff-provider-internal.h) is a refusal, not a pass: the provider
- * does not answer, but rules the pair out of identity service and
- * out of recording, so from then on the walk consults only the
- * computing provider, and a walk that ends unanswered carries the
- * no-record verdict.  With a fill callback the terminal provider
- * computes instead of passing, so an emit walk returns only
- * answered or error.
+ * The walk shared by diff_provider_consult() and
+ * diff_provider_emit_hunks(): consult the chain in order and map its
+ * dispositions onto the outcome set.  The first answer ends the
+ * walk.  A stop-no-record disposition (diff-provider-internal.h)
+ * is a refusal, not a pass: the provider does not answer, but rules
+ * the pair out of identity service and out of recording, so from
+ * then on the walk consults only the computing provider, and a walk
+ * that ends unanswered carries the no-record verdict.  With a fill
+ * callback the terminal provider computes instead of passing, so an
+ * emit walk returns only answered or error.
  */
 static enum diff_provider_outcome
 walk_providers(const struct diff_provider_request *req,
@@ -89,6 +94,28 @@ walk_providers(const struct diff_provider_request *req,
 	struct diff_provider *p;
 	int no_record = 0;
 
+	if (req->diffopt && req->diffopt->repo != req->repo)
+		BUG("diff provider request walks one repository's chain "
+		    "with another repository's diff options");
+
+	/*
+	 * An object replacement redirects a blob's content
+	 * (OBJECT_INFO_LOOKUP_REPLACE) while leaving the id that names it
+	 * unchanged, so an answer keyed on the raw id would be the
+	 * pre-replacement diff.  A replacement is therefore a parameter
+	 * outside the recording key: no provider may serve a replaced pair
+	 * from its identity, and a result computed for it must not be
+	 * recorded under the raw id.  Mark the walk no-record so the
+	 * identity providers step aside and the builtin computes from the
+	 * replaced content.  The check is a no-op when the repository has
+	 * no replace refs.
+	 */
+	if ((req->old_oid &&
+	     lookup_replace_object(req->repo, req->old_oid) != req->old_oid) ||
+	    (req->new_oid &&
+	     lookup_replace_object(req->repo, req->new_oid) != req->new_oid))
+		no_record = 1;
+
 	for (p = provider_chain(req->repo); p; p = p->next) {
 		enum diff_provider_disposition disp;
 
@@ -118,6 +145,13 @@ walk_providers(const struct diff_provider_request *req,
 		DIFF_PROVIDER_UNANSWERED;
 }
 
+enum diff_provider_outcome
+diff_provider_consult(const struct diff_provider_request *req,
+		      xdl_emit_hunk_consume_func_t hunk_cb, void *cb_data)
+{
+	return walk_providers(req, NULL, NULL, hunk_cb, cb_data);
+}
+
 enum diff_provider_hunks_error
 diff_provider_check_hunk(struct diff_provider_hunks_check *c,
 			  long old_start, long old_count,
diff --git a/diff-provider.h b/diff-provider.h
index 1a7e4e299c..c5478b5700 100644
--- a/diff-provider.h
+++ b/diff-provider.h
@@ -26,6 +26,8 @@
  * sees it.
  */
 
+struct diff_options;
+struct object_id;
 struct repository;
 
 /*
@@ -89,15 +91,35 @@ enum diff_provider_outcome {
  * A consultation request.  The interface consults providers from
  * these fields alone; no content is loaded before an answer.
  *
- * repo owns the provider chain the request walks.  xpp carries the
- * parameters the diff runs with.  Each provider gates itself on the
- * fields that concern it.
+ * repo owns the provider chain the request walks.  old_oid/new_oid
+ * name the blobs whose bytes are diffed; pass NULL for a side whose
+ * bytes are not a stored blob (a working-tree file, textconv output,
+ * a gitlink), so no provider answers from an id it cannot look up.
+ * diffopt carries the diff settings that live outside xpp; xpp
+ * carries the parameters the diff runs with.  Each provider gates
+ * itself on the fields that concern it.
  */
 struct diff_provider_request {
 	struct repository *repo;
+	const struct object_id *old_oid;
+	const struct object_id *new_oid;
+	struct diff_options *diffopt;
 	const xpparam_t *xpp;
 };
 
+/*
+ * Consult the providers for the request's pair without computing.
+ * On DIFF_PROVIDER_ANSWERED the hunks were emitted through hunk_cb
+ * (0-based emission coordinates, context 0) and were validated
+ * before the first callback ran, so a consumer may accumulate
+ * directly into its result.  Never returns DIFF_PROVIDER_ERROR.
+ * The callback's return value is not consulted: emission of a
+ * validated answer has no error leg, so the callback must return 0.
+ */
+enum diff_provider_outcome
+diff_provider_consult(const struct diff_provider_request *req,
+		      xdl_emit_hunk_consume_func_t hunk_cb, void *cb_data);
+
 /*
  * Load the pair's content.  Called at most once per request, only
  * when the ranges are computed rather than provided.  The buffers
diff --git a/diff.c b/diff.c
index 11ec88dc8c..0d4cb3fcfd 100644
--- a/diff.c
+++ b/diff.c
@@ -17,6 +17,7 @@
 #include "quote.h"
 #include "diff.h"
 #include "diff-hunks.h"
+#include "diff-provider.h"
 #include "diffcore.h"
 #include "delta.h"
 #include "hex.h"
@@ -35,6 +36,7 @@
 #include "tmp-objdir.h"
 #include "graph.h"
 #include "oid-array.h"
+#include "trace2.h"
 #include "packfile.h"
 #include "pager.h"
 #include "parse-options.h"
@@ -2992,6 +2994,11 @@ void diff_hunks_attach(struct diff_options *o)
 
 void diff_hunks_detach(struct diff_options *o)
 {
+	unsigned long hits, misses;
+
+	diff_hunks_read_stats(o->repo, &hits, &misses);
+	if (hits)
+		trace2_data_intmax("diff-hunks", o->repo, "read-hits", hits);
 	diff_hunks_writer_finish(o->hunks_writer);
 	o->hunks_writer = NULL;
 }
@@ -4321,18 +4328,35 @@ static const char *get_compact_summary(const struct diff_filepair *p, int is_ren
 }
 
 /*
- * Fill data->added/deleted for a modified pair by collecting its hunk
- * coordinates, and record them into the store. Runs only on a warming
- * run; returns 1 when it produced the counts, 0 when the caller must
- * compute the diffstat itself.
+ * Hunk callback for the provider interface: sum counts into a
+ * diffstat entry.
+ */
+static int diffstat_sum_hunk_cb(long start_a UNUSED, long count_a,
+				long start_b UNUSED, long count_b,
+				void *cb_data)
+{
+	struct diffstat_file *data = cb_data;
+
+	data->added += count_b;
+	data->deleted += count_a;
+	return 0;
+}
+
+/*
+ * Fill data->added/deleted for a modified pair through the hunk provider
+ * interface: on an answer, sum the provided counts; on a warming run,
+ * compute and record them. Returns 1 when it produced the counts, 0 when
+ * the caller must compute the diffstat itself.
  *
- * --ignore-blank-lines is excluded: that flag is part of the store
- * key, but it coalesces hunks differently between the emit and
- * hunk-callback paths, so a recorded entry would not match a
- * store-less run's --stat output. (--inter-hunk-context is not
- * excluded: it only groups hunks, and diffstat sums their counts,
- * which grouping does not change.) Recording requires both sides to
- * be valid regular files whose blobs the key can name.
+ * The providers own the exclusions the request can express (-B, -I,
+ * and --anchored are outside the store key). This consumer additionally
+ * excludes --ignore-blank-lines before consulting: that flag is part of
+ * the key, but it coalesces hunks differently between the emit and
+ * hunk-callback paths, so a served answer would not match a store-less
+ * run's --stat output. (--inter-hunk-context is not excluded: it only
+ * groups hunks, and diffstat sums their counts, which grouping does not
+ * change.) Recording requires both sides to be valid regular files whose
+ * blobs the key can name.
  */
 static int diffstat_from_hunks(struct diff_options *o,
 			       struct diff_filespec *one,
@@ -4347,23 +4371,37 @@ static int diffstat_from_hunks(struct diff_options *o,
 			  .ignore_regex_nr = o->ignore_regex_nr,
 			  .anchors = o->anchors,
 			  .anchors_nr = o->anchors_nr };
+	struct diff_provider_request req = {
+		.repo = o->repo,
+		.old_oid = (one->oid_valid && !S_ISGITLINK(one->mode)) ?
+			   &one->oid : NULL,
+		.new_oid = (two->oid_valid && !S_ISGITLINK(two->mode)) ?
+			   &two->oid : NULL,
+		.diffopt = o,
+		.xpp = &xpp,
+	};
 
 	if (o->xdl_opts & XDF_IGNORE_BLANK_LINES)
 		return 0;
+	/* format-patch keeps its diffstat off the store (see the flag). */
+	if (o->flags.no_precomputed_hunks)
+		return 0;
 
-	/* Not a warming run: the caller computes the diffstat. */
-	if (!o->hunks_writer)
+	switch (diff_provider_consult(&req, diffstat_sum_hunk_cb, data)) {
+	case DIFF_PROVIDER_ANSWERED:
+		return 1;
+	case DIFF_PROVIDER_UNANSWERED:
+		break;
+	case DIFF_PROVIDER_ERROR: /* not returned by a consult */
+	case DIFF_PROVIDER_UNANSWERED_NO_RECORD:
 		return 0;
-	/*
-	 * -I patterns, --anchored anchors, and break detection (-B)
-	 * shape the diff outside the store key, so what they compute
-	 * must not be recorded under it.
-	 */
-	if (o->ignore_regex_nr || o->anchors_nr || o->break_opt != -1)
+	}
+
+	/* A miss on a read-only run: let the caller compute the diffstat. */
+	if (!o->hunks_writer)
 		return 0;
 	/* Recording needs blobs the key can name, on both sides. */
-	if (!one->oid_valid || !two->oid_valid ||
-	    S_ISGITLINK(one->mode) || S_ISGITLINK(two->mode) ||
+	if (!req.old_oid || !req.new_oid ||
 	    !DIFF_FILE_VALID(one) || !DIFF_FILE_VALID(two) ||
 	    !S_ISREG(one->mode) || !S_ISREG(two->mode))
 		return 0;
@@ -4453,9 +4491,9 @@ static void builtin_diffstat(const char *name_a, const char *name_b,
 
 	else if (may_differ) {
 		/*
-		 * Record into the diff-hunks store on a warming run. A
-		 * "log -L" range-scoped stat is not the whole-pair diff
-		 * the store keys, so it does not record. Otherwise diff
+		 * Serve or record via the diff-hunks store. A "log -L"
+		 * range-scoped stat is not the whole-pair diff the store
+		 * keys, so it neither reads nor records. Otherwise diff
 		 * normally.
 		 */
 		if (p->line_ranges || !diffstat_from_hunks(o, one, two, data)) {
diff --git a/diff.h b/diff.h
index 3d44de39ff..6166598eea 100644
--- a/diff.h
+++ b/diff.h
@@ -206,6 +206,13 @@ struct diff_flags {
 	unsigned suppress_diff_headers;
 	unsigned dual_color_diffed_diffs;
 	unsigned suppress_hunk_header_line_count;
+
+	/*
+	 * Do not serve the diffstat from the precomputed-hunks store.
+	 * Set by format-patch so a generated patch carries the builtin
+	 * counts and does not depend on the sender's local store state.
+	 */
+	unsigned no_precomputed_hunks;
 };
 
 static inline void diff_flags_or(struct diff_flags *a,
@@ -422,9 +429,11 @@ struct diff_options {
 	int max_depth_valid;
 
 	/*
-	 * Precomputed diff hunks (see diff-hunks.h). When hunks_writer is
-	 * set (a warming run), diffstat records the hunks it computes;
-	 * the writer is attached only for the stat output formats.
+	 * Precomputed diff hunks (see diff-hunks.h). diffstat consults the
+	 * hunk provider interface before running xdiff, keyed by each file
+	 * pair's blob object IDs. When hunks_writer is set (a warming run),
+	 * diffstat also records the hunks it computes; the writer is
+	 * attached only for the stat output formats.
 	 */
 	struct diff_hunks_writer *hunks_writer;
 };
@@ -679,8 +688,9 @@ void diff_free(struct diff_options*);
 /*
  * Attach a diff-hunks writer to a diff producing a stat format, so a
  * warming run records the hunks it computes; a no-op when writing is off
- * or for other formats. Pair with diff_hunks_detach() once the diff is
- * done.
+ * or for other formats. (Reading is separate: consumers consult the
+ * providers through diff_provider_consult(); see diff-provider.h.) Pair
+ * with diff_hunks_detach() once the diff is done.
  */
 void diff_hunks_attach(struct diff_options *o);
 void diff_hunks_detach(struct diff_options *o);
diff --git a/t/t4220-diff-hunks.sh b/t/t4220-diff-hunks.sh
index c677831946..62329e1070 100755
--- a/t/t4220-diff-hunks.sh
+++ b/t/t4220-diff-hunks.sh
@@ -81,6 +81,79 @@ test_expect_success 'a second warming run refreshes the store in place' '
 	test_cmp expect actual
 '
 
+test_expect_success 'log --stat matches with and without the store' '
+	no_store log --stat >expect &&
+	warm &&
+	git log --stat >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'log --numstat and --shortstat match' '
+	no_store log --numstat >expect_num &&
+	no_store log --shortstat >expect_short &&
+	warm &&
+	git log --numstat >actual_num &&
+	git log --shortstat >actual_short &&
+	test_cmp expect_num actual_num &&
+	test_cmp expect_short actual_short
+'
+
+# A built store must reproduce diffstat output at every context
+# length.  Only trim-stable pairs are recorded, so one entry serves
+# every context; a trim-divergent pair is never recorded and always
+# computed.  Zero context is where trim_common_tail runs, which is
+# what makes the two diffs differ.
+test_expect_success 'diffstat matches at several context lengths' '
+	no_store log --stat >expect_def &&
+	no_store log -U0 --stat >expect_u0 &&
+	no_store log -U7 --stat >expect_u7 &&
+	warm &&
+	git log --stat >got_def &&
+	git log -U0 --stat >got_u0 &&
+	git log -U7 --stat >got_u7 &&
+	test_cmp expect_def got_def &&
+	test_cmp expect_u0 got_u0 &&
+	test_cmp expect_u7 got_u7
+'
+
+test_expect_success 'store built at a nonzero context stays correct at that context' '
+	no_store -c diff.context=5 log --stat >expect &&
+	warm -c diff.context=5 &&
+	git -c diff.context=5 log --stat >actual &&
+	test_cmp expect actual
+'
+
+# This blob pair (a real git test file being modernized) has different
+# valid diffs at different contexts: at zero context, where
+# trim_common_tail runs, "diff -U0" reports 9/6, while "diff -U3"
+# reports 10/7. Such a trim-divergent pair is exactly what the writer
+# must never record, since no single entry could serve both readers.
+# A compact synthetic pair cannot show this count split: on small
+# inputs xdiff produces minimal diffs, minimal diffs of one pair all
+# add and delete the same number of lines, and trimming the common
+# tail preserves minimality, so the counts agree by construction (a
+# search over thousands of synthetic pairs up to 8 lines found no
+# split). The split needs the cost-capping heuristics that only larger
+# inputs trigger, so the pair is shipped as a fixture under t4220/.
+test_expect_success 'a trim-divergent file is correct at each context' '
+	cp "$TEST_DIRECTORY/t4220/trim-divergent-old" div.sh &&
+	git add div.sh &&
+	git commit -m divergent-old &&
+	cp "$TEST_DIRECTORY/t4220/trim-divergent-new" div.sh &&
+	git add div.sh &&
+	git commit -m divergent-new &&
+	no_store log -1 --format= --stat -- div.sh >expect_def &&
+	no_store log -1 --format= -U0 --stat -- div.sh >expect_u0 &&
+	warm &&
+	git log -1 --format= --stat -- div.sh >got_def &&
+	git log -1 --format= -U0 --stat -- div.sh >got_u0 &&
+	test_cmp expect_def got_def &&
+	test_cmp expect_u0 got_u0 &&
+	# The fixture must actually diverge, or the test would pass without
+	# exercising the split; fail loudly if a diff change ever levels it.
+	! test_cmp expect_def expect_u0
+'
+
 # A warming run displays the diffstat it computes. At zero context xdi_diff
 # trims, so the displayed counts must be the trimmed ones (what a store-less
 # run shows), not the untrimmed ones the writer compares against when it
@@ -99,6 +172,16 @@ test_expect_success 'warming --stat at zero context matches a store-less run' '
 	)
 '
 
+test_expect_success 'diff --stat matches with and without the store, both directions' '
+	no_store diff --stat second fourth >expect_fwd &&
+	no_store diff --stat fourth second >expect_rev &&
+	warm &&
+	git diff --stat second fourth >got_fwd &&
+	git diff --stat fourth second >got_rev &&
+	test_cmp expect_fwd got_fwd &&
+	test_cmp expect_rev got_rev
+'
+
 test_expect_success 'show and diff-tree --stat use the store' '
 	test_when_finished "git diff-hunks clear" &&
 	# diff_hunks_attach() runs for show and diff-tree: a write-enabled
@@ -121,6 +204,193 @@ test_expect_success 'show and diff-tree --stat use the store' '
 	test_cmp expect_dt got_dt
 '
 
+test_expect_success 'log -R --stat matches (reversed pairs keyed apart)' '
+	no_store log -R --stat >expect &&
+	warm &&
+	git log -R --stat >actual &&
+	test_cmp expect actual
+'
+
+# The diffstat read path produces identical output on a hit or a miss, so
+# it emits a trace2 "read-hits" count to prove it consulted the store.
+test_expect_success 'diffstat consults the store (trace shows read hits)' '
+	warm &&
+	GIT_TRACE2_EVENT="$PWD/trace_on.json" git log --stat >/dev/null &&
+	test_grep read-hits trace_on.json &&
+	test_env GIT_TRACE2_EVENT="$PWD/trace_off.json" no_store log --stat >/dev/null &&
+	test_grep ! read-hits trace_off.json
+'
+
+# Diff settings that change hunks but are not part of the store key must
+# bypass it in both directions, so output stays byte-identical to a
+# store-less run.
+test_expect_success 'setup ignore fixture' '
+	git init ignore-repo &&
+	(
+		cd ignore-repo &&
+		test_write_lines code keep "# c" >f &&
+		git add f &&
+		git commit -m c1 &&
+		test_write_lines codeCH keep "# cX" >f &&
+		git add f &&
+		git commit -m c2 &&
+		warm
+	)
+'
+
+# Output parity alone cannot prove the guard: served counts can
+# coincide with computed ones, so each bypass below also asserts the
+# consultation itself (no read hit with the option, a hit without it)
+# and that a warming run under the option records nothing.
+test_expect_success '-I bypasses the store in both directions' '
+	(
+		cd ignore-repo &&
+		no_store diff -I"^#" --numstat HEAD~ HEAD >expect &&
+		git diff -I"^#" --numstat HEAD~ HEAD >actual &&
+		test_cmp expect actual &&
+		# -I does not change the key, so only the ignore_regex
+		# guard keeps the warmed entry from serving here.
+		GIT_TRACE2_EVENT="$PWD/trace_i.json" \
+			git diff -I"^#" --numstat HEAD~ HEAD >/dev/null &&
+		test_grep ! read-hits trace_i.json &&
+		GIT_TRACE2_EVENT="$PWD/trace_i_ctl.json" \
+			git diff --numstat HEAD~ HEAD >/dev/null &&
+		test_grep read-hits trace_i_ctl.json &&
+		git diff-hunks clear &&
+		GIT_DIFF_HUNKS_WRITE=1 git diff -I"^#" --numstat HEAD~ HEAD >/dev/null &&
+		test_path_is_missing .git/objects/info/diff-hunks &&
+		# Restore the warmed fixture for the tests below.
+		warm
+	)
+'
+
+test_expect_success '-B bypasses the store in both directions' '
+	git init break-repo &&
+	(
+		cd break-repo &&
+		test_write_lines a b c d e f g h >f &&
+		git add f &&
+		git commit -m orig &&
+		test_write_lines 1 2 3 4 5 6 7 8 >f &&
+		git add f &&
+		git commit -m rewrite &&
+		warm &&
+		no_store diff -B --stat HEAD~ HEAD >expect &&
+		git diff -B --stat HEAD~ HEAD >actual &&
+		test_cmp expect actual &&
+		GIT_TRACE2_EVENT="$PWD/trace_b.json" \
+			git diff -B --stat HEAD~ HEAD >/dev/null &&
+		test_grep ! read-hits trace_b.json &&
+		GIT_TRACE2_EVENT="$PWD/trace_ctl.json" \
+			git diff --stat HEAD~ HEAD >/dev/null &&
+		test_grep read-hits trace_ctl.json &&
+		git diff-hunks clear &&
+		GIT_DIFF_HUNKS_WRITE=1 git diff -B --stat HEAD~ HEAD >/dev/null &&
+		test_path_is_missing .git/objects/info/diff-hunks
+	)
+'
+
+test_expect_success '--anchored bypasses the store in both directions' '
+	(
+		cd ignore-repo &&
+		no_store diff --stat --anchored=keep HEAD~ HEAD >expect &&
+		git diff --stat --anchored=keep HEAD~ HEAD >actual &&
+		test_cmp expect actual &&
+		# Anchors do not change the key, so only the anchors guard
+		# keeps the warmed entry from serving here.
+		GIT_TRACE2_EVENT="$PWD/trace_anchor.json" \
+			git diff --stat --anchored=keep HEAD~ HEAD >/dev/null &&
+		test_grep ! read-hits trace_anchor.json &&
+		GIT_TRACE2_EVENT="$PWD/trace_plain.json" \
+			git diff --stat HEAD~ HEAD >/dev/null &&
+		test_grep read-hits trace_plain.json &&
+		git diff-hunks clear &&
+		GIT_DIFF_HUNKS_WRITE=1 \
+			git diff --stat --anchored=keep HEAD~ HEAD >/dev/null &&
+		test_path_is_missing .git/objects/info/diff-hunks
+	)
+'
+
+test_expect_success '--ignore-blank-lines bypasses the store in both directions' '
+	git init ibl-repo &&
+	(
+		cd ibl-repo &&
+		printf "a\n\nx\ny\nb\n" >f &&
+		git add f &&
+		git commit -m v1 &&
+		printf "a\nx\ny\nB\n" >f &&
+		git add f &&
+		git commit -m v2 &&
+		warm &&
+		no_store diff --stat --ignore-blank-lines HEAD~ HEAD >expect &&
+		git diff --stat --ignore-blank-lines HEAD~ HEAD >actual &&
+		test_cmp expect actual &&
+		# The flag is an xdl_opts bit and thus part of the key; the
+		# stat consumer excludes it before consulting at all.
+		GIT_TRACE2_EVENT="$PWD/trace_ibl.json" \
+			git diff --stat --ignore-blank-lines HEAD~ HEAD >/dev/null &&
+		test_grep ! read-hits trace_ibl.json &&
+		GIT_TRACE2_EVENT="$PWD/trace_plain.json" \
+			git diff --stat HEAD~ HEAD >/dev/null &&
+		test_grep read-hits trace_plain.json &&
+		git diff-hunks clear &&
+		GIT_DIFF_HUNKS_WRITE=1 \
+			git diff --stat --ignore-blank-lines HEAD~ HEAD >/dev/null &&
+		test_path_is_missing .git/objects/info/diff-hunks
+	)
+'
+
+test_expect_success 'a whitespace-ignoring diff is not served default entries' '
+	git init ws-repo &&
+	(
+		cd ws-repo &&
+		test_write_lines alpha beta gamma >f &&
+		git add f &&
+		git commit -m c1 &&
+		test_write_lines "  alpha" beta gamma delta >f &&
+		git add f &&
+		git commit -m c2 &&
+		warm &&
+		no_store diff -w --numstat HEAD~ HEAD >expect &&
+		git diff -w --numstat HEAD~ HEAD >actual &&
+		test_cmp expect actual
+	)
+'
+
+test_expect_success 'a driver algorithm override keeps output correct and keys apart' '
+	git init driver-algo &&
+	(
+		cd driver-algo &&
+		echo "file.foo diff=foo" >.gitattributes &&
+		git add .gitattributes &&
+		git commit -m attributes &&
+		test_write_lines 1 2 3 4 5 >file.foo &&
+		git add file.foo &&
+		git commit -m one &&
+		test_write_lines 1 2 X 4 5 6 >file.foo &&
+		git add file.foo &&
+		git commit -m two &&
+		warm -c diff.foo.algorithm=histogram &&
+		no_store -c diff.foo.algorithm=histogram log --stat >expect &&
+		git -c diff.foo.algorithm=histogram log --stat >actual &&
+		test_cmp expect actual &&
+		# The driver algorithm is an xdl_opts key bit: entries
+		# warmed at the default settings must not serve a
+		# driver-forced histogram read, and output stays correct.
+		git diff-hunks clear &&
+		warm &&
+		no_store -c diff.foo.algorithm=histogram log --stat >expect2 &&
+		git -c diff.foo.algorithm=histogram log --stat >actual2 &&
+		test_cmp expect2 actual2 &&
+		GIT_TRACE2_EVENT="$PWD/trace_algo.json" \
+			git -c diff.foo.algorithm=histogram log --stat >/dev/null &&
+		test_grep ! read-hits trace_algo.json &&
+		GIT_TRACE2_EVENT="$PWD/trace_algo_ctl.json" \
+			git log --stat >/dev/null &&
+		test_grep read-hits trace_algo_ctl.json
+	)
+'
+
 # Cover the pair shapes an object walk encounters: binary and
 # mode-only changes produce no text hunks to record.
 test_expect_success 'binary and mode-only changes do not break the writer' '
@@ -141,6 +411,16 @@ test_expect_success 'binary and mode-only changes do not break the writer' '
 	test_cmp expect actual
 '
 
+test_expect_success 'log -L --stat neither reads nor records' '
+	warm &&
+	GIT_TRACE2_EVENT="$PWD/trace_linelog.json" \
+		git log -L1,1:file.txt --stat >/dev/null &&
+	test_grep ! read-hits trace_linelog.json &&
+	git diff-hunks clear &&
+	GIT_DIFF_HUNKS_WRITE=1 git log -L1,1:file.txt --stat >/dev/null &&
+	test_path_is_missing $STORE
+'
+
 test_expect_success 'verify succeeds on a valid store and on an absent one' '
 	warm &&
 	git diff-hunks verify &&
-- 
2.54.0

[RFC PATCH v7 06/10] blame: read precomputed hunks

[Michael Montalbo]

Before diffing a target blob against a parent, offer the pair's identity
to the hunk provider interface.  Blame's requests have gone through
diff_provider_emit_hunks() since the interface arrived, but carried no
identity, so nothing could answer them.  Now blame fills in the pair's
blob object ids and its diff options, and the chain serves the pair from
the store, keyed by the ids and the request's xdiff flags, before the
terminal provider falls back to fill-and-compute.  Blame diffs at zero
context, which is not part of the key.  An answer replays the recorded
hunks through blame_chunk_cb without loading either blob; a request
carrying -I patterns or anchors is outside the key and always computes.

Blame withholds the identity where its diff is not the plain blob-pair
diff the key describes: reverse blame, ignored revisions, textconv
paths, and the working-tree or --contents pseudo-commit, whose blob is
not a stored object.  Those requests always compute.  Whitespace and
algorithm options such as -w instead change blame's xdl_opts, so the
consult keys a different entry and misses a store warmed without them.
Blame's default xdl_opts now come from DIFF_HUNKS_DEFAULT_XDL_OPTS, new
here, which records the key-relevant defaults a diff_options-based
consumer already carries (today the indent heuristic), so a default
blame run and a default "log --stat" warming run share keys by
construction.

"--show-stats" reports how many pairs the store served and how many
consultations it could not, read from diff_hunks_read_stats(); the store
counts its own consultations, so blame keeps no tally.

Extend t4220 with the blame side:

- parity for plain, --porcelain, and --incremental output, and hit and
  miss accounting across warming runs;
- the blame inputs that must bypass or miss the store: -w, indent
  heuristics, --reverse, textconv, -M/-C, and the --ignore-rev pass;
- rename and merge handling, and --contents;
- reading a truncated or corrupt store as absent, and a crafted
  zero-hunk record as a miss that verify flags.

Add p4218, measuring the cost of a warming run and the read speedups.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 blame.c                    |  39 ++++
 builtin/blame.c            |   8 +-
 diff.h                     |  11 ++
 t/meson.build              |   1 +
 t/perf/p4218-diff-hunks.sh |  48 +++++
 t/t4220-diff-hunks.sh      | 355 +++++++++++++++++++++++++++++++++++++
 6 files changed, 461 insertions(+), 1 deletion(-)
 create mode 100755 t/perf/p4218-diff-hunks.sh

diff --git a/blame.c b/blame.c
index c3ef9c17f7..7d7671ef5d 100644
--- a/blame.c
+++ b/blame.c
@@ -24,6 +24,7 @@
 #include "bloom.h"
 #include "commit-graph.h"
 #include "diff-provider.h"
+#include "userdiff.h"
 
 define_commit_slab(blame_suspects, struct blame_origin *);
 static struct blame_suspects blame_suspects;
@@ -1937,6 +1938,24 @@ static int blame_chunk_cb(long start_a, long count_a,
 	return 0;
 }
 
+/*
+ * A hunk provider's key names the (old blob, new blob) pair and may only
+ * serve a diff whose result is determined by that pair and the xdiff
+ * settings. Textconv rewrites the buffers being diffed away from the
+ * blob contents the key names, so any origin whose path has a textconv
+ * driver must withhold the pair's identity.
+ */
+static int blame_textconv_active(struct blame_scoreboard *sb,
+				 const char *path)
+{
+	struct userdiff_driver *drv;
+
+	if (!sb->revs->diffopt.flags.allow_textconv)
+		return 0;
+	drv = userdiff_find_by_path(sb->repo->index, path);
+	return drv && drv->textconv;
+}
+
 struct blame_diff_fill_data {
 	struct blame_scoreboard *sb;
 	struct blame_origin *parent, *target;
@@ -1973,6 +1992,7 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 	struct blame_diff_fill_data fill_data = { sb, parent, target, ignore_diffs };
 	xpparam_t xpp = { .flags = sb->xdl_opts };
 	struct diff_provider_request req = { .repo = sb->repo, .xpp = &xpp };
+	int provider_usable;
 
 	if (!target->suspects)
 		return; /* nothing remains for this target */
@@ -1983,6 +2003,25 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 	d.ignore_diffs = ignore_diffs;
 	d.dstq = &newdest; d.srcq = &target->suspects;
 
+	/*
+	 * Offer the pair's identity only where blame's diff is the plain
+	 * blob-pair diff the recording key describes; reverse blame,
+	 * ignored revisions, and textconv paths withhold it and always
+	 * compute.  The working-tree/--contents pseudo-commit (marked by
+	 * its null commit id) holds a blob that is not a stored object,
+	 * so its pairs withhold identity too: no id may be sent that
+	 * names bytes a provider cannot look up.
+	 */
+	provider_usable = !sb->reverse && !ignore_diffs &&
+		!is_null_oid(&target->commit->object.oid) &&
+		!blame_textconv_active(sb, target->path) &&
+		!blame_textconv_active(sb, parent->path);
+
+	if (provider_usable) {
+		req.old_oid = &parent->blob_oid;
+		req.new_oid = &target->blob_oid;
+	}
+	req.diffopt = &sb->revs->diffopt;
 	if (diff_provider_emit_hunks(&req, blame_diff_fill, &fill_data,
 				     blame_chunk_cb, &d) == DIFF_PROVIDER_ERROR)
 		die("unable to generate diff (%s -> %s)",
diff --git a/builtin/blame.c b/builtin/blame.c
index 48d5251c6d..7891d82ae6 100644
--- a/builtin/blame.c
+++ b/builtin/blame.c
@@ -15,6 +15,7 @@
 #include "hex.h"
 #include "commit.h"
 #include "diff.h"
+#include "diff-hunks.h"
 #include "revision.h"
 #include "quote.h"
 #include "string-list.h"
@@ -1060,7 +1061,7 @@ int cmd_blame(int argc,
 parse_done:
 	revision_opts_finish(&revs);
 	no_whole_file_rename = !revs.diffopt.flags.follow_renames;
-	xdl_opts |= revs.diffopt.xdl_opts & XDF_INDENT_HEURISTIC;
+	xdl_opts |= revs.diffopt.xdl_opts & DIFF_HUNKS_DEFAULT_XDL_OPTS;
 	revs.diffopt.flags.follow_renames = 0;
 	argc = parse_options_end(&ctx);
 
@@ -1315,9 +1316,14 @@ int cmd_blame(int argc,
 	output(&sb, output_option);
 
 	if (show_stats) {
+		unsigned long hunk_hits, hunk_misses;
+
+		diff_hunks_read_stats(sb.repo, &hunk_hits, &hunk_misses);
 		printf("num read blob: %d\n", sb.num_read_blob);
 		printf("num get patch: %d\n", sb.num_get_patch);
 		printf("num commits: %d\n", sb.num_commits);
+		printf("num precomputed hits: %lu\n", hunk_hits);
+		printf("num precomputed misses: %lu\n", hunk_misses);
 	}
 
 cleanup:
diff --git a/diff.h b/diff.h
index 6166598eea..380a258878 100644
--- a/diff.h
+++ b/diff.h
@@ -231,6 +231,17 @@ static inline void diff_flags_or(struct diff_flags *a,
 
 #define DIFF_WITH_ALG(opts, flag)   (((opts)->xdl_opts & ~XDF_DIFF_ALGORITHM_MASK) | XDF_##flag)
 
+/*
+ * The xdl_opts bits git turns on by default that a from-scratch xdl_opts
+ * (git blame's own option parsing) does not set, and so must OR in to match
+ * a store warmed at the default diff settings; a diff_options-based consumer
+ * (diffstat) already has them in o->xdl_opts. Today this is only the indent
+ * heuristic. It does NOT cover a non-default diff.algorithm: a repo that
+ * configures one records under that algorithm, and a consumer keying without
+ * it misses (a lost hit, not wrong output).
+ */
+#define DIFF_HUNKS_DEFAULT_XDL_OPTS XDF_INDENT_HEURISTIC
+
 enum diff_words_type {
 	DIFF_WORDS_NONE = 0,
 	DIFF_WORDS_PORCELAIN,
diff --git a/t/meson.build b/t/meson.build
index c63c30ba63..3f45b09dd6 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -1158,6 +1158,7 @@ benchmarks = [
   'perf/p4205-log-pretty-formats.sh',
   'perf/p4209-pickaxe.sh',
   'perf/p4211-line-log.sh',
+  'perf/p4218-diff-hunks.sh',
   'perf/p4220-log-grep-engines.sh',
   'perf/p4221-log-grep-engines-fixed.sh',
   'perf/p5302-pack-index.sh',
diff --git a/t/perf/p4218-diff-hunks.sh b/t/perf/p4218-diff-hunks.sh
new file mode 100755
index 0000000000..f849e97832
--- /dev/null
+++ b/t/perf/p4218-diff-hunks.sh
@@ -0,0 +1,48 @@
+#!/bin/sh
+
+test_description='diff-hunks store performance'
+. ./perf-lib.sh
+
+test_perf_default_repo
+
+# Pick a file to blame pseudo-randomly. The sort key is the blob
+# hash, so it is stable.
+test_expect_success 'select a file' '
+	git ls-tree -r HEAD | grep ^100644 |
+	sort -k 3 | head -n 1 | cut -f 2 >filelist
+'
+
+file=$(cat filelist)
+export file
+
+# Warm the store the way an owner would: a stat walk with writing on.
+test_perf 'warm the store' '
+	git diff-hunks clear &&
+	GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
+'
+
+test_expect_success 'ensure the store is warm for the timed reads' '
+	GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null
+'
+
+test_perf 'log --stat -1000 (store)' '
+	git log --stat -1000 >/dev/null
+'
+
+test_perf 'log --stat -1000 (no store)' '
+	git -c core.diffhunks=false log --stat -1000 >/dev/null
+'
+
+test_perf 'blame $file (store)' '
+	git blame "$file" >/dev/null
+'
+
+test_perf 'blame $file (no store)' '
+	git -c core.diffhunks=false blame "$file" >/dev/null
+'
+
+test_expect_success 'clean up store' '
+	git diff-hunks clear
+'
+
+test_done
diff --git a/t/t4220-diff-hunks.sh b/t/t4220-diff-hunks.sh
index 62329e1070..086c53f651 100755
--- a/t/t4220-diff-hunks.sh
+++ b/t/t4220-diff-hunks.sh
@@ -81,6 +81,49 @@ test_expect_success 'a second warming run refreshes the store in place' '
 	test_cmp expect actual
 '
 
+test_expect_success 'core.diffhunks=false disables lookups' '
+	warm &&
+	git -c core.diffhunks=false blame --show-stats file.txt >out 2>&1 &&
+	test_grep "num precomputed hits: 0" out
+'
+
+# Writing seeds from the current store and merges into it, so a later
+# warming run keeps the entries an earlier one recorded rather than
+# rebuilding. Warm one pair, then a different pair, and confirm the first
+# is still served.
+test_expect_success 'a later warming run preserves earlier entries' '
+	git init incr &&
+	(
+		cd incr &&
+		test_commit a1 f.txt "1" &&
+		test_commit a2 f.txt "1
+2" &&
+		test_commit a3 f.txt "1
+2
+3" &&
+		GIT_DIFF_HUNKS_WRITE=1 git diff --stat a1 a2 >/dev/null &&
+		git diff-hunks verify &&
+		GIT_DIFF_HUNKS_WRITE=1 git diff --stat a2 a3 >/dev/null &&
+		git diff-hunks verify &&
+
+		# Blaming as of a2 diffs the a1..a2 pair. If seeding had
+		# dropped it when the a2..a3 pair was warmed, this would
+		# report zero precomputed hits.
+		git blame --show-stats a2 -- f.txt >out 2>&1 &&
+		test_grep "num precomputed hits: [1-9]" out &&
+
+		# The second warm ADDED the a2..a3 pair; blaming a3 diffs
+		# both a2..a3 and a1..a2, so a hit on each shows the store
+		# gained the new pair while keeping the earlier one.
+		git blame --show-stats a3 -- f.txt >out3 2>&1 &&
+		test_grep "num precomputed hits: 2" out3 &&
+
+		no_store log --stat >expect &&
+		git log --stat >actual &&
+		test_cmp expect actual
+	)
+'
+
 test_expect_success 'log --stat matches with and without the store' '
 	no_store log --stat >expect &&
 	warm &&
@@ -211,6 +254,14 @@ test_expect_success 'log -R --stat matches (reversed pairs keyed apart)' '
 	test_cmp expect actual
 '
 
+# One warm serves both diffstat and blame: the blob pairs a blame
+# walks are the same parent-child pairs the diffstat warm recorded.
+test_expect_success 'a single warming run serves both blame and diffstat' '
+	warm &&
+	git blame --show-stats file.txt >out 2>&1 &&
+	test_grep "num precomputed hits: [1-9][0-9]*" out
+'
+
 # The diffstat read path produces identical output on a hit or a miss, so
 # it emits a trace2 "read-hits" count to prove it consulted the store.
 test_expect_success 'diffstat consults the store (trace shows read hits)' '
@@ -221,6 +272,23 @@ test_expect_success 'diffstat consults the store (trace shows read hits)' '
 	test_grep ! read-hits trace_off.json
 '
 
+test_expect_success 'blame matches with and without the store' '
+	no_store blame file.txt >expect &&
+	warm &&
+	git blame file.txt >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'blame --porcelain and --incremental match' '
+	no_store blame --porcelain file.txt >expect_p &&
+	no_store blame --incremental file.txt >expect_i &&
+	warm &&
+	git blame --porcelain file.txt >got_p &&
+	git blame --incremental file.txt >got_i &&
+	test_cmp expect_p got_p &&
+	test_cmp expect_i got_i
+'
+
 # Diff settings that change hunks but are not part of the store key must
 # bypass it in both directions, so output stays byte-identical to a
 # store-less run.
@@ -357,6 +425,26 @@ test_expect_success 'a whitespace-ignoring diff is not served default entries' '
 	)
 '
 
+test_expect_success 'blame -w stays correct and does not hit default entries' '
+	(
+		cd ws-repo &&
+		no_store blame -w f >expect &&
+		git blame -w --show-stats f >out 2>&1 &&
+		test_grep "num precomputed hits: 0" out &&
+		git blame -w f >actual &&
+		test_cmp expect actual
+	)
+'
+
+test_expect_success 'blame with indentHeuristic off stays correct and misses' '
+	warm &&
+	git -c diff.indentHeuristic=false blame --show-stats file.txt >out 2>&1 &&
+	test_grep "num precomputed hits: 0" out &&
+	no_store -c diff.indentHeuristic=false blame file.txt >expect &&
+	git -c diff.indentHeuristic=false blame file.txt >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'a driver algorithm override keeps output correct and keys apart' '
 	git init driver-algo &&
 	(
@@ -391,6 +479,94 @@ test_expect_success 'a driver algorithm override keeps output correct and keys a
 	)
 '
 
+test_expect_success 'blame --reverse never consults the store' '
+	warm &&
+	git blame --reverse HEAD~3..HEAD file.txt >actual 2>/dev/null &&
+	no_store blame --reverse HEAD~3..HEAD file.txt >expect 2>/dev/null &&
+	test_cmp expect actual &&
+	# Reverse blame withholds the pair identity. Zero hits alone
+	# cannot prove that: reverse pairs are never warmed, so a
+	# consulted pair would miss, not hit. Zero misses is what shows
+	# the store was never consulted.
+	git blame --reverse --show-stats HEAD~3..HEAD file.txt \
+		>stats 2>/dev/null &&
+	test_grep "num precomputed hits: 0" stats &&
+	test_grep "num precomputed misses: 0" stats
+'
+
+test_expect_success 'blame with a textconv driver bypasses the store' '
+	echo "tc.txt diff=tc" >>.gitattributes &&
+	git add .gitattributes &&
+	git commit -m tc-attr &&
+	git config diff.tc.textconv "sed -e s/1/one/" &&
+	test_commit tc1 tc.txt "line 1" &&
+	test_commit tc2 tc.txt "line 1
+line 2" &&
+	warm &&
+	git blame --show-stats tc.txt >out 2>&1 &&
+	test_grep "num precomputed hits: 0" out &&
+	no_store blame tc.txt >expect &&
+	git blame tc.txt >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'a replaced blob makes the store step aside' '
+	git init replace-repo &&
+	(
+		cd replace-repo &&
+		test_commit r1 f.txt "a" &&
+		test_commit r2 f.txt "a
+b" &&
+		warm &&
+		# Control: without a replacement the pair is served.
+		GIT_TRACE2_EVENT="$PWD/trace_ctl.json" \
+			git log -1 --format= --numstat -- f.txt >/dev/null &&
+		test_grep read-hits trace_ctl.json &&
+		# Replace r2 blob: the diff now reads different content
+		# (through OBJECT_INFO_LOOKUP_REPLACE) under the id the store
+		# keyed, so a served answer would be the pre-replacement diff.
+		# Identity is withheld, the store steps aside, and the builtin
+		# computes from the replaced content.
+		new_blob=$(git rev-parse HEAD:f.txt) &&
+		repl=$(printf "a\nB\nC\nD\n" | git hash-object -w --stdin) &&
+		git replace "$new_blob" "$repl" &&
+		no_store log -1 --format= --numstat -- f.txt >expect &&
+		git log -1 --format= --numstat -- f.txt >actual &&
+		test_cmp expect actual &&
+		GIT_TRACE2_EVENT="$PWD/trace_repl.json" \
+			git log -1 --format= --numstat -- f.txt >/dev/null &&
+		test_grep ! read-hits trace_repl.json
+	)
+'
+
+test_expect_success 'blame -M and -C stay correct with the store' '
+	warm &&
+	no_store blame -M file.txt >expect_m &&
+	no_store blame -C file.txt >expect_c &&
+	git blame -M file.txt >got_m &&
+	git blame -C file.txt >got_c &&
+	test_cmp expect_m got_m &&
+	test_cmp expect_c got_c
+'
+
+# Copy-detecting (and reverse) blame still diff blob pairs through
+# pass_blame_to_parent, so they must use the real blame xdl_opts. A
+# whitespace-only change is invisible under -w; if -w were dropped on
+# these paths the -w and non-w results would coincide.
+test_expect_success 'blame -C honors -w' '
+	git init -q blame-cw &&
+	(
+		cd blame-cw &&
+		printf "one\ntwo\nthree\n" >f &&
+		git add f && git commit -q -m base &&
+		printf "one\n  two  \nthree\n" >f &&
+		git add f && git commit -q -m reindent &&
+		git blame -C -w f >with_w &&
+		git blame -C f >without_w &&
+		! test_cmp with_w without_w
+	)
+'
+
 # Cover the pair shapes an object walk encounters: binary and
 # mode-only changes produce no text hunks to record.
 test_expect_success 'binary and mode-only changes do not break the writer' '
@@ -411,6 +587,92 @@ test_expect_success 'binary and mode-only changes do not break the writer' '
 	test_cmp expect actual
 '
 
+test_expect_success 'blame across a rename matches' '
+	echo "original content" >rename-src.txt &&
+	git add rename-src.txt &&
+	git commit -m "add rename-src" &&
+	echo "more" >>rename-src.txt &&
+	git add rename-src.txt &&
+	git commit -m "modify rename-src" &&
+	git mv rename-src.txt rename-dst.txt &&
+	git commit -m "rename" &&
+	echo "post" >>rename-dst.txt &&
+	git add rename-dst.txt &&
+	git commit -m "modify after rename" &&
+	no_store blame rename-dst.txt >expect &&
+	warm &&
+	git blame rename-dst.txt >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'blame handles merge commits' '
+	git checkout -b merge-side main~2 &&
+	test_commit merge-change merge-file.txt "side content" &&
+	git checkout main &&
+	git merge --no-edit merge-side &&
+	no_store blame merge-file.txt >expect &&
+	warm &&
+	git blame merge-file.txt >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'distinct --contents against one revision do not collide' '
+	warm &&
+	test_write_lines "line 1" "appended line" >c1 &&
+	test_write_lines "rewritten line" >c2 &&
+	# Ground truth without the store.
+	no_store blame -s --contents=c2 file.txt initial >expect &&
+	# With the store, an intervening c1 run must not poison the c2 lookup.
+	git blame -s --contents=c1 file.txt initial >/dev/null &&
+	git blame -s --contents=c2 file.txt initial >actual &&
+	test_cmp expect actual &&
+	# The --contents side is a working-tree pseudo-commit (a null commit
+	# id), so its pairs withhold identity and never consult the store.
+	# Output parity alone cannot show that: a consulted unwarmed pair
+	# would miss, not hit, so zero misses is what proves the pair was
+	# never looked up.
+	git blame -s --show-stats --contents=c2 file.txt initial >stats 2>&1 &&
+	test_grep "num precomputed hits: 0" stats &&
+	test_grep "num precomputed misses: 0" stats
+'
+
+test_expect_success 'blame --ignore-rev bypasses the store for ignored pairs' '
+	git init ignore-rev-repo &&
+	(
+		cd ignore-rev-repo &&
+		test_commit ir1 f.txt "base" &&
+		test_commit ir2 f.txt "base
+more" &&
+		warm &&
+		# Control: the ordinary pass is served, nothing is computed.
+		git blame --show-stats f.txt >ctl 2>&1 &&
+		test_grep "num precomputed hits: 1" ctl &&
+		test_grep "num get patch: 0" ctl &&
+		no_store blame --ignore-rev ir2 f.txt >expect &&
+		git blame --ignore-rev ir2 f.txt >actual &&
+		test_cmp expect actual &&
+		# The ignored revision adds a pass that withholds identity:
+		# it computes its diff (get patch rises) instead of being
+		# served or even counted as a store consultation.
+		git blame --ignore-rev ir2 --show-stats f.txt >stats 2>&1 &&
+		test_grep "num precomputed hits: 1" stats &&
+		test_grep "num precomputed misses: 0" stats &&
+		test_grep "num get patch: 1" stats
+	)
+'
+
+test_expect_success 'blame counts misses for pairs the store does not hold' '
+	(
+		cd ignore-rev-repo &&
+		test_commit ir3 f.txt "base
+more
+third" &&
+		git blame --show-stats f.txt >stats 2>&1 &&
+		test_grep "num precomputed hits: 1" stats &&
+		test_grep "num precomputed misses: 1" stats
+	)
+'
+
 test_expect_success 'log -L --stat neither reads nor records' '
 	warm &&
 	GIT_TRACE2_EVENT="$PWD/trace_linelog.json" \
@@ -421,6 +683,67 @@ test_expect_success 'log -L --stat neither reads nor records' '
 	test_path_is_missing $STORE
 '
 
+# Integrity: a structurally broken header is read as absent (the reader
+# falls back to xdiff and stays correct); a checksum mismatch is caught
+# by verify, which is when integrity is checked.
+test_expect_success 'a truncated store is read as absent' '
+	warm &&
+	test_copy_bytes 20 <$STORE >truncated &&
+	mv truncated $STORE &&
+	no_store blame file.txt >expect &&
+	git blame file.txt >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'a corrupt signature is read as absent' '
+	warm &&
+	printf "XXXX" >corrupt &&
+	tail -c +5 <$STORE >>corrupt &&
+	mv corrupt $STORE &&
+	no_store blame file.txt >expect &&
+	git blame file.txt >actual &&
+	test_cmp expect actual
+'
+
+# Byte 6 of the header is the chunk count; a value larger than the file
+# can hold must be rejected before the chunk table is walked.
+test_expect_success 'an over-claimed chunk count is read as absent' '
+	warm &&
+	printf "\377" | dd of=$STORE bs=1 seek=6 count=1 conv=notrunc 2>/dev/null &&
+	no_store blame file.txt >expect &&
+	git blame file.txt >actual &&
+	test_cmp expect actual
+'
+
+# A record with no hunks would replay as an equivalence claim, which
+# the writer never records; the reader must treat such a record as a
+# miss and recompute, and verify must flag it.
+test_expect_success 'a zero-hunk record is read as a miss and fails verify' '
+	git init zero-hunk &&
+	(
+		cd zero-hunk &&
+		test_commit z1 f.txt "base" &&
+		test_commit z2 f.txt "base
+more" &&
+		warm &&
+		# The store holds one entry of one hunk: a 4-byte count and
+		# one 16-byte hunk record, just before the trailing
+		# checksum. Zero the count to craft the record the writer
+		# refuses to produce.
+		rawsz=$(test_oid rawsz) &&
+		fsize=$(test_file_size $STORE) &&
+		printf "\\0\\0\\0\\0" | dd of=$STORE bs=1 \
+			seek=$((fsize - rawsz - 20)) count=4 conv=notrunc \
+			2>/dev/null &&
+		no_store blame f.txt >expect &&
+		git blame --show-stats f.txt >stats 2>&1 &&
+		test_grep "num precomputed hits: 0" stats &&
+		git blame f.txt >actual &&
+		test_cmp expect actual &&
+		test_must_fail git diff-hunks verify
+	)
+'
+
 test_expect_success 'verify succeeds on a valid store and on an absent one' '
 	warm &&
 	git diff-hunks verify &&
@@ -454,6 +777,38 @@ test_expect_success 'a warm discards a corrupt store rather than seeding from it
 	test_cmp expect actual
 '
 
+# A generated patch must carry the builtin diffstat, not one served from
+# the sender's local store, so its counts do not depend on whether the
+# sender warmed the store. Poison the store so a served answer diverges
+# from the builtin, then confirm format-patch shows the builtin counts.
+test_expect_success 'format-patch keeps its diffstat off the store' '
+	git init fp-repo &&
+	(
+		cd fp-repo &&
+		test_commit p1 f.txt "a" &&
+		test_commit p2 f.txt "a
+b" &&
+		warm &&
+		# Bump the new-side count of the single recorded hunk. The
+		# record stays structurally valid, and a read skips the
+		# trailing checksum, so the store serves this poisoned count.
+		rawsz=$(test_oid rawsz) &&
+		fsize=$(test_file_size .git/objects/info/diff-hunks) &&
+		printf "\\0\\0\\0\\7" | dd of=.git/objects/info/diff-hunks bs=1 \
+			seek=$((fsize - rawsz - 4)) count=4 conv=notrunc 2>/dev/null &&
+		# The store now serves a divergent count, proving the poison
+		# is live and observable through a store consumer.
+		printf "7\t0\tf.txt\n" >poisoned &&
+		git log -1 --format= --numstat -- f.txt >served &&
+		test_cmp poisoned served &&
+		# format-patch does not consult the store, so its output is
+		# identical with the store poisoned and with it disabled.
+		no_store format-patch -1 --stdout --stat -- f.txt >expect &&
+		git format-patch -1 --stdout --stat -- f.txt >actual &&
+		test_cmp expect actual
+	)
+'
+
 test_expect_success 'diff-hunks clear removes the store file' '
 	warm &&
 	test_path_is_file $STORE &&
-- 
2.54.0

[RFC PATCH v7 07/10] sub-process: separate process lifecycle from hashmap management

[Michael Montalbo]

subprocess_start() and subprocess_stop() couple two concerns: managing a
child process (setup, handshake, teardown) and managing a hashmap that
indexes running processes by command string.  The hashmap suits callers
like convert.c where many files may share one filter process looked up
by name, but callers that manage process membership under their own
rules do not need the coupled operations.

Extract subprocess_start_command() and subprocess_stop_command() so
callers can reuse the child process setup and handshake machinery
without the map operations.  subprocess_start() and subprocess_stop()
become thin wrappers that add hashmap operations on top.

The diff process support added later in this series keeps its processes
in a pool owned by a per-repository provider object, and an entry for a
failed command must stay behind there so the command is not retried.
That membership follows rules subprocess_start() and subprocess_stop()
do not know.  The pool therefore uses the _command variants for process
lifecycle and manages its own map.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 28 +++++++++++++++++++++++-----
 sub-process.h |  9 ++++++++-
 2 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/sub-process.c b/sub-process.c
index 2d5c965169..3cef42b088 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,7 +49,7 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
-void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
 		return;
@@ -57,7 +57,14 @@ void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
 	entry->process.clean_on_exit = 0;
 	kill(entry->process.pid, SIGTERM);
 	finish_command(&entry->process);
+}
 
+void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry)
+{
+	if (!entry)
+		return;
+
+	subprocess_stop_command(entry);
 	hashmap_remove(hashmap, &entry->ent, NULL);
 }
 
@@ -72,7 +79,7 @@ static void subprocess_exit_handler(struct child_process *process)
 	finish_command(process);
 }
 
-int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
 	subprocess_start_fn startfn)
 {
 	int err;
@@ -96,15 +103,26 @@ int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, co
 		return err;
 	}
 
-	hashmap_entry_init(&entry->ent, strhash(cmd));
-
 	err = startfn(entry);
 	if (err) {
 		error("initialization for subprocess '%s' failed", cmd);
-		subprocess_stop(hashmap, entry);
+		subprocess_stop_command(entry);
 		return err;
 	}
 
+	return 0;
+}
+
+int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
+	subprocess_start_fn startfn)
+{
+	int err;
+
+	err = subprocess_start_command(entry, cmd, startfn);
+	if (err)
+		return err;
+
+	hashmap_entry_init(&entry->ent, strhash(cmd));
 	hashmap_add(hashmap, &entry->ent);
 	return 0;
 }
diff --git a/sub-process.h b/sub-process.h
index bfc3959a1b..45f1b8e5e3 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -52,10 +52,17 @@ int cmd2process_cmp(const void *unused_cmp_data,
  */
 typedef int(*subprocess_start_fn)(struct subprocess_entry *entry);
 
-/* Start a subprocess and add it to the subprocess hashmap. */
+/* Start a subprocess and run the startfn (typically handshake). */
+int subprocess_start_command(struct subprocess_entry *entry, const char *cmd,
+		subprocess_start_fn startfn);
+
+/* Start a subprocess, run startfn, and add it to the subprocess hashmap. */
 int subprocess_start(struct hashmap *hashmap, struct subprocess_entry *entry, const char *cmd,
 		subprocess_start_fn startfn);
 
+/* Kill a subprocess. */
+void subprocess_stop_command(struct subprocess_entry *entry);
+
 /* Kill a subprocess and remove it from the subprocess hashmap. */
 void subprocess_stop(struct hashmap *hashmap, struct subprocess_entry *entry);
 
-- 
2.54.0

[RFC PATCH v7 08/10] sub-process: add a gentle status read

[Michael Montalbo]

subprocess_read_status() reads "status=<key>" packets up to a flush with
packet_read_line_gently(), which is gentle only about EOF.  A malformed
length header still dies inside pkt-line, and an empty packet is
indistinguishable from the flush that ends the section.  A protocol
violation in a status section therefore either kills the whole command
or silently truncates the section.  That posture fits the filter
protocol's callers, which treat their process as required
infrastructure; the diff process consult added later in this series
treats its process as optional, and any protocol error must degrade to
the builtin diff rather than abort the command.

Add subprocess_read_status_gently(): the same status loop, reading
through packet_read_with_status() with the gentle options, returning
-1 on a truncated or malformed packet and on an empty packet where a
status line or the terminating flush belongs.  subprocess_read_status()
and its callers are unchanged.

The handshake has its gentle counterpart in 061a68e443 (sub-process:
use gentle handshake to avoid die() on startup failure, 2026-06-01),
which turned truncated handshake reads into error returns for every
caller.  This series' base includes that commit, so a process that
dies during the handshake feeds the same non-fatal fallback as a
status failure here, and an optional diff process degrades to the
builtin diff on either kind of protocol error.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 sub-process.c | 24 ++++++++++++++++++++++++
 sub-process.h | 10 ++++++++++
 2 files changed, 34 insertions(+)

diff --git a/sub-process.c b/sub-process.c
index 3cef42b088..33bd789618 100644
--- a/sub-process.c
+++ b/sub-process.c
@@ -49,6 +49,30 @@ int subprocess_read_status(int fd, struct strbuf *status)
 	return (len < 0) ? len : 0;
 }
 
+int subprocess_read_status_gently(int fd, struct strbuf *status)
+{
+	for (;;) {
+		int pktlen = -1;
+		enum packet_read_status rs;
+		const char *value;
+
+		rs = packet_read_with_status(fd, NULL, NULL, packet_buffer,
+					     sizeof(packet_buffer), &pktlen,
+					     PACKET_READ_CHOMP_NEWLINE |
+					     PACKET_READ_GENTLE_ON_EOF |
+					     PACKET_READ_GENTLE_ON_READ_ERROR);
+		if (rs == PACKET_READ_FLUSH)
+			return 0;
+		if (rs != PACKET_READ_NORMAL || !pktlen)
+			return -1;
+		if (skip_prefix(packet_buffer, "status=", &value)) {
+			/* the last "status=<foo>" line wins */
+			strbuf_reset(status);
+			strbuf_addstr(status, value);
+		}
+	}
+}
+
 void subprocess_stop_command(struct subprocess_entry *entry)
 {
 	if (!entry)
diff --git a/sub-process.h b/sub-process.h
index 45f1b8e5e3..8655b38897 100644
--- a/sub-process.h
+++ b/sub-process.h
@@ -101,4 +101,14 @@ int subprocess_handshake(struct subprocess_entry *entry,
 
 int subprocess_read_status(int fd, struct strbuf *status);
 
+/*
+ * Like subprocess_read_status(), but a malformed status section fails
+ * instead of dying: a truncated or malformed packet, and an empty
+ * packet where a status line or the terminating flush belongs, return
+ * -1 and leave the stream unusable.  subprocess_read_status() cannot
+ * tell an empty packet from the flush that ends the section, and dies
+ * on a framing error inside packet_read_line_gently().
+ */
+int subprocess_read_status_gently(int fd, struct strbuf *status);
+
 #endif
-- 
2.54.0

[RFC PATCH v7 09/10] userdiff: add diff.<driver>.process config

[Michael Montalbo]

Add the process field to struct userdiff_driver and teach the
config parser to populate it from diff.<driver>.process.

The field names a long-running hunk provider process.  Nothing
reads it yet: the consult, the protocol, and the documentation
arrive with the next commit, which starts and pools processes keyed
by this field's command string.

Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 userdiff.c | 7 +++++++
 userdiff.h | 2 ++
 2 files changed, 9 insertions(+)

diff --git a/userdiff.c b/userdiff.c
index b5412e6bc3..7547874aa2 100644
--- a/userdiff.c
+++ b/userdiff.c
@@ -509,6 +509,13 @@ int userdiff_config(const char *k, const char *v)
 		drv->algorithm = drv->algorithm_owned;
 		return ret;
 	}
+	if (!strcmp(type, "process")) {
+		int ret;
+		FREE_AND_NULL(drv->process_owned);
+		ret = git_config_string(&drv->process_owned, k, v);
+		drv->process = drv->process_owned;
+		return ret;
+	}
 
 	return 0;
 }
diff --git a/userdiff.h b/userdiff.h
index 827361b0bc..51c26e0d41 100644
--- a/userdiff.h
+++ b/userdiff.h
@@ -31,6 +31,8 @@ struct userdiff_driver {
 	char *textconv_owned;
 	struct notes_cache *textconv_cache;
 	int textconv_want_cache;
+	const char *process;
+	char *process_owned;
 };
 enum userdiff_driver_type {
 	USERDIFF_DRIVER_TYPE_BUILTIN = 1<<0,
-- 
2.54.0

[RFC PATCH v7 10/10] diff: consult oid-only hunk providers via diff.<driver>.process

[Michael Montalbo]

The provider chain so far holds the diff-hunks store in front of the
terminal builtin computation.  Open it to external processes: a pair on
a path whose driver configures diff.<driver>.process is answered by a
long-running process speaking a pkt-line protocol (following the filter
process protocol), registered at the head of the chain and consulted
before the store and before any blob is loaded.

The protocol starts with the smallest request that can carry an answer:
object names alone.  A request is the pathname and the pair's
old-oid/new-oid, with no content.  The process answers with hunk lines,
with a zero-hunk success that asserts the blobs equivalent (trailing
newlines included), or with status=need-content, on which the pair
falls through to the builtin answer.  This serves the two shapes that
need no content pushed to them: a cache keyed on the blob pair, and a
process that fetches the blobs itself (for example over "git cat-file
--batch").  A pair whose side is not a stored blob carries a NULL id;
the provider sends no request and passes it.  Because Git holds no
content for the exchange, the answer is used as sent: hunks are
validated for order, overlap, lockstep alignment, and magnitude, then
replayed without the normalization xdiff applies to diffs it computes
itself.  The magnitude bound is the blobs' sizes, read from the object
database without loading content: a blob of N bytes holds at most N
lines.

Because the process's answer is authoritative, it outranks the store,
and its head-of-chain position says so.  A pair the process answers
never reaches the store and is never recorded, so nothing it produces
enters the store, which holds the builtin answer only.  A request it
does not answer, whether need-content, a missing capability, or a
missing id, passes down the chain to the builtin answer, which is what
the store serves, so the store may serve such a pair and a warming run
may record it.  Entries recorded before a process was configured are
not purged; a pair the process answers ignores them, and "git diff-hunks
clear" discards them.

The provider gates itself per request.  The driver is looked up by the
old-side path, so a renamed file resolves to the same driver, and by
the repository-relative path, so a diff.relative run from a
subdirectory names the pair the same way.  Options the process is never
told about select no process: the whitespace-ignoring options, -I,
--anchored, and an algorithm forced by option or configuration (blame
routes its algorithm through xdl_opts, so --histogram is covered).  The
request gains its last field, the path; the consumers change only by
filling it, and neither names the process.

The provider's state is its repository's pool of running processes,
keyed by the configured command, so drivers sharing a command share a
process, a submodule speaks to its own, and releasing the provider
(from repo_clear()) stops them.  The pool owns a copy of each command
string, so an entry outlives a config re-read.  A command that fails
stays as an entry that is not retried: its request and every later one
pass, so the store may serve the path for the rest of the command.

A protocol error in a response never kills the command.  The response
is read through a packet reader gentle about framing, so an error takes
one path: a single warning, the process stopped and marked failed, and
the builtin diff for the rest of the command.  That covers garbage
bytes, a truncated response, an empty packet, a bare status, and an
unrecognized status.  Semantically invalid coordinates cost only their
pair: the response is drained, the pair is computed, and the process
stays alive.  A path the protocol cannot carry (an embedded newline, or
one too long for a packet) falls back per path rather than costing the
command its process.  The handshake keeps one fatal check: a process
that announces a capability Git did not request aborts the command, as
the long-running filter protocol does.

Consulting is allowed per command, following the allow_textconv
precedent.  "git diff", "git log" and "git show", and "git blame" set
allow_diff_process; the plumbing diff commands and the interactive-patch
machinery never set it, so scripted and staging output stays builtin.
The options adjust the flag:

- --no-ext-diff clears it and --ext-diff sets it;
- --diff-process and --no-diff-process set and clear it alone, leaving
  external diff drivers as they were;
- format-patch clears it unconditionally, so a generated patch applies
  for recipients without the process;
- range-diff passes --no-ext-diff to the "git log" it compares.

git blame and the summary formats consult the process.  For blame, a
pair reported equivalent emits no hunks, so the whole commit passes to
its parent.  In the stat formats such a pair sums to a zero-count entry,
which the "nothing changed" rule omits, as under -w.  The subprocess is
long-running: one startup cost across a traversal, one round-trip per
consulted pair.  Answers travel in struct xdl_hunk, new in
xdiff-interface.h, holding xdiff's 1-based coordinates; nothing feeds
them back to xdiff, since only coordinate consumers consult.

A content-carrying request is the natural extension: it would serve
sides that are not stored blobs and processes that want content pushed
to them, and bring patch output and log -L's range tracking to the same
answer.  As it stands, a process's answers show in blame and the summary
formats while patch output stays builtin.

t4080 exercises the protocol, the per-command gate, and the error paths:

- each adversarial response shape warns and falls back to builtin, the
  request log proving which failures disable the process and which keep
  it alive (a malformed hunk line, coordinates past the blob size, a
  count overflowing strtol(), overlapping or misaligned hunks, an
  unrecognized status, a bare status, an empty packet, a mid-response
  crash, and raw garbage);
- a capability-less process and status=abort degrade without noise, and
  a failed start warns once and returns the path to the store;
- a trailing token on a hunk line is ignored, pinning field
  appendability;
- positive consults for git diff, git show, and diff-tree under
  --ext-diff and --diff-process; textconv output and gitlink sides are
  never identified; a diff.relative run consults by the repo-relative
  path;
- the equivalence answer is pinned from both consumers, and a warming
  run past a deferring process records the pair for a later read.

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Michael Montalbo <mmontalbo@gmail.com>
---
 Documentation/config/diff.adoc           |   6 +
 Documentation/diff-algorithm-option.adoc |   3 +
 Documentation/diff-options.adoc          |  15 +-
 Documentation/gitattributes.adoc         | 160 ++++++
 Makefile                                 |   2 +
 blame.c                                  |  10 +-
 builtin/blame.c                          |   1 +
 builtin/diff.c                           |   1 +
 builtin/log.c                            |  11 +-
 diff-process.c                           | 669 +++++++++++++++++++++++
 diff-provider-internal.h                 |   1 +
 diff-provider.c                          |  12 +-
 diff-provider.h                          |  39 +-
 diff.c                                   |  42 +-
 diff.h                                   |   9 +
 meson.build                              |   1 +
 range-diff.c                             |   6 +
 t/helper/meson.build                     |   1 +
 t/helper/test-diff-process-backend.c     | 349 ++++++++++++
 t/helper/test-tool.c                     |   1 +
 t/helper/test-tool.h                     |   1 +
 t/meson.build                            |   1 +
 t/t4080-diff-process.sh                  | 593 ++++++++++++++++++++
 xdiff-interface.h                        |  12 +
 24 files changed, 1916 insertions(+), 30 deletions(-)
 create mode 100644 diff-process.c
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/t4080-diff-process.sh

diff --git a/Documentation/config/diff.adoc b/Documentation/config/diff.adoc
index 1135a62a0a..349bdbe492 100644
--- a/Documentation/config/diff.adoc
+++ b/Documentation/config/diff.adoc
@@ -218,6 +218,12 @@ endif::git-diff[]
 	Set this option to `true` to make the diff driver cache the text
 	conversion outputs.  See linkgit:gitattributes[5] for details.
 
+`diff.<driver>.process`::
+	The command to run as a long-running process that answers
+	which line ranges changed between two blobs.  See
+	linkgit:gitattributes[5] for the protocol and when it is
+	consulted.
+
 `diff.indentHeuristic`::
 	Set this option to `false` to disable the default heuristics
 	that shift diff hunk boundaries to make patches easier to read.
diff --git a/Documentation/diff-algorithm-option.adoc b/Documentation/diff-algorithm-option.adoc
index 8e3a0b63d7..16e6fc7261 100644
--- a/Documentation/diff-algorithm-option.adoc
+++ b/Documentation/diff-algorithm-option.adoc
@@ -18,3 +18,6 @@
 For instance, if you configured the `diff.algorithm` variable to a
 non-default value and want to use the default one, then you
 have to use `--diff-algorithm=default` option.
++
+Explicitly choosing a diff algorithm on the command line also
+bypasses `diff.<driver>.process` (see linkgit:gitattributes[5]).
diff --git a/Documentation/diff-options.adoc b/Documentation/diff-options.adoc
index c8242e2462..cf359d88b4 100644
--- a/Documentation/diff-options.adoc
+++ b/Documentation/diff-options.adoc
@@ -833,7 +833,20 @@ endif::git-format-patch[]
 	to use this option with linkgit:git-log[1] and friends.
 
 `--no-ext-diff`::
-	Disallow external diff drivers.
+	Disallow external diff drivers and processes, including
+	`diff.<driver>.command` and `diff.<driver>.process`
+	(see linkgit:gitattributes[5]).
+
+`--diff-process`::
+`--no-diff-process`::
+	Allow (or forbid) consulting a diff process configured with
+	++diff.++__<driver>__++.process++ (see linkgit:gitattributes[5]),
+	leaving external diff drivers unaffected.  `git diff`, `git log`,
+	`git show`, and `git blame` allow consulting by default; the
+	plumbing diff commands forbid it unless this option or
+	`--ext-diff` is given.  linkgit:git-format-patch[1] accepts the
+	option but ignores it: a generated patch is always based on the
+	builtin diff.
 
 `--textconv`::
 `--no-textconv`::
diff --git a/Documentation/gitattributes.adoc b/Documentation/gitattributes.adoc
index da773e2924..dd4fa0ad2d 100644
--- a/Documentation/gitattributes.adoc
+++ b/Documentation/gitattributes.adoc
@@ -832,6 +832,166 @@ NOTE: If `diff.<name>.command` is defined for path with the
 (see above), and adding `diff.<name>.algorithm` has no effect, as the
 algorithm is not passed to the external diff driver.
 
+Answering diffs from a long-running process
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Unlike `diff.<name>.command`, which replaces the textual patch, the
+process configured in `diff.<name>.process` feeds hunks back into
+Git's own machinery:
+it answers "which line ranges changed between these two blobs", and
+Git's output is produced from that answer.  A process is started
+lazily, once per configured command string and per repository, and
+consulted over a pkt-line protocol (following the long-running filter
+process protocol; see the "Long Running Filter Process" section above
+for the filter analogue).
+
+The process is asked by object names alone: a request carries the
+pathname and the `old-oid`/`new-oid` of the blob pair, and no content.
+The pathname is the repository-relative old-side (preimage) path, not
+the shortened display path a `--relative` diff shows, so a driver
+scoped to a directory matches whatever directory the command runs from.
+This suits a process that keeps a persistent cache keyed on the pair, and
+a process that fetches the blobs itself (for example via
+`git cat-file --batch`) to compute its own notion of the changed
+lines.  Pairs with a side that is not a stored blob (a working-tree
+file, textconv output) are not sent; Git computes those itself.
+
+The exchange opens with a handshake: Git announces its role and
+version, the process replies in kind, and then Git lists the
+capabilities it supports and the process replies with the ones it
+implements.  A process that announces a capability Git did not list
+aborts the command.
+
+-----------------------
+packet:          git> git-diff-client
+packet:          git> version=1
+packet:          git> 0000
+packet:          git< git-diff-server
+packet:          git< version=1
+packet:          git< 0000
+packet:          git> capability=hunks-by-oid
+packet:          git> 0000
+packet:          git< capability=hunks-by-oid
+packet:          git< 0000
+-----------------------
+
+After the handshake, each request and response looks like:
+
+-----------------------
+packet:          git> command=hunks-by-oid
+packet:          git> pathname=path/file.c
+packet:          git> old-oid=<hex>
+packet:          git> new-oid=<hex>
+packet:          git> 0000
+packet:          git< hunk <old_start> <old_count> <new_start> <new_count>
+packet:          git< 0000
+packet:          git< status=success
+packet:          git< 0000
+-----------------------
+
+Start values are 1-based and counts are non-negative; a count of 0
+describes a pure insertion or deletion at the 1-based line the change
+sits before (a start of 0 is accepted for an empty file side).  Hunks
+must be listed in order, must not overlap, and must keep the unchanged
+runs between them the same length on both sides; Git validates this
+and, with a warning, falls back to its builtin diff on a response
+that violates these rules.
+
+A `status=success` response with zero hunks asserts that the blobs are
+equivalent, including their trailing newlines.  A process that cannot
+answer a pair from its object names (or cannot rule out a
+trailing-newline-only difference) responds `status=need-content`, and
+Git produces that pair's diff itself.  An asserted equivalence makes
+the pair vanish from the summary formats, but the pair still counts
+as changed for `--exit-code`, the same way a whitespace-only pair
+does under `-w`.
+
+The status names the disposition of the whole request.  Git
+understands three: `success` (the hunk lines are the answer),
+`need-content` (Git produces this pair's diff itself), and `abort`, which
+withdraws the capability the request used: Git stops sending
+`hunks-by-oid` requests to that process for the rest of the command,
+while the process stays alive for request forms negotiated under
+other capabilities.  Any other status is a protocol error: Git warns,
+stops the process, and uses the builtin diff for the remainder of the
+command.
+
+Every response has the same shape whatever its status: zero or more
+hunk lines, a flush packet, and a status packet terminated with a
+flush packet.  A response that carries no hunks, `need-content`
+included, still begins with the empty hunk section's flush packet; a
+bare status packet is a protocol error:
+
+-----------------------
+packet:          git< 0000
+packet:          git< status=need-content
+packet:          git< 0000
+-----------------------
+
+The process must read the entire request before it responds; Git
+writes the whole request before it reads the response.
+
+The protocol extends without breaking deployed processes: a process
+must ignore request keys it does not recognize, and Git ignores
+trailing space-separated tokens after the last field of a hunk line,
+so a later protocol version can append request keys and hunk fields.
+New request forms arrive as capabilities, which a process may decline
+to announce; announcing a capability Git did not request aborts the
+command, as it does under the long-running filter process protocol.
+
+There is no shutdown handshake: Git's side of the pipes closes when
+the command exits, and the process should exit when it reads EOF.  No
+flush point is guaranteed, so a process that maintains persistent
+state (such as a cache) should persist as it answers rather than at
+exit.  Git applies no timeout to a response; a process that hangs
+hangs the command, as with the long-running filter processes.
+
+`git blame` and the `--stat`, `--numstat`, and `--shortstat` formats
+consult the process; the textual patch and `git log -L` range
+tracking are produced by the builtin machinery, so a process whose
+answers deliberately differ from the builtin diff shows that
+difference only in blame and those formats.  `--dirstat=lines` routes
+through the diffstat path and consults; the other `--dirstat` modes do
+not.  A merge's `--stat` (including under `--cc`) is computed against
+the first parent, so it consults like any other stat; the combined
+patch itself compares one merge result against all of its parents at
+once, which the pairwise request above does not express, so that patch
+uses the builtin diff, and extending the protocol to combined diffs is
+left for future work.  A content-carrying extension of this
+protocol would bring patch output and `git log -L` range tracking to
+the same answer.
+
+Consulting is allowed per command, as with textconv: `git diff`,
+`git log` (`git whatchanged` included) and `git show`, and
+`git blame` consult a configured process; the plumbing diff commands
+do not unless `--ext-diff` or `--diff-process` is given explicitly,
+and the interactive-patch commands (`git add -p` and friends), which
+build the hunks they present from plumbing output, always stage from
+the builtin diff.  `git range-diff` generates the patches it
+compares with `--no-ext-diff`.  `--diff-process` and
+`--no-diff-process` allow or forbid only the consulting;
+`--no-ext-diff` disables all external diff mechanisms, this one
+included.  Options the process is never told about never select it:
+with the whitespace-ignoring options, `--ignore-matching-lines`, and
+`--anchored`, the pair is answered as when no process is configured.
+`--diff-algorithm` (or a configured `diff.algorithm`) forces a builtin
+algorithm and bypasses the process the same way.  A per-path
+++diff.++__<driver>__++.algorithm++ does so for `git diff` and the stat
+formats, which build their diff parameters from it; `git blame` builds
+its parameters from its own diff options, so a per-driver algorithm
+does not by itself keep blame from consulting the process.
+`git format-patch` never
+consults the process, so generated patches are always based on the
+builtin diff and apply for recipients without the process.  On a
+path whose driver has a process, the process is consulted before the
+diff-hunks store (see linkgit:git-diff-hunks[1]): a pair the process
+answers is never served from the store and never recorded into it.
+A pair the process does not answer, for example with
+`status=need-content`, gets the builtin diff, so the store may serve
+it and a warming run may record it: the store holds builtin results,
+and for such a pair the builtin result is what would be computed
+anyway.
+
 Defining a custom hunk-header
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
diff --git a/Makefile b/Makefile
index 11a06934b3..853e3cdacd 100644
--- a/Makefile
+++ b/Makefile
@@ -819,6 +819,7 @@ TEST_BUILTINS_OBJS += test-csprng.o
 TEST_BUILTINS_OBJS += test-date.o
 TEST_BUILTINS_OBJS += test-delete-gpgsig.o
 TEST_BUILTINS_OBJS += test-delta.o
+TEST_BUILTINS_OBJS += test-diff-process-backend.o
 TEST_BUILTINS_OBJS += test-dir-iterator.o
 TEST_BUILTINS_OBJS += test-drop-caches.o
 TEST_BUILTINS_OBJS += test-dump-cache-tree.o
@@ -1151,6 +1152,7 @@ LIB_OBJS += diff-delta.o
 LIB_OBJS += diff-merges.o
 LIB_OBJS += diff-lib.o
 LIB_OBJS += diff-no-index.o
+LIB_OBJS += diff-process.o
 LIB_OBJS += diff-provider.o
 LIB_OBJS += diff.o
 LIB_OBJS += diffcore-break.o
diff --git a/blame.c b/blame.c
index 7d7671ef5d..3189bcdaba 100644
--- a/blame.c
+++ b/blame.c
@@ -2010,17 +2010,25 @@ static void pass_blame_to_parent(struct blame_scoreboard *sb,
 	 * compute.  The working-tree/--contents pseudo-commit (marked by
 	 * its null commit id) holds a blob that is not a stored object,
 	 * so its pairs withhold identity too: no id may be sent that
-	 * names bytes a provider cannot look up.
+	 * names bytes a process cannot look up.
 	 */
 	provider_usable = !sb->reverse && !ignore_diffs &&
 		!is_null_oid(&target->commit->object.oid) &&
 		!blame_textconv_active(sb, target->path) &&
 		!blame_textconv_active(sb, parent->path);
 
+	/*
+	 * Look up the driver by the parent (old) path, as builtin_diff()
+	 * does with name_a, so a renamed file resolves to the same driver
+	 * across diff and blame.  A process that reports a pair
+	 * equivalent emits no hunks, so blame passes the whole commit
+	 * through and looks past it.
+	 */
 	if (provider_usable) {
 		req.old_oid = &parent->blob_oid;
 		req.new_oid = &target->blob_oid;
 	}
+	req.path = parent->path;
 	req.diffopt = &sb->revs->diffopt;
 	if (diff_provider_emit_hunks(&req, blame_diff_fill, &fill_data,
 				     blame_chunk_cb, &d) == DIFF_PROVIDER_ERROR)
diff --git a/builtin/blame.c b/builtin/blame.c
index 7891d82ae6..13e3ff9e36 100644
--- a/builtin/blame.c
+++ b/builtin/blame.c
@@ -1025,6 +1025,7 @@ int cmd_blame(int argc,
 	repo_init_revisions(the_repository, &revs, NULL);
 	revs.date_mode = blame_date_mode;
 	revs.diffopt.flags.allow_textconv = 1;
+	revs.diffopt.flags.allow_diff_process = 1;
 	revs.diffopt.flags.follow_renames = 1;
 
 	save_commit_buffer = 0;
diff --git a/builtin/diff.c b/builtin/diff.c
index a2ad63ac8c..a39ffe69a4 100644
--- a/builtin/diff.c
+++ b/builtin/diff.c
@@ -510,6 +510,7 @@ int cmd_diff(int argc,
 	init_diffstat_widths(&rev.diffopt);
 	rev.diffopt.flags.allow_external = 1;
 	rev.diffopt.flags.allow_textconv = 1;
+	rev.diffopt.flags.allow_diff_process = 1;
 
 	/* If this is a no-index diff, just run it and exit there. */
 	if (no_index)
diff --git a/builtin/log.c b/builtin/log.c
index 0b72477f56..59ecd34334 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -209,6 +209,7 @@ static void cmd_log_init_defaults(struct rev_info *rev,
 	init_diffstat_widths(&rev->diffopt);
 	rev->diffopt.flags.recursive = 1;
 	rev->diffopt.flags.allow_textconv = 1;
+	rev->diffopt.flags.allow_diff_process = 1;
 	rev->abbrev_commit = cfg->default_abbrev_commit;
 	rev->show_root_diff = cfg->default_show_root;
 	rev->subject_prefix = cfg->fmt_patch_subject_prefix;
@@ -2226,11 +2227,15 @@ int cmd_format_patch(int argc,
 		die(_("unrecognized argument: %s"), argv[1]);
 
 	/*
-	 * A patch generated by format-patch carries the builtin diffstat,
-	 * not one served from a local store, so its counts do not depend
-	 * on whether the sender warmed the store.
+	 * Patches generated by format-patch must be based on the builtin
+	 * diff so recipients without the store or the process can apply
+	 * them, and so the emitted diffstat does not depend on the sender's
+	 * local cache: the precomputed-hunks store is not consulted for the
+	 * diffstat, and the diff process is not consulted even when
+	 * --ext-diff enables the external diff command.
 	 */
 	rev.diffopt.flags.no_precomputed_hunks = 1;
+	rev.diffopt.flags.allow_diff_process = 0;
 
 	if (rev.diffopt.output_format & DIFF_FORMAT_NAME)
 		die(_("--name-only does not make sense"));
diff --git a/diff-process.c b/diff-process.c
new file mode 100644
index 0000000000..121903a6c8
--- /dev/null
+++ b/diff-process.c
@@ -0,0 +1,669 @@
+/*
+ * The process provider of the hunk provider interface: consult a
+ * long-running external process via the pkt-line protocol for the
+ * hunks of a blob pair.  The process answers from the pair's object
+ * names alone: it can serve a persistent cache keyed on the pair, or
+ * fetch the blobs from the repository itself (e.g. via "git cat-file
+ * --batch") and compute its own notion of which lines changed.  The
+ * provider sits at the head of its repository's chain and gates
+ * itself per request; its state is the repository's pool of running
+ * processes, one per configured command, stopped when the provider
+ * is released.
+ *
+ * Protocol: pkt-line over stdin/stdout, following the pattern of
+ * the long-running filter process protocol (see convert.c).
+ *
+ * Handshake:
+ *   git> git-diff-client / version=1 / flush
+ *   process< git-diff-server / version=1 / flush
+ *   git> capability=hunks-by-oid / flush
+ *   process< capability=hunks-by-oid / flush
+ *
+ * Per-pair, when both sides are stored blobs:
+ *   git> command=hunks-by-oid / pathname=<path>
+ *   git> old-oid=<hex> / new-oid=<hex> / flush
+ *   process< hunk <old_start> <old_count> <new_start> <new_count>
+ *   process< ... / flush
+ *   process< status=success / flush
+ *
+ * No content is sent.  Because Git holds no content for the exchange,
+ * the answer is used as the process sent it: the hunks are not re-run
+ * through xdiff's compaction, and a status=success response with zero
+ * hunks asserts that the blobs are equivalent, including their
+ * trailing newlines.  A process that cannot answer from the object names
+ * (or cannot rule out a trailing-newline-only difference) responds
+ * status=need-content; the pair then gets the builtin answer, served
+ * from the diff-hunks store or computed.  A later
+ * protocol extension can define a content-carrying request for such
+ * processes and for sides that are not stored blobs.
+ */
+
+#include "git-compat-util.h"
+#include "diff.h"
+#include "diff-provider-internal.h"
+#include "gettext.h"
+#include "hex.h"
+#include "odb.h"
+#include "repository.h"
+#include "sigchain.h"
+#include "userdiff.h"
+#include "sub-process.h"
+#include "pkt-line.h"
+#include "strbuf.h"
+
+#define CAP_OID_HUNKS (1u << 0)
+
+/*
+ * The provider's state: the repository's diff processes, keyed by
+ * their command string, so drivers that configure the same command
+ * share one process.  An entry whose process failed stays in the
+ * pool with the failed bit set, so the command is not retried while
+ * the entry lives; the pool and its entries last until the provider
+ * is released.
+ */
+struct diff_process_state {
+	struct hashmap subprocesses;
+};
+
+struct diff_subprocess {
+	struct subprocess_entry subprocess;
+	/*
+	 * Owns the string subprocess.cmd and the hashmap key borrow: the
+	 * entry outlives the userdiff config a re-read may replace.
+	 */
+	char *cmd;
+	unsigned int supported_capabilities;
+	unsigned failed : 1;
+};
+
+static int start_diff_process_fn(struct subprocess_entry *subprocess)
+{
+	static int versions[] = { 1, 0 };
+	static struct subprocess_capability capabilities[] = {
+		{ "hunks-by-oid", CAP_OID_HUNKS },
+		{ NULL, 0 }
+	};
+	struct diff_subprocess *entry =
+		container_of(subprocess, struct diff_subprocess, subprocess);
+
+	return subprocess_handshake(subprocess, "git-diff",
+				    versions, NULL,
+				    capabilities,
+				    &entry->supported_capabilities);
+}
+
+/*
+ * The pool entry for a command, or NULL when its process fails to
+ * start here: the failure leaves a failed entry in the pool, so only
+ * the request that observed it maps it to an error and later
+ * requests pass the provider by.
+ */
+static struct diff_subprocess *get_or_launch_process(
+		struct diff_process_state *state,
+		struct userdiff_driver *drv)
+{
+	struct subprocess_entry *running;
+	struct diff_subprocess *entry;
+
+	running = subprocess_find_entry(&state->subprocesses, drv->process);
+	if (running) {
+		entry = container_of(running, struct diff_subprocess,
+				     subprocess);
+		return entry->failed ? NULL : entry;
+	}
+
+	entry = xcalloc(1, sizeof(*entry));
+	entry->cmd = xstrdup(drv->process);
+	if (subprocess_start_command(&entry->subprocess, entry->cmd,
+				     start_diff_process_fn))
+		entry->failed = 1;
+	hashmap_entry_init(&entry->subprocess.ent, strhash(entry->cmd));
+	hashmap_add(&state->subprocesses, &entry->subprocess.ent);
+	if (entry->failed) {
+		warning(_("diff process '%s' failed to start;"
+			  " using the builtin diff"), drv->process);
+		return NULL;
+	}
+	return entry;
+}
+
+/*
+ * A hunk in the diff process's presentation coordinates: the line
+ * numbering it reports over the protocol.  Kept distinct from struct
+ * xdl_hunk (xdiff's coordinates) so that only translated hunks ever
+ * reach a consumer; diff_process_hunk_to_xdl() is the single
+ * crossing point.
+ */
+struct diff_process_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
+/*
+ * Parse one non-negative decimal field of a hunk line into *out and
+ * advance *line past it.  Fields must be plain decimal with no leading
+ * whitespace or sign (isdigit() takes an unsigned char to stay defined
+ * for high-bit bytes).  The first three fields are followed by a single
+ * space; the last (is_last) is followed by end-of-string or a space.
+ * Trailing space-separated tokens after the last field are allowed and
+ * ignored, so a future protocol version can append fields (e.g. a
+ * "moved" marker) without an older Git rejecting the line, mirroring
+ * the request-side rule that processes ignore unknown keys.
+ *
+ * A value that overflows strtol() is not a parse failure: the line is
+ * well-formed, so the stream stays in protocol sync.  It is reported
+ * through *out_of_range, and the caller skips the pair the same way
+ * it skips any other out-of-range coordinate.
+ */
+static int parse_hunk_field(const char **line, long *out, int is_last,
+			    int *out_of_range)
+{
+	const char *p = *line;
+	char *end;
+
+	if (!isdigit((unsigned char)*p))
+		return -1;
+	errno = 0;
+	*out = strtol(p, &end, 10);
+	if (end == p)
+		return -1;
+	if (errno == ERANGE)
+		*out_of_range = 1;
+	else if (errno)
+		return -1;
+	if (is_last) {
+		if (*end != '\0' && *end != ' ')
+			return -1;
+	} else {
+		if (*end != ' ')
+			return -1;
+		end++;
+	}
+	*line = end;
+	return 0;
+}
+
+static int parse_hunk_line(const char *line,
+			   struct diff_process_hunk *presented,
+			   int *out_of_range)
+{
+	*out_of_range = 0;
+	/* Format: "hunk <old_start> <old_count> <new_start> <new_count>" */
+	if (!skip_prefix(line, "hunk ", &line))
+		return -1;
+	if (parse_hunk_field(&line, &presented->old_start, 0, out_of_range) ||
+	    parse_hunk_field(&line, &presented->old_count, 0, out_of_range) ||
+	    parse_hunk_field(&line, &presented->new_start, 0, out_of_range) ||
+	    parse_hunk_field(&line, &presented->new_count, 1, out_of_range))
+		return -1;
+	return 0;
+}
+
+/*
+ * Translate a hunk from the diff process's presentation coordinates
+ * into xdiff's.
+ *
+ * Protocol starts are already 1-based positions (the line a change
+ * sits before), the same numbering xdiff uses, so the only adjustment
+ * is for an empty file side: "git diff" addresses it with a start of 0
+ * and a count of 0 (e.g. "0 0 1 5" adds five lines to an empty old
+ * side), and since xdiff uses start-1 as an array index that 0 becomes
+ * 1 here.  This is NOT the full inverse of xdl_emit_hunk_hdr()
+ * (xdiff/xutils.c): that emitter shifts a count-0 range to start-1 for
+ * the displayed "@@" header, but the protocol keeps the unshifted
+ * 1-based position for a mid-file insert or delete.  This is the single
+ * point where presentation coordinates become xdiff coordinates, so
+ * any consumer of these coordinates may assume 1-based starts.
+ *
+ * Returns -1 for a start of 0 paired with a nonzero count, which names
+ * no line in either coordinate system.  (parse_hunk_line() already
+ * guarantees non-negative starts and counts.)
+ */
+static int diff_process_hunk_to_xdl(const struct diff_process_hunk *presented,
+				    struct xdl_hunk *xdl)
+{
+	long old_start = presented->old_start;
+	long new_start = presented->new_start;
+
+	if ((!old_start && presented->old_count) ||
+	    (!new_start && presented->new_count))
+		return -1;
+	if (!old_start)
+		old_start = 1;
+	if (!new_start)
+		new_start = 1;
+
+	xdl->old_start = old_start;
+	xdl->old_count = presented->old_count;
+	xdl->new_start = new_start;
+	xdl->new_count = presented->new_count;
+	return 0;
+}
+
+/*
+ * Validate the process's hunks (already in xdiff coordinates) before they
+ * bypass the diff algorithm.  The content-independent rules (in-order,
+ * non-overlapping, lockstep-aligned, int32-bounded coordinates) are the
+ * provider interface's shared rule, diff_provider_check_hunk(); this
+ * function adds the two checks that need the blobs' line counts (a hunk
+ * past the end of a file, the run after the last hunk) and the
+ * per-rule diagnostics naming the process.  On a bad response we warn
+ * and the caller falls back to the builtin diff.  Returns 0 if valid,
+ * -1 (after warning) otherwise.
+ *
+ * old_lines/new_lines bound the line count of each side, or are
+ * negative when no bound is known.  An oid-only answer arrives without
+ * content, so its caller passes upper bounds derived from the blobs'
+ * byte sizes, which caps coordinate magnitude but cannot support the
+ * run-after-the-last-hunk check: that one compares exact line counts,
+ * so it runs only when lines_exact is set, which no caller does today.
+ * It is kept for a content-carrying request, whose loaded buffers
+ * would provide exact counts.
+ */
+static int validate_external_hunks(const struct xdl_hunk *hunks, size_t nr,
+				   long old_lines, long new_lines,
+				   int lines_exact,
+				   const char *process, const char *path)
+{
+	struct diff_provider_hunks_check c = { 0 };
+	size_t i;
+
+	for (i = 0; i < nr; i++) {
+		const struct xdl_hunk *h = &hunks[i];
+
+		if (old_lines >= 0 &&
+		    (h->old_count > old_lines - h->old_start + 1 ||
+		     h->new_count > new_lines - h->new_start + 1)) {
+			warning(_("diff process '%s' returned a hunk past the "
+				  "end of '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+		switch (diff_provider_check_hunk(&c, h->old_start,
+						  h->old_count, h->new_start,
+						  h->new_count)) {
+		case DIFF_PROVIDER_HUNKS_OK:
+			break;
+		case DIFF_PROVIDER_HUNKS_RANGE:
+			warning(_("diff process '%s' returned out-of-range "
+				  "coordinates for '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		case DIFF_PROVIDER_HUNKS_OVERLAP:
+			warning(_("diff process '%s' returned overlapping hunks "
+				  "for '%s'; using the builtin diff"),
+				process, path);
+			return -1;
+		case DIFF_PROVIDER_HUNKS_MISALIGNED:
+			warning(_("diff process '%s' returned hunks that leave "
+				  "'%s' misaligned; using the builtin diff"),
+				process, path);
+			return -1;
+		}
+	}
+	if (lines_exact &&
+	    old_lines - c.prev_old_end != new_lines - c.prev_new_end) {
+		warning(_("diff process '%s' returned hunks that leave '%s' "
+			  "misaligned; using the builtin diff"),
+			process, path);
+		return -1;
+	}
+	return 0;
+}
+
+/*
+ * The most lines a blob can hold, from its size alone: every line,
+ * even an empty one, costs at least one byte, so a blob of N bytes
+ * holds at most N lines.  Returns -1 when the size is unavailable,
+ * leaving the response bounded only by the shared int32 rule.  A size
+ * beyond INT32_MAX clamps to it, which loses nothing: a coordinate
+ * that large fails the shared rule anyway.  In a partial clone the
+ * size lookup must not fetch the blob from the promisor remote:
+ * validating an answer that exists to avoid loading content must not
+ * itself download that content, so a missing blob reads as size
+ * unavailable instead.
+ */
+static long blob_line_cap(struct repository *r, const struct object_id *oid)
+{
+	unsigned long size;
+	struct object_info oi = OBJECT_INFO_INIT;
+
+	oi.sizep = &size;
+	if (odb_read_object_info_extended(r->objects, oid, &oi,
+					  OBJECT_INFO_SKIP_FETCH_OBJECT) < 0)
+		return -1;
+	if (size > INT32_MAX)
+		return INT32_MAX;
+	return (long)size;
+}
+
+/*
+ * The driver whose process a consultation for path would ask, or NULL
+ * when none applies (no driver, process not allowed, or xpp carries
+ * options the process is never told about).  Needs no content, so
+ * the driver is picked before any blob is loaded.
+ */
+static struct userdiff_driver *diff_process_driver(struct diff_options *diffopt,
+						   const char *path,
+						   const xpparam_t *xpp)
+{
+	struct userdiff_driver *drv;
+
+	if (!diffopt || !path)
+		return NULL;
+	if (!diffopt->flags.allow_diff_process || diffopt->ignore_driver_algorithm)
+		return NULL;
+	/*
+	 * Whitespace-ignoring, regex-ignore (-I) and anchored options
+	 * change which lines count as different, but the process is never
+	 * told about them, so its hunks could not honor them.  A forced
+	 * diff algorithm (an option or configured algorithm setting)
+	 * requests a specific builtin computation, which an
+	 * authoritative answer would override.  Rather than silently
+	 * override the user's request, fall back to the builtin diff,
+	 * which does honor these flags.  Key this off xpp (the
+	 * parameters this diff actually runs with) rather than diffopt,
+	 * so a caller like blame, which keeps its algorithm and
+	 * whitespace flags outside diffopt, is covered without a
+	 * separate guard of its own.
+	 */
+	if ((xpp->flags & (XDF_WHITESPACE_FLAGS | XDF_IGNORE_BLANK_LINES |
+			   XDF_DIFF_ALGORITHM_MASK)) ||
+	    xpp->ignore_regex_nr || xpp->anchors_nr)
+		return NULL;
+
+	/*
+	 * A path the protocol cannot carry never selects a process: an
+	 * embedded newline would let the rest of the path forge further
+	 * request keys, and the pathname must fit one packet.  Passing
+	 * here keeps the cost local to the path; a failed write would
+	 * instead cost the whole command its process.
+	 */
+	if (strchr(path, '\n') ||
+	    strlen(path) > LARGE_PACKET_DATA_MAX - strlen("pathname=\n"))
+		return NULL;
+
+	drv = userdiff_find_by_path(diffopt->repo->index, path);
+	if (!drv || !drv->process)
+		return NULL;
+	return drv;
+}
+
+/*
+ * Without content there is no size-derived bound on a response, so cap
+ * accumulation at a constant instead.  A response that exceeds the
+ * cap is a protocol error: the process is disabled for the rest of
+ * the command and the caller falls back to the builtin diff.
+ */
+#define OID_HUNKS_MAX (1 << 20)
+
+enum diff_process_result {
+	DIFF_PROCESS_ERROR = -1, /* failed; caller falls back to builtin */
+	DIFF_PROCESS_OK = 0,     /* the process supplied hunks */
+	DIFF_PROCESS_SKIP,       /* process did not apply: use builtin */
+	DIFF_PROCESS_EQUIVALENT, /* process says files are equivalent */
+};
+
+/*
+ * Ask drv's diff process to answer the request from the blob pair's
+ * object ids alone (the "hunks-by-oid" capability): no content is
+ * loaded or sent.  On DIFF_PROCESS_OK the process's hunks are emitted
+ * through hunk_cb in 0-based emission coordinates, validated for order,
+ * overlap, and lockstep alignment first; because Git holds no content,
+ * the answer is used as the process sent it, without xdiff's compaction.
+ * DIFF_PROCESS_EQUIVALENT means the process asserts the pair equal.
+ * DIFF_PROCESS_SKIP covers everything that should fall through to the
+ * builtin computation: a missing capability, a missing object id, a
+ * status=need-content answer, or an invalid response.
+ */
+static enum diff_process_result diff_process_query_hunks(
+		struct diff_process_state *state,
+		struct userdiff_driver *drv,
+		const struct diff_provider_request *req,
+		xdl_emit_hunk_consume_func_t hunk_cb,
+		void *cb_data)
+{
+	const char *path = req->path;
+	struct diff_subprocess *entry;
+	struct child_process *process;
+	int fd_in, fd_out;
+	struct packet_reader reader;
+	struct strbuf status = STRBUF_INIT;
+	struct xdl_hunk *hunks = NULL;
+	struct diff_process_hunk presented;
+	struct xdl_hunk hunk;
+	size_t nr_hunks = 0, alloc_hunks = 0, i;
+	int bad_coords = 0;
+	long old_cap, new_cap;
+	enum diff_process_result res;
+
+	if (!req->old_oid || !req->new_oid)
+		return DIFF_PROCESS_SKIP;
+
+	entry = get_or_launch_process(state, drv);
+	if (!entry)
+		return DIFF_PROCESS_ERROR;
+	if (!(entry->supported_capabilities & CAP_OID_HUNKS))
+		return DIFF_PROCESS_SKIP;
+
+	process = subprocess_get_child_process(&entry->subprocess);
+	fd_in = process->in;
+	fd_out = process->out;
+
+	sigchain_push(SIGPIPE, SIG_IGN);
+
+	if (packet_write_fmt_gently(fd_in, "command=hunks-by-oid\n") ||
+	    packet_write_fmt_gently(fd_in, "pathname=%s\n", path) ||
+	    packet_write_fmt_gently(fd_in, "old-oid=%s\n",
+				    oid_to_hex(req->old_oid)) ||
+	    packet_write_fmt_gently(fd_in, "new-oid=%s\n",
+				    oid_to_hex(req->new_oid)) ||
+	    packet_flush_gently(fd_in))
+		goto comm_error;
+
+	packet_reader_init(&reader, fd_out, NULL, 0,
+			   PACKET_READ_CHOMP_NEWLINE |
+			   PACKET_READ_GENTLE_ON_EOF |
+			   PACKET_READ_GENTLE_ON_READ_ERROR);
+	for (;;) {
+		enum packet_read_status rs = packet_reader_read(&reader);
+		int out_of_range;
+
+		if (rs == PACKET_READ_FLUSH)
+			break;
+		/*
+		 * Only a hunk line may precede the flush.  EOF and a
+		 * malformed frame end the session; an empty packet, which
+		 * a length-only read cannot tell from a flush, would
+		 * truncate the hunk section here and leave the status
+		 * section to poison the next request, so it is a protocol
+		 * error too.
+		 */
+		if (rs != PACKET_READ_NORMAL || !reader.pktlen)
+			goto comm_error;
+		if (parse_hunk_line(reader.line, &presented,
+				    &out_of_range) < 0)
+			goto comm_error;
+		if (bad_coords)
+			continue;
+		if (out_of_range ||
+		    diff_process_hunk_to_xdl(&presented, &hunk) < 0) {
+			/*
+			 * Semantically invalid coordinates in a well-formed
+			 * response: the stream stays in protocol sync, so
+			 * drain the rest and fall back for this file while
+			 * keeping the process alive, the same treatment
+			 * validate_external_hunks() failures receive.
+			 */
+			bad_coords = 1;
+			continue;
+		}
+		if (nr_hunks >= OID_HUNKS_MAX) {
+			warning(_("diff process '%s' sent too many hunks"
+				  " for '%s'; disabling it for the"
+				  " remainder of this command"),
+				drv->process, path);
+			goto disable;
+		}
+		ALLOC_GROW(hunks, nr_hunks + 1, alloc_hunks);
+		hunks[nr_hunks++] = hunk;
+	}
+
+	if (subprocess_read_status_gently(fd_out, &status))
+		goto comm_error;
+
+	if (!strcmp(status.buf, "success")) {
+		if (bad_coords) {
+			warning(_("diff process '%s' returned out-of-range "
+				  "coordinates for '%s'; using the builtin diff"),
+				drv->process, path);
+			res = DIFF_PROCESS_SKIP;
+			goto out;
+		}
+		if (!nr_hunks) {
+			res = DIFF_PROCESS_EQUIVALENT;
+			goto out;
+		}
+		/*
+		 * Bound the coordinates by the blobs' sizes, read from the
+		 * object database without loading content.  Either both
+		 * bounds hold or neither is applied: a partial bound would
+		 * misclassify a response that the other side's size would
+		 * have caught.
+		 */
+		old_cap = blob_line_cap(req->repo, req->old_oid);
+		new_cap = blob_line_cap(req->repo, req->new_oid);
+		if (old_cap < 0 || new_cap < 0)
+			old_cap = new_cap = -1;
+		if (validate_external_hunks(hunks, nr_hunks, old_cap, new_cap,
+					    0, drv->process, path) < 0) {
+			res = DIFF_PROCESS_SKIP;
+			goto out;
+		}
+		/*
+		 * Replay in the coordinates a hunk consumer receives from
+		 * xdiff's emission: 0-based starts.  The answer is used as
+		 * the process sent it; with no content in hand it cannot be
+		 * re-run through xdiff's compaction.
+		 */
+		for (i = 0; i < nr_hunks; i++)
+			hunk_cb(hunks[i].old_start - 1, hunks[i].old_count,
+				hunks[i].new_start - 1, hunks[i].new_count,
+				cb_data);
+		res = DIFF_PROCESS_OK;
+		goto out;
+	}
+	if (!strcmp(status.buf, "need-content")) {
+		/*
+		 * The process cannot answer this pair from its object names;
+		 * the caller computes the diff itself.
+		 */
+		res = DIFF_PROCESS_SKIP;
+		goto out;
+	}
+	if (!strcmp(status.buf, "abort")) {
+		/* The process withdrew: stop asking it for this session. */
+		entry->supported_capabilities &= ~CAP_OID_HUNKS;
+		res = DIFF_PROCESS_SKIP;
+		goto out;
+	}
+	/*
+	 * An unrecognized status is a protocol error, not a per-pair
+	 * failure: this Git did not request anything it does not know,
+	 * so the process is answering some other protocol, and asking
+	 * it again would warn on every pair of the traversal.
+	 */
+	warning(_("diff process '%s' sent unrecognized status '%s' for "
+		  "'%s'; disabling it for the remainder of this command"),
+		drv->process, status.buf, path);
+	goto disable;
+out:
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return res;
+
+comm_error:
+	warning(_("diff process '%s' failed for '%s'; disabling it"
+		  " for the remainder of this command"),
+		drv->process, path);
+disable:
+	subprocess_stop_command(&entry->subprocess);
+	entry->failed = 1;
+	free(hunks);
+	strbuf_release(&status);
+	sigchain_pop(SIGPIPE);
+	return DIFF_PROCESS_ERROR;
+}
+
+/*
+ * The process outranks every later provider through its chain
+ * position: when it answers, the walk ends, so no later provider
+ * serves the pair, and an answered pair is never recorded.  When it
+ * does not answer (it defers with need-content, lacks the
+ * capability, or failed), the caller computes the builtin diff for
+ * that pair.  The store holds builtin results and nothing else, so
+ * an identity answer for such a pair equals what the caller would
+ * compute.  Every non-answer is therefore a pass: a refusal would
+ * suppress that equal answer, and would keep a warming run from
+ * recording the builtin result the caller computes anyway.
+ */
+static enum diff_provider_disposition
+diff_process_consult(struct diff_provider *provider,
+		     const struct diff_provider_request *req,
+		     diff_provider_fill_fn fill UNUSED, void *fill_data UNUSED,
+		     xdl_emit_hunk_consume_func_t hunk_cb, void *cb_data)
+{
+	struct diff_process_state *state = provider->state;
+	struct userdiff_driver *drv;
+	struct subprocess_entry *running;
+
+	drv = diff_process_driver(req->diffopt, req->path, req->xpp);
+	if (!drv)
+		return DIFF_PROVIDER_DISP_PASS;
+	running = subprocess_find_entry(&state->subprocesses, drv->process);
+	if (running && container_of(running, struct diff_subprocess,
+				    subprocess)->failed)
+		return DIFF_PROVIDER_DISP_PASS;
+
+	switch (diff_process_query_hunks(state, drv, req,
+					 hunk_cb, cb_data)) {
+	case DIFF_PROCESS_OK:
+	case DIFF_PROCESS_EQUIVALENT:
+		return DIFF_PROVIDER_DISP_ANSWERED;
+	case DIFF_PROCESS_SKIP:
+	case DIFF_PROCESS_ERROR:
+		break;
+	}
+	return DIFF_PROVIDER_DISP_PASS;
+}
+
+static void diff_process_release(struct diff_provider *provider)
+{
+	struct diff_process_state *state = provider->state;
+	struct hashmap_iter iter;
+	struct diff_subprocess *entry;
+
+	/* A failed entry's process is already stopped or never ran. */
+	hashmap_for_each_entry(&state->subprocesses, &iter, entry,
+			       subprocess.ent) {
+		if (!entry->failed)
+			subprocess_stop_command(&entry->subprocess);
+		free(entry->cmd);
+	}
+	hashmap_clear_and_free(&state->subprocesses,
+			       struct diff_subprocess, subprocess.ent);
+	free(state);
+}
+
+struct diff_provider *diff_process_provider_new(void)
+{
+	struct diff_process_state *state = xcalloc(1, sizeof(*state));
+	struct diff_provider *p = xcalloc(1, sizeof(*p));
+
+	hashmap_init(&state->subprocesses, cmd2process_cmp, NULL, 0);
+	p->consult = diff_process_consult;
+	p->release = diff_process_release;
+	p->state = state;
+	return p;
+}
diff --git a/diff-provider-internal.h b/diff-provider-internal.h
index 8ad3e481e7..cba1fa271a 100644
--- a/diff-provider-internal.h
+++ b/diff-provider-internal.h
@@ -93,6 +93,7 @@ struct diff_provider {
  * diff-provider.c holds itself.  Each call returns a fresh provider
  * for one repository's chain.
  */
+struct diff_provider *diff_process_provider_new(void);
 struct diff_provider *diff_hunks_store_provider_new(void);
 
 /*
diff --git a/diff-provider.c b/diff-provider.c
index 66a9909eaa..c8aaf8e857 100644
--- a/diff-provider.c
+++ b/diff-provider.c
@@ -41,11 +41,11 @@ static struct diff_provider *builtin_provider_new(void)
 
 /*
  * The repository's chain, assembled on first walk.  The composition
- * is fixed, and the order is the authority resolution: the store is
- * consulted before the builtin computation, the terminal provider,
- * so the chain always ends in an implementor that can answer.
- * Nothing is decided per repository here; each provider gates itself
- * per request.
+ * is fixed, and the order is the authority resolution: the process
+ * outranks the store, and the builtin computation is the terminal
+ * provider, so the chain always ends in an implementor that can
+ * answer.  Nothing is decided per repository here; each provider
+ * gates itself per request.
  */
 static struct diff_provider *provider_chain(struct repository *r)
 {
@@ -53,6 +53,8 @@ static struct diff_provider *provider_chain(struct repository *r)
 
 	if (*tail)
 		return *tail;
+	*tail = diff_process_provider_new();
+	tail = &(*tail)->next;
 	*tail = diff_hunks_store_provider_new();
 	tail = &(*tail)->next;
 	*tail = builtin_provider_new();
diff --git a/diff-provider.h b/diff-provider.h
index c5478b5700..061e1c2f5c 100644
--- a/diff-provider.h
+++ b/diff-provider.h
@@ -11,19 +11,23 @@
  * its content is loaded.
  *
  * A hunk provider answers a consumer's request from the pair's
- * identity (its blob object ids) and the parameters that determine
- * the diff; a request no provider answers falls through to the
- * consumer's own computation.  A provider is either authoritative for
- * its requests, meaning its answer may deliberately differ from the
- * builtin diff, or not, meaning its answer must reproduce the builtin
- * result exactly.  The interface resolves that authority through a
- * provider chain owned by the repository, built on first consultation
- * and released by repo_clear(): chain order is the resolution, and
- * the builtin computation itself is the chain's terminal provider.  A
- * consumer never names a provider; it reads the outcome below.
- * Every answer a provider serves from identity passes the shared
- * coordinate check (diff-provider-internal.h) before any consumer
- * sees it.
+ * identity, its blob object ids and the settings that determine the
+ * diff, before any content is loaded; a request no provider answers
+ * falls through to the consumer's own computation.  Two providers implement this
+ * interface with different authority.  The diff-hunks store
+ * (diff-hunks.h) is in-process and not authoritative: it may only
+ * reproduce the builtin result, so it never asserts a pair
+ * equivalent, and it stands aside wherever a process outranks it.  A
+ * process configured in diff.<driver>.process (diff-process.c) is
+ * authoritative for its paths: its answer may deliberately differ
+ * from the builtin diff, including asserting a pair equivalent.  The
+ * interface resolves that authority through a provider chain owned
+ * by the repository, built on first consultation and released by
+ * repo_clear(): chain order is the resolution, and the builtin
+ * computation itself is the chain's terminal provider.  A consumer
+ * never names a provider; it reads the outcome below.  Every answer a
+ * provider serves from identity passes the shared coordinate check
+ * (diff-provider-internal.h) before any consumer sees it.
  */
 
 struct diff_options;
@@ -95,14 +99,17 @@ enum diff_provider_outcome {
  * name the blobs whose bytes are diffed; pass NULL for a side whose
  * bytes are not a stored blob (a working-tree file, textconv output,
  * a gitlink), so no provider answers from an id it cannot look up.
- * diffopt carries the diff settings that live outside xpp; xpp
- * carries the parameters the diff runs with.  Each provider gates
- * itself on the fields that concern it.
+ * path names the file the pair is diffed as; a provider selected by
+ * path applies only where it is set.  diffopt carries the diff
+ * settings that live outside xpp; xpp carries the parameters the
+ * diff runs with.  Each provider gates itself on the fields that
+ * concern it.
  */
 struct diff_provider_request {
 	struct repository *repo;
 	const struct object_id *old_oid;
 	const struct object_id *new_oid;
+	const char *path;
 	struct diff_options *diffopt;
 	const xpparam_t *xpp;
 };
diff --git a/diff.c b/diff.c
index 0d4cb3fcfd..ea79d80c26 100644
--- a/diff.c
+++ b/diff.c
@@ -4377,6 +4377,13 @@ static int diffstat_from_hunks(struct diff_options *o,
 			   &one->oid : NULL,
 		.new_oid = (two->oid_valid && !S_ISGITLINK(two->mode)) ?
 			   &two->oid : NULL,
+		/*
+		 * Attribute lookup and the process protocol need the
+		 * repo-relative path; the display name a caller passes
+		 * around may be stripped of o->prefix and would miss a
+		 * driver scoped to a directory.
+		 */
+		.path = one->path,
 		.diffopt = o,
 		.xpp = &xpp,
 	};
@@ -4491,12 +4498,14 @@ static void builtin_diffstat(const char *name_a, const char *name_b,
 
 	else if (may_differ) {
 		/*
-		 * Serve or record via the diff-hunks store. A "log -L"
+		 * Serve from a hunk provider (the process, then the store),
+		 * or record into the store on a warming run. A "log -L"
 		 * range-scoped stat is not the whole-pair diff the store
 		 * keys, so it neither reads nor records. Otherwise diff
 		 * normally.
 		 */
-		if (p->line_ranges || !diffstat_from_hunks(o, one, two, data)) {
+		if (p->line_ranges ||
+		    !diffstat_from_hunks(o, one, two, data)) {
 			/* Crazy xdl interfaces.. */
 			xpparam_t xpp;
 			xdemitconf_t xecfg;
@@ -6252,6 +6261,27 @@ static int diff_opt_submodule(const struct option *opt,
 	return 0;
 }
 
+static int diff_opt_ext_diff(const struct option *opt,
+			     const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_external = !unset;
+	options->flags.allow_diff_process = !unset;
+	return 0;
+}
+
+static int diff_opt_diff_process(const struct option *opt,
+				 const char *arg, int unset)
+{
+	struct diff_options *options = opt->value;
+
+	BUG_ON_OPT_ARG(arg);
+	options->flags.allow_diff_process = !unset;
+	return 0;
+}
+
 static int diff_opt_textconv(const struct option *opt,
 			     const char *arg, int unset)
 {
@@ -6582,8 +6612,12 @@ struct option *add_diff_options(const struct option *opts,
 			 N_("exit with 1 if there were differences, 0 otherwise")),
 		OPT_BOOL(0, "quiet", &options->flags.quick,
 			 N_("disable all output of the program")),
-		OPT_BOOL(0, "ext-diff", &options->flags.allow_external,
-			 N_("allow an external diff helper to be executed")),
+		OPT_CALLBACK_F(0, "ext-diff", options, NULL,
+			       N_("allow an external diff helper to be executed"),
+			       PARSE_OPT_NOARG, diff_opt_ext_diff),
+		OPT_CALLBACK_F(0, "diff-process", options, NULL,
+			       N_("allow a configured diff process to be consulted"),
+			       PARSE_OPT_NOARG, diff_opt_diff_process),
 		OPT_CALLBACK_F(0, "textconv", options, NULL,
 			       N_("run external text conversion filters when comparing binary files"),
 			       PARSE_OPT_NOARG, diff_opt_textconv),
diff --git a/diff.h b/diff.h
index 380a258878..e0b58a8105 100644
--- a/diff.h
+++ b/diff.h
@@ -173,6 +173,15 @@ struct diff_flags {
 	 */
 	unsigned allow_external;
 
+	/**
+	 * Allows diff.<driver>.process to be consulted.  Set by the
+	 * porcelain commands whose output may reflect a diff process
+	 * (diff, log, show, blame) and by --ext-diff or --diff-process;
+	 * plumbing does not set it by default, so its output stays
+	 * builtin.  Cleared by --no-ext-diff or --no-diff-process.
+	 */
+	unsigned allow_diff_process;
+
 	/**
 	 * For communication between the calling program and the options parser;
 	 * tell the calling program to signal the presence of difference using
diff --git a/meson.build b/meson.build
index 391d9da93c..6d6ac8e753 100644
--- a/meson.build
+++ b/meson.build
@@ -356,6 +356,7 @@ libgit_sources = [
   'diff-merges.c',
   'diff-lib.c',
   'diff-no-index.c',
+  'diff-process.c',
   'diff-provider.c',
   'diff.c',
   'diffcore-break.c',
diff --git a/range-diff.c b/range-diff.c
index 8e2dd2eb19..3cadbcd7e1 100644
--- a/range-diff.c
+++ b/range-diff.c
@@ -52,6 +52,12 @@ static int read_patches(const char *range, struct string_list *list,
 	int ret = -1;
 
 	strvec_pushl(&cp.args, "log", "--no-color", "-p",
+		     /*
+		      * The patches being compared must be the builtin
+		      * diff's: an external diff command or diff process
+		      * could change either side of the comparison.
+		      */
+		     "--no-ext-diff",
 		     "--reverse", "--date-order", "--decorate=no",
 		     "--no-prefix", "--submodule=short",
 		     /*
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 3235f10ab8..6abcda4afb 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -12,6 +12,7 @@ test_tool_sources = [
   'test-date.c',
   'test-delete-gpgsig.c',
   'test-delta.c',
+  'test-diff-process-backend.c',
   'test-dir-iterator.c',
   'test-drop-caches.c',
   'test-dump-cache-tree.c',
diff --git a/t/helper/test-diff-process-backend.c b/t/helper/test-diff-process-backend.c
new file mode 100644
index 0000000000..b0bb78d9f3
--- /dev/null
+++ b/t/helper/test-diff-process-backend.c
@@ -0,0 +1,349 @@
+/*
+ * Test process implementing the diff process protocol (diff.<driver>.process).
+ *
+ * Speaks the long-running process protocol over stdin/stdout and
+ * answers command=hunks-by-oid requests from the blob object names
+ * alone; no content is exchanged.  The --mode= switch selects the
+ * response shape:
+ *
+ *   oid-fixed         packet: git< hunk 5 2 5 2
+ *   oid-equal         packet: git< status=success (zero hunks: equivalent)
+ *   oid-need-content  packet: git< status=need-content
+ *   oid-empty         packet: git< hunk 0 0 1 2 (empty old side)
+ *
+ * and the adversarial shapes the protocol error paths are tested
+ * with:
+ *
+ *   oid-trailing        a hunk line with a trailing token to ignore
+ *   oid-malformed       a hunk line that does not parse
+ *   oid-huge            coordinates far past the end of any test blob
+ *   oid-erange          a count too large for any long
+ *   oid-overlap         two hunks out of order
+ *   oid-misaligned      two hunks whose unchanged runs differ in length
+ *   oid-badstart        a start of 0 paired with a nonzero count
+ *   oid-unknown-status  status=frobnicate
+ *   oid-abort           status=abort
+ *   oid-bare-status     a status packet without the hunk-section flush
+ *   oid-empty-packet    an empty packet (0004) inside the hunk section
+ *   oid-crash           one hunk line, then exit with no flush or status
+ *   oid-garbage         raw non-pkt-line bytes, then exit
+ *   cap-none            handshake announcing no capability at all
+ *
+ * Success responses end with:
+ *
+ *   packet:          git< 0000
+ *   packet:          git< status=success
+ *   packet:          git< 0000
+ *
+ * Each request is logged to --log as:
+ *
+ *   command=<cmd> pathname=<path> old-oid=<hex> new-oid=<hex>
+ */
+
+#include "test-tool.h"
+#include "pkt-line.h"
+#include "parse-options.h"
+#include "strbuf.h"
+
+static FILE *logfile;
+
+enum mode {
+	MODE_OID_FIXED,
+	MODE_OID_EQUAL,
+	MODE_OID_NEED_CONTENT,
+	MODE_OID_EMPTY,
+	MODE_OID_TRAILING,
+	MODE_OID_MALFORMED,
+	MODE_OID_HUGE,
+	MODE_OID_ERANGE,
+	MODE_OID_OVERLAP,
+	MODE_OID_MISALIGNED,
+	MODE_OID_BADSTART,
+	MODE_OID_UNKNOWN_STATUS,
+	MODE_OID_ABORT,
+	MODE_OID_BARE_STATUS,
+	MODE_OID_EMPTY_PACKET,
+	MODE_OID_CRASH,
+	MODE_OID_GARBAGE,
+	MODE_CAP_NONE,
+};
+
+static enum mode parse_mode(const char *s)
+{
+	if (!strcmp(s, "oid-fixed"))
+		return MODE_OID_FIXED;
+	if (!strcmp(s, "oid-equal"))
+		return MODE_OID_EQUAL;
+	if (!strcmp(s, "oid-need-content"))
+		return MODE_OID_NEED_CONTENT;
+	if (!strcmp(s, "oid-empty"))
+		return MODE_OID_EMPTY;
+	if (!strcmp(s, "oid-trailing"))
+		return MODE_OID_TRAILING;
+	if (!strcmp(s, "oid-malformed"))
+		return MODE_OID_MALFORMED;
+	if (!strcmp(s, "oid-huge"))
+		return MODE_OID_HUGE;
+	if (!strcmp(s, "oid-erange"))
+		return MODE_OID_ERANGE;
+	if (!strcmp(s, "oid-overlap"))
+		return MODE_OID_OVERLAP;
+	if (!strcmp(s, "oid-misaligned"))
+		return MODE_OID_MISALIGNED;
+	if (!strcmp(s, "oid-badstart"))
+		return MODE_OID_BADSTART;
+	if (!strcmp(s, "oid-unknown-status"))
+		return MODE_OID_UNKNOWN_STATUS;
+	if (!strcmp(s, "oid-abort"))
+		return MODE_OID_ABORT;
+	if (!strcmp(s, "oid-bare-status"))
+		return MODE_OID_BARE_STATUS;
+	if (!strcmp(s, "oid-empty-packet"))
+		return MODE_OID_EMPTY_PACKET;
+	if (!strcmp(s, "oid-crash"))
+		return MODE_OID_CRASH;
+	if (!strcmp(s, "oid-garbage"))
+		return MODE_OID_GARBAGE;
+	if (!strcmp(s, "cap-none"))
+		return MODE_CAP_NONE;
+	die("unknown --mode=%s", s);
+}
+
+/*
+ * Read "key=value" packets up to a flush, capturing "command" and
+ * "pathname".  Returns 1 if a request was read, 0 on EOF.
+ *
+ * The first packet uses the gentle variant so that a clean shutdown
+ * by Git (EOF) does not produce a spurious "the remote end hung up
+ * unexpectedly" on stderr.  Subsequent packets use the non-gentle
+ * variant: once inside a request, truncation is a protocol violation
+ * and dying loudly is the correct response.
+ */
+static int read_request_header(char **command, char **pathname,
+			       char **old_oid, char **new_oid)
+{
+	int first = 1;
+	char *line;
+
+	*command = *pathname = *old_oid = *new_oid = NULL;
+	for (;;) {
+		const char *value;
+
+		if (first) {
+			if (packet_read_line_gently(0, NULL, &line) < 0)
+				return 0;
+			first = 0;
+		} else {
+			line = packet_read_line(0, NULL);
+		}
+		if (!line)
+			break;
+		if (skip_prefix(line, "command=", &value))
+			*command = xstrdup(value);
+		else if (skip_prefix(line, "pathname=", &value))
+			*pathname = xstrdup(value);
+		else if (skip_prefix(line, "old-oid=", &value))
+			*old_oid = xstrdup(value);
+		else if (skip_prefix(line, "new-oid=", &value))
+			*new_oid = xstrdup(value);
+	}
+	return 1;
+}
+
+static void send_status(const char *status)
+{
+	packet_flush(1);
+	packet_write_fmt(1, "%s\n", status);
+	packet_flush(1);
+}
+
+static void command_loop(enum mode mode)
+{
+	for (;;) {
+		char *command = NULL, *pathname = NULL;
+		char *old_oid = NULL, *new_oid = NULL;
+
+		if (!read_request_header(&command, &pathname,
+					 &old_oid, &new_oid))
+			break; /* EOF: Git closed its end */
+
+		if (!command || strcmp(command, "hunks-by-oid"))
+			die("unexpected command: '%s'",
+			    command ? command : "(none)");
+
+		if (logfile) {
+			fprintf(logfile,
+				"command=%s pathname=%s old-oid=%s new-oid=%s\n",
+				command,
+				pathname ? pathname : "(none)",
+				old_oid ? old_oid : "(none)",
+				new_oid ? new_oid : "(none)");
+			fflush(logfile);
+		}
+
+		switch (mode) {
+		case MODE_OID_FIXED:
+			packet_write_fmt(1, "hunk 5 2 5 2\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_EQUAL:
+			send_status("status=success");
+			break;
+		case MODE_OID_EMPTY:
+			/*
+			 * An empty old side: the "git diff" convention
+			 * addresses it with a start of 0 and a count of 0.
+			 * Claims two lines added, fewer than the builtin
+			 * would show, so the answer is observable.
+			 */
+			packet_write_fmt(1, "hunk 0 0 1 2\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_TRAILING:
+			/*
+			 * Git must ignore trailing space-separated tokens
+			 * on a hunk line (the appendability rule), so this
+			 * must behave exactly like oid-fixed.
+			 */
+			packet_write_fmt(1, "hunk 5 2 5 2 moved=yes\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_MALFORMED:
+			packet_write_fmt(1, "hunk five two 5 2\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_HUGE:
+			/*
+			 * In-range for int32 (and for a 32-bit long), so
+			 * only the blob-size bound can reject it.
+			 */
+			packet_write_fmt(1, "hunk 1 1000000000 1 1000000000\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_ERANGE:
+			/* Overflows strtol() even where long is 64-bit. */
+			packet_write_fmt(1, "hunk 1 99999999999999999999 1 1\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_OVERLAP:
+			packet_write_fmt(1, "hunk 3 2 3 2\n");
+			packet_write_fmt(1, "hunk 2 2 2 2\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_MISALIGNED:
+			packet_write_fmt(1, "hunk 2 1 2 1\n");
+			packet_write_fmt(1, "hunk 5 1 6 1\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_BADSTART:
+			/*
+			 * A start of 0 names an empty side, so a nonzero
+			 * count beside it names no line; the coordinate is
+			 * rejected per pair while the process stays alive.
+			 */
+			packet_write_fmt(1, "hunk 0 2 1 2\n");
+			send_status("status=success");
+			break;
+		case MODE_OID_UNKNOWN_STATUS:
+			send_status("status=frobnicate");
+			break;
+		case MODE_OID_ABORT:
+			send_status("status=abort");
+			break;
+		case MODE_OID_BARE_STATUS:
+			/* No hunk-section flush: a protocol violation. */
+			packet_write_fmt(1, "status=success\n");
+			packet_flush(1);
+			break;
+		case MODE_OID_EMPTY_PACKET:
+			/*
+			 * An empty packet is not a flush; inside the hunk
+			 * section it is a protocol violation.
+			 */
+			if (write(1, "0004", 4) < 0)
+				die_errno("write empty packet");
+			send_status("status=success");
+			break;
+		case MODE_OID_CRASH:
+			packet_write_fmt(1, "hunk 5 2 5 2\n");
+			exit(0);
+		case MODE_OID_GARBAGE:
+			if (write(1, "@@@@ not a pkt-line @@@@", 24) < 0)
+				die_errno("write garbage");
+			exit(0);
+		default:
+			send_status("status=need-content");
+			break;
+		}
+
+		free(command);
+		free(pathname);
+		free(old_oid);
+		free(new_oid);
+	}
+}
+
+static void handshake(enum mode mode)
+{
+	char *line;
+
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "git-diff-client"))
+		die("bad welcome: '%s'", line ? line : "(eof)");
+	line = packet_read_line(0, NULL);
+	if (!line || strcmp(line, "version=1"))
+		die("bad version: '%s'", line ? line : "(eof)");
+	if (packet_read_line(0, NULL))
+		die("expected flush after version");
+
+	packet_write_fmt(1, "git-diff-server\n");
+	packet_write_fmt(1, "version=1\n");
+	packet_flush(1);
+
+	/* Drain capabilities advertised by Git */
+	while ((line = packet_read_line(0, NULL)))
+		; /* drain */
+
+	if (mode != MODE_CAP_NONE)
+		packet_write_fmt(1, "capability=hunks-by-oid\n");
+	packet_flush(1);
+}
+
+static const char *const usage_str[] = {
+	"test-tool diff-process-backend --mode=<mode> [--log=<path>]",
+	NULL
+};
+
+int cmd__diff_process_backend(int argc, const char **argv)
+{
+	const char *mode_str = NULL, *log_path = NULL;
+	enum mode mode = MODE_OID_FIXED;
+	struct option options[] = {
+		OPT_STRING(0, "mode", &mode_str, "mode",
+			   "response shape (default oid-fixed);"
+			   " see the file header for the full list of modes"),
+		OPT_STRING(0, "log", &log_path, "path",
+			   "append per-request summary to this file"),
+		OPT_END()
+	};
+
+	argc = parse_options(argc, argv, NULL, options, usage_str, 0);
+	if (argc)
+		usage_with_options(usage_str, options);
+
+	if (mode_str)
+		mode = parse_mode(mode_str);
+
+	if (log_path) {
+		logfile = fopen(log_path, "a");
+		if (!logfile)
+			die_errno("failed to open log '%s'", log_path);
+	}
+
+	handshake(mode);
+	command_loop(mode);
+
+	if (logfile && fclose(logfile))
+		die_errno("error closing log");
+	return 0;
+}
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index b71a22b43b..3c3f95269c 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -22,6 +22,7 @@ static struct test_cmd cmds[] = {
 	{ "date", cmd__date },
 	{ "delete-gpgsig", cmd__delete_gpgsig },
 	{ "delta", cmd__delta },
+	{ "diff-process-backend", cmd__diff_process_backend },
 	{ "dir-iterator", cmd__dir_iterator },
 	{ "drop-caches", cmd__drop_caches },
 	{ "dump-cache-tree", cmd__dump_cache_tree },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f2885b33d5..a5bb755516 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -15,6 +15,7 @@ int cmd__csprng(int argc, const char **argv);
 int cmd__date(int argc, const char **argv);
 int cmd__delta(int argc, const char **argv);
 int cmd__delete_gpgsig(int argc, const char **argv);
+int cmd__diff_process_backend(int argc, const char **argv);
 int cmd__dir_iterator(int argc, const char **argv);
 int cmd__drop_caches(int argc, const char **argv);
 int cmd__dump_cache_tree(int argc, const char **argv);
diff --git a/t/meson.build b/t/meson.build
index 3f45b09dd6..a76edfbf81 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -518,6 +518,7 @@ integration_tests = [
   't4072-diff-max-depth.sh',
   't4073-diff-stat-name-width.sh',
   't4074-diff-shifted-matched-group.sh',
+  't4080-diff-process.sh',
   't4100-apply-stat.sh',
   't4101-apply-nonl.sh',
   't4102-apply-rename.sh',
diff --git a/t/t4080-diff-process.sh b/t/t4080-diff-process.sh
new file mode 100755
index 0000000000..a8efa19416
--- /dev/null
+++ b/t/t4080-diff-process.sh
@@ -0,0 +1,593 @@
+#!/bin/sh
+
+test_description='diff.<driver>.process: oid-only hunk requests'
+
+TEST_PASSES_SANITIZE_LEAK=true
+. ./test-lib.sh
+
+# See t/helper/test-diff-process-backend.c for the process implementation
+# and available --mode= options.
+
+BACKEND="test-tool diff-process-backend"
+
+test_expect_success 'setup' '
+	echo "*.c diff=cdiff" >.gitattributes &&
+	git add .gitattributes &&
+
+	# 10 lines, changes at 5-6 and 9-10 between the two commits.
+	cat >pair.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	original5
+	original6
+	line7
+	line8
+	line9
+	line10
+	EOF
+	git add pair.c &&
+	git commit -m "add pair.c" &&
+
+	cat >pair.c <<-\EOF &&
+	line1
+	line2
+	line3
+	line4
+	changed5
+	changed6
+	line7
+	line8
+	changed9
+	changed10
+	EOF
+	git add pair.c &&
+	git commit -m "change pair.c"
+'
+
+test_expect_success 'an oid-capable process answers blame by object names alone' '
+	test_when_finished "rm -f backend.log" &&
+	ORIG=$(git rev-parse --short HEAD~1) &&
+	CHANGE=$(git rev-parse --short HEAD) &&
+	# The process reports only lines 5-6 as changed, so blame attributes
+	# lines 9-10 to the original commit even though the builtin diff
+	# would show them as changed.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		blame pair.c >actual &&
+	sed -n "9p" actual >line9 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "$ORIG" line9 &&
+	test_grep "$ORIG" line10 &&
+	sed -n "5p" actual >line5 &&
+	test_grep "$CHANGE" line5 &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'an oid-capable process answers --numstat by object names alone' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		log -1 --format= --numstat -- pair.c >actual &&
+	printf "2\t2\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'need-content falls through to the builtin diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-need-content --log=backend.log" \
+		log -1 --format= --numstat -- pair.c >actual &&
+	printf "4\t4\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'a warmed hunk store does not override process hunks' '
+	test_when_finished "git diff-hunks clear" &&
+	ORIG=$(git rev-parse --short HEAD~1) &&
+	GIT_DIFF_HUNKS_WRITE=1 git log -2 --stat -- pair.c >/dev/null &&
+
+	# Control: without a process, blame is served from the store.
+	git blame --show-stats pair.c >stats &&
+	test_grep "num precomputed hits: 1" stats &&
+
+	# The store holds the builtin hunks, but a process-capable driver
+	# makes the process authoritative, so blame must reflect the
+	# process hunks (only lines 5-6), not a store hit.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed" \
+		blame pair.c >actual &&
+	sed -n "9p" actual >line9 &&
+	test_grep "$ORIG" line9
+'
+
+test_expect_success 'a worktree side is not asked by object names' '
+	test_when_finished "rm -f backend.log && git checkout -- pair.c" &&
+	echo "worktree edit" >>pair.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		diff --numstat -- pair.c >actual &&
+	printf "1\t0\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process bypassed by --no-ext-diff' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		log -1 --format= --numstat --no-ext-diff -- pair.c >actual &&
+	printf "4\t4\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'format-patch keeps its diffstat off the process' '
+	test_when_finished "rm -f backend.log" &&
+	# format-patch emits a diffstat, and a diffstat consults the
+	# process, but the gate keeps it builtin so a generated patch
+	# applies for recipients without the process.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		format-patch -1 --stdout --stat -- pair.c >actual &&
+	test_grep "^+changed9" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'format-patch --ext-diff keeps its diffstat off the process' '
+	test_when_finished "rm -f backend.log" &&
+	# The gate holds even when --ext-diff enables the external command.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		format-patch -1 --stdout --ext-diff --stat -- pair.c >actual &&
+	test_grep "^+changed9" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'diff process not consulted by plumbing diff commands' '
+	test_when_finished "rm -f backend.log && git checkout -f HEAD -- pair.c" &&
+	# diff-tree diffs the two commits, a real pair a defeated gate would
+	# consult the process for.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		diff-tree --numstat HEAD >actual &&
+	test_grep "pair.c" actual &&
+	# diff-index needs a change to diff, or there is no pair and a
+	# missing log proves nothing; stage one and diff it against HEAD.
+	echo "staged change" >>pair.c &&
+	git add pair.c &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		diff-index --cached --numstat HEAD -- pair.c >actual &&
+	test_grep "pair.c" actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'add -p stages from the builtin diff with a process configured' '
+	test_when_finished "rm -f backend.log" &&
+	cat >gate.c <<-\EOF &&
+	int gate(void) { return 1; }
+	EOF
+	git add gate.c &&
+	git commit -m "add gate.c" &&
+	cat >gate.c <<-\EOF &&
+	int gate(void) { return 2; }
+	EOF
+	# add -p builds its hunks from patch text, which is not a provider
+	# consumer today, so a configured process cannot shape what it
+	# offers. This pins that interactive patch stays builtin for the
+	# current consumers, rather than exercising the plumbing gate.
+	test_write_lines y |
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		add -p gate.c &&
+	git diff --cached -- gate.c >staged &&
+	test_grep "return 2" staged &&
+	test_path_is_missing backend.log &&
+	git commit -m "gate.c v2"
+'
+
+test_expect_success 'blame withholds identity for the working-tree pair' '
+	test_when_finished "rm -f backend.log && git checkout -- pair.c" &&
+	echo "uncommitted" >>pair.c &&
+	wt=$(git hash-object pair.c) &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		blame pair.c >actual &&
+	# The dirty working-tree side is not a stored blob: no request
+	# may name its bytes by object id.
+	test_grep ! "new-oid=$wt" backend.log
+'
+
+test_expect_success 'a replaced blob makes the process step aside' '
+	new_blob=$(git rev-parse HEAD:pair.c) &&
+	test_when_finished "rm -f backend.log && git replace -d $new_blob" &&
+	# Replacing the new-side blob redirects the content the diff reads
+	# under the id the process would be sent, so a raw-id request would
+	# name bytes other than the ones diffed. Identity is withheld: the
+	# process is not consulted and the builtin computes the pair from
+	# the replaced content.
+	repl=$(printf "just one line\n" | git hash-object -w --stdin) &&
+	git replace "$new_blob" "$repl" &&
+	git log -1 --format= --numstat -- pair.c >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		log -1 --format= --numstat -- pair.c >actual &&
+	test_cmp expect actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'an equivalence answer omits the pair from the stat' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-equal --log=backend.log" \
+		log -1 --format= --numstat -- pair.c >actual &&
+	# An equivalent pair sums to a zero-count entry, and the stat
+	# code omits zero-count modified entries, the same way a
+	# whitespace-only pair prints nothing under -w. The builtin
+	# diff would print nonzero counts here, and the log proves the
+	# process was consulted.
+	test_must_be_empty actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'blame passes equivalent pairs through to the boundary' '
+	ORIG=$(git rev-parse --short ":/add pair.c") &&
+	# The process asserts every consulted pair equal, so no line is
+	# ever treated as changed: every line passes through to the
+	# commit that added the file, marked as the blame boundary.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-equal" \
+		blame pair.c >actual &&
+	sed -n "5p" actual >line5 &&
+	test_grep "^\^$ORIG" line5 &&
+	sed -n "10p" actual >line10 &&
+	test_grep "^\^$ORIG" line10
+'
+
+test_expect_success 'a warming run records a pair the process defers' '
+	test_when_finished "git diff-hunks clear" &&
+	git diff-hunks clear &&
+	# The process owns the path but defers this pair with
+	# need-content, so the pair gets the builtin diff; that is the
+	# result the store holds, so the warming run records it and a
+	# later read is served from the store.
+	GIT_DIFF_HUNKS_WRITE=1 git -c core.diffHunks=true \
+		-c diff.cdiff.process="$BACKEND --mode=oid-need-content" \
+		log -1 --format= --stat -- pair.c >/dev/null &&
+	git -c core.diffHunks=true blame --show-stats pair.c >stats 2>&1 &&
+	test_grep "num precomputed hits: 1" stats
+'
+
+# The protocol error paths: each adversarial response shape must warn,
+# fall back to the builtin output, and either keep the process alive
+# (a per-pair rejection) or disable it for the rest of the command (a
+# protocol error).  The request log tells the two apart: the log walk
+# below consults two pairs (gate.c first, then pair.c), so a disabled
+# process shows one logged request and a live one shows two.
+
+test_expect_success 'a malformed hunk line disables the process for the command' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-malformed --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "disabling it for the remainder" err &&
+	test_line_count = 1 backend.log
+'
+
+test_expect_success 'coordinates past the blob size skip the pair, process stays alive' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-huge --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "past the end" err &&
+	test_line_count = 2 backend.log
+'
+
+test_expect_success 'a count that overflows long skips the pair, process stays alive' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-erange --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "out-of-range coordinates" err &&
+	test_line_count = 2 backend.log
+'
+
+test_expect_success 'overlapping hunks are rejected per pair' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-overlap --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "overlapping hunks" err &&
+	test_line_count = 2 backend.log
+'
+
+test_expect_success 'misaligned hunks are rejected per pair' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-misaligned --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "misaligned" err &&
+	test_line_count = 2 backend.log
+'
+
+test_expect_success 'a start of zero with a nonzero count is rejected per pair' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	# A start of 0 names an empty side, so a nonzero count beside it
+	# names no line; the coordinate is rejected and the pair falls back
+	# to the builtin diff while the process stays alive.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-badstart --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "out-of-range coordinates" err &&
+	test_line_count = 2 backend.log
+'
+
+test_expect_success 'an unrecognized status disables the process for the command' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-unknown-status --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "unrecognized status .frobnicate." err &&
+	test_line_count = 1 backend.log
+'
+
+test_expect_success 'status=abort withdraws the capability without a warning' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-abort --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep ! "disabling" err &&
+	test_line_count = 1 backend.log
+'
+
+test_expect_success 'a bare status without the hunk-section flush is a protocol error' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-bare-status --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "disabling it for the remainder" err &&
+	test_line_count = 1 backend.log
+'
+
+test_expect_success 'an empty packet in the hunk section is a protocol error' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-empty-packet --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "disabling it for the remainder" err &&
+	test_line_count = 1 backend.log
+'
+
+test_expect_success 'a process that dies mid-response fails the command over to builtin' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-crash --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "disabling it for the remainder" err &&
+	test_line_count = 1 backend.log
+'
+
+test_expect_success 'garbage bytes on stdout fail the command over to builtin' '
+	test_when_finished "rm -f backend.log err" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-garbage --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual 2>err &&
+	test_cmp expect actual &&
+	test_grep "disabling it for the remainder" err &&
+	test_line_count = 1 backend.log
+'
+
+test_expect_success 'a process announcing no capability is never asked' '
+	test_when_finished "rm -f backend.log" &&
+	git log --format= --numstat -- "*.c" >expect &&
+	git -c diff.cdiff.process="$BACKEND --mode=cap-none --log=backend.log" \
+		log --format= --numstat -- "*.c" >actual &&
+	test_cmp expect actual &&
+	test_must_be_empty backend.log
+'
+
+test_expect_success 'a trailing token on a hunk line is ignored' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-trailing --log=backend.log" \
+		log -1 --format= --numstat -- pair.c >actual &&
+	printf "2\t2\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'a failed start warns once and the store may serve the path' '
+	git init failrepo &&
+	(
+		cd failrepo &&
+		echo "*.c diff=cdiff" >.gitattributes &&
+		git add .gitattributes &&
+		test_commit f1 f.c "one" &&
+		test_commit f2 f.c "one
+two" &&
+		test_commit f3 f.c "one
+two
+three" &&
+		GIT_DIFF_HUNKS_WRITE=1 git log --format= --stat -- f.c >/dev/null &&
+		# The command has no shell metacharacters, so it fails at
+		# exec time; a shell-wrapped command would fail at the
+		# handshake, which the gentle handshake in the base
+		# (061a68e443) likewise degrades to the builtin diff.
+		git blame f.c >expect &&
+		git -c diff.cdiff.process=/does-not-exist-diff-backend \
+			blame f.c >actual 2>err &&
+		test_cmp expect actual &&
+		# One warning even though the blame consults two pairs.
+		test $(grep -c "failed to start" err) = 1 &&
+		# A failure is a non-answer like any other: the request
+		# that observes it and every later request pass to the
+		# store, so the warmed store serves both pairs.
+		git -c diff.cdiff.process=/does-not-exist-diff-backend \
+			blame --show-stats f.c >stats 2>&1 &&
+		test_grep "num precomputed hits: 2" stats
+	)
+'
+
+test_expect_success 'the store serves a pair the process defers' '
+	test_when_finished "git diff-hunks clear" &&
+	git diff-hunks clear &&
+	GIT_DIFF_HUNKS_WRITE=1 git log --format= --stat -- pair.c >/dev/null &&
+	git blame pair.c >expect &&
+	# need-content defers the pair to the builtin diff, which is
+	# what the store holds, so the walk continues past the process
+	# and the store serves the pair.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-need-content" \
+		blame pair.c >actual &&
+	test_cmp expect actual &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-need-content" \
+		blame --show-stats pair.c >stats 2>&1 &&
+	test_grep "num precomputed hits: 1" stats
+'
+
+test_expect_success 'git diff between commits consults the process' '
+	test_when_finished "rm -f backend.log" &&
+	ORIG=$(git rev-parse ":/add pair.c") &&
+	CHANGE=$(git rev-parse ":/change pair.c") &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		diff --numstat $ORIG $CHANGE -- pair.c >actual &&
+	printf "2\t2\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'git show consults the process' '
+	test_when_finished "rm -f backend.log" &&
+	CHANGE=$(git rev-parse ":/change pair.c") &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		show --format= --numstat $CHANGE -- pair.c >actual &&
+	printf "2\t2\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'diff-tree --ext-diff consults the process' '
+	test_when_finished "rm -f backend.log" &&
+	CHANGE=$(git rev-parse ":/change pair.c") &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		diff-tree --ext-diff --no-commit-id --numstat $CHANGE >actual &&
+	printf "2\t2\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success '--no-diff-process forbids consulting alone' '
+	test_when_finished "rm -f backend.log" &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		log -1 --no-diff-process --format= --numstat -- pair.c >actual &&
+	printf "4\t4\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success '--diff-process allows plumbing to consult' '
+	test_when_finished "rm -f backend.log" &&
+	CHANGE=$(git rev-parse ":/change pair.c") &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		diff-tree --diff-process --no-commit-id --numstat $CHANGE >actual &&
+	printf "2\t2\tpair.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=pair.c" backend.log
+'
+
+test_expect_success 'a forced blame diff algorithm bypasses the process' '
+	test_when_finished "rm -f backend.log" &&
+	CHANGE=$(git rev-parse --short ":/change pair.c") &&
+	# The process would attribute lines 9-10 to the original commit
+	# (see the oid-fixed blame test above); a forced builtin
+	# algorithm must produce the builtin attribution and never start
+	# the process.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		blame --histogram pair.c >actual &&
+	sed -n "9p" actual >line9 &&
+	test_grep "$CHANGE" line9 &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'textconv output is never identified to the process' '
+	test_when_finished "rm -f backend.log" &&
+	echo "*.tcv diff=tcv" >>.gitattributes &&
+	git add .gitattributes &&
+	git commit -m tcv-attr &&
+	test_config diff.tcv.textconv cat &&
+	test_commit tcv1 file.tcv "alpha" &&
+	test_commit tcv2 file.tcv "alpha
+beta" &&
+	git blame file.tcv >expect &&
+	git -c diff.tcv.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		blame file.tcv >actual &&
+	test_cmp expect actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'a gitlink side is never identified to the process' '
+	test_when_finished "rm -f backend.log" &&
+	echo "sub diff=cdiff" >>.gitattributes &&
+	git add .gitattributes &&
+	git commit -m sub-attr &&
+	C1=$(git rev-parse HEAD) &&
+	C2=$(git rev-parse HEAD~1) &&
+	git update-index --add --cacheinfo 160000,$C1,sub &&
+	git commit -m sub-1 &&
+	git update-index --add --cacheinfo 160000,$C2,sub &&
+	git commit -m sub-2 &&
+	git -c diff.cdiff.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+		log -1 --format= --numstat -- sub >actual &&
+	printf "1\t1\tsub\n" >expect &&
+	test_cmp expect actual &&
+	test_path_is_missing backend.log
+'
+
+test_expect_success 'a relative diff consults by the repo-relative path' '
+	test_when_finished "rm -f backend.log" &&
+	echo "reldir/*.rel diff=rdrv" >>.gitattributes &&
+	git add .gitattributes &&
+	git commit -m rel-attr &&
+	mkdir reldir &&
+	test_write_lines line1 line2 line3 line4 original5 original6 \
+		line7 line8 line9 line10 >reldir/x.rel &&
+	git add reldir/x.rel &&
+	git commit -m "add x.rel" &&
+	test_write_lines line1 line2 line3 line4 changed5 changed6 \
+		line7 line8 changed9 changed10 >reldir/x.rel &&
+	git add reldir/x.rel &&
+	git commit -m "change x.rel" &&
+	# diff.relative strips the prefix from the displayed name; the
+	# driver lookup and the request pathname must still use the
+	# repo-relative path, or the directory-scoped attribute above
+	# would not match and the process would never be consulted.
+	# The process runs at the repository root, so its log lands there.
+	(
+		cd reldir &&
+		git -c diff.relative=true \
+			-c diff.rdrv.process="$BACKEND --mode=oid-fixed --log=backend.log" \
+			log -1 --format= --numstat -- x.rel
+	) >actual &&
+	printf "2\t2\tx.rel\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=reldir/x.rel" backend.log
+'
+
+test_expect_success 'an empty file side is answered with a start of zero' '
+	test_when_finished "rm -f backend.log" &&
+	>empty.c &&
+	git add empty.c &&
+	git commit -m "add empty.c" &&
+	printf "x\ny\nz\n" >empty.c &&
+	git add empty.c &&
+	git commit -m "fill empty.c" &&
+	# The process addresses the empty old side with a start of 0 and a
+	# count of 0, and claims two of the three added lines. The answer is
+	# used as sent, so the stat shows the two lines it named, not the
+	# three the builtin would.
+	git -c diff.cdiff.process="$BACKEND --mode=oid-empty --log=backend.log" \
+		log -1 --format= --numstat -- empty.c >actual &&
+	printf "2\t0\tempty.c\n" >expect &&
+	test_cmp expect actual &&
+	test_grep "command=hunks-by-oid pathname=empty.c" backend.log
+'
+
+test_done
diff --git a/xdiff-interface.h b/xdiff-interface.h
index 71e5dffefb..23db1d2a46 100644
--- a/xdiff-interface.h
+++ b/xdiff-interface.h
@@ -6,6 +6,18 @@
 
 struct object_database;
 
+/*
+ * Hunk descriptor for externally computed diffs, in xdiff's own
+ * coordinates: line numbers are 1-based and a hunk's start is the
+ * first line it covers.  A caller translates any external "empty side"
+ * idiom (such as git diff's start-0/count-0) to a 1-based start before
+ * storing hunks in this struct.
+ */
+struct xdl_hunk {
+	long old_start, old_count;
+	long new_start, new_count;
+};
+
 /*
  * xdiff isn't equipped to handle content over a gigabyte;
  * we make the cutoff 1GB - 1MB to give some breathing
-- 
2.54.0